By admin 
At the recent VB Transform 2026 conference, Ryan Phillips, Head of Enterprise Product at Runway ML, offered a compelling glimpse into the innovative strategies that power the company’s cutting-edge generative AI tools, arguing that these insights hold profound relevance for virtually any organization navigating the complexities of AI development and deployment. Runway, an applied AI research firm dedicated to building general world models that fuel generative applications, has consistently pushed the boundaries of what’s possible. Phillips’ presentation, "Demystifying Evals, Model Training, and Infrastructure Details," not only showcased Runway Characters, a revolutionary real-time video generation model capable of zero-latency avatar interactions, but also provided a detailed roadmap for building, evaluating, and shipping AI models effectively.
"I think even if you are not all building models yourselves, it’s helpful to learn how we do it because I think almost all of the lessons are applicable to what you all are doing day-to-day," Phillips emphasized, underscoring the universal applicability of Runway’s methodologies. He painted a vivid picture of the rapid evolution in AI-powered video creation, contrasting the painstaking, hundreds-of-hours process of stitching individual frames for low-framerate, illegible text videos just five years ago, with today’s on-the-fly, interactive video generation powered by Runway’s advanced models. "Studying how we build these real-time models can inspire how you build and deploy real-time experiences, whether agentic or not, in your companies today," he added.
A cornerstone of Runway’s success, as highlighted by Phillips, lies in its meticulous approach to model evaluation. Building a robust AI product, he stressed, begins with a high-quality evaluation set, but this crucial step transcends mere engineering. It necessitates deep, cross-functional alignment, bringing together product, design, research, and sales teams to collaboratively define what "quality" truly means in the context of AI-generated content. Runway achieves this through internal workshops where team members collectively review generated examples, fostering a unified organizational understanding of specific failure modes and, consequently, a shared definition of a successful generation. "We spent a lot of time working with our team, running through examples… of what success and failure looks like, down to the very, very detailed and picky things," Phillips recounted.
This rigorous internal alignment ensures that the evaluation set is not only comprehensive in covering broad customer use cases but also robust enough to capture extreme edge cases. To illustrate, Phillips cited the use of "Tooth," a non-human character with unusual facial features – no nose and distinctively shaped teeth – to test the model’s behavior when pushed beyond conventional human facial structures. Such deliberate testing ensures predictable performance even in unconventional scenarios. The evaluation process itself scrutinizes subtle artifacts. Phillips shared an example where a video was strictly graded as a failure because, while the character’s face remained intact, background elements like a net began to morph inexplicably. Despite the sophisticated nature of their AI, Runway employs a surprisingly straightforward tool for tracking these evaluations: an Excel spreadsheet. Daily tests are logged, categorizing outputs as "minor" or "major" failures against a pre-determined pass rate. "We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model," Phillips stated. "So it’s not magical."
Recognizing that manual evaluation at scale can be a significant bottleneck for enterprise developers grappling with non-deterministic quality drift in their own real-time pipelines, Phillips pointed towards leveraging language models (LLMs) for automating the visual grading process. "LLMs are getting quite good at being a judge for a lot of this content, especially the types of morphing or changing that you would see in an evaluation set," he observed. Furthermore, teams can equip LLMs with behind-the-scenes context, such as a hand-drawn sketch or an ad’s structural layout, to guide generation and validation processes, thereby ensuring quality without imposing additional cognitive load on the end user. This approach democratizes quality assurance, making it more scalable and efficient.
The journey from a massive, resource-intensive foundation model to a real-time generative video product is paved with intricate engineering and optimization. Runway’s process begins with the pre-training of these large foundation models, which, while powerful, are inherently slow to generate outputs. To bridge this latency gap and achieve real-time performance, Runway employs a technique called "distillation." This involves training a smaller, faster "student" model to accurately mimic the behavior and outputs of the larger "teacher" model. Phillips indicated that distillation alone can reduce generation time by "80 to 90%."
Following distillation, the team applies Adversarial Post-Training (APT) to the distilled model. This advanced technique continuously refines the model by pitting it against a system specifically designed to identify and exploit its weaknesses. APT is crucial for recapturing the visual sharpness that can sometimes be lost during the distillation process, ensuring that the generated content is not only fast but also high in fidelity.
However, the inherent complexity of modifying model architectures inevitably introduces new challenges. During the distillation and APT phases, Runway encountered a persistent bug: AI-generated characters would exhibit a noticeable sway or drift away from the center of the frame during real-time generation. This was a significant hurdle that consumed weeks of effort as the team attempted to engineer a backend patch to resolve the core model’s instability. The breakthrough came not from a complex algorithmic fix, but from an astute observation during their rigorous evaluation process. They discovered that if the user’s initial input image was perfectly centered, the generated video remained stable. Instead of continuing to invest more resources into a potentially elusive backend engineering solution, Runway ingeniously pivoted to a user experience-centric approach.
"What we did was, when we noticed this in our evaluations, we then said, ‘What if we just offered that as a feature?’ If a user gives us a character that is turned to the left, we know the video is going to morph. Let’s just fix it for them," Phillips explained. This led to the development of a front-end feature aptly named "Optimize for Image Quality." This feature automatically re-centers the user’s input image before the generation process commences. By strategically framing a backend model limitation as a user-friendly feature, Runway transformed a potential flaw into a perceived enhancement, thereby improving the user experience without compromising on the underlying technology. Phillips’ advice to other companies is clear: "Turn model limitations into product features so that you can actually expand how the model works. It might feel like a limitation internally, but your customers will not see it that way if you’re kind of building this in as a product feature." This philosophy underscores the importance of user-centric design in the AI product lifecycle.
The successful global delivery of video at a consistent 24 frames per second (fps) hinges on optimizing every facet of the infrastructure stack. This involves a multi-layered approach, encompassing techniques such as intelligent caching, parallel decoding, and even deep kernel-level modifications in collaboration with hardware partners like Nvidia. Runway’s commitment to performance was tested shortly after the launch of Runway Characters when they observed that approximately 8% of API calls were experiencing a significant drop to 16 fps, resulting in noticeable stuttering for users.
Unraveling the root cause of such performance degradation demanded deep observability into the system’s operations. The team deployed a sophisticated debugging strategy, employing an AI agent powered by Claude in conjunction with established monitoring tools like Datadog and Sentry. This comprehensive diagnostic effort allowed them to trace the anomaly with precision, ultimately isolating the problem to a specific data center within the us-east-1 region. The solution, remarkably, did not involve intricate software adjustments or configuration changes. "The solution actually wasn’t [to] go fix anything or change a config," Phillips elaborated. "They actually went and physically replaced those GPUs in the data center to fix it, and that ultimately solved the problems." This anecdote serves as a potent reminder for enterprise teams deploying real-time applications: hardware and infrastructure anomalies can have a direct and substantial impact on model performance, necessitating rigorous, full-stack debugging capabilities. "Don’t forget about all the small details, because there’s so many of them when you’re deploying these models," Phillips cautioned.
The development of advanced AI systems is rarely a linear or predictable path. Teams often find themselves entrenched in what Phillips aptly termed "failure hell" – extended periods where they are grappling with a single, intractable problem with no clear resolution in sight. "We think you have to go through that pain and really struggle with the problem for a little bit before you can get the breakthrough," he asserted. This period of intense struggle, however, is often a precursor to rapid, exponential improvement. As underlying models surmount these technical challenges, the landscape of enterprise creative work undergoes a fundamental transformation. Traditionally, marketing and design teams focused on the creation of singular assets, such as a specific advertisement or illustration. In an era characterized by real-time generation and increasingly sophisticated agentic workflows, this paradigm is shifting towards a more strategic role: defining parameters, establishing aesthetic guidelines, and safeguarding intellectual property. "You might not be designing a single ad, but you might design a world that then the agent or a real-time video model can generate ads from," Phillips concluded, painting a vision of a future where creatives become architects of generative universes, empowering AI to execute on their vision at an unprecedented scale. This evolution signifies a move from micro-creation to macro-design, where the emphasis is on setting the stage for AI to perform.
