By Titik Puspa 
In the sprawling industrial landscape of San Leandro, California, a company named Encord is quietly orchestrating a revolution in the nascent field of physical artificial intelligence. Far from the sterile server farms and abstract algorithms that define much of the AI discourse, Encord’s headquarters is a bustling warehouse, a veritable playground for robotic training data. Here, amidst towering shelves stocked with everything from fake flowers to bags of wires, a team of human "pilots," as Encord calls its robotic trainers, are engaged in a meticulous, often precarious, dance with the physical world. Their mission: to generate the high-fidelity, real-world data that is becoming the ultimate bottleneck in the quest for truly capable humanoid and warehouse robots.
At the heart of this operation, Andrew Ceja, a seasoned Encord pilot, is engaged in a delicate balancing act with a Jenga tower. His task, seemingly simple, is a proxy for the complex manipulation skills robots need to acquire. What sets Ceja’s endeavor apart, however, is the sophisticated array of technology he wears. A headset, equipped with cameras to meticulously track his every gaze and movement, is also fitted with an array of sensors designed to capture his brain waves. This integration of human cognitive data with physical actions represents a bold leap into a new paradigm of robot training, aiming to imbue AI models with a deeper understanding of human intent, error, and surprise.
Encord, along with a growing cohort of ambitious startups, is betting that the next significant hurdle in robotics won’t be the elegance of an AI model’s architecture, but rather the sheer scarcity of authentic, real-world physical training data. These companies are shifting from merely managing existing data to actively manufacturing the data that doesn’t yet exist. "The data simply does not exist," is a recurring refrain from Vineeth Velmurugan, Encord’s head of robot learning, a veteran with experience at OpenAI’s robot lab and the warehouse automation firm Berkshire Grey. Velmurugan joined Encord specifically to spearhead the company’s internal data-creation initiatives, recognizing that the explosion of interest in end-to-end learning for robotic manipulation tasks had outstripped the available training datasets.
The Jenga game, while illustrative, is just one facet of Encord’s data generation efforts. The company is at the vanguard of exploring novel data modalities, including the integration of brainwave data, in collaboration with Zander Labs, a German neuroscience startup. Zander Labs posits that by measuring brain activity, they can deduce crucial mental states such as error detection, intention, and surprise. This data, when correlated with physical actions, promises to create a richer, more nuanced dataset for training AI models. Encord’s current work with Zander is a pilot program, with the ultimate goal of creating an initial dataset tagged with brainwave data. This dataset will then be tested against customer robotics models to rigorously evaluate whether it demonstrably improves performance, a crucial step before considering any large-scale deployment.
Lucas Gehrke, a neuroscientist from Zander Labs overseeing the trials, elaborates on the significance of this approach. He explains that the intensity of brain activity during a task can provide invaluable insights for model builders, helping them to determine when a robot might need to engage its most computationally intensive or "highest-effort" models. This cognitive-aware data can lead to more efficient and adaptable robotic systems, capable of dynamically adjusting their processing power based on the perceived complexity or cognitive load of a task.
This multi-modal data generation strategy positions Encord at the "bleeding edge" of tackling the robotics data bottleneck. The company’s genesis lies in assisting companies building machine-vision applications with data annotation and model evaluation. However, as their clientele – a constellation of leading robotics firms that Velmurugan is authorized to speak about only in general terms – began to embrace end-to-end learning for complex robotic manipulation, a critical realization emerged: they needed to become producers of data, not just consumers or managers.
The aspiration to replicate the generative AI revolution seen in large language models (LLMs) for robotics faces a persistent obstacle: the fundamental difference in data acquisition. While LLMs could be trained by scraping vast quantities of text from the internet at minimal cost, generating physical training data for robots is an inherently expensive and labor-intensive undertaking. Self-driving car companies, for instance, have long collected their own physical-world data, but scaling this process is a formidable challenge. Training solely from video, while useful, often lacks the precise fidelity and depth of information captured by direct physical interaction. Velmurugan estimates that a breakthrough in robot learning might require a dataset five times the size of YouTube’s entire video corpus, a staggering scale that underscores why data generation has evolved from a research problem into a significant business opportunity.
The core of Encord’s data generation strategy revolves around two primary sources: "egocentric" video, captured by workers wearing cameras during their tasks, and data from robots operated remotely. Egocentric data offers a first-person perspective, mimicking how a robot might perceive its environment. This is often augmented with additional camera angles and other sensor data to provide a more comprehensive view. Encord draws this egocentric data from numerous factories across the globe. However, their San Leandro facility serves as an experimental hub, a place to push the boundaries by exploring new data modalities like brain waves or to meticulously collect datasets for specific skills that require fine-tuning.
During a recent visit by TechCrunch, pilots were utilizing "leader-follower rigs." These setups involve paired robotic arms, where one is directly controlled by a human operator, and the other meticulously mimics its movements. This allows for the generation of data for tasks such as pouring liquids – a notoriously "sloshy" and challenging maneuver for robots – and the precise stacking of objects like poker chips. "Every humanoid company has asked us for these pieces," Velmurugan states, highlighting the immediate and pressing demand for such fine-grained manipulation data.
The warehouse itself is a testament to the diverse and often mundane objects that form the building blocks of robotic competence. Storage racks held an eclectic inventory: cartons of fake flowers in vases, stacks of books, an assortment of plastic fruits and vegetables, kitty litter trays and scoops, and bundles of wires. This seemingly disparate collection represents the "stock in trade" for training robotic manipulators to perform a wide range of household and industrial tasks.
At one of these meticulously arranged stations, another pilot, Sofia Infante, was demonstrating the intricate process of plugging and unplugging Ethernet cables from the back of a server. This is precisely the kind of task that data center operators would be thrilled to automate, provided robots could achieve the necessary dexterity and precision. Experiencing the controls firsthand reveals the current limitations: robotic pincers, while improving, still lack the nuanced dexterity and the vast degrees of freedom inherent in human fingers and arms, making such delicate operations a significant challenge for current robotic capabilities.
Encord is also developing another innovative data modality that leverages a set of sensors strapped to the forearm to detect electrical signals in muscles, known as electromyography (EMG). While video footage of human hands manipulating objects often fails to capture the entire hand or the subtle muscle movements, Velmurugan envisions using forearm sensor data to construct a more robust 3D depiction of hand position and movement. This approach aims to provide AI models with a more comprehensive understanding of human motor control and intent.
The data generated by Encord is not merely raw footage; it is meticulously annotated with physical descriptions of each action, such as "right hand tightens bolt." This dense annotation is crucial for helping LLM-based models to truly understand the context and meaning of the actions being performed. Velmurugan estimates that this level of detailed annotation is approximately 100 times more valuable than "junky ego data" for training specific tasks, even though it incurs only about 20 times the production cost. This favorable cost-benefit ratio, on paper, makes such high-quality data a compelling investment.
However, the "20 times more" cost is a significant consideration. The economic disparity between generating physical training data and scraping text from the internet for LLMs remains a stark reality. The latter was virtually free for frontier labs, whereas the former requires substantial investment in specialized equipment, human expertise, and controlled environments. This fundamental difference in data acquisition economics highlights a key divergence between the development of LLMs and physical AI. Physical AI requires active data manufacturing, a process that fundamentally alters the economic calculus of building and deploying these advanced robotic systems.
Despite these challenges, Velmurugan expresses optimism about the progress being made across the industry. Encord’s unique position, with visibility into the programs of numerous robotics companies, allows them to observe firsthand which data generation and annotation techniques are proving effective and which are falling short. This bird’s-eye view of the industry’s collective learning curve is a key part of Encord’s value proposition. By analyzing trends across multiple clients, Encord can identify promising data techniques that are gaining traction industry-wide before any single customer might realize their potential.
This constant influx of new data challenges and experimental approaches ensures that the dozen or so pilots at Encord’s San Leandro facility remain exceptionally busy. Both Infante and Ceja represent a burgeoning workforce specializing in the creation of the fundamental building blocks for neural networks. Many, like them, previously honed their skills at Scale, another prominent AI data annotation firm, before joining Encord. Ceja’s own journey into this field is a testament to the evolving nature of technology and its applications. His prior experience at a waste management company, where he was responsible for maintaining a robotic trash sorter, ignited his interest in robotics. Now, as the Jenga tower inevitably topples, he finds satisfaction in the intellectual challenge of solving these complex training tasks. "It’s something new every day!" he exclaims, capturing the dynamic and ever-evolving nature of the frontier of physical AI.
