On the Job Learning: How Physical Intelligence is Putting Robots to Work 

Welcome to the physical AI liminal space. It’s robot limbo, waiting to give the data flywheel a sufficient nudge. Ken Goldberg has memorably described the problem as the 100,000-year data gap, referring to the sizable headstarts for sets used to train large language models versus those we’re looking to kickstart a general-purpose robot revolution.  

Roboticists have introduced plenty of creative approaches to the problem, synthetic data being one of the most notable. The method utilizes simulation to train systems on seemingly infinite loops.  

Each method has its strengths and drawbacks, and all will likely contribute to building the vast expanses of data necessary for approaching generalization. That data won’t always be clean and will often require post-training methods like tele-operation to further refine models.   

But what of the more straightforward method? What about brute force? How far can physical AI models advance on the backs of data collected by robots interacting with the real world? More to the point: how many robots will it take to get there? 

We sit at the precipice of that vast limbo. The goal is to generate enough data to kickstart the physical AI flywheel that allows systems to be deployed at sufficient scale to consistently refine and improve their own models. But deploying robots — even at a small scale — requires them to be good at at least one or two things.  

The simple knowledge that robots are on the floor, slowly getting better isn’t enough for a factory or warehouse to justify deployment — those systems need to do something useful. It can (and mostly will for the time being) be something simple like loading and unloading totes, just so long as the robot is putting in the work.  

When I suggest that bridging the gap this way could prove difficult, Physical Intelligence founder Sergey Levine attempts to put things into perspective.  

“In the grand scheme of things, maybe sometimes something that we as researchers get a little bit mixed up about is easy for us versus easy in the context of human civilization,” the AI pioneer explains. “And in the context of the world as a whole, actually getting real world data and then deploying robots that are going to collect more experience and get better and better is a lot easier than inventing some other technologies just to avoid having to do that. And because it's a bootstrap problem, it's easier once things are out in the world.” 

Levine adds that researchers may soon (relatively speaking) have the opposite problem of too much physical AI data. That is to say the flywheel has spun up one million robot deployments, which have suddenly put researchers in the position of determining what data to prioritize and what to throw out. Of course, he’s getting ahead of himself on that one, and from where we’re perched in the robot data liminal zone, it looks to be a pretty good problem to have. 

ROI Calculator

Discover the potential cost savings of robotic automation over a 20-year system life

This calculator compares your current manual labor costs against the total cost of owning and operating a robotic system over its 20-year lifespan.

EXPLORE TODAY

For now, an increasing number of robotics companies I talk to are preaching the gospel of baby steps. In recent months, I’ve spoken with executives at Agility, Apptronik, and Boston Dynamics about the importance of focused deployments. Get good at one thing, deploy, build up a skill set, diversify.  

Levine says Physical Intelligence is “experimenting” with its own deployment.  

 “For example, our robot assembling boxes at the Dandelion Chocolate Factory,” he explains. “That's very convenient because Dandelion is across the street from our offices, so we could set up a robot there and it spends all day kind of building these boxes that they use to pack up chocolates. We also have a coffee service at the office. Right outside where I'm standing right now, there's a station set up where somebody can go on the company Slack system, type in, ‘I want a latte,’ and the robot will go and make it. These are not really high horsepower kind of commercial efforts. They're really experiments to see what happens when the robot has to solve real-world tasks, and they're also experiments on how data from these kinds of deployments can be used to further improve the system.” 

The key to creating the right deployments is developing situations in which the system is collecting optimal data. Levine cites a concept in educational psychology known as the “zone of proximal development,” which represents a kind of sweet spot for childhood education.  

“The zone of proximal development is when you are doing something that you're not completely clueless about, where you have some sense for how to get started, but that provides just enough challenge that there's room for improvement,” he says. “So a good early childhood education system puts children in that zone of proximal development. And what we want to do is we want to put our robots in that zone of proximal development.” 

In the end, it comes back to collecting real world data. Levine acknowledges that the so-called gap is a large one, but perhaps not so impossible to traverse as some of his fellow researchers have made it out to be. 

“Real world data is not actually that costly,” he says. “It's an industrial effort. You have to set it up. You have to get the robots. People have to actually do it. People have to be trained to do it properly. But it's not somehow profoundly impossible relative to other kinds of industrial efforts. It looks impossible from a very research-centric academic standpoint because it costs money, costs resources, and it costs work, typically the kind of work that professional researchers don't typically enjoy doing. But it's not harder than building a house or building a bridge, right? It's work that can be done. And if we are serious about taking physical AI to the next level, it makes sense to put in that work and get it right.”