Why Generalist is Starting With Robotics' Hardest Problem

Andy Barry describes it as Generalist’s first “jaw on the floor moment.” 

“We had taught the robot to pick this baggie up with the left hand and then like shake it,” the startup’s cofounder and CTO explains. “We were running the task and on one of the rollouts, it picked the baggie up with the right hand and shook it and did the task. We all stopped. We never taught it to do that.” 

The staff Slack was overcome with messages of disbelief that the system had seemingly managed to make the jump on its own.  

It was still early days for Generalist. The Physical AI firm had yet to amass an unfathomable amount of code. The base was small enough, in fact, that and chief scientist and fellow cofounder, Andy Zeng, used the downtime afforded him by a bout with the flu to watch every minute of the firm’s data collect up to that point. “He was sure that somebody had done it with the right hand,” says Barry. 

“That was the first ‘wow’ moment where like there is something here that is just very different than what we have done before,” he adds. “We've seen many more moments like that, but that was kind of, that's burned in my memory because it was the first time and we all just didn't believe it.” 

Generalist has similarly managed to capture public interest in the weeks following GTC. The firm chose NVIDIA’s developer conference to publicly debut its GEN-0 model. It was a noisy week, of course, leaving its demos buried among an avalanche of physical AI news. Ultimately, however, the young team didn’t have to wait too long. Rapid iterations showcased in viral videos have afforded the startup subsequent news cycles. 

Five months after releasing GEN-0, Generalist debuted GEN-1.  

“We believe it to be the first general-purpose AI model that crosses a new performance threshold: mastery of simple physical tasks,” the company wrote in a post announcing the release earlier this month. “It improves average success rates to 99% on tasks where previous models achieve 64%, completes tasks roughly 3x faster than state of the art, and requires only one hour of robot data for each of these results. GEN-1 unlocks commercial viability across a broad range of applications — and while it cannot solve all tasks today, it is a significant step towards our mission of creating generalist intelligence for the physical world.” 

The words “commercial viability” no doubt caused many a brow to furrow. The notions of generalist intelligence and real ROI still feel impossibly disconnected for most intents and purposes. Barry says real world returns are a major priority for the startup, however, and something well within grasping distance. 

“I have worked on many robots in my career that have been very difficult to commercialize,” he explains. When we started the company, we said we’re not going to just build technology and then somehow they will come. We said from the very beginning, we're going to go out and make sure that our goals and our benchmarks are real tasks people are paying money for today. Because that means if we can hit our benchmarks, we will have reached commercial viability.” 

ROI Calculator

Discover the potential cost savings of robotic automation over a 20-year system life

This calculator compares your current manual labor costs against the total cost of owning and operating a robotic system over its 20-year lifespan.

EXPLORE TODAY

Part of that focus on commercialization has bringing customers into the fold early on in the process, and utilizing their metrics as goal posts for Generalist’s models. 

“We’ve taken exactly their tasks, sometimes even their specific parts,” Barry adds. “We've brought them in-house and then we've run our benchmarks on those. We've trained models on them and we've used them to guide our research. When I say ‘glimpses of commercial viability,’ what I mean is some partners are getting very excited about where their tasks are and how close we could be to deploying those.” 

Generalist began building its models by focusing on, arguably, industrial robotics’ hardest problem: manual dexterity. Subsequent videos have showcased impressive results, as manipulators engage in extremely precise behaviors, such as plugging in an Ethernet cable, placing pens into a zipper bag, and paper money out of a bill fold and putting it back in.  

“Why didn't we go work on a mobile base?” Barry rhetorically posits. “Because we said dexterity is where all the value is. Why aren't we starting with a humanoid? Because again, dexterity is where the value is and if you have dexterity, you can put it on a human arm. Why do we buy other people's arms as opposed to building our arms? Because we know from tele-op alone that the arms are good enough, so that's not where we should focus the company's risk. I don't want to take any risk at all except in the thing that really matters —  in dexterity. That's what we said at the beginning, and I think the investors recognized that.” 

Much of that focus involves collecting the right data. Earlier attempts didn’t go as planned. 

“People were mostly confused,” says Barry. “They were like, ‘what do you want?’ And we were like, just put these gloves on and go do your stuff, and then mail us the data.’ A bunch of people were like, ‘I don't know what this is. I'm too confused. No. A bunch of people were like, ‘okay, this seems cool.’ Then they dug a little bit, and then pretty quickly, we had robots that actually did some things.” 

After experimenting with a number of different methods, Generalist landed on a model that extracts data from a device worn by people performing the intended task. Some are professionals, who are already being compensated for their work, while others are non-professionals who receive money for their participation in the project.