Discussing Embodied AI and Humanoids with Agility, Physical Intelligence, and Dyna Robotics

Depending on topic, guest lineup, and your own familiarity with the subject matter, moderating a panel can feel like a million years or the blink of an eye. I’ve been doing this job long enough to have experienced both scenarios more times than I care to mention. Given the choice, I’ll opt for the latter every time. I prefer carrying on a conversation backstage, after getting the flashing red end of session light to the awkward farewells of people who fully run out of things to discuss.  

I’ll be the first to admit that I overstuffed Tuesday night’s panel from a content perspective, but 20 minutes only affords you time to scratch the surface of embodied AI and humanoids with panelists from Agility, Dyna Robotics, and Physical Intelligence. We jumped around a bit, but from the afterglow of late checkout at the Sunnyvale Radisson the next morning, I think we touched on some pertinent topics to cap off a lively year for robotics. 

The Nebius Robotics & Physical AI Awards and Summit was a one-day conference/ceremony held at the Computer History Museum situated conveniently between the Googleplex and Microsoft’s Silicon Valley Campus. Amit Goel, NVIDIA’s head of Robotics and Edge Computing Ecosystem, teed us up with his own keynote.

My panel was as follows: The Pioneer Frontier: Architecting Truly Autonomous Systems featuring Jonathan Hurst (co-founder, Agility AI), Kevin Black (researcher, Physical Intelligence), Lindon Gao (co-founder/CEO, Dyna Robotics).  

A few subjects have been top of mind for me as we look forward to what’s certain to be another wild 12 months for the industry. To avoid writing a story with a wordcount longer than the panel itself, I’ll break down some answers by topic below. 

Why is grasping/manipulation such a difficult problem for embodied AI? 

Black: From an RL (reinforcement learning) background, what we would say is, in locomotion/walking, it’s very easy to cover the same space, meaning that if you take random actions, you can see the robot fall down in all sorts of different ways and learn from that. Manipulation is very sparse, because you reach for something and you only interact with it a little bit. It’s much harder to get a signal and cover all possible ways you can interact with it. 

Hurst: I think it’s one of the grand challenges of robotics, and part of it, too, is because the hardware is so challenging. Even if we get the AI tools to a point where they can find optimal solutions with whatever piece of hardware — which itself is a pretty big ask — the hardware is also really, really tough. It has to be so robust and so compliant and so stable in ways I don’t think science understands yet. We have a lot of exploration left to do before robot manipulators can get anywhere close to human dexterity from a hardware point of view. 

Generalized vs. specialized models for robotics. 

Black: The way it’s gone in language models and visual language models is the pre-training is very important — learning from as much data as possible to build really general representations and being able to handle any situation you encounter. Even in LLMs I would argue there’s been a place for specialized models, like coding models, dialog models, and really advanced stuff. They’re definitely training specialized models for that. In robotics, if I were to guess, I would say it’s even more important, at least in the short term, because you’ll have some complication or some weird embodiment where you’ll need a specialized model to get high performance there. I feel like it will be quite a while before we can have a model that can do literally everything.  

Gao: Pre-training is generally a generalist model. What we see is that, when we pre-train a model on specialized asks, it doesn’t benefit a lot at the post-training level. But for pre-training, it has a wide distribution. You can transfer from one task to another, and thereby give the model a better understanding of the 3D environment around it, and actually boost the performance downstream.  
 

Humanoids in the home. 

Hurst: The benefit of the humanoid form factor is it can operate in human spaces. The value of it in the long run is its versatility and its ability to do many, many things in human environments. The technology is not there today, though. It’s coming, but it’s not there today. Agility’s strategy has been to find the first use case that justifies that form factor, that needs bipedal, bimanual balancing, and that why we’re starting to just move totes around. It’s incredibly hard to deploy hundreds of robots and then thousands of robots and support them all. We’re going to learn all of that as we then go to the next use case of bin picking and stocking shelves and working our way out. But it’s going to be a while before you have this generally capable humanoid that can operate in all of our spaces and homes.