Martial Hebert on Why Self-Driving Cars Took So Long and What Everyone Got Wrong About AI

Transcript

[00:00:00] Martial Hebert: Sometimes people are surprised that it took so long for self-driving cars to be deployed. The basic technology was ready quite some time ago, but to go from that - from something that can truly be validated - that's a completely different game. I've been at the Robotics Institute 42 years. I came here as a postdoc on a project called the Autonomous Land Vehicle Project. There used to be a program called the DARPA Image Understanding Program, and that program has been instrumental in placing the country as the leader of that field of research, which is essential for everything that has to do with physical AI.

[00:00:36] Brian Heater: I'd love to talk about the arc that's led us to where we are now.

[00:00:40] Martial Hebert: When we talk about that arc of development, we miss sometimes the angle of talent, expertise, and culture over decades. We have built this culture, this body of expertise, this environment where talent emerges around those topics of autonomy and robotics.

[00:01:00] Brian Heater: If I had asked you ten years ago where we would be as far as actually having self-driving cars on the road - you're smiling - do you think that we would be a little further along than we are right now?

[00:01:16] Martial Hebert: Well, the real answer is a very boring answer: it depends, and here is why.

[00:01:33] Brian Heater: Hello and welcome back to Automated. I'm Brian Heater, the managing editor at the Association for Advancing Automation. I'm excited to share a great conversation with you this week. Martial Hebert is the Dean of CMU's School of Computer Science. We recorded this conversation back in March to discuss the opening of the school's massive 150,000-square-foot Robotics Innovation Center. It's a really great chat with a lot of insight into the state of robotics and physical AI research. Thanks to Martial and thanks to CMU. Thanks to you, as always - if you are enjoying the program, please like, subscribe, and check out our newsletter, all of that, over at Automated.fm. And with all of that, please enjoy this conversation with Martial Hebert.

We talk a lot about what's coming up next in automation on the show, but if you really wanna see the future in motion, you've got to be there in person. Automate 2026 is where the world's leading innovators, builders, and dreamers come together to show you what's possible. Robots, AI, machine vision, motion control - you name it. All automation under one roof. And as part of Automate this year, the Humanoid Robot Forum brings together leaders, engineers, and researchers for a two-day deep dive into the real-world development, deployment, and commercialization of humanoid robotics. Register for free at automateshow.com to join

[00:02:59] Martial Hebert: us in Chicago, June 22nd through the 25th. We will see you there.

[00:03:05] Brian Heater: I'm really curious about some of the logistics behind creating a 75,000-gallon water tank. That must have been far and away one of the more difficult processes of opening this Robotics Innovation Center.

[00:03:25] Martial Hebert: Thank you for asking about this, and thank you for being careful calling it a water tank, by the way. Most people call it a pool, which involves about a thousand pages of regulation. So please, let's call it a water tank and not a pool.

[00:03:39] Brian Heater: Is there a difference beyond just government regulation?

[00:03:43] Martial Hebert: No, just government regulations. So, more seriously - I don't think that was a particularly logistical nightmare, really. The people who are doing that are very professional and used to doing those kinds of things. The more important point about it is that we had a smaller water tank here in one of the buildings on campus in the second basement - but it was much, much smaller. Now we have a much larger one. What that means concretely is that we are now able to conduct research and conduct projects that we could not conduct before. And this is basically the story of this entire facility. This is the story of what we're trying to do in robotics - to continue expanding, and expanding means being able to experiment on a larger scale and in ways that we could not before.

[00:04:46] Brian Heater: This is interesting, and I feel like you probably have as good a perspective on this as anybody. This is something we talk about quite a bit on the show - the data gap. Right now with physical AI, you're somebody who has been involved with autonomous vehicles for about four decades. To a certain extent those have been in the real world collecting real-world data, but getting robots out there collecting information is still the biggest challenge. This seems like a step - it's a larger tank, a way to get them collecting data within a large body of water.

[00:05:30] Martial Hebert: It's not just collecting data. It's also simply experimenting - experimenting on a larger scale, experimenting with more realistic environments. When it comes to physical AI, and in fact the fundamental difference between what we call physical AI and what we might call regular AI - the ChatGPT, LLMs, and so forth - a critical difference is of course the physical aspects. What does that mean? That means two things. One, it's very hard to collect data. Framing this in the context of underwater robotics - this is even harder when you want to collect data in settings that involve interaction with people, for example, coworkers or home robots that interact with people.

There's the data collection aspect, but there's another aspect, which is the experimentation. You need to experiment with physical things in real settings, and no matter how much simulation you do, at some point you need real experimentation to be able to validate those things. In fact, we have a long history of doing this. We have another facility off campus called the National Robotics Engineering Center, which is in the north of the city, where we do large-scale systems integration and field experimentation. The difference now with the new wave of AI models is this idea of the importance of data - the importance of collecting data not just physically, but in simulation - and bringing those AI models and those physical systems together.

[00:07:31] Brian Heater: I really want to go way back - back to NavLab. I want to go back to really what I think is the beginning for you, at least, of autonomous vehicles. What was the state of the art of this technology when you got involved?

[00:07:52] Martial Hebert: Since you're challenging me to talk about my own history, we might be in trouble - that might take a long time.

[00:08:02] Brian Heater: We've got about 45 minutes, so let's go.

[00:08:04] Martial Hebert: So I arrived here as a postdoc, and in fact I've been at the Robotics Institute 42 years and two weeks today - but who's counting? I came here as a postdoc on a project called the Autonomous Land Vehicle Project, which was the first project actually funded by DARPA. The first federally funded project on what is now autonomy or self-driving. And the way that started is not with something that looks like a car or a truck or a vehicle, really. It was a six-wheeled vehicle called the Terregator that was trying to navigate the sidewalks of campus autonomously. Then there was development by a large team - I was just a postdoc and a starting scientist at the time - of NavLab, which was a Chevy van equipped for those early days of self-driving. There was a camera, and I believe what was the very first lidar used for autonomous driving. There were two copies of it: one here at CMU and another in Denver at what is now Lockheed Martin - used to be Martin Marietta.

[00:09:19] Brian Heater: It's interesting to hear that DARPA has played a huge role in the development of robotics generally here in the States. And it's interesting to hear, as you said, that this specific project wasn't - I tend to think of DARPA as a lot of the Boston Dynamics work, a lot of the pack robots, or the humanoids. But it sounds like they were working on something a little more humble at the time.

[00:09:55] Martial Hebert: It's interesting you use that word humble. I would've said exactly the opposite - they were working on something amazingly ambitious.

[00:10:03] Brian Heater: Sure - I suppose, I mean, as far as you know, this is the defense department, so it's just an interesting place to start this sort of research on the sidewalks of campus.

[00:10:15] Martial Hebert: On the surface it would look that way, except if you generalize the mission of DARPA and the mission of federal funding in general - not just a narrow view of defense, but a broader view of national security and of leadership in areas of technology and science. If you view this with a much broader aperture, then it makes sense. As a matter of fact, the ALV project was part of a much broader project called the Strategic Computing Initiative, and in fact the real motivation originally for the ALV was not just giving things autonomy. A deeper motivation was to find a task or application that was so hard that it would motivate a revolution in the field of computing. That's what I meant by ambitious - it was a very ambitious view of the world. This was not narrowly concentrating on a particular defense application. It was saying, what do we need to look at to really transform the technology? And incidentally, make sure that the US leads in that transformation.

[00:11:39] Brian Heater: That's really the space race argument, right? We'll go to space and then all these other breakthroughs will come.

[00:11:49] Martial Hebert: Exactly. And in fact, sometimes we forget that DARPA was created originally to address the space race - the rockets and all that.

[00:11:58] Brian Heater: Which is - and I'm sure most of the people watching this are familiar with the DARPA challenges - but that's the idea behind a lot of these, right? We put out a specific DARPA challenge and then all of this interesting innovation comes from the work that goes into those challenges.

[00:12:15] Martial Hebert: Yeah, absolutely. And I'll give you another example that's probably not as well known by a broad audience that has come to my field. My field is computer vision - that's where my research is. There used to be a program called the DARPA Image Understanding Program. This was one of the longest-running DARPA programs - about 15 years or something like that. It involved basically all of the research outfits in the country, mostly universities, doing fundamental research in computer vision. And that program has just been instrumental in placing the country as the leader of that field of research, which is essential, by the way, for everything that has to do with physical AI.

The spirit of that program again was not looking at some narrow defense application. It was taking a very broad view, fostering the exchange of ideas, the creativity to discover new things and invent new things. I think it's important to keep that in mind when we talk about federal funding. Sometimes there's that conversation about, well, do we need that - industry is doing a lot of things. This is where federal funding has its place: the idea of having a very large sandbox where people can really create and innovate - and not innovate in the sense of creating new products, but innovate in the sense of having completely new ideas and completely new directions of research that would not be possible otherwise. The DARPA Image Understanding Program is one example, the Strategic Computing ALV is another, and there are many like this in the history of federal funding.

[00:14:20] Brian Heater: Is there an extent to which federal funding or research of that nature is perhaps more pliable or more open to change than VC or startup funding, which is very focused on a specific goal and bringing a specific technology to market?

[00:14:48] Martial Hebert: It's different, and it's complementary. I think part of the issue sometimes in the conversation is that they're seen as either in competition or as completely different universes that don't need each other, but in fact they are truly complementary. The aspect you mentioned - concentrating on a narrower objective - is one difference. But another difference is long term versus short term. If you look at all the major discoveries and go back to the research, the arc and history of research that created them - this is a very long-term process. So that's the kind of difference between the two approaches, but again, a difference in a good way that is actually complementary.

[00:15:54] Brian Heater: I'd love to talk specifically about the arc that's led us to where we are as far as autonomous driving, and how Pittsburgh - CMU being the major player here - became such an epicenter for autonomous driving. It sounds like the Autonomous Land Vehicle program and NavLab were a big part of that. How do you get from point A to point B?

[00:16:23] Martial Hebert: The ALV program was certainly a big part of it, but there were other parts. In the Robotics Institute, for example, there's a group called the Field Robotics Center that was instrumental in really elevating that idea of autonomy to a much higher level.

Part of the answer to your question is something that we miss sometimes when we talk about investment in research and how that arc of development works. We tend to think about technology only - this particular technique, this other technique is being deployed, et cetera. But we miss sometimes the angle of talent, expertise, and culture. Part of the reason why we have so many individuals in technical or management leadership positions in the robotics and self-driving industry who come from CMU - a big part of that reason is that over all those decades you describe, we have built this culture, this body of expertise, this environment where talent emerges around those topics of autonomy and robotics. It's a much more subtle effect than just pointing to a particular bit of technology.

As a matter of fact, you were asking about the early days of the ALV. Clearly some of the algorithms we used at the time are not in use anywhere anymore, of course. So you could say, well, what's the connection? The connection is that by working on those things we learned, we tried other things, we developed other things, and we have this community of people that gets developed, this community of expertise that gets developed. And eventually, when things are ready for commercialization and deployment, we have this community, we have this expertise. That's the connection to the industry - not just the technology.

[00:18:47] Brian Heater: So in this context, what does culture mean? And what do you do to promote that?

[00:18:55] Martial Hebert: There are two things that we do here that I think are pretty unique - not unique as ideas when I say them, but the degree to which we do this is a little different from other places.

The first one is to take a very broad view of the field. And by the field, I don't mean just self-driving - I mean all of robotics, in fact all of computer science and everything that has to do with computing in general. A broad view that goes from AI and algorithms all the way to physical systems and the design of physical systems, through the human aspects: modeling human behavior, understanding human interaction, et cetera. Something that integrates all of those aspects. This is something we've done in the School of Computer Science here since the beginning - from our founders and their definition of the field. The mistake we need to always avoid is to take a very narrow view of the field that tends - to use a technical machine learning term - to overfit on a particular technique, a particular shiny object. The reality is that when you look at having safe, trustworthy systems that actually make an impact, it takes all of those things.

The second aspect is more on the cultural side: understanding the connection between fundamental research and the impact and the systems and the integration, and not putting artificial barriers between the two. You may hear sometimes in discussions people say, well, this is not really fundamental or academic research - it's very applied. Or vice versa: this is nice, but it's really theoretical stuff that's not going to have an impact. Those are the very things to absolutely avoid, because those are the things that can completely stifle new ideas. On the cultural side, what we try to do is have an environment where there is a direct connection and a deep respect for all aspects of research and development - from the most fundamental theoretical aspects all the way to the most physical systems integration aspects. And I think that's part of what I've seen making progress in all these years.

[00:22:11] Brian Heater: From the outside, we tend to think of people working on bleeding-edge technologies as being very forward-thinking. But I think to a certain extent what you're getting at is that even researchers working on the edge of technology can still be too hung up on certain orthodoxies and certain ways of doing things.

[00:22:40] Martial Hebert: Yes. And especially in academia, the tendency is great to look at one particular side of that spectrum.

[00:22:53] Brian Heater: I was thinking about this recently - I think neural networks are probably a perfect example of this, right? As far as the degree to which they changed the game for artificial intelligence, particularly over the last few years. They completely changed research for a lot of people, and I suspect because of that, a lot of people have had to adapt their work quite a bit.

[00:23:22] Martial Hebert: Yes. And the same applies to those techniques - it does not mean that those techniques are the end of the story, and we need to be very careful not to act like they are the end of the story. It's a constant evolution of ideas - I don't even want to say technology, but of the ideas and the thinking. We need to have that frame of mind, and we need to prepare people - for example, through the way we train students - to have that frame of mind of constant change. And frankly, to keep an open mind. That sounds like a silly thing, but in a scientific or technical field it's sometimes actually very hard to keep an open mind. It's much easier to center on a particular direction or particular technique.

[00:24:17] Brian Heater: What's really interesting to me is entering this field - even 10, 15, 20 years ago versus today, nevermind 30 or 40 years ago - it seems that so much of what you would've been working on would have felt very theoretical, very academic.

[00:24:42] Martial Hebert: No - not in our case. I think the better characterization is that we had that spectrum from the theoretical to - for lack of a better term - the more applied or integrated side. The difference over the years is that what could be achieved on that applied side: we could not achieve nearly as much as what can be achieved now. Right now, one can go much further, much faster in terms of impact. But that spectrum was always there.

[00:25:51] Brian Heater: It's been very fascinating speaking with people who have been working on autonomous vehicles over the past 10 to 15 years with all of the money and research that's gone into them. Obviously we've seen a bit of a boom-and-bust cycle as far as funding goes. I know this isn't specific to self-driving, but there are a lot of things that happen faster than anybody anticipates, and a lot of things that have taken longer. If I had asked you ten years ago where we would be as far as actually having self-driving cars on the road - you're smiling - do you think we would be a little further along than we are right now?

[00:26:47] Martial Hebert: First of all, I'm smiling because at the time that was the number one question that would be asked, of course. Which I consistently refused to answer, by the way. I was very skilled at that.

A couple reasons for not answering the question. Well, the real answer is a very boring answer: it depends. And here is why it's an important 'it depends.' The exact conditions under which the technology is used matter, and they matter not just in terms of whether it works or not or whether it's commercially viable. They matter at a fundamental level. Are you doing driving in a place that has been mapped extensively, like Waymo does in San Francisco? Or are you trying to drive in a place you have never seen before? Are you driving in conditions that have rules of the world or conditions that have never been seen before? What is the density of pedestrians? All of those questions are fundamental as to what can be done. In some cases they may require radically different solutions in terms of sensors, in terms of the way the system is trained, and all that. That's why I consistently - to everybody's chagrin - refused to give a broad statement, because general statements like this from a technical standpoint don't really apply.

The second thing, on self-driving - but any physical AI, especially that has potentially any interaction with people, and especially catastrophic interaction - the amount of effort, and by effort I don't mean just money, I mean ideas, processes, and all that - to do testing and validation and actually get to a level where the system can be validated to a point where it can be deployed, is consistently underestimated. And I think that's where your question was going. Sometimes people are surprised that it took so long for self-driving cars to be deployed. If you talk to people at those companies - and we have a lot of CMU connections - the basic technology was ready quite some time ago. That's true. But to go from that - from something that can truly be validated to a level where you can have the general public around - that's a completely different game.

[00:30:07] Brian Heater: Something I've been talking about quite a bit with physical AI companies - FieldAI is a great example, and they're opening up a space in your new building - is this idea of the robotics data gap and the flywheel. We have more data from cars than we do far and away from robots. So if we need a lot of hard real-world data, how much of this is going to come from simulation? How much from real-world data? And if it is real-world data, how do we get to a point where we can deploy enough robots to start getting that flywheel moving?

[00:31:07] Martial Hebert: You just gave a wonderful summary - except for one item you did not mention that I will - of all the key challenges in research on robotics data. And exactly to your point, the work is basically figuring out how to strike the right balance between physically acquired data - which is limited - and simulation, and perhaps other sources.

There are other companies and research groups who work with videos: just like you have an enormous corpus of text from what people write on the internet, there's an enormous corpus of videos of people doing things that you could use as input to training as well. Accurate physical simulation is another source, and it's not just simulation of the actions and interactions, but simulation of entire scenarios at scale. So all of those things are on the table. And coming back again to not giving a specific answer, which I'm apparently very good at -

[00:32:32] Brian Heater: Obstacle avoidance, as they say.

[00:32:33] Martial Hebert: Obstacle avoidance - yes. But it's actually still the case that the real answer depends in a technical way - not in ways of investment - on the environment, on the task, and all that. FieldAI, as the name indicates, looks at field robotics applications - what we call the three D's: dull, dirty, and dangerous - things that are typically done where you don't have people around. They have certain characteristics. There's a universe of scenarios that informs what it means to create data in that scenario. There are other entities that look more at situations where the quality of interaction with a person becomes critical and the safety aspects are very different. So you might think about data in a completely different way. That's what actually makes it exciting - it's a very rich area. It's not as straightforward as finding the ultimate way of fabricating data for robotics. It's actually a very rich field of its own.

[00:34:00] Brian Heater: You're a longtime Yinzer now, right?

[00:34:04] Martial Hebert: Yes - 42 years or so.

[00:34:06] Brian Heater: And obviously CMU has been a really large part of that. Something I was curious about - I spent the last nine or so years of my career at TechCrunch, and the last several visits I made to Pittsburgh were coming out there and speaking with startups. Prior to the pandemic there was a big question of brain drain - how do we maintain all of these really smart people, how do we keep them in this industry versus going to San Francisco or New York? I have to imagine the pandemic probably changed that a little bit, allowing people to work remotely to a certain extent. How has that changed in recent years, and to what degree is a large project like this robotics center a bid to really keep people local and keep people innovating in Pittsburgh?

[00:35:14] Martial Hebert: It's part of this effort to expand the ecosystem here in Pittsburgh. This goes back to what I said earlier about the harder-to-quantify effect of the environment and the culture. It's basically expanding that environment and, hopefully by doing this, creating more talent and retaining more talent.

[00:35:45] Brian Heater: One of the things that you were speaking about around the opening of the center that I thought was very interesting was this idea of cross-pollination - getting a lot of people in physical AI and robotics who work in different corners of that field into close proximity with each other and giving them opportunities to work with one another.

[00:36:12] Martial Hebert: Yeah - co-location. It all boils down to this idea of exchanging ideas and being able to work together on ideas. It's not just being able to work on systems or being in the same office writing code. It's the idea of being able to think together and create new ideas. And yes, the intent of something like the RIC is to enhance our ability to work like this with companies, with startups, et cetera.

[00:36:50] Brian Heater: So we talked a bit about the 75,000-gallon water tank. What are some of the other features in this new building?

[00:37:01] Martial Hebert: A very large drone cage - far larger than anything we've had access to. That drone cage is connected inside to a very large motion capture facility, so we can have motion capture both inside and outside. We have a large outdoor testing area that can be reconfigured, as well as an indoor testing area. Interestingly, we have a significant wet lab space - for example, for materials research as a component of robotics.

This building is home to the School of Computer Science jointly with the College of Engineering, and one of the things the College of Engineering does in particular - together with our colleagues in robotics - has to do with soft robotics. It has to do with designing new materials, both for tactile sensing, for artificial skin, and for artificial muscles for robots. So there are all kinds of activities around the physical form of robots, looking at radically different ways of thinking about what physical devices look like - things that require wet lab facilities that you would not naturally think of as being part of robotics or physical AI.

[00:38:30] Brian Heater: A few times ago when I was out there I had spoken with some of the drone team, and if I remember correctly, swarming behavior was a big part of what was being studied at CMU.

[00:38:44] Martial Hebert: That's one aspect. Swarming behavior, resilience of systems - we go from things that are very autonomy-focused, like autonomous swarming, all the way to things that are more systems-oriented. For example, in the computer science department we look at the issue of how to distribute computation between a device at the edge - could be a drone, could be a phone, could be another device - and a central or intermediate facility.

[00:39:26] Brian Heater: You're talking about sort of a shared brain across devices.

[00:39:28] Martial Hebert: To an extent, yes. How do you distribute the AI processing and data based on the computational resources, based on the network - whether you have a good connection or a bad connection - how do you do that optimally and seamlessly in a way that maximizes performance overall? That's another example. Coming back to my repeated point about connecting fundamental research to what is traditionally called applied: the way I describe it, it's a very practical thing - you want your device to work as optimally as it can in all conditions. That's very practical. But underneath that, there are very hard fundamental research questions. So that's the kind of thing I love, and I think that's where progress is - this combination.

And then coming back to your question about the drone group: one of our big projects - the first one that started at the Robotics Innovation Center - is called the Triage Project. It's basically a first responder project. The idea is that in a disaster situation - an earthquake or similar event with a lot of casualties - to have robots and drones that can assess from a distance the state of different victims and be able to make decisions about what intervention needs to be taken. Ground robots that can apply those interventions all the way through to autonomous evacuation. That's an example of a very ambitious project that motivates considerable fundamental research - for example, you need to develop sensors that can sense medically relevant signs at a distance. That involves fundamental research in sensing and computer vision, et cetera.

[00:41:52] Brian Heater: This speaks to a large part of your own work and your own focus. I know you've done quite a bit of work in what we used to call elder care, what we would call age tech now. And it sounds like, to an extent, those are motivated in a similar way - by serving the needs of people first, rather than having the technology itself come first and then kind of backwards-engineering what function it could serve.

[00:42:27] Martial Hebert: Yes. You're referring to something called the Quality of Life Technology Center, which was a National Science Foundation Center. I like to think in historical terms, and it's interesting to look at this center and the history around it. This center was, I think, at least ten years ago now, and the ideas behind it were quite forward-looking in terms of having systems help the elderly. The important thing was helping - not replacing capabilities, but actually helping - and the idea was to maintain people in their everyday life environment as long as possible.

The interesting observation there is that if I compare what we were able to do at the time and what could be done now, the progress is considerable. When it comes to robots and physical AI in particular, we were at the time very limited by the fact that we needed fairly complex, expensive, and because they're big and complex, not necessarily safe robots. The evolution now with physical AI is that there is at least the possibility - the potential - of having devices that are much simpler and much safer in their physical form, that can be used in the home and in proximity to people. So the evolution is very interesting - it's now going to enable those things.

[00:44:35] Brian Heater: Moving from the historical view to the future - the center has been a long time in the works and I imagine everyone's been lobbying for their projects to get in there. You mentioned the materials lab, the drones, the water tank - it's a really big center, but you do have to future-proof it. You have to create a center that will be of use for hopefully decades and generations to come. How do you look forward into what research around physical AI and robotics is going to evolve into, and make sure that this center continues to serve its function in the future?

[00:45:34] Martial Hebert: There are some things that will always remain when we talk about physical AI. Coming back to earlier in the conversation - when we talk about physical AI, we need to have physical testing. We need that space, and we need it to be designed in a way that it can evolve and be reconfigured.

So the testing and development space that we have - if you look at the building and the space - they are very open. And this was a little bit controversial by the way. We have not prescribed particular fixed configurations of those spaces with some kind of preconceived idea that this is where we are going to test X, Y, and Z. We kept those things very open and by construction reconfigurable and changeable over time. Exactly because we cannot predict what the future configuration of the projects is going to be and what kind of work is going to take place.

[00:47:01] Brian Heater: Well, I think we're actually at about time right now. It was an absolute pleasure - thank you so much for taking the time.

[00:47:07] Martial Hebert: Thank you for the opportunity. Thank you.

[00:47:10] Brian Heater: Thanks so much to Martial and CMU. I hope to visit that huge facility someday - I'm inviting myself and the rest of A3 to come visit. Hopefully we'll do that in the very near future. Thank you so much for joining us, and thanks to you as always for tuning in. If you like the show, please like and subscribe, please tell a friend. We also have a newsletter with original features that drops every Thursday morning. You can find all of that and more over at Automated.fm. And with all of that, we will catch you just about this time next week with another episode of Automated.