FieldAI is Building Embodied Intelligence For the World's Harshest Terrains

Ali Agha started with the hardest problems first. There’s the Mars helicopter and the search for extraterrestrial life, for starters. The researcher turned FieldAI founder says planetary caves are a prime spot to looking for the latter, as the structures shield any potential life from harmful cosmic rays.

“We were exploring solutions with different types of mobility systems, including four-legged robots, to go into these environments on Mars, and also on Titan,” Agha told me at a recent live recording of the Automated podcast. “We published quite a bit on this, and did the first deployment in human history of legged robots traveling several kilometers with zero human touch — no prior map, no GPS, no predefined information — into Mars-analog caves, from cave networks in Oregon to Kentucky and beyond.”

It was a kind of robotics kismet, then, when DARPA launched a Subterranean Challenge focused on outcomes of the variety Agha’s team was already determined to solve. Missions focused on industrial settings, underground environments, digital twins. 

“We submitted a proposal, were one of the very few funded teams, and our team grew at one point to more than 70 or 80 people — we ended up winning the urban phase of that competition worldwide, right before COVID,” Agha says. “That was an incredible experience, because we were putting brains and autonomy onto a heterogeneous set of assets. At the peak of our operations, we sent 11 robots — four with legs, four with wheels, three flying — into fully unknown environments, discovering them, carrying out a variety of missions as they explored the world and autonomously dispatched themselves.”

Agha and company soon moved onto another altogether different DARPA challenge whose goals aligned with the team’s work.

“Throughout these years of space missions and DARPA challenges, we kept pushing the boundaries of field robotics and kept deploying these solutions,” Agha tells me. “We were understanding the corner cases, developing this appreciation that once you go into the real world, everything very quickly becomes off-nominal — everything becomes an edge case, a corner case.”

Founded in 2023, FieldAI was built on each of these learnings. It was trained some of the planet's most unforgiving terrains, for the messy, noisy world outside the lab. A world built of cobbled together edges case. Not Martian caves, but one that is predictable only in its unpredictability. 

Once again, the team is starting with the hard problem. In this case, it’s construction, which offers its own special brand of unstructured environment. It also stands as one of automation's biggest opportunities. 

“Sometimes you have applications where you can control the environment to a certain degree, optimize it for the robot's operation,” he says. “Then you have applications like self-driving cars, where you don't control the world, but the world still has quite a bit of structure — you know the number of lanes ahead of time, you know where the exit is, you have GPS, and a limited set of objects you expect to encounter — cars, pedestrians, sidewalks. Then you go to a construction site, where every room has as much variation as you might see across an entire self-driving-car setting, because you're dealing with objects that are sometimes hard to even describe — broken pieces, hanging wires, dripping water, half-built scaffolding, stairs being installed, people walking around.”

It turns out construction is also an excellent way to collect a ton of rich, real-world training data, all at once. 

“One of the reasons we're in environments like construction isn't just that it's a massive market — $17 trillion, extremely hungry for new technology to increase productivity and safety,” Agha adds. “It's also that the site changes every minute, every hour. So the richness and quality of the data is unbelievably higher than in any other domain we've deployed in.”