Tackling Automation's Biggest Challenges: Dexterous Manipulation

Moravec’s paradox is one of the most counterintuitive and thought-provoking observations in AI and robotics. Reasoning, it says, requires very little computation, but sensorimotor skills require massive computational resources.

The result is that machines find complex problems — such as advanced algebra, for example — a cinch to solve but they struggle with easy problems like walking across a room or handling objects with human-like precision. Steven Pinker put it best: “The hard problems are easy, and the easy problems are hard.”

Dexterous manipulation fits perfectly into this challenging and counterintuitive category. In robotics, “full” dexterous manipulation refers to human-like dexterity, precision, and object handling in real-time, in structured and non-structured environments, across many materials, with minimal reprogramming required across different application domains and objects. These activities are a cinch for humans, but incredibly challenging for machines.

Many technologies must come together to achieve dexterous manipulation in a robot. It means robots that can safely handle objects of different shapes, sizes, and across a wide range of materials. It means being able to perform delicate tasks such as small component assembly without causing damage. It means robots with human-like awareness of the position of its body in space.

Achieving full dexterous manipulation also means providing robots with tactile feedback capabilities, enabling them to adjust the grip and force applied to various objects. It means developing control systems that can coordinate all the gripper parts and their degrees of freedom. And it means robots with the real-time intelligence and adaptability to generalize its handling skills to objects it has never encountered before.

Defining dexterity

The assumption that dexterous manipulation requires human-like grippers is not necessarily true, says Michael Perry, vice president of marketing, Dexterity Inc.

“In our world, when you start talking about dexterous manipulation, some customers expect that with the same end effector you can pick up a screw, a cola, and a box. There's some level of generality that you can achieve, but there are weight and size limitations that start making sense with a more dedicated form factor. In a typical application, we're picking goods that are most relevant to our customers rather than trying to pick everything all the time,” explains Perry.

Instead of trying to create “a one-size-fits-all solution that addresses all customers all the time and every corner case.” Dexterity uses grippers that can most effectively meet their customers' needs, says Perry.

“The question to answer is ‘What are you using hands for?’ If it's truly in a world where you have no idea what the robot is going to be picking up at any given time, then maybe human hands are the right solution. However, most industrial applications are not that open-ended and while you could get two hand-like grippers to manipulate a variety of objects, you will quickly run into speed, handling, and cost tradeoffs that make the application infeasible in practical terms.”

Dexterity’s application-centered approach is based on porting a wide range of human skills over to robots.

“There isn’t one core technology that unlocks skills like dexterous manipulation. Ultimately, people use a combination of vision, sense of touch, and proprioception to make decisions about where to pick things from and where to put them to. All of that needs to roll up into a robot that's capable of handling all those types of problems,” says Perry.   

Dexterity’s robots leverage multiple sensing capabilities and combine that with algorithmic motion planning, enabling robots to smoothly execute tasks created by an AI-powered decision-making engine. The company is focused on “sensor fusion.” That is, combining multiple intelligences and inputs and enabling robots to efficiently decide the best course of action based on those inputs.

“Our world requires robots with vision, touch, [and] two arms that can work together without colliding into each other, and the ability to understand the workspace so that the robot can move efficiently through it. All of that requires multiple intelligences working together in the most processor efficient way,” explains Perry.  

A sense of touch

Tactile perception plays a crucial role in tasks that require precise manipulation and interaction with different materials and surfaces.

HaptX is developing systems that can help to replicate human-like tactile intelligence in humanoid robots. Originally designed to bring a sense of touch to the virtual reality experience, the company’s HaptX Gloves incorporate hundreds of microfluidic tactile actuators that physically displace the skin when worn, enabling virtual objects to feel real.

Join us at the A3 Humanoid Robot Forum , October 7 in Memphis, Tennessee, to see HaptX chief revenue officer Joe Michaels talk about humanoids, dexterity, and touch control. Register now!

In 2019, a HaptX R&D project in collaboration with the Shadow Robot Company incorporated HaptX Gloves into an award-winning “tactile telerobot.” The system provides realistic tactile and force feedback to remote robot operators, enabling them to control human-like robot grippers from any distance.

But the technology has found yet another powerful use case in training humanoid robots to achieve human-like dexterity.

“Dexterous manipulation offers massive opportunity for the robotics industry. It will enable robots to achieve the same kinds of capabilities that humans have and will, over time, address the global manufacturing labor shortage, “ says Joe Michaels, chief revenue officer, HaptX.

“But none of this works unless robots have the same kinds of technical capabilities that humans have. And the hand — with its five digits and 27 degrees of freedom — is the trickiest part. It is so far beyond the capabilities of a simple parallel gripper that it requires completely different tools.”

To train a humanoid robot in dexterous manipulation, a human pilot dons the HaptX gloves which are mapped to a dexterous end effector or gripper. The human moves their hands naturally as they complete a task and the end effector responds like a human hand.  

“By operating the robot using our gloves, you teach the robot’s AI how humans use our hands to perform different tasks. When you provide robots with dexterity, you give them the power to do all the work that humans do. And you're expanding the robots’ capabilities way beyond simple two finger parallel grippers and into general purpose robots that can perform nearly any task that a human can do. And that increases productivity on a massive scale,” explains Michaels.

Industry is still in the “very early” stages of the race to achieve full dexterous manipulation, says Michaels.

“We’ve just now been given all of the tools that we need to start solving some of these giant problems and it will take a few years to really begin to see full dexterous manipulation at work at scale in industrial applications. But we're already seeing early indications of success and within a decade, I envision humanoid robots doing the work that the humans do at scale.”  

This doesn’t mean that general purpose humanoid robots will replace traditional industrial automation, however.

“You don't need a Ferrari to do the job that a Toyota will do. There are certain jobs that make more sense for simpler, maybe cheaper, more robust robots that do the same task over and over. However, the kinds of work that require human intelligence, human decision making, human skill and dexterity, that's where humanoid robots will shine,” says Michaels.

Muscles

Achieving human-like dexterity may require entirely new approaches to actuation. Kyber Labs is building a robotic manipulation platform based on artificial muscle fiber actuators that mimic the functionality of human muscle.

Designed from the ground up for AI-based controls, the company’s low-cost, artificial muscles enable fluid and dexterous movement that is not feasible with conventional actuation systems, says Tyler Habowski, co-founder Kyber Labs. It achieves these goals by combining bio-inspired neural net machine learning technology with its bio-inspired artificial muscles.

“Typically, robots take the dynamics of the world as humans experience it and override it with super rigid, super precise dynamics. But for humans, these dynamics exist more fluidly and naturally,” explains Habowski.

Kyber Labs’ system is in part designed to help robots perform tasks as intuitively as humans.

“Humans don’t need to think much about path planning and trajectory when opening a door, for example. We just set a low stiffness in our joints, grasp the handle and pull ‘backwards-ish’. Current robots have to perform more explicit measuring and planning to open the same door.”

The company’s artificial muscles, which are more powerful than human muscles by volume, use electromagnetic fields to move, eliminating the need for gearboxes.

“This makes our technology unbreakable, unlike traditional gearboxes, so we can throw a lot of machine learning at it. And quickly take much bigger steps towards robots with full dexterous manipulation. Conventional motors and gearboxes are reaching their optimization limit and they weren’t designed for machine learning controls,” says Habowski.

The system enables joint position and stiffness control via antagonistic actuation, which mimics how biological muscles work. This removes the need for robots to explicitly calculate every joint angle, instead relying on inherent compliance to handle uncertainty.

Habowski sees early use cases in applications like fabric handling, but there is virtually no limit to uses for dexterous, bio-inspired actuation technology that can easily be scaled for different applications.

“It’s going to start off relatively small. ‘Threading the needle’ is going to come much later. Tasks, such as handling textiles, which are really challenging for current robot designs, are just one domain where this technology can quickly deliver real benefits. Ultimately, the entire spectrum of handling and manipulation tasks can benefit from this technology.”

Achieving widespread adoption of full dexterous manipulation may be several years away, but the technologies required are either already available or in rapid development. As always, commercialization and adoption trends will follow their own trajectories with external factors such as the state of the supply chain and macroeconomic conditions playing significant roles, but it’s only a matter of time before robots achieve human-level dexterity. Moravec never said it would be easy.