Industrial Automation’s Biggest Challenges: Real-Time Adaptation

Automation that can adapt in real time presents new opportunities for robot users and developers. A3 members InOrbit and Realtime Robotics share their insights on what real-time adaptation means and the benefits to end users and robot developers.

Technology improvements enable some robots to adapt to changing conditions very quickly — sometimes faster than the blink of a human eye, which typically takes from 100–400 milliseconds to complete.

With continuous monitoring, machine learning, and path planning features, real-time adaptation offers significant benefits to end users, from enhanced efficiency and productivity to improved safety and greater flexibility in handling product mixes.

Robots that can adjust their actions based on real-time variances enable increased throughput, optimized workflows, and streamlined processes. Furthermore, adaptive robots can operate safely in dynamic environments, such as busy factory floors with people, robots, and other vehicles moving around.

Degrees of adaptability

Real-time adaptation is key to maximizing automation in dynamic applications, but there are different levels involved, says Ville Lehtonen, vice president of product at Realtime Robotics, Inc.  

“In fully dynamic applications, the robot doesn't really know what's coming next. For example, self-driving cars require an extremely high level of real-time adaptation because there’s an enormous amount going on for the sensors to handle — all the richness of life from turtles on the road to thunderstorms lashing your cameras,” explains Lehtonen.

A robot arm that picks pieces from a bank of CNC machines or 3D printers also benefits from real-time adaptation capabilities, albeit in less complex environments.

“The robot knows it’s going to be in the same area, but the parts can be wildly different, so it has to adapt all the time. You can't just say ‘go to these coordinates with these joint positions,’ because that's not going to work when there is variability in part size and shape,” says Lehtonen.

Realtime Robotics’ RapidPlan technology generates optimized motion plans and interlocks for robots to achieve the shortest possible cycle time in multi-robot cells. RapidPlan enables a PLC to command robots safely to a destination with the ability to quickly adapt to design changes by automatically regenerating paths.

The company’s automated collision-free motion planning software provides real-time adaptation, but it also reduces — and in some cases, virtually eliminates — manual programming time.

This functionality enabled one of its partners to develop a self-programming automated welding system for the railway sector that automatically generated code for ten welding robots so that they could perform 25,000 welds without having to manually program every weld point. In fact, Lehtonen says this project was considered undoable by humans given the huge number of welds. 

Part of the “secret” of Realtime Robotics technology is how it processes the point clouds it receives from robot sensors, says Lehtonen. 

“You just stream a point cloud at us and we have a way of collapsing this probability space incredibly fast. We use voxelization to make the math a little easier and because it speeds up the compute. Then we do a lot of proprietary, clever things to ‘collapse’ the space, enabling us to calculate the safest and most effective path for two robot arms in 10 to 50 milliseconds,” explains Lehtonen, adding that the system makes robot path plans so fast that it appears to a human that the robot is not stopping to think at all.

Changing fundamentals

The conceptual framework underlying automation today has evolved significantly from the approach that underpinned traditional, fixed production lines for decades, says Florian Pestoni, CEO at InOrbit.AI.

“Automotive manufacturing facilities might be the ultimate embodiment of traditional thinking, where the automation is as deterministic as possible, meaning that you try to control the environment and the conditions, you pre-program everything that each production cell is going to do, and you try to make it as identical as possible in every step.”

Technological advancements, from improved sensors to machine learning and AI, allow automation and non-robotic equipment to work in more changeable conditions and with greater flexibility.

“Random stuff happens in our stochastic world, especially when you have people around. It's fascinating to see how robots are going from traditional, highly controlled environments into semi-structured and completely unstructured environments. You can't survive in those environments without real-time adaptation. It just wouldn't be safe. Moreover, automation won’t be at its most effective if it can’t perceive and react to its environment,” explains Pestoni.

Challenges

The main challenge facing those developing real-time adaptation solutions is balancing adaptability with extremely low failure rates, says Realtime Robotics’ Lehtonen.

“Automation has to work safely and without collisions 99.99999% of the time. Anything less than seven nines is a potential disaster waiting to happen and robots move too much for five nines or six nines to be safe.”  

Vision libraries for robot perception have unleashed a lot of robotics opportunities for developers and end users. Realtime Robotics aims to bring that type of intelligence to the kinematics space.

“Kinematics can be incredibly difficult. In truck unloading, for example, six-jointed robot arms operate close to walls. The way a robot arm works is not at all like how a human arm works so plotting the right path for a robot arm to take can be quite counterintuitive and extremely challenging if you start writing code from scratch. This is hard, and now it's pointlessly hard, because you can get our kinematics library and do it out-of-the-box without having to program everything from scratch,” explains Lehtonen.   

For robot developers this means radically lower barriers to entry. Developers can download a vision library and kinematics library to take care of those aspects of robot building without having to reinvent the wheel each time, leaving them free to focus on the core problem they are trying to solve with automation.

Beyond the factory and warehouse

Real-time adaptation enables maximum efficiency and synchronization between automation and other devices in busy warehouses and factories, but it also enables robot deployments in other semi-structured, cluttered, and even more unpredictable environments.

InOrbit provides a scalable cloud-based robot management and analytics platform designed to enable robotics companies and their customers to develop, deploy, and operate smart robots at global scale. Its RobOps Platform system includes cloud-robot communications, incident management, localization and teleoperation, adaptive diagnostics, and predictive analytics among other features.

InOrbit technology is currently being used to increase the efficiency of hospital delivery robots at a large medical facility in the United States. Running a major hospital is “almost like running a small town,” says Pestoni, with robots having to safely and efficiently navigate multiple floors and buildings to make their deliveries.

“There’s a lot of equipment and medicine to move around, corridors to navigate, and escalators for robots to take. This is work that was historically done by people pushing carts and is now performed by autonomous delivery robots.”

Complex environments like these require two levels of real-time adaptation, explains Pestoni.

On a basic level, robots use sensing and various forms of AI from computer vision to obstacle detection and path planning to understand the environment in which they're operating and to respond and act accordingly.

“The simplest example is a robot that needs to transport something from point A to point B, but along the way encounters an obstacle. Modern robots can detect the obstacle so they don't run into it, and they can sometimes navigate around it. This level of real-time adaptation is enabled by the local AI running on the robot, but it’s limited by how far the individual robot can sense its environment.”

A more advanced level of real-time adaptation can be achieved by augmenting the local AI of a robot with a global view that includes other robots, machines, manually driven forklifts, and smart devices. Such a global view provides greater observability —“a fancy word for continuous monitoring,” says Pestoni — and enables orchestration of fleets of robots and other devices. It also allows for robot optimization and continuous improvement.

Moreover, it enables robots in busy medical facilities to interact effectively with infrastructure like elevators. The robots are able to abide by any number of user-defined rules, like waiting for the next elevator if there are too many people already in it or if the weight exceeds a certain value.

“All of a sudden, real-time adaptation isn't just in the immediate vicinity of a robot. The complexity expands exponentially, but the potential also expands exponentially in terms of the number of situations the robot can handle.”

In warehouse environments, for example, InOrbit technology could be used to identify the optimum robot from among a fleet of AMRs to collect a product identified by the warehouse management system.

“Picking the right robot to fulfil such tasks depends on lots of factors from proximity to the product, whether the robot is carrying a load, and how much battery each robot has left,” explains Pestoni. The platform can also be used as a “traffic cop” at busy intersections, he adds. This ensures that robots can safely navigate myriad other vehicles — from manual and autonomous forklifts to other AMRs — moving around busy facilities.

New opportunities

“Real-time adaptation should be thought of as a continuum,” says Realtime Robotics’ Lehtonen. 

“Many people see it as a case of you’re either real-time or you’re not, but this attitude implies that you're giving up on robot arms and we're going to use humanoids for these applications because of their wider range of real-time adaptability. Instead, I think we’re going to see robot arms, humanoids, humans, and AMRs working together. There's a lot of room on the continuum, and I would encourage people to be aware of that, rather than just writing off robot arms as old tech. It's never that simple.”  

Real-time adaptation presents a massive opportunity for introducing automation to industries that may have never used robots before, concludes InOrbit’s Pestoni.

“Some industries might have experience with mechanization, but not automation, like agriculture. We work with a company that makes fully autonomous tractors. We work with hospitals and everything in between. None of that would be possible without real-time adaptation.”