Robots & Vision: Who Runs This Workcell, Anyway?

The booths were larger, and rows filled with attendees at the Automated Technologies Council’s recent Automate 2011 Show and Conference. But among the hoopla, new products, and corporate announcements, a hidden revolution was taking shape - a revolution of cameras, robots, and controllers’, struggling to determine who will run tomorrow’s automated workcell.

Robot Domination

During the past 10 to 15 years, the machine vision system in a vision-guided robot workcell was treated like slave labor - literally. The camera was simply another sensor mounted on the robot, and the robot controller called the shots when it came to directing the robot and requesting offsets from the machine vision system, as necessary, to fine-tune the robots programmed path.

Interestingly, the robot controller rarely could match the computational power of the vision system, but in the case of controlling the workcell, this weakness worked to the robot controllers’ advantage. Robot controllers used embedded single-board-computer (SBC) electronics with real-time operating systems (RTOS), which made the robot controller a better, safer supervisor for the robot workcell than a PC running some version of Windows or Linux that was plagued with latencies from non-essential (to the workcell) routines. A smaller brain also meant the robot controller programming codes more closely resembled PLC programming than a machine vision’s image analysis software programs. Shorter learning curve, faster adoption, robot controller wins.

The above description - where the vision systems fine-tune robot motion - is the essence of visual servoing, but not the complete picture. Visual servoing has come to mean that the vision system directs the robot. In the past, the virtual world of the robotic simulation software ruled the workcell, using real-world data from the vision system to adjust robotic movement. The robot may have a general path to follow as part of its routine, but in essence, the vision system is calling the shots based on object locations and real-time motion in the workcell. In the new world order, the vision system using real-world data may control the robot, or in other words, data will move from the real to the virtual, rather than from the virtual to the real, opening up new vistas in vision-guided robotic applications.

At this year’s Automate 2011 show, companies demonstrated some of the first commercially available workcell solutions where the vision system was actually in control, resulting in what some attendees describe as the first shots of a revolutionary change in the control of a vision-guided robot workcell.

Vision Steps Up

“Software is going to be the enabler of this next phase in vision-guided robotic workcells,” explains Phil Baratti, Manager, Applications Engineering at EPSON Robotics (Carson, California). “Today, a lot of people use stereovision and displacement sensors, lasers, etc., to give depth and skew information, but now there are [machine vision] algorithms that use a single camera to determine skew and depth of part. There is still stereovision, and it’s come a long way, but the single camera control is going to drive vision-guidance in the future.”

MVTec Software GmbH’s (Munich, Germany) HALCON 10 image processing software was at the heart of several visual servoing applications at the recent Automate 2011 show.

“There are two approaches to providing 3D vision for robotic guidance,” explains Dr. Wolfgang Eckstein, Managing Director at MVTec. “One is to have 3D data from stereovision, structured light, or time of flight camera acquisition, the other is to extract 3D information using 2D data, or what we call 3D shape-based matching.”

According to Eckstein, extracting 3D from 2D data using a single camera is gaining in popularity because of the improved algorithms and simplified hardware designs. While extracting 3D information from 3D data is essentially comparing 3D point clouds to find the best fit, the extraction from 2D data starts with 3D models of the imaged object. “We rotate the 3D [model] until a 2D projection of the model fits what the camera sees. When we find the best fit, we determine the angle and know the object’s location and orientation based on the stored 3D model,” he said.

The larger the 3D model, the more processing power is required to extract 3D data from a 2D image. “It’s easier to use a single surface that can be treated as a 2D planar object - also called 2.5D vision - but in the case of an engine block that may be turned upside down, you need the full 3D model data to determine the object’s location and orientation in 3D space.

With HALCON 10, MVTec was able to increase the speed of its algorithms by 2x to 3x running on the same computing hardware as the previous version. But it’s improvements in both algorithm efficiency and processor speeds that are contributing to the growth in 3D robotic guidance.

“Overall, there is no single approach that always works,” says Eckstein. “If you’re doing bin picking, or picking parcels from a conveyor belt, or tracking moving parts with a robot, you need different image processing methods. That’s what we offer: different methods that an engineer can use to optimize each application.”

Steps to the Future

As one might expect, the demos at this year’s Automate 2011 were mainly limited to large robotic workcells with repeatability and precision measured in inches or fractions thereof, rather than microns. To take the next step will require several developments.

“It’s interesting that you have vision technologists that know vision, the algorithms, and how to make a field of view do what they need it to, but when you try to cross over to the motion side, you realize there’s a gap of knowledge and experience there,” says EPSON’s Baratti. “There’s still a learning curve between the guys doing the best in vision and those in robotics.”

Unfortunately, a holistic, vision-guided programming simulator that automates all these functions isn’t available today, and may never be. Users and workcell designers are used to programming their robot systems separately from their vision systems - even companies that provide both automation technologies. But as vision systems get better at mapping out a 3D world, it could significantly change the way robots are programmed today. Rather than telling a robotic simulator where pallets, conveyors, and other production systems are located, a 3D camera on the robot could simply map out the full 3D environment, and continue to calibrate and update the environment on the fly.

And while most of these future developments fall within the realm of software, improved optics with autofocusing capabilities and internal processing for 3D coordinate extraction could greatly simplify the process while expanding robotic-guided automation to the micron level necessary for electronics and medical devices, among other applications.

“Visual servoing has been considered an art more than a science for a long time, but the demonstrations at Automate 2011 showed that it’s a commercially viable product today for many applications,” adds EPSON’s Baratti. “Once upon a time, single-camera 3D vision wasn’t doable either. Now look where we are.”