3D Vision Thinks Outside the Envelope

Ask a child to find Waldo hiding on a page in a book, and you’ll keep them busy while you drive to the grocery store. Ask them to find their best friend on the opposite side of the stadium at a college football stadium, and you can probably make it to the concession stand and back with time to pick out a new football jersey.

The point is that while it’s hard to find a feature in a 2D image, it’s even harder to find it in the ‘real world’ with all three dimensions. And when the scene is so large you can’t see it in one look, teaching a machine to find features across multiple mosaicked images is a challenge for the new millennium. Luckily, machine vision specialists are working hard to simplify the basics of 3D vision while adding new capabilities for large scenes and applications that demand more than just finding Waldo. Some want you to read his shirt, too.

K.I.S.S. It Better
“2D [machine vision] is hard due to all the factors that can cause variations in part appearance: lighting, contrast spatial relationships, etc.,” explains David J. Michael, Ph.D., Director of Core Vision R&D at Cognex Corporation (Natick, MA). “3D is even harder.”

It wasn’t 10 years ago that most 3D systems were custom made. “If a customer was using structured light to do volumetric measurement of the inside of a combustion chamber, it required a system integrator to build the system at very high costs,” says John Lewis, Market Development Manager at Cognex. “Now, pre-calibrated sensors like the DS1100 make that application much easier to solve. And the system is more easily tailored to accomplish a variety of applications.”

“For applications such as bin picking, robot guidance, and meat cutting, for example, there really is no replacement for 3D machine vision,” adds Bettina Hörmann, Media Communications Manager at IDS Imaging Development Systems GmbH (Obersulm, Germany). “The key is to make 3D vision easy to use and cost-effective by offering a fully integrated solution, such as our Ensenso N10 stereo vision camera solution.

“The Ensenso API provides telecentric height maps which are already combined from the full 3D data of multiple cameras,” Hörmann notes. “Through this concept the software hides the entire complexity of multi-view geometry and reduces the point clouds to a simple, metric matrix of Z values. This reduced representation is sufficient for the majority of 3D applications and at the same time simplifies data processing dramatically by allowing the use of well-established 2D algorithms. We think that such simplifications will essentially help to develop a whole bunch of completely new application areas.”

And it’s not just about making a 3D measurement. “It took 2D machine vision a long time to conquer the four verbs of the automation value chain: scan, visualize, measure, and control,” explains Terry Arden, Chief Executive Officer of LMI Technologies (Delta, BC, Canada). “And it took 2D vision even longer to put it all into the first smart camera and make it easy to use. Now, through fully integrated 3D systems such as LMI’s Gocator, we’ve raised the bar on the user experience with web browser technology so customers can easily access the benefits of 3D over 2D.”

More than 3D
The machine vision market has offered highly integrated 3D vision systems for several years, so what’s new today? The answer lies in adding features that simplify existing in-line applications while opening up new markets to 3D vision.

Within the last year, Cognex introduced VisionPro® 3D-Locate™ software for its PC vision systems, as well as a new line of dedicated 3D displacement systems, the DS1000 series. Both make it easier for automation customers to adopt 3D machine vision and combine it with other inspection steps.

“VisionPro 3D has been very successful in the automotive industry because it tackles the difficult 3D problem in a step-by-step manner to ensure that customers have every opportunity for success,” explains Michael. In fact, Kuka Robotics Corporation (Augsburg, Germany) has selected VisionPro as the base for its new 2D and 3D VGR solution.

According to Michael, VisionPro software addresses all three aspects of 3D calibration: It directly supports intrinsic calibration, which estimates the camera sensor and lens parameters themselves. It also handles extrinsic calibration, which means calibrating the position of the camera in the world, whether it’s mounted to the robot end effector or fixed somewhere in the robot cell. Finally, it supports hand-eye calibration, which is how users convert vision results into the robot’s motion coordinate system.

“Calibration is important for all vision systems,” adds Michael, “but especially for 3D systems that need to make accurate measurements for robot guidance and gauging applications. That’s why the DS1100, the first in the 1000 series, comes fully calibrated from the factory and includes important functions like OCR [optical character recognition] for industries that produce molded parts, such as tires. If you set the camera up with a working envelope as specified, the DS1100 needs no calibration and is ready to go.”

LMI Technologies, a specialist in 3D sensor solutions, is tackling a new set of “off-line” 3D applications by combining 3D meshing and visualization software from recent acquisition 3D3 Solutions (Vancouver, Canada) with its well-established Gocator 3D sensor line.

“We think of ‘in-line’ applications as dealing with a part moving through a factory,” explains Arden. “But there’s an ‘off-line’ task where the focus is building a high-quality 360-degree digital version of reality, possibly with color, for applications in entertainment, medical, archaeology, reverse engineering, and rapid prototyping or 3D printing applications. The acquisition of 3D scanning software company 3D3 Solutions gives us an industry-leading toolset for aligning, stitching, and combining 3D point clouds from our latest stereovision Gocator using structured white light. 3D mesh processing is very hard to do in a seamless, intuitive way for the end user — especially if you add color blending. 3D3 Solutions’ mesh software does all this very well and on a PC, so we can push our sensor lines into off-line applications while bringing the 3D alignment and deviation map software capability to industrial applications.”

Arden is excited about rapid prototyping driven by the 3D printing industry. Rapid prototyping systems are being touted as the next great disruptive technology on par with personal computers. “By combining our sensors with 3D3 Solutions’ FlexScan3D software, we have all four of the ‘verbs for success’ in the automation value chain: scan, visualize, measure, and control,” he says. “And more to the point, it’s not just about solving these problems but creating a solution that is intuitive and easy for the user. That’s been LMI’s focus when it comes to bringing 3D vision to the marketplace.”