3D Vision Technology Advances to Keep Pace With Bin Picking Challenges

Innovative machine vision technologies have helped increase productivity and efficiency in many different industries, while allowing human workers to perform more meaningful work elsewhere within a company. In manufacturing and e-commerce settings, for example, many companies deploy robotic bin picking solutions to automate material handling tasks. Doing so presents challenges but also adds tremendous value when successfully implemented. This article looks at the some of the latest advancements in bin picking, specifically in 3D imaging.

Picking Multiple Part Types

A bin picking system typically consists of a 3D machine vision system, a robotic manipulator, and software. The system helps pick up objects from a container such as a bin and place them onto another area such as a conveyor or pallet or into a process. While these systems help streamline production and protect workers from repetitive and potentially hazardous tasks, bin picking presents challenges because parts are loose and randomly piled in the bin, making it hard for the system to differentiate and pick up the parts.

Bin picking challenges go beyond just randomly oriented, loose parts, according to Øyvind Theie, Vice President of Product at 3D imaging company Zivid.

“The demand for quality and speed in bin picking is remarkably high because the task is not to just match but outperform human capabilities across a wide span of applications,” he said. “For most of our customers, delivering high quality bin picking capabilities over time requires 3D cameras that can see everything with low noise, high resolution, and high accuracy — regardless of shape, size, reflectivity, and color — with minimal efforts and failures.”

To that end, Zivid has worked to make its cameras more accurate, robust, and versatile to meet these needs. Over the course of a year, CapSen Robotics, one of Zivid’s customers, has used the Zivid Two camera to perform tens of millions of pick-and-place operations involving different objects in a 3D vision-guided medical kitting application. Seeking high-resolution image capture and processing along with ultrafast cycle times, CapSen found that Zivid’s camera provides the 3D sensing accuracy the application requires.

“The sheer number of different medical products that could be picked from a bin and placed within a carton is truly vast, anything from pipettes, syringes, and swabs to medicines in packets, tubes, or bottles,” said Mark Stevens, Director of Business Development at CapSen Robotics.

With the Zivid Two camera, CapSen achieved a combined image capture and object detection speed of under 0.5 seconds and an average cycle time of 4 seconds per pick. In addition, thanks to the camera’s artifact reduction technology, native color operation, and high dynamic range, CapSen’s solution has operated error free, 24/7, while picking a wide variety of matte, glossy, semitransparent, and plastic-wrapped products.

Adapting to Customer Needs

No one-system-fits-all approach exists yet, since so much of an application depends on the objects and the environment, said Martin Hennemann, Product Manager, Ensenso 3D Cameras, IDS Imaging Development Systems.

“Surface materials, reflection, and transparency remain major issues that 3D sensing has to cope with. At the same time, challenging coverage, resolution, and performance requirements have to be met,” he said.

In addition to offering flexible 3D camera models with fast and accurate 3D point clouds — covering small to large volumes — IDS takes feedback from customers all over the world and develops improvements and features to keep pace with the growing needs of end users and systems integrators alike. For example, the company developed the new Ensenso PartFinder localization algorithm for challenging objects and scenes. PartFinder allows users to create a model and search for parts, and when the search is completed, the software dialogue shows the number of detected parts at the bottom as well as in a 3D visualizer that shows good matches as green and less confident matches in orange.

IDS cameras see use in bin picking applications of all types. At psb intralogistics in Germany, the “autopick” fully automated system consists of a robot with gripper and two Ensenso 3D cameras. In this application, the robot automatically picks small parts from bulk materials and places them into target containers at 300 to 500 parts per hour, depending on the object. In another application, Dutch company Fizyr developed an automated vision solution featuring up to four Ensenso cameras that enables logistics automation in various conditions and applications, such as item picking, parcel handling, depalletizing, truck unloading, or baggage handling.

Innovating With Deep Learning, Novel Sensor Technology

When a bin has one type of object with a fixed shape, bin picking is straightforward, as CAD models can easily recognize and localize individual items. But randomly positioned objects can overlap or become entangled, presenting one of the greatest challenges in bin picking. Identifying objects with varying shapes, sizes, colors, and materials poses an even larger challenge, but by deploying deep learning algorithms, it is possible to find and match objects that do not conform to one single geometrical description but belong to a general class defined by examples, according to Andrea Pufflerova, Public Relations Specialist at Photoneo.

“A well-trained convolutional neural network [CNN] can recognize and classify mixed and new types of objects that it has never come across before,” she said. 

Photoneo trains its CNN on large datasets of objects to allow the software to recognize mixed object types, including cartons, parcels, hang-on boxes, tubes, ropes, bags, and even food or organic items.

By combining its PhoXi family of 3D scanners and MotionCam-3D camera with robotic intelligence, the company aims to provide flexible automation solutions for applications requiring robotic recognition and pick-and-place tasks, according to Pufflerova.

While PhoXi 3D scanners provide high resolution and accuracy for scanning static scenes, the  company’s MotionCam-3D uses a unique technology the company calls “parallel structured light,” which enables the capture and high-quality 3D reconstruction of moving scenes. This, according to Pufflerova, has enabled automation in a whole new range of applications. Photoneo’s recent customer projects include a piece-picking application in the automotive industry that enabled automated car body production, the picking of heavy metal spheres, the picking and placing of randomly positioned pipe parts, and the picking and placing heavy furniture parts.

An Autonomous, Stand-Alone Solution

For its part, Universal Robots (UR) helped enhance its cobots’ capabilities in several applications, including machine tending when it introduced its ActiNav flexible machine-loading solution. ActiNav combines intelligent vision with autonomous motion control software and UR’s cobots to offer precise, consistent, agile bin picking capabilities, according to the company. At full-service molding company Allied Moulded Products, for example, the company recently faced labor shortages and needed to keep processes running while protecting employees during the COVID-19 pandemic. To do so, the company opted to use ActiNav, which Manufacturing Engineer Technician Nate Gilbert said was the first system that fit within the available space without creating extra obstacles.

“Just the simplicity of it picking parts out of the bin and onto the conveyor is what we were really looking for, and ActiNav seemed to meet that demand,” said Gilbert, noting that ActiNav constitutes a major improvement to the vibratory feeders that are often used for part positioning.

For Allied Moulded, providing ActiNav part data and teaching the robot to pick from the bin were straightforward processes. “We just touch the robot to each part of the bin,” said Maintenance Group Lead Nathan Wells. “You do a few spots of that, and ActiNav learns the bin itself. I would say after the first day of setup, it took maybe a shift to learn how to actually do the programming, and after that we were good on our own.”

Deploying UR’s system allowed Allied Moulded operators to take on more meaningful work, becoming trainers and mentors on the system and contributing design ideas for future automation projects.

Four Benchmarks for System Success

Four main considerations to make when it comes to assessing the efficiency of a bin picking cell include cell cycle time, bin emptying rate, pick-and-place accuracy, and the ability for the cell to be reused in other applications, according to Sébastien Paille, Head of Sales & Marketing, Visio Nerf.

“These four criteria are intrinsically linked,” he said. “For example, a suboptimal emptying rate may require the addition of mechanical systems to shake the bin or a change to more appropriate tools to complete the task.”

He continued, “Integration of such elements extends the cycle time and penalizes the flexibility of the cell. Appropriate bin picking cell sizing is essential and involves finding the right balance between these criteria.”

Beyond these factors are important choices such as the type of 3D sensor, which Paille says is crucial when it comes to designing the bin picking cell.

“No matter how good the processing software may be, a 3D sensor providing a bad point cloud will negatively impact the performance of the system, especially on materials that are shiny, matte, glossy, black, and so on,” he said. “Several different 3D sensor types exist for the acquisition of a point cloud, and each of these sensor types are based on different physical principles that depend on the use case under consideration.”

Robustness represents another key consideration in 3D bin picking. The 3D sensors used in the system must be able to work in hot or otherwise harsh conditions in the vicinity of the robotic cell. Dust, oily air, sand, or metal particles cannot damage the bin picking system, so protecting all components represents another important step, according to Paille.

Visio Nerf offers 3D cameras based on proprietary cameras and LED projectors that also incorporate FPGAs and high-quality lenses, all enclosed within a robust IP65 enclosure safe for deployment into harsh environments.

“By combining two 4-megapixel cameras with our own high-power [80 W] LED projector inside a protective housing, our 3D cameras can localize/identify objects located at a depth of more than 1 meter, precisely at 0.5 millimeters, in a range of demanding application types.”

Bin Picking, Beyond

Certain applications in machine vision and automation will help drive trends in bin picking into the future. On-arm 3D cameras, for example, will usher in a new paradigm in bin picking, according to Theie.

“The ability to perform bin picking tasks with on-arm 3D cameras at the same cycle times as with traditional stationary 3D cameras — but with fewer miss picks, higher flexibility, smaller footprint, lower costs, and less maintenance — will advance bin picking applications even further.”

Additional 3D Bin Picking Solutions  

In addition to the companies mentioned here, several other players exist when it comes to developing machine vision solutions for bin picking applications, including Fanuc, Pickit, Solomon, Canon, Apera, Keyence, Sick, and more. Learn more about 3D vision companies here: www.automate.org/vision.