PhoXi® 3D Scanner in a Bin Picking Application

In recent years, many vision system vendors have introduced 3D scanners and cameras into their product portfolios. System integrators have been continuously seeking the most natural and efficient hardware integrations to raise the demand for this type of sensors. Besides metrology, quality control and trajectory correction; bin picking application is the most common and breakthrough, as the financial payoff of even a very basic application is precious. While the idea of the robotic arm picking of randomly placed objects from a container is rather simple to imagine, real implementation brings big challenges. The article elaborates on such challenges experienced in an actual implementation, and the capability of hardware and software developed by Photoneo to tackle the issues. Only applications in static environments are considered. All the experience mentioned was being gathered during an installation of the bin picking solution by our team for a producer of hydraulic components. Contractual cycle time was set to 7 seconds. It was met successfully, solely by one PhoXi® 3D Scanner and one ABB robot. Now, the 7-second cycle time covers picking a small metal object from a big scanning volume loaded with 10 000 parts, nailed by oriented placing. The area ratio of the size of parts being picked to the bin size is approximately 1:900.

Vision system selection
As one may correctly assume, the main pillar of each bin picking application is the quality of a 3D vision system and related quality of the point cloud it generates. It is fair to mention that some system implementation can still be worked around with a 2D vision system. These applications are usually combined with mechanical systems like vibration feeders, used to separate objects into a very specific pose which can be recognized by the 2D camera. The specific orientation is a result of cyclic shuffles and the oriented pose occurs after a random repetition of each procedure. With this approach, cycle time increases with the decreasing number of parts, so it is problematic to unload the container completely. Moreover, such a solution consumes more space, which is an issue clients complain about and the reason why they do not favor 2D in the end. One should also consider the fact that a combined investment for the feeder and the 2D camera is comparable with costs for a 3D scanner. Due to the limitation of this technique, we will omit it herein.

In a 3D vision system application, the quality of a point cloud is always a trade-off among various scanner parameters. The best practices of successful 3D systems have proven that it is rather complicated to cover a full range of applications with a single device. Very often we see categorizing the applications by scanning volumes, while a hardware equipment of the 3D scanner in each category can be very similar. Most of the well-established vendors on the market have built their systems on the structured light principle and emitting the patterns. Most of the scanning devices are able to deal with applications with smaller scanning volumes. However, they have troubles with volumes of a euro pallet size or with a size even bigger than that (mostly due to the depth of field restrictions). In general, bin picking in an extended volume is a challenge, as the point cloud density is lower and the noise level higher, resulting in a decrease of an absolute accuracy. Therefore, the quality and depth of field of PhoXi® 3D Scanner XL is crucial when the bigger scanning volume is combined with a proportionally smaller object to handle. Bigger scanning volumes require a stronger source of light or, alternatively, a longer exposition time to secure the reasonable point cloud quality.

Implementation
For the implementation needed by the producer of hydraulic components, Photoneo® PhoXi® 3D Scanner XL had been selected. The device can serve a scanning volume of 2400x1800x2200 mm. Images are captured by a 3.2 megapixel sensor and an average acquisition time is approximately 800ms. Another unique characteristic is a lightweight (up to 1.2 kg) but durable carbon body with a minimal thermal expansion.

Selecting the object suitable for the bin picking application
Parts to be picked per se are a factor for consideration before deciding on any implementation of the bin picking solution. First, for now, highly reflective and transparent objects are more or less out of the application scope from the material point of view. Material of the object is decisive for the quality and usability of a 3D scan as well as for the process of localization, object extraction from a container and oriented placement. Secondly, in a bin full of parts every object becomes a possible source of collision with a robot arm. In the case of two or more parts locked together they may also collide with the body or accessories of the robot cell. Thirdly, a deformity of objects as a result of the field of gravity is also challenging in terms of localization and oriented placement. Last but not least, the integrator also needs to think of a shape of the part. Highly asymmetric parts may get stuck in a twin part and overcharge the robot arm load.

Implementation
The object selected for the implementation is depicted in the following CAD model. The customer had requested 99% rate of bin unloading with 7-second cycle time. During feasibility studies conducted at Photoneo, the probability of the parts locking together was detected to be minor. However, we tackled a problem with interreflections, as the metal shiny part was on the edge of feasibility. The condition to extract the object from any pose brought a challenge with a gripper design. The question of the gripper design is addressed thereinafter.

Cycle time
For industrial process implementations, cycle time is one the of main measures of success. Simply put, cycle time is composed of the time required for scanning, localization algorithm, robot path planning and the time consumed by the robot arm operation. Based on experience in the field, one should avoid attempts to shrink the time available for 3D scanning. With a longer time for the image capture procedure, the system is able to output a point cloud with higher density, less noise and higher accuracy. This, in turn, enables the localization algorithm to identify the object successfully in a shorter time. It is worth to mention that a single scan usually captures enough objects for more than one system cycle. Localization algorithm can continuously search for new objects asynchronously in the background. Exposition time for the best in class scanners depending on the scanning volumes ranges from 300 to 800 ms. Scanning time closer to 1 second does not limit the application with the cycle time still lower than 2 seconds. With this scanning time, asynchrone localization execution and parallel deployment of multiple robots and scanners, pretty low cycle times can be achieved.

Implementation
PhoXi® 3D Scanner chosen for this implementation works with the exposition time starting at the level of 800ms. The localization has been running asynchronously and there has not been a need of any additional scanners or robots in the application for the producer of hydraulic components. Contractual cycle time is at the level of 7 seconds, achieved solely by our scanner and one robot, even in the case of the big scanning volume loaded with 10 000 parts, topped by the oriented placement . One of the unique features of our vision system ensuring the success is called Smart Memory. While the robot arm is leaning to pick a part up, the scanner still senses asynchronously and saves the perception of the space around the part to be picked to its memory. While the arm is putting the part to a desired location, the scanner sends a signal for another path planning. One may wonder, what if the parts are slightly moved as a result of the previous pick, and the actual placement doesn’t fit the memory trail of the scanner. Photoneo’s answer is that we’ve implemented a special solution into our vision system, able to analyze the changes of positions of objects very quickly, and concentrate on the parts that did not move. The logic behind the PhoXi® 3D Scanner Smart Memory can thus be explained in a way that it works continuously like a human memory, while some of the actions in a given time during the process are able to program several future steps, not just one.

Object localization
Object localization is a crucial part of every bin picking solution. Precision of the localization algorithm should be determined according to the speed and orientation. Implementations requiring a high precision of oriented placing are designed to utilize the CAD model of a part. For applications with objects of a certain shape variation, the CAD model is not the first option to consider.

Implementation
In our case, the approach with the CAD model was chosen. At first, we analyzed the CAD model and extracted a vast network of connected features that served as an intelligent object representation. For the application, more than a million of such features were used to map the part, creating a mathematical model of the part. This enabled us to compare mathematically similar objects and decrease the time of localization. The algorithm pre-scans the scene to look for a bunch of potential object occurrences. In the second run, it compares the parts of the scene with the remembered mathematical model that can not only say if the part is similar to the model, but it also computes an approximate position and orientation of the part (Smart Memory). While the fine alignment of the part is necessary for the oriented placement, we use multiple approaches to fine-tune the position to the part. While the localization needs to process thousands of inputs, a right balance between speed and accuracy has to be chosen. Higher accuracy can make a fine grasp more simple, while faster localization can create more candidates to choose from.
 
Tool - gripper design
Every object extraction procedure is only as efficient as the robot tool point. There is a range of grippers out there designed to suit as many as possible applications. For the majority of metallic part extractions, localization precision can be supplemented by a plain force of a gripper with an active magnet. These applications are reasonably robust but the precision of the oriented placement is limited. Likewise, vacuum grippers are universal in applications for objects with a smooth surface but require periodic maintenance. In order to secure the desired precision of a bin picking application, the gripper should have an application-specific design. Investment into the customized gripper design usually pays off, as it leads to lower maintenance costs and less service time needed.

Implementation
In the implementation for the customer with the described part, we were solving an issue of how to grasp the object from any position. When the part being picked was positioned with the smaller opening upwards, a small two-finger gripper have been used from the inside of the smaller hole. When the object was oriented with the bigger opening upwards, understandably, a bigger two-finger tool have been used from the inside of the bigger hole. For grasping the object while lying in a horizontal position, a vacuum gripper have been used.
 
Photoneo Bin Picking solution provides an option to define the invariance of the tool. The vacuum gripper used is specific because of the invariance in z-axes. In the real application, it doesn't really matter what the real rotation of the tool point with the vacuum cup is when making a contact with the object. The sampling rate of the invariance was set at the level of 15 degrees. Algorithm then automatically calculated plausible poses of the tool point relative to the object. The invariance step was set with respect to the calculation time of inverse kinematics. From a certain level of the step of invariance, the probability of a suitable pose does not increase while the calculation still time does.

Lastly, it is worth to mention that we decided to mount the gripper under an angle that can help to avoid collisions and extend a reach. The small design change significantly improved the extraction from the corners of the bin. The bending and the extended length of the gripper as per the illustration also helped to reach the very bottom of the bin, as the reach of the robot arm itself was only 1.45m.

Implementation of the robot path planning
Once the selection of the right object for next pick and the appropriate gripper were secured, the actual trajectory for the robot arm represented the last challenge. Photoneo engineers decided to utilize one of the available open source path planners. Photoneo’s Bin Picking solution utilizes the ROS (Robot Operating System) and one of the motion planners available there. In this case, the testing led the team to STOMP (Stochastic Trajectory Optimization for Motion Planning). STOMP provides a collision avoidance functionality, it also takes into account an energy for the robot movement and reduces a potential for abrasion of motors of the robot. The most crucial part of the configuration was a complete listing of potential collision objects like the robot cell, bin, scanner, etc.

Conclusion
The Bin Picking application by Photoneo works well with all the major industrial robot brands - ABB, Kuka, Fanuc, Mitsubishi Electric, Omron, Stäubli, Universal Robots and Yaskawa. For the installation described above, the client had decided to implement the ABB model IRB 1600 with a carrying capacity of 10kg and reach of 1.45m. At the time of writing the article, the installation has been in production for about a half year.