Getting Inside Machine Vision’s Head

Using a machine vision system is like being a parent. Teaching your “child” the wrong data can yield unexpected and unwanted results.

To develop a successful machine vision application, you have to understand how a computer thinks – or more to the point – how the machine vision industry and its suppliers have developed machine vision systems to process information. It always starts with teaching the system about the object under test, and then programming the system to make the right decisions.

Starting With the Product
“This is critical to Teledyne DALSA,” explains Ben Dawson, Director of Strategic Development at Teledyne DALSA (Billerica, Massachusetts). “We’re always looking at how to make teaching the system and decision making easier. The challenges are to balance simplicity for the user, the system’s ability to solve vision tasks, and the cost of computation.”

System designers typically have three ways to teach machine vision about the product it will one day inspect: electronic CAD files and models; images of good (‘golden’) and defective parts; and, to invent a term, ‘feature selection’.  Feature selection is programming the vision system to detect and measure part details (features) that are critical for quality.  An example is the spacing of threads on a bolt, as measured by image intensity variations.  Each approach has its uses, as well as challenges and advantages. 

CAD, HDL or other engineering documents would seem to be the best starting point for setting a baseline for the perfect part.  But this excludes ‘real world’ conditions that introduce variations.  A camera doesn’t see a part as a CAD drawing.  Its view of the part can drastically change due to ambient lighting, vibration, movement, rotation and many other issues that impact how the object appears to the camera from one instant to the next.

“There’s a lot of effort that has to go into using a CAD drawing or HDL part description, and it’s mostly done in the semiconductor and electronics industries where companies have large engineering staffs and can amortize development cost over millions of parts,” explains Teledyne DALSA’s Dawson. This approach can take a lot of engineering time because, “…you have to build the model and the abstract expression of the how the part should look.  The camera image doesn’t look much like the CAD model because there are part reflections, surface structures, and textures. So you have to layer texture and lighting onto your model, or have smart algorithms to extract the right information and disregard natural and acceptable variations in the part.”

Templates, or images of the part itself, take the ‘teaching’ out of the abstract and put it one step closer to the real world products the vision system will evaluate. “The big advantage [to using an image template] is that the user does not have to specify any explicit geometric description,” explains Wolfgang Eckstein, Managing Director at MVTec Software GmbH (Munich, Germany). “Besides this, all other parameters, like edge contrast, image noise or other optimization parameters can be derived from the same template. The user only has to specify intuitive parameters, like maximum rotation or scaling. By offering multiple templates, even these parameters can be derived automatically.”

“Similar is the golden template used for print inspection,” continues Eckstein. “Here it is very common to provide multiple samples to automatically determine acceptable variations of the pattern. Quite useful is also the option to add samples with a defect to narrow down the parameters more precisely. In the next years we will see this idea of ’teaching by sample‘ also in other applications areas. A typical candidate is the defect detection. In this case the system is being offered defects of different classes. From this, one automatically derives a method to locate the defect, to classify it to its specific class, and optionally a defect grade. Besides the ease of development and use, it has a big advantage for quality assurance - all defect samples are stored explicitly, so the whole inspection process can be tracked reliably. The aim is to get rid of inputting thresholds and other non-intuitive parameters. The software either has to derive them automatically for good and/or bad samples or from other information in the image.”

Certified System Integrator Program

Set Yourself at the Forefront of the Global Vision Market

Vision system integrators certified by A3 are acknowledged globally throughout the industry as an elite group of accomplished, highly skilled and trusted professionals. You’ll be able to leverage your certification to enhance your competitiveness and expand your opportunities.

GET CERTIFIED

Learning image-based templates work best when the vision system developer has a large sample of images of manufactured products with and without defects. “One problem is that customers tend to send a couple of images that weren’t taken in the lighting you’ll use for the vision system,” adds Teledyne DALSA’s Dawson. “And the images often don’t have all the defects the vision system will have to find during production. Golden parts work if you don’t have a lot of natural variation in the product, but if you’re inspecting vegetables, for instance, forget about the golden part.”

Most vision applications use image template methods to find parts in the camera’s field of view.  Once the part is found, the vision system measures selected features, such as part dimensions and texture, and comes to a decision on the part quality.  “‘Feature selection’ is currently the most common way to teaching a vision system about a part,” says Dawson.  “So we try to make it easy and intuitive, for example, by letting users graphically place feature measuring ‘tools’ on the part image.  In essence, you are building a very abstract model of the part and that requires users to select and place the right features.”

Just Decide, Darn You!
Making a final ‘pass/fail’ determination for a product can be viewed as a two step process: first, find and classify potential defects, then make a final decision based on acceptable tolerances and business criteria about the cost of the defect. Model- or template-based methods often include a classifier that learns from the model’s parameters.  As a simple example, if the model includes part intensity variance, then parts that vary some amount from the learned variance of good parts are classified as bad.  This is easy to use but you are limited by what the model or template measures.  ‘Feature selection’ methods give you many more measurements to use in classification, but require that you program the classification. This is often a simple matter of applying thresholds to measurements and logically combining the results. For example, the classification might call a part bad only if the intensity variance in one area is larger than in another part area.

“For example, finding and evaluating a scratch is a common problem,” explains Dawson. “The vision system takes measurements that look for linear structures in the image that could be scratches. Then you compare these measurements with context knowledge and classification rules learned from a large sample of good and ‘scratched’ part measurements to estimate the probability that this part is scratched.”

Many vision systems stop with classification, but according to Dawson, good vision systems take it one step further into decision theory. “In a perfect world a classifier would make perfect decisions, but in practice, the world is fuzzy and classifiers have imperfect knowledge. So you have to bias the classifier’s decision based on the cost of defect ‘escapes’ and the cost of false alarms – rejecting good product. If I’m an automotive parts supplier for Honda, and I allow a defective part to escape into their production line, the potential downside is huge. Honda won’t just reject the part; they’ll reject the entire shipment, and possibly add fines or costs for auditing my production. So I’ll bias the classifier so the chance of a bad part escaping is very low, but at the cost of more good parts being rejected.”

While teaching a machine vision system remains a technical challenge that often requires special imaging and manufacturing expertise, the body of knowledge contains in each image processing software package continues to grow. Like OCR and OCV before them, more suppliers are bundling software algorithms into tools that walk designers 80% of the way through a solid system design…but in the end, designing a successful system for all but the simplest of vision problems still requires a ‘village’ comprised of imaging specialists, product design, and manufacturing engineers to keep the system on the straight and narrow – just like your teenager back home.