U.S., China Say Nuts to Agricultural Machine Vision

It hasn’t been an easy row to hoe, that’s for sure.

Machine vision has always been about the automated extraction of useful information from the interaction of light with physical objects. With light as one of its foundations, machine vision counts on control of light for its success, which is why machine visions excels in controlled environments, such as labs and production floors where the vagaries of ambient light can be controlled. This is one of the reasons that machine vision technology isn’t as common in farms as it is in automotive plants.

But as demand for foodstuffs grows along with the number of developed nations, agricultural industries are looking to automation to help keep labor costs down while increasing production and maintaining high quality.

“The whole pecan market has really increased during the last seven to eight years,” explains Roger Holt, president and CEO of Inspection Masters LLC (Las Cruces, New Mexico), “primarily because China has discovered nuts. At one point, pecans were down to 78 cents per pound. Now, they’re about $4 per pound, which is causing people to plant pecan trees everywhere they can across the southern United States, as well as Central America and South America. In the Southwest U.S., the chili pepper capital of the world, farmers are planting pecan trees in their chili fields. South Africa is becoming the world’s new grower of pecans and macadamia nuts.”

While working as an engineer at NASA 14 years ago, Holt’s father in law — a pecan grower — asked if there was any way automation could help reduce his labor costs in light of a dwindling number of willing seasonal workers. “It was getting harder to find people willing to spend hot weekends in the sun, sorting in-shell pecans from those still in their husks,” Holt recalls.

Large food-processing equipment OEMs already had advanced systems for separating shells from the nut meat inside, at a starting cost of around $250,000. But for small growers, manual labor was the primary way for separating ripe nuts from nuts still in their husks or rocks and other "defects" of similar shape.

After four years of development, Inspection Masters launched its first nut-sorting machine vision system. Today, Inspection Masters uses “dual wavelength” area array cameras from JAI that can acquire both color and grayscale near-infrared (NIR) images of the nuts as they fall from a chute at a rate of about 5 tons per hour.

Holt explains that the color images are important because the pecans will change color from the beginning of the harvest season to the end, and the machine vision system needs to be able to adapt to changing requirements to separate nuts in husks to the quality nuts without a husk. Right in the middle of the growing season, the nuts and husks will be about the same brown color, which is where the NIR images come in. Nuts in their shells are harder on the outside then nuts in husks. Hard shells reflect more NIR light, making them appear white in the NIR image; nuts in the husks absorb more light, making them appear darker in the NIR light. By tracking the position of each, Inspection Masters can use high-speed air jets to reject nuts still in their husks. The process is pretty much the same whether it’s pecans, walnuts, or macadamias, Holt adds.

Inspection Masters is currently developing a modular system that will work on a conveyor, increasing throughput from 5 tons/hour to the industry's current benchmark of around 20 tons/hour.

Real-Time Grading, Sorting
While Inspection Masters is adding more cameras to accommodate customers’ needs for higher throughput, real-time Windows specialists Kithara Software (Richmond, Virginia), working with their startup client VMEK, have combined a real-time Windows operating system extension environment to a machine vision sorting operation to include grading as well as sorting.

Kithara has been developing real-time PC-based software environments for PCs for the past 17 years. Three years ago, Kithara turned its attention to the machine vision market. That led to the company’s first real-time GigE Vision driver that allows users to acquire images directly from the camera and feed them into a real-time kernel running on a Windows-based machine.

“It’s not a virtual machine but an extension of Windows running in parallel for real-time image processing and EtherCAT for real-time downstream automation control,” explains Kent Lovvorn, director of business development at Kithara Software. Today, Kithara’s RealTime Suite can accommodate OpenCV and Halcon image-processing libraries.

With this new functionality, Lovvorn says Kithara’s RealTime Suite can achieve less than 10-microsecond jitter and 100% downstream determinism through EtherCAT. In the case of the waterfall sorting application, this allows the machine to identify contaminants but also to make spatial and color measurements on each individual object as it falls at a rate of 4 m/s, or 2 tons/hour. “We can also estimate the weight of the object based on volumetric calculations,” adds Lovvorn.

UAV Hyperspectral Imagers Could Boost Ag Production
Hyperspectral specialists Resonon (Bozeman, Montana) just fielded their third almond inspection system using a 240 spectral band hyperspectral imager. The camera head puts a diffraction grating in front of a monochromatic sensor. The sensor then essentially becomes a line-scan camera with spatial location along one axis and intensity of the spectral band along the other axis. The hyperspectral imager can differentiate between almonds and very similarly colored defects, including almond shells and hulls. A robot picks out the defects labeled by the hyperspectral imager and accompanying software.
 
“What we’ve done is make hyperspectral imagers affordable and trainable,” explains Resonon’s research scientist, Casey Smith. “The ability to train the software makes the system adaptable for a wide variety of food-sorting applications."
 
While the almond inspection system was developed for product purposes, most of Resonon’s customers in the agricultural space use airborne hyperspectral imagers. “You can tell a lot more about what crops need water or pesticides from the air than can you can by walking the rows,” says Smith. “We’re pretty excited to see what the [Federal Aviation Administration] does with [unmanned aerial vehicles]. Someday, most farmers will have UAVs that fly their crops regularly to give them updates on crop health.”