Thermal Imaging and Machine Vision

One of the most interesting technology advancements has been the vast improvement in the price/performance of infrared imagers and cameras over the last couple of years. While heretofore these cameras were awkward to use and just plain cumbersome for shop floor use largely because of cooling requirements (not to mention expensive), today one finds infrared cameras designed to be compatible with shop floors. While still expensive the prices are coming down rapidly. With the convergence of breakthroughs in the underlying infrared camera technology it is now reasonable to consider them for use in machine vision applications.

This was made apparent at the International Robots & Vision Show in June, 2003, which found four suppliers of thermal imaging cameras exhibiting at the show. This in turn piqued my curiosity to determine where these cameras are being used in industrial or typical machine vision applications. A review of the market determined that there are at least 15 companies that offer thermal imaging cameras. Input for this article was solicited from these 15 companies. Several responded that while they do indeed offer IR cameras their main market is the military and they have not found their cameras engaged in machine vision applications. In the end four did take the time to respond and provide their insights for this article:

• Dr. Austin Richards – Indigo Systems
• Jon Chynoweth – Mikron Instrument Company
• Dr. Arn Adams – Santa Barbara Focalplane
• Erick Argueta – Sensors Unlimited

1.  How would you define thermal imaging or IR imaging?
Dr. Austin Richards: ‘’Thermal imaging is a technology that creates a photographic image or video sequence of light emitted by an object at terrestrial temperatures, like people, animals, cars, etc.  The light wavelength is typically 6-20 times longer than the wavelength of visible light.’‘ 

Jon Chynoweth: ‘’The ability to create a picture out of thermal profiles.’‘

Dr. Arn Adams: ‘’Thermal imaging is imaging in the region of the electromagnetic spectrum from about 3 to beyond 14 microns (where room temperature objects emit significant amounts of IR radiation).  IR imaging is performed at wavelengths longer than about 800 nm.’‘ 

Erick Argueta: ‘’Thermal imaging and IR imaging are intersecting but not identical fields.  Thermal imaging is the imaging of temperature distributions and normally uses IR wavelengths to do so, but IR imaging is much broader, encompassing any imaging system that employs wavelengths between roughly one and ten microns whether or not temperature is the parameter of interest.  For example, imaging of moisture content or chemical composition is usually accomplished with an IR system, but this would not be thermal imaging.’‘

2. What constitutes a thermal imaging-based machine vision system?
Chynoweth: ‘’Consists of an IR camera with or without a lens, a typically custom built enclosure hardwired or tethered to Ethernet or server computer with application-specific custom software and automatic decision-making. For process related applications one finds two types of cameras; 1) thermal imaging and 2) radiometric providing a pixel-by-pixel 16-bit temperature map.’‘

Adams: ‘’Thermal IR camera controlled by a computer system that makes decisions based on camera output.’‘

Richards: ‘’A thermal imaging-based machine vision system is one that has a camera directed at some process for remotely measuring the temperature or the emissions from the given process.  Sometimes the infrared emission strength or radiance (not temperature) is what is measured.’‘

3. Can you distinguish between active and passive thermal imaging systems? What does one find mostly in machine vision applications?
Argueta: ‘’Active thermal imaging systems use a laser or other means to raise the temperature of the target, usually over a small area at a given time, whereas passive thermal imaging systems sense the inherent temperature of the target.  Passive systems are the simplest, and the most common, but active systems have capabilities that passive systems do not.’‘

Richards: ‘’Definitely, passive thermal imaging is what is generally used in machine vision. In passive thermal imaging, the available light is used to look at a scene, which is heated or cooled by some process, e.g. chocolate chip cookies being baked. You can image the temperature variations induced on the product during the baking process. Active thermal imaging is when you illuminate the surface of the object under study with a continuous or pulsed light source and image the surface with a thermal imaging camera.  A pulsed system can see defects or structures below the object’s surface under the right conditions.  A continuous source illumination can reveal markings under paint.’‘

Chynoweth: ‘’In an active system one is continuously monitoring a target while heating or cooling to obtain the delta temperature. In one automotive application where composite body panels are glued to the frame, application requires creating the delta temperature by blowing warm air. Result is guaranty that glue line is continuous and without gaps.’‘

Adams: ‘’Active thermal imaging systems typically make real-time decisions based on the camera output, while passive systems generally are used as monitors.’‘

4. How is a thermal imaging-based machine vision system different from one that typical operates in the 400 - 700 nm region?
Adams: ‘’Thermal imaging-based machine vision systems generally make decisions based on thermal parameters.’‘

Chynoweth: ‘’They are very similar, operating typically at 30Hz or 60Hz with either an analog or digital output. Instead of adjusting contrast and brightness, with IR camera one adjusts gain and level.’‘

Richards: ‘’Thermal imaging-based machine vision systems see radiant energy emitted from objects at room temperature. Thermal imaging systems are usually a lot lower in resolution.  Higher resolution visible imaging is a much more mature technology and it is still cheaper and simpler to make, particularly CMOS imaging systems.  But these visible-light imagers can only see thermal emission from extremely hot objects.  The uses of these two types of cameras are generally distinct and separate.’‘

Argueta: ‘’In addition to the obvious differences in wavelength, camera technology and optics, in a (passive) thermal imaging system the target is also the light source.  The optimum type of thermal imaging system near-infrared (NIR) 1-3 um, mid-wave infrared (MWIR) 3-5 um or long-wave infrared (LWIR) 8-12 um for a particular application depends on the temperature range to be sensed.  For target temperatures near or slightly above room temperature, 3-5 or 8-12 micron systems are usually optimal, whereas for target temperatures above roughly 200 degrees C near-infrared systems are usually optimum.  On a more mundane level, IR imaging systems (whether thermal or not) are generally more expensive than visible-light systems of comparable scale.’‘

5. What is the history of thermal imaging leading up to today's applications in machine vision?
Richards: ‘’Thermal imaging systems were developed for the military in the 1960s.  Until about 10 years ago, they were large systems that were very expensive (>$100K) and consumed relatively large amounts of power.  Now the smallest thermal imaging cameras are under $10K, fit in the palm of your hand, and are becoming commodity items rather than special custom systems.’‘ 

Chynoweth: ‘’In the case of industrial applications the biggest market is for online process control. However, the challenge has been the cost of the cameras especially since typical process applications require multiple cameras. Consequently, to date most applications have been based on handheld cameras examining scenes to predict when maintenance is required. However, the price barrier is coming down quickly.’‘

6. What makes it possible today to consider thermal imaging-based applications in manufacturing today?
Argueta: ‘’Near-Infrared cameras are now very rugged and do not require external cooling like regular thermal cameras.  In manufacturing where fast acquisition times are required, the speed of acquisition in near-infrared cameras is also comparable to its 400nm-800nm cousins.  Additionally, higher resolutions are now available, from 640x480 pixels in 2-dimensional focal plane arrays to 1,024 pixels in 1-dimensional linear arrays.  The support of standards like Camera LinkTM and others make it a versatile tool that has moved from the laboratory bench to the industrial environment for several years now with a reasonable price.  Reliability is also important in industrial settings and the cameras can now withstand operating temperatures as high as 40oC standard, higher with minimal customization.  Extremely easy to use (composite video compatible) and no warm-up time required, the NIR cameras make it a very attractive solution for process control needs.’‘

Chynoweth: ‘’Thermal resolution is greater than it was 3-4 years ago, the package size of the camera has been reduced and availability of software platforms have yielded greater ease of use.’‘ 

Adams: ‘’Lower cost and higher performance are enabling an increasing number of applications.  For example, we've (Santa Barbara Focalplane) recently introduced our ImagIR LC computer-based camera system for US$10,000; it comes with a Windows PC, all cables and software and has a 256x256 element sensor that operates at over 70 Hz frame rate with better than 20 milliKelvin NETD performance.  Higher frame rates (>10 kHz) and larger format sensors (>1Kx1K) are also enabling new applications.’‘

Richards: ‘’Thermal cameras continue to come down in cost, especially when you compare them to the cost of discrete thermal sensors, in some applications.  For example, if you wanted to measure the exterior temperature of a vacuum furnace used for making steel, you could cover the outside of the furnace with thousands of discrete sensors at various points and measure the temperatures, checking for any hot spots. But it would be much more cost-efficient to remotely thermal image the exterior of the furnace, in this example, than it would be to attach and then measure thousands of thermal sensors on the outside of the furnace.   As the price for thermal-based imaging falls, the applications that didn’t make sense in the past, are now very affordable and do make sense.’‘

7. Are there calibration issues associated with thermal imaging? Are there some standardized ways that one calibrates them?
Richards: ‘’Yes, calibration is a very big issue in a lot of cases.  There are methods for calibrating cameras and we’ve found the most reliable and accurate method is to utilize two black body sources, one hotter than the subject and the other, colder.  They then bracket the temperature range that you want to measure.  Indigo Systems and WinSoft Corporation have jointly developed a skin temperature monitoring station, ThermaSTAT  which utilizes the two black body boxes as calibration sources and the tiny Omega  thermal imaging camera.  The Omega has a real-time calibration source within the camera.  It images the facial area of a human subject, remotely and non-invasively, in real time.  The system then automatically self-calibrates, the computer converts all the real-time calibration information from all three sources and if the human’s temperature is out of range of a preset value, an alarm sounds.’‘

Argueta: ‘’Yes there are.  To obtain absolute temperature readings, the system must always be calibrated.  The ‘’system’‘ includes both the camera and the target, since the target's spectral emissivity is a major factor in obtaining accurate temperature measurements.’‘
Chynoweth: ‘’Of course. Since output of microbolometers has a tendency to shift, cameras need a mechanism to compensate for the shift. In cameras today this compensation is performed automatically at some fixed frequency – every 20 minutes, for example. Absolute calibration is also required at some time interval, However, this would be a function of the application. For many applications calibration once a year is adequate.’‘

Adams: ‘’Non-uniformity correction (NUC) is necessary for all FPA (focal plane array) thermal imagers.  Temperature readout vs NUCed camera intensity is an additional calibration issue.  Typically, 2-point NUCs are performed on at least a daily basis for the most sensitive applications (e.g., when better than 50 milliKelvin resolution is required).’‘

8. Are standards like Firewire (IEEE 1394), CameraLink and USB 2 factors in thermal imaging?
Adams: ‘’Yes.  Firewire is rapidly becoming the standard for relatively slow thermal IR cameras (data rates less than about 40 MBytes/sec).  CameraLink will become the standard for high frame rate thermal IR cameras (e.g., SBF's 1024x1024 camera operates at data rates up to 320 MBytes/sec, and our new digital output FPAs will operate at even higher rates).’‘

Richards: ‘’Yes, to all.  Indigo Systems now has an optional FireWire accessory available with the Omega thermal imaging camera.’‘ 

Chynoweth: ‘’Mikron now offers Firewire-based cameras and will be coming out with a USB 2 version in September.’‘

9. What are some machine vision features that one might expect to find in thermal imaging systems today?  Specific types of software?  Specific hardware features in cameras?
Argueta: ‘’Because of the proliferation of interface standards across IR imaging and specially NIR imaging, traditional software like National InstrumentsTM (NI-DAQTM) and other packages can be used to address every manufacturing imaging need.  Together with off the shelf frame grabber image boards, they make it a very cost effective solution with all the bells and whistles that could be expected from a ‘visible’ imaging system.  Some of the typical hardware features include:

•   Analog composite video
•   Digital interface (USB, Camera Link, Serial)
•   Exposure time control
•   On board corrections’‘

Chynoweth: ‘’The features are really in the software package: ability to create region-of-interest (ROI), extract min/maxc/standard deviation, trigger alarms based on target temperature, etc.’‘ 

Adams: ‘’Computer control of the thermal IR camera with the ability to stream data to disk at the full frame rate of the camera while viewing the live display, and making real-time processing decisions.  Software development kits that don't tie the customer down to a specific set of computer hardware is necessary in most machine vision environments.’‘

Richards: ‘’Machine vision involves image processing, e.g. pattern recognition, edge detection, or circle detection.  An example might be if you wanted to find the liquid level of spray paint cans on the assembly line, thermal imaging could detect the temperature differential between the paint and the air in the can.  Or one could set up a thermal camera to remotely image the liquid level in several tanks or barrels in the manufacturing plant. Thermal imaging could look for temperature variants in glass bottles, if the imaging point was outside the specified temperature norm, the quality control process would be served.  Back end software might include image processing modules that do edge detection and filtering, background subtraction, etc.  Cameras might come with CDs of software developer’s kits that allow a programmer to easily integrate the camera into a PC-based system: ActiveX controls or DLLs that enable image acquisition and camera control.  Alternatively, LabVIEW drivers might be provided to make it easy to integrate the camera with National Instruments HW and SW.

Hardware might include environmentally sealed cameras for harsh factory floors, analog video output for easy integration with existing machine vision infrastructure, 12 VDC power requirements which are easy to meet.’‘

10. Do today's thermal imaging cameras come with features typically found in conventional cameras targeted at the machine vision market? Exposure control? Asynchronous frame reset? Progressive scanning? Electronic shutters?
Richards: ‘’Some, yes.  The thermal imaging industry is borrowing even more features from the visible cameras arena. Our thermal imaging cameras do have exposure control and in addition, Indigo Systems has lenses with a quick-change feature.  Most thermal imaging companies have ‘external triggering’ on their cameras, known as ‘asynchronous reset’ in the visible camera market.  Shutter speed, known as ‘’integration time’‘ is available, depending on specifications, with shorter or longer performance time on our infrared cameras. Indigo’s thermal camera systems feature NTSC and S-video (for analog formats) and RS-422 (digital), offering the same analog and digital video interfaces that visible cameras feature. Indigo Systems also has standard tripod mounts and standard 24V or 6V (for the Omega) power sources.  Our objective is to make infrared imaging systems that easily interface with the features of standard visible imaging systems, making the crossover more cost-effective and convenient than ever.  All of Indigo’s FPAs are ‘’snapshot’‘ mode, meaning all pixels are integrated at the same moment in time, permitting simultaneous capture of every pixel and then read out of the camera (some other cameras have rolling-mode readouts, which cannot image dynamic scenes well).  We are committed to making IR cameras as standard as CCDs, high performance and full featured.’‘

Adams: ‘’Yes, thermal IR cameras generally come with exposure control and electronic shutters (frame rate and integration time adjustment from about a microsecond to 10's of milliseconds), and asynchronous frame reset (the cameras can output sync pulses or accept input sync pulses).’‘

Argueta: ‘’Some of today's IR cameras have exposure control with electronic shutters with various exposure time control and asynchronous triggering.  Some other features include windowing for faster frame rates and low power standby modes when the camera in not heavily in use.’‘

11. What are some specific machine vision applications for thermal imaging-based systems - both online and offline?
Argueta: ‘’In the area of near-infrared thermal imaging, applications include temperature mapping of high-temperature processes such as brazing, metal smelting, and plastic container inspection.  Non-thermal applications of near-infrared imaging include moisture content imaging of paper and wood, sorting of fruit and other food products, pharmaceutical tablet inspection, bottle fill inspection through translucent or colored bottles, and plastic sorting for recycling.’‘

Chynoweth: ‘’Applications include continuously monitoring electric bus bars, looking for good/bad populated printed circuit boards, monitor plastic seal on coffee cans, torpedo ladle cars in steel looking for burn throughs, inspection of clutch plates for presence of graphite, etc.’‘

Adams: ‘’Non-destructive testing (NDT) and imaging spectrometry seem to be the largest machine vision applications for our high frame rate & large format cameras.’‘

Richards: ‘’Online – inspecting cookies coming out of an oven – inspecting glass bottles just coming out of the mold, inspecting boxes or canisters for defective seals, inspecting silicon wafers for quality assurance, etc. Offline- Any research in the lab or in the field – measuring the temperature of a burning cigarette, for example or looking at a solid rocket motor plume when the motor is in a test stand.’‘

12. What are some limitations associated with thermal imaging-based machine vision systems? Speed?
Richards: ‘’Frame rates can be comparable (down to a microsecond e.g. speeding bullet - frame captured at about one microsecond; the number of pixels is limited so it’s hard to see a lot of details. Resolution is generally lower and there are limits on the minimum resolvable feature size – because of diffraction, you cannot see something that is smaller than a wavelength of light (wavelength is about 10 microns and you cannot see anything smaller than that – microscopic imaging begins with feature sizes larger than 8 microns for a long-wave camera.’‘

Adams: ‘’Almost everyone is asking for larger format arrays running at ever-higher frame rates, but many applications are unable to support the increased costs of such systems ($250K for our 1Kx1K thermal IR imaging system that runs at over 140 Hz frame rate, full window; even higher for some systems using custom-designed FPGAs).’‘

Argueta: ‘’Typically, IR cameras have fewer pixels than visible-light cameras; IR cameras with more than 640 x 480 pixels are rare.’‘

13. Are there merchant system integrators that offer to integrate thermal imaging-based machine vision systems? Can you elaborate on the services that they offer?
Adams: ‘’SBF has worked with and is working with several system integrators.  Often they are integrators who specialize in specific market niche's, such as NDT analysis of aerospace structure or NDT of electronic components, or NDT of automotive parts.’‘

Richards: ‘’Yes, several companies offer this.  WinSoft can take a thermal imaging camera and incorporate that into a production line.’‘

Chynoweth: ‘’More and more all the time. One of the biggest hurdles to using system integrators is that typically customers would like to have one somebody in charge of the entire system. But system integrators do not assume total responsibility. While they might buy the camera and write application software, the image data is typically in a proprietary format. Hence, they supply system integrator with data format or set of ActiveX controls built in. While system integrators will buy camera, enclosure, I/O, etc. and write software, when system does not operate, component supplier is blamed. This is where there is an advantage in buying from camera company that offers to provide turnkey solution as they have the skills to design and implement the enclosure which stabilizes the environment of the imager, which is the key to a successful application.’‘

14. What have you introduced that is new in the last year targeted at the thermal imaging-based machine vision market?
Argueta: ‘’Sensors Unlimited introduced a high resolution 1,024 pixel InGaAs photodiode array.’‘

Chynoweth: ‘’Mikron introduced several packaged systems: can end monitoring, critical vessel monitoring and substation monitoring.’‘

Adams: ‘’Santa Barbara Focalplane introduced our FireWire ImagIR LC camera product, as well as our 1K x 1K format thermal imaging camera.  Both of these products will find applications within the machine vision market.’‘

Richards: ‘’Indigo Systems introduced FireWire & Omega targets machine vision because then you don’t need a frame grabber.  You can have a number of different cameras all connected to a single host computer without having to have multiple frame grabbers. The Phoenix camera line from Indigo Systems captures high-speed events such as turbine blades in operation to monitor the status of the thermal barrier coating that protects the blades from combustion gases.’‘

15. Are there tips that you can give to prospective end users of thermal imaging-based machine vision systems to assure that they succeed with the implementation?
Richards: ‘’Yes, before making a final decision on the thermal imaging system that is best for you, be specific with your requirements – be very careful to get really detailed information on all the specifications of the camera that you are purchasing.  Customers need to understand their current requirements and then they need to anticipate that they might need a thermal imaging machine vision system for something else.  Some cameras have more features and can fulfill not only what you need right now, but what you might need for future applications.’‘

Argueta: ‘’For thermal imaging take a look at the temperature range you are measuring and decide which wavelength range would be optimal.  A black body radiation curve can aid in determining the wavelength.  The maximum sensitivity is achieved in one wavelength band, even though several wavelength bands can conduct the measurement some are better than others depending on the temperature range the user is interested in.’‘

Chynoweth: ‘’Buy entire package from someone who will take full responsibility for long term operation. Do not piece part and do it yourself as there are lots of different constraints to be handled in most applications.’‘

Adams: ‘’When determining what thermal IR camera you need, be sure to let the camera manufacturer know what your application is, what frame rate and pixel format you think is necessary, and what wavelength region is important.  Also, how will you need to interface to the camera, both hardware- and software-wise?  My experience with many of the thermal IR camera manufacturers has been that they will let you know if their camera isn't a good fit for your application and, if not, they can sometimes steer you to someone else who does offer a good fit for the application.’‘

16. Are there specific activities that you as a supplier of thermal imaging-based machine vision systems have to provide to assure that your customers are successful? 
Richards: ‘’Good customer support, good documentation, good software (DLLs, drivers) so they do not have to reinvent the wheel.’‘

Chynoweth: ‘’Success of the system is in the software. Hence, suppliers should offer software for the entire application – offer to be one-stop shop.’‘

Adams: ‘’Our customers are able to talk with the engineers that design and build the thermal IR camera systems.  This helps our customers more rapidly develop their applications.’‘