Are Smart Cameras Smart Enough?

Conventional analog RS-170 cameras are finding competitive approaches in machine vision applications. One can find a number of alternatives: digital signal processed cameras, digital output cameras, digital imagers, and smart cameras. There are significant distinctions in these classes of cameras.

In true digital signal processed cameras, the complete video processing is digital. These cameras offer the opportunity to improve video quality within the camera, inserting special enhancements during the processing of the pre-processed signal: reduce bloom, enhance gamma control, reduce vertical smear, increase S/N - all via remote control set ups. These cameras are used widely in security applications. For example, backlight compensation is performed based on processing selected areas, creating an image with little noise. These cameras involve video processing prior to converting into an analog final composite video output. In other words, these cameras still output an analog RS-170 signal.

Digital output cameras incorporate an analog-to-digital converter in the camera. The signal processing circuits and the A/D are close to the solid-state imager. This reduces the effects of ambient electrical noise on the video signal put out by the imaging device. Digital output cameras control the timing and digitization right at the camera so that the frame grabber does not have to synchronize analog video signals and digital clock signals, thereby simplifying the circuitry on frame grabbers. Digital output cameras generate signals in which each photosite in the imager generates its own digital gray level, not affected by contiguous photosites.

Transmitting the data to the vision processor can be serial or parallel based on a number of different standards emerging today or based on a proprietary approach. These transmission standards are not image transmission standards like RS 170 that dictate how the pixel data is to be arranged into an image array. In addition to digital output cameras made to RS-422 and RS-644 standards, versions are now available compatible with new serial ports - universal serial bus and IEEE 1394, often referred to as 'Firewire.' Companies, such as PPT Vision have introduced their own proprietary Digital Serial Link. Channel Link is another standard that converts serial to parallel format and from parallel to serial for high speed transmission. These latter lend themselves to feeding image data directly into a PC for host-based processing and analysis of the image data.

Some of these digital output cameras incorporate compute power to perform specific imaging operations within the camera. For example, the Duncan Technologies camera uses a field programmable gate array ( FPGA) to process pixels on the fly so that pre-processed data is fed into the PC. These cameras can implement automatic flat-field normalization in the camera - send it the command to look at a flat field once, then send it the command to turn on normalization and it will automatically apply pixel-by-pixel adjustments to correct for uneven illumination. The nature of the processing to be performed in the camera FPGA is application-specific. Some examples are in object inspection where things like ratios, thresholds, etc. are used to reduce the image to a state that can be analyzed by the decision portion of the systems - food sorting, produce sorting, etc. 

Digital imagers are not cameras per se. They typically include an imager connected directly to a control board with an A/D embedded in a PC. Significantly, the entire operation of the digital imager is dependent on the PC. The PC issues exposure commands as well as commands to dictate which specific pixel regions to furnish back to the PC. The digitized image is then resident in the PC memory for subsequent processing and analysis.

None of these cameras are 'smart cameras.' Smart cameras are self-contained machine vision systems that incorporate intelligence with sufficient compute power to not only control the cameras but to also perform image processing within the camera head itself. Only results, such as measurements, are communicated to some form of plant automation equipment. A smart camera typically includes the lens mount, an imaging sensor, memory, embedded processor, a serial interface and a digital I/O.  The embedded intelligence might be an Intel microprocessor or Motorola Power PC or DSP or FPGA (Field Programmable Gate array) or some combination of these.

Given the imager and A/D are an integral part of the camera, smart cameras enjoy the same advantages of the digital output cameras when it comes to noise and the effects of noise on pixel jitter - pixel geometric positioning fidelity. In other words, there is less variation in assignment of pixel space to object space. This feature makes sub-pixel processing generally more robust. With subpixel positional repeatability of 1/10, a 500 pixel camera essentially yields a ruler with 5000 markings (a 1000 pixel camera, 10,000 markings) that can be applied across an object. With appropriate calibration, accurate and repeatable dimensional measurements over a wide range of part sizes is possible.

Smart cameras are different than products referred to as embedded vision systems. These products require a camera be tethered to a separate vision box. This vision box includes the embedded compute power in the form of a microprocessor, DSP, application-specific integrated circuits or FPGA or some combination of these elements as well as memory.

In the case of smart cameras, most often the application program is downloaded on the camera so that it operates independently of any other computer. Sometimes the smart camera is interfaced to a PC for purposes of configuring for set up and for output data collection and analysis. In the former case, the PC may be removed from the line after set up. In the latter case, the PC may actually be a plant wide automation system to which the smart camera is interfaced via a local area network, such as Ethernet.

The typical smart camera can be considered a general-purpose system capable of being programmed to address any number of applications. However, just like any general-purpose machine vision system or vision processor there are constraints based on the implementation. The software available for the smart camera and the type and speed associated with the embedded intelligence in a smart camera dictate how much actual image processing and analysis can be performed within the camera itself. Similarly it can dictate the throughput that can be handled for a given complement of processing and analysis. 

Uwe Furnter of Matrix-Vision suggests that while DSP-based smart cameras are typically cheaper than those with embedded microprocessors, they suffer from the fact that they are more difficult to program and that programs really become application-specific rather than general-purpose. If 'smart ' is associated with the amount of compute power included in the camera head, one of the smartest cameras would be those offered by Wintriss Engineering. Its smart cameras include a microprocessor, DSP and multiple FPGAs with up to 130,000 gates. They offer both area and line scan versions of their smart cameras. Their line scan version performs imaging related processes on 5150 pixel lines at 40 MHz.

They have developed an application-specific version of the camera targeted specifically at web scanner applications - their WebRanger. One FPGA performs image sensor control and pixel correction. The combination of the compute power in the camera head permits running real-time digital filters, lighting correction, streak correction and input/output capability. Ultimately geometric and photometric manifested flaws are discriminated based on connectivity analysis, all performed within the camera. Their design is scaleable so they can stack as many cameras as required for the web scanner application. All real time imaging tasks are performed in the camera. While a PC serves as the user interface, it receives only the defect data and provides analysis tools and data management.

On the opposite end, Banner Engineering offers a low end smart camera. Jeff Schmitz, Corporate Business Manager - Vision Systems at Banner Engineering suggested its 'PresencePLUS is an easy to use, economical 2D area sensor. In essence it contains nearly 200,000 MiniBeams bundled together that can be taught 'good' and 'bad' products for go/no-go applications.'  The unit operates on user adjusted thresholded images to count pixels (either black or white) and compares the count to a taught or user-adjusted reference pixel count range.  This approach is well suited to applications where gross changes result in substantial contrast changes that characterize the condition to be detected such as presence of a label, presence of product in a tray, etc.

Cameras with more embedded intelligence are offered by quite a few companies, most of them located offshore. These include: American Eltec, DS-GmbH/Neurocheck, DVT, Fastcom, Matrix Vision, Nanosystems, Omron, Pulnix America, QualiCam/QualiVision, Siemens, Vision Components, Vision and Controls, Wintriss Engineering and Wizcam America. While most of these are based on nominally 640 X 480 imagers, Wintriss and Nanosystems offer versions based on line scan imagers and at least Wintriss and Vision Components offer versions that use nominally 1000 X 1000 imagers.

Most of these camera ultimately require conventional programming to make them suitable for machine vision applications. Several, such as those offered by Omron, Pulnix and DS-GmbH/Neurocheck are based on neural network processing. This approach typically involves training the system by showing it a number of different classes of product - those that are good and those that are bad. After this train-by-showing approach, the system sorts 'good' from 'bad' product.

Most of these cameras also require some form of 'configuration kit.' This may be a PC and includes a monitor, physical interface (keyboard, mouse, etc.) and graphic user interface reflecting machine vision applications. Often the GUI includes icons corresponding to a fixed set of algorithms optimized for a specific machine vision application: robot guidance or machine guidance, gaging parts, OCR/OCV, etc. Often these are the same 'canned' tool libraries one finds in general-purpose machine vision systems under application-specific icons. However, the underlying algorithms may not be as robust as might be found in a general-purpose machine vision system or vision processor with more 'intelligence' - products that might have more compute power capable of running more compute intensive algorithms.

One interesting observation is that application-specific smart camera systems are emerging. As noted above, Wintriss Engineering now offers a turnkey solution called WebRanger for web scanning applications. Mehrdad Agah of Fastcom cited their turnkey system Smart Bending Angle Calculator targeted at measuring the bending angle during the tube bending process. In this case, using a CMOS imager, they are able to take advantage of random addressability.

Most of these companies were asked to respond to a number of questions related to smart cameras. What follows is a summary of their responses.

How do smart cameras compare with the functionality of
 Embedded vision systems
 General purpose machine vision systems, vision processors
 PC/frame grabbers/software
 PC/digital camera (Firewire, Hotlink, etc.)

Mike Williams of DVT provided a summary to this question that generally reflected the other responses: 'The idea of a smart camera is that it should be easy to use, it should be self-contained, and it should have on-board intelligence. Smart cameras are a subset of embedded machine vision systems.' And 'the big frame grabber or PC-based systems typically cost more and are designed for integration houses rather than end-users.'

Market Intelligence News & Insights: 

How Will Apple’s $500 Billion Investment for US Manufacturing, Education, and AI Play Out?

On February 24^th, Apple announced a “commitment… to spend and invest more than $500 billion in the U.S. over the next four years”, with plans for new manufacturing plants, enhanced support for manufacturing partners via their Advanced Manufacturing Fund, and a myriad of other initiatives. As part of this plan, Apple projects 20,000 new hires, with roles focused on R&D, silicon engineering, software development, and AI/ML engineering.

Read More

Mark Sippel of Omron, referring to capabilities of above mentioned product types, added 'these same capabilities and improvements have given these devices (smart cameras) functionality that approaches the capabilities of many PC based systems. As new functionality and smart camera models are introduced in the near future, much of the functionality differences between PC based or high-end systems will begin to disappear altogether.'

What are the main applications of smart cameras?

The response reads like a litany of conventional machine vision applications, a good summary of which was provided by Marcel Singleton representing Vision Components and several other offshore smart camera companies in North America: 

• Precision registration/alignment
• Determining the orientations of objects/robot guidance
• Confirming the presence, proper positioning, and correctness of labels on products and packaging
• Reading characters or deciphering 1D or 2D matrix codes that have been machine marked on manufactured parts and packaged goods (OCR and bar/matrix decoding)
• Checking the position, correctness and quality of logos and/or text that have been machine marked on/into manufactured parts and packaged goods (OCV)
• Gauging/dimensional characterization
• Verifying that all of the components associated with a given assembly are in place and undamaged
• Verifying the presence and integrity of blister-packaged pills and tablets
• Verifying the proper settings of switches
• Inspecting items for bent features, bridging, burrs, mouse-bites, chips, cracks, scratches, flash, etc.
• Surveillance/security applications
• Sorting items of interest into different categories for further processing or packaging

What are advantages/disadvantages of smart camera implementations?

The following represents a compilation from all of those responding.
Advantages:

• Easy to set up and get running quickly with minimal engineering support
• Easy to use interfaces
• Compact camera and controller sizes/small footprint
• Typically more economical than alternatives often costing less than comparably configured vision systems that are 'industrial PC'-based
• Industrially hardened hardware packages/factory floor ready
• Built in inputs and outputs
• Communication capability like RS-232C/422, DeviceNet - networkable without a PC
• Better quality image data than analog camera/frame grabber combinations due to reductions in noise and other image degradation factors
• No PC card driver or configuration problems

Disadvantages:

• Limited functionality/Lack the processing power of high end, PC-based systems because even the Pentium-based units feature processors that lag behind what is currently available on desktop and high end laptops
• May lack the expandability and flexibility of PC-based systems
• Non-PC compatible platforms can require appreciably greater programming effort and may offer inferior development tools
• Fixed lenses that limit the field of view size
• May have limits due to pixel array sizes that can limit the resolution on large fields of view
• Proprietary nature of smart cameras can limit choices of hardware, like imagers, I/O, lighting, lens, communications format
• Not as many software applications and libraries as already exist for PC/frame grabber-based systems

One interesting disadvantage cited by Mike Williams of DVT was that since these systems are inexpensive they are not taken seriously by many in the marketplace.

How does one interface - GUI? Line?

Each supplier has their own interface approach. What Mark Sippel suggests of Omron's approaches to the man-machine interface for its various smart camera products fairly represents the spectrum across the different suppliers: 'interfacing ranges from simple push button setup and teach feature ….(to) simple, easy to use menu driven interfaces that are imposed over the camera images in combination with a handheld keypad. This allows total setup and configuration to be done on the same monitor as the camera's images without the use of PC hardware. …. (another product) uses Windows based flow chart and tool library software interface in combination with the on-screen menu system. This interface can be used to setup, configure and monitor the vision sensor using a PC. When ready to run stand-alone, the PC can be disconnected and (the system) monitored and adjusted using the monitor's on-screen menus.'

Some of these products are targeted at OEMs who in turn would design their own man-machine interfaces and graphic user interfaces. As suggested by Mr. Agah, companies like Fastcom offer versions that 'use an Integrated Development Environment (IDE) which permits the programmer to work on different projects and develop application code in ANSI-C (could be different language with different vendor) on a PC and then download the software onto the Flash memory of the camera, so it can operate independently afterwards.'

Marcel Singleton summarized on line interfaces indicating they are typically handled by the digital I/O: communications with PLCs, part-in-place sensors (for triggering), air eject mechanisms, alarm devices/lights, machine controllers, etc. He also observed that 'the reporting of X/Y/theta offset positions, outputting of characters read or deciphered bar or matrix code information, transfer of other data, and sometimes triggering is done using serial communications.'

What tools are typically included?

The tools provided vary from supplier to supplier and in some cases are based on design implementations or specific applications targeted by the supplier. DVT suggests it offers measurement, OCR, barcode reading, object finding and communications drivers. Marcel Singleton notes that 'Vision & Controls Pictor is targeted specifically at gauging applications programmed by a user pointing, clicking and entering tolerances and other setup parameters in dialog boxes presented by the product's MS Windows-based VCWin software. System measures distances and angles, determining part orientation, calculating best fit lines and circles, counting the number of edges or objects along a straight line or around a circular/ring element, checking roundness (eccentricity), inspecting surfaces, doing completeness checks, carrying out normalized gray scale correlation-based inspections, blob analysis, OCR, deciphering ECC Data Matrices, and verifying assemblies.'

In Omron's case the tools are a function of their models. According their Mark Sippel, 'the F-10 offers up to 8 models of gray scale pattern matching. The F30 offers binary pixel counting. The F150 series vision sensor offers binary and gray scale tools. These include area, center of gravity, 360 degree rotational and precise model search, surface detection, edge detection, edge pitch inspection, labeling or simple blob analysis, classification for sorting models, position compensation, branching if/then type inspection flow, field of view calibration, and electronic filters, like edge extraction. The F400 color Vision Sensor offers many of the same tools as the F150 series plus added benefit of up to 8 color pickup detection or 5 color filters to choose from.'

What are prices and what do the prices typically include?

The prices of smart cameras depend on their compute power as much as anything. Those that perform simple pixel counting functions sell for $995 to $1500. As they increase in functionality the prices are in the $4-8K range. Those with more intelligence can range up to $26K in price. The prices may or may not include the software, lighting, monitors, configuration kit, etc. Those targeted at OEM customers (only development software) can even sell for less than $995.

What is the future for smart cameras?

Jeff Schmitz echoed what most of the respondents said - 'future smart cameras will offer even more functionality. They will be more flexible to easily connect and integrate into factory floor networks of components. Continuing advances in consumer CCD and CMOS imagers will continue to make industrial smart cameras more powerful and affordable.'  Additional advances cited by Mark Sippel  of Omron included greater power in the tool sets, greater flexibility in communications and higher resolution. 'This means a future where the power and functionality of these smart cameras will be equal or better than existing PC-based or high-end vision systems.'

For the user Mr. Sippel suggested 'with the continued low cost of these devices, the trend of distributing machine vision across the entire line at points before value is added will also continue.'  Mr. Agah added 'intelligent cameras will play a significant role for system integrators (and OEMS) who wish to add vision functionality to machines without purchasing and integrating a variety of components from different manufacturers.'

Smart Camera Impact on the Machine Vision Market

As said about many things today, smart cameras will get cheaper, faster, smarter, and better! Significantly, the underlying compute technology that will make these results possible will have a similar impact on all machine vision product designs.

The marketplace question is 'are smart cameras 'cannibalizing' the market for general-purpose machine vision systems/vision processors and frame grabber/PC-based systems?'  To a certain extent the answer is 'yes.' If the performance of a smart camera is adequate for an application that might otherwise have been addressed by alternate machine vision products, because of the lower cost the smart camera is likely to get the sale.

However, smart cameras are actually contributing to the expansion of the machine vision market. There are many applications that would just not be considered because of the cost of a system based on a general-purpose machine vision system/vision processor or frame grabber/PC-based system. Smart cameras are being deployed for these applications. A case in point is when a plant has the requirement to equip multiple lines with machine vision. At $30 - 50K per line, the likelihood of equipping multiple lines decreases rapidly with the number of lines. A smart camera-based solution at $5 - 10K can not only be more easily justified, but given the potential gain in quality and productivity, it may be unjustifiable not to equip every line accordingly. The same can be said where avoiding adding value would be possible using a distributed arrangement of smart cameras along a production line.

All in all, the lower cost associated with a smart camera-based machine vision application implementation will result in an increase in the machine vision market, especially in terms of units deployed. The key for the user is to recognize that smart cameras may not be as smart as alternate machine vision-based approaches and consequently can not be expected to handle all machine vision applications. In other words, smart cameras complement the machine vision products in the market and are not always a substitute for them.