Optics for Digital Image Processing

In comparison to traditional analog cameras, the use of digital CCD and CMOS cameras for machine vision and other industrial imaging applications requires a different approach in order to select the appropriate lens to achieve good image quality.

1. The transfer chain in digital imaging
Image quality is affected by a whole series of links (components) in the imaging system used in the process (i.e., components of the digital camera). It is customary to speak of the transfer chain with its individual links (components), as shown in Figure 1.

Each link in this chain influences the image quality, either positively or negatively. This applies especially to the lens, since it stands at the beginning of this chain. What is lost at this stage can only be recovered with difficulty, if at all. The lens transfers the visual information of the object to be depicted onto the plane of the image sensor. This process will be examined in detail on the following pages, in order to describe the different factors that influence the imaging capability of an optical system.
What, then, constitutes ‘‘good image quality’‘ in an optical system?

2. Image sharpness and contrast

The intuitive answer is: the ability to resolve the finest details. The measurement for this is a resolution test, consisting of alternating dark and bright bars of the same width and with varying patterns of smaller bars and spaces, such as seen in Figure 2.

The measurement for the fineness of structures in this example is the number of black/white line pairs per unit of length, thus, for example 5 line pairs per centimeter. This totals 10 lines (black and white) per centimeter. In the analog world, such a test pattern is imaged through a lens and the image is then examined by counting the number of line pairs per unit of length that are just barely recognizable. The limit of resolution on the image side is usually expressed in line pairs per millimeter. Note that line pairs per millimeter (lp/mm) is the optics' industry's standard unit for expressing resolution.

Let us now carry out this experiment. The result is shown in Figure 3. Our test object is, again, the resolution test pattern from figure 2. The optical system images this pattern. As the fineness of the test pattern increases (line pairs per millimeter increases), the picture becomes ‘‘flatter and flatter,’‘ until the finest structures are barely able to be recognized, at which point we have reached the limit of resolution. If we designate the brightness of pure white with a value equal to ‘‘1’‘ and the darkness of black with value equal to ‘‘0,’‘ then the difference in brightness between light and dark bars becomes smaller and smaller as the number of line pairs increase. For the coarse structures, the difference between bright and dark is still equal to 1. For the medium structures the difference drops to 0.65-0.35 = 0.3, therefore 30%. And for the very fine structures, the difference is only 5%!

Actually, however, the goal is to capture also the very finest patterns with a great amount bright/ dark difference (high contrast or modulation), so that they remain easily visible!

The limit of resolution alone is therefore not an adequate measurement of picture sharpness; the modulation with which the pattern is reproduced (in line pairs per millimeter) must also be considered. The higher the modulation, the better the optics!
We must therefore indicate the modulation of the reproduction as a function of the fineness of the pattern (in line pairs per millimeter) in a diagram. This is shown in Figure 3 at the bottom, where the modulation reproduced for the three line pairs per millimeter shown in the example is indicated by the circles on the graph curve.

The result shown is the modulation transfer function abbreviated ‘‘MTF.’‘
The MTF represents the relationship between the fineness of the pattern (in Lp/mm) and their reproduced modulation.

You may wonder what these tests with simple bar grids and patterns have to do with reality, where complicated structures having soft tone transitions and fine surface details occur simultaneously. If we are to be scientifically accurate, the answer is fairly complicated, but it can be reduced to a simple denominator. Every brightness and pattern distribution in the test subject can be thought of as being composed of a sum of periodic structures of different fineness and orientation. The bar tests are only a simple example. Under actual imaging conditions, instead of the abrupt bright-dark transitions of our test patterns, only softer transitions have to be considered, with a ‘‘more harmonic’‘ progress and more precisely sinusoidal changes. The sum of all these lines (line pairs/mm) of different orientation comprise the object structure. The lens weakens the contrast of the individual components according to its modulation transfer function and produces an image that corresponds more or less well to the object. In this regard, the answer to the question posed at the beginning, ‘‘What, then, constitutes good image quality in an optical system?’‘ becomes apparent. Picture quality [sharpness] is: The highest possible modulation reproduction for coarse and fine object structures (expressed in line pairs per mm), up to a maximum line pair number dependent on the application.

Now, a very important question must be clarified: up to what pattern fineness does a modulation transfer makes sense at all, because what the imaging system can no longer resolve does not need to be transferred (imaged).

3. Image Sensors

After the above discussion of image sharpness and modulation transfer, you will undoubtedly already recognize that the requirements of digital imaging are linked to the type of image sensor employed. The smallest representable image element of a semiconductor image sensor (CCD or CMOS) is called a ‘‘pixel,’‘ which is an abbreviation for the expression ‘‘picture element.’‘ The pixel has a strictly regular geometric structure and is arranged in rows with a square or rectangular surface. Today's high-performance image sensors have up to 30 million pixels for area sensors and up to 12 thousand pixels for line scan sensors. The size of these pixels are about 0.002 mm (2 microns) to 0.010 mm (10 microns). Therefore, we can transfer the results concerning image sharpness as follows:
The lens for a 5 micron pixel size, for example, must transmit up to 100 Lp/mm with the highest possible modulation.

The sensor can transmit these line pair numbers only if a dark bar falls specifically on a pixel and a bright bar on the neighboring pixel. Or, conversely: The highest line pair number, which a sensor with a pixel dimension ‘‘p’‘ can transmit and which is also called Nyquist frequency, is equal to RMAX = 1/(2p) Lp/mm

In the above examples, this is 50 Lp/mm (for pixel size 0.010 mm) and 100 Lp/mm (for pixel size 0.005 mm).
 
The tendency in the semiconductor industry to house smaller and smaller structures in the tiniest space has its driving force in the fact that the cost of a component (for example, an image sensor) increases at least in proportion to the surface area. Currently, the size of the smallest reproducible structures are 1/4 µm (that is, 1/4 of a thousandth of a millimeter!). In the research laboratories of the semiconductor industry, they are working on achieving still smaller structures, down to 1/10 µm. It is therefore reasonable to assume that this development will benefit semiconductor image sensors as well, with a lower limit for the pixel size being imposed by a decreasing sensitivity. However this limit is not likely to be much smaller than 2 µm.
There exists the possibility (1) of reducing the surface area of the sensor with the same number of pixels, in order to make it less expensive, or (2) of increasing the number of pixels in the same surface area in order to increase image quality.

Another special fact about digital image memories that must be taken into account is the regular arrangement of pixels.

If we examine more carefully the modulation transfer of the digital image memory in the vicinity of the highest transferable pair of lines Rmax (maximum resolution), then we note that for greater numbers the modulation does not suddenly fall to zero; instead, there is a characteristic decrease in the line pair number reproduced. Figure 4 should clarify this.

The bar test can be seen in the figure at the top (TEST PATTERN), as well as the associated brightness distribution between 1 and 0 of the bright and dark bars. On the horizontal axis, the dimensions of the pixels are shown, in the example from pixel No. 1 through pixel No. 40. Note that there are about 7 bright/dark bar pairs (line pairs) for 10 pixels. The bar widths are now smaller than a pixel (by a factor of approximately 0.7), so that a bright bar but also a certain portion of a dark bar falls onto a single pixel. The pixel can no longer distinguish between these and averages the brightness (in approximately the same way as an integrating exposure meter). Therefore, the succeeding pixels now have differently bright gray tones, depending on the surface portion of the bright and dark bars on the pixel, as can be seen in the middle of figure 4. The brightness distribution shown under REPRODUCTION BY IMAGE SENSOR indicates that now only three (roughly stepped) transitions from maximum brightness to maximum darkness per 10 pixels occur. Actually, however, there should be 7 line pairs per 10 pixels, as in the test pattern.
The line pair number reproduced therefore decreased instead of increasing.
The image no longer matches the object, it consists only of false information! If we consider that the highest transferable line pair number is equal to 5 line pairs per 10 pixels, then we also recognize the mathematical relationship for this characteristic information loss: the line pairs exceeding the maximum line pair number of 5 Lp/10 pixels (7 Lp/10 pixels - 5 Lp/10 pixels = 2 Lp/10 pixels) are subtracted from the maximum reproducible line pair number (5 Lp/10 pixels - 2 Lp/10 pixels = 3 Lp/10 pixels) and therefore result in the actually reproduced (false) structure of 3 Lp/10 pixels. The modulation transfer function is therefore mirrored at the maximum reproducible line pair number Rmax, as shown in Figure 4, bottom.

Naturally, we are not interested in the reproduction of this false information and it would be ideal if the modulation transferred would suddenly drop to zero at the maximum line pair number. Unfortunately, this is not possible, neither for the optics nor the image sensor.
We must therefore pay attention to ensuring that the total modulation transferred (from lens and image sensor) at the maximum line pair number Rmax = 1/(2 x p) is sufficiently small, so that these disturbing patterns are of no consequence. Otherwise, it can happen that good optics with high modulation are judged to be worse than inferior optics with a lower modulation. The total modulation transfer (from lens and semiconductor sensor) is composed of the product of the two modulation transfer functions. This is also valid for the modulation at the maximum transferable line pair number Rmax. For a typical semiconductor image sensor, the modulation at Rmax is about 30-50%. Therefore it is reasonable to demand about 20-30% for the optics at this line pair number Rmax, so that the false information will definitely lie below 10% (0.5 x 0.2 = 0.1).

Our knowledge from the format-bound arrangement must therefore be supplemented:

(1) The modulation transferred above the highest line pair number Rmax, which can be reproduced by the sensor, must be sufficiently small so that no false information is transferred.

(2) On the other hand, the modulation transferred must be as high as possible below Rmax.

4. Requirements of lenses for high-resolution line scan applications

The driving factors for surface and web inspection in quality assurance in manufacturing processes are that the sizes of the defects to be detected get smaller and smaller, currently  down to below 10 microns. The reliability of the inspection process is continuously improved (‘zero defect’) and cost reduction is achieved by more efficient component use and faster scan speeds that result in increased throughput.
 
The corresponding impact on machine vision components is that the camera manufacturers have increased the resolution of their line scan cameras from 4k and 6k to 8k and up to 12k. The sizes of the pixels become smaller and smaller (from 10 microns to 7 microns and down to even 5 microns). These trends require high performance optics for macro imaging.

High resolution line scan applications use the most advanced technology:

             resolution         pixel size           Nyquist frequency       sensor length

  8k     ~ 8192 pixels       7 microns                  72 lp/mm                 58 mm
  8k     ~ 8192 pixels       5 microns                100 lp/mm                 41 mm
12k   ~ 12288 pixels       7 microns                   72 lp/mm                86 mm
12k   ~ 12288 pixels       5 microns                 100 lp/mm                62 mm

Special requirements for lenses suitable for these kinds of high resolution line scan applications include a large image circle to cover the long linear arrays and high optical resolution to provide the appropriate MTF for the small pixel sizes.

As described in 2., the MTF of a lens depends on the number of line pairs per mm. In addition it depends on the magnification ratio that the lens is used to accomplish the actual set-up: ß’ = image size divided by object size which is also equivalent to sensor pixel size divided by object resolution. The MTF depends also on the iris opening (F- number) and the field angle (w’). A common way to provide MTF data of a lens is shown in figure 5.

Figure 5: MTF curves – contrast in [%] of a lens as function of
image height u’/u’max for 3 different number of lp/mm as parameter

The MTF for test grids in tangential orientation is different from sagittal (radial) orientation to the optical axis, as shown in figure 6. For line scan applications you must always use the tangential MTF, since this is what corresponds to the orientation of the line scan sensor.  

Considering all of the above, it seems obvious that anyone designing an imaging system or application that will incorporate modern CCD or CMOS technology would be well-advised to consult with an expert optical engineer and a high-quality lens manufacturer as early as possible in the design cycle.

Jos. Schneider Optische Werke GmbH
Ringstrasse 132
55543 Bad Kreuznach
Germany
Telephone: +49 67-16 01-0
www.schneiderkreuznach.com

Schneider Optics Inc.
285 Oser Avenue
Hauppauge, NY 11788 USA
Telephone: (631) 761-5000
www.schneideroptics.com