Lighting for Facial Biometrics

Facial biometrics provides a user-friendly, inexpensive, and easy to use method of verifying an individual’s identity.  However, the accuracy of the verification is significantly less than other biometric methods.  One of the shortcomings is that many facial biometric algorithms are very sensitive to even minor changes in lighting.  One approach is to require standardized lighting for facial biometrics.  This can prove impractical for a facial biometric kiosk that is incorporated into an existing building. As demonstrated here, modulated light sources synchronized with video cameras can be used to provide images that are lighting independent for these situations.  The amount of modulated light needed is minimally obtrusive.  The lighting arrangement can be arbitrary, but should be appropriate for the application and the site.  Good lighting can suppress the background in closed and open spaces.  While motion artifacts in the background can appear, identification and removal is easily achieved.

Introduction to Biometrics
Modern biometrics is a method of associating unique characteristics of an individual with a database entry.  The biometric system ‘‘recognizes’‘ the individual when the data acquired from an individual person reasonably approximates the information in their database entry.  In this article, we will concern ourselves with biometric verification, determining if an individual is who they claim to be.  Verification is frequently used in everyday security, for granting (or denying) access to valuable materials, currency or information.  In these situations, the individual desiring access is generally knowledgeable and compliant with the procedure and knows that biometric information is being gathered.  In many ways, this is an ideal situation.  However, biometric verification remains error-prone because of limitations in hardware and software.

There exists a large host of options for biometric identification and verification.  A short list of techniques may include, fingerprint, facial, hand geometry, iris, retina, token and knowledge based identification.  Each of these methods has limitations in accuracy in identification, ease of use, cost, speed, robustness to fraud, and user acceptance.¹  Several biometric vendors have proposed combinations of biometric measures to enhance identification, assuming that a combination of methods is less error prone and more difficult to defeat.  Facial biometrics is often part of that combination because of low cost, easy integration and a data record that is useful in forensic investigations.  However, facial recognition has high error rates compared with other types of recognition.¹ A simple, but not simplistic, statistical analysis shows that combining a strong and a weak biometric method to reduce errors does not produce a significant improvement.^{2, 3}

Because of the characteristics of facial recognition described above, the goal of improving facial biometrics is very appealing.  Standards organizations have assembled documents describing appropriate methods for acquiring images for facial biometrics and challenge datasets for biometric algorithms.^4-6  The goal of these methods is to provide a uniform, and perhaps ideal, image for facial biometrics databases.  Significant portions of these documents deal with how to control lighting in images used for facial recognition.  It is often difficult to achieve the needed degree of control of ambient lighting in commercial spaces, where natural lighting can vary depending on time of day, weather conditions and outdoor vehicular traffic.  Artificial lighting can vary depending on the aging and maintenance of lighting fixtures as well as from reflections of nearby objects.  These difficulties are highlighted in the FERET and FRVT challenges, as well as review articles.^6-9

Significant improvements in facial biometrics performance can be made if the images are independent of ambient light.  The majority of the effort in the biometrics field in removing ambient lighting has been in software solutions.  Hardware solutions, like the lighting solution discussed here, are a worthy alternative approach.  Understanding the current technology is important in bringing hardware and software approaches together.

Overview of the State-of-the-Art

Algorithms
In understanding the role of lighting in facial biometrics, it is useful to have a general appreciation of the algorithms used for identification and verification.  It is difficult to give an in-depth analysis of commercial facial biometrics systems because the algorithms are proprietary and rarely disclosed.  There are several algorithms that are used in the facial biometrics field.9  In their simplest forms, the methods are objective, i.e. they make no assumptions about what characteristics define a face.  This is both a strength and a weakness.  It has been shown that human assumptions about facial recognition often do not represent the best basis for discrimination.  The effectiveness of objective methods is strongly dependent on the examples presented to them.

The accuracy and speed of each method are dependent on the qualities of the set of images used to train the algorithm.  For facial biometric verification, the training set should include a set of images with a reasonable set of variations that the subject may present to the camera, for example: a variety of poses, facial expressions, and cosmetic changes like facial hair and hair style.  It should be expected that the methods will produce more errors if the individual presents a variation not represented within the training set.

There are several approaches to minimizing these effects.  One is to minimize the effects of these variations by emphasizing certain features, like eyes, ears and mouth by using weighting factors to increase the importance of particular parts of the image.  For example, the weight, or importance, of the image background should be zero. Accounting for pose and lighting can be accomplished by many techniques, including 3-D modeling.  In accounting for pose, there are six additional parameters that need to be considered.  There are three rotational degrees of freedom, yaw, pitch and roll and three translational degrees, left-right, up-down and inward-outward as defined by the optical axis of the camera.⁵  Changes in lighting change the reflections and shadows on the face.  Optical ray-tracing models can identify the light sources present and allow for removal of their effects. 

There are significant challenges for facial biometrics in terms of robustness, speed and accuracy.  These three challenges interact with each other, since gains in accuracy through increased model complexity reduce speed and robustness.  Biometrics continues to provide useful challenges for both academia and industry.

Lighting Technologies
The Applied Mathematics and Computer Science communities dominate research in biometrics, because many of the challenges require skill and knowledge in dimensionality reduction, databases and algorithm design.  There are remarkably few attempts at solving facial biometric issues in hardware.  Many of these attempts are focused on improving the computational hardware, such as implementing hardware neural networks.   There are even fewer attempts at lighting solutions despite the overwhelming emphasis on lighting in the machine vision literature.

A significant number of the illumination-oriented inventions for biometrics focus on the near infrared.^{3, 10-17}  This approach is successful because even intense near infrared lighting is unobtrusive; photons with wavelengths greater than 700 nm are invisible to the human eye.  In addition, the atmosphere and window glazing significantly reduce the available light with wavelengths between 700 and 2000 nm.  Naturally-lit interior spaces can be considered to be dim within this wavelength range. Commercial CMOS and CCD cameras are sufficiently sensitive to light between 700 and 1100 nm for acceptable results.  The difficulties lie with interference from interior incandescent lighting and obtaining inexpensive, and bright, near infrared light sources for the measurement.

Infrared thermography is a relatively new area of interest, with the cost of infrared imaging detectors rapidly falling toward $4000/unit due to technological advances.  Thermography senses the temperature of an object.  Human beings stand out against the background due to their higher temperature.  It is capable of mapping not only facial features, but also the pattern of blood vessels within the skin.¹⁰

Modulated visible and infrared light sources, when synchronized with image capture devices, hold the promise of capturing images that are independent of ambient lighting. The goal is to capture images of object illuminated by the modulated light source.  In 1992, Scientific Generics considered the possibility of acquiring facial images using modulated light sources (a 5 kW near-infrared flash lamp), they were forced to discard the idea due to the limitations of 1992-era technology.¹¹  A similar technique was developed at Siemens in 2000 for gesture recognition using an infrared strobe.¹² However, a detailed evaluation of flash lamp performance in a biometric environment has not been attempted.  The flash lamp approach is worthy of renewed interest in the facial biometrics field. 

Using Modulated Illumination to Remove Ambient Light Signatures
Because lighting presents such an overwhelming challenge to facial recognition, considerable computational power or constraints to ambient lighting is used to reduce even minor variations.  Ray tracing is a complex task, even when the surfaces are simple, let alone considering all of the possible shapes and curvatures of the human head.  The traditional approach to standardized lighting is to use flash or flood lighting to ‘‘overwhelm’‘ ambient illumination sources.  When the lighting is in the visible, this is obtrusive and can make the subject uncomfortable and less compliant.  Infrared lighting can be used, but near-infrared-only light sources are expensive and lie outside the most sensitive spectral region of CMOS and CCD cameras.  Infrared images are also incompatible with current facial recognition databases because the reflectance of facial features change dramatically in the near-infrared.  Modulated light sources can provide an easily implemented solution for both subtle and dramatic changes in ambient lighting.

The Role of Photometry in Biometrics
In photometry, modulated light sources have long been used to minimize the signature of ambient light entering the instrumentation.18  In the simplest arrangement, the measurement light source is modulated by turning the lamp on and off and the intensity entering the instrument is measured during both the on (Ibright) and off (Idark) period of the lamp.  The measured intensity (Imeas).

Measured Intensity
Using research instrumentation, very small differences in light intensity can be measured when the modulation is faster than the changes in ambient light.  In biometrics applications, the photometric quality of the digital image (signal to noise ratio) is limited by digitization quantization, readout and shot noise.

A picture is a record of a photometric measurement.  In digital images, the intensity of each pixel is directly proportional to the number of detected photons.  In flash portrait photography, the light from the flash lamp is added to the ambient lighting to increase illumination of the subject.  Because intensity diminishes proportionally to the inverse square of the distance from the flash, objects near the flash are much more strongly illuminated than distant ones.  The flash lamp provides two advantages, it is a modulated light source and its effect is localized.  This means that when illuminated (bright) and dark images are taken, the difference image will contain a record of objects near the flash.  These two advantages have not been previously recognized in the biometrics field.

Only a small change (25% or less) in intensity from ambient light level reflected from the face is needed to correct for gross changes in ambient lighting.  Flash intensities at this level are marginally intrusive.  The intensity level is similar to the red-eye reduction flashes from pocket cameras.  The lower limit of flash intensity is set by the camera’s inherent photometric noise.  Usable pictures will result when the recorded intensity is 2 to 3 times the noise level. Equation 2, which describes the effect of noise, ignoring digitization quantization, can be derived from Equation 1.¹⁸

Certified System Integrator Program

Set Yourself at the Forefront of the Global Vision Market

Vision system integrators certified by A3 are acknowledged globally throughout the industry as an elite group of accomplished, highly skilled and trusted professionals. You’ll be able to leverage your certification to enhance your competitiveness and expand your opportunities.

GET CERTIFIED

As with all flash photography, care needs to be taken not to saturate the camera.  Ideal cameras have large dynamic ranges, such as 10-bit machine vision cameras and products by Pixim and SMaL Camera Technologies, which can provide the optimal dynamic range for the difference image. In principle, the arrangement of lighting is arbitrary, but thought should be given to which arrangements best suit the physical site, reveal biometrically important features, and allow for accurate matching with existing facial recognition databases.  In particular, attention needs to be paid to the ISO standard, which requires diffuse light.  On-axis lighting leads to potentially unacceptable glare on facial features, eyewear and jewelry.

Because the lighting is controlled and the effect of ambient sources is significantly reduced, the difference image is ideally suited as input to current biometric algorithms.  The shading in the image is now a strong clue to curvature of the facial features, allowing accurate measurement of pose. A reduction in background clutter allows for more accurate localization of the face.  

A Simple Test
This method was implemented using a consumer-grade digital camera on two different days.  The results are shown in Figure 2.  The sampling rate for this configuration is about 0.05 Hz, the flash illumination is approximately coaxial with the lens.  The bit depth of the pixels is 8 bits and the photometric quality of the images is limited by digitization quantization.  The intensity of the reflected light from the flash is 20% of the total intensity arriving at the camera, based on exposure settings.  The camera was used with automatic exposure settings.  Neither the camera nor the illumination parameters are ideal, but represent the limitations of the equipment available for this initial study.

The results of difference imaging are dramatic.  When the exposure conditions are constant, i.e. all automatic functions in the camera are disabled; the light from the flash unit is the only light present in the flash image. There is no evidence of highlights from natural or artificial ambient light sources. In figure 2, the subject stands out strongly from the background in both difference images.

Some hints of the background remain in the panel 1 difference image.  This is due to the flash reflecting off the white walls of the room.  Significantly, the complex background features, the sunlit parking lot and the ceiling light fixture, have been removed.  Only faint borders from some objects remain due to artifacts from the JPEG compression algorithm in the camera.  The JPEG compression algorithm is ‘‘lossy’‘ and does not accurately reproduce intensity at the borders of high-contrast areas in images.

In panel 2 of Figure 2 we have a more complex background, there is more contrast and details.  There are a number of chairs with black enamel paint in the background.  Sunlight is reflecting from the walkway above the subject and provides a strongly directional illumination.  In the ambient light photograph, b, the walkway is overexposed.  Two of the five lamps to the left of the subject are lit.  There are a number of chairs with black enamel paint in the background.  With the exception of the lamps, the majority of the background has been eliminated in the image.  Like portions of the walkway, the two lamps appear in this image because in the original ambient light image, b, the lamps are overexposed.

As an example of practical hardware, the video equipment in self-service ATMs is appropriate to the task.  The cameras have a sampling rate of 30 to 60 Hz.  Digitized images are available from the cameras when video is sent over Ethernet cables. Some ATM camera modules can do the processing using the on-board signal processor.  The only new requirement is the addition of an illumination source to the ATM fascia.

Special Considerations
There are three important limitations in implementing difference images using flash lamps.  The simplest problem to solve is glare from specular reflections, which is especially prevalent with on-axis illumination. Like all flash photography, specular reflections from windows and other glossy objects saturate portions of the image.  The specular reflections can be reduced using polarizers or diffused light.  Secondly, difference images have an inherently low dynamic range.  There are two approaches to this, using corresponding high dynamic range data from the original images, and using nonlinear response functions for the camera to emphasize the added intensity from the flash.   Finally, unlike using standard video frames, the images are slow to acquire.  The low acquisition rate means that the effect of motion is more difficult to suppress.  The effect of motion and its suppression, is worthy of detailed discussion.

It is instructive to consider an NTSC format camera that is in use with commercial ATM machines.  The cameras can have a 50° to 110° field of view. Typically, NTSC cameras have a horizontal resolution of 300-500 horizontal pixels with a frame rate of 30 to 60 Hz.  At the upper end of this range are VGA cameras with 640 horizontal pixels. Lower resolution cameras will suffer less from motion effects while significant increases in resolution do not aid biometric verification.  Using a 60° field of view, 30 Hz frame rate, 640-pixel resolution, typical velocities, distances and trigonometry, the ‘‘worst case’‘ can be calculated for object motion between different frames.  Figure 3 shows a schematic for a hypothetical office entryway, or a stand-alone ATM. 

The effect from objects moving through the field of view at different rates using the calculation described in the above paragraph is shown in Table 1.  The speeds indicated are in the range for vehicular and pedestrian motions.  It is fortunate that the subject is relatively stationary during other verification measures, like fingerprint scanning or token presentation.  The range in speeds for token presentation and keyboard entry is from 1 to 0.03 mph.  High rates of speed perpendicular to the optical axis cause the moving object to be recorded at different locations in subsequent frames.  These shifts would appear as bright smears in the background of the difference image.  Inspection of the table shows us that commonplace events, like pedestrian and vehicular traffic, have a small effect on the image.  It is also possible to test the quality of the difference image by measuring the variance of the images taken just before and just after the difference image.  If the variance in the set of images is large, the suspect pixels can be assigned a weight of zero or the difference image can be retaken.

Summary and Future Directions
The utilization of illumination technology in biometrics can serve an important role in improving facial recognition, where the primary source of variability is a change in natural and artificial lighting.  The modulated lighting method proposed here provides images that are ideally suited for current biometric algorithms, because they are independent of ambient lighting and background variability is significantly reduced.

Remarkably, even when using a consumer grade camera to test the method, the results are impressive.   In the difference image the subject appears to be illuminated only by the flash lamp, there is no evidence of highlights or shadows from other light sources. In addition, the image of the subject is clearly isolated from the background, and retains a significant degree of detail.

The change in lighting required for an acceptable difference image is remarkably small.  In the examples above, the intensity provided by the flash was only 20% of the ambient intensity.  Further reductions can be achieved with more sophisticated cameras and modulation techniques.  With an optimized optical system, it is conceivable that an ATM-style biometrics kiosk could utilize brightness changes in the video display to provide the modulation for difference imaging.

The modulated lighting method has two drawbacks, low speed and low signal to noise.  The low speed may not be a practical disadvantage in most environments because the subject is unlikely to be moving quickly.  Movement in the background can be easily identified and corrected using existing data.  The low signal to noise is a fundamental problem of the technique.  Using cameras with nonlinear response functions, or using the original images, can mitigate the issue when utilizing flash lamps. Clearly, the next steps are testing the method in a high-traffic environment using an optimized camera, lighting and automated software.

Acknowlegements
The author would like to thank David Krzic at Diebold, Inc. for technical specifications of self-service ATMs.

References

1. B. Dawson, Vision-Based Biometrics; Machine Vision Online, (2000).
https://www.machinevisiononline.org/public/articles/archivedetails.cfm?id=364
Accessed: September 3 2004.

2. J. Daugman, Combining Multiple Biometrics; The Computer Laboratory, Cambridge University, (2004).
http://www.cl.cam.ac.uk/users/jgd1000/combine/combine.html
Accessed: August 26 2004.

3. A. Ross; A. Jain, Information Fusion in Biometrics. Pattern Recognition Letters 24, 2115 (2003).

4. P. J. Philips; H. Moon; S. A. Rizvi, The FERET Verification Testing Protocol for Face Recognition Algorithms; NIST Image Recognition, (1998).
ftp://sequoyah.nist.gov/pub/nist_internal_reports/ir_6281.ps.Z
Accessed: September 3 2004.

5. ISO/IEC, Biometric Data Interchange Formats - Part 5: Face Image Data - Draft; ISO, (2003).
http://www.icao.int/mrtd/download/documents/Annex%20D%20-%20Face%20Image%20Data%20Interchange.pdf
Accessed: September 2 2004.

6. P. J. Philips; A. Martin; C. L. Wilson; M. Przyboki, An Introduction to Evaluating Biometric Systems. IEEE Computer 33, 56 (2000).

7. M. Bone; J. L. Wayman; D. Blackburn, Evaluating Facial Recognition Technology for Drug Control Applications, ONDCP International Counterdrug Technology Symposium, June 26-28 2001.

8. A. Jain; S. Pankanti, Biometrics Systems: Anatomy of Performance. IEICE Trans. Fundamentals E84-D, 788 (2001).

9. A. Pentland; T. Choudhury, Face Recognition for Smart Environments. IEEE Computer 33, 50 (2000).

10. A. Green, Information Concerned With the Body of an Individual. Kodak Ltd., International Patent WO8804153 (1988).

11. N. Farahati; A. Green; N. Piercy; L. Robinson, Real-Time Recognition Using Novel Infrared Illumination. Optical Engineering 31, 1658 (1992).

12. H. Roettger; W. Radlik; C. Maggioni; J. Simmerer, Procedure to Reduce Extraneous Light on Object to be Recognized for Gesture Operated Touchscreen. Siemens AG, German Patent DE19918633 (2000).

13. J. B. Dowdall; I. Pavlidis, Near-Infrared Method and System for Use in Face Detection. Honeywell, International Patent WO03023695 (2003).

14. F. J. Cusak, Jr.; J. Bortolussi; D. C. Ehn, Method and System for Eliminating Unwanted Shadows on a Subject in a Facial Recognition System. Lau Technologies, International Patent WO010116 (2000).

15. K. Kondo; K. Uormori, Eye Position Detection Method and Device. Matsushita Electric Industrial Co. Ltd., Japanese Patent JP2002056394 (2003).

16. F. J. Prokoski, Dual Band Biometric Identification System. Prokoski, United States Patent US2002136435 (2002).

17. J. H. Lemelson; L. J. Hoffman, Vehicle Security Systems and Methods Employing Facial Recognition Using a Reflected Image. Lemelson and Hoffman, United States Patent US2003142849 (2003).

18. J. D. Ingle; S. R. Crouch, Spectrochemical Analysis (Prentice Hall, Upper Saddle River, NJ, 1988).