3D Machine Vision in the Aerospace Industry

The military/aerospace industry has generally pioneered the development of much technology, given their deep pockets and emphasis on quality and safety. This is somewhat the case when it comes to 3D-based machine vision systems in manufacturing applications. While they have not supported much research in this field, this industry has generally been an early technology adopter of this technology.

What is interesting are the many technical approaches that have been deployed in this industry: interferometric approaches of one type or another, stereo correspondence including versions with three cameras, various structured light approaches, etc. Today these approaches are shop floor tools to guarantee that assemblies and individual parts are correct.
The availability of commercial point cloud management software, as well DMIS standards and advances in hardware-based compute power, have made it possible to make comprehensive 3D measurements in near real-time. Applications range from measuring relatively small parts like airfoils to measuring complete dimensions of an aircraft.

Input for this article was canvassed from all known suppliers of 3D-based machine vision systems for aerospace/military manufacturing applications. The following kindly responded to our questions:

• Dr. Bernd-Dietmar Becker, Director, Laser Scanner Marketing & Product Management, FARO Technologies, Inc. (Lake Mary, Florida)
• James Gardner, USA Business Development Director for Large Volume Metrology, Metris (Brighton, Michigan)
• Don Waterman, Vice President of Business Development, Neptec (Kanata, Ontario)
• Giles Gaskell, Director of Business Development, NVision, Inc. (South Lake, Texas)

1. Can you describe your 3D machine vision product line that has addressed applications in the aerospace/defense manufacturing industry specifically and discuss how you differentiate your products that address these applications, if you offer more than one?
[Bernd Becker – Faro Technologies, Inc.] We do not have a specific machine vision product line. The customers use our Laser Scanner LS 420 (20m range) or 880 (80m range). It is very fast - 120,000 3D points per second. 

[James Gardner – Metris] We offer two solutions for the optical tracking of the 3D position of single points or 6D positions of objects (equipped with multiple tracking sensors):

• K-Series - In this case, a “camera” equipped with 3 linear array digital cameras observes the position of infrared LEDs and calculates their position based on triangulation. The measurement volume of a single camera is limited to 17m³ but the tracking frequency can go up to 1,000 Hz enabling true dynamic tracking of objects.
• iGPS (infrared GPS) - Metris’ iGPS is a modular system that provides the capability to build an indoor GPS enabled environment with almost unlimited size by installing a network of transmitters within a facility. Within this measurement environment, sensors receive the signals from these transmitters, and on the basis of signal related information, the precise position of these sensors in 3D space is calculated. In support of aircraft parts alignment and assembly, sensors are mounted on aircraft parts in order to capture the precise position and orientation of these parts.

The advantage of this technology is that:

• The measurement volume can be easily expanded by adding more transmitters to the network
• The measurement accuracy is independent from the measurement volume
• There are less line-of-sight issues: it is sufficient for the sensors to have two transmitters within range creating a redundancy within the network.
• System performance does not depend on the number of sensors in use as each sensor works independently. This means that when a room is iGPS-enabled, a multitude of measurement applications can be operated in parallel.

Compared to K-Series, the measurement frequency of iGPS is between 1 and 40 Hz depending on the required accuracy. This means that iGPS is limited to pseudo-static applications.

iGPS is a modular, large volume tracking system enabling factory-wide localization of multiple objects with metrology accuracy, applicable in manufacturing and assembly. iGPS is mainly used by aerospace manufacturers, but is also adopted by automotive and industrial manufacturers both for positioning and tracking applications.

For non-contact surface scanning, Metris offers two solutions:

• Laser line scanners - In this case a laser line is projected on a surface while the projection of this line is observed by a CCD camera. Through triangulation, the position of each point on the line relative to the scanner is calculated through triangulation. The scanner is mounted in a “localizer” that determines the position of the scanner. This localizer can be a CMM, an articulated arm or the position of the probe can be monitored using a dynamic tracking system like K-Series. 
  
  Laser line scanners from Metris are available in multiple versions:
  • LC15 for CMM–based high-accuracy scanning (e.g. of turbine blades)
  • LC50 for general purpose CMM-based scanning
  • XC50 cross scanner incorporating three laser lines for high productivity inspection of features like holes in sheet metal or composite parts
  • ModelMaker series handheld scanners to be mounted on an articulated arm
  • K-Scan handheld scanner with large working volume and without a need for a mechanical localizer like a CMM or articulated arm (K-Series optical tracking is used)
  • K-Robot - This is an industrial version of the K-Scan used for automated scanning on a robot.

Metris not only offers the scanner but also CMM, articulated arms (MCA) and point cloud acquisition and processing software (Focus series).

The advantage of laser line scanners is that they provide a large amount of measurement points in a short time interval, e.g. 80,000 points/second. On the other hand, line laser scanners have to be positioned relatively close to the surface (typically between 50 and 200 mm stand-off).

• Laser Radar - With Laser Radar, an infrared laser beam is projected on the measurement surface. The coordinates of the measurement point are calculated by combining the azimuth and elevation angle of the laser beam with the distance between laser source and reflection point. Different from a laser tracker, the Laser Radar only needs a fraction of the laser power to be reflected in order to calculate the distance to the reflection point. As there is no reflector needed, Laser Radar measurements don’t need an operator to climb on the measurement object and measurements can be fully automated for overnight inspection.

Laser Radar has a measurement volume up to a 60m radius. Compared to a laser line scanner, Laser Radar measures points one-by-one. This new generation of metrology instruments precisely measures large-scale geometry without requiring photogrammetric dots, laser tracker spherically mounted retro-flectors or probes. The automatic measuring capability saves measuring time and manpower.

Last but not least, Metris Integration Services assists customers in successfully deploying metrology-enabled manufacturing solutions incorporating in-house solutions like iGPS, Laser Radar or Optical CMM, third party 3D measurement solutions, and the related hardware and software to deliver completely automated systems for large scale projects.

[Don Waterman, Neptec] Neptec has more than 10 years experience in the design, build and operations support of space-borne vision systems and 3D imaging systems for NASA’s Space Shuttle.  During that time, Neptec has delivered two major vision systems: a Space Vision System (SVS) and a Laser Camera System (LCS).

The SVS analyzes and processes video imagery of objects from one or more cameras.  These objects are typically payloads that are manipulated by the Space Station Remote Manipulator System (SSRMS) and the Shuttle Remote Manipulator System (SRMS).  Target areas are located at pre-determined positions on the objects.  With knowledge of the geometry of the target arrays on each object in a scene, the SVS, utilizing photogrammetric processes, calculates the position, orientation, and rate of movement of each object relative to the viewing camera.  With knowledge of the viewing camera's position and orientation, the SVS then derives the position and orientation of each object relative to any other point of reference.

The position, orientation and rate information is presented to an operator in both character and graphical representation.  Once configured, the SVS operates continuously, tracking an object as long as the targets remain within the camera's field of view.  It operates in real-time, generating a new measurement of the object's position and orientation with each video frame.

The LCS is an infrared, auto synchronous triangulation laser sensor developed to operate in the rigors of space on the end of the Orbiter Boom Sensor System. This sensor provides both 2D imagery and 3D data of the shuttle thermal protection system (TPS) used for on-orbit inspection of the TPS.  Software for the LCS includes Neptec-developed machine vision algorithms and tools to fuse 2D and 3D data to maximize the information available to Shuttle Flight Operations during periods of critical decision-making.

Neptec has, over the last several years, expanded its business into the industrial automation market which has led to the development of a 3D non-contact Laser Metrology System (LMS) for both in-line and off-line short range and medium volume metrology applications offering a solution for rapid, high precision, three-dimensional metrology.  The LMS design is based on the LCS technology with improved performance characteristics – a measurement precision of 0.002 inches or better while scanning at 20,000 points per second.  The system incorporates a red, fully steerable laser, and is capable of a 30° by 30° field of view.  The steerable laser beam allows the user to scan the entire field of view in seconds, select a part of the field of view and overlap a second, higher density scan, onto the first scan.  It also allows the user to define a specific beam trajectory to capture only the data of the features that matter. 

As the LMS scans the object of interest, it generates a cloud of independent spatial data points which, after calibration, are transformed into the 3D spatial coordinates of points located on the surface of the object of interest.  The LMS computes a high-resolution, 3-dimensional (3D) map of the surface features of the object of interest or the location of discrete target points on the object of interest. Measurements can be made between points on the model that can be translated into real world coordinates or compared to design models (e.g. CAD) providing a high resolution, detailed inspection capability of complex geometries and areas of damage. 

Other differentiating features of the LMS include:

• Precision - Provides a precision of 25 microns in all three axis.
• Large Scan Area - The Neptec LMS system can scan a 1.15m² area. The Neptec LMS optical scanner enjoys a significant advantage over competitive laser stripe scanners in the marketplace which scan relatively small volumes. 
• Size – The Neptec LMS system is small, lightweight (<5 kg), compact and portable, as are many of the other laser stripe scanner cameras. The Neptec system enjoys a competitive advantage over laser stripe systems where the scanner configuration requires that it is fixed to a CMM or robotic arm. 
• Ease of Use - The Neptec LMS system is simple to set up and easy to use as are single point laser and laser stripe systems.
• Eliminates Hard Tooling - The Neptec LMS metrology system does not require jigs or fixtures to locate parts within the field of view of the camera. Model and design changes are incorporated into the system through the software. 

[Giles Gaskell – Nvision]  Unlike most laser or white light point data collection devices, our system is intended for inspection in a production environment. Its multi-axis automation and the fact it can scan very shiny or even transparent objects without coating them marks it out.

2. What specific 3D-based machine vision applications related to the aerospace/defense manufacturing industry do your products address?
[James] 

• Laser line scanners are used to inspect the dimensions of turbine blades and the shape of free-form composite components.
• Laser Radar aims at large scale inspection applications, e.g. inspection of wings, fuselage sections, engine housings, antenna’s, etc.
• The main application of iGPS in aerospace is the assembly of large parts such as fuselage and wing assembly. By putting sensors on each of the parts to be assembled, their relative position can be assessed with high accuracy and corrected automatically in order to get both parts aligned perfectly before assembly.
• iGPS and K-Series can also used to accurately position and track manufacturing tools such as drilling, riveting or painting robots or laser projection systems.
• K-Series-based adaptive robot control enables high accuracy positioning of industrial robots under variable loading conditions.

[Don] The LMS is being developed to address the following application areas:

• Part Identification (Recognition) Intelligence - Part identification has the capability to provide identification of the part being scanned to the user.  This feature requires the user to input geometric parameters to the LMS system that can be used to identify the part.  The geometric parameters may be diameter, depth, location and number, part width and height. 
• Part Accept/Reject Capability - Part accept/reject capability includes comparing a feature of a scanned part to a pre-defined tolerance requirement.   The dimension tolerance requirement exists in an as-designed 3D model developed using CAD tools. 
• Part Request/Hand-off Capability - Part request/hand-off capability in production lines having in-line machining centers.  When a part has completed a quality check, a new part may be requested or if the part has failed a quality check, the part may be handed off to another machining center for a rework operation.  Assessment of the part request/hand-off capability involves comparison of measured features with CAD part dimensions, accept or hand-off to a machining center for rework and notification to the user.
• Deformation Detection Capability – Deformation (impact damage, corrosion, etc.) detection by measuring the out-of-plane displacement (positive or negative) of a surface to detect damage in aircraft panel assemblies. 
• Gap & Flush Measurement Capability - Gap and flush measurement is an assembly quality inspection.   Gap and flush measurements are used to verify the fit of mating assemblies such as, door to door frames, automotive dash boards to window frames, and seats to interior car frames. 

[Giles] Machined parts that need to be 100% inspected such as aircraft turbine parts.

[Bernd] Quality assurance (QA) of large pump housings, centerline detection of large forged object e.g. crank shafts, capture of volumes in stockpile applications, capture of arbitrary environments for machine vision simulation and optimization. There will be more to come in the area of 3D scale bridges in harbors, quality assurance of moulds, crane arms, etc.

3. What has been the most difficult 3D-based machine vision application in the aerospace/defense manufacturing industry that you have addressed and why? What were some of the specific application issues (throughput, appearance variables, position variables, line integration issues, etc.)?
[Giles] Measuring the profiles of vanes placed closely together in a gas turbine nozzle. All the surfaces to be measured can be seen, but the line of sight has a very acute angle to the surface. Our system requires only a 10 degree angle to the surface to measure, which allows it to address this extremely demanding application.

[Bernd] None, really, as long as our accuracy and noise and ambient qualities are OK. We develop further noise and accuracy qualities in order to expand our application range. 

[James] Most aerospace applications are characterized by important challenges due to their size, the stringent manufacturing schedules, need for in-depth automation and often customer specific integration. The Metris Integration Services group is specifically established to overcome these challenges by working in close cooperation with the customer to smoothly integrate the Metris technology into the manufacturing processes.

One of the most challenging projects has been metrology guidance during the assembly of a full airplane at a major airplane vendor. The objective was to accurately align the parts to be assembled in the shortest possible time in order to reduce the total assembly time. This requires high accuracy measurements in an assembly hall where temperature changes and vibrations are likely to occur. Metris Integration Services developed a dedicated solution to overcome these issues.

4. Can you provide some insight into the principles embedded in your 3D-based machine vision products?
[Bernd] Phase Shift distance sensor. 

[Giles] The system used a focused white light spot and a camera mounted in a sensor with up to 7 axes of movement available.

5. Can you provide some insights into the specific hardware/software implementation designs of your 3D-based machine vision products?
[Don] Auto synchronous triangulation scanning.

[James] The Metris Integration Services group has developed a process-centric work cell management toolkit to simplify, automate and error proof aerospace manufacturing processes. Maximum productivity comes from software capable of driving these tools centered on the manufacturing process itself. With a core foundation layer architecture, scriptable process engine, and con?gurable user interface, the process management enables the management and guidance of all alignment, inspection, projection, and machine automation tasks. Whether there is a need to semi-automate manual assembly or inspection processes via laser guidance, automate your jig inspection routine, or simultaneously track and align mating components in 6 DOF, the core software controls the entire process. Spatial analyzer is the most used software application to analyze results with Laser Radar.

[Giles] The sensor is mounted on a conventional CMM frame.

6. What new 3D-based machine vision products or advances to your existing products have you introduced in the past year? Are they products specifically targeted at the aerospace/defense manufacturing industry?
[James] Metris has introduced the Metris Integration Services group.  As such, Metris answers the trend in which major aerospace manufacturers expect their suppliers to integrate their solutions into their manufacturing processes rather than doing these themselves.
A few of the successful implementations of this team has been the aircraft assembly solution mentioned before or the adaptive robot control solution used to accurately position aircraft skin components under a riveting machine.

[Giles] The MAXOS machine and its ability to scan highly polished and even transparent surfaces.

[Bernd]  None yet but will have better noise and accuracy levels in 2008. 

7. What historically were the barriers to the adoption of 3D-based machine vision systems in the aerospace/defense manufacturing industry and what are today’s barriers to more widespread application of 3D-based machine vision systems in the aerospace/defense manufacturing industry?
[Bernd]  Measurement quality, ruggedness of unit, price, interface to integration, volume of sensor, weight of sensor. 

[Giles] There has never been an accepted ISO standard for calibration of non-contact scanning devices. The MAXOS can be calibrated according to 10360-2 as it is CMM-based and can measure the shiny calibration artifacts.

[James]

• The main barrier for laser line scanner adoption is the fact that their principles are totally different from the traditional touch trigger approach. This means that measurement approaches have to be adapted, which is not obvious within the metrology community. Also, the lack of laser scanning accuracy standards doesn’t help.
• The barrier for innovative products like Laser Radar and iGPS was mainly due to the lack of industrial references. With the successful implementations that we currently have, this will change rapidly.
• The introduction of metrology into the manufacturing process requires a smooth integration between measurement and production tools. This is the reason why Metris established the Integration Services team.

8. What advances associated with the technology infrastructure of your 3D-based machine vision products has lead to more rigorous performance (reliability, repeatability, accuracy, etc.) in your newest products (optics, lighting, vision hardware, vision software, cameras, etc.)? And, what are the specific advantages of these advances in terms of price/performance?
[Giles] Improved optics that increase accuracy and software improvements to raise the speed of data collection. The benefit to price/performance is the increase in the number of conventional CMMs it can replace.

[James] Use of improved optics, accurate encoders for automated measurements, optimized calibration algorithms to maintain constant quality, development of software that aims to integrate the measurement tools into the manufacturing process. 

[Bernd] Price performance improves as we increase performance of units as above and keeps prices stable. 

9. What changes in the underlying 3D-based machine vision technology (vision engines, lighting, cameras) do you anticipate in the next 2 – 3 years that will yield even better performance and the ability to address even more aerospace/defense manufacturing industry applications?
[James] Improvements in system performance will mainly come from advances in underlying DSP and application software. DSP software will enable improvement in accuracy (noise filtering, error rejection, more detailed calibration models), increased measurement robustness (automatic adaptation of measurement settings to material and environmental conditions) and increased measurement speed (more DSP power).

On the application side, the focus will be on integrating the measurement tools into the manufacturing and assembly process. 

[Bernd]  As above, reduction in size, weight, noise, price, increase in ruggedness, accuracy and interfaces. 

[Giles] Faster data collection. Better point resolution and accuracy.

10. How will those changes impact the aerospace/defense manufacturing industry?
[Giles] Speed up the ability of manufacturers to perform 100% inspection of complex machined parts and remove what in many cases is an expensive bottleneck.

[Bernd] Do not know this in general - but it will impact density of sensors and their intelligence, thus make machinery more independent, running with less personnel, as well as safer and more efficient (e.g. in mining).  

[James] Faster and more accurate production and assembly. Typically, this will shift skilled personnel from labor-intensive measurement tasks to engineering and analysis tasks. 

11. What are some market/process changes that are taking place in the aerospace/defense manufacturing industry that are driving the adoption of 3D-based machine vision systems?
[Bernd]  The trend to move lower end mass market machinery production and engineering to China means new product developments at the higher end in western countries.

[James]

• Higher manufacturing volumes of airplanes combined with shorter throughput times. This will require further automation of both the manufacturing and inspection process. The tendency to manufacture first-time-right will increase the need for metrology-driven manufacturing processes.
• Manufacturing and assembly that are located in different locations, meaning that in-depth inspection is a key requirement before the components are shipped to the final assembly
• Development/Manufacturing of larger airplanes
• Use of special materials (composites) will requires a review of the traditional inspection processes (e.g. non contact measurements will gain importance)

[Giles] The need to reduce costs (skilled manpower) from the manufacturing inspection process.