DXOMARK Benchmarks Displays Using Real-World Conditions and Scientific Data from ProMetric® Imaging Colorimeters

About DXOMARK

DXOMARK provides independent device quality benchmarks based on scientific assessment. Based in Paris, France, the company operates state-of-the-art labs with documented test protocols to evaluate the objective and perceptual performance of DSLR and DSC cameras and their lenses, smartphone cameras, chipsets & ISPs, battery life, and displays under real-world conditions; DXOMARK also evaluates audio quality of speakers and smartphones. The results of hundreds of tests are aggregated as numerical values, which are a small set of sub-scores for the key quality attributes and an overall quality index, the device’s DXOMARK score. Referenced widely by the global consumer electronics community, a DXOMARK score is used by manufacturers and news media to communicate a device’s quality ranking.

Since launching its website in 2008, DXOMARK has rapidly established a reputation among consumers and manufacturers as a trusted source of reviews based on scientific data. The company began by testing DSLRs and camera equipment, later adding smartphone camera systems. With touchscreen displays the pivotal interface for smartphone operation, DXOMARK extended its testing to include display performance. Opened in 2020, DXOMARK’s display test lab reflects the company’s dedication to device benchmarking using scientific testing, applying the latest display test equipment and methods to evaluate quality according to the user experience.

Display Quality as a Factor of User Experience

Device manufacturers and display panel makers commonly perform visual inspection and display calibration as part of their quality control operations. Typically, displays are tested in dark labs, with a measurement system aligned at a specific distance and angle relative to the display. These settings limit the effects of variables like ambient light, view angle, and focus, isolating the display to characterize its brightness, color, and other visual qualities under optimal conditions.  Of course, this is not the way a user visualizes their smartphone or any device display.

DXOMARK takes a different approach to measurement. “We measure aspects like brightness, contrast, color, flicker—all sorts of parameters, which are physical attributes of the display,” explains Nicolas Touchard, Vice President of Marketing at DXOMARK. “But we make assessments under real-life conditions. That is something quite different from the test conditions as defined in the industry standards as used by manufacturers.” Users view their devices under a range of ambient light conditions—at night, in sunlight, indoors, outdoors—and at variable distances, angles, and visibility conditions. Manufacturer testing measures specifications of the display itself, while DXOMARK provides an objective understanding of the quality of the display under real-world conditions.

“All the standards right now are based on measurements made without ambient lighting, but that’s not a consumer use case,” explains François Grandclerc, Display Quality Director at DXOMARK. “DXOMARK’s approach combines perceptual analysis and objective measurements, all being performed in conditions consistent with actual use cases. Our objective was to change the spirit of existing standards by doing measurement where nobody else wanted to go.”

Fundamental to DXOMARK’s test processes are scientific measurements and repeatable test protocols. Since user experience of a device is inherently subjective—and subjective analysis alone cannot provide a benchmark—the combination of perceptual analysis and scientific measurements are required. Both analyses should be performed in conditions that reproduce the user’s environment for the most relevant measurements. Additionally, measurement systems, test setups, and protocols should be thoroughly defined and always performed the same way so that measurements are reliable regardless of who performs testing and when testing occurs. As Touchard explains, “If the instruments, the way the measurements are done, the setup, the protocols are not defined, you can’t have consistent data.”

Scientific data includes photometric values measured from the display under various conditions, which must be acquired in a controlled environment, limiting human intervention to standardize the test process across devices and ensure objectivity.

DXOMARK’s test processes employ scientific metrology tools, visual inspection systems and protocols, software, and robotics. Devices like spectrometers and imaging colorimeters are measurement systems that acquire photometric data such as a display’s luminance value (Lv), chromaticity coordinates (u?,v? on the CIE 1976 chromaticity diagram), brightness and color uniformity, and contrast. Robotics automate test equipment and devices, including alignment, relative angle, and positioning, to match real-world test scenarios. A centralized software platform controls all aspects of the test process remotely, such as sending test patterns to the display via WiFi, operating measurement equipment and robotics, and adjusting variables like ambient light.

DXOMARK’s test protocols define the procedure and conditions relevant to measuring display attributes (performance parameters to be tested) according to each device use case (how a device would be used in the real world). “When we do our scoring, we test across many use cases like web-browsing, night reading, in-car navigation, gaming, watching videos. All of these use cases encompass what people will do with their smartphone,” explains Touchard. “In each of these scenarios, we check attributes and do measurements of physical characteristics across these attributes, and we do this considering the ambient environment—the environment of the user.”

Not all display attributes must be tested for every possible use case. DXOMARK’s test protocols ensure the most relevant attributes are evaluated for each scenario. For a use case like web browsing, for example, most display attributes (readability, color, motion, touch, artifacts) should be tested as part of the user’s web browsing experience. For in-car navigation, however, fewer display attributes (readability, artifacts) require testing since the user is not typically interacting with the display.

A matrix of display quality attributes, testing conditions, and use cases that illustrate the complexity of DXOMARK display tests based on real-world conditions. Image courtesy of DXOMARK.

Components of a Scientific Display Test Lab

To meet the demand for display quality benchmarking, the DXOMARK team set out to develop a new test lab dedicated to evaluating display performance. DXOMARK hired several specialists to outline their display quality benchmarking procedure. Several tools would also need to be selected to accomplish the objectives for testing according to the defined attributes and use cases.

DXOMARK evaluated a range of display test instruments for the display test lab, including imaging colorimeters, specialized lenses, gloss meters, flicker measurement devices, spectroradiometers, high-speed cameras, and even robotics to test touchscreen response. Bringing together all of the pieces for automated display evaluation also meant integrating measurement equipment, robotics, and software into a controlled fixture. The DXOMARK lab features a custom-engineered test chamber—the Display Bench—that automates DXOMARK’s display testing under repeatable conditions. The Display Bench includes robotic mounts for remote alignment of smartphones and measurement equipment and an LED panel to mimic ambient lighting conditions from darkness to daylight.

Inside DXOMARK’s Display Bench, featuring a Radiant ProMetric® I Imaging Colorimeter, multi-axis robotic mounting equipment, and LED array to emulate lighting in use-case scenarios. Image courtesy of DXOMARK.

For its automated visual inspection system, DXOMARK chose the ProMetric® I-series Imaging Colorimeter and FPD Conoscope Lens from Radiant Vision Systems. Within the Display Bench, the ProMetric I Imaging Colorimeter measures luminance, chromaticity, and uniformity of the display in each test scenario, as ambient lighting and other conditions are varied for each display attribute and use case. Each measurement uses multiple test images, such as a white image for luminance and uniformity, colored images for chromaticity measurement, and a checkerboard contrast image for contrast and modulation transfer function (MTF).

Radiant’s equipment relies on imaging, enabling the ProMetric system to capture the entire display at once and measure multiple attributes in a matter of seconds. Test images are pushed to the display in sync with camera and software operations, which are specific to each measurement scenario, providing fully automated testing without downtime.

The ProMetric I Imaging Colorimeter synchronizes test operations with images on the display for comprehensive evaluation in seconds.

Outside or inside of the Display Bench, Radiant’s FPD Conoscope Lens is paired with the same ProMetric Imaging Colorimeter to capture a polar plot of the display as viewed to ±70° in a single image. This measurement characterizes luminance, chromaticity, and contrast changes as the display is viewed by a user from different angles. A typical use case would be when multiple users are viewing photos or movies on the same device, and each user’s vantage point varies based on their position relative to the display.

A Radiant Vision Systems FPD Conoscope Lens measures the view angle performance of a display (left), providing a polar plot of luminance as seen across angles to ±70° (right).

The DXOMARK display test lab also incorporates equipment from Radiant’s sister company, Konica Minolta Sensing in Europe. In the Display Bench, a Konica Minolta CS-2000 Spectroradiometer is used to measure brightness and color fidelity of the device over changing ambient-light conditions and to provide calibration data for the Radiant ProMetric Imaging Colorimeter. A Konica Minolta CM-25d Spectrophotometer is also used to evaluate gloss based on light reflected from the display surface.

Benefits of Radiant Vision Systems Equipment

The capabilities of Radiant’s test and measurement equipment align well with DXOMARK’s goals for user-focused display testing in several ways.

Radiant’s imaging colorimeters not only provide objective, scientific data according to colorimetric values, but are designed with tristimulus optical filter systems that measure light as it is perceived by the human eye. During measurement, subsequent exposures of the display are captured through each of the camera’s filters, which are weighted to human spectral sensitivity curves (color-matching functions standardized by CIE), ensuring the truest measure of light and color quality according to human perception.

Radiant’s ProMetric imaging systems also come in a range of high-resolution sensor options to accommodate the increasing resolutions of today’s smart device displays, and even measure multiple displays at once.

 “For a smartphone now, you can have a 6-inch display with around 4K resolution. So, we needed a tool that shows us a luminance or color map with enough pixels to measure one device—at least,” states Grandclerc. “But in our approach, we measure four devices at the same time, and we need sufficient information measured across all devices. We needed the highest resolution available on the market, and it was Radiant.”

Radiant equipment also gives DXOMARK flexibility to meet a range of test criteria. When testing view angle performance, the ProMetric Imaging Colorimeter’s standard lens can be swapped for the FPD Conoscope Lens to acquire angular measurements, providing “the ability to switch from one to the other without buying another tool,” comments Grandclerc.

Another benefit of Radiant’s equipment is electronic control. The focus and aperture settings of the ProMetric system’s lenses are remotely adjustable from software. Integrated with automated controls, specific lens settings can be applied for different device positions and alignments set by robotics to ensure repeatable testing without human intervention. “We can develop a smart algorithm to detect where the best focus is—same for the aperture,” states Grandclerc. “We wanted something that we could control electronically without putting an operator inside of the Bench to check everything—to avoid any mistakes, to be sure we are repeatable. Repeatability is key in our business.”

Similarly, each ProMetric imaging system is controlled by software to capture images and apply measurements in sync with automated processes. Radiant’s TrueTest™ Automated Visual Inspection Software is the control platform for each Radiant device and features programmable API and SDK to enable integration into fully automated test benches and testing processes. Commenting on the importance of software integration, Grandclerc recalls, “It was maybe one of my first questions to Radiant—what can you do with the SDK to drive the camera?” With this capability, DXOMARK was able to incorporate the Radiant hardware and software functionality into its turnkey test system.

Equipment is only one part of a complete solution, however. Global support from both Radiant Vision Systems in the USA and Konica Minolta Sensing in Europe were equally integral to DXOMARK’s test setup and operations. “The quick and professional support from the team in the United States and here in Europe is something that is very important,” states Grandclerc. “The Radiant camera is a key tool for us, so when there is an issue, we need quick support.” Although development of the display test lab extended into 2020, amid the COVID-19 pandemic, team members from both Radiant and Konica Minolta came together to ensure DXOMARK’s goals were met. “We even did some calibrations via video-conferencing,” notes Grandclerc.

Beyond a Benchmarking Score

Looking to the future, DXOMARK plans extend its display quality testing, adding additional equipment and protocols to evaluate displays across new use cases. However, the company’s focus is not limited to applying scientific measurement to produce benchmark scoring for consumers.

As a leader in measurement across the entire device component ecosystem, DXOMARK aims to give manufacturers and consumers alike a view of display performance that is easy-to-understand, relevant, and furthers device innovation. DXOMARK’s test data not only allows apples-to-apples comparison of quality across devices but provides manufacturers and display panel makers with scientific metrics that help drive development of devices with improved performance. This knowledge is key for any device manufacturer whose goal is to give their customers the highest quality user experiences, beyond display performance specifications measured in a dark lab.

“Our primary purpose is to make accurate, repeatable measurements and perceptual analyses that correlate with human perception, and which (at the end of the day) enable us to give a score—which is very consumer oriented,” explains Touchard. “Then our goal is to deliver very detailed reports of the characteristics of the device we’re testing—serving our customers and providing very accurate and in-depth analysis so that they can understand how they are positioned relative to competition and how they can make improvements.”

With scientific data from Radiant’s ProMetric Imaging Colorimeters, Konica Minolta systems, and other leading display test and measurement equipment, DXOMARK guides the consumer electronics industry toward a better understanding of device quality based on the ultimate standard—the user experience. By qualifying the performance of displays under real-world conditions, DXOMARK helps set the industry's focus on user-centric design.

Learn more about DXOMARK: www.dxomark.com

“DXOMARK develops and uses test methods to mimic smartphone users under different conditions, and the Radiant camera provides very accurate, repeatable, extensive measurements to test all photometric parameters that count for visual display quality.”
– Nicolas Touchard, Vice President of Marketing, DXOMARK