Machine Vision Brings Interactive Displays to Life In a 3D World

Empowerment is not easy when it comes to human-machine interactions. Moving digital representations of reality through electronic conduits is relatively simple, but try to do it without a keyboard, mouse or specialized hardware and the limitations of modern interactivity are quickly revealed. 

The answer is a new generation of interactive displays and signage that takes input from the world around it without the need for tangible I/O devices that require physical touch. True empowerment comes without limitations, and it is this three-dimensional, interactive world that machine vision is helping to build below the radar of mainstream media. 

IBM’s Everywhere Concept

A recent IBM global survey of advertising experts predicts that the next 5 years will see more changes to the advertising industry than the previous 50 years. Think about that. The last 15 years have seen the birth of the Internet, which nearly brought the print publishing world to its knees and significantly changed how people get their news and information. 

Specifically, IBM points to four drivers of change: Attention, where consumers are in control; Creativity, where amateurs can product information similar to professionals; Measurement, where advertisers know what works and what doesn’t; and Advertising inventories, which is related to new ways to buy and exchange advertising space. Interactive displays and signage are likely to be part of this revolution by giving users unfettered access to information, while unobtrusively observing and learning from their actions. 

IBM’s (Hawthorne, New York) Everywhere Display is a perfect example the potential between machine vision and interactive displays and signage. The IBM Everywhere Display is a prototype projection and imaging system that transforms the consumer shopping experience, giving customers a faster, more convenient way to access inventory and product information.

The Everywhere Display projects images containing store data onto any surface in the store and then reacts to customer responses to the projected information. Shoppers simply touch the projected surface to find information, as if they are interacting with a touchscreen computer. Everywhere can project onto walls, shelves, floors or any other surface and can be used for digital merchandising to show advertisements or other promotions when the customer is not directly interacting with it.

The Everywhere Display achieves this by combining an overhead projector, a rotating mirror, a video camera, and advanced software, to create a virtual touch screen projected on any surface. For example, by touching the projected surface, a customer would be able to find information such as jean sizes and styles in stock in a clothing store, the type of memory card needed for a digital camera, or a store map. 

In Rheinberg, Germany’s METRO Group Future Store, the Everywhere Display is linked to a wine information kiosk. When a patron asks about a particular vintage, the Everywhere Display projects direction arrows on the floor of the store, leading customers to the precise location of the wine they have just chosen using the online kiosk. The device also shows animated content about wines on promotion on the floor of the wine aisle and directs shoppers to these as well as pointing out the best selling wines in the store.

Immersive Environments

IBM’s Everywhere Display may be the future of in-store, consumer communication, but for professional applications in energy, education, defense and aerospace, precision counts for more than 10%-off coupons floating across store shelves. EON Reality Inc. (Irvine, California) is tackling the professional interactive display market with two products: Icube (pronounced, “eye cube”) and TouchLight, a collaborative project with Microsoft Research (Redmond, Washington). 

Icube is a large area work cell with displays on three sides, creating virtual “cubes” that range from 10 feet on a side to 30 feet. “Icube delivers real time feedback to the user based on markers fitted on the 3D glasses worn by the user,” explains Mats Johansson, CEO and Founder of EON Reality. “If you’re looking at the interior of an aircraft cabin – which is a big application area for us – and you crouch to look underneath a table, the video displays change perspective to show you the underside of the table.” 

Icube requires visual ‘markers’ on the person as well as any other interactive devices, such as wands, remotes, or handheld controllers, because Johansson says professional applications need precision and fluidity within large spaces. Between four and 12 cameras track the markers within the Icube, and feed images back to a PC that provides the image processing horsepower along with advanced display and rendering capabilities. 

“If you’re looking at the horizon to infinity, even small movements can lead to be perspective changes,” explains EON Reality’s Johansson. “That’s why we use a minimum of 3 cameras to see the target and do a good extrapolation and determine the position of the user [with full 6 degrees of freedom.] It’s like the ancient sea farers that needed three stars to determine their position. We could do it with 2, or even 1, but 3 gives us better accuracy. We could use wireless or gyro-based tracking systems, but in these environments, there is often a lot of wireless interference, which can interfere with the system. The optical approach has proven the most robust.” 

While consumer graphics and display cards have become so powerful that EON Reality can use COTS equipment, Johansson says he typically uses professional optical elements, including cameras and image processing for the motion tracking elements. “One thing we’d like to see - and has been missing for quite a while - is very good augmented reality display headsets, or head mounted displays. If someone could truly solve that problem and bring it to the consumer, it would open a world of applications in many areas,” says Johansson. 

While machine vision may not solve the head mounted display problem, it may have offer improvements to the 3D motion tracking system. Companies such as AIA member, TYZX Inc. (Menlo Park, California), and Canesta Inc. offer compact solutions for determining 3D positions for all pixels within an image. 

TouchLight is based on a Microsoft patent that places a pair of IR stereoscopic cameras behind the display. By making gestures near the screen, TouchLight interacts as if you had touched the display. Placing the cameras behind the screen allows participants to get very close to the display without having to worry about obstruction or shadows interfering with the motion tracking system. 

Today, EON Reality is adding collaboration to the Icube concept through EON Coliseum, a software program that allows Icube installations to work with virtually any computing platform, from Icubes on other continents, to laptops viewed through 3D glasses in an airport lounge. “It’s the next step in the virtual, or as we call it, immersive meetings. When you can do real product demonstrations from half a world away and make a sale without the travel costs and lost time, you start to see the need for immersive display systems.” 


Footnote:
Learn more about interactive displays and machine vision in the world of art by reading, "The Show Must Go On."