Elevating the Art of Entertainment – Robotically

Don’t blink! Robots are popping up in unexpected places. They’re hoisting pop stars over our heads, dancing with pianos, making cameos in blockbuster movies, and creating spectacles of color and light in venues large and small. 

These aren’t your toy bots or typical infotainment robots. These are industrial robots. Often heavy-payload, production-ready robots you commonly see on the automotive assembly line in Detroit.

Put them on the stage and now you have a performer’s best friend. An extension of their art. A trusted partner to enhance the audience’s experience and create a unique, memorable event. 

Behind the scenes, they’re even more intriguing. Serving double-duty on filmmakers’ sets, capturing awe-inspiring shots with superhuman control and repeatability. Finding the nuances in form and fluidity to create magical images of products and iconic brands. Robots in the entertainment industry are marvels of the sublime.

Concerts and Performing Arts
ABB Robotics was one of the first industrial robot manufacturers to take the stage. Their intro to the entertainment industry was a high-octane rock and roll show.

Typically at home on the industrial assembly line, ABB robots toured the globe with Bon Jovi, set sail on Royal Caribbean, competed on “America’s Got Talent,” nearly upstaged Lady Gaga at the Grammys, and recently performed on the same stage with icons of the famed Peking Opera. 

Profiled in this magazine article, Andy Flessas (aka andyRobot®) is the artist and robot whisperer that brought many of these robots to life. He recently relocated his business from Las Vegas to Nashville where he has a new studio under construction. Flessas likens robots to creatures of nature, such as gazelles, lions or cheetahs, noting that each robot has its own personality. His canvas is the stage.

“I’m using robots to amplify traditional stagecraft. It’s re-examining what stagecraft is and then amplifying it with the robot.”

Stagecraft Robotized
Flessas has a degree in film animation and is a 3D animator. In the 1990s, ABB Robotics contracted Flessas to create computer-animated dancing versions of its robots to entertain customers while they loaded ABB’s software onto their PCs. 

This was the first time Flessas saw a robot. He was instantly smitten. So captivated, in fact, that he attended ABB’s training school at its U.S. headquarters in Auburn Hills, Michigan, where he learned to program robots.

“What I found with robot programming is that it’s very linear,” says Flessas. “At that time, it wasn’t concerned with how the robot looked. With theater and live stuff, it’s all about the way the robots look and the way they move. The way they look when they move.”

Flessas created a plug-in for Autodesk’s computer animation software Maya. Robot Animator® enables computer animators to program robots much like animating a character. The software works with the smallest to the largest ABB robots, from the IRB 120 to the IRB 8700. He’s currently working on a version for ABB’s YuMi dual-arm collaborative robot that will animate the cobot’s pair of seven-axis arms. 

Robot Animator is also compatible with other robot brands. Flessas says anyone can use the software. No programming knowledge is required. He’s been tweaking the software for 20 years and it’s been used in live stage events, brand PR, and in the making of films and commercial spots. The software will be released for licensing later this year.

Flessas was just granted his seventh patent. In 2009, he patented the RoboScreen® entertainment system, which combines an ABB six-axis robot with the latest in LED display and media control technology to offer eye-catching video content and synchronized animatronics. Within a few months, Bon Jovi licensed that technology and took five RoboScreens on the road for the band’s worldwide Circle Tour. 

In 2014, Royal Caribbean Cruise Lines debuted the RoboScreen system on board its newest ship, the Quantum of the Seas. It marked the first permanent installation of industrial robots as part of a major performance venue and the first on an oceangoing vessel. See how it all came together.

Now check out this compilation video for a behind-the-scenes look at some of andyRobot’s eye-popping work with ABB robots (and Lady Gaga).

Live Theatre
These days, Flessas is studying how different cultures perceive robotic entertainment. For example, he says Americans tend to like the big super moves of these robotic performers, while the British prefer to see the near misses at high speeds. By contrast, the Chinese market prefers a more formal presentation, like an opera, where you can appreciate the beauty and subtleties of movement. He says his latest event in China was a “real class” in that study.

In June, four ABB robots took the stage at the National Centre for the Performing Arts in Beijing. They were part of an international ensemble of Chinese folk-art performers combined with technology and performance artists from the U.S. and Europe. Produced by L!VE Kingway and directed by Zhang Yimou, who directed “The Great Wall” starring Matt Damon and the 2008 Beijing Olympics opening and closing ceremonies, “2047 Apologue” explores how humans interact with technology in a series of short theatrical performances.

The ABB IRB 4600 robots act as puppeteers (pictured in Image 1), performing with a modern dancer from Sweden and traditional Chinese marionettes. Watch brief vignettes of the different performances. “2047 Apologue” is slated to tour other cities in China and play in Edinburgh, Scotland, later this year.  

The Touring Machine
Flessas does more than write the software and program the robots. He optimizes both the software and hardware for a successful installation, much the way a robot integrator works in the industrial realm.

“In all cases our projects are with very large entertainment organizations, so we try to use the resources they already have in place and to create autonomy, so it’s easy for the customer to understand the robots and bring them into their touring machine.”

Flessas says he’s usually on site for the first few shows until the crews are trained. “Then we set them free and we have them working for years going around the planet.”

ABB’s global coverage and support network help ensure these installations perform as planned. Robots are renowned for their repeatability and flexibility, but it’s their reliability that’s critical to the performing arts. After all, these industrial robots are built to work 24/7 in demanding environments for up to 8 to 10 years with minimal maintenance. Because when that curtain goes up, failure is not an option.

Safety on the Set
Coming from the industrial world where we’re used to having robots behind safety fences, the sight of Bon Jovi’s frontman dancing atop the RoboScreens might stop you in your tracks. But the safety procedures on these live sets are just as rigorous if not more rigorous than industrial environments.

Flessas says these A-list performers are like stunt people. At one point in the show, the RoboScreens transform into a staircase and Jon Bon Jovi climbs to the top.

“Everything is heavily choreographed and the set is tightly controlled,” says Flessas. “Security is heavy. Take away the robots, and the stage and all its platforms alone are still very dangerous. But these are well-practiced super professionals.

“Safety protocols actually become more intense,” he says. “There are more layers. We hardcode SafeMove.”

SafeMove is ABB’s safety-certified robot monitoring software that resides on the robot controller. When coupled with sensors that monitor safety zones around the robot for proximity with people and objects, SafeMove allows people to safely work (or perform) alongside ABB robots regardless of the payload.

Small but Nimble Performers
These entertainment installations don’t always involve high-payload robots like those in the Bon Jovi and deadmau5 concerts (pictured), or aboard the cruise ships. The robots that had Lady Gaga’s piano dancing at the 2016 Grammy Awards are IRB 120 robots, ABB smallest. 

“We use those because they have to roll on and roll off stage, and be super light and ready for action,” says Flessas. 

These nimble manipulators wowed the audience with a laser show (pictured) to kick off the opening keynote at LiveWorx 2017 in Boston.

“They are very versatile robots,” says Flessas. “They can do a camera, a light, a laser, Gaga’s piano. In the studio, I have nine of them.”

Flessas’ robot inventory is for his own experimentation and development. They’re generally not used in the field. The production design companies usually buy or lease robots for events. Often the robots can be repurposed for other types of performances and venues.

Product Promotion
At tradeshows and conferences, brand promotion events, or new product launches, robots often take center stage to engage audiences and make a lasting impression.

Athletic apparel giant NIKE commissioned andyRobot to create a robotics display for the 2016 NIKE Innovation Summit in New York City. Check it out.

ABB robots helped celebrate the 50th anniversary of the iconic Porsche 911. Sports car fans got their first glimpse of a new limited-edition Porsche in an online interactive experience where they control two camera-equipped ABB robots to move around the vehicle so they can view it from almost any angle. This video takes you behind the scenes of the robotic exhibit.

Sometimes the robots themselves are the showstoppers. Watch this live event video from the opening ceremony of the Yaskawa Electric Robot Village in 2015. KUKA Robotics is known for posing robot duels with sports celebrities, such as table tennis champ Timo Boll, to promote the speed and agility of its robots.

Robots on the Big Screen
Robots are also on the big screen, and increasingly making appearances on the “small” screen. Let’s face it, robots are cool. That puts them on the short list for casting calls for movies, and also commercials for high-tech products like cars and electronics.

ABB robots appeared in the “Iron Man 3” movie. Both ABB and FANUC robots had cameos in “Terminator Salvation” where up to 18 robots from each manufacturer were staged to mass-produce Skynet’s Terminator army.

Two Kawasaki robots (pictured) appear in “Transformers: The Last Knight” now in theatres. In the fifth installment of this popular action-movie franchise, Transformers and humans are at war. The movie’s characters explore the depths of the sea in a submarine to find the key to save our world. You’ll have to watch the flick for the rest of the story.

Cobots are getting into the act, too. Universal Robots’ collaborative robot appeared in “Alien: Covenant,” the latest installment in the Alien movie franchise. Watch the UR in this scene. The UR also appears with Halle Berry in the sci-fi TV series “Extant.” 

Meanwhile, a team of UR robots assemble the “Maytag Man” in this TV commercial. The lightweight UR can be used alongside humans without the need for safety guarding. The Universal Robots’ distributor on the project, Numatic Engineering, says the UR robot was chosen because the cobot could be easily reprogrammed on the spot. Training a UR robot is easy with the provided touchscreen tablet, or by simply grabbing the robot arm to demonstrate desired movements. UR robots not only share the small screen with household appliances, they help build them at Whirlpool’s dryer factory in Marion, Ohio.

Now how’s this for a blast from the past? A Fiat Strada car commercial with robots, circa 1979!

Robots Behind the Scenes
When robots go behind the scenes to help automate stage production, or special-effects photography and videography for commercials and movies, the plot thickens.

“A lot of what we do is not identifiable even if you’re looking right at it,” says Joe Campbell, Vice President and Business Unit Manager-Products for gantry robot manufacturer Güdel Inc. in Ann Arbor, Michigan. He’s surprised by how automated the stages are in today’s premier theatres.

Güdel developed stage automation for the Metropolitan Opera House in New York City were The Met used Güdel’s TrackMotion Overhead (TMO) modules to automate the set for a 100-year anniversary production of “The Pearl Fishers.” Check out the case study.

Campbell says this is the same device they use to move robots on linear tracks to extend their reach. But The Met had it mounted in the rafters of the theatre. The play’s aerialists were suspended from the track.

“They looked like they were swimming against a blue backdrop,” says Campbell. “Our automation modules were moving them back and forth, scripted and totally controlled as part of the play.”

Güdel builds multi-axis servo-driven actuators. Although these systems don’t look like traditional articulated arms, they are industrial robots.

Broadcasting
Coupled with cameras, robots are also behind the scenes in broadcasting. Universal Robots’ operator-free camera solution is used for live sportscasts and on the set of TV newsrooms.

French television company FTV used a UR10 robot (pictured) at the 2016 Rio Summer Olympics in Brazil. Mounted on a trilight structure to allow for a wide range of movement, the compact UR10 was ideal for FTV’s tight workspace, which prohibited the use of a crane. The FTV director appreciated the many trajectories enabled by Universal Robot’s proprietary software, noting that he never lost any quality even in loop mode.

Filmmaking
Robots are increasingly playing critical roles in filmmaking and videography. Whether they’re shooting movies or commercial spots, robots provide unprecedented precision, repeatability, and speed behind the scenes.

One of the best examples is the blockbuster movie “Gravity” starring Sandra Bullock and George Clooney. They play astronauts stranded in space after the mid-orbit destruction of their space shuttle. Robots help create the breathtaking scenes where the movie’s characters are wildly flailing and floating through outer space as they try to catch their bearings in microgravity.

Güdel provided the linear tracks for the KUKA robots that shot those tricky film sequences. Bot & Dolly (acquired by Google in 2013) designed and assembled the robotic technology and custom software for the shoot. In this video Bot & Dolly explain how their robotic motion-control camera system was used in the making of “Gravity.”

“Bot & Dolly was ultra high-tech,” says Güdel’s Campbell. “It was very complicated to make those amazing floating shots.

“If you wanted an actor to be upside down in space, you could either spin the actor or spin the camera,” he explains. “A lot of the shooting that they did gave us the illusion that the ‘Gravity’ characters were wildly thrashing and floating in space, when in fact they were not. If Sandra is on a green background and she’s waving her arms and moving her feet, but the camera now moves upside down, it looks like she has just spun upside down.”

Flessas says several companies are using his Robot Animator software on movie sets with high-payload robots moving actors around green screen environments. He sees the use of smaller-format robots mounted with cameras becoming more prevalent as well, where they are interwoven with the robot which is moving the person. Together, they move in synchrony to create jaw-dropping special effects.

Camera Bots
Robotic cameras are also used to create stop-action or slow motion images for ads and TV commercials. Look fast! A match ignites in air, effervescence springs from soda pop … robotic cameras capture images barely distinguishable by the human eye. See for yourself.

Check out this behind-the-scenes camera work by a modified Stäubli industrial robot (aka "Spike") to create breathtaking imagery that would make “Mad Men” protagonist Don Draper swoon.

Another Stäubli robot (aka ‘Bolt’ High-Speed Cinebot) coupled with a 4K camera capable of shooting at 1,000 frames per second makes short work of difficult slow-motion shots. 

Flessas sees more on the horizon for robotic camera work. “I think you’re going to start seeing it in projection mapping (“Box” is one of the best examples of robotic projection mapping). All of the movies are going to be robotic cameras. I predict that within two years, in every movie, unless it’s some type of handheld shot, the cameras will be robotic. 

“Because any type of traditional motion camera system cannot compete,” says Flessas. “You cannot compete with a robot. Filmmaking is going to be all robotic. It’s not if, it’s when.”

Precision, repeatability, flexibility, reliability – just a few of the reasons robots are a great fit for the stage and screen. As robot prices continue to drop and integration gets easier, we expect to see more robots in the entertainment industry. Because like great performers, robots will keep pushing their limits.

RIA Members featured in this article:
ABB Robotics
Güdel, Inc.
Kawasaki Robotics (USA), Inc.
Stäubli North America
Universal Robots USA Inc.