Human-Robot Interaction: A Team Sport

How many people does it take to make a team? Two or more? How about a human-robot team?

Human-robot interaction (HRI) and collaboration (HRC) are becoming less novelty and more modus operandi. Still the conversation is more about coexistence versus direct interaction with robots. There are however emerging examples in the service robot sector for healthcare, rehabilitation, hospitality and cleaning robots. This is especially true in Asia and Europe, where the fan base is larger.

Collaborative robots, often dubbed cobots, come in many flavors, including autonomous mobile robots and mobile manipulators. Their ability to operate safely within close proximity or directly with humans – plus adaptability, portability and ease of use – make them likely teammates now and in the future.

Ergonomic Assistants
In another collaborative application, a Doosan cobot works among a team of operators at Moodng, a South Korea-based Tier 1 automotive supplier, whose customers include Hyundai, Kia, and Tata Daewoo. With a robotic helper by your side, the ergonomically challenging job of gluing interior carpet liners is much easier and safer.

His workers used to suffer wrist pain and other repetitive stress injuries from applying glue to wide areas of the carpet material while at the same time trying to keep a firm hold on the glue dispenser’s injection button. The Doosan cobot sprays a specific amount of glue at a certain speed consistently throughout the day, improving quality and efficiency.

He also comments on how easy it is to teach the cobots. In conjunction with a touchscreen teach pendant and by pressing a button on the robot arm, you can easily program a Doosan cobot by freely moving it to specific arm positions. Templates for common tasks, such as pick and place, palletizing, screw driving and insertion, also make it easy for operators to control the cobots without having to write complicated programming scripts.

Accustomed to working with conventional industrial robots, Moodng operators feel safer working next to these collaborative robots. Curved edges and smooth movements make them less intimidating. Torque sensors mounted on all six joints detect impacts as little as 0.2 N, making the cobots highly sensitive to any collisions.

From automotive components to fried chicken, you’ll find robotic teammates popping up in interesting applications. Take this robotic duo collaborating with a human chef to prepare a staple of the Korean diet, while sparing the head chef’s flesh from hot oil splatter and other hazards.

From Teamwork
Meanwhile, on the other side of the globe, researchers at the Georgia Institute of Technology have a vision for the future of human-robot interaction. Rather than thinking of robots as merely tools, they want to move toward a paradigm in which robots can learn through interaction and experience how to be effective peers for human professionals in manufacturing, healthcare and other fields.

Georgia Tech’s latest rising star is leading the charge. Matthew Gombolay is a professor in the School of Interactive Computing at the Institute for Robotics and Intelligent Machines (IRIM) at the Atlanta-based university. With a doctorate in autonomous systems from the Massachusetts Institute of Technology in 2017 and experience in robotics, AI and human-robot interaction, Gombolay leads the CORE Robotics Lab at Georgia Tech. Only a year old, the lab develops advanced algorithmic techniques to enable robots to collaborate with human teammates.

For as much as they study robots, Gombolay’s students study humans. They study human behavior and conduct experiments with human subjects in attempts to understand how and why we make certain decisions before acting upon them. (This is not uncommon among research institutions around the country studying HRI. See Dream Labs of the Future.) By better understanding humans, Georgia Tech researchers hope to better control robot behavior, especially when they are interacting with us.

Fortunately, Gombolay studied with one of the pioneers in HRI research. Julie Shah is a professor in the Department of Aeronautics and Astronautics at MIT, where she leads the Interactive Robotics Group of the Computer Science and Artificial Intelligence Laboratory, better known as CSAIL. Before joining the faculty at MIT, Shah worked at Boeing Research and Technology on robotics applications for aerospace manufacturing. She earned international recognition for her innovative research in enabling human-robot teamwork. Back in 2013, Shah was one of the only researchers in the U.S. using industrial robots to study human-robot interaction.

“Julie really pushed doing research in real-world settings,” says Gombolay. “What I found special about my time as a Ph.D. student was working directly with the people in the factories or healthcare providers in a hospital setting, to make sure whatever problem you’re trying to solve will have operational relevance. That’s a culture I’ve brought with me to Georgia Tech.”

Leveraging a strong partnership with Lockheed Martin, Gombolay’s lab is “rethinking” a novel concept. They’re exploring how they can expand on the original idea behind the collaborative robots Baxter and Sawyer (progeny of the once defunct Rethink Robotics), which were supposed to be systems you could easily drop into a factory and have the workers on the floor program them to do arbitrary tasks.

“We’re looking at specifically drilling, fastening and riveting an aerospace fuselage,” says Gombolay. “A person will teach a robot what to look for and how to physically complete that task. The robot will go around and look for all the tasks that it can possibly complete. If it has any doubts, let’s quantify that doubt, and then ask the human for help.”

It’s all about teamwork, from a human’s point of view.

“One of the things that makes HRI-focused machine learning and robotics different is that we really care about the fact that we are getting information from humans,” says Gombolay.

In a previous discussion with Shah in 2014, RIA highlighted her sentiments, “It’s about harnessing the strengths of humans and robots to achieve new levels of efficiency and productivity that neither can achieve alone.”

The crux of human-robot interaction and collaboration, in which humans and robots are independently superior in certain capabilities, but infinitely better together, still holds true. Gombolay’s research builds on that notion.

To Apprenticeship Learning
After defending his PhD dissertations, Gombolay served as a technical staff member at MIT’s Lincoln Laboratory transitioning his research for the U.S. Navy. His work earned him an R&D 100 Award for developing a technique for optimizing human-machine collaboration via apprenticeship scheduling, where robots assist humans in complex scheduling tasks.

“Factories across the country have these unique individuals who are able to run their factories. Without those individuals, the factory would fall apart,” explains Gombolay. “I saw that time and time again, and wanted to understand what they were doing. Research has shown that you can’t just ask somebody what they are doing. We really need to watch them and infer their strategies by watching the actions they take. I used what was kind of new at the time, a counter factual-reasoning approach, to look at what they did relative to what they did not do, in order to predict what actions experts are likely to take. Now I have a decision-making model in a box that can inform observation algorithms.”

Gombolay and fellow researchers at MIT took the apprenticeship scheduling model and applied it to two very different scenarios, a missile-defense simulator for the Navy and the labor and delivery ward of Boston’s Beth Israel hospital. See the full story.

Each year, the U.S. Defense Advanced Research Projects Agency recognizes up-and-coming researchers with breakthrough technologies deemed of high interest for national security. In 2018, Gombolay was named a DARPA Riser for his work in apprenticeship scheduling.  He also received a three-year NASA Early Career Fellowship grant to continue exploring the ideas of apprenticeship learning with robots in foreign environments, namely the moon and Mars.

“A lot of these things that we’re trying to do for manufacturing, and for NASA, work well in healthcare settings,” says Gombolay. “When I was at Beth Israel, the charge nurses I worked with are the ones that decide which nurses and doctors go where and when to take care of which patients. It’s basically air traffic control, but in a hospital. Half of them quit the job because it’s so stressful.”

To Teammates
Now at Georgia Tech, he’s working to build more clinical relationships with nurses and physicians to continue to explore robotic apprenticeship scheduling and understand expert nurses’ thought processes so they can use that to expedite training, and reduce operator burnout and attrition in healthcare environments.

“Every time I showed up at Beth Israel, people teased me, ‘Matthew, where’s my robot? I want my robot.’ They genuinely wanted help!”

The old “robots stealing jobs” argument that often plagues discussions about automation rarely breaches the conversation in these settings. Gombolay says 99.9 percent of the time, that’s not even a concern.

“Their job is insanely difficult. Robots aren’t even close to 5 percent of what a nurse or doctor does in interacting and caring for patients, or facilitating teamwork. With machine learning, we’re getting close to maybe automating 90 percent of 1 percent of some of these decision-making tasks in very narrow settings.”

He draws an analogy to the way a surgeon works with a scrub nurse. The tech knows what instrument the surgeon needs before they even have to ask for it. That’s the kind of vision he has for robotics.

“The robot knows what you want without you having to tell it what you need,” says Gombolay. “That’s what we really need for robots to become a seamless partner or teammate.”

Looking ahead five or more years, he thinks we’ll see more research work in teaming. His lab will continue to work with the Navy and the MIT Lincoln Laboratory on human-machine teaming. He says it’s important to continue to have multiple humans in his research, to understand human behavior.

“It could make it easier if we redesign the world to support robots,” says Gombolay. “But I want robots to live in our world and support us.”

Robots may have a ways to go until they can lend the kind of support that anticipates our needs. But that doesn’t mean collaborative robots are static. Entire fleets of mobile robots come to play.

Mobile Collaborators
Whirlpool deployed three MiR autonomous mobile robots to transport components between the preassembly and assembly lines of the home appliance manufacturer’s facility in Poland. The Shared Services Centre in Lodz produces dryers and freestanding cookers.

“At our factory, a dryer leaves the production line every 15 seconds. This requires transporting a huge number of components,” says Szymon Krupinski, Site Leader at Whirlpool. “Mobile robots provide us with a completely new way of delivering parts without human involvement. This enables employees to focus on higher value-added areas. Collaborative mobile robots also significantly improve safety, allowing us to avoid all potential collisions between people and devices, such as forklifts or tuggers.”

Don't Miss These Industry-Leading Events!

October 1-3, 2024
Cincinnati, OH

October 7, 2024
Memphis, TN

October 8-10, 2024
Memphis, TN

November 12-13, 2024
Atlanta, GA

During plant operation two MiR200 robots, with a load capacity of up to 200 kg, transport components while the third mobile robot serves as a backup, docked in a charging station. The two robots transport dryer doors from the preassembly area to the assembly line. On every run, each robot carts 12 doors at a time and on the way back transports the empty packaging. The full loop is around 130 meters.

The robot drives to the preassembly line, moves under a cart loaded with boxes, and spreads the “wings” of its top module to engage with flanges on the underside of the cart, creating a secure link. The cart is then transported to the assembly line and the boxes are unloaded. At the same time, empty boxes are collected on the cart’s upper flow rack. Loaded and empty boxes flow between the cart and the assembly line via gravity. Afterwards the robot returns to its starting point and repeats the transportation cycle. A full cycle takes just under 4 minutes.

Watch the MiR robots in action at Whirlpool.

Autonomous Adaptability
Along its route, MiR’s sensors and scanners allow the robot to detect and avoid obstacles such as forklifts or tuggers, or operators walking in the manufacturing area. Since the first mobile robot was implemented in December 2018, the plant layout and thus the robots’ route has changed several times. Simplicity in operation, programming, and advanced navigation technology enable the MiR robots to quickly adapt to changes in the production area layout.

“The ease of operation of MiR robots allows them to be used by staff without any engineering or programming background,” says Paolo Aliverti, Logistic Program Manager Industry 4.0 for Whirlpool. “This enables us to effectively utilize the robots without making big investments in training the employees in the context of the new technology.”

Adam Bakowicz, Process Technology Senior Engineer Industry 4.0 says factory workers were at first very interested in the new mobile technology for sheer curiosity. Then they would tentatively challenge the robot to test its safety worthiness. Now he says they have completely accepted the mobile collaborators as just another piece of automation. The user-friendliness of the robots had a positive impact and reduced the need for special workforce training.

The fleet management system, MiRFleet, allows the robots to properly queue the requests from the line and monitor their battery charge levels to ensure continuous work. Poland-based distributor ProCobot supported Whirlpool’s implementation and brought in local integration partner Octant to help with designing the top module for hooking the carts and to integrate an app with MiRFleet. The specialized app allows operators to select manual or automatic queue requests for deliveries.

“By changing the system from human-operated to automated delivery, we can boost productivity and engage employees in final product manufacturing,” says Bakowicz. “The MiR robots provide us with low cost of automation and flexibility in changing the plant layout.”

The Lodz factory is the newest plant in Whirlpool EMEA (Europe, Middle East, and Africa). They expect an ROI of less than 2 years. Similar solutions are undergoing pilot testing at two Whirlpool plants in Italy and other locations.

Collaborative robots don’t have to be autonomously mobile to be adaptable. Sometimes they just need to be portable. Built-in safety and easy setup allow cobots to play nice with others.

Seamless Integration
Facing the challenge of having to staff round-the-clock production for fast-changing processes, custom injection molder EVCO Plastics turned to collaborative robots to ease the load. Four Universal Robots (UR) cobots now handle a wide range of tasks such as dispensing, assembly, harvesting of 3D printers, quality inspection, and packaging. As production needs change, the cobots are placed on wheels and can be moved around the factory floors of the Wisconsin-based facility as required.

Bernie Degenhardt, EVCO Automation Manager, emphasizes that the cobots are not replacing his existing workforce. “Having a human do monotonous, repeatable tasks is kind of a waste of intelligent labor. The operator will go to a higher level, something that requires more dynamic thinking such as quality checks, other things like that,” he explains, adding that operators don’t see the cobots as a threat. “I think they see the UR as a tool. You put it on the floor, and it’s not caged off or isolated. The operator feels they’re working with the robot, not against it.”

EVCO is not new to robotics. They have numerous traditional, Cartesian robots tending injection molding machines.

“The biggest difference between hard automation and collaborative robots is the setup time,” says EVCO’s Automation Engineer Jason Glanzer. “These cobots interface well with many products. UR has really been on top of continuously improving compatibility, which was really important for us,” he says, highlighting the UR+ platform that certifies grippers, vision cameras, software, and other peripherals to work seamlessly with UR cobots.

Watch the UR robots on the job at EVCO Plastics.

Intricate Assembly, Easy Setup
One of the UR cobots is deployed in an intricate assembly task. The cobot picks up a gearbox used in lawnmowers, places it into a grease dispenser and then inserts a cap to seal the grease port. Placing the cap correctly is a tricky task made simpler by the UR+ certified FT300 Force Torque Sensor from Robotiq.

“Adding the sensor to the end of the UR arm allows the cobot to ‘feel’ when the cap is inserted correctly, using a spiral motion,” explains Glanzer.

The UR+ software handshake means that all programming of the sensor happens directly through the cobot’s teach pendant, with the same intuitive interface used to program the cobot itself.

“Without the UR+ integration we would have to create a considerable amount of script code to accomplish a task like this,” says Glanzer.

The cobot teach pendant has two other UR+ interfaces, a Cognex vision camera and an Asycube Flexible Part Feeder from Asyril. The feeder distributes the caps on a surface that vibrates until the caps are spread out facing the right side up, then the vision camera mounted above the surface snaps an image of the caps’ positions, instructing the robot where to pick them up.

“Being able to control every part of the cell through the cobot means that it essentially becomes the PLC,” says Glanzer. “It reduces a lot of system costs and programming time upfront. Traditionally it would take up to several weeks or more to automate a new project, whereas now, you roll the cobot out to the floor, you do some programming, and it can be up and running in two days.”

The cobot’s simple setup and operation means EVCO can run them on the third shift. “If something goes wrong in the middle of the night, one of our setup guys can go over there and get the cell up and running without having to call an automation technician or engineer onto the plant floor,” says Glanzer.

The 3rd Shift
While UR cobots are assembling components, others are harvesting 3D printers and packaging parts. EVCO’s 3D printer farm consists of six polymer printers that run 24/7 to serve EVCO’s own internal tooling needs and produce parts for outside customers.

The printer cell can now work continuously without human oversight for as long as build plates are available. The printers communicate with the cobot via Wi-Fi. If designers want to change the quantity of prints or models that need to be built, they can send an email from their desk to change the value for how many times the robot needs to restart the printer or clear it for the next build.

EVCO can now do even very small runs cost-efficiently. A recent example is a tripod mount for a spotting scope used by hunters and bird watchers. The customer is an online retailer that needed just 50 of these parts. It’s a job that EVCO normally wouldn’t be able to do cost-effectively, given the small batch size.

“We plan to take on more jobs like this,” says Glanzer. “By mirroring the existing array, adding an additional six printers, with the UR10 on a track to travel between them, it’s infinitely flexible. The possibilities are part of the beauty of the solution.”

Located in a region with low unemployment, EVCO had trouble staffing the third shift. Manning cells with repetitive and tedious tasks was especially challenging.

“Getting people to stick around long term is difficult, leading to a lot of turnover,” says Andy Prell, EVCO’s Production Manager. “Universal Robots allows us to do the same amount of work with fewer people, so it helps mitigate that challenge,” he explains, noting how operators have traditionally handled one or maybe two machines at a time. “That really limits what we can produce on any given shift. Now we can use one operator to run several machines concurrently, which allows us a lot of flexibility in our operation.”

Rapid ROI
Automation Manager Degenhardt is quick to summarize what that means for the bottom line. “Eliminating the labor cost allows us to run basically two jobs for the cost of one UR robot, so the payback comes quite quickly that way.”

He estimates ROI on EVCO’s cobots is between six and nine months. Adding to the quick payback is savings on workers’ compensation insurance.

“This is a big deal,” he says. “The cobots help reduce any kind of repetitive strain injuries so we actually get a lower rate, which is a huge cost savings to us.”
 

Built-In Safety Saves Cost, Space
The built-in safety system in the UR cobots makes them automatically stop when they encounter obstacles in their route. Based on a risk assessment of the application, a vast majority of UR cobots operate right alongside workers with no safety guarding, as is the case at EVCO.

“Real estate is important; our production floor gets really tight. We’re constantly changing out molds. We have a lot of forklift and crane traffic,” says Glanzer. “Not having to put up guarding really opens up a lot of space for us, enabling us to keep production running on one cell while we’re doing a mold change on the other.”

On the packaging line, you see direct human-robot collaboration where an operator takes the packaged box directly from the UR cobot and tapes it shut. The benefit is not only space savings, but it’s also a significant cost saver.

“With standard robots, all the guarding, safety relays and light curtains can add thousands of dollars to the cost,” says Glanzer. “With cobots, none of this is necessary.”

With cobots and HRI, more is possible. As we learn more about how and why humans behave the way we do, it opens our eyes to what we can become and what we can create. In our next installment, we’ll cover more ways technology is opening new, exciting opportunities for human-robot collaboration and augmentation.