LLMs Help Robot Guide Dogs Give Spoken Directions

Guide dogs help visually impaired people navigate life's obstacles and avoid potentially dangerous situations. But they're not the right fit for every case. Training takes years, and even then, only half attain official guide dog status.

Caring for dogs can be difficult, and some people also have allergies. In the US, only around 2% of visually impaired people use a guide dog. In China, there are only about 400 guide dogs for more than 10 million visually impaired people.

While guide dogs can be a powerful resource, another option is emerging. Researchers from New York State's Binghamton University have developed robotic guide dogs that can speak to their humans, vastly surpassing biological dogs' maximum 20 cues. 

Shiqi Zhang, a leading researcher on the study told A3, "only a small portion of blind or visually impaired individuals use a guide dog. Some applicants are too old or too young to be eligible, and others have dog allergies or cannot walk a dog for miles each day. On the supply end, breeding and training a guide dog is costly, and the graduation rate from guide dog schools is low. We have been working on mobile robotics research for more than 15 years and thought robot dogs could provide an alternative”.

Yohei Hayamizu, another leading researcher on the study told A3 the teams, "research focus is on human-robot interaction and developing robots that are socially and spatially aware. For robots to contribute to a human-inhabited environment, they need to recognize their surroundings, make their own decisions, and interact with people. A robotic guide dog is one of the platforms that advances robotics one step forward.”

In this work, large language models (LLMs) were used to create a talking robot guide dog, using Chat GPT-4 to develop strong language capabilities. Using the in-built AI, the robotic dog can determine the ideal route for its handler and safely guide them to their destination with the bonus of providing real-time feedback and dialogue along the way.

“Current research on robot guide dogs mostly focuses on control methods and hardware design. There was no research on the language capabilities of robotic guide dogs. We believe it's important because this is one of the aspects where robots can do better than dogs,” says Zhang “It's important to note that people generally like dogs because of the strong emotional support and companionship. Robot dogs are flexible and versatile, capable of navigating stairs and uneven surfaces. With new natural language capabilities, robot guide dogs are one step closer to becoming a component of our everyday lives, whether sighted or not."

This research is not the first foray into robotic guide dogs for the team, which previously taught systems to respond to a leash tug. But integrating LLMs allows the handler and the guide dog to have a back-and-forth dialogue —something that’s not possible with biological dogs — which provides visually impaired people with more control over their journey and a better understand of the different situations they face along the way.

The AI-enhanced robot guide dog was tested using seven legally blind participants in a larger multi-room office environment. The guide dog asked the handlers where they wanted to go and offered different routes to take. Once the handler selected the desired route, the robot guide dog would navigate the person to their destination, continually communicating the surroundings to the handler and avoiding obstacles along the way. After the tests, the participants filled in questionnaires about their experience and the consensus was that the combination of pre-planning and real-time narration from the guide dog was preferred by the participants.

“While we have respect for the bond between humans and living guide dogs, robotic versions offer several practical advantages in scalability, accessibility, and information density. They can be manufactured at scale, unlike the years-long biological training process,” says Hayamizu. “They can also interact with people with rich context, which a living dog simply cannot, and they can expand their capability by integrating with GPS and digital maps."

However, the researchers say robotic guide dogs are not intended to be a direct replacement for biological guide dogs. “Our goal has never been to replace living service dogs," says Zhang. "People love dogs, and many consider them family members. Robot dogs, at least for now, are still rigid and lack the warmth of a real animal."

Zhang also confirmed to A3 that the team's "goal is to develop a robotic alternative for individuals who would benefit from a guide animal but cannot use a biological guide dog. There are already other assistive navigation tools, such as smart white canes and smartphone apps. Robots, as embodied AI systems, offer unique advantages, especially their ability to actively perceive and interact with the physical world. Still, the intention is not to replace existing assistive technologies, but to provide an additional option for those who need it”.

Going forward, the researchers plan to conduct more user studies, improve the autonomy of the dogs, and test the navigation capabilities of the guide dogs over long distances in both indoor and outdoor environments. Zhang concluded to A3 that, “the technology is still in its early stages. We are looking into directions including intelligent power management to improve travel distance and mapless navigation in outdoor environments."