U.S. Army Research Transforms Soldier-Robot Communications

U.S. Army researchers have developed ground-breaking technology that will enhance how soldiers and robots communicate and perform tasks in tactical environments.

According to Army researcher Dr. Claire Bonial from the U.S. Army DEVCOM, Army Research Laboratory (ARL), “the research sets out to develop a Natural Language Understanding (NLU) pipeline for robots that can be easily ported over to any computational system or agent and incrementally tames the variation that we see in natural language.”

This means that regardless of how a soldier chooses to express themselves to the robot, the underlying intent of that language is understood and can be acted on, given both the current conversational and environmental or situational context.

To do this, the NLU pipeline first automatically parses the input language into Abstract Meaning Representation (AMR) which captures the basic meaning of the content of the language and then converts and augments it into Dialogue-AMR, which captures additional elements of meaning needed for two-way human robot dialogue, such as what the person is trying to do with the utterance in the conversational context – for example give a command, ask a question, state a fact about the environment, etc.

“This is unique in comparison to other NLU components within dialogue systems, many of which forego any kind of semantic representation in favor of a deep-learning approach,” Bonial said.

This research was presented at the 14th International Conference on Computational Semantics (IWCS 2021) where it received the Outstanding Paper Award. The paper represents a body of research spanning over a decade for researchers like Bonial.

The research includes work to represent language expressed through semi-idiomatic constructions, most recently extending AMR so that it can better capture two-way dialogue and specifically task-oriented, situated dialogue between people and robots, in an augmented version of the representation called Dialogue-AMR.

Efforts to develop Dialogue-AMR started in 2018 with dialogue expert Dr. David Traum as part of the University Affiliated Research Center previously established between DEVCOM ARL and the Institute for Creative Technologies (ICT) at the University of Southern California.

Dialogue-AMR draws upon ARL and ICT’s Bot Language research collaboration focused on robot dialogue systems, which was initiated by one of the IWCS paper’s co-authors, Dr. Clare Voss, in 2012.

The paper also reports on a critical step in the research trajectory: the researchers’ experiments to robustly and critically evaluate Dialogue-AMR as a computational semantic representation, as well as the NLU pipeline used to automatically obtain the correct Dialogue-AMR representation when given unconstrained natural language input.

The researchers evaluate in two problem domains: first, the domain of human-robot dialogue for collaborative search and navigation tasks, and second the domain of human-human dialogue in the virtual Minecraft gaming environment, where participants are collaboratively building block structures.

The Minecraft dialogue data that made this comparative evaluation possible was obtained from Dr. Martha Palmer of the University of Colorado, Boulder, and Dr. Julia Hockenmaier of the University of Illinois at Urbana-Champaign.

Evaluating in these two domains gets at a key question regarding the utility of such a representation and the NLU pipeline in general.

Bonial questioned how efficiently and with what accuracy the pipeline and representation can be applied in one problem domain, and then transferred and refined to a new problem domain. 

“Any Army application requires a system that can efficiently and flexibly act in dynamic circumstances,” Bonial said. “Our evaluation demonstrates that our NLU pipeline can obtain human-level performance in the first, human-robot dialogue domain, such that the output Dialogue-AMR representations agree with, or are similar to, human gold-standard representations at a rate similar to representations manually created by different humans.”

Their evaluation demonstrates that with additional training data (several hours’ worth of annotation work amounting to about 200 additional training instances for the machine learning elements of the pipeline), the NLU pipeline obtains promising performance in the second domain of human-human dialogue in the Minecraft domain.

The researchers noted that while the performance is somewhat lower than performance on the original human-robot dialogue domain, it is comparable or better than the performance of other automatic semantic parsers for other semantic representations and language domains.

“These promising results demonstrate that this NLU pipeline leverages a valid approach for communicating with robots in collaborative tasks across multiple domains,” Bonial said. “I am very excited for the promise of this research to provide transformational overmatch in our computational systems’ ability to understand the underlying meaning of Soldiers’ instructions, despite difference in their dialects, accents and other types of noise sure to arise in Army-relevant applications.”

The team’s next steps involve connecting the output semantic representation with a system that grounds the pieces of the representation to both entities in the environments and the executable behaviors of the robot in joint work with Dr. Thomas Howard of the University of Rochester.

“We are optimistic that the deeper semantic representation will provide the structure needed for superior grounding of the language in both the conversational and physical environment such that robots can communicate and act more as teammates to soldiers, as opposed to tools,” Bonial said.