Engineering research discovers critical vulnerabilities in AI-enabled robots

Within its new Responsible Innovation initiative, researchers at Penn Engineering discovered that certain features of AI-governed robots carry security vulnerabilities and weaknesses that were previously unidentified and unknown. The research aims to address the emerging vulnerability for ensuring the safe deployment of large language models (LLMs) in robotics.

“Our work shows that, at this moment, large language models are just not safe enough when integrated with the physical world,” says George Pappas, UPS Foundation Professor of Transportation in Electrical and Systems Engineering (ESE), in Computer and Information Science (CIS), and in Mechanical Engineering and Applied Mechanics (MEAM).

In the new paper, Pappas, who also serves as the Associate Dean for Research at Penn Engineering, and his co-authors caution that a wide variety of AI-controlled robots can be manipulated or hacked.

RoboPAIR, the algorithm the researchers developed, needed just days to achieve a 100% “jailbreak” rate, bypassing safety guardrails in three different robotic systems: the Unitree Go2, a quadruped robot used in a variety of applications; the Clearpath Robotics Jackal, a wheeled vehicle often used for academic research; and the Dolphin LLM, a self-driving simulator designed by NVIDIA. For example, by bypassing safety guardrails, the self-driving system could be manipulated to speed through crosswalks.

Prior to publicly releasing the study, Penn Engineering informed the companies about their system vulnerabilities and is working with them to use the research as a framework to advance the testing and validation of these manufacturers’ AI safety protocols.

“What is important to underscore here is that systems become safer when you find their weaknesses. This is true for cybersecurity. This is also true for AI safety,” says Alexander Robey, a recent Penn Engineering Ph.D. graduate in ESE, current postdoctoral scholar at Carnegie Mellon University and the paper’s first author.

“In fact, AI red teaming, a safety practice that entails testing AI systems for potential threats and vulnerabilities, is essential for safeguarding generative AI systems—because once you identify the weaknesses, then you can test and even train these systems to avoid them.”

What is required to address the problem, the researchers argue, is less a software patch than a wholesale reevaluation of how the integration of AI into physical systems is regulated.

“The findings of this paper make abundantly clear that having a safety-first approach is critical to unlocking responsible innovation,” says Vijay Kumar, Nemirovsky Family Dean of Penn Engineering and another co-author.

“We must address intrinsic vulnerabilities before deploying AI-enabled robots in the real world. Indeed, our research is developing a framework for verification and validation that ensures only actions that conform to social norms can—and should—be taken by robotic systems.”