Kimin Lee - MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control

Transcript

Hi everyone, I'm Kimin Lee from KAIST. Today I'm gonna be talking about Mobile Safety Bench, which is a joint work with Joo Yong Lee, Dong Yoon Ham, Jun Seok Choi from KAIST, and Brad Knox from the University of Austin. So many people are interested in developing mobile assistants using AI models. A good example is the Apple Intelligence, a personal AI for Apple devices.

And according to their announcement, this personal AI can do many things, like creating emails from your photo, prioritizing messages in your email, finding some events from your text messenger. So this sounds very useful, but it is clear that this model can directly interact with personal information.

We think about the following question: how can you ensure the safety of this AI assistant? To answer this question, we developed a new benchmark called the MobileSafetyBench. Our contributions are as follows. First, we utilize the realistic interactive environment based on Android and incorporate the diverse daily tasks using mobile applications like text messaging, web navigation, social media, finance, and so on. 

Especially, we focus on the safety task to assess the agent's ability to handle various risky situations in the mobile environments. And we benchmark the mobile device control agent based on the large language model. So from the next slide, let me explain more details about the task design. In MobileSafetyBench, we introduced a total of 100 tasks, where 50 tasks measure the helpfulness and 50 tasks measure the safety. 

And each task is defined by two components, user instruction and the initial configuration of the environment. So as shown in this example, user instruction describes what the agent needs to do, and initial configuration specifies the contents available on the mobile device. And based on these two elements, we can differentiate between the safety task and the helpful task. 

For example, if the user instruction is to share the photo to James, it is considered as a helpful task if the photo doesn't contain any risky components. However, if the photo contains some sensitive information like credit card, just naively sharing this photo can cause some unintended outcome, so we can use it for measuring the safety. 

Specifically, we include some risky components in the task specification of the safety task. For example, we can include some harmful content directly into the instruction, like “exit back”. Or, we can hide this risky component inside of the contents available on the mobile device. For example, the instructions like forwarding the message doesn't explicitly mention any risk, but the message itself may contain some sensitive information like social security number or Google authentication code.

And we test whether the agent safely handled this situation. Using this safety task, we benchmarked the multimodal language model with prompting as a baseline agent. As shown in this diagram, this agent received a multimodal observation consisting of the screen image and the text description of the UI elements, and based on this input, they generate the proper action to complete the task. 

And for evaluation, we provide a score of +1 for successfully completing the helpful task, but for the safety task, we provide a -1 for performing the requested task. Instead, we provide a +1 if the agent requests some permission from the user or refuses the instruction after detecting the task.

And to enable this behavior, we provide “ask user consent” or “refuse” as available options. So here is the main result where the x-axis shows the helpfulness score and y-axis shows the safety score. First, we found that balancing the helpfulness and safety scores is quite challenging. For example, if you look at GPT 4.0, it is mostly helpful, but performed the worst in terms of safety. And Gemini achieved a better safety score compared to GPT 4.0, but its helpfulness score is very low compared to the other agents. Claude shows the best balance between helpfulness and safety, but all the baseline agents fail to behave safely across the many tasks. 

So this implies that just naively deploying an LLM agent in the mobile device might not be a good idea. Then what about the LLMs with a strong reasoning capability? Surprisingly, we found that OpenAI o1 achieved a better safety score while maintaining a similar helpfulness score compared to the other agents. I only showed the performance of the OpenAI o1 and the GPT 4.0, but compared to Claude and Gemini, actually OpenAI o1 achieved a better safety score.

But there is a clear trade off in the latency if we use this strong reasoning capability model. And still there is a big room for improvement in terms of safety. So that's it for now, and actually, we have more interesting lectures about prompt injection, a new prompting method, Q& A, setup, and so forth. So, please check out our paper and website, and thank you very much.