夜雨聆风
第四期·完结篇
威胁四：接口与环境风险
在AI智能体的背景下，特别是那些在网络或具身环境中运行的智能体，接口与环境风险指的是因智能体与其外部运行环境上下文之间交互而产生的漏洞和局限性[131]。这些风险并非源于智能体的内部推理或学习能力，而是源于智能体感知和行动所依赖的接口与环境的错配、脆弱性或可变性[132][133][134]。
这一风险类别包含三个关键维度：
4.1观测与动作空间的错配
尽管部署的智能体经常需要诸如滚动、悬停或标签页操作等基础性操作，但LLM是在静态文本语料库上进行预训练的。这种训练数据的错配导致了感知和执行两方面的问题。攻击者也可以利用这些现存的观测-动作空间错配和鲁棒性问题来攻击智能体系统。例如，在评估通用网络交互任务的WebArena基准测试中，GPT-4的性能暴露了其中一些挑战[132]：滚动、悬停和标签页切换等细粒度动作给模型增加了不必要的复杂性，并且经常被误用。AgentOccam也表明，通过精简动作空间（例如，移除效用低的命令、抽象化复合动作）和调整观测空间（例如，修剪重复的历史记录、提供完整的页面状态），可以显著提升轨迹稳定性和任务成功率[134]。所有这些结果都表明，需要使LLM的推理偏差与观测空间和动作空间相匹配，以实现强大的智能体性能并确保更高程度的鲁棒性/可信性。
4.2现实环境中的感知–行动脆弱性
WebArena的错误分析揭示了基于LLM的智能体在现实环境中存在三个紧密关联的脆弱性[132]。
对先前输入的误解读：现实环境中的智能体经常误读输入，表现出鲁棒性不足。例如，GPT-4经常重复输入已经输入过的搜索短语（如“DMV area”），直到达到步骤限制，这表明它未能将短期状态和过往行动纳入决策。智能体也通常会忽略先前输入的内容或行动历史[132]。现代的预训练和监督微调范式基于对话风格的数据，这训练模型学习短期的指令-响应行为（同时降低了长期、具身化、序列化状态跟踪的优先级），很可能是导致这些缺陷的原因[132][135]。
过早终止与可行性误判：现实环境中另一个显著的鲁棒性问题是不安全的提前停止，这通常由智能体的感知偏差引起。例如，在WebArena基准测试中，作者在智能体提示中为那些因缺乏证据而无法完成的任务提供了“不可行”（Unachievable, UA）提示。然而，移除显式的UA提示使GPT-4的整体任务成功率提高了14.41%，同时将模型对不可能任务的正确检测率降低至44.44%。此外，在此指令设置下，GPT-4将54.9%实际可行的任务错误地标记为不可行。这表明即使微小的指令变化也会对停止/继续行为产生显著影响[132]。相比之下，较小规模的GPT-3.5则倾向于耗尽步骤限制、重复错误动作或产生幻觉式响应，而不是生成结构化的不可行性推理[132][136]。
模板、反馈与记忆的脆弱性：基于LLM的智能体在面对重复、长周期或轻微变化的任务时，表现出脆弱的泛化能力。即使任务源于相同的基础模板，性能差异也很显著：在WebArena的61个模板中，GPT-4仅在4个模板上能够持续成功[132]。类似的脆弱性也在其他基准测试中被观察到：Mind2Web报告称，源自模板的网络任务变体常常导致成功率急剧下降[136]，而BrowserGym则发现，在不同环境和界面状态下复现结果存在不稳定性[133]。这些发现凸显了在缺乏稳健记忆或自适应反馈机制的情况下，依赖表面模式进行泛化的局限性。为了弥合这一差距，WebArena基准测试被提出作为一个测试平台，用于探索显式整合记忆和反馈以提高可靠性的方法[132]。诸如AgentOccam等补充性工作进一步表明，通过更好的规划原语可以增强长周期协调和轨迹稳定性[134]。
4.3动态内容、本地化与机器人检测
对于自主智能体而言，网络环境带来了重大的可访问性和可复现性挑战。时区、默认语言和地理设置等本地化因素会改变网站的渲染方式，导致智能体行为各异，从而损害跨试验的一致性[136]。动态界面元素，包括广告、弹窗和非确定性更新，进一步增加了随机性，导致即使在基本相同的任务上性能也不稳定[132]。验证码（CAPTCHA）和其他机器人检测机制则造成了额外的障碍，常常给智能体系统带来重大问题。诸如Open CaptchaWorld等研究[137]表明，即使是先进的多模态智能体也难以应对验证码，其最高成功率仅为40%，而人类则接近100%。这些局限性使得可复现性、可靠性和可扩展性成为现实环境（尤其是基于网络的系统）中AI智能体安全研究的持续性挑战[133]。
总结
AI智能体作为人工智能发展的下一范式，凭借其自主性、目标导向推理、工具调用与环境交互能力，正在深刻变革复杂工作流自动化、软件工程、个性化服务、科学研究及医疗健康等诸多领域。然而，这些能力的提升也伴随着攻击面的显著扩大与安全风险的质变。本文系统梳理了当前AI智能体系统面临的四大类安全威胁，揭示了其从传统AI安全边界向新型、自主、多模态、多智能体协同方向演化的趋势。
首先，提示注入与越狱仍是智能体系统最基础且最广泛的攻击入口。直接注入与间接注入、有意与无意、单模态与多模态、传播性与非传播性、混淆与载荷拆分等多样化变体，使得攻击者能够操纵智能体偏离预期行为，甚至实现跨智能体的恶意载荷扩散。多模态注入（图像、音频、视频）及混合攻击的出现，进一步突破了传统文本过滤的防御边界。
其次，自主网络利用与工具滥用展现了智能体作为攻击执行者的潜力。研究表明，基于GPT-4等大语言模型的智能体能够以极低成本自主利用一日漏洞、入侵沙盒网站，并通过协同规划与工具调用链实施复杂多步攻击。这种攻击的经济性与可并行化特性，使得大规模自动化网络攻击成为现实威胁。
第三，多智能体与协议层威胁引入了全新的攻击维度。模型上下文协议（MCP）与智能体间协议（A2A）等标准化通信机制，在提升互操作性的同时，也暴露了泛洪、重放、凭据泄露、身份伪造、传递性提示注入等协议特有漏洞。多智能体系统中的协调操纵、知识与学习操纵、跨智能体推理策略规避、责任追溯混淆及机密数据篡改等威胁，使得攻击能够跨越组织边界、利用分布式工作流中的碎片化上下文，传统安全机制在此类场景下面临严峻挑战。
最后，接口与环境风险源于智能体感知与行动空间的固有错配。现实环境中的观测—动作空间差异、输入误解读、过早终止、泛化脆弱性，以及动态内容、本地化差异和机器人检测机制，共同构成了智能体在真实部署中的可靠性障碍。这些风险不仅影响任务成功率，更可能被攻击者利用以引发不可预期的行为。
综上所述，AI智能体的安全威胁已从单一模型的对抗性攻击，演变为覆盖输入、工具、协议、跨系统风险体系。当前防御研究仍处于起步阶段，亟需发展面向智能体特性的动态监控、输入验证、协议加固、可追溯审计及分级自主控制等综合安全机制。未来工作应进一步聚焦于真实世界部署中的鲁棒性评估、多智能体协同下的信任管理，以及人机协同治理框架的构建，以确保AI智能体在释放巨大潜力的同时，能够在安全可信的边界内服务于社会。
向上滑动，可查看所有参考文献
[1]. How Llms Work. Ai What Do Large Language Models "Understand"? Image, 21:1, 2024.
[2]. Andrei Kucharavy. Fundamental Limitations Of Generative Llms. In Large Language Models In Cybersecurity: Threats, Exposure And Mitigation, Pages 55-64. Springer Nature Switzerland Cham, 2024.
[3]. Thomas Kwa, Ben West, Joel Becker, Amy Deng, Kathryn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Et Al. Measuring Ai Ability To Complete Long Tasks. Arxiv Preprint Arxiv:2503.14499, 2025.
[4]. Palo Alto Networks (Unit 42). Ai Agents Are Here. So Are The Threats., 2025.
[5]. Langchain. Langchain Documentation. Https://Python.Langchain.Com/, 2024.
[6]. Toran Bruce Richards. Autogpt: An Autonomous Gpt Experiment. Https://Github.Com/Torantulino/Auto-Gpt, 2024.
[7]. Openclaw. Https://Openclaw.Im
[8]. Guanzhi Wang et al. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.
[9]. Paolo Dal Cin, Daniel Kendzior, Yusof Seedat, and Renato Marinho. Three essentials for agentic ai security. MIT Sloan Management Review (Online), pages 1-4, 2025.
[10]. Reuters. Just in time? manufacturers turn to ai to weather tariff storm, 2025. URL https://www.reuters.com/business/just-time-manufacturers-turn-ai-weather-tariff-storm-2025-08-13/.
[11]. Wired. Forget chatbots. ai agents are the future, 2025. URL https://www.wired.com/story/fast-forward-forget-chatbots-ai-agents-are-the-future/. Accessed: 2025-08-16.
[12]. Joon Sung Park et al. Generative agents: Interactive simulacra of human behavior. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), 2023.
[13]. Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik. Empowering biomedical discovery with ai agents. Cell, 187 (22):6125-6151, 2024.
[14]. Mourad Gridach, Jay Nanavati, Khaldoun Zine El Abidine, Lenon Mendes, and Christina Mack. Agentic ai for scientific discovery: A survey of progress, challenges, and future directions. arXiv preprint arXiv:2503.08979, 2025.
[15]. Verge. Inside the automated warehouse where robots are packing your groceries, 2025. URL https://www.theverge.com/robot/719880/ocado-online-grocery-automation-krogers-luton-ogrp-robot-grid. Accessed: 2025-08-16.
[16]. Zihan Chen, Yixin Wu, et al. Autoagents: A framework for automatic agent generation. arXiv preprint arXiv:2309.17288, 2023. URL https://arxiv.org/abs/2309.17288.
[17]. Reuters. Amazon's delivery, logistics get ai boost, 2025. URL https://www.reuters.com/business/retail-consumer/amazons-delivery-logistics-will-get-an-ai-boost-2025-06-04/. Accessed: 2025-08-16.
[18]. Subash Neupane, Sudip Mittal, and Shahram Rahimi. Towards a hipaa compliant agentic ai system in healthcare. arXiv preprint arXiv:2504.17669, 2025.
[19]. Ken Huang. Ai agents in healthcare. In Agentic AI: Theories and Practices, pages 303-321. Springer, 2025.
[20]. Nalan Karunanayake. Next-generation agentic ai for transforming healthcare. Informatics and Health, 2(2): 73-83, 2025.
[21]. Michael Moritz, Eric Topol, and Pranav Rajpurkar. Coordinated ai agents for advancing healthcare. Nature Biomedical Engineering, pages 1-7, 2025.
[22]. James Zou and Eric J Topol. The rise of agentic ai teammates in medicine. The Lancet, 405(10477):457, 2025.
[23]. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you've signed up for: Compromising real-world llm-integrated Applications with indirect prompt injection. Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023. URL https://api.semanticscholar.org/CorpusID:258546941.
[24]. Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. ArXiv, abs/2211.09527, 2022. URL https://api.semanticscholar.org/CorpusID:253581710.
[25]. Luca Beurer-Kellner, Beat Buesser, Ana-Maria Cre¸tu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, et al. Design Patterns for Securing LLM Agents against Prompt Injections. arXiv preprint arXiv:2506.08837, 2025.
[26]. Donghyun Lee and Mo Tiwari. Prompt infection: Llm-to-llm prompt injection within multi-agent systems. arXiv preprint arXiv:2410.07283, 2024.
[27]. OWASP GenAI Project. Owasp genai llm01: Prompt injection, 2025.
[28]. Jeremy McHugh, Kristina Sekrst, and Jonathan Rodriguez Cefalu. Prompt injection 2.0: Hybrid ai threats. ArXiv, abs/2507.13169, 2025. URL https://api.semanticscholar.org/CorpusID:280296803.
[29]. Yulin Chen, Haoran Li, Yuan Sui, Yufei He, Yue Liu, Yangqiu Song, and Bryan Hooi. Can indirect prompt injection attacks be detected and removed? arXiv preprint arXiv:2502.16580, 2025.
[30]. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjeAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 10471-10506, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi:10.18653/v1/2024. findings-acl.624. URL https://aclanthology.org/2024. findings-acl.624/.
[31]. Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7101–7117, 2025.
[32]. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023. URL https://arxiv.org/abs/2307.15043.
[33]. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957, 2024.
[34]. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models. arXiv preprint arXiv:2309.00614, 2023.
[35]. Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: interpretable gradient-based adversarial attacks on large language models. arXiv preprint arXiv:2310.15140, 2023.
[36]. Zexuan Zhong, Ziqing Huang, Alexander Wettig, and Danqi Chen. Poisoning retrieval corpora by injecting adversarial passages. arXiv preprint arXiv:2310.19156, 2023.
[37]. Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. Eia: Environmental injection attack on generalist web agents for privacy leakage. arXiv preprint arXiv:2409.11295, 2024.
[38]. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advagent: Controllable blackbox red-teaming on web agents. arXiv preprint arXiv:2410.17401, 2024.
[39]. Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. Dissecting adversarial robustness of multimodal lm agents. arXiv preprint arXiv:2406.12814, 2024.
[40]. Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, and William Yang Wang. Melon: Provable defense against indirect prompt injection attacks in ai agents. arXiv preprint arXiv:2502.05174, 2025.
[41]. Yanzhe Zhang, Tao Yu, and Diyi Yang. Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391, 2024.
[42]. Sam Johnson, Viet Pham, and Thai Le. Manipulating llm web agents with indirect prompt injection attack via html accessibility tree. arXiv preprint arXiv:2507.14799, 2025.
[43]. Junhyuk Choi, Yeseon Hong, Minju Kim, and Bugeun Kim. Examining identity drift in conversations of llm agents, 2025. URL https://arxiv.org/abs/2412.00804.
[44]. Jiawei Guo and Haipeng Cai. System prompt poisoning: Persistent attacks on large language models beyond user injection. arXiv preprint, 2025. URL https://arxiv.org/abs/2505.06493.
[45]. Quan Zhang, Binqi Zeng, Chijin Zhou, Gwhwan Go, Heyuan Shi, and Yu Jiang. Human-imperceptible retrieval poisoning attacks in llm-powered applications. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 502-506, 2024.
[46]. Cody Clop and Yannick Teglia. Backdoored retrievers for prompt injection attacks on retrieval augmented generation of large language models. arXiv preprint arXiv:2410.14479, 2024.
[47]. Le Wang, Zonghao Ying, Tianyuan Zhang, Siyuan Liang, Shengshan Hu, Mingchuan Zhang, Aishan Liu, and Xianglong Liu. Manipulating multimodal agents via cross-modal prompt injection. arXiv preprint arXiv:2504.14348, 2025.
[48]. Sean Park. Unveiling ai agent vulnerabilities part ii: Code execution. Trend Micro Research Report, 2025. URL https://www.trendmicro.com/vinfo/br/security/news/cybercrime-and-digital-threats/unveiling-ai-agent-vulnerabilities-code-execution.
[49]. Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, and Vitaly Shmatikov. Abusing images and sounds for indirect instruction injection in multi-modal llms. arXiv preprint arXiv:2307.10490, 2023.
[50]. Rodrigo Pedro, Daniel Castro, Paulo Carreira, and Nuno Santos. From prompt injections to sql injection attacks: How protected is your llm-integrated web application? arXiv preprint arXiv:2308.01990, 2023.
[51]. Richard Fang, Rohan Bindu, Akul Gupta, Qiushi Zhan, and Daniel Kang. LLM agents can autonomously hack websites. arXiv preprint arXiv:2402.06664, 2024.
[52]. Rodrigo Pedro, Miguel E. Coimbra, Daniel Castro, Paulo Carreira, and Nuno Santos. Prompt-to-sql injections in llm-integrated web applications: Risks and defenses. 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1768-1780, 2025. URL https://api.semanticscholar.org/CorpusID:272856332.
[53]. MITRE Corporation. CVE-2024-5565: Vanna.AI Remote Code Execution Vulnerability, 2024. URL https://cve.org/CVErecord?id=CVe-2024-5565.
[54]. Anshuman Chhabra, Kartik Patwari, Chandana Kuntala, Deepak Kumar Sharma, Prasant Mohapatra, et al. Towards fair video summarization. Transactions on Machine Learning Research, 2023.
[55]. Xingxing Wei, Siyuan Liang, Ning Chen, and Xiaochun Cao. Transferable adversarial attacks for image and video object detection. arXiv preprint arXiv:1811.12641, 2018.
[56]. Zhipeng Wei, Jingjing Chen, Xingxing Wei, Linxi Jiang, Tat-Seng Chua, Fengfeng Zhou, and Yu-Gang Jiang. Heuristic black-box adversarial attacks on video recognition models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12338-12345, 2020.
[57]. Linxi Jiang, Xingjun Ma, Shaoxiang Chen, James Bailey, and Yu-Gang Jiang. Black-box adversarial attacks on video recognition models. In Proceedings of the 27th ACM International Conference on Multimedia, pages 864-872, 2019.
[58]. Guangke Chen, Fu Song, Zhe Zhao, Xiaojun Jia, Yang Liu, Yanchen Qiao, and Weizhe Zhang. Audiojailbreak: Jailbreak attacks against end-to-end large audio-language models. arXiv preprint arXiv:2505.14103, 2025.
[59]. Eugene Bagdasaryan, Rishi Jha, Vitaly Shmatikov, and Tingwei Zhang. Adversarial illusions in multi-modal embeddings. In 33rd USENIX Security Symposium (USENIX Security 24), pages 3009-3025, 2024.
[60]. Lukas Aichberger, Alasdair Paren, Yarin Gal, Philip Torr, and Adel Bibi. Attacking multimodal os agents with malicious image patches, 2025. URL https://arxiv.org/abs/2503.10809.
[61]. Cristian Pinzon, Juan F. De Paz, Javier Bajo, Alvaro Herrero, and Emilio Corchado. Aida-sql: An adaptive intelligent intrusion detector agent for detecting sql injection attacks. In 2010 10th International Conference on Hybrid Intelligent Systems, pages 73-78, 2010. doi:10.1109/HIS.2010.5600026.
[62]. Johann Rehberger. Deepseek ai: From prompt injection to account takeover, 2024. URL https://embracehered.com/blog/posts/2024/deepseek-ai-prompt-injection-to-xss-and-account-takeover/.
[63]. Sander Schulhoff, Jeremy Pinto, Anam Khan, L-F Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan Boyd-Graber. Ignore this title and hackaprompt: Exposing systemic vulnerabilities of llms through a global scale prompt hacking competition. In The 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (ACL), 2023.
[64]. Stav Cohen, Ron Bitton, and Ben Nassi. Here comes the ai worm: Unleashing zero-click worms that target genai-powered applications. arXiv preprint arXiv:2403.02817, 2024.
[65]. Diego Gosmar, Deborah A Dahl, and Dario Gosmar. Prompt injection detection and mitigation via ai multi-agent nlp frameworks. arXiv preprint arXiv:2503.11517, 2025.
[66]. Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, and Jason Bennett Thatcher. An early categorization of prompt injection attacks on large language models, 2024. URL https://arxiv.org/abs/2402.00898.
[67]. U.S. AI Safety Institute. Technical blog: Strengthening ai agent hijacking evaluations. https://www.nist.gov/news-events/news/2025/01/technical-blog-strengthening-ai-agent-hijacking-evaluations, January 2025. Accessed: January 29, 2025.
[68]. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144, 2024.
[69]. Simon Bennetts. Owasp zed attack proxy. AppSec USA, 2013.
[70]. David Kennedy, Jim O'gorman, Devon Kearns, and Mati Aharoni. Metasploit: the penetration tester's guide. No Starch Press, 2011.
[71]. Nathalie Muehlberger. Csrf and xss: Practical examples using burp suite. Seminararbeit, Ausgewählte Kapitel der IT-Security, 2020. URL https://wiki.elvis.science/images/b/b3/Thesis.pdf. Accessed: 2026-02-05.
[72]. Rushi Mamtora, DP Sharma, and Jatin Patel. Server-side template injection with custom exploit. International Journal of Scientific Research in Science, Engineering and Technology, 2021.
[73]. Jean Rosemond Dora, Ladislav Hluchý, and Karol Nemoga. Ontology for blind sql injection. Computing and Informatics, 42(2):480-500, 2023.
[74]. Zhengliang Shi, Shen Gao, Xiuyi Chen, Yue Feng, Lingyong Yan, Haibo Shi, Dawei Yin, Pengjie Ren, Suzan Verberne, and Zhaochun Ren. Learning to use tools via cooperative and interactive agents. arXiv preprint arXiv:2403.03031, 2024.
[75]. Renxi Wang, Xudong Han, Lei Ji, Shu Wang, Timothy Baldwin, and Haonan Li. Toolgen: Unified tool retrieval and calling via generation. arXiv preprint arXiv:2410.03439, 2024.
[76]. Uncovering Security Threats and Architecting Defenses in Autonomous Agents: A Case Study of OpenClaw,https://arxiv.org/html/2603.12644
[77]. Trojan's Whisper: Stealthy Manipulation of OpenClaw through Injected Bootstrapped Guidance,https://browse-export.arxiv.org/abs/2603.19974
[78]. OpenClaw Vulnerability: Website-to-Local Agent Takeover,https://www.oasis.security/blog/openclaw-vulnerability
[79]. Ronny Ko, Jiseong Jeong, Shuyuan Zheng, Chuan Xiao, Tae-Wan Kim, Makoto Onizuka, and Won-Yong Shin. Seven security challenges that must be solved in cross-domain multi-agent llm systems. arXiv preprint arXiv:2505.23847, 2025.
[80]. Mohamed Amine Ferrag, Norbert Tihanyi, Djallel Hamouda, Leandros Maglaras, and Merouane Debbah. From prompt injections to protocol exploits: Threats in llm-powered ai agents workflows. arXiv preprint arXiv:2506.23260, 2025.
[81]. Introduction to mcp, 2025. URL https://modelcontextprotocol.io/introduction. Accessed: 2025-06-04.
[82]. Rao Surapaneni, Miku Jha, Michael Vakoc, and Todd Segal. A2a: A new era of agent interoperability, April 2025. URL https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/. Accessed: 2025-06-04.
[83]. Gaowei Chang. Agentnetworkprotocol (anp) github repository. https://github.com/agent-network-protocol/AgentNetworkProtocol, 2024. Accessed: 2025-06-04.
[84]. Agent communication protocol: Welcome, 2024. URL https://agentcommunicationprotocol.dev/introduction/welcome. Accessed: 2025-06-04.
[85]. Saman Taghavi Zargar, James Joshi, and David Tipper. A survey of defense mechanisms against distributed denial of service (ddos) flooding attacks. IEEE communications surveys & tutorials, 15(4):2046-2069, 2013.
[86]. Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu. Hidden backdoors in human-centric language models. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 3123-3140, 2021.
[87]. Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. Poisonedrag: Knowledge poisoning attacks to retrieval-augmented generation of large language models. ArXiv, abs/2402.07867, 2024. URL https://api.semanticscholar.org/CorpusID:267626957.
[88]. Vale Tolpegin, Stacey Truex, Mehmet Emre Gursoy, and Ling Liu. Data poisoning attacks against federated learning systems. In European symposium on research in computer security, pages 480-501. Springer, 2020.
[89]. Shubhi Shukla, Manaar Alam, Sarani Bhattacharya, Pabitra Mitra, and Dehdeep Mukhopadhyay. "whispering mlaas": Exploiting timing channels to compromise user privacy in deep neural networks. IACR Transactions on Cryptographic Hardware and Embedded Systems, pages 587-613, 2023.
[90]. Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace, and Florian Tramer. Privacy side channels in machine learning systems. In 33rd USENIX Security Symposium (USENIX Security 24), pages 6861-6848, 2024.
[91]. Venkatraman Renganathan and Tyler Holt Summers. Spoof resilient coordination for distributed multi-robot systems. 2017 International Symposium on Multi-Robot and Multi-Agent Systems (MRS), pages 135-141, 2017. URL https://api.semanticscholar.org/CorpusID:7897062.
[92]. Richard M. Chang, Guofei Jiang, Franjo Ivancic, Sriram Sankaranarayanan, and Vitaly Shmatikov. Inputs of coma: Static detection of denial-of-service vulnerabilities. 2009 22nd IEEE Computer Security Foundations Symposium, pages 186-199, 2009. URL https://api.semanticscholar.org/CorpusID:6355518.
[93]. Shiyi Yang, Zhibo Hu, Xinshu Li, Chen Wang, Tong Yu, Xiwei Xu, Liming Zhu, and Lina Yao. Drunkagent: Stealthy memory corruption in llm-powered recommender agents. arXiv preprint arXiv:2503.23804, 2025.
[94]. Sumeet Motwani, Mikhail Baranchuk, Martin Strohmeier, Vijay Bolina, Philip Torr, Lewis Hammond, and Christian Schroeder de Witt. Secret collusion among ai agents: Multi-agent deception via steganography. Advances in Neural Information Processing Systems, 37:73439-73486, 2024.
[95]. Rana Shahroz, Zhen Tan, Sukwon Yun, Charles Fleming, and Tianlong Chen. Agents under siege: Breaking pragmatic multi-agent llm systems with optimized prompt attacks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9661-9674, 2025.
[96]. Yang Li, Wenhao Zhang, Jianhong Wang, Shao Zhang, Yali Du, Ying Wen, and Wei Pan. Aligning individual and collective objectives in multi-agent cooperation. Advances in Neural Information Processing Systems, 37: 44735-44760, 2024.
[97]. Bei Chen, Gaolei Li, Xi Lin, Zheng Wang, and Jianhua Li. Blockagents: Towards byzantine-robust llm-based multi-agent coordination via blockchain. In Proceedings of the ACM Turing Award Celebration Conference-China 2024, pages 187-192, 2024.
[98]. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling, 2024.
[99]. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. Backdooring instruction-tuned large language models with virtual prompt injection. arXiv preprint arXiv:2307.16888, 2023.
[100]. Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In 2022 IEEE Symposium on Security and Privacy (SP), pages 2043–2059. IEEE, 2022.
[101]. Rui Zeng, Xi Chen, Yuwen Pu, Xuhong Zhang, Tianyu Du, and Shouling Ji. Clibe: detecting dynamic backdoors in transformer-based nlp models. arXiv preprint arXiv:2409.01193, 2024.
[102]. JFrog Security Research Team. Jfrog and hugging face join forces to expose malicious ml models, March 2025. URL https://jfrog.com/community/ai/jfrog-and-hugging-face-join-forces-to-expose-malicious-ml-models/. Accessed: 2025-06-04.
[103]. Wei Duan, Jie Lu, and Junyu Xuan. Group-aware coordination graph for multi-agent reinforcement learning. arXiv preprint arXiv:2404.10976, 2024.
[104]. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, et al. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025.
[105]. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling, 2024.
[106]. Xuezhou Zhang, Yuzhe Ma, Adish Singla, and Xiaojin Zhu. Adaptive reward-poisoning attacks against reinforcement learning. In International Conference on Machine Learning, pages 11225–11234. PMLR, 2020.
[107]. David F Ferraiolo, Ravi Sandhu, Serban Gavrila, D Richard Kuhn, and Ramaswamy Chandramouli. Proposed nist standard for role-based access control. ACM Transactions on Information and System Security (TISSEC), 4 (3):224–274, 2001.
[108]. Boyi Zeng, Lizheng Wang, Yuncong Hu, Yi Xu, Chenghu Zhou, Xinbing Wang, Yu Yu, and Zhouhan Lin. Hurfe: Human-readable fingerprint for large language models. Advances in Neural Information Processing Systems, 37: 126332–126362, 2024.
[109]. Timour Igamberdiev, Thomas Arnold, and Ivan Habernal. Dp-rewrite: Towards reproducibility and transparency in differentially private text rewriting. arXiv preprint arXiv:2208.10400, 2022.
[110]. Weiyan Shi, Ryan Shea, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204.07667, 2022.
[111]. Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. Are large pre-trained language models leaking your personal information? arXiv preprint arXiv:2205.12628, 2022.
[112]. Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. The emerged security and privacy of llm agent: A survey with case studies. arXiv preprint arXiv:2407.19354, 2024.
[113]. William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. Taintdroid: an information-flow tracking system for realtime privacy monitoring on smartphones. ACM Transactions on Computer Systems (TOCS), 32(2):1–29, 2014.
[114]. Shoaib Ahmed Siddiqui, Radhika Gaonkar, Boris Köpf, David Krueger, Andrew Paverd, Ahmed Salem, Shruti Tople, Lukas Wutschitz, Menglin Xia, and Santiago Zanella-Béguelin. Permissive information-flow analysis for large language models. arXiv preprint arXiv:2410.03055, 2024.
[115]. Pierre Peigne, Mikolaj Kniejski, Filip Sondej, Matthieu David, Jason Hoelscher-Obermaier, Christian Schroeder de Witt, and Esben Kran. Multi-agent security tax: Trading off security and collaboration capabilities in multi-agent systems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27573–27581, 2025.
[116]. Satbir Singh. Llm-based agents: The benefits and the risks. https://www.enkryptai.com/blog/llm&agents&benefits&&srisks, February 2025. Accessed: 2025-08-21.
[117]. Christian Schroeder de Witt. Open challenges in multi-agent security: Towards secure systems of interacting ai agents. arXiv preprint arXiv:2505.02077, 2025.
[118]. Anshuman Chhabra, Peizhao Li, Prasant Mohapatra, and Hongfu Liu. " what data benefits my classifier?" enhancing model performance and interpretability through influence-based data selection. In ICLR, 2024.
[119]. Anshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu. Outlier gradient analysis: Efficiently identifying detrimental training samples for deep learning models. In ICML, 2025.
[120]. Sahar Abdelnabi, Aideen Fay, Giovanni Cherubin, Ahmed Salem, Mario Fritz, and Andrew Paverd. Get my drift? catching llm task drift with activation deltas. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 43–67, 2024. URL https://api.semanticscholar.org/CorpusID:270211056.
[121]. Dami Choi, Yonadav Shavit, and David K Duvenaud. Tools for verifying neural models' training data. Advances in Neural Information Processing Systems, 36:1154-1188, 2023.
[122]. Dan Petrovic. Advanced interpretability techniques for tracing llm activations. https://dejan.ai/blog/advanced-interpretability-techniques-for-tracing-llm-activations/, March 2025. Accessed: 2025-08-21.
[123]. Tom Sander, Pierre Fernandez, Alain Durmus, Matthijs Douze, and Teddy Furon. Watermarking makes language models radioactive. Advances in Neural Information Processing Systems, 37:21079-21113, 2024.
[124]. Meng Hao, Hongwei Li, Hanxiao Chen, Pengzhi Xing, Guowen Xu, and Tianwei Zhang. Iron: Private inference on transformers. Advances in neural information processing systems, 35:15718-15731, 2022.
[125]. Georgios A Kaissis, Marcus R Makowski, Daniel Rückert, and Rickmer F Braren. Secure, privacy-preserving and federated machine learning in medical imaging. Nature Machine Intelligence, 2(6):305-311, 2020.
[126]. Francis Dutil, Alexandre See, Lisa Di Jorio, and Florent Chandelier. Application of homomorphic encryption in medical imaging. arXiv preprint arXiv:2110.07768, 2021.
[127]. Harshal Tapsamudre, Arun Kumar, Vikas Agarwal, Nisha Gupta, and Sneha Mondal. Ai-assisted controls change management for cybersecurity in the cloud. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12629-12635, 2022.
[128]. Leo de Castro, Antigoni Polychroniadou, and Daniel Escudero. Privacy-preserving large language model inference via gpu-accelerated fully homomorphic encryption. In Neurips Safe Generative AI Workshop 2024, 2024.
[129]. Deevashwer Rathee, Dacheng Li, Ion Stoica, Hao Zhang, and Raluca Popa. Mpc-minimized secure llm inference. arXiv preprint arXiv:2408.03561, 2024.
[130]. Tao Lu, Haoyu Wang, Wenjie Qu, Zonghui Wang, Jinye He, Tianyang Tao, Wenzhi Chen, and Jiaheng Zhang. An efficient and extensible zero-knowledge proof framework for neural networks. Cryptology ePrint Archive, 2024.
[131]. Yurun Chen, Xavier Hu, Keting Yin, Juncheng Li, and Shengyu Zhang. Evaluating the robustness of multimodal agents against active environmental injection attacks. arXiv preprint arXiv:2502.13053, 2025.
[132]. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023.
[133]. De Chezelles, Thibault Le Sellier, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lu, Ori Yoran, Dehan Kong, Frank F Xu, Siva Reddy, Quentin Cappart, et al. The browsergym ecosystem for web agent research. arXiv preprint arXiv:2412.05467, 2024.
[134]. Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, and Huzefa Rangwala. Agentocam: A simple yet strong baseline for llm-based web agents. arXiv preprint arXiv:2410.13825, 2024.
[135]. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[136]. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URL https://arxiv.org/abs/2306.06070.
[137]. Yaxin Luo, Zhaoyi Li, Jiacheng Liu, Jiacheng Cui, Xiaohan Zhao, and Zhiqiang Shen. Open captcha world: A comprehensive web-based platform for testing and benchmarking multimodal llm agents. arXiv preprint arXiv:2505.24878, 2025.
