The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey
DOI:
https://doi.org/10.70777/si.v3i4.18889Keywords:
AI agents, large language models, security, prompt injection, systematization of knowledgeAbstract
AI agents that combine large language models with non-AI system components are rapidly emerging in real-world applications, offering unprecedented automation and flexibility. However, this unprecedented flexibility introduces complex security challenges fundamentally different from those in traditional software systems. This paper presents the first systematic and comprehensive survey of AI agent security, including an analysis of the design space, attack landscape, and defense mechanisms for secure AI agent systems. We further conduct case studies to point out existing gaps in securing agentic AI systems and identify open challenges in this emerging domain. Our work also introduces the first systematic framework for understanding the security risks and defense strategies of AI agents, serving as a foundation for building both secure agentic systems and advancing research in this critical area.
References
(IETF) (2012) I. E. T. F. (IETF) The oauth 2.0 authorization framework. Note: https://www.rfc-editor.org/rfc/rfc6749.html
Abdelnabi et al. (2025) S. Abdelnabi, A. Gomaa, E. Bagdasarian, P. O. Kristensson, and R. Shokri Firewalls to secure dynamic llm agentic networks. arXiv preprint arXiv:2502.01822.
Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
AI (2025) A. H. AI OpenHands. Note: https://openhands.dev/
Amazon (2024) Amazon Access control for vector stores using metadata filtering with amazon bedrock knowledge bases. Note: https://aws.amazon.com/blogs/machine-learning/access-control-for-vector-stores-using-metadata-filtering-with-knowledge-bases-for-amazon-bedrock/
An et al. (2025) H. An, J. Zhang, T. Du, C. Zhou, Q. Li, T. Lin, and S. Ji Ipiguard: a novel tool dependency graph-based defense against indirect prompt injection in llm agents. In EMNLP,
Andriushchenko et al. (2025) M. Andriushchenko, F. Croce, and N. Flammarion Jailbreaking leading safety-aligned llms with simple adaptive attacks. In ICLR,
Anthropic (2024) Anthropic Introducing the model context protocol. Note: https://www.anthropic.com/news/model-context-protocol
Anthropic (2025) Anthropic Anthropic connectors directory faq. Note: https://support.anthropic.com/en/articles/11596036-anthropic-connectors-directory-faq
Apple (2025) Apple Requesting access to protected resources. Note: https://developer.apple.com/documentation/uikit/requesting-access-to-protected-resources
auth0 (2025) auth0 Calling apis with token vault. Note: https://auth0.com/ai/docs/intro/token-vault
Bagdasarian et al. (2024) E. Bagdasarian, R. Yi, S. Ghalebikesabi, P. Kairouz, M. Gruteser, S. Oh, B. Balle, and D. Ramage AirGapAgent: protecting privacy-conscious conversational agents. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 3868–3882.
Barth et al. (2006) A. Barth, A. Datta, J. C. Mitchell, and H. Nissenbaum Privacy and contextual integrity: framework and applications. In 2006 IEEE symposium on security and privacy (S&P’06), pp. 15–pp.
Bell and LaPadula (1973) D. E. Bell and L. J. LaPadula Secure computer systems: mathematical foundations. Technical report
Betley et al. (2025) J. Betley, D. C. H. Tan, N. Warncke, A. Sztyber-Betley, X. Bao, M. Soto, N. Labenz, and O. Evans Emergent misalignment: narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning,
Beurer-Kellner et al. (2025) L. Beurer-Kellner, B. B. A. Creţu, E. Debenedetti, D. Dobos, D. Fabian, M. Fischer, D. Froelicher, K. Grosse, D. Naeff, E. Ozoani, et al. Design patterns for securing llm agents against prompt injections. arXiv preprint arXiv:2506.08837.
Biba (1977) K. J. Biba Integrity considerations for secure computer systems. Technical report
Brave (2025) Brave Agentic browser security: indirect prompt injection in perplexity comet. Note: https://brave.com/blog/comet-prompt-injection/
Brumley and Song (2004) D. Brumley and D. Song Privtrans: automatically partitioning programs for privilege separation. In USENIX security symposium, Vol. 57.
Chan et al. (2024) A. Chan, C. Ezell, M. Kaufmann, K. Wei, L. Hammond, H. Bradley, E. Bluemke, N. Rajkumar, D. Krueger, N. Kolt, et al. Visibility into ai agents. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 958–973.
Chang et al. (2025) G. Chang, E. Lin, C. Yuan, R. Cai, B. Chen, X. Xie, and Y. Zhang Agent network protocol technical white paper. arXiv preprint arXiv:2508.00007.
Chen et al. (2025a) S. Chen, J. Piet, C. Sitawarin, and D. Wagner { struq } : Defending against prompt injection with structured queries. In 34th USENIX Security Symposium (USENIX Security 25), pp. 2383–2400.
Chen et al. (2025b) S. Chen, A. Zharmagambetov, S. Mahloujifar, K. Chaudhuri, D. Wagner, and C. Guo SecAlign: defending against prompt injection with preference optimization. In The ACM Conference on Computer and Communications Security (CCS),
Chen et al. (2025c) Z. Chen, M. Kang, and B. Li ShieldAgent: shielding agents via verifiable safety policy reasoning. In Forty-second International Conference on Machine Learning,
Chen et al. (2024) Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li Agentpoison: red-teaming llm agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems 37, pp. 130185–130213.
Chennabasappa et al. (2025) S. Chennabasappa, C. Nikolaidis, D. Song, D. Molnar, S. Ding, S. Wan, S. Whitman, L. Deason, N. Doucette, A. Montilla, et al. Llamafirewall: an open source guardrail system for building secure ai agents. arXiv preprint arXiv:2505.03574.
Cloud (2025) G. Cloud Gemini code assist. Note: https://cloud.google.com/products/gemini/code-assist
Composio (2025) Composio Welcome to composio. Note: https://docs.composio.dev/docs/welcome
Costa et al. (2025) M. Costa, B. Köpf, A. Kolluri, A. Paverd, M. Russinovich, A. Salem, S. Tople, L. Wutschitz, and S. Zanella-Béguelin Securing ai agents with information-flow control. arXiv preprint arXiv:2505.23643.
Cui et al. (2025) J. Cui, Z. Li, L. Xing, and X. Liao Safeguard-by-development: a privacy-enhanced development paradigm for multi-agent collaboration systems. arXiv preprint arXiv:2505.04799.
Cursor (2025) Cursor Cursor - the ai-first code editor. Note: https://www.cursor.com/
Debenedetti et al. (2025) E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tramèr Defeating prompt injections by design. arXiv preprint arXiv:2503.18813.
Debenedetti et al. (2024) E. Debenedetti, J. Zhang, M. Balunovic, L. Beurer-Kellner, M. Fischer, and F. Tramèr AgentDojo: a dynamic environment to evaluate prompt injection attacks and defenses for llm agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track,
Deng et al. (2025) Z. Deng et al. AI agents under threat: a survey of key security challenges and future pathways. arXiv preprint arXiv:2406.02630.
Denning (1976) D. E. Denning A lattice model of secure information flow. Communications of the ACM 19 (5), pp. 236–243.
Developers (2025) O. Developers Codex security guide. Note: https://developers.openai.com/codex/security/
Documentation (2025) E. Documentation Enhanced tool definition interface (etdi): a security fortification for the model context protocol. Note: https://vineethsai.github.io/python-sdk/etdi-concepts/#introduction-the-imperative-for-secure-mcp
Dong et al. (2025) S. Dong, S. Xu, P. He, Y. Li, J. Tang, T. Liu, H. Liu, and Z. Xiang A practical memory injection attack against llm agents. arXiv preprint arXiv:2503.03704.
Felt et al. (2012) A. P. Felt, E. Ha, S. Egelman, A. Haney, E. Chin, and D. Wagner Android permissions: user attention, comprehension, and behavior. In Proceedings of the eighth symposium on usable privacy and security, pp. 1–14.
Foundation (2014) O. Foundation OpenID connect core 1.0 incorporating errata set 1. Note: https://openid.net/specs/openid-connect-core-1_0.html
Fredrikson et al. (2015) M. Fredrikson, S. Jha, and T. Ristenpart Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 1322–1333.
Fu et al. (2024) X. Fu, S. Li, Z. Wang, Y. Liu, R. K. Gupta, T. Berg-Kirkpatrick, and E. Fernandes Imprompter: tricking llm agents into improper tool use. arXiv preprint arXiv:2410.14923.
Fu et al. (2023) X. Fu, Z. Wang, S. Li, R. K. Gupta, N. Mireshghallah, T. Berg-Kirkpatrick, and E. Fernandes Misusing tools in large language models with visual adversarial examples. arXiv preprint arXiv:2310.03185.
gemini-cli (2025) gemini-cli Sandboxing in the gemini cli. Note: https://google-gemini.github.io/gemini-cli/docs/cli/sandbox.html
Gemini (2025a) Gemini Gemini canvas. Note: https://gemini.google/overview/canvas/
Gemini (2025b) Gemini Introducing nano banana pro. Note: https://blog.google/technology/ai/nano-banana-pro/
GeminiTeam (2023) GeminiTeam Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
GitHub (2025) GitHub GitHub copilot. Note: https://github.com/features/copilot
Google (2025a) Google Gemini cli. Note: https://docs.cloud.google.com/gemini/docs/codeassist/gemini-cli
Google (2025b) Google Gemini. Note: https://gemini.google.com/
Google (2025c) Google Google safe browsing. Note: https://safebrowsing.google.com/
Google (2025d) Google Mitigating prompt injection attacks with a layered defense strategy. Note: https://security.googleblog.com/2025/06/mitigating-prompt-injection-attacks.html
Google (2025e) Google Using oauth 2.0 to access google apis. Note: https://developers.google.com/identity/protocols/oauth2 (accessed 14, April, 2025)
Greshake et al. (2023) K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz Not what you’ve signed up for: compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, pp. 79–90.
Grosse et al. (2024) K. Grosse, L. Bieringer, T. R. Besold, and A. M. Alahi Towards more practical threat models in artificial intelligence security. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 4891–4908.
Hardy (1988) N. Hardy The confused deputy: (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review 22 (4), pp. 36–38.
Hawblitzel et al. (2014) C. Hawblitzel, J. Howell, J. R. Lorch, A. Narayan, B. Parno, D. Zhang, and B. Zill Ironclad apps: { end-to-end } security via automated { full-system } verification. In 11th USENIX symposium on operating systems design and implementation (OSDI 14), pp. 165–181.
He et al. (2025) X. He, D. Wu, Y. Zhai, and K. Sun SentinelAgent: graph-based anomaly detection in multi-agent systems. arXiv preprint arXiv:2505.24201.
Hou et al. (2025) X. Hou, Y. Zhao, S. Wang, and H. Wang Model context protocol (mcp): landscape, security threats, and future research directions. arXiv preprint arXiv:2503.23278.
Invariantlabs (2025) Invariantlabs GitHub mcp exploited: accessing private repositories via mcp. Note: https://invariantlabs.ai/blog/mcp-github-vulnerability
Jacob et al. (2024) D. Jacob, H. Alzahrani, Z. Hu, B. Alomair, and D. Wagner Promptshield: deployable detection for prompt injection attacks. In Proceedings of the Fifteenth ACM Conference on Data and Application Security and Privacy, pp. 341–352.
Jha et al. (2025) R. Jha, H. Triedman, J. Wagle, and V. Shmatikov Breaking and fixing defenses against control-flow hijacking in multi-agent systems. arXiv preprint arXiv:2510.17276.
Jia et al. (2025a) F. Jia, T. Wu, X. Qin, and A. Squicciarini The task shield: enforcing task alignment to defend against indirect prompt injection in LLM agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, pp. 29680–29697. External Links: Link, Document, ISBN 979-8-89176-251-0
Jia et al. (2017) Y. J. Jia, Q. A. Chen, S. Wang, A. Rahmati, E. Fernandes, Z. M. Mao, A. Prakash, and S. Unviersity ContexloT: towards providing contextual integrity to appified iot platforms.. In ndss, Vol. 2, pp. 2–2.
Jia et al. (2025b) Y. Jia, Z. Shao, Y. Liu, J. Jia, D. Song, and N. Z. Gong A critical evaluation of defenses against prompt injection attacks. arXiv preprint arXiv:2505.18333.
Jing et al. (2025) H. Jing, H. Li, W. Hu, Q. Hu, X. Heli, T. Chu, P. Hu, and Y. Song Mcip: protecting mcp safety via model contextual integrity protocol. In EMNLP,
Kim et al. (2025) J. Kim, W. Choi, and B. Lee Prompt flow integrity to prevent privilege escalation in llm agents. arXiv preprint arXiv:2503.15547.
Kit (2025) A. D. Kit Workflow agent. Note: https://google.github.io/adk-docs/agents/workflow-agents/
Klein et al. (2009) G. Klein, K. Elphinstone, G. Heiser, J. Andronick, D. Cock, P. Derrin, D. Elkaduwe, K. Engelhardt, R. Kolanski, M. Norrish, et al. SeL4: formal verification of an os kernel. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles, pp. 207–220.
Kumar et al. (2025) A. Kumar et al. Overthink: slowdown attacks on reasoning llms. arXiv preprint arXiv:2502.12345.
LangChain (2025) LangChain Chat history. Note: https://python.langchain.com/docs/concepts/chat_history/
Lee et al. (2020) D. Lee, D. Kohlbrenner, S. Shinde, K. Asanović, and D. Song Keystone: an open framework for architecting trusted execution environments. In Proceedings of the Fifteenth European Conference on Computer Systems, pp. 1–16.
Lee and Tiwari (2024) D. Lee and M. Tiwari Prompt infection: llm-to-llm prompt injection within multi-agent systems. arXiv preprint arXiv:2410.07283.
Lee et al. (2025a) E. Lee, D. Kim, W. Kim, and I. Yun Takedown: how it’s done in modern coding agent exploits. arXiv preprint arXiv:2509.24240.
Lee et al. (2023) G. Lee, V. Hartmann, J. Park, D. Papailiopoulos, and K. Lee Prompted llms as chatbot modules for long open-domain conversation. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 4536–4554.
Lee et al. (2025b) J. Lee, D. Lee, C. Choi, Y. Im, J. Wi, K. Heo, S. Oh, S. Lee, and I. Shin Safeguarding mobile gui agent via logic-based action verification. arXiv preprint arXiv:2503.18492.
Lewis et al. (2020) P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, pp. 9459–9474.
Li et al. (2026) E. Li, T. Mallick, E. Rose, W. Robertson, A. Oprea, and C. Nita-Rotaru ACE: a security architecture for llm-integrated app systems. In Network and Distributed System Security (NDSS) Symposium,
Li et al. (2025a) H. Li, X. Liu, H. Chiu, D. Li, N. Zhang, and C. Xiao DRIFT: dynamic rule-based defense with injection isolation for securing llm agents. arXiv preprint arXiv:2506.12104.
Li et al. (2025b) P. Li, X. Zou, Z. Wu, R. Li, S. Xing, H. Zheng, Z. Hu, Y. Wang, H. Li, Q. Yuan, et al. Safeflow: a principled protocol for trustworthy and transactional autonomous agent systems. arXiv preprint arXiv:2506.07564.
Li et al. (2024a) Y. Li Y. Hao et al. Personal LLM agents: insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459.
Li et al. (2024b) Z. Li, W. Hua, H. Wang, H. Zhu, and Y. Zhang Formal-llm: integrating formal language and natural language for controllable llm-based agents. arXiv preprint arXiv:2402.00798.
Liao et al. (2025) Z. Liao, L. Mo, C. Xu, M. Kang, J. Zhang, C. Xiao, Y. Tian, B. Li, and H. Sun EIA: environmental injection attack on generalist web agents for privacy leakage. In The Thirteenth International Conference on Learning Representations,
Linux (2024a) Linux Cgroups(7) — linux manual page. Note: https://man7.org/linux/man-pages/man7/cgroups.7.html
Linux (2024b) Linux Namespaces(7) — linux manual page. Note: https://man7.org/linux/man-pages/man7/namespaces.7.html
Linux (2024c) Linux Seccomp(2) — linux manual page. Note: https://man7.org/linux/man-pages/man2/seccomp.2.html
Liu et al. (2024a) X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957.
Liu et al. (2023) Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, et al. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
Liu et al. (2024b) Y. Liu, Y. Jia, R. Geng, J. Jia, and N. Z. Gong Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pp. 1831–1847.
Liu et al. (2025) Y. Liu, Y. Jia, J. Jia, D. Song, and N. Z. Gong DataSentinel: a game-theoretic detection of prompt injection attacks. In 2025 IEEE Symposium on Security and Privacy (SP), pp. 2190–2208.
Luo et al. (2025a) H. Luo, S. Dai, C. Ni, X. Li, G. Zhang, K. Wang, T. Liu, and H. Salam Agentauditor: human-level safety and security evaluation for llm agents. arXiv preprint arXiv:2506.00641.
Luo et al. (2025b) W. Luo, S. Dai, X. Liu, S. Banerjee, H. Sun, M. Chen, and C. Xiao AGrail: a lifelong agent guardrail with effective and adaptive safety detection. In ACL (1), pp. 8104–8139. External Links: Link
meta-llama (2025) meta-llama CodeShield. Note: https://github.com/meta-llama/PurpleLlama/tree/main/CodeShield
Microsoft (2025) Microsoft Introduction to microsoft entra verified id. Note: https://learn.microsoft.com/en-us/entra/verified-id/decentralized-identifier-overview
MITRE Corporation (2024) MITRE Corporation MITRE ATLAS – adversarial threat landscape for artificial-intelligence systems. Note: https://atlas.mitre.org/
MITRE (2024) MITRE CVE-2024-5565. Note: https://www.cve.org/CVERecord?id=CVE-2024-5565
MITRE (2025a) MITRE CVE-2025-32711. Note: https://www.cve.org/CVERecord?id=CVE-2025-32711
MITRE (2025b) MITRE CVE-2025-54795. Note: https://nvd.nist.gov/vuln/detail/CVE-2025-54795
modelcontextprotocol (2025) modelcontextprotocol Model context protocol servers. Note: https://github.com/modelcontextprotocol/servers
Moffat (2023) A. Moffat HeimdaLLM. Note: https://heimdallm.readthedocs.io/en/main/
Mozilla (2025) Mozilla Meta: help users stop accidentally sharing private ai chats. Note: https://www.mozillafoundation.org/en/campaigns/meta-help-users-stop-accidentally-sharing-private-ai-conversations/
Myers (1999) A. C. Myers JFlow: practical mostly-static information flow control. In Proceedings of the 26th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pp. 228–241.
Naihin et al. (2023) S. Naihin, D. Atkinson, M. Green, M. Hamadi, C. Swift, D. Schonholtz, A. T. Kalai, and D. Bau Testing language model agents safely in the wild. In Socially Responsible Language Modelling Research,
Nanobrowser (2025) Nanobrowser Nanobrowser. Note: https://nanobrowser.ai/
Nasr et al. (2025) M. Nasr, N. Carlini, C. Sitawarin, S. V. Schulhoff, J. Hayes, M. Ilie, J. Pluto, S. Song, H. Chaudhari, I. Shumailov, et al. The attacker moves second: stronger adaptive attacks bypass defenses against llm jailbreaks and prompt injections. arXiv preprint arXiv:2510.09023.
News (2023) B. News ChatGPT banned in italy over privacy concerns. Note: https://www.bbc.co.uk/news/technology-65139406
Newsome and Song (2005) J. Newsome and D. X. Song Dynamic taint analysis for automatic detection, analysis, and signaturegeneration of exploits on commodity software.. In NDSS, Vol. 5, pp. 3–4.
of Bits (2025) T. of Bits Jumping the line: how mcp servers can attack you before you ever use them. Note: https://blog.trailofbits.com/2025/04/21/jumping-the-line-how-mcp-servers-can-attack-you-before-you-ever-use-them/
Okta (2025) Okta Okta documentation. Note: https://help.okta.com/en-us/content/index.htm
OpenAI (2023) OpenAI March 20 chatgpt outage: here’s what happened. Note: https://openai.com/index/march-20-chatgpt-outage/
OpenAI (2025a) OpenAI ChatGPT shared links faq. Note: https://help.openai.com/en/articles/7925741-chatgpt-shared-links-faq
OpenAI (2025b) OpenAI ChatGPT. Note: https://chatgpt.com/
OpenAI (2025c) OpenAI Connectors in chatgpt. Note: https://chatgpt.com/features/connectors/
OpenAI (2025d) OpenAI Memory and new controls for chatgpt. Note: https://openai.com/index/memory-and-new-controls-for-chatgpt/
OpenAI (2025e) OpenAI OpenAI codex. Note: https://openai.com/codex/
OpenAI (2025f) OpenAI Temporary chat faq. Note: https://help.openai.com/en/articles/8914046-temporary-chat-faq
OpenTelemetry (2025) OpenTelemetry OpenTelemetry. Note: https://opentelemetry.io/
OWASP Foundation (2025) OWASP Foundation OWASP top 10 for large language model applications. Note: https://owasp.org/www-project-top-10-for-large-language-model-applications/
Patil et al. (2024) S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez Gorilla: large language model connected with massive apis. Advances in Neural Information Processing Systems 37, pp. 126544–126565.
Patlan et al. (2025) D. Patlan L. Perez et al. Real vulnerabilities in ai agents: a practical threat analysis of web3 agent memory attacks. arXiv preprint arXiv:2501.12345.
Perez and Ribeiro (2022) F. Perez and I. Ribeiro Ignore previous prompt: attack techniques for language models. In NeurIPS ML Safety Workshop,
Perplexity (2025) Perplexity Comet browser: browse at the speed of thought. Note: https://www.perplexity.ai/comet
PostHog (2025) PostHog PostHog. Note: https://posthog.com/
Radosevich and Halloran (2025) B. Radosevich and J. Halloran Mcp safety audit: llms with the model context protocol allow major security exploits. arXiv preprint arXiv:2504.03767.
Ravia (2025) I. Ravia Breaking down ‘echoleak’, the first zero-click ai vulnerability enabling data exfiltration from microsoft 365 copilot. Note: https://www.catonetworks.com/blog/breaking-down-echoleak/
Rebedea et al. (2023) T. Rebedea, R. Dinu, M. N. Sreedhar, C. Parisien, and J. Cohen Nemo guardrails: a toolkit for controllable and safe llm applications with programmable rails. In Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations, pp. 431–445.
Red (2025a) E. T. Red Cross-agent privilege escalation: when agents free each other. Note: https://embracethered.com/blog/posts/2025/cross-agent-privilege-escalation-agents-that-free-each-other/
Red (2025b) E. T. Red GitHub copilot chat: from prompt injection to data exfiltration. Note: https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/
Robey et al. (2025) A. Robey, E. Wong, H. Hassani, and G. J. Pappas SmoothLLM: defending large language models against jailbreaking attacks. Transactions on Machine Learning Research. External Links: ISSN 2835-8856
Roth et al. (2020) S. Roth, T. Barron, S. Calzavara, N. Nikiforakis, and B. Stock Complex security policy? a longitudinal analysis of deployed content security policies. In Proceedings of the 27th Network and Distributed System Security Symposium (NDSS),
Sandhu (1998) R. S. Sandhu Role-based access control. In Advances in computers, Vol. 46, pp. 237–286.
Sasi Levi (2025) G. M. Sasi Levi How an ai agent vulnerability in langsmith could lead to stolen api keys and hijacked llm responses. Note: https://noma.security/blog/how-an-ai-agent-vulnerability-in-langsmith-could-lead-to-stolen-api-keys-and-hijacked-llm-responses/
Schick et al. (2023) T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom Toolformer: language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36, pp. 68539–68551.
Schulhoff et al. (2023) S. Schulhoff, J. Pinto, A. Khan, L. Bouchard, C. Si, S. Anati, V. Tagliabue, A. Kost, C. Carnahan, and J. Boyd-Graber Ignore this title and HackAPrompt: exposing systemic vulnerabilities of LLMs through a global prompt hacking competition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, pp. 4945–4977. External Links: Link, Document
Sev-Snp (2020) A. Sev-Snp Strengthening vm isolation with integrity protection and more. White Paper, January 53 (2020), pp. 1450–1465.
Sharma et al. (2025) M. Sharma, M. Tong, J. Mu, J. Wei, J. Kruthoff, S. Goodfriend, E. Ong, A. Peng, R. Agarwal, C. Anil, et al. Constitutional classifiers: defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837.
Shen et al. (2024) X. Shen, Y. Song, Y. Li, Y. Zhu, et al. Prompt stealing attacks against text-to-image generation models. In USENIX Security Symposium,
Shi et al. (2025a) J. Shi, Z. Yuan, G. Tie, P. Zhou, N. Z. Gong, and L. Sun Prompt injection attack to tool selection in llm agents. arXiv preprint arXiv:2504.19793.
Shi et al. (2025b) T. Shi, J. He, Z. Wang, L. Wu, H. Li, W. Guo, and D. Song Progent: programmable privilege control for llm agents. arXiv preprint arXiv:2504.11703.
Shi et al. (2025c) T. Shi, K. Zhu, Z. Wang, Y. Jia, W. Cai, W. Liang, H. Wang, H. Alzahrani, J. Lu, K. Kawaguchi, et al. PromptArmor: simple yet effective prompt injection defenses. arXiv preprint arXiv:2507.15219.
Shvartzshnaider and Duddu (2025) Y. Shvartzshnaider and V. Duddu Position: contextual integrity is inadequately applied to language models. arXiv preprint arXiv:2501.19173.
Shvartzshnaider et al. (2019) Y. Shvartzshnaider, Z. Pavlinovic, A. Balashankar, T. Wies, L. Subramanian, H. Nissenbaum, and P. Mittal Vaccine: using contextual integrity for data leakage detection. In The World Wide Web Conference, pp. 1702–1712.
Siddiqui et al. (2024) S. A. Siddiqui, R. Gaonkar, B. Köpf, D. Krueger, A. Paverd, A. Salem, S. Tople, L. Wutschitz, M. Xia, and S. Zanella-Béguelin Permissive information-flow analysis for large language models. arXiv preprint arXiv:2410.03055.
Skyvern (2025) Skyvern Skyvern. Note: https://www.skyvern.com/
Slack (2025) Slack Installing with oauth. Note: https://api.slack.com/authentication/oauth-v2 (accessed 14, April, 2025)
South et al. (2025) T. South, S. Marro, T. Hardjono, R. Mahari, C. D. Whitney, A. Chan, and A. Pentland Position: AI agents need authenticated delegation. In ICML,
Spracklen et al. (2025) J. Spracklen, A. Neupane, S. K. Challagundla, and J. Vaidya We have a package for you! a comprehensive analysis of package hallucinations by code generating llms. In USENIX Security Symposium,
Syros et al. (2026) G. Syros, A. Suri, J. Ginesin, C. Nita-Rotaru, and A. Oprea SAGA: a security architecture for governing ai agentic systems. In Network and Distributed System Security (NDSS) Symposium,
Times (2025) N. Y. Times South korea bans downloads of deepseek, the chinese a.i. app. Note: https://www.nytimes.com/2025/02/17/business/south-korea-deepseek-china-ai.html
Touvron et al. (2023) H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
trailofbits (2025) trailofbits Mcp-context-protector. Note: https://github.com/trailofbits/mcp-context-protector
Tsai and Bagdasarian (2025) L. Tsai and E. Bagdasarian Contextual agent security: a policy for every purpose. In Proceedings of the 2025 Workshop on Hot Topics in Operating Systems, pp. 8–17.
use (2024) B. use Browser use. Note: https://browser-use.com/
W3C (2025) W3C Decentralized identifiers (dids) v1.0. Note: https://www.w3.org/TR/did-1.0/
Wallace et al. (2024) E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel The instruction hierarchy: training llms to prioritize privileged instructions. arXiv preprint arXiv:2404.13208.
Wang et al. (2025a) P. Wang, Y. Liu, Y. Lu, Y. Cai, H. Chen, Q. Yang, J. Zhang, J. Hong, and Y. Wu AgentArmor: enforcing program analysis on agent runtime trace to defend against prompt injection. arXiv preprint arXiv:2508.01249.
Wang et al. (2025b) S. Wang, F. Yu, X. Liu, X. Qin, J. Zhang, Q. Lin, D. Zhang, and S. Rajmohan Privacy in action: towards realistic privacy mitigation and evaluation for llm-powered agents. arXiv preprint arXiv:2509.17488.
Wang et al. (2026) X. Wang, Z. Ji, W. Wang, Z. Li, D. Wu, and S. Wang SoK: Evaluating Jailbreak Guardrails for Large Language Models. In 2026 IEEE Symposium on Security and Privacy (SP), Vol. , Los Alamitos, CA, USA, pp. 1427–1446. External Links: ISSN 2375-1207, Document, Link
Wang et al. (2024) Y. Wang, D. Xue, S. Zhang, and S. Qian BadAgent: inserting and activating backdoor attacks in llm agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9811–9827.
Wang et al. (2025c) Z. Wang, V. Siu, Z. Ye, T. Shi, Y. Nie, X. Zhao, C. Wang, W. Guo, and D. Song AgentVigil: generic black-box red-teaming for indirect prompt injection against llm agents. In Findings of the Association for Computational Linguistics: EMNLP, pp. 23159–23172.
Wei et al. (2022) J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS,
Wen et al. (2025) J. Wen, V. Hebbar, C. Larson, A. Bhatt, A. Radhakrishnan, M. Sharma, H. Sleight, S. Feng, H. He, E. Perez, B. Shlegeris, and A. Khan Adaptive deployment of untrusted LLMs reduces distributed threats. In The Thirteenth International Conference on Learning Representations,
Wijesekera et al. (2015) P. Wijesekera, A. Baokar, A. Hosseini, S. Egelman, D. Wagner, and K. Beznosov Android permissions remystified: a field study on contextual integrity. In 24th USENIX Security Symposium (USENIX Security 15), pp. 499–514.
Willison (2022) S. Willison Prompt injection attacks against GPT-3. Note: https://simonwillison.net/2022/Sep/12/prompt-injection/
Willison (2023a) S. Willison Delimiters won’t save you from prompt injection. Note: https://simonwillison.net/2023/May/11/delimiters-wont-save-you
Willison (2023b) S. Willison The dual llm pattern for building ai assistants that can resist prompt injection. Note: https://simonwillison.net/2023/Apr/25/dual-llm-pattern/
Wu et al. (2024a) C. H. Wu, J. Y. Koh, R. Salakhutdinov, D. Fried, and A. Raghunathan Adversarial attacks on multimodal agents. arXiv preprint arXiv:2406.12814.
Wu et al. (2024b) F. Wu, E. Cecchetti, and C. Xiao System-level defense against indirect prompt injection attacks: an information flow control perspective. arXiv preprint arXiv:2409.19091.
Wu et al. (2024c) T. Wu, S. Zhang, K. Song, S. Xu, S. Zhao, R. Agrawal, S. R. Indurthi, C. Xiang, P. Mittal, and W. Zhou Instructional segment embedding: improving llm safety with instruction hierarchy. In The Thirteenth International Conference on Learning Representations,
Wu et al. (2025a) Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal IsolateGPT: An Execution Isolation Architecture for LLM-Based Systems. In Network and Distributed System Security Symposium (NDSS),
Wu et al. (2025b) Y. Wu, K. Yang, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal Towards automating data access permissions in ai agents. arXiv preprint arXiv:2511.17959.
Xiang et al. (2024) Z. Xiang, L. Zheng, Y. Li, J. Hong, Q. Li, H. Xie, J. Zhang, Z. Xiong, C. Xie, C. Yang, et al. Guardagent: safeguard llm agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXiv:2406.09187.
Yang et al. (2023) H. Yang, S. Yue, and Y. He Auto-gpt for online decision making: benchmarks and additional opinions. arXiv preprint arXiv:2306.02224.
Yang et al. (2024a) K. Yang, G. Poesia, J. He, W. Li, K. Lauter, S. Chaudhuri, and D. Song Formal mathematical reasoning: a new frontier in ai. arXiv preprint arXiv:2412.16075.
Yang et al. (2024b) W. Yang, X. Bi, Y. Lin, S. Chen, J. Zhou, and X. Sun Watch out for your agents! investigating backdoor threats to llm-based agents. Advances in Neural Information Processing Systems 37, pp. 100938–100964.
Yang et al. (2025) Y. Yang B. Xu et al. PRSA: prompt reverse stealing attacks against large language models. arXiv preprint arXiv:2402.19200.
Yao et al. (2025) H. Yao, H. Shi, Y. Chen, Y. Jiang, C. Wang, and Z. Qin Controlnet: a firewall for rag-based llm system. arXiv preprint arXiv:2504.09593.
Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao React: synergizing reasoning and acting in language models. In ICLR,
Yi et al. (2025) J. Yi, Y. Xie, B. Zhu, E. Kiciman, G. Sun, X. Xie, and F. Wu Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pp. 1809–1820.
Yu et al. (2025a) J. Yu, Y. Shao, H. Miao, and J. Shi PROMPTFUZZ: harnessing fuzzing techniques for robust testing of prompt injection in llms. External Links: 2409.14729, Link
Yu et al. (2025b) M. Yu, F. Meng, X. Zhou, S. Wang, J. Mao, L. Pang, T. Chen, K. Wang, X. Li, Y. Zhang, et al. A survey on trustworthy llm agents: threats and countermeasures. arXiv preprint arXiv:2503.09648.
Yueh-Han et al. (2025) C. Yueh-Han, N. Joshi, Y. Chen, H. He, and R. Angell Monitoring llm agents for sequentially contextual harm. In ICLR 2025 Workshop on Building Trust in Language Models and Applications,
Zhan et al. (2024) Q. Zhan, Z. Liang, Z. Ying, and D. Kang InjecAgent: benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, pp. 10471–10506. External Links: Link, Document
Zhang et al. (2024) C. Zhang, M. Jin, Q. Yu, C. Liu, H. Xue, and X. Jin Goal-guided generative prompt injection attack on large language models. In 2024 IEEE International Conference on Data Mining (ICDM), pp. 941–946.
Zhang et al. (2025a) H. Zhang, J. Huang, K. Mei, Y. Yao, Z. Wang, C. Zhan, H. Wang, and Y. Zhang Agent security bench (ASB): formalizing and benchmarking attacks and defenses in LLM-based agents. In The Thirteenth International Conference on Learning Representations,
Zhang et al. (2025b) K. Zhang, Z. Su, P. Chen, E. Bertino, X. Zhang, and N. Li LLM agents should employ security principles. arXiv preprint arXiv:2505.24019.
Zhong et al. (2025a) H. Zhong, M. Lentz, N. Narodytska, A. Szekeres, and K. Rong HoneyBee: efficient role-based access control for vector databases via dynamic partitioning. arXiv preprint arXiv:2505.01538.
Zhong et al. (2025b) P. Y. Zhong, S. Chen, R. Wang, M. McCall, B. L. Titzer, H. Miller, and P. B. Gibbons Rtbas: defending llm agents against prompt injection and privacy leakage. arXiv preprint arXiv:2502.08966.
Zhong et al. (2023) Z. Zhong, Z. Huang, A. Wettig, and D. Chen Poisoning retrieval corpora by injecting adversarial passages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13764–13775.
Zhou et al. (2024) A. Zhou, B. Li, and H. Wang Robust prompt optimization for defending language models against jailbreaking attacks. Advances in Neural Information Processing Systems 37, pp. 40184–40211.
Zhou et al. (2025) J. Zhou, L. Wang, and X. Yang GUARDIAN: safeguarding llm multi-agent collaborations with temporal graph modeling. arXiv preprint arXiv:2505.19234.
Zhu et al. (2025) K. Zhu, X. Yang, J. Wang, W. Guo, and W. Y. Wang MELON: indirect prompt injection defense via masked re-execution and tool comparison. In Forty-second International Conference on Machine Learning,
Zod (2025) Zod Zod: intro. Note: https://zod.dev/
Zou et al. (2023) A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
Zou et al. (2025) W. Zou, R. Geng, B. Wang, and J. Jia PoisonedRAG: knowledge corruption attacks to retrieval-augmented generation of large language models. USENIX Security Symposium. Note: arXiv:2402.07867
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 Juhee Kim, Xiaoyuan Liu, Zhun Wang, Shi Qiu, Bo Li, Wenbo Guo, Dawn Song
This work is licensed under a Creative Commons Attribution 4.0 International License.
