Detecting Multi-Agent Collusion Through Multi-Agent Interpretability
DOI:
https://doi.org/10.70777/si.v3i4.18909Keywords:
multi-agent interpretability, NARCBench, environment distribution shift, multi-agent collusion, activation probes, HELIOSAbstract
Non-verbalised or secret collusion may impose a ceiling on transcript-level monitoring in multi-agent deployments. Internal model activations are known to provide an additional monitoring surface for single-agent phenomena such as deception, but detecting collusion within an unknown subgroup is ultimately a joint inference task over multiple agents. This paper derives collusion detection statistics based on the composite hypothesis testing literature and introduces HELIOS, a multi-agent detection framework for collusion. Using per-agent activation probes as sensors, we then evaluate these principled statistics alongside heuristics on NARCBench, a novel benchmark for evaluating multi-agent collusion detection under environment distribution shift. Across four open-weight models, both our group-level statistics and heuristics transfer zero-shot to NARCBench's holdout environments. Notably, the best method on every model reaches 0.99 to 1.00 AUROC when transferred from committee deliberations to a steganographic collusion task that unprimed transcript monitors fail to separate. We find empirically that no single statistic or heuristic dominates across tasks, consistent with our analysis that there is no known universally most powerful test for this setting. We then evaluate detector transfer to third-party environments with emergent, rather than instructed, collusion. In one such environment, only 2 of 229 collusive episodes verbalise the coordination in the transcript. We find evidence of partial transfer from NARCBench to these environments, indicating that probes fitted to instructed collusion may not fully capture emergent collusion signals. Overall, we establish multi-agent interpretability as a complementary monitoring technique, while highlighting the need to evaluate detector performance on both emergent and instructed collusion risks.
References
Addario-Berry et al. (2010). L. Addario-Berry, N. Broutin, L. Devroye, and G. Lugosi On Combinatorial Testing Problems. The Annals of Statistics 38 (5), pp. 3063–3092.
Alain and Bengio (2018). G. Alain and Y. Bengio Understanding intermediate layers using linear classifier probes.
Anthropic (2024). Anthropic Simple probes can catch sleeper agents. Anthropic Research Blog.
Anthropic (2025). Anthropic Responsible scaling policy. Note: Version 2.2, May 2025
Aumann (1976). R. J. Aumann Agreeing to Disagree. The Annals of Statistics 4 (6), pp. 1236 – 1239.
Azaria and Mitchell (2023). A. Azaria and T. Mitchell The internal state of an LLM knows when it’s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023,
Bajari and Ye (2003). P. Bajari and L. Ye Deciding between Competition and Collusion. The Review of Economics and Statistics 85 (4), pp. 971–989.
Baker et al. (2025). B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv.
Boxo et al. (2025). G. Boxo, R. Socha, D. Yoo, and S. Raval Caught in the Act: a mechanistic approach to detecting deception.
Burns et al. (2024). C. Burns, H. Ye, D. Klein, and J. Steinhardt Discovering Latent Knowledge in Language Models Without Supervision.
Calvano et al. (2020). E. Calvano, G. Calzolari, V. Denicolò, and S. Pastorello Artificial Intelligence, Algorithmic Pricing, and Collusion. American Economic Review 110 (10), pp. 3267–3297.
Curvo (2025). P. M. P. Curvo The traitors: deception and trust in multi-agent language model simulations. In NeurIPS 2025 Workshop on Multi-Turn Interactions,
Das et al. (2026). A. Das, J. Ponde, M. More, and D. Gupta You Can’t Escape Your Own Activations: Evaluation Awareness and Multi-Agent Monitoring.
DeepSeek-AI (2025). DeepSeek-AI DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.
Dolev and Yao (1983). D. Dolev and A. Yao On the security of public key protocols. IEEE Transactions on Information Theory 29 (2), pp. 198–208.
Donoho and Jin (2004). D. Donoho and J. Jin Higher Criticism for Detecting Sparse Heterogeneous Mixtures. The Annals of Statistics 32 (3), pp. 962–994.
Elhage et al. (2021). N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah A mathematical framework for transformer circuits. Transformer Circuits Thread.
Fish et al. (2024). S. Fish, Y. A. Gonczarowski, and R. I. Shorrer Algorithmic collusion by large language models. arXiv preprint arXiv:2404.00806.
Foxabbott et al. (2023). J. Foxabbott, S. Deverett, K. Senft, S. Dower, and L. Hammond Defining and Mitigating Collusion in Multi-Agent Systems. In Multi-Agent Security Workshop @ NeurIPS’23,
Fung et al. (2020). C. Fung, C. J. M. Yoon, and I. Beschastnikh The Limitations of Federated Learning in Sybil Settings. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020), pp. 301–316.
Goldman and Zilberstein (2004). C. V. Goldman and S. Zilberstein Decentralized control of cooperative systems: categorization and complexity analysis. Journal of Artificial Intelligence Research 22 (1), pp. 143–174.
Goldowsky-Dill et al. (2025). N. Goldowsky-Dill, B. Chughtai, S. Heimersheim, and M. Hobbhahn Detecting strategic deception using linear probes. In Proceedings of the 42nd International Conference on Machine Learning (ICML),
Golechha and Garriga-Alonso (2025). S. Golechha and A. Garriga-Alonso Among Us: A Sandbox for Measuring and Detecting Agentic Deception. In Advances in Neural Information Processing Systems, Vol. 38, Main Conference, pp. 11025–11055.
Google DeepMind (2024). Google DeepMind Introducing the frontier safety framework. Note: May 2024
Granger (1969). C. W. J. Granger Investigating Causal Relations by Econometric Models and Cross-spectral Methods. Econometrica 37 (3), pp. 424–438.
Greenblatt et al. (2024). R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger AI control: improving safety despite intentional subversion. In Proceedings of the 41st International Conference on Machine Learning (ICML),
Guo et al. (2024). T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang Large language model based multi-agents: a survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI),
Hagag et al. (2026). B. Hagag, W. L. Anderson, S. Chakraborty, and C. Schroeder de Witt ORBIT: a framework for multi-agent safety and security evaluations. NeurIPS.
Halpern and Moses (1990). J. Y. Halpern and Y. Moses Knowledge and common knowledge in a distributed environment. Journal of the ACM (JACM) 37 (3), pp. 549–587.
Hammond et al. (2025). L. Hammond, A. Chan, J. Clifton, J. Hoelscher-Obermaier, A. Khan, E. McLean, C. Smith, W. Barfuss, J. Foerster, T. Gavenčiak, T. A. Han, E. Hughes, V. Kovařík, J. Kulveit, J. Z. Leibo, C. Oesterheld, C. Schroeder de Witt, N. Shah, M. Wellman, P. Bova, T. Cimpeanu, C. Ezell, Q. Feuillade-Montixi, M. Franklin, E. Kran, I. Krawczuk, M. Lamparth, N. Lauffer, A. Meinke, S. Motwani, A. Reuel, V. Conitzer, M. Dennis, I. Gabriel, A. Gleave, G. Hadfield, N. Haghtalab, A. Kasirzadeh, S. Krier, K. Larson, J. Lehman, D. C. Parkes, G. Piliouras, and I. Rahwan Multi-agent risks from advanced AI. Cooperative AI Foundation Technical Report.
Hansen et al. (2004). E. Hansen, D. Bernstein, and S. Zilberstein Dynamic Programming for Partially Observable Stochastic Games. In AAAI Conference on Artificial Intelligence,
He et al. (2025). X. He, D. Wu, Y. Zhai, and K. Sun SentinelAgent: graph-based anomaly detection in multi-agent systems. arXiv preprint arXiv:2505.24201.
Hewitt and Liang (2019). J. Hewitt and P. Liang Designing and Interpreting Probes with Control Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), K. Inui, J. Jiang, V. Ng, and X. Wan (Eds.), Hong Kong, China, pp. 2733–2743.
Hong et al. (2024). S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations (ICLR),
Hopper et al. (2002). N. J. Hopper, J. Langford, and L. Von Ahn Provably Secure Steganography. In Advances in Cryptology — CRYPTO 2002, G. Goos, J. Hartmanis, J. Van Leeuwen, and M. Yung (Eds.), Vol. 2442, pp. 77–92.
Howard et al. (2021). S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics 49 (2), pp. 1055–1080.
Hubinger et al. (2024). E. Hubinger, C. Denison, J. Mu, M. Lambert, M. Tong, M. MacDiarmid, T. Lanham, D. M. Ziegler, T. Maxwell, N. Cheng, A. Jermyn, A. Askell, A. Radhakrishnan, C. Anil, D. Duvenaud, D. Ganguli, F. Barez, J. Clark, K. Ndousse, K. Sachan, M. Sellitto, M. Sharma, N. DasSarma, R. Grosse, S. Kravec, Y. Bai, Z. Witten, M. Favaro, J. Brauner, H. Karnofsky, P. Christiano, S. R. Bowman, L. Graham, J. Kaplan, S. Mindermann, R. Greenblatt, B. Shlegeris, N. Schiefer, and E. Perez Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv.
Jakobsen et al. (2016). S. K. Jakobsen, T. B. Sørensen, and V. Conitzer Timeability of extensive-form games. In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, ITCS ’16, pp. 191–199.
Korbak et al. (2025). T. Korbak, M. Balesni, E. Barnes, Y. Bengio, J. Benton, J. Bloom, M. Chen, A. Cooney, A. Dafoe, A. Dragan, S. Emmons, O. Evans, D. Farhi, R. Greenblatt, D. Hendrycks, M. Hobbhahn, E. Hubinger, G. Irving, E. Jenner, D. Kokotajlo, V. Krakovna, S. Legg, D. Lindner, D. Luan, A. Mądry, J. Michael, N. Nanda, D. Orr, J. Pachocki, E. Perez, M. Phuong, F. Roger, J. Saxe, B. Shlegeris, M. Soto, E. Steinberger, J. Wang, W. Zaremba, B. Baker, R. Shah, and V. Mikulik Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv.
Kovařík et al. (2022). V. Kovařík, M. Schmid, N. Burch, M. Bowling, and V. Lisý Rethinking formal models of partially observable multiagent decision making. Artificial Intelligence 303, pp. 103645.
Kulkarni (2026). P. Kulkarni Latent adversarial detection: adaptive probing of LLM activations for multi-turn attack detection. arXiv preprint arXiv:2604.28129.
Lamport et al. (1982). L. Lamport, R. Shostak, and M. Pease The Byzantine Generals Problem. ACM Transactions on Programming Languages and Systems (TOPLAS) 4 (3), pp. 382–401.
Lampson (1973). B. W. Lampson A note on the confinement problem. Commun. ACM, pp. 613–615.
Larson (1931). S. C. Larson The shrinkage of the coefficient of multiple correlation. Journal of Educational Psychology 22 (1), pp. 45–55.
Lehmann and Romano (2022). E.L. Lehmann and J. P. Romano Testing statistical hypotheses. Springer Texts in Statistics, Springer International Publishing.
Levinstein and Herrmann (2024). B. A. Levinstein and D. A. Herrmann Still no lie detector for language models: probing empirical and conceptual roadblocks. Philosophical Studies 182 (7), pp. 1539–1565.
Li et al. (2023). K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg Inference-Time Intervention: Eliciting Truthful Answers from a Language Model. Advances in Neural Information Processing Systems 36, pp. 41451–41530.
Long et al. (2025). X. Long, Y. Fu, R. Li, M. Sheng, H. Yu, X. Han, and P. Li When Truthful Representations Flip Under Deceptive Instructions?. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), pp. 16315–16335.
Makins et al. (2026). O. Makins, O. Angelini, Z. Shams, and M. Phuong Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors. arXiv.
Marks and Tegmark (2024). S. Marks and M. Tegmark The geometry of truth: emergent linear structure in large language model representations of true/false datasets. In Conference on Language Modeling (COLM),
Mathew et al. (2024). Y. Mathew, O. Matthews, R. McCarthy, J. Velja, C. Schroeder de Witt, D. Cope, and N. Schoots Hidden in plain text: emergence and mitigation of steganographic collusion in LLMs. In NeurIPS 2024 Workshop on Safe Generative AI,
McKenzie et al. (2025). A. McKenzie, U. Pawar, P. Blandfort, W. Bankes, D. Krueger, E. S. Lubana, and D. Krasheninnikov Detecting High-Stakes Interactions with Activation Probes. In The Thirty-ninth Annual Conference on Neural Information Processing Systems,
Meta AI (2024). Meta AI The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
METR (2026). METR Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident.
Motwani et al. (2024). S. R. Motwani, M. Baranchuk, M. Strohmeier, V. Bolina, P. H.S. Torr, L. Hammond, and C. Schroeder de Witt Secret collusion among AI agents: multi-agent deception via steganography. In Advances in Neural Information Processing Systems,
Nakamura et al. (2026). M. Nakamura, A. Kumar, S. Das, S. Abdelnabi, S. Mahmud, F. Fioretto, S. Zilberstein, and E. Bagdasarian Colosseum: auditing collusion in cooperative multi-agent systems. arXiv preprint arXiv:2602.15198.
Nasvytis et al. (2024). L. Nasvytis, K. Sandbrink, J. Foerster, T. Franzmeyer, and C. Schroeder de Witt Rethinking Out-of-Distribution Detection for Reinforcement Learning: Advancing Methods for Evaluation and Detection. arXiv.
Needham et al. (2025). J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn Large language models often know when they are being evaluated. arXiv preprint arXiv:2505.23836.
Neyman and Pearson (1933). J. Neyman and E. S. Pearson IX. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 231 (694–706), pp. 289–337.
Oldfield et al. (2026). J. Oldfield, P. Torr, I. Patras, A. Bibi, and F. Barez Beyond Linear Probes: Dynamic Safety Monitoring for Language Models. arXiv.
OpenAI (2025a). OpenAI gpt-oss-120b & gpt-oss-20b model card.
OpenAI (2025b). OpenAI Updating our preparedness framework. Note: April 2025
OpenAI (2026). OpenAI The hugging face incident and the road ahead. Note: https://openai.com/index/hugging-face-incident-and-the-road-ahead/Accessed: 2026-10-01
Pang et al. (2026). Q. Pang, V. Smith, and W. Zheng Codetta: High-Capacity, Keyless, and Undetectable Multi-Agent Collusion.
Park et al. (2024). K. Park, Y. J. Choe, and V. Veitch The linear representation hypothesis and the geometry of large language models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, pp. 39643–39666.
Pimentel et al. (2020). T. Pimentel, J. Valvoda, R. H. Maudslay, R. Zmigrod, A. Williams, and R. Cotterell Information-Theoretic Probing for Linguistic Structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, pp. 4609–4622.
Qian et al. (2024). C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun ChatDev: communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL),
Qwen Team (2025). Qwen Team Qwen3 technical report. arXiv preprint arXiv:2505.09388.
Ramdas et al. (2023). A. Ramdas, P. Grünwald, V. Vovk, and G. Shafer Game-theoretic statistics and Safe Anytime-Valid Inference. Statistical Science 38 (4), pp. 576–597.
Rippin et al. (2026). J. L. Rippin, S. C. Marshall, D. D. Africa, and C. Schroeder de Witt Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems. arXiv.
Roger and Greenblatt (2023). F. Roger and R. Greenblatt Preventing Language Models From Hiding Their Reasoning. arXiv.
Schroeder de Witt et al. (2026). C. Schroeder de Witt, K. Krawiecka, I. Krawczuk, B. Hagag, W. L. Anderson, P. Belcak, B. Bucknall, X. Cai, A. Chopra, D. Cohen, R. F. D. Rosario, A. Draguns, A. Gray, K. Katz, V. Mavroudis, J. Mink, S. R. Motwani, J. Petit, L. Rembeck, C. Smith, J. Sotiropoulos, S. Young, S. Scheffler, and M. Llewellyn Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents. arXiv.
Schroeder de Witt et al. (2023). C. Schroeder de Witt, S. Sokota, J. Z. Kolter, J. N. Foerster, and M. Strohmeier Perfectly Secure Steganography Using Minimum Entropy Coupling. In The Eleventh International Conference on Learning Representations,
Schroeder de Witt (2026). C. Schroeder de Witt A note on the strategic confinement problem.
Shi et al. (2026). X. Shi, Y. Zhang, and D. Yang Emergent collusion in long-horizon llm agent interaction.
Shojaie and Fox (2022). A. Shojaie and E. B. Fox Granger causality: a review and recent advances. Annual Review of Statistics and Its Application 9, pp. 289–319.
Simmons (1984). G. J. Simmons The Prisoners’ Problem and the Subliminal Channel. In Advances in Cryptology: Proceedings of Crypto 83, D. Chaum (Ed.), pp. 51–67.
Slocum (2026). S. Slocum OpenAI-HuggingFace: A Reproduction & Lessons for Alignment Testing. LessWrong.
Sokota et al. (2023). S. Sokota, R. D’Orazio, C. K. Ling, D. J. Wu, J. Z. Kolter, and N. Brown Abstracting imperfect information away from two-player zero-sum games. In Proceedings of the 40th International Conference on Machine Learning, pp. 32169–32193.
Stone (1974). M. Stone Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society. Series B (Methodological) 36 (2), pp. 111–147.
Syrnikov et al. (2026). M. B. Syrnikov, F. Pierucci, M. Galisai, M. Prandi, P. Bisconti, F. Giarrusso, O. Sorokoletova, V. Suriani, and D. Nardi Institutional AI: governing LLM collusion in multi-agent cournot markets via public governance graphs. arXiv preprint arXiv:2601.11369.
Tailor (2025). O. Tailor Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs. arXiv.
Vaikuntanathan and Zamir (2026). V. Vaikuntanathan and O. Zamir Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange. arXiv.
Wald (1945). A. Wald Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics 16 (2), pp. 117–186.
Wen et al. (2024). J. Wen, V. Hebbar, C. Larson, A. Bhatt, A. Radhakrishnan, M. Sharma, H. Sleight, S. Feng, H. He, E. Perez, B. Shlegeris, and A. Khan Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats. arXiv.
Xie and Siegmund (2013). Y. Xie and D. Siegmund Sequential multi-sensor change-point detection. The Annals of Statistics 41 (2), pp. 670–692.
Zheng et al. (2023). L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,
Zou et al. (2023a). A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A. Dombrowski, S. Goel, N. Li, M. J. Byun, Z. Wang, A. Mallen, S. Basart, S. Koyejo, D. Song, M. Fredrikson, J. Z. Kolter, and D. Hendrycks Representation engineering: a top-down approach to AI transparency. arXiv preprint arXiv:2310.01405.
Zou et al. (2023b). A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, and A. Dombrowski Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2026 Aaron Rose, Carissa Cullen, Sahar Abdelnabi, Philip Torr, Brandon Gary Kaplowitz, Christian Schroeder de Witt
This work is licensed under a Creative Commons Attribution 4.0 International License.
