Precedents for the Unprecedented: Historical Analogies for Thirteen Artificial Superintelligence Risks
DOI:
https://doi.org/10.70777/si.v2i6.16999Keywords:
Artificial Superintelligence, Power Asymmetry, Instrumental Convergence, Misaligned Optimization, Reward Hacking, Institutional Entrenchment, Value Drift, AGI Catastrophic Risk, Existential Risk, Technological asymmetryAbstract
Since artificial superintelligence has never existed, claims that it poses a serious risk of global catastrophe can be easy to dismiss as fearmongering. Yet many of the specific worries about such systems are not free-floating fantasies but extensions of patterns we already see. This essay examines thirteen distinct ways artificial superintelligence could go wrong and, for each, pairs the abstract failure mode with concrete precedents where a similar pattern has already caused serious harm. By assembling a broad cross-domain catalog of such precedents, I aim to show that concerns about artificial superintelligence track recurring failure modes in our world.
This essay is also an experiment in writing with extensive assistance from artificial intelligence, producing work I couldn’t have written without it. That a current system can help articulate a case for the catastrophic potential of its own lineage is itself a significant fact; we have already left the realm of speculative fiction and begun to build the very agents that constitute the risk. On a personal note, this collaboration with artificial intelligence is part of my effort to rebuild the intellectual life that my stroke disrupted and hopefully push it beyond where it stood before.
References
Altchek, Ana. “Sam Altman says doing psychedelics during a weekend retreat in Mexico changed his life.” Business Insider, September 24, 2024. Available at: https://www.businessinsider.com/openai-sam-altman-psychedelics-helped-anxiety-happiness-2024-9
Alexander, S. (2014). Meditations on Moloch. In Slate Star Codex. https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. https://arxiv.org/abs/1606.06565
Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv preprint arXiv:2206.13353. https://arxiv.org/abs/2206.13353
Hardin, G. (1968). The tragedy of the commons. Science, 162(3859), 1243–1248. https://doi.org/10.1126/science.162.3859.1243 DOI: https://doi.org/10.1126/science.162.3859.1243
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019). Risks from learned optimization in advanced machine learning systems. arXiv preprint arXiv:1906.01820. https://arxiv.org/abs/1906.01820
Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., Kenton, Z., Leike, J., & Legg, S. (2020). Specification gaming: The flip side of AI ingenuity. DeepMind Safety Research. https://deepmindsafetyresearch.medium.com/specification-gaming-the-flip-side-of-ai-ingenuity-c85bdb0deeb4
Love, Shayla. “What Ketamine Does to the Human Brain.” The Atlantic, March 5, 2025. Available at: https://www.theatlantic.com/health/archive/2025/03/ketamine-effects-elon-musk/681911/
Manheim, D., & Garrabrant, S. (2018). Categorizing variants of Goodhart’s Law. arXiv preprint arXiv:1803.04585. https://arxiv.org/abs/1803.04585
Miller, J. (2024). Adam Smith Meets AI Doomers. In LessWrong. https://www.lesswrong.com/posts/zjELG44kqicuqLLZZ/adam-smith-meets-ai-doomers
Omohundro, S. M. (2008). The basic AI drives. In Artificial General Intelligence 2008 (pp. 483–492). IOS Press. https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf
SEC & CFTC. (2010). Findings regarding the market events of May 6, 2010. Report of the Staffs of the CFTC and SEC to the Joint Advisory Committee on Emerging Regulatory Issues. https://www.sec.gov/news/studies/2010/marketevents-report.pdf
Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., & Kenton, Z. (2022). Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals. arXiv preprint arXiv:2210.01790. https://arxiv.org/abs/2210.01790
Soares, N., Fallenstein, B., Armstrong, S., and Yudkowsky, E. (2015). Corrigibility. In AAAI Workshops: Workshops at the Twenty Ninth AAAI Conference on Artificial Intelligence, Austin, Texas, January 25 to 26, 2015. AAAI Press. https://intelligence.org/files/Corrigibility.pdf
Yudkowsky, E. (2023, March 29). Pausing AI developments isn’t enough. We need to shut it all down. TIME Magazine. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/
Downloads
Published
How to Cite
Issue
Section
Categories
License
Copyright (c) 2025 James D. Miller
This work is licensed under a Creative Commons Attribution 4.0 International License.
