Alignment - Scalable - Agent Foundations - Control Theory

4 Items

Focuses on mathematical formulations, decision theories, and control protocols ensuring that advanced autonomous systems remain aligned with human intent even when system capability exceeds human domain competence.

  • Scalable oversight and debate: Weak-to-strong generalization, recursive oversight systems, scalable oversight, and game-theoretic debate formulations.
  • Agent foundations and decision theory: Logical inductors, causal decision theory under self-reflection, embedded agency, and formal value learning dynamics.
  • Assistance games and objective specification: Assistance games, cooperative inverse reinforcement learning, resolving the King Midas problem, and learning under-specified user objectives under uncertainty.
  • Misalignment dynamics: Theoretical limits of alignment, goal misspecification, goal misgeneralisation, and emergent misalignment resulting from narrow fine-tuning.
  • Reward hacking and reward tampering: Reward gaming, specification gaming, reward tampering, and reward hacking during training loops and pre-deployment evaluations.
  • Corrigibility and containment: Mathematical guarantees for sub-agent creation, system interruptibility, corrigibility, and safe off-switch mechanisms.
  • AI control and untrusted monitoring: Protocols and containment architectures for AI control, ensuring safety even in the presence of untrusted or potentially subverting models.

All Items

  • Superintelligence values vs. divergent intelligent species' values

    AI Alignment: Humanity’s Final Ptolemaic Conceit

    Brian M. Delaney
    DOI: https://doi.org/10.70777/si.v3i4.18758
  • Diagram comparing a single-agent attack with a distributed three-agent attack. A monitor flags the single malicious commit with score 0.92, but three individually less suspicious commits score 0.68, 0.60, and 0.60 and jointly complete the attack. When combined, the commits are flagged with score 0.90, illustrating the fragmentation effect.

    Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors

    Oliver Makins, Orazio Angelini, Zohreh Shams, Mary Phuong
    83
    DOI: https://doi.org/10.70777/si.v3i4.18890
  • Attack and defense landscape of agentic AI

    The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey

    Juhee Kim, Xiaoyuan Liu, Zhun Wang, Shi Qiu, Bo Li, Wenbo Guo, Dawn Song
    DOI: https://doi.org/10.70777/si.v3i4.18889

  • White House Accord on Super Intelligence Joint Commitment on Frontier Responsibilities

    Donald J. Trump, Sundar Pichai, Dario Amodei, Mark Zuckerberg, Greg Brockman, Elon Musk, Jensen Huang
    2
    DOI: https://doi.org/10.70777/si.v3i4.18908