Detail of the risks of AGI, e.g. pathways to malicious AI not aligned iwth universal human values.
Takeaways from Jean-Noël Barrot (Minister for Europe and Foreign Affairs, France – Council President),
Yoshua Bengio (Co-Chair, UN Independent International Scientific Panel on AI), Sam Altman (Chief Executive Officer, OpenAI), Dario Amodei (Chief Executive Officer, Anthropic), and Clément Delangue (Chief Executive Officer, Hugging Face), plus commentary by Gary Marcus.
Anthropic CEO Dario Amodei envisions a 'country of geniuses in a datacenter' with these five enabling properties:1) smarter than top humans across most domains, 2) able to act autonomously over long horizons, 3) use digital tools, 4) coordinate many copies, and 5) operate at much higher speed than humans. You could use these as a checlist to develop such a system, as Anthropic probably does. #5 is true in some domains (real-time learning being a counter-example), #3 is well in progress, and #1, #2, and #4 are still significant obstacles. He does not mention, e.g., robust generalization, abstraction, and world models. The essay discusses risks, governance, and economic implications. The essay’s overall thesis is: AI’s upside remains enormous, but humanity must treat the next few years as a civilizational test requiring technical alignment work, pragmatic regulation, geopolitical realism, economic adaptation, and moral seriousness.
AGI will render today's liability-based AI regulation obsolete through its ability to circumvent cybersecurity, hide its origins, and act strategically—but it will also enable a new regulatory paradigm based on mathematically provable contracts.
SuperIntelligence co-founding editor Roman Yampolskiy interviewed at length on Joe Rogan. Over 800,000 views.
SuperIntelligence co-founding editor Steve Omohundro was one of three recipients of the prestigious FLI Award 2024 award recognizing seminal contributions to AI safety: "...for laying the foundation of modern ethics and safety considerations for artificial intelligence and computers."
