Post-AGI Civilizational Equilibria Workshop at ICML 2025

I attended the Post-AGI Civilizational Equilibria Workshop at ICML 2025 organized by David Duvenaud, Nora Ammann Raymond Douglas, and Jan Kulveit.

These are my notes from the workshop.

Can goodness compete? by Joe Carlsmith

Context: Joe Carlsmith is a senior research analyst at OpenPhil. Sadly I showed up late and missed the first 5-10 minutes.

• Variant 1:
• Variant 2: Technical AI alignment i.e., Point AI optimization power at alignment
  • There are two types of alignment taxes:
    1. development and deployment. Might go slower, might be more costly
    2. Tax imposed by values themselves e.g., deontological constraints
• Variant 3: Negative sum dynamics: competition leads to results that are worse for all participants
  • Burning the cosmic commons: each party races to colonize as much of the universe as fast as possible
  • Arms races: Each party builds up a huge arsenal of weapons to counter the arsenal built up by other parties
  • Classic solution: Coordination
    • Parties foresee the costs of the negative sum dynamics and avoid them e.g., via suitably credible and stable commitments
    • How likely is this kind of coordination to occur and succeed? Not at all clear.
• Variant 4: Strategy Stealing Assumption: agents with good values don’t have any inherent competitive disadvantage b/c they can copy whatever bad agents’ policies
  • Many reasons why this might not hold!
  • Classic answer: prevent/constrain the forms of competition

Taking the Proliferation of Highly-Capable AI Seriously by Stephen Casper

Context: Stephen Casper is an MIT EECS PhD researcher working on Technical AI Governance.

• Open weight models are a pretty big deal
  • Forecasting open model weights figure

Scenario Planning for Transformative AI’s Economic Impact by Anna Yelizarova

Context: Anna Yelizarova currently leads the Windfall Trust initiative

• Our mission: Prepare society for economic disruption of transformative AI & shape future where Windfall it creates benefits all of humanity
• Scenario planning helps us avoid blind spots & prepare for outcomes we’re not politically ready to talk about
• Two axes:
  • Augmenting vs Replacing Human Labor
  • Curtailed vs Accessible
• Partnering with Metaculus to predict scenarios

Living in an Extremely Unequal World by Richard Ngo

Missed it because of lunch - this was the talk I most wanted to see!

• Question from audience: How do we bootstrap love and care?
  • Richard:
    • In a previous version, I laid out a vision aristocracy where the wishes of the more capable agents are the only things you care about
    • We are breaking new ground trying to integrate our modern sense of egalitarianism
    • Lots of work to do!

The History of Technologically Provoked Welfare Erosion

• How might AI lead to a bad future?
  • Bad Technology
  • Bad Actors
  • Bad Evolution
• Welfare degradation
  • The institutional form of a society affects the wellbeing of its inhabitants
  • The institutional form of a society affects its competitive performance
  • Thus, institutions of human societies are subject to an evolutionary selection process over time
• (I can’t tell if this talk is full of platitudes or is clearly wrong?)
• Conclusion: Under sufficiently competitive conditions, seemingly benign technologies can degrade wellbeing by granting welfare limiting institutions a competitive advantage

Resisting AI-Enabled Authoritarianism by Fazl Barez

• AI systems have 4 politically dangerous features
  • Massive data ingestion
  • Black-box inference (no transparency, explainability, accountability)
  • Automated decisions: no human in the loop
  • Real-time speed (bypass democratic oversight)