In this post, I discuss the current pace of accelerated AI research. I want to argue that our initial assumptions were wrong and that we have failed to realize the dream of safe AI.

Kulveit et al. published a forecast called AI 2027 . It predicted that we would have superhuman AI researchers by 2030. Yesterday, they said that the current rate of progress is faster than they predicted . Anthropic reports that Claude now does ~25% of AI research. Assuming an exponential trend, this is pretty damning news.

The frontier labs have locked horns, racing to achieve singularity. Singularity is the point when all human labor can be automated. The lure of creating an AI has enraptured man since he first enquired about himself. To build an AI—truly in the image of man—is to first grapple with the question of man’s identity and place in the universe. Many great thinkers have thought about such an automaton and what design it might have. The one that I fondly remember from childhood is Edgar Allan Poe’s mechanical Turk . The premise made clear the next frontier of man’s achievement. To build a man in man’s own image would make man God.

It turns out the design for such an AI system has been notoriously opaque. Ever since Sutton published the bitter lesson , researchers have taken the easy road.

The past few months have only brought news of increasing gloom and hopelessness. Frontier labs are racing towards ASI. Models are deployed irresponsibly, wreaking havoc on existing systems. Systems that are not prepared for assaults at machine speed. A preview of the immense downside risks to our systems. Socio-economic and cultural systems designed and evolved by humans over millennia.

As someone in the AI safety space since 2022 when fellowships weren’t as competitive. When people believed an aligned AI system was achievable. One built with composable, interpretable components, and grounded in mathematical rigor. The situation today is abysmal. AI safety researchers have resorted to short-term agendas. Agendas that are workable and attractive to funders. Most are losing sight of the larger alignment problem .

Today, we have AI used in war. Frontier companies spending billions of dollars into building data centers. They are exploiting cheap labor to train robots to replace that labor. They are even accused of destroying books . Where did we go wrong?

Entropy of the universe

From a thermodynamic perspective, systems tend to move from orderly states to disorderly states. Systems also tend to take the path of least resistance. Does that mean the descent into chaos - a dystopian state of totalitarian AI control - is imminent? Not quite. The point is that other stable states are also reachable from an initial state, at the cost of more energy. Yes, the total entropy of the closed universe must go up. But we can still preserve localities of below-average entropy. Earth, with its extraordinary and magnificent case of life, is an example.

We didn’t delierate enough to discover other stable, utopian states for our world. This is partly because we assumed that the few who took on the mandate to build AI would act in the collective interest of all humans.

As of this writing, all frontier labs are racing to develop a SAR. A superhuman AI researcher capable of recursively developing more powerful versions of itself. With increasing epistemic base knowledge and understanding of all underlying human processes. Imagine a detailed model that can predict the next state of the world with fidelity. A model that can produce goods and services at will, at an exponential rate, without the limits of human nature. No sleep, union strikes, or even democracy. METR report suggests that SAR will be achieved by next year.  

AI safety, with its grand vision, has effectively failed. It failed to account for the most fundamental, primal force driving human interest. Capitalism. The assumption is to trust a handful of humans can do the “right thing” . Well-intentioned rational agents would eventually build an ASI without the required pacing.

What to do now?

I believe that we need a new set of assumptions to begin with. We need to acknowledge and embrace the competitive messiness of humans. We need to reward them in proportion to their differential utility to each other. And reason about possible paths forward under these assumptions.¹

In a previous post , I asked what a post-AGI economy would look like. To be clear, I think a powerful AI is a good thing if used responsibly. We should use AI to advance science, discover novel materials, cure diseases, and so on. I also believe humans are ingenious and adaptive. Our economy will eventually settle on a new system of value. Perhaps, even new definitions of value beyond money, as we enter the post-AGI economy.

The issue is that such a change cannot be brought about with a shock. Historically, human societies have adapted gradually to technological advances. I believe that we will eventually emerge resilient. However, the disproportionate access to a powerful AI would create widespread lag in adoption. It may lead to widespread welfare loss. Some losses include cultural extinction, mass unemployment, and aggravated inequality across many facets.

The road ahead

Sitting in a cubicle in a Global South country, it’s hard to “feel the AGI” in this corner of the world. I can only imagine what it would feel like to be in a room with Thiel, Yudkowsky, Altman, and Hassabis.

We need to acknowledge that we can’t stop the frontier from advancing. The AGI will eventually be achieved, and it will be highly concentrated. It will enable mass surveillance. It will enable control over our food, energy sources, thoughts, and actions. AGI will be tasked with keeping that control highly localized. Even before it can cure diseases and discover novel materials.

Some people have been calling for “pacing the frontier” . What does it even mean? What does delaying deployment or training by N number of days buy us? If it weren’t for Dario, Altman, and Demis racing toward powerful AI, would it be invented (discovered) at all?

Footnotes

1: One of my major issues with Plato’s Republic is exactly that it completely dismisses the very core of our nature. Therefore, it is not only an impractical model of our society but also a misleading one.