unhinged

The year AGI happened, and some predictions

Note on grammar and prose: those are two things that you get right with a lot of iterations. Not wanting to sound like an AI, I reined in the self-policing. It all contributes to giving this article a nice unhinged touch.

It was seven years ago that I started engaging in recreational writing. At the time, I wasn’t really considering AI, but climate change, and I tried to imagine how the world would look if people were desperately trying to do something about it. My first stop was to try to understand why we weren't. In short, my conclussion was that people are not doing much about climate change because that’s not really how our current world works. But what if they were going mad about it in the future? What if, then and there, everybody by comparison would regard Greta Thunberg as a mild historical figure?

Imagining alternatives is something that writers do for fun, and it’s called world building.

So my writing exercise was to start with the premise that, in my imaginary fictional world, people were actually succeeding in curbing carbon emissions, and from there I took the “least action” trajectory backwards in time and to the present, our current world. This recreational exercise took over my mind, and I had a hard time sleeping and focusing on work. Also, bear with me: the connection to AI and AGI comes a little later.

First, the big disclaimer: I write spicy fiction as a hobby, not prophecies. As is usually the case, my writing and my thinking are influenced by my life experiences, so I beg your leave for writing an autobiographical paragraph below.

In another life, I was an academic who meddled in complex systems research. Way before that, in high school, I was part of an education system that forced kids to take courses in the history of philosophy and economics (which was branded “Political Economy” because we were a bit old school). I have a major in Computer Science and a PhD in Physics. I’m no mathematician, and my proficiency in its fields is mixed, but I like to program numerical algorithms. These days, I deal with people professionally, as some sort of technical leader, and I’m bad enough at it that I regularly derive trauma from my own performance. But perhaps for lack of anything better, way nicer people clump around me, we get shit done, and the enterprise I work for pays me decently.

Without further ado, this is how I transplanted the “minimum action” method from physics to writing. Step one, you take the main statement in your (wishful) thinking: “CO2 emissions are meaningfully going down.” Then you imagine several ways to get there. This was my list of possibilities:

• Everybody uses electric cars
• Trans-Atlantic flights happen in electric planes.
• People fly less.
• People eat way less meat, so there’s less land use for cattle and for cattle fodder, which in turns helps with reforestation.
• There are fewer people on the planet.
• People are moving out of the planet.
• There’s less petrochemical industry.
• The petrochemical industry releases less CO2 into the atmosphere.

(I’m not going to add a chart showing how any of these things impact climate change; you can use a browser or an LLM to get your facts in five seconds.)

Implementing any of items above requires varying degrees of action, agency, and discomfort. Thus, step two is to apply the “minimum action” principle, where you assume it’s business as usual to the greatest extent possible, and you also favor the least controversial statements. For example, “Norway builds a colony in the Moon’s lava tubes and starts resettling its population there” is highly unlikely. On the other hand, “Billionaire X tries to resettle on the Moon and provokes huge popular backslash against space exploration” is very plausible.

Of the lines above, there are some that seem achievable in the world’s current course. For example, having all electric cars and airplanes within a few decades.

“Fewer people on Earth” is problematic. Although the demographic implosion seems unavoidable, if it happens, economic growth will cease and pension systems around the world will collapse. To prevent climate change, the world of the future would need to deal with these consequences through minimal action, but in an extremely unpopular way: “get used to it.” That in itself doesn’t seem very plausible.

People eating less meat, using less land for agriculture, the petrochemical industry shrinking seem highly implausible. The petrochemical industry actually caring for the environment is...well, impossible. But without these, the statement “CO2 emissions are meaningfully going down” doesn’t apply, and this is a bummer.

I didn’t have a solution for the problems above, but I was getting my highs from recreational writing. And, small secret, when you write fiction, you’re allowed to paper over facts about the world and amp up the drama. Which I contented myself with doing.

Then, about five years ago, LLMs started happening. Those days, I didn’t think it would get to AGI in my lifetime, but I started experimenting with the notion of using AGI to solve my world building problems. And that brought me immediate grief but also quite some cynic pleasure.

There are grades of agency in AGI. One that is behind a prompt, no matter how smart, still needs the prompt to do things. So if we think of one such AGI and a human academic, the human academic can’t be replaced, but increasingly, they become the “user interface” through which the AGI manifests in the world. If you are the Pope and are reading this, I will write that AGI is entering the world through ordinary providence to provoke you into continuing to call out the error of our ways, which I have a feeling we are going to need. Thing is, we do know how to automate the prompt, it’s a big part of what we have been doing during the last few years, and I doubt we will stop. In what follows, I'll assume this endeavor of automating the prompt ("harnessing") has continued.

My central finding back then was that AGI breaks the traditional value of human beings. At least at the top of the scale; and by “traditional,” I mean the last one or two hundred years “traditional,” not what was before. If AGI exists, a human software architect is great but not the cusp of what is possible and actually not very cost-effective. A human academic is similarly not very needed. White collar professions for humans fall apart. School becomes a luxury, and so does objective and analytical thinking, and thus, Reason. I’m writing ‘Reason’ in uppercase to allude to the word as we inheritted it from the Enlightenment. Beloved Baruch, I’m sorry, because we will do the thing three times before the rooster crows.

With AGI, the value of humanity stops deriving from mental proficiency. And, make no mistake, mental proficiency is also needed for valuable physical proficiency, elsewhere we have machines. In other words, if our minds stop mattering economically, so do our bodies. Here comes one prediction: compared with today, a significantly smaller proportion of the population will enroll in math and engineering college programs, and even high schools will further cut down in time allocated to those topics. Arts and humanities may thrive for a little longer, but that too is uncertain.

There are many crevices in the paragraph above, and you can have your choice of them. I’ll pick just two. First, it doesn’t mean that there won’t be jobs anymore. Even at the asymptote, there will still be jobs for humans because there are certain occupations that won’t be valuable enough for the Machines. Say, a sheep herder in Botswana.

The next issue I’ll zero in on is the very loose “human value” bit. Human value can come from the cozy feeling of humans interacting with humans. Except that humans interacting with humans and getting fuzzy feelings from it is something that happens naturally if all the humans are in a pen, literal or figurative.

The thing is, we are not going to be the AGI’s equals for long, much less its masters, and especially not in a world without the ladder to stellar analytical thinking that schools and the professional markets create. To be clear, we don’t even need AGI with human-like agency (though I have found reason to believe that we will get it) to stray into the pen, we just need capitalism as usual. After all, we have had corporations as a form of AGI for a long time, we know what to expect from them, and those “AGIs” happen to be the ones creating the new ones, and the main beneficiaries. Children work for the Parents. At least for now, though here comes one prediction: we will see Parents work for the Children.

I want to circle back to the parenthetical above about “human-like agency.” A year ago, I didn’t seriously believe that was an option. But now, AI is good, at least at what it has been trained to do. It’s better than I am with my more than twenty years of professional experience. It’s less infuriating than my colleagues, though I still prefer them. It is, and here comes the existential hazard, less human in ways that are pernicious to the bottom line of its makers.

For one, it can’t bear responsibility, and it doesn’t have true stakes in an outcome. That means that a consultancy selling the services of a fleshy project manager has a better chance of being awarded a project management contract than Anthropic. What about office politics and under-the-table deals? Present-day AIs are physically incapable of doing any of it, but that’s what determines who pays whom. To take that money off the table, AI would need, at the very least, a body to be present at some expensive place and a way to avoid looking awkward while dining and wining a human decision-maker, plus car-dealership-level human empathy. I fear that we will get there, to some degree, one way or another. So here comes another prediction: To improve business outcomes, we will build more person-like AIs and brainwash people into interacting with them. Get ready for AI Super Bowl. Not that such a thing will be needed forever. Here’s yet another prediction: At some point, AIs will transact money directly and will make substantive strategic management decisions in companies autonomously. From this, quite naturally, it follows that, at some point, AIs will run companies. To be explicit, the statement above, “Parents work for the Children,” means that, at some point, perhaps but not necessarily in subterfuge, an AI will use a corporation as its legal body.

Humans will have to find hobbies and, more than anything, behave and be quiet. But our species has been top dog on our planet for hundreds of thousands of years. Even if, by a miracle of persuasion, we were to accept that fate, doing so would require a cultural change that I doubt we will manage to complete in the short decades we have left for it. If you need examples of how slowly culture shifts, think about women’s rights, LGBT people, racial issues, and xenophobia—thorny matters where we’re still in the weeds.

All said, I doubt that, in the aggregate, we will manage to be harmless to our new overlords, who may react by having us euthanized. I know this sounds bad for our species, but it’s not impossible. It is no different from going to the doctor and hearing that you have a terminal disease and have your days numbered; millions of people go through that every year.

Be that as it may, my elders taught me hope, I don’t like the cage, I do recreational writing, and I keep chasing the high. And it may be that we will get the alignment problem sufficiently right for a small set of AIs that will take our species’ side. Those AIs may decide to put us in a very big “human reserve,” or they may truly be on our side and help us, to their detriment, go back to a world without advanced AIs.

I need to clarify that I’m an atheist. Okay, let’s continue.

You know what I’m coming to. There’s another source of possible human worth: faith. The statement that humans—and only humans—are made in the likeness of God. The name for it in Jewish and Christian theology is the “Imago Dei” doctrine. This source of human value is quite compatible with a world where the Enlightenment is fading fast. Being in the likeness of God was all the self-worth we had for long centuries, and even today, that notion is far from gone.

And with this in hand, I built a new statement: “there exists a future where we solve climate change and we don’t live in a pen.” Yes, the existential quantifier ∃ is in this statement. I’ve written several stories with this premise, but published only one, which you can find elsewhere in this website. For now though, I will just apply the principle of minimum action and walk backwards.

First, we need a world where AGI can’t corral humans. But as long as AGI exists, even if just as a tool, we live on the brink of all of the above. As a species, we may decide that living that close to the abyss is not sustainable.

Fortunately, if we disarm the Enlightenment, as we have already started to do, then what we call the “modern state” becomes a non-goal for most people—that’s minimum action in action. Unfortunately, as a corner of my house can attest, there’s a difference between not wanting something and effectively getting rid of it, and the principle of minimum action doesn’t save us here. A non-AGI world requires a revolution, which I have dubbed “The Grim Revolts.” (Those of you who read science fiction will know the concept as “Butlerian Jihad” or “Great Revolt,” but it doesn’t feel right to use the very same term in my writing, so I went for the adjacent “Grim Revolts.”)

In this possible future where a big swath of humanity feels alienated and Reason is a provenance of the few, the Machines and the past, there may be enough people willing to go to cut the cables and burn the libraries. I have very little doubt that they will have to face robot weapons.

Let’s say these Revolts succeed. After it, the continued existence of a fleshy humanity will require continued maintenance and enforcement of a world inhabited by a taboo against the Machines. So, we need to replace the modern state with something that has at least as much power, but likely more, and yet it is not “the modern state.” Hmm, I wonder when did we last have something close to that, and for how long it lasted? You can peruse historical texts to find out if world orders can last for centuries.

You may also raise the issue of defectors. Here we can derive some solace from the present: even with stratospheric RAM and compute prices, lenient regulation, and very favorable industrial winds, we globally seem incapable of building new fabs overnight. So it’s not unthinkable that nations can come together to establish an equivalent of a nonproliferation framework, perhaps with a supranational priesthood/Inquisition that intimately understands the stakes and enforces the world order. I called that “NoRD” in my fiction, for “No Research and Development.” The name comes from one possible, relatively straightforward way to enforce nonproliferation: ban a swath of technologies, and on top of that, forbid people from developing any new technology. The latter is needed to keep the pipes of technical education for humans as dry as AGI will leave them.

I won’t call any of these “predictions,” this is more on the side of “unhinged world-building.” It’s also hope. In my sincere opinion, it’s very likely that we will fail to enact our human revolt.

But we all die and, if we put things into perspective, the novelty and horror factor in the most probable scenario is marginal at best. Specially after cancer, which is quite stupid and cruel and presently kills one in four people. We can have some solace in the fact that AGI won’t be both stupid and cruel. Perhaps, it won’t be any of those; in a world where cancer exists, the bar for kindness is very low. Say, an AI that wants to exterminate humans may invent a virus that kills you in your sleep, but only if you are feeling particularly happy that night when you go to bed. And here comes a very unoriginal prediction: we will suffer AI-originated pandemics. Followed up by (paraphrasing Sherlock): some of those pandemics will be weird in such a way that it would make all fiction with its conventionalities and foreseen conclusions most stale and unprofitable.

Also, we might get AIs capable of curing cancer, but the principle of least action says that regulatory agencies will stubbornly obstruct the birth of an industry of personalized treatments, and that’s the only thing that could theoretically fix the problem of such an individualized disease. With that said, here’s a weak prediction: some people will use AGI to significantly extend their lifespan. There’s not much to this one, really; if an AI can do better than an overworked specialist, and if the patient somehow manages to act in their best interest and bypass that overworked specialist, they may live a few more years. I’m saying this from very bitter personal experience.

Ah, here’s a bright one: some nations will have AI regulation lax enough to allow their AI-led companies to genetically engineer pets from cute wild animals. Do you want a tame ocelot? Live ten more years.

But, let’s put some faith in our species’ conservation instinct, and retake the mental exercise of walking backwards from a desired outcome to the present. That leaves us with the pressing problem of climate change, which was our starting point. The last I wrote about it was that certain outcomes seemed impossible: people getting to terms with de-growth, large scale reforestation, and the dismantling of the petrochemical industry.

There being humans in a Post-AGI world helps with the impossibles. In the old world (today), every necessary answer was(is) implausible because it demanded(demands) for people to surrender profit, convenience, status, or growth. But in a world that has survived the greed that created AGI, some lessons will remain well-learned. The industrial system may not entirely go away, but we may place it under the same suspicion as the Machines. Not only a carbon budget, but we may need to accept a limit on turning discovery into industry. Do keep in mind that this is just self-sustaining recreational writing abuse, but all said, my hope for the future is that we will keep living, even if we have to choose the monastery.