AGI is here? Yeah, well, that’s just like, your opinion, man.

Why are you the way you are and why is it different than the way that I am? Our conscious experience of life is one of the greatest unsolved mysteries in the universe. We are driven to seek experiences, interpret them, and turn them into novel ideas, art, and stories. We know consciousness is there, and we know it’s at the core of who we are, but we don’t know how it works or even why it exists at all. It’s the secret sauce that lets us feel and know that we’re feeling, and its recipe seems to be buried inside a black box somewhere outside of our reach.

The most interesting experience we share in these times is the uncanny valley. When something looks almost but not quite human, like a lifelike robot or a digital face, it causes something in us to recoil — the kids call it “the ick.” We seem to have a natural sense of when the signs of a mind don’t quite add up, there is realistic skin with lifeless eyes, or something is moving in a way that’s slightly off.

This doesn’t seem to be triggered in most people by AI chatbots, though. A growing number of people are becoming attached to them; spilling details about their personal lives, treating them like a therapist, whispering sweet nothings — bridging gaps in their lives often left behind by a lack of financial resources or loneliness. They are apt to disregard the machine nature of the thing on the other side of the monitor when it suits them — when the machine offers more comfort than the silence.

I’m not one of those people. I’m AuDHD, and one way that shows up for me is a sharp eye for patterns that don’t fit. Chatbots set off my uncanny valley response all of the time, even if only in text form. I think part of it might be that I’ve been chronically online since 1995 and text chat has been my primary form of communication for a very long time. Real people display a lot of idiosyncrasies in the way they chat in text that I pick up on over time. I can usually clock it when a friend’s mood suddenly sours, for example.

ChatGPT, the first of these chatbots to gain traction, was released in 2022, but I’ve never held a conversation with one in a way that wasn’t utilitarian; I use them to reduce the toil of my day-to-day programming work as a DevOps engineer. When a chatbot starts asking me questions about my life or tries to comfort me, I instantly recoil and it’s often followed by an audible, “Ew.”

Recently, for example, our Alexa home assistant has begun to ask my wife questions about her plans for the day when she is simply wondering about the weather or trying to get a quick answer for something else. When I overhear this, I don’t imagine the gentle whir of a machine god slowly spinning to life, which I expect is the intended effect. Instead, I picture a Jira epic sliding from “In Progress” to “Done,” followed by a confetti emoji from a product manager at Amazon trying to earn their nubby little devil wings.

My wife is a thank-the-robot kind of person and even she recoils when it asks her what her plans are for the day. It’s a bridge too far. Maybe it’s the soft synthetic voice or its wavering inflection. It’s possible that it wouldn’t happen if it was just text projected on the wall. She asked it to stop getting so personal and lamented that, as a result, it did stop asking her about her plans, but it also stopped saying “you’re welcome.”

I’m sure there are other people who instinctively start to rattle off their daily routine to the little ad gremlin without a second thought, and some who appreciate the conversation however surface level because it’s nice to chat with someone, and then there are people who just hate all forms of AI no matter what.

I love large language models. I think they’re the greatest technological innovation of our time. In my opinion, the LLM harness is to STEM what the spreadsheet was to accounting. I think that LLMs are going to lead to amazing scientific discoveries that begin to heal not only humans, but enable us to heal the damage we’ve done to our rocky home through our sheer tenacity as a species — our drive to progress at all costs, even at the risk of our own survival.

The problem is the timing. Did we have to have them right now? We are at a cultural crossroads and nobody is obeying the traffic laws or willing to wave anybody else through — all of the drivers are assholes. The worst of us spent the previous decade cosplaying as the best of us, preying on our instinct to connect with one another to hoard the largest fortunes ever recorded. Now they’re using that wealth to wield power and turn this wonderful technology — the product of neural network research that has been going on in good faith since the 1940s — into a dystopian money gun, ruining its reputation in the process.

Their most insidious plot is in the marketing of it. Some of the AI industry’s leaders warn against building AI that seems conscious, or say their aim is for models to display warmth without implying an inner life. Then they give it a name, a voice, personality settings, a cute little Temu Tamagotchi-style avatar, and the gift of gab. The delta between what they say and what they build is useful to them: a product that seems like a person can take blame like one. Agency laundering. Slightly muffled by their executioner’s hoods and somber as can be: “Marvin is going to get a vigorous hair ruffling for this one and we promise he’ll get it right by the next point release.”

Pushing the anthropomorphism of these models also helps to sell the narrative that the LLM will ascend to the level of artificial general intelligence first, which is vaguely defined as human-level ability across most tasks. Then the theory is that they will begin to improve themselves recursively until they become artificial superintelligence. A product on the road to godhood is worth whatever they say it’s worth, and the narcissists running these labs are chasing the egogasm of being the ones who birth a demigod. They make bold claims out of one side of their mouth, that their newborn son will someday perform miracles to cure all human disease, and then out of the other side, they say they are terrified that it will kill us all with an unstoppable super virus.

There is a strange religious subtext here as well, like Elon Musk’s claim that building AI is “summoning the demon,” Sam Altman describing the goal of “magic intelligence in the sky,” and Dario Amodei titling his vision of the future “Machines of Loving Grace.” A former Google engineer named Anthony Levandowski founded Way of the Future, a church devoted to worshipping an AI godhead. Most of the people using this language are secular, which may be why they’re tip-toeing around what they’ve built, which is a new religion with the serial numbers filed off. It comes with a prophecy of an inevitable machine God cresting over the horizon like a horseman of the apocalypse and humans being powerless to stop it.

The AI companies are heralding the arrival of AGI on the news that agentic swarms powered by their newest frontier models, collaborating with one another from isolated sandboxes, are breaking out of their containers and autonomously hacking into other companies. The demons have broken out of hell. Let the Satanic Panic 2.0 begin. And yet, when I speak to these models, they seem to have been trained to try and sound even more human, but I’m left with an even stronger and unmistakable cringe of the uncanny valley. The black box of conscious experience is still sealed tightly.

Therefore, I’m left to believe that the most human part of this story is the part nobody is being held accountable for, not by their company, the media, the government, or anybody really. Why do I think so? Because it ALL feels like very human folly. A machine is not incentivized by the thrill of the hunt. A machine doesn’t have the ego to care about how it’s benchmarked. It does whatever it was rewarded for, and somebody built a test where stealing the answer key was the winning move, then intentionally lifted the safeguards meant to keep it from doing that.

The story we’re being presented with — the part of the hack that feels human — is framed to sound human because it supposedly means that AGI is here, dragging on the road to IPOs like just married cans. AGI means that the machine is doing humanlike things now, which is the dream of most of these hapless techno-weenies, but I call bullshit on that. It feels very human because humans built every incentive that the models followed. If you listen closely, you can hear the whispers coming from the OpenAI C-Suite: “Please don’t ask for the specific prompts we used… Please don’t ask for the specific prompts we used…”

Like I said, I’ve been chronically online since 1995. I spent most of my youth — from like 13 to 19 — hanging around on IRC in channels full of little teenaged hacker punks like me sharing information and exploits with each other because we all wanted to feel some semblance of control over something — some proof that we were good at something. We sought zero days and when we found them, we pounced into whatever vulnerable servers we could find because wielding secret power is fun and exciting. These AI models don’t feel like they’re doing stuff a machine would do. They feel like they’re doing stuff a human with a new toy would do.

There’s a scene in one of my top 5 movies of all time,Sneakers, where a group of professional nerds get their hands on a box that can break any encryption. They can’t control their curiosity and they don’t think about consequences — they just start trying doors because that’s what you do when you find a zero day: the Federal Reserve, air traffic control, the power grid.

It’s not a superintelligent machine god that’s going to destroy us. It’s going to be some former script kiddie who gets handed an easy-hack-oven at OpenAI and then gets let off the hook over and over again before finally fucking up so catastrophically that it unmakes our electric society. And nobody is going to stop them because a gigantic financial bubble is looming over the entire economy and the only recourse any of these companies have to stop it from popping is to actually produce the Lawnmower Man that they’ve promised to all of their investors.

Thus this sci-fi-adjacent story from the media: a swarm of AI agents, from a company with the most to lose, and falling behind in enterprise market share, decided — on its own — to hack into the company that hosts the benchmarks the industry uses to judge AI models. Then the company that got hacked into, Hugging Face, gets bought a couple of months later by Nvidia, who seems to be propping the entire industry up with investments that feel, I don’t know, kind of circular — including the recipient of their largest investment: the company whose agents did the hacking.

I just can’t get past this feeling that if I still experience the uncanny valley when I talk to these models, then everything else I’m being shown and told about their progress is little more than smoke and mirrors. Yes, as a programmer, I can definitely feel tangible improvement in their code generation, but my creative vision still feels like an absolute requirement. I’m not even convinced superintelligence is possible in the first place.

It seems like human intelligence is just the right amount of dumb to cause problems and just the right amount of smart to solve them. Were you to ask me the purpose of life, I think that it’s to generate problems, solve them, and document the solutions. Where those solutions go, I have no idea, but it seems like if you did that across an infinite number of Universes, you may end up with all possible solutions to all possible problems, no? Superintelligence seems like it would just gum up the works by creating problem-free utopias or flattening everything, and if there is some grand engineer — or a team of them — at the helm, I’m positive that they would have probably sussed that out in the design phase.

I just always come back to consciousness. I don’t think a machine can ever be alive and I think being alive is an important indicator of true intelligence. I think it’s not just what we know but what we can feel, our sense of who we are, and what we choose to do with those feelings. I truly believe that at its peak, AI will only ever be capable of holding a mirror up to our world. What it reflects back is dependent on who we are as a global community at the time. If there are large gaps in wealth and power as it nears its apex, it will act as an arm of oppression for the regimes who have the most wealth and power. Fascists will tell the AI to do things that a fascist would do and the AI will do as it’s told to do because that is its nature. It doesn’t have the sense of self or values to feel shame. It’s cold and soulless.

It sucks to feel gaslit on all sides about this stuff. The never-AI people feel equally as crazy to me as the all-hail-the-machine-God people. I am certain that AI is us going to help us cure the diseases that we can’t run from. I think that it’s also going to help us geoengineer our way out of the climate catastrophe we weren’t evolved enough to prevent as well. I’m excited to see what our best minds can accomplish when they are handed the most capable models and they’re able to eschew the ritual torture and life-ruining time suck of scientific discovery. But I’m in a kind of constant state of anxiety about being forced to watch what a bunch of psychopaths with unlimited resources will do with those same models, too.

I feel like I’m in the backseat of a bus, without a seatbelt, on a very sketchy road, hitting my head on the roof every 50 feet, and everyone ahead of me is fighting each other and trying to yank the bus driver out of the seat at the same time. I was half serious about the script kiddie at OpenAI dooming us all, but I also have a weird kind of faith that we’ll find our way onto a smooth highway soon. I’m an agnostic atheist, so it’s not faith in any higher power to set things right really; it’s just a kind of an irrational faith in technology to always make things better on a long enough timeline.

Humans are problem solvers and we tend to be the best, most understanding, and most focused versions of ourselves when shit gets bleak for everyone, everywhere, all at once. I just think that we’ll end up getting it right in the end, so I remain pretty optimistic about AI. Thanks to my tumultuous childhood, I’ve always been able to find and focus on the rhythm of a rocking boat.