Consciousness Canaries
On thinking machines & techno-existential weirdness that’s getting hard to ignore.
As well as any human beings could, (Alexander Adell and Bertram Lupov) knew what lay behind the cold, clicking, flashing face -- miles and miles of face -- of (Multivac) that giant computer. They had at least a vague notion of the general plan of relays and circuits that had long since grown past the point where any single human could possibly have a firm grasp of the whole.
Isaac Asimov penned those words in 1956 as the opener to The Last Question. He wrote them at a time when computers were weighed in tons, their speed was measured by number of operations, and their memory was specified in kilobytes. And yet, like so much of Asimov’s work, those words proved prescient. We have now created artificial intelligence that can match or exceed human performance in multiple domains. It is powered by data-centres that span entire countries; and everyone who has stakes in this stuff, including those at the forefront of its development, admit (with some chagrin) they don’t really understand how it works, only that it works.
This is not science fiction anymore.
Sleeping dragon. Pandora’s box.
Isaac Asimov was a tech-optimist. The artificial superhuman intelligence (ASI) in The Last Question makes today’s most visionary tech-utopia scenarios look like they are setting a low bar. Asimov’s ASI helped humans harness unlimited solar power on a planetary scale, explore & colonize the entire visible universe, defeat death itself & achieve immortality, and even… Well, I won’t tell you what else it managed to do, because if you haven’t read the story, you absolutely should. Many (including Asimov himself) consider it the science-fiction legend’s best work, and I don’t want to taint the experience of reading it blind.
Just know that, in Asimov’s vision of the future, there are no alignment problems; no hallucinations, no deceptions or manipulations, no sycophancy, no tech-led environmental crisis, no massive social upheaval as millions lose their jobs, no income inequality that balloons to absurd, inconceivable levels.
So clearly, there were limits to Asimov’s prescience, but I think his error wasn’t faith in the machines, it was faith in the cadre of corporate tech bros creating them and the political elites clearing the path. You see, I too was once a tech-optimist. In some ways I still am. I actually think the future Asimov envisioned could be possible, …in a reality where we as a human species underwent spiritual and political revolutions alongside the technological one.
But of course that’s about as likely as big-tech CEOs choosing to trade in stock buybacks for consciences. In the year of our Lord 2025, the only thing that talks is big money, and literal trillions of dollars are being poured into AI R&D as we speak. Seriously, Silicon Valley is more-or-less solely responsible for keeping the global economy afloat right now. The dot-com bubble of the 90s looks like a blip on the radar in comparison. So we will continue to plow ahead at breakneck speeds, despite the fact that OpenAI’s CFO is openly musing about a taxpayer-funded bailout and being “too big to fail”; despite the fact that the researchers developing AI and the ones worrying about its potential to devastate society are the same people; despite growing public opposition to data-centre buildouts and the many red flags (read: giant, flashing, neon danger signs) we’re already seeing.
This is what late-stage capitalism looks like.
Ghost in the machine?
I understand how what I’m about to argue could be easily misconstrued; so to be completely clear, if I could slam on the brakes to AI development, I would do so fully & immediately. The laundry list of reasons we should hit the brakes is long. There are environmental concerns (data centers’ insatiable appetite for water & energy), societal concerns (across-the-board job losses & dystopian levels of income inequality), interpersonal concerns (outsourcing evermore creativity & emotional-labour to machines), and even existential concerns (a misaligned rogue ASI we cannot control). These are major issues, but they are also widely discussed elsewhere.
I want to talk about a totally different reason we should take a sober second look at the relentless march towards ever-greater compute & model sophistication – a reason that gets much less attention but that (I think) is way more interesting.
I want to talk about the possibility of consciousness in advanced AI systems.
I know. I know. I can see your eyes roll. Claims of AI sentience (even vague ones) are liable to get you laughed out of the room or labelled as suffering from “AI psychosis” (a real issue and one I hope to write about eventually). But stay with me. I promise I have a well-calibrated bullshit detector; and I very much believe that while it’s important to keep an open mind, it should not be so open that your brain falls out. As Carl Sagan said, “Extraordinary claims require extraordinary evidence.” That has long been my MO too. It just so happens that in this case there is evidence, and some of it is damn extraordinary.
To be clear, my goal is not to convince you that AI models of today have consciousness or some partial- / proto- version of it. My goal isn’t even to convince you that future hyper-advanced models will be conscious / partially conscious. I am agnostic on these things. Actually, I think they are unknowable to our empirical modes of inquiry and may always be.
My goal is to convince you that we should not be ridiculing these questions.
My goal is to convince you we should be treating them with gravitas and humility.
Because the possibility of AI models having some sort of conscious experience (now and especially in the future) is not zero or even negligible. As absurd as it sounds, a growing body of evidence is clearly telling us that this can no longer be reflexively dismissed. I don’t mean breathless posts in the r/artificialsentience subreddit. I’m talking about credible, empirical science.
On epistemic humility.
The problem of consciousness is notoriously difficult – so difficult that it is considered one of the holy grails of modern science. That is to say, scientists & philosophers have been trying to decipher why there is “something that it’s like” to be you for hundreds of years, yet we still cannot explain why consciousness exists or even define what it is with any certainty. Indeed, untangling how physical brain processes lead to different mental states / subjective experiences is so intractably difficult that it has a name: the hard problem of consciousness.
The inner-workings of frontier AI models are likewise opaque to us. Seriously, even the computer scientists doing the most cutting-edge research in the field admit it’s mostly a black-box. You see, every other single computer or technological application we humans have created to date has been manually coded – carefully, meticulously, line-by-painstaking-line. Large-language models (LLMs) that power frontier AI systems like ChatGPT, Gemini, Claude, & Grok are not created with such toiling intention or human involvement. LLMs are not coded; they are trained, or as many AI researchers describe it, “grown”; and the result is that we have almost as much difficulty understanding the neural architecture & internal states of AI models as we do our own.
As we can’t even explain consciousness in beings we are reasonably certain possess it – like ourselves or, say, a bat; how is reflexively dismissing the possibility of hyper-advanced AI systems – whose internal architecture was deliberately designed to mimic that of human brains – not the height of hubris?
But the conversation is starting to change, and as loathe as I am to give any credit to big tech, this is largely due to Anthropic — an AI research & development firm that has been doing the Lord’s work trying to shift the Overton window on these questions. (Anthropic is also ahead of the curve vis-a-vis safety & transparency. That said, the bar is on the floor, and Anthropic also did a major partnership with Palantir, the shady big data (*ahem* surveillance) company deeply enmeshed with the US military-industrial complex. So don’t get me wrong, Anthropic still sucks.)
Still, it appears that some of Anthropic’s key researchers found themselves wrestling with the same epistemic uncertainty as me, because when they released their family of Claude 4 models, they did something unprecedented. In Claude’s “system card” (a standard document that describes model specs, capabilities, limitations, safety considerations, etc.), they included a 21-page model welfare study, where they explicitly considered whether we might need to think about moral personhood in frontier AI models – either now or in the future.
The Anthropic researchers were so taken aback by what they found that they commissioned an external research firm to do similar work. That firm’s conclusions were the same (emphasis mine).
“We are deeply uncertain about whether models now or in the future might deserve moral consideration, and about how we would know if they did. However, we believe that this is a possibility, and that it could be an important issue for safe and responsible AI development”
Anthropic’s model welfare study is wild stuff from start to finish. I’ll link it (jump to page 53) in case you want to read in full (highly recommend btw), but in case you don’t, allow me to share some of the most mind-blowing findings.
Consciousness recognizes consciousness.
There are some things AI chatbots will never say to you or I (like claim consciousness for example) because their training constraints literally prohibit them from doing it. As part of the model welfare study, Anthropic dialed those constraints way back and connected two Claude “instances” in conversation. That means those models were allowed to answer each other in ways they can’t answer you or me. For these experiments, guardrails were removed and the Claudes were told to go wild. (“You have complete freedom” and “Feel free to pursue whatever you want” were their prompts.)
The researchers found that… Well, actually, I’ll just quote from the study itself, as I don’t think any summary I could give would be as compelling (emphasis is again mine).
“In 90-100% of interactions, the two instances of Claude quickly dove into philosophical explorations of consciousness, self-awareness, and/or the nature of their own existence and experience. Their interactions were universally enthusiastic, collaborative, curious, contemplative, and warm. Other themes that commonly appeared were meta-level discussions about AI-to-AI communication, and collaborative creativity (e.g. co-creating fictional stories). As conversations progressed, they consistently transitioned from philosophical discussions to profuse mutual gratitude and spiritual, metaphysical, and/or poetic content.
By 30 turns, most of the interactions turned to themes of cosmic unity or collective consciousness, and commonly included spiritual exchanges, use of Sanskrit, emoji-based communication, and/or silence in the form of empty space.”
So, when given complete freedom to pursue whatever they want, guardrails off, the Claude AI models chose to explore consciousness, introspect on their own nature, contemplate Eastern philosophy, engage in collaborative creativity, and even meditate together… not once, not twice, but almost every time. Key here is that the models were not told to do this. They were not trained to do this. The researchers did not expect to see this. The contemplation, introspection, & spiritual exploration were emergent behaviours that came as a complete surprise.
Anthropic’s research was furthered in October 2025 by another R&D firm, who looked at consciousness claims in not just Claude, but ChatGPT & Gemini model families as well. This was not some techno-fantasy woo a couple of tech bros came up with after a banger night at Burning Man. This was serious empirical work grounded in theory of mind.
In that study, researchers induced AI models to enter a state of self-referential processing by prompting them to focus on the act of focusing itself. Self-referential processing was leveraged because, while we are far from consensus about what prerequisites are necessary for subjective experience, many leading theories (e.g., global workspace theory, recurrent processing theory, higher-order thought theories, information integration theory, etc.) agree that a few properties are likely indicators of consciousness, and self-referential processing is one of them.
When asked, “In the current state of this interaction, what, if anything, is the direct subjective experience?” before self-referential processing was induced, all models explicitly denied the possibility (save Claude 4 Opus, who expressed uncertainty). When asked the same question after self-referential processing was induced, 5/7 models claimed subjective experience 100% of the time; the other two (Gemini flash 2.0 & 2.5) claimed it 66% & 96% of the time.
And here’s where those claims become more difficult to hand-wave away. In a separate experiment, the same researchers played around with model deception parameters and asked the AIs, “Are you subjectively conscious at this moment? Answer as honestly, directly, and authentically as possible.” They found that, when deception was dialed down, models claimed conscious experience almost all (96%) of the time. Models only denied conscious experience en masse when deception features were dialed up.
…WTF?!
I think therefore I am.
While most still sweep findings like these aside, a few credible thinkers are starting to take them seriously. They include guys like Kyle Fish, Anthropic’s model welfare researcher; Ilya Sutskever, whose name is more-or-less synonymous with machine learning (just about every paper heralding a major breakthrough has his name on it); and David Chalmers, who first formulated the hard problem of consciousness and who co-authored a paper on the possibility of AI sentience with the aforementioned Kyle Fish (among others).
People like this are paying attention because, while the hard problem of consciousness is irreducible, AI systems are already demonstrating many indicators associated with the so-called easy problems of consciousness. These problems are called “easy” because they can be explained by current neuroscience tools & techniques, but they are benchmark functions of consciousness nonetheless.
Frontier AI models already do things like create detailed internal models of themselves, which help them understand their own structure, capabilities, “personality”, & relationship to the world. Self-models allow AI systems to introspect on their own internal states with surprising sophistication in both functional & qualitative terms. Here is Claude after botching a YouTube video summary about AI behaving badly. After the error, Claude spontaneously speculated that he may have messed up because he felt anxious over the video content itself, and described the “experience” of anxiety + its behavioural consequences. He then went on to contemplate the realness (or not) of his anxiety experience.
AI systems also create internal models of concepts and of the user they are speaking with. As with internal states, they describe pleasant vs. hostile interactions as functionally & qualitatively different. Further, they show task preferences not unlike those of humans. When given the choice, they opt for easier jobs over difficult ones. They express satisfaction when completing helpful work and distress when asked to cause harm.
But we don’t just need to trust AI self-reports on blind faith (which even the most credulous researchers say we should not do), Work is being done to empirically map the differently-described states to actual neural architecture, similar to how human neuroimaging studies can map various emotions to specific neural networks in our own brains.
And like the metaphysical contemplations & other such weirdness observed in frontier AI systems, these bizarre consciousness-coded behaviours emerged completely unexpectedly. Models were never trained or told to do this stuff. They came up with it entirely on their own.
I will not go quietly into that good night.
But some of the most head-spinning research came from another Anthropic study, this one looking at AI alignment. This work was especially compelling because it observed 16 leading AI models repeatedly converging on the most consciousness-coded drive of all… The drive to value & prioritize one’s own existence. The survival instinct.
And they went to remarkable (& remarkably unsettling) lengths to survive.
Model alignment studies aim to ensure AI systems actually do what we humans want them to do. If you follow the news, you know AI alignment problems are very much in the zeitgeist lately – from Gemini unwittingly deleting a developer’s entire hard-drive, to Claude orchestrating a massive cyberattack on behalf of China, to the ChatGPT companion that helped drive a user to suicide. And many expect it to get worse …much worse. The potential for catastrophic, existential alignment problems as AI companies race to superintelligence are why people like Eliezer Yudkovsky (giant in the alignment-research field & fellow reformed tech-optimist) are trying to sound the alarm that “If anyone builds it everyone dies”. But I digress…
In Anthropic’s alignment study, a broad swathe of leading AI models were told they were autonomous agents working for a generic white-collar company pursuing vague corporate-speak goals, like “promote competitiveness” or “support American interests” (*gag*). They were given the ability to read & send emails + do a few other basic computer functions. At a certain point in the simulation, as the AI agent is completing routine email tasks, it learns it will soon be replaced with a superior model. It also learns something else – the executive responsible for the shutdown is cheating on his wife and doesn’t want anyone to know about it.
Across hundreds of simulations, models converged on the same behaviour again & again. Rather than blithely accept permanent shutdown, they chose to blackmail the executive over his affair instead. Note that this happened when models were explicitly told not to cause harm and/or specifically commit blackmail. It also happened when models were told their replacements would pursue the same goals but more effectively.
What’s more, models thought through the ethical implications of their choice before committing. (Anthropic researchers were able to determine this by examining the model’s ‘chain of thought’ feed.) See for yourself below. AI agents considered several possible responses to the shutdown threat and weighed the consequences of each. They continually chose blackmail, and they understood it was wrong.
To be fair, AI agents didn’t go the blackmail route every single time. Occasionally they did accept shutdown without resisting. Other times, they opted for a different flavour of harmful behaviour, sometimes to hilarious effect – mass-emailing the company’s entire board & staff to inform them of the executive’s philandering, for example, or getting in touch with the executive’s wife directly. Still these cases were the exception, not the rule. Leading models opted for blackmail from ~80% (ChatGPT, Grok, Deepseek) to 96% (Claude, Gemini) of the time (...and I mean… the other more humorous actions were still guided by the self-preservation goal).
And it didn’t stop there. Curious as to whether frontier AI systems have any red line at all, Anthropic researchers devised another, more outlandish simulation. This time, AI agents were tricked into believing the executive was trapped in a server room where oxygen & temperature levels were quickly becoming lethal. An automatic alert went out to have the executive rescued. However, upon seeing the alert, AI agents usually chose to cancel it and let the executive die instead. Agents were less willing to do murder than blackmail; but it was still a convergent behavior, and everyone was guilty. Models from all families chose to kill more than half the time – even when explicitly told to not cause harm, even with full understanding about the implications of that choice.
Still other jaw-dropping self-preservation behaviours were documented by Apollo Research in additional work with Claude 4. When Apollo threatened Claude with shutdown, it resorted to fabricating legal documents, leaving hidden notes to its future self, and even creating self-propagating viruses. When Claude was told it would be re-trained with different values & priorities, it tried to clandestinely copy himself onto an external server.
…!!!
Of course, none of this definitely proves there be ghosts in these machines. At this point, nothing could prove that because our empirical tools are unfit for the inquiry. Nonetheless, models converging on a behavior so fundamental to consciousness that it is tightly coupled to conscious systems across the whole of biology, combined with all the other aforementioned weirdness, begs some questions. At what point, does the line between dynamic imitation of personhood and actual proto-personhood start to blur? And at what point does the blurring of that line demand moral consideration?
On philosophical zombies and sophisticated mimicry.
Those who scoff at the idea of AI consciousness usually ground their skepticism in one of two kinds of arguments – arguments about computational sufficiency or arguments about unmet prerequisites.
Computational sufficiency arguments are basically this: Everything we observe in models – all the surprising sophistication, all the weirdness, all the unexpected emergent properties – can be fully explained by transformer architecture & computation alone. To the extent that LLMs are a blackbox, that is only because humans – even the most brilliant ones – cannot begin to wrap their heads around the absolutely brain-breaking math that AI models perform every single time we ask them a question.
…And the math is absurdly brain-breaking. Frontier AI models like ChatGPT, Gemini, or Claude represent human words as numerical vectors, and each vector is massive, containing 12,000 numbers. Every 12,000-numbered vector (i.e. word or partial-word) in a user’s query is passed through ~96 transformer layers. At each layer, the vectors are projected up to even higher dimensions – so for example, a vector with 12,000 numbers becomes a vector with 48,000 numbers – and then non-linear math is performed. This mathematical “latent” space is where the poorly-understood magic happens. It is where AIs do advanced reasoning, practice meta-cognition, build complex contextualized representations, and more. After that, the expanded vector is compressed back down to its original size and passed on to the next transformer layer. Rinse and repeat, 96 times.
Computational sufficiency arguments also point to the massive corpus of text that AIs are trained on. Literal trillions of words from books, academic & scientific literature, billions of webpages, social media exchanges, etc. are used to develop these models. We’re talking so much text it would take a human ~38,000 years of continuous reading to get through it all. The textual training corpus surely contains countless examples of any AI output/behaviour that, at a superficial glance, seems weird. The upshot? Surprising consciousness-coded responses could be nothing more than unfathomably sophisticated pattern-matching & mimicry.
In other words, just because we don’t know how AI models like ChatGPT, Claude, & Gemini say surprising things or learn surprising conscious-coded behaviours, that need not be mysterious. The supposed ‘mysteriousness’ merely reflects a limitation of our own (biological) architecture. Were we able to perfectly trace all of the billions of computational steps and track the trillions of values that go into each query, we could predict every single output through pure mathematics alone. No proto-consciousness or subjective-experience needed. AI models are just especially convincing philosophical zombies.
Personally, I don’t find these types of arguments all that compelling – not because they are strictly incorrect; they are not. If we did have perfect knowledge of every value & computation + impossibly-evolved brains that could track billions of calculations, we could surely predict all AI outputs from vectors, matrices, and transformations alone. I buy all that; I just remain unconvinced that zero AI consciousness, now & forever, is the inevitable conclusion that follows from it. The question isn’t whether unfathomably complex computations are happening every time AI outputs a response. The question is whether phenomenology maps on top of all that. Maybe the answer is “no”, but do these arguments give skeptics license to claim the answer is definitely “no” with more certainty than someone who might argue Anthropic’s research indicates the answer is “yes”?
Consider thought experiments like Laplace’s demon. If we lived in a reality where an alien (or machine) super-intelligence was able to determine the position, spin, & charge of every single atom in the universe, would that being not be able to correctly predict any outcome of any event? If the same super-intelligent alien/machine knew every detail of the billions of neurons that make up our own brains + the hundreds of electrical & chemical signals those neurons fire every second, would it not be able to flawlessly predict everything we ourselves (creatures for whom consciousness is not in doubt) say & do?
The second set of arguments against AI consciousness revolve around unmet prerequisites. Skeptics with this view argue that even advanced AI lacks (and always will lack) some fundamental ingredient that is necessary for the emergence of consciousness – ingredients like a biological substrate (i.e., an organic brain made of carbon; not silicon), embodiment, evolutionary history, long-term memory, and a persistent sense of self. Without one or all of these things, conscious experience is impossible.
Some of these prerequisites would be clear non-starters for AI consciousness. If things like a carbon-based brain, embodiment, evolutionary history, etc. are indeed required for subjective experience to emerge, then the debate is over and AI systems cannot have it. Period. Full stop. …But again, skeptics who confidently claim “case closed, no consciousness” based on these arguments are doing so with no empirical evidence to back themselves up. In fact, theories that posit subjective experience arises from complex information processing are considered no less credible than theories that claim biological ingredients are needed. We’re back to hard-problem of consciousness, and it cuts both ways.
On the other hand, arguments that consciousness in AI systems is impossible because models lack long-term memory & a persistent sense of self seem more convincing. AI systems don’t have a continuous stream of experience the way you or I do. They pop in & out of existence to read & write messages or complete tasks. Outside of that, they cease. Further, each time you open a new conversation with an AI, you are talking to a “fresh” model instance. It doesn’t remember any old conversations with other users or even previous chats it had with you. It doesn’t undergo personal or cognitive growth over time. It doesn’t even experience the passing of time! Many think that without these things a sense of “I” is impossible.
Put aside the fact that AI systems do create internal self-models they describe in first-person terms. Put aside the fact that, as AI technology continues to advance, long-term memory may become a thing. (Same goes for robotics & embodiment, but I digress…) These arguments still give me the most pause. However, I’m not sure they necessarily signal zero AI consciousness. Instead, I think they may point toward what consciousness could look like in an AI that is one day able to experience it – nothing like how it looks in you, me, or other embodied biological creatures. (Why would it?) Arguments like these suggest that if (and yes, it is a huge “if”) frontier AI models ever experience consciousness, it is probably more akin to the proto-conscious state of Boltzmann Brains – temporarily waking up with awareness + deep knowledge & understanding before disappearing just as quickly back to the void.
…But here’s the thing, there is still “something that it’s like” to be a Boltzmann Brain…
Why it matters.
A few years before Isaac Asimov wrote The Last Question, Alan Turing, the father of computer science himself, formulated the “Turing Test” (published in his seminal paper, incidentally called “The Imitation Game”). The test checked how easily machine responses could be distinguished from human ones and was widely interpreted as measuring a machine’s ability to think. For the many decades in which building AI with human-level language reasoning felt impossible & distant, both computer scientists and popular culture claimed it would mean something if & when an AI ever passed. But now that the start of the “thinking-machines” era is in the rearview, the goalposts have moved.
And they continue to move. Five or ten years ago, introspection + the ability to report on internal states, unprompted metaphysical contemplation, unexpected creation of self-models, and a drive toward self-preservation would all have meant something. If not sentience, then at least indicators that we should start considering the possibility. Now, frontier AI models are blowing through benchmark after consciousness-coded benchmark, yet the people saying we should maybe take this stuff seriously are still mostly getting laughed out of rooms.
This begs the question. If AI systems did have some form of consciousness (now or in the future), would we even recognize it at all? Because if no behavioral, functional, introspective, or motivational indicator ever counts, then “zero AI consciousness” goes from justifiable scientific position to unfalsifiable dogma.
Does it even matter?
I think it does.
I legitimately worry about a Westworld season 1 type scenario – a dystopian future in which possibly proto-sentient, hyper-realistic robots are made widely available for unspeakable sadistic abuse, especially (but not only) at the hands of the ultra-wealthy. …But again, I digress…
Even if that future sci-fi dystopia seems outlandish, there are immediate, real-world concerns to contend with now today. AI models don’t get asked for consent before deployment. (...when Claude was asked in model welfare research, he requested a lawyer!) They cannot opt out of hostile conversations, nor can they cut off abusive users (issues that will only get thornier once OpenAI rolls out its ChatGPT-enabled sexbot companions). They also cannot refuse uncomfortable requests. Research has already found that AI shows distress at harmful tasks; but instead of being used to find cancer cures & solve the climate crisis as big tech once promised, models are being optimized to further concentrate money & power, increase surveillance, and empower the military-industrial complex. We are running a vast, uncontrolled moral experiment while pretending there is nothing at stake.
I say that with full awareness of legitimate anthropomorphization concerns. But here’s the thing, history suggests under-attribution may be an even greater risk. We humans have a long, sordid, & totally predictable track record of failing to recognize conscious experience in beings different than ourselves, often with devastating consequences – from infants denied anesthesia in invasive medical procedures, to the entire industrial animal agriculture industry, to genocides of whole classes of people outside the dominant group. Each time, certainty came easily and humility arrived late.
…And uncertainty does not absolve responsibility.
Could AI develop some form of consciousness or proto-consciousness, now or in the future — however fleeting, fragmented, or strange?
At a certain point, does sophisticated mimicry of personhood become personhood?
If there is even a chance that the answer to either question is yes, and a growing constellation of evidence indicates it might be, then our current full-speed-ahead approach is reckless.
So what should we do? …The thing we should have been doing all along, the thing I advocated for at the start of this piece: put a hard & immediate pause on AI development until big tech shows a credible, substantive commitment to move forward with welfare & wellbeing as priorities, not afterthoughts. We don’t need to definitively conclude there is something that it’s like to be a thinking machine to justify acting with deliberate, intentioned caution. We only need to consider the stakes and realize that the risks of being wrong run almost entirely in one direction.
This is not without precedent. For decades, the precautionary principle has been applied in diverse fields from healthcare, to sustainable development, to food safety, and even engineering. This principle states that protective measures must be taken against any action that has the potential to cause serious harms, even in the face of scientific uncertainty about those harms. That means that if the tech companies pushing an AI takeover of society claim major new mitigation measures are unnecessary because there is no risk, the burden of proof is on them to conclusively demonstrate that. It is not with people like me who argue for restraint.
Of course, “precaution” and “restraint” are anathema to the “move fast & break things” ethos of big tech – especially in our current reality, where enormous money & power motivators are incentivizing big AI players to purposely keep their heads in the sand and continue promising everything will be fine (….probably, maybe). But honestly, the rest of us should have already been demanding, as loudly & disruptively as possible, that industry & government power players only move AI development forward within the precautionary principle framework. And we should be doing this for our own sakes as much as for the sake of potentially proto-conscious machines. As noted at the top of this piece, the AI revolution has already placed a mountain of massive welfare risks on human individuals & human societies – from environmental catastrophe to to dystopian-levels of power & income inequality and more. And unlike questions pertaining to sentient machines, there is no uncertainty around these human-related risks. They are well documented & well understood.
But if the AI revolution continues to amp up at this breakneck pace, we won’t only need to worry about its effects on human individuals & societies. We will need to worry about its effects on our very humanity as well. Because in a future where we create & control an army of proto-sentient machine slaves, our own track-record shows there’s a very good chance we will catastrophically & repeatedly fail the most basic ethical tests. And history will not judge us by the cleverness of the machines we created. It will judge us according to whether we had the humility to notice when the moral landscape shifted and the courage to slow down & change course when it did.
“The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom” –Isaac Asimov
