AI will not gain consciousness, but is becoming society's unconscious
TechFlow Selected TechFlow Selected
AI will not gain consciousness, but is becoming society's unconscious
We can do more and more, yet understand less and less. This is a wake-up call for every practitioner who relies on AI tools.
Navigating Web3 tides with focused insightsAuthor: L. M. Sacasas
Compiled/Translated by: TechFlow
TechFlow Editor's Note: As we outsource more and more decisions to AI, human society is generating an "unconscious layer" detached from active judgment. This is not a sci-fi superintelligence threat, but a more隐蔽 civilizational transformation—we can do more and more, but understand less and less. This is a wake-up call for every practitioner relying on AI tools.
In the early 20th century, the British mathematician and philosopher Whitehead said a frequently quoted line: "We should form the habit of thinking about what we are doing—this is a great wrong-headed old cliché repeated by all textbooks and celebrity speeches." "Quite the contrary," he believed, "the progress of civilization lies in extending the number of important operations we can perform without thinking." He also used an analogy: "Thinking about operations is like a cavalry charge in battle—strictly limited in number, requiring fresh horses, and only to be launched at decisive moments."
You have likely seen this quote. It does convey some truth. But in my view, this statement always seems short-sighted or insufficient. Perhaps because I often see people using this passage to defend indiscriminately outsourcing human cognitive activities to machines, without properly calculating the costs, or even realizing there are costs.
To be fair, Whitehead's passage comes from An Introduction to Mathematics in 1911, and he was actually talking about the advantages of symbols and signs in mathematical calculations—although he said these operations were "without thinking," they were still learned and applied by thinking minds. However, he talked about "civilization," plus that ornate rhetorical ending, probably indeed invites this (mis)reading.
The problem is not whether some thinking automation, or even some externalization of mental processes, is useful in specific cases. Frankly, I think using physical activity rather than mental activity as an analogy yields a more convincing argument. I have long departed from the athletic skills I might have had back in the day, but the lesson remains. Almost every young person learning a sport will frustratingly discover: you must spend a lot of seemingly excessive and painful time, over and over again, executing basic and repetitive training. Wax on, wax off—those who have seen The Karate Kid understand. But what actually happens is precisely the dynamic Whitehead describes. You are automating certain physical movements so you can execute them without thinking. Only then can you exert any creativity or extraordinary skill. Learning to dance, learning an instrument is the same. Automating basic movements is an indispensable foundation for achieving mastery.
But does this dynamic apply equally to all fields of human activity? Are there situations where outsourcing certain activities weakens, rather than promotes, the higher good pursued? Or do "low-order" activities come with certain goods we do not want to lose? Even, like in mathematics, can we always easily separate specific subroutines from a given process or activity? Are there indivisible holistic activities that would disintegrate once attempting to outsource or automate any element? Moreover, as I mentioned before, is there no difference between internally mastered thinking automation and thorough externalization or complete outsourcing of cognitive labor? Returning to the analogy of physical activity: if an athlete imagines using a machine to complete all "trivial" training for him, so he can directly enter the wonderful part of the competition—this person will actually never reach that step. These are crucial questions before we easily endorse those promises of efficiency and liberation made in the name of new technology.
However, my discomfort with this frequently cited ode to cognitive automation by Whitehead, perhaps also stems from its stark contrast with Hannah Arendt's admonition: we should "think about what we are doing." Whitehead wrote nearly half a century earlier than Arendt, I do not think Arendt was alluding to Whitehead. It's just that these two sentences happen to form a confrontational echo, and I think exploring this tension is useful.
"We should form the habit of thinking about what we are doing—this is a great wrong-headed old cliché." —Whitehead
"Therefore my proposal is very simple: nothing more than thinking about what we are doing." —Hannah Arendt
Interestingly, the context in which Arendt's admonition is situated is closer to our current anxieties about technology and human existence. This sentence comes from the preface of The Human Condition, first published in 1958. She explains:
"What I propose in the following is very simple: it is nothing more than to think about what we are doing. 'Thinking what we are doing' is the question that has become the central question of our time. This is obviously a matter of thinking, and thoughtlessness—reckless rashness, desperate confusion, or the complacent repetition of 'truths' that have become trivial and empty—seems to me one of the most prominent features of our time. Therefore my proposal is very simple: nothing more than thinking about what we are doing."
The "latest experiences" that triggered Arendt's research included the launch of artificial satellites and the arrival of automation. Although until recently, one might have thought Arendt's concerns about the impact of automation on labor were misjudged, now it seems her concerns may just have come too early.
When I think about the divergence between Whitehead and Arendt on the question of "do we need more thinking or less thinking," I find it useful to distinguish between two things: on the one hand, establishing a set of unconscious routines to develop higher capabilities; on the other hand, being subject to unconscious forces that unstable drive your behavior, potentially causing destruction.
Is there a tipping point for Whitehead's principle, a flip or reversal? "The progress of civilization," Whitehead said, "lies in extending the number of important operations we can perform without thinking." But what if there is a threshold? Beyond a certain threshold—whether in quantity or quality—will civilization degenerate because of extending the number of important operations humans can perform without thinking? Or at least, will we get a civilization completely different from what we know? We may know the answer soon, because the adoption of AI often means the automation of our thinking, and creating a vast field of unconscious action in the world, these actions stemming from the externalized storage of our personal and collective memories.
Arendt worried that as our technological capabilities (mainly interpreted in mathematical language) surpass our ability to understand them in ordinary language, we might lose the ability to "think and talk about those things we can indeed do." (This suggests the explainability problem existed before large language models and so-called black-box algorithms appeared.) In a passage that has now become relevant and urgent again, she continues, "It is as though our brain—which constitutes the physical, material condition of our thought—cannot keep up with what we do, so that from now on we indeed need artificial machines to think and speak for us." This is a first-order political problem. Under such conditions, we will no longer be self-governing in any meaningful sense. Politics as a realm of human action, constituted by speech, will cease to exist.
On Christmas 1958, the same year Arendt published The Human Condition, W. H. Auden published The Sons of Light, with such thoroughly Arendtian lines:
When we observe
The observing mind that observes itself
Neither worrying nor cold
But thoroughly banal
Though the tools it controls
Can make wishes and counter-wishes come true
It clearly cannot understand
What it can clearly do
Auden and Arendt were friends, and Auden also wrote a positive review for The Human Condition, but I am not sure of the direction of influence.
Although Arendt straightforwardly explains how we arrived at the situation of "clearly unable to understand what we can clearly do"—that is, when our technology-enabled actions cannot be fully understood by our everyday language—there is another interesting framework.
Actions that language cannot penetrate, and thus conscious thinking cannot penetrate, can also be explained through the unconscious. I am not someone who instinctively resorts to unconscious language, but it might be a useful analogy to help understand the relationship between us—whether individually or collectively—and the AI devices increasingly mediating our world experience.
Simply put, I can think of two ways to analogize certain instances of consumer-grade AI as the unconscious. The first is relatively straightforward: as we outsource more and more tasks to agent AI at the personal, organizational, and institutional levels, we generate a layer of activity in the world that is functionally detached from human active judgment and supervision. Moreover, as long as this layer structures and shapes our experience, the ratio of conscious human action to unconscious human action is shrinking.
Obviously, at least since the industrial era (if not earlier), there have been relatively opaque systems operating in the human lifeworld. But I think the difference lies in that these systems were relatively isolated from daily human activities. Whereas AI systems are increasingly deeply woven into our experience. In many cases, these are not only processes imposed on us by external forces, but also processes we release and initiate, they act for us, and react upon us. It is this close intertwining of human and machine that makes me think of the unconscious analogy.
Erik Hoel recently made a similar argument with more elegant pen and longer length. His conclusion is:
"The traditional danger of AI is usually considered to be an existential threat posed by superintelligence. However, this may overlook the real and more subtle danger: the AI revolution is a mechanism that transfers the processes of our civilization from the supervision of consciousness to the unconscious. But when AI removes consciousness from the operation of the world, it makes the world increasingly inexplicable, increasingly strange and difficult to understand. So far, 'the great sludgeification of shared spaces' has confirmed this as the main risk of the large language model revolution. Moreover, as AI systems become smarter, especially if they remain (or are likely to remain) unconscious, then a more significant risk is the decline of the importance of consciousness in culture."
This line of thinking runs parallel to the view I have argued intermittently for ten years: digital technology re-enchants the world in another way. That is, it makes the world mysterious and elusive.
We are confronted by these technologically enchanted objects in a sense that cannot be ignored, and we must deal with them. We seek help from technology, pinning our hopes on their power. We also fear technology, viewing them as the source of trouble. The technological forces we encounter are sometimes benign, but equally often malicious forces, undermining our efforts and derailing our plans.
Technological objects not only have the potential to empower us, sometimes even filling us with wonder. We also view these objects and forces as important factors determining our fortune and misfortune, acting upon us outside our control and beyond our understanding. In other words, we are vulnerable, our autonomy is weakened by the circuits of technological distributed agency, which intersect with our will and desires.
The second way of analogizing AI with the unconscious may be a bit esoteric, please allow me to explain. In Understanding Media, Marshall McLuhan claimed that "with the advent of electrical technology, man extended, or set outside himself, a live model of his own central nervous system." He returned multiple times to this idea that electrical media are an externalization of our nervous system. For example:
"A major characteristic of the electric age is that it establishes a global network that largely has the characteristics of our central nervous system. Our central nervous system is not just an electrical network, it constitutes a single unified field of experience."
McLuhan was not the first to propose this view. As early as 1950, the Catholic priest and paleontologist Teilhard de Chardin claimed that media technology constitutes "creating a true nervous system for humanity" and "an elaborate construction of common consciousness." Through emerging computing and communication technology networks, a new level of reality is emerging, wrapping around the biosphere. This is a technologically mediated, unified layer of human consciousness, which he called the "Noosphere," and described as "an amazing thinking machine."
(I admit this is not my usual way of expression, but let's play with these concepts a bit. In fact, I am already trying to convince you that viewing AI in a psychoanalytic way might be a useful angle.)
Electrical media extended our ability to perceive the world, bringing distant events before our eyes and ears, compressing the time to transmit information to near instant. But since the time of Teilhard de Chardin and McLuhan, something else has emerged: a huge growth in artificial memory or data storage capabilities. So while electrical media extended our nervous system, recently it has also facilitated the collection and storage of unprecedented amounts of information in the form of text, images, video, etc. Our nervous system is supplemented by a digital memory of huge scale. It is this digital memory that is feeding large language models, which now have almost become synonymous with artificial intelligence.
This vast library of human knowledge and culture is basically the internet, or at least the internet is the way it becomes accessible to us. It is for this reason that digital media theorist Gregory Ulmer called the internet "the prosthesis of our collective unconscious" in the early 2000s.
I think Ulmer means this. We have always been social animals, that is, we are intersubjective beings of culture. But most of what shapes us, especially during childhood, usually also fades from our conscious awareness over time. However, the internet, as a prosthesis of this unconscious dimension in our formation process as social beings, makes our collective cultural memory accessible. The internet is a huge network composed of cultural artifacts, symbols, images, texts, and various trivialities that shape our lives, their connection not by linear logic, but by an associative logic almost like dreams, which is very much like the human unconscious in psychoanalytic theory.
Ulmer believed this provides us with a profound opportunity. By making the collective unconscious something we can reflect upon, something accessible to conscious thinking, we thereby have a way to overcome the profound blind spots that are usually at the core of our experience. Needless to say, I have never been too optimistic about this possibility. But however you evaluate Ulmer's optimistic vision—I am greatly simplifying here, but I hope not unjustly—in my view, the emergence of commercialized chat-based AI has indeed led us in a very different direction.
Interestingly, inserting a natural language interface between ourselves and the digitized collective unconscious actually makes it more blurred and difficult to understand for us. The chatbot interface reconstructs the agency of our navigation in the collective unconscious, arguably, it becomes an anti-therapist, leading us away from self-knowledge and insight (no matter how disturbing or surprising) towards a comfortable and soothing encounter. It provides a false clarity, letting us fall into self-satisfaction, protecting us from self-doubt, from staying too long in the awareness of our own ignorance, or staying in disturbing uncertainty. It shields the tangled forest of human experience, lighting up an artificially clear path for us, leading to the promise of knowledge and wisdom. But in doing so, it gently but decisively sinks us back into the unconscious. Perhaps this is the root of AI psychosis.
Regardless of the association with AI psychosis, it is worth thinking about: the madness, listlessness, compulsiveness, aggressiveness, anxiety, and despair in our public and collective existence, to some extent stems from the gradual retreat of human consciousness in the face of a new form of collective unconscious, which re-establishes itself in the form of artificial intelligence.
This non-human action layer in the world was already being constructed before the emergence of agent AI. Algorithmic processes were already constructing elements of human experience, especially in the context of bureaucratic systems.
I am not sure when and by whom the conceptual connection between electricity, nervous systems, and thought was first established, but in his 1851 novel The House of the Seven Gables, Nathaniel Hawthorne wrote: "Then there is electricity, the devil, the angel, the mighty physical force, the omnipresent intelligence!" And: "Is it a fact—or have I dreamt it—that by means of electricity, the world of matter has become a great nerve, vibrating thousands of miles in a breathless space of time? Rather, the round globe is a vast head, a brain, instinct with intelligence! Or, shall we say, it is itself a thought, nothing but a thought, and no longer the substance which we deemed it!"
Join TechFlow official community to stay tuned
Telegram:https://t.me/TechFlowDaily
X (Twitter):https://x.com/TechFlowPost
X (Twitter) EN:https://x.com/BlockFlow_News
