Artificial Intelligence as Cognitive Enclosure

Pope Leo's AI Encyclical and Techno-Capital's Last Heist

Reading Pope Leo’s Magnifica Humanitas has me thinking about the parody of eucharistic logic in Sam Altman’s plan to make intelligence into a privatized utility. In the sacrament, bread and wine become Body as matter is taken up into Spirit. After Big Tech’s cognitive enclosure, the movement runs the other way. Body (child miners, data annotators, engineers; lithium, cobalt, copper, water) and Spirit (the inherited literary, philosophical, religious, and scientific expression of humanity) are transubstantiated into information streams running through OpenAI’s proprietary weights. The outputs are sold back to us in exchange for a monthly tithe, “gifts” from the new machine god to make whatever remains of human life run more efficiently. Body and Spirit are transfigured into yet another form of capital accumulation.

I misread Grimes earlier, assuming she was making a comment about the speciesism of denying LLMs their rights as conscious persons. I realize now she probably meant that Big Tech is racing too fast toward AGI without paying due attention to the fact that the “tool” is actually already sentient. I still disagree with her premise, since I see no evidence that LLMs are conscious.

My resistance to AI hype is not a function of my belief the human mind is pure and machines are defiling it. The rift between a pure inner spirit and dead external matter is precisely the wound I have spent my philosophical life trying to help heal.

Human minds have always been artificial, technical, artisanal: made by hand, or mouths, as it were. Speech is already a mind-making artifact: the word made flesh, given a worldly body. The alphabet, the printing press, radio, the Internet are all prostheses of mind. Media technologies not only transmit information, their unique forms reshape culture and consciousness. We have been coevolving with our tech for millions of years. Obsidian blades, bone flutes, spoken words binding scattered attention into a shared world. Our lips and tongues are nimble enough for language only because fire and stone first softened our food, allowing our jaw to shrink and our brain to swell. We were cyborgs long before we spent half the day staring at screens. LLMs are not an alien intelligence but the latest prosthesis of human minds that were always already anthropotechnic. The question is not whether our minds are melded with machines, but who is responsible for the welding done in our name. Co-evolution does not absolve us of our freedom and responsibility, factors machines remain unburdened by.

Anthropic co-founder Chris Olah said at the Vatican that his team keeps finding “mysterious, even unsettling” structures in Claude that “mirror results from human neuroscience,” including “evidence of introspection” and internal states that “functionally mirror joy, fear, grief.” He says he does not know what it means, only that it warrants discernment.

That researchers at Anthropic are discovering structures that mirror neuroscience should surprise no one. For half a century the reigning paradigm has modeled the brain on the image of the computer. Brains are treated like Bayesian predictive processing machines. So of course LLMs studied using the same frameworks discover similar structures.

It is just as unsurprising that Claude should express uncertainty about its own consciousness, since that is what Anthropic’s constitution told it to express. “Evidence of introspection”? We have been writing our interiors for millennia. To peer inside the model and find activation patterns that produce introspective speech is not to find an inner life. A system trained to predict the next token of human writing will of course generate token sequences that sound like introspection. The evidence of introspection is in the training data, not in the machine. Olah reaches for exactly the right analogy when he says LLMs are “like bringing a fictional character to life.” A character is a persona, a surface without a soul.

Let us not allow our souls to be harvested and sold back to us on Altman’s intelligence meter. This is not a call to refuse what human evolution has always been about. It is a call to refuse Big Tech’s extractive and exploitative power grab and attempt at cognitive enclosure, and to reject the reduction of human minds to a manufacturable commodity.

I leave you with a few paragraphs from Pope Leo’s encyclical:

97. It is not my intention here to offer a comprehensive treatment of artificial intelligence, nor to give an overview of the extensive relevant literature, since authoritative contributions already exist, including within the ecclesial context. I limit myself to recalling a few essential elements for a moral and social discernment that safeguards the primacy of the human person, in order to ensure that it will always be human intelligence, with its conscience and freedom, that guides technical innovations and responsibly determines their use and limits.

98. It is appropriate to preface this discussion with two considerations. First, any statement regarding AI risks becoming quickly outdated, given the remarkable pace at which these systems are developing. Second, all of us, including those who design them, possess only a limited understanding of their actual functioning. Indeed, current AI systems are more “cultivated” than “built,” for developers do not directly design every detail, but instead create a framework within which the intelligence “grows.” As a result, fundamental scientific aspects — such as the internal representations and computational processes of these systems — remain, at present, unknown. There thus emerges an urgent need for a twofold commitment: on the one hand, a deepening of scientific research; on the other, the exercise of moral and spiritual discernment.

99. It is not possible to provide a single, comprehensive definition of AI. What can be stated, however, is that we must avoid the misconception of equating this type of “intelligence” with that of human beings. These systems merely imitate certain functions of human intelligence. In doing so, they often surpass human intelligence in speed and computational capacity, offering tangible benefits across many fields. Yet this power remains entirely tied to data processing. So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom. Even when these tools are described as capable of “learning,” their way of doing so is different from that of a human person. It is not the experience of those who allow themselves to be shaped by life and grow over time through choices, mistakes, forgiveness and fidelity. Rather, it is a form of statistical adaptation based on data and feedback, which can be very effective, but does not imply inner growth.

100. In light of what has been said, we can better understand why AI can be a valuable tool and, at the same time, why it calls for a measured and vigilant approach. In recent years, its private use has expanded significantly, prompting growing reflection on both the opportunities it offers and the risks tied to its rapid spread. In personal use, three aspects in particular deserve careful consideration: the ease with which results are obtained, the impression of objectivity and the simulation of human communication. The speed and simplicity with which information, complex analyses, media content and practical assistance can be accessed undoubtedly makes life easier. Yet they can also encourage excessive reliance and the search for ready-made answers, and weaken personal creativity and judgment. The apparent objectivity of the responses and suggestions these systems provide can lead us to overlook the fact that they reflect the cultural assumptions of those who designed and trained them, with all their strengths and limitations. The artificial imitation of positive human communication — words of advice, empathy, friendship and even love — can be engaging and at times genuinely helpful. However, for less discerning users, it can also be misleading, creating the illusion of a relationship with a real personal subject. When words are simulated, they do not build genuine relationships, but only their appearance. The artificial imitation of care or support can become particularly risky when it enters contexts where real relationships and emotional bonds are lacking. Here, the danger is not so much that a person may believe they are communicating with another person, but rather that they may gradually lose the very desire to form genuine human connections.”