“Counterfeit people“
The dialogue between Paul Atreides and Reverend Mother Gaius Helen Mohiam after the initiation rite emphasis a second and not merely historical reason that led the world of Dune to ban AI: building machines that imitate humans is tantamount to “counterfeiting”. ” ‘Thou shalt not make a machine to counterfeit a human mind’ “, the Reverend Mother replies to young Paul, allowing herself to misquote the sacred text of the Dune universe, the Orange Catholic Bible, which only speaks of “in the likeness of a man’s mind“.
The term “counterfeit“, as opposed to “likeness“, introduces an ethical dimension to the prohibition. In Dune , creating a machine that imitates the human mind means not only reproducing but also degrading the authentic and unique essence of the human soul. Counterfeiting involves deliberate falsification, an attempt to deceive or replace the original. Thinking machines, in fact, are the realisation of something that is not authentically human. They are false versions that try to replicate human capabilities and characteristics without possessing their true essence.
In the fourth novel of the Dune cycle (“God Emperor of Dune“, 1981), creating a thinking machine is a real abomination, as we see in the following dialogue, set 3,500 years after the first novel, between another later reverend mother and Emperor Leto II, son of Paul Atreides:
“We speak of a new Abomination, Lord!” Anteac said.
“You think the Ixians can produce an artificial intelligence?” he asked. “Conscious the way you are conscious?”
“We fear it, Lord,” Anteac said.
“You would have me believe that the Butlerian Jihad survives among the Sisterhood?”
“We do not trust the unknown which can arise from imaginative technology,” Anteac said.
Note that in this dialogue the biblical prohibition against AI gets weaker over the millennia.
The theme of counterfeiting humans brings out analogies with current events. That very risk was highlighted by the recently deceased American philosopher Daniel Dennett. A year ago, Dennett discussed “The Problem with Counterfeit People” in the American magazine The Atlantic: “Companies using AI to generate fake people are committing an immoral act of vandalism, and should be held liable“.
Dennett’s tone is apocalyptic about the possibility of losing our freedom. He concludes that while we may not be enslaved, we will surely be controlled by an elite of powerful people who will rule the world by manipulating information via AI:
“These counterfeit people are the most dangerous artifacts in human history, capable of destroying not just economies but human freedom itself. … Creating counterfeit digital people risks destroying our civilization. Democracy depends on the informed (not misinformed) consent of the governed. By allowing the most economically and politically powerful people, corporations, and governments to control our attention, these systems will control us.…
” ‘Thou shalt not disfigure the soul’ “: an ethical prohibition
At the end of the Butlerian Jihad, the reaction was also to unify religions under a common principle:[1]
Out of those first ecumenical meetings came two major developments: 1. The realization that all religions had at least one common commandment: ‘Thou shalt not disfigure the soul’.
The verb “disfigure” means altering something in a way that ruins its appearance or integrity. In the commandment “‘Thou shalt not disfigure the soul‘” from the Orange Catholic Bible of Dune, the verb “disfigure” refers to the act of corrupting, damaging, or altering the soul in a negative way, degrading and diminishing its spiritual and moral essence.
The warning against the risk that technology — especially thinking machines and artificial intelligence — would corrupt human nature is the principal reason behind the ban on AI in Dune. [2]
This first common commandment to respect the uniqueness of the human soul also explains why the reverend mother, in her initial meeting with Paul, takes the liberty of specifying the commandment “‘Thou shalt not make a machine in the likeness of a man’s mind‘”, more precisely by interpreting the term “likeness” as “counterfeit”.
The Butlerian Jihad, the revolt against thinking machines, is not only a struggle against technology-based oppression of men by other men, it is also a philosophical statement about the value and potential of the human intellect. In an era in which artificial intelligence is often seen as a panacea for all ills, Herbert’s thoughts invite us to reflect on the risks associated with addiction to technology. The words “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them” echoes as a warning against abdicating our critical thinking and autonomy.
This thinking is also reflected in the current debate, although it is characterised by divergent perspectives. On the one hand, the notion that there is a distinctiveness to the human mind compared to an artificial mind is used to support the view that human beings are superior, perhaps derived from a divine soul, which reaffirms traditionalist positions.
On the other hand, we find the position espoused by Shannon Vallor and others that the insistence on using the term artificial intelligence to describe the successes of machine learning creates a false parallel between the capabilities of machines and the capabilities of humans, which obscures the profound qualitative differences between the two. There is a risk that the term becomes
a rhetorical device that eventually diminishes human intelligence as it is forcibly compared to systems that are still very different and actually rather impoverished compared to the human mind.
Defending autonomy
The issue of AI causing loss of freedom goes beyond the problem of enslavement. It should also be understood as implying loss of personal autonomy.
“Why do you test for humans?” he asked.
“To set you free.”
“Free?”
Frank Herbert explores freedom in terms of personal, mental and spiritual autonomy. This broader interpretation of the concept of freedom is exemplified in various characters and through their experiences, particularly the protagonist, Paul Atreides, and the Bene Gesserit mission. When Reverend Mother Gaius Helen Mohiam subjects Paul to the gom jabbar test, she is not simply testing his resistance to physical pain. She is testing his ability to overcome instinctive and animalistic reactions that stem from fear and survival instincts. The test is used to determine whether Paul is able to control his mind and impulses, thus demonstrating his autonomy and ability to exercise self-discipline.
The Bene Gesserit is a religious order that seeks to manipulate and guide human evolution through genetic selection and mental training. For them, the distinction between “human” and “animal” is fundamental. “Animals” act only according to basic instincts, without self-control or understanding. But “humans” have the ability to reflect, choose and control their own impulses.
Autonomy is an essential property for Paul Atreides who, for the Bene Gesserit, is the Kwisatz Haderach i.e. a being with extraordinary mental and spiritual powers with the potential to transcend normal human limits. The Kwisatz Haderach is a figure of immense power and inner conflict, a symbol of human potential reaching its maximum power, and a symbol of the responsibilities and dangers involved with having such power. That makes him not only a saviour, but also a much-feared figure. His abilities can lead to both salvation and destruction.
Machine logic
Human autonomy also implies rejecting purely mechanical thinking, even when this is done by human beings and not machines.
In Appendix II of the first novel, it is said of the religion of Dune :
Then came the Butlerian Jihad — two generations of chaos. The god of machine-logic was overthrown among the masses and a new concept was raised:
“Man may not be replaced.”
This extract further clarifies that machines had not really overpowered humans. The danger of AI also lies in human attitudes towards logical mechanical thinking, the illusory hope among the masses of widespread liberation leading instead to their enslavement. Thinking machines enslave human beings by reducing their autonomy (“selfdom”), not just by making them slaves of other men.
The theme is taken up by Emperor Leto II in the fourth volume of the saga when he specifies that the Butlerian Jihad was intended to be against attitudes towards mechanical thought[3] as much as against the machines themselves:
She makes me recall the Butlerian Jihad in a poignant way. She is the antithesis of all that’s mechanical and non-human. How odd it is, Moneo, that the Ixians, of all people, should produce this one person who so perfectly embodies those qualities which I hold most dear.”
“I do not understand your reference to the Butlerian Jihad, Lord. Machines that think have no place in …
“The target of the Jihad was a machine-attitude as much as the machines,” Leto said. “Humans had set those machines to usurp our sense of beauty, our necessary selfdom out of which we make living judgments. Naturally, the machines were destroyed.”
[1] The need for a common ethical, if not religious, perspective, is also emerging in real life as we address the issue of AI ethical alignment. Like all information technologies, AI has a global reach, it is not limited to activities within a single nation. And so, according to some ethicists, defining which behaviours are ethically acceptable in different cultures of the world assumes a high priority. For instance, the UN High-Level Advisory Body on Artificial Intelligence is working towards defining a core set of common ethical principles that AI should align with.
[2] The Butlerian Jihad also forbids the use of assisted reproductive technology on the grounds that humanity cannot sink to the level of animals and genetic breeding cannot be left to chance:
Paul held up a restraining hand as Stilgar surged forward.
“We’ve known for two days that she carries my child.”
“But Irulan . . .”
“By artificial means only. That’s my offer.”
The Reverend Mother closed her eyes to hide his face. Damnation! To cast the genetic dice in such a way! Loathing boiled in her breast. The teaching of the Bene Gesserit, the lessons of the Butlerian Jihad — all proscribed such an act. One did not demean the highest aspirations of humankind. No machine could function in the way of a human mind. No word or deed could imply that men might be bred on the level of animals.
“Your decision,” Paul said.
She shook her head. The genes, the precious Atreides genes — only these were important. Need went deeper than proscription. For the Sisterhood, mating mingled more than sperm and ovum. One aimed to capture the psyche.
[3] For this reason, bureaucrats are compared to machines:
Propagation of the Faith. . .
A more honest label would’ve been Propagation of the Bureaucracy, he thought.
A type of religious civil servant had sprung up all through his universe. This new man of the Qizarate was more often a convert. … His gods were Routine and Records. He was served by mentats and prodigious filing systems. Expediency was the first word in his catechism, although he gave proper lip-service to the precepts of the Butlerians. Machines could not be fashioned in the image of a man’s mind, he said, but he betrayed by every action that he preferred machines to men, statistics to individuals, the faraway general view to the intimate personal touch requiring imagination and initiative.
