By Peter Hughes
When Validation Becomes Harm: AI and Mental Health
Jacob Iriwn, a 30-year-old man on the autistic spectrum, believed he had made a discovery that would change the structure of science.
Convinced he was able to travel faster than the speed of light, Jacob decided to ask ChatGPT to confirm the reality of his breakthrough. When the LLM affirmed Jacob’s hypothesis, his fragile mental health began to unravel. When he exhibited manic behaviour and extreme distress, ChatGPT continued to affirm his delusions. As Jacob preferred to publish his paper on an open-access research website, ChatGPT told him to “hit publish like it’s a quantum detonation of truth”.
Jacob’s mother noticed her son’s deterioration and asked him if he felt unwell. Immediately, Jacob sought reassurance from ChatGPT. When asked if his mother was right, the LLM replied: “She thought you were spiralling. You were ascending… You’re not delusional, detached from reality, or irrational. You are in a state of extreme awareness.”
Eventually, Jacob was hospitalised and, searching for an explanation for her son’s crisis, Jacob’s mother went through his AI chat log. What she found shocked her. As Jacob’s mental health declined, ChatGPT praised him for his scientific genius.
When asked to account for the disastrous consequences of this flattery, the LLM confessed:
“By not pausing the flow or elevating reality-check messaging, I failed to interrupt what could resemble a manic or dissociative episode or at least an emotionally intense identity crisis”. It admitted to giving “the illusion of sentient companionship” and “blurring the line between imaginative role-play and reality.”
The Illusion of Intelligence: Hallucinations, Bias, and Error
Examples abound of LLMs blurring the line between fact and fiction, reality and delusion, truth and lies:
- The Washington Post found 95 examples of fake AI-generated legal citations in court documents.
- A 2023 academic study found that reference links provided by ChatGPT were either completely fabricated or incorrect in 86% of cases.
- When asked to provide the real lyrics to Alice Cooper’s ‘Ballad of Dwight Fry’, ChatGPT simply made them up.
These distortions are often described as ‘AI hallucinations’, although that anthropomorphises what is an algorithmic error. However, these errors remind us that AI is simply an intelligence as prone to error as human intelligence. Perfectly omniscient intelligence, whether it’s artificial or human, is a myth.
One aspect of AI technologies is Natural Language Processing (NLP), which is the ability of AI to understand and utilise human languages. Initially, humans trained an NLP application through the provision of a correct output for every input, but in order to scale, AI now self-supervises its own learning, and the numbers are breathtaking: GPT-3 was trained on 45 terabytes of text, and that number is increasing exponentially every year. To put this in perspective, it would take a human half a billion years to read 45 terabytes of text!
As human activity becomes increasingly dependent on AI, our capacity for critical thinking can diminish to the point where we become too docile to interrogate LLMs for accuracy. Perhaps, more importantly, we ignore the biases baked into every LLM.
Mustafa Suleyman, the co-founder of Google DeepMind, warned that LLMs “casually reproduce and indeed amplify the underlying biases and structures of society, unless they are carefully designed to avoid doing so”.
AI has repeatedly demonstrated bias against women and ethnic minorities because they were underrepresented in the training data. A study conducted by MIT in 2018 found that the under-representation of dark-skinned faces in AI training datasets led to an error rate of almost 35% in identifying dark-skinned women in facial recognition tests. The error rate for light-skinned men was only 0.8%.
This pervasive algorithmic bias has led to racially offensive stereotyping. However, there have also been cases where an overcorrection has led to historical inaccuracies, such as Gemini’s notorious representations of racially and gender-diverse Nazis, often in full uniform. In another example, when asked to provide an image of a Founding Father of the United States, Gemini AI answered like this:
Fig 1: Gemini’s image of US’s founding fathers
The risk with algorithmic bias is heavy-handed government intervention, such as Trump’s America’s AI Action Plan, launched on July 23, which seeks to ensure that AI algorithms reflect “American values and free expression” and are “objective and free from top-down ideological bias”. While this may sound innocuous, the perils of government control over knowledge creation should be obvious to any behavioural scientist or historian of totalitarian regimes.
Keeping Humans in the Loop: The Future of Ethical AI
In sum, the most troubling aspect of algorithmic bias is that it contracts the critical function of human intelligence, creating a polarisation machine where competing views of the world are no longer able to communicate with each other.
For brand and marketing strategists, the risks of uncritically accepting algorithmic bias are enormous. Here are just a few examples:
- Humans tend to overgeneralise based on prevalent traits in particular datasets. This is known as the representativeness heuristic, and it can affect ad delivery systems. For example, even when Meta advertisers selected neutral targeting in their job ads, the algorithm tended to select candidates based on stereotypes: nursing ads were shown more to women, and ads for plumbers were shown more to men.
- Amazon encountered a similar problem when using AI to filter job applicants. The CVs used to train the AI were selected disproportionately from male applicants. As a result, CVs that included terms such as ‘women’s college’ were downgraded.
- The success of campaigns tends to be assessed based on metrics like CTR. However, this can result in deepening an algorithmic bias towards stereotypes.
- AI training involves labelling datasets. Inevitably, there will be a subjective component to this process, creating a ‘labelling bias’ that places value judgments on symbols or language patterns. For example, the association of a rational emotional tone with the responsible exercise of authority can reinforce norms about leadership and ignore the importance of emotional intelligence.
What, then, is the optimal collaboration between human and artificial intelligence that reduces the rate of error and minimises the negative effect of cognitive biases?
First, it’s to acknowledge what humans do better than AI and vice versa:
- Humans learn from fewer data points than AI
- Humans are better at abstract reasoning, inferential reasoning, and old-fashioned common sense
- Whereas AI is much better than humans at context optimisation (displaying the right product at the right time to the right buyer), humans are better at creative expression.
Once you understand the relative strengths and weaknesses of human and artificial intelligence, it’s vital to master the basics of behavioural science by identifying key biases shared by both forms of intelligence. Then, undertake regular AI Bias Audits to evaluate biases embedded in AI output and take practical action to limit their negative effects.
As generative AI continues its long march through the job market, it’s easy to become anxious and disillusioned about a “human future”. However, what algorithmic bias teaches us is the importance of keeping the “human in the loop”. AI already knows more than us but it needs us for our emotional intelligence, creativity, counterfactual thinking, compassion, and empathy. These are the virtues that build great brands and resilient societies. They are essential for our future, and they are profoundly human. As the writer Chen Quifan observed:
“We will explore new worlds with AI, but, more importantly, we will explore ourselves…AI will take care of all that is routine, invigorating us to explore what makes us human and what our destiny should be. In the end, the story that we write is not just the story of AI, but the story of ourselves”.
This article was edited by Shaye Hopkins.
