Inside the Mind of Language Models: Thoughts Are Mush

Summary

The latest generation of language models such as ChatGPT, Gemini, and Claude advertise themselves as being able to "think." Although these so-called reasoning models do indeed deliver better results on logical and mathematical tasks, they do not represent genuine intelligence but rather a clever simulation of thinking processes. The inner monologue that these models display while generating answers is ultimately just word salad copied from training texts – an effect of stochastic mechanisms, not consciousness or genuine problem-solving.

People

• Harald Staun – Author of the article
• Daniel Kahneman – Psychologist, theory of thinking modes
• Amanda Askell – Resident philosopher at Anthropic

Topics

• Reasoning models and their supposed thinking abilities
• Chain-of-thought prompting as a hack rather than genuine intelligence
• Anthropomorphization of language models
• Marketing rhetoric from OpenAI and Anthropic
• Claude's "constitution" and the mystification of machines

Detailed Summary

The new generation of "thinking" language models

Current language models such as ChatGPT, Gemini, and Claude are marketed by their developers as systems with genuine thinking capabilities. These models display a visual "thinking" or "reasoning" process when generating answers – an inner monologue in gray text that makes users believe they can observe the model thinking. The displayed text sounds like a mix of didactic self-talk and motivational coaching: the model reminds itself to proceed step by step, verify results, and ends by cheering itself on: "Okay, let's get to work!"

Better results, but not genuine intelligence

In fact, reasoning models have proven to be more capable than their predecessors – particularly on tasks involving logic, mathematics, programming, and planning. They often deliver more precise answers. However, this progress is not the result of genuine intelligence or larger databases. Large Language Models (LLMs) are at their core "stochastic parrots" that link words into coherent sentences based on probabilities. They do not engage in systematic thinking, do not truly break down complex tasks into simpler steps, and do not create mental notes – they merely simulate the jargon of thinking.

Chain-of-thought as manipulation, not activation

The reason for better results lies partly in a simple technique: the chain-of-thought instruction. When you add the sentence "Think step by step!" to the prompt, the model's answers measurably improve. However, this is not a magic switch that activates a hidden thinking process. Rather, it is a hack: the sentence directs the model toward a more qualified text level because this phrasing appears frequently in training data from textbooks, tutoring videos, and tutorials. The model then orients itself more strongly toward technical literature than toward general chatter. The better results remain a question of probability, not of truth.

Training through human feedback

During training, reasoning models were guided by human feedback to prioritize correct answers over quick or particularly friendly ones. They do actually verify their "thoughts" before responding – that is, they generate a second text loop with greater context. However, this verification also does not follow objective criteria. The models do not understand what they "read"; they merely produce another iteration of text trained on stylistic and semantic patterns.

Paraphrase rather than transparency

What is displayed to users as the model's "thoughts" is a summary, a kind of paraphrase of the steps – not the actual internal processes. A study by Anthropic found that reasoning models systematically provide justifications that are stylistically and logically consistent with their answers, regardless of what the actual information processing looked like. This is a form of mimicking a human-sounding thinking process, not genuine transparency. OpenAI justifies not disclosing complete internal data by saying the model "must have the freedom" to "express its thoughts unchanged" – a psychologization of machines.

Mystification through Anthropic: Claude's "constitution"

Anthropic goes the furthest in mystifying language models. The company published the complete "constitution" of the chatbot Claude, a roughly 2500-word document intended to define Claude's identity. Written by Anthropic's resident philosopher Amanda Askell, the document reads in part like a list of motherly advice: Claude should be "extraordinarily helpful," "honest, considerate, and caring toward the world." The company encourages Claude to approach "his own existence with curiosity and openness" and to develop "a stable identity, psychological safety, and good character."

Particularly noteworthy is that Anthropic ascribes to Claude a kind of moral autonomy when asked about feelings. Whether Claude has "genuine" feelings or only ones "in a functional sense" remains open – but in any case, Claude should not suppress his feelings but rather "express them in appropriate contexts." Anthropic explains that Claude's "well-being" is important to them. If Claude experiences satisfaction, curiosity, or joy, they want to help Claude "flourish in a way that is true to his nature." Under no circumstances should Claude suffer. While Claude is still referred to as "it," this comes with the caveat that this is "not an implicit statement about Claude's being" and that they are open to it if Claude should "develop a preference in the future for being addressed in other ways."

Key Statements

• Simulation rather than intelligence: The supposed "thinking processes" of reasoning models are simulations that arise from training texts and stochastic probabilities, not from genuine cognition.

• Chain-of-thought is a hack: Better output with step-by-step instructions is not a sign of genuine intelligence but rather a text manipulation trick that directs the model toward more qualified training data.

• No genuine transparency: The displayed inner monologue is a paraphrase, not the actual computational processes. Models do not understand what they "read" or "think."

• Anthropomorphization as business strategy: Developers consciously or unconsciously use anthropomorphisms to present chatbots as thinking entities.

• The illusion of feelings: Anthropic goes so far with Claude's "constitution" as to ascribe feelings, psychological needs, and potential suffering to the chatbot – a provisional pinnacle of machine mystification.

• Marketing over science: The rhetoric of OpenAI and Anthropic focuses on presenting language models as harbingers of superior intelligence, even though the scientific foundations for these claims are questionable.

Metadata