Linked Out
Some AI is an unfaithful servant, even to itself.
How should we think about machines that “think?” Cautiously, as it turns out. It’s a useful thing to do, however, not least because it reminds us of the productive turmoil between our ears. That stuff we also call “thinking.”
A couple of weeks ago I heard about a recent paper addressing two crucial aspects of AI. It was called “Chain-of-Thought is Not Explainability.” The second term, “explainability”, is a concept: How did an AI program come to its conclusion? This is an important idea for many kinds of things. Explainability might show whether an AI refused a loan fairly, or how it reached a diagnosis, or was transparent in making a legal judgement. These are, by the way, all areas in which AI is increasingly applied.
Explainability sounds a lot like the first term, “chain-of-thought,” or CoT. But as the title of the paper suggests, while CoT offers the steps to a conclusion, these steps often do not accurately reflect the way the AI reached its conclusion.
This mismatch occurs because the CoT is an entirely separate AI program from the AI that delivered the original answer. CoT is best-guessing its way to the conclusion reached by the original program. It has no feeling for the steps that the original program took, but is guessing at them from afar.
The paper notes that the original AI program is processing information in a distributed way. That is, it’s gathering information in a nonlinear fashion, and assembling the results in an unknown order. On the other hand, the CoT presents the result as a sequence of steps, as if that’s what the original AI did. In fact, these are quite different ways of coming to a conclusion.
This and other problems with CoT that the paper discusses are a big deal. As the authors of the paper put it, there’s “a risk of misplaced confidence in model outputs, particularly when explanations appear logical but fail to reflect the true reasons behind a decision.” And since AI presents decisions with unshakable confidence about itself, humans tend to believe the answers it gives them.
I wanted to write about this. Thinking like a human, I wondered if the way people use CoT as if it were explainability might be considered as something like a guilty afterthought - there is a conclusion, so now the humans need to figure out the steps by which it was reached. This led me to think about the famous line, “it’s the cover-up that kills you,” which, as I found with a little research, is really a folkloric shortening of the late Sen. Howard Baker’s line during the Watergate hearings, “It is almost always the cover-up rather than the event that causes trouble.” My recall of the quote was not accurate to Sen. Baker, but well suited to what might be the theme of this post: “AI - It’s the Cover-Up That Kills You.”
Then I decided it was better to have a pithy headline that referred to the absence of any link between the activities of answer-making and method-explaining. “LinkedIn” came to mind as something to play against. I threw away all that other thinking, in favor of the headline you’ve read.
In between these dubiously-linked ideas about CoT and cover-ups, and on my way to the punning title, I wondered in passing about food, shelter, sex, weather, United Airlines’ on time record, my social media popularity, that weird buzzing sound, the previous night’s strangely powerful dream in the laundromat, the headline for this piece, the subhead, Howard Johnson’s tuba work on “Unfaithful Servant,” and who knows what else.1 As one does. It’s unlikely that any of this “thinking” affected what I wrote, but it was all within my thought process.
A similar concatenation would probably be there if I was working through something less diffuse than writing. Say, a mathematical proof. I’m not proud of it, but human reasoning, such as it is, is messy like that. We have different expectations of AI, however: we assume a less diffuse and more coherent approach to the goal, the pure reasoning that some of us fancy is the way we reason (We don’t.)
Our coherence as biological entities, complete with emotion and the memory of past emotion, and our strange and tragic relationship with time, must of necessity be far more messy and capacious than the coherence of an AI program. Yet according to the CoT paper, AI can’t even manage to recall the relatively uncomplicated process by which it came to a conclusion.
Unlike with people, the answer and the explanation are unlinked activities. What gives?
That gets to the “cover-up” part I was thinking about. Something not necessarily guilt-driven, in this case. Yet like a cover-up, something done after the fact, with little connection to the actuality.
And as such, with computers, a misdirection.
Humans are far messier than we think in our approach to figuring out the world. Paradoxically, however, while we are far more diffuse, we are also far more comprehensive. We carry with us the steps we take, in memory and often by encoded habits and the regulations we follow, which are all part of our selves. We’re parking and retrieving information, and probably reshaping it as we go, thanks to our interesting capacity for taste. When it comes to talking about how we do things though, we don’t create a whole different method. We stay inside the same practice, inside our selves.
Does this difference matter? Absolutely, particularly as contemporary AI is rapidly generating a computer-assisted world of answers rather than searches. Google used to provide links to likely websites where we might find answers, from which we chose our best option.2 We were part of the process of decision. Today, ChatGPT and other AI agents present people with a finished product, but give no sense of how the answer was derived. A misleading CoT explanation of a single pathway, delivered persuasively, is worse than no explanation at all.
The authors of the paper suggest that the CoT problems might be resolved by adapting human-type reasoning strategies to the process. These include assigning confidence scores (aka questioning your own bullshit), recognizing narrative drift (aka figuring out that your story is departing from the facts), and creating a new program to question and pressure-test the quality of the original CoT work.
Stated another way, one of the great challenges in AI is learning how to encode doubt. Good luck with that.
Fortunately, we humans remain excellent at this. Perhaps we could even cultivate it, as we go further into the Age of Pure Answers.
When genius fiction writers of a century ago attempted to portray the stream of consciousness, they evoked something interesting, but their result wasn’t close to the actual three-dimensional pinball of mental life.
These have been increasingly subject to their own disinformation, thanks to both farms, Search Engine Optimization, clickbait headlines, and other efforts to game the system, but that was still inside a system where people applied critical thought.