When the history of AI is told, it's a fair bet that there will be some post-hoc head scratching on who originated the bright idea of rebranding lying as mere hallucination. As if AI was on some sort of acid trip, and its human users along for the ride.

Perhaps it's the reticence to attribute agency to AI; hallucination is something passive, something that just happens, a natural result of ingesting (illicit?) substances. Lying suggests intent, an active choice to deceive for profit.

In any event, all the hand wringing about AI hallucination-lying had tended to frame the core problem as an issue with being given wrong or mistaken or confabulated answers.

Less attention has been paid to the perhaps more debilitating problem for human users: wrong answers are given with confidence.

1/ Transmission effects

It’s intuitive to believe that better knowledge leads to better decision making. What experimental research shows, however, is a much more convoluted picture of the link between knowledge and decision making, mediated by the confidence with which information is transmitted.

For example, various studies show that the more confidence a person exudes the more competent he or she may be perceived as being and more persuasive they are likely to be.[1] But it also seems that this effect happens independent of knowledge. In other words, saying something with confidence is as important to shaping behavior as the actual truth of what's been said.[2] Not exactly breaking news for anybody following GOP politics but still.

We also know that providing new or more information increases confidence, regardless of whether that information is actually useful in increasing knowledge.[3] In a context where acting on unfounded knowledge can have serious ramifications - citing non-existent case law in court, for example - the supreme confidence baked in to AI answers and transmitted to users is a major worry.

In a sense these transmission effects aren’t particularly surprising. AI’s human users are psychologically primed to respond positively to overconfidence, because we seem hard-wired to hate uncertainty.[4] Follow this thought through and it means we’re geared to gravitate towards knowledge sources, media commentators, politicians and - perhaps - AI agents that confidently assert “the truth”, justified or not.[5]

2/ Compound effects

These transmission effects pluck at a second string of the overconfidence problem. Overconfidence is already an issue for human decision making, and AI is making it worse. As Anders Sandberg has put it “when people are overconfident they make more stupid decisions, ignore countervailing evidence and set up policies that increase risk”.[6] This, for Sandberg, helps explain why it’s so difficult to mobilize decision makers to act on long term and existential risks; they have an over-inflated and under-evidenced confidence in their own ideas about risk.

For organizations and societies that are already dealing with fallout from human overconfidence it seems inevitable that introducing superconfident AI machines into the mix will compound or deepen the challenge of compensating for cognitive bias. Finance in particular has attempted to attenuate irrational confidence - viz. building cultures where traders and analysts are more aware of behavioral economic and decision theory lessons about framing biases. Unless the organizational roll out of overconfident AI comes with a health warning - “may exacerbate poor judgment” - we’ll see more risk percolate through these knowledge intensive systems.[7]

3/ Misalignment indicator

A third string of the superconfidence problem has been teased out by AI alignment researchers. The claim is that uncertainty is a virtue when it comes to ensuring AI agents make decisions aligned with humanity. Confidence - an excess of certainty - is an altogether damning attribute for an AI to possess.

The short version of this argument is that we humans have spent the last 2500 years or so philosophically failing to solve the alignment problem. Questions abound, for instance, about the nature of freedom, or the relative hierarchy of values like justice and security, the circumstances where natural moral rights override civil law obligations and so on.

If you ask an AI for the proper definition of peace, it would be disappointing to get a singular answer talking about the absence of violence. Ideally you’d want at least a nod to competing traditions that stress the need to address indirect and structural forms of violence. If you’re looking for a quick answer to a question this might be annoying, but it feels like rather a large slippery slope to start eliding reality because users don’t have the patience for complexity. TL;DR.

The point, as Peter Eckersley put it (with a nod to Isaiah Berlin) is that “agents that are not completely sure of the right thing to do . . . are much more likely to tolerate the agency of others, than agents that are completely sure that they know the best way for events to unfold. This appears to be true not only of AI systems, but of human ideologies and politics, where totalitarianism has often been built on a substructure of purported moral certainty.”[8]

Eckersley goes on to suggest two rules or conjectures to guide AI value alignment: (1) “powerful agents with mathematically certain, monotonically increasing, open-ended objective functions will adopt sub-goals to disable or dis-empower other agents in all or almost all cases.” And the inverse: (2) “powerful agents with mathematically uncertain objectives will not adopt sub-goals to disable or dis-empower other agents unless those agents constitute a probable threat to a wide range of objectives.”

In other words, overconfidence is a canary in the AI value alignment coal mine. AI developers need to drop their pick axes and start encoding uncertainty into AI.

Implications

The dangers of hallucinatory AI are driving discussions around how to integrate AI into organizational workflows. There are emerging tools, like NVIDIA’s Guardrails, that attack the hallucination problem by allowing human users to better understand the source knowledge. There will be other tools or iterations of AI that go further, engineering AI with ever-increasingly accurate levels of knowledge.

What we’ve tried to pull out in here is that, irrespective of progress on hallucinated or confabulated knowledge, there are distinct questions to ask about the effects of AI “superconfidence”. Or, if you like, the manner in which AI interacts with human users. Organizations - particularly in knowledge intensive sectors like law - need to make space for discussion about how to track and manage the framing effects of AI on decision making.