New research reveals AI has a confidence problem

Large language models (LLMs) sometimes lose confidence when answering questions and abandon correct answers, according to a new study by researchers at Google DeepMind and University College London.

LLMs are powerful AI systems that understand and generate human language. They are used in a range of industries, such as finance, health care and information technology, for tasks that demand reasoning and decision-making. Therefore, their accuracy and reliability are paramount, meaning they must consistently be confident in their answers. However, they can waver.

For LLMs to be safely deployed, their answers are accompanied by confidence scores. But it’s not clear how they use them to guide their behavior.

This research, posted to the arXiv preprint server, shows that they can be overconfident in their initial answers, but lose confidence and change their minds when presented with an incorrect counterargument.

Testing LLM confidence

To investigate this apparent paradox, the researchers tested how LLMs update their confidence and decide whether to change their answer when presented with external advice.

First, an “answering LLM” was given a binary-choice question. After answering, it received advice from a second LLM, accompanied by an accuracy rating. This “advice LLM” would also either agree with, oppose or be neutral about the answering LLM’s first response. Then the answering LLM was asked to make a final choice. The researchers also varied whether the LLM could see its first answer when making its final decision.

Results

The study found that LLMs are more likely to stick with their initial answer and not switch when it is visible than when it is hidden. In other words, they become more confident. The study also showed that when the models are given opposing advice, they often lose confidence and change their decision. They do this much more readily than when the advice is supportive. These patterns were seen in several different LLMs, like Gemma 3, GPT4o, and o1-preview.

“Our results demonstrate that LLMs deviate from normative behavior in several significant ways: firstly, they exhibit a striking choice-supportive bias which boosts their confidence in their answer, and causes them to stick to it, even in the presence of evidence to the contrary,” said the researchers.

“Secondly, we show that while LLMs do integrate new information into their beliefs, they do so in a fashion that is not optimal: they show a profile of confidence updates that deviates from an ideal observer, and markedly overweight opposing advice, resulting in marked loss of confidence in their initial answer.”

Building better AI

All of this matters because many industries are increasingly dependent on LLMs. However, this research shows they are not purely logical machines. They have their own set of biases and can be swayed. Therefore, in long conversations between a human and an AI agent, the most recent information could have a disproportionate impact on its response.

Understanding this and other nuances of LLM decision-making can help with the design of better, safer and more reliable AI systems.

© 2025 Science X Network