Google and Technion study: LLMs can recover up to 65% of unrecalled facts by thinking longer

Full text

When large language models (LLMs) hallucinate, developers typically assume the model lacks the required facts. Engineering teams diagnose the error as missing knowledge. The standard response is to increase model size, expand training data, or build complex retrieval architectures.

A new study by researchers at Google Research and Technion demonstrates that the knowledge is often not missing. The model has the information encoded parametrically but fails to surface it during generation.

Their experiments show that frontier models like GPT-5 and Gemini-3 encode 95-98% of tested facts. This indicates that in many cases, recall, rather than encoding, is the primary bottleneck for factual accuracy.

By understanding how to unlock existing knowledge through inference-time computation, engineering teams can build more reliable applications without necessarily relying on larger models or external databases.

Knowledge profiling: measuring what models actually know

To map this gap between storage and retrieval, the researchers propose shifting the evaluation focus from question-level accuracy to fact-level profiling. Instead of simply scoring whether an LLM answers an isolated prompt right or wrong, fact-level profiling tests a single underlying piece of information across multiple conditions, evaluating whether the fact is stored in the model's parameters at all, whether it can be queried from different directions and phrasings, and what computational effort is required to retrieve it.

This framework distinguishes between whether a fact is parametrically "encoded" and whether it is "known". A model encodes a fact if it can accurately reproduce it when primed with its original training context. A model knows a fact if it can reliably answer questions about it across varied phrasings and directions.

"Encoding and recall failures are indistinguishable under accuracy metrics, yet they imply different limitations and solutions,” the researchers write. “Encoding failures call for pre-training interventions, such as scaling model size or data coverage. Recall failures suggest post-training interventions that often improve how models utilize what they already encode."

The paper illustrates this using a sample fact: Oasis played their first gig at the Boardwalk club. Based on how models process this information, the study categorizes knowledge into five distinct profiles:

Direct recall: The model encodes the fact and readily accesses it to answer direct questions without extra inference compute.

Encoding failure (empty shelves): The model neither encodes nor knows the fact. It cannot complete a Wikipedia-style sentence about Oasis’s early days, nor can it answer questions about the event. This signals a need for more pre-training data or greater model capacity.

Recall failure (lost keys): The model has the fact encoded but cannot access it. It can seamlessly complete the original training text about Oasis, but fails to answer "Where did Oasis play their first show?" even when given time to think.

Recall with thinking: The fact is encoded, but inaccessible to direct generation. It is only successfully recalled when the model uses inference-time computation, such as Chain-of-Thought, to bridge the gap. The researchers refer to this mechanism as recall facilitation. The model might initially fail to answer the direct question. By generating intermediate thoughts about the band's early history in Manchester, it structurally primes itself to locate and recall the locked answer.

Inference without encoding: The model never explicitly encoded the Oasis fact. Instead, it successfully answers the question by making an educated guess or reasoning across other encoded facts it does know. It might deduce the answer by chaining together separate data points, such as "Oasis formed in Manchester," "the Boardwalk was a famous 90s music club there," and "the Boardwalk hosted early gigs by emerging bands.”

Scaling illusions, long-tails, and tip-of-the-tongue recoveries

The researchers evaluated 13 LLMs on over 4 million responses. They used WikiProfile, a benchmark containing 2,150 facts extracted from Wikipedia, testing each fact across formats ranging from exact context completion to multiple-choice verification.

For frontier models like GPT-5 and Gemini-3, encoding is nearing saturation. These models successfully encode 95-98% of the tested facts. However, they still fail to directly recall 26-34% of those encoded facts without thinking.

Inference-time thinking acts as a vital recovery mechanism. Providing models with extra computational effort successfully retrieves 40-65% of the encoded facts that models initially fail to directly recall. The researchers compare this to the human tip-of-the-tongue state, where deliberate effort, such as mentally retracing context, eventually helps remember the information.

Scaling up model size does not automatically resolve this gap. For example, they found that scaling the Gemma3 model from 1 billion to 27 billion parameters decreased encoding failures from 85% to 23%. But at the same time, the share of recall failures increased, peaking at 40% without thinking.

This suggests that scaling mainly solves the storage problem rather than the access problem. As the model memorizes vastly more facts, a larger pool of knowledge becomes trapped in an "encoded but inaccessible" state. The bulk of model errors shifts from missing data to failed recall.

"Our findings suggest that recall is tightly coupled to the conditions under which facts were learned, degrading when queries diverge from training-time patterns," the researchers write. How a user asks a question directly dictates whether the model can unlock the stored answer.

For example, the experiments showed that rare facts are encoded at rates similar to popular facts. Yet they found a large recall gap between long-tail and highly popular facts that exceeds 25% for frontier models.

Similarly, models struggle to generate answers to reverse questions (i.e., asking for the subject instead of the object). For example, a model might easily answer that Oasis played their first gig at the Boardwalk club, but fail to answer who played their first gig at that same club. At the same time, the same models show that they know the correct answer when given the same question in multiple-choice format.

"Whereas these failures are often interpreted as limitations of memorization or bidirectional encoding, our results suggest a different picture: rare facts are often encoded but inaccessible, and reverse facts can be recognized even when they cannot be generated,” the researchers write. “This reframes both phenomena as recall failures rather than 'missing knowledge.'"

The ROI of thinking and tips for developers

The high encoding rates of frontier models require a shift in how developers approach factuality and pipeline architecture.

Don’t treat every factual failure as a retrieval problem: The default enterprise reaction to hallucinations is often to deploy Retrieval-Augmented Generation (RAG), scale up vector databases, or ingest more domain documents. While necessary for real-time updates or entirely absent data, this study shows up to 95-98% of standard facts are already parametrically encoded. Many failures can be recovered without external retrieval.

Use inference-time reasoning selectively: Thinking recovered 40–65% of encoded facts that models failed to directly recall. However, thinking is computationally expensive. A practical architecture involves a fast first-pass model call, followed by a higher-reasoning-effort retry when confidence is low. Teams can optimize costs by dynamically routing challenging conditions, like rare entities or reverse questions, to thinking models.

Test semantic access, not just benchmark accuracy: Standard accuracy metrics mask underlying model capabilities. Evaluation sets should probe the same underlying fact across different phrasings, contexts, and directions to truly understand what a model knows versus what it can reliably access.

Leverage query reformulation and retries: Because recall is highly context-dependent, query framing dictates success. Changing the structure of a prompt, generating relevant intermediate context, or prompting the model to generate a reasoning chain before answering are legitimate reliability mechanisms that surface information direct prompts miss.

Limitations and practical takeaways

The WikiProfile benchmark relies on encyclopedic Wikipedia facts. These findings might not perfectly generalize to proprietary or highly specialized enterprise domains. A model's ability to store and recall a niche internal company metric may behave differently than its handling of public encyclopedic data.

Fully profiling a frontier model on the WikiProfile suite costs approximately $500. Developers can significantly reduce this cost by omitting multiple-choice variants or using fewer response samples per question.

Teams can access the WikiProfile benchmark on Hugging Face to evaluate their own systems and build custom knowledge profiles. Identifying whether an enterprise application suffers from empty shelves or lost keys is the first step toward fixing it.