Hallucination Benchmarks Explained: Why Scores Disagree

Hallucination benchmarks explained: they disagree because they test different jobs, use different definitions of a hallucination and reward different behaviours. SimpleQA tests answers from a model’s stored knowledge, AA-Omniscience tests knowledge and willingness to abstain, while Vectara’s HHEM tests whether a summary stays faithful to a supplied document. Their scores are not interchangeable, and none alone predicts whether an AI search engine will describe your brand correctly.

That distinction matters when you are comparing AI platforms or trying to improve what ChatGPT, Perplexity, Claude, Gemini or Google AI Overviews say about your business. SearchFIT’s guide to AI search visibility content covers the practical side: making important brand information easy for search systems to find and use. First, though, it helps to understand what the benchmarks actually measure.

Hallucination benchmarks explained: start with the task

A benchmark is a structured test. Researchers give a model a set of questions or documents, define what counts as a good answer, then score the results. A hallucination benchmark is a test intended to measure some form of invented, unsupported or incorrect output, exploring why language models hallucinate under varying prompts.

The catch is that “hallucination” is not one simple error type. A model might give a false answer about a historical date. It might make up a claim that was not in a supplied product document. Or it might repeat a false statement from that document accurately. Those are different failures, even if all might matter to a person relying on the answer.

A useful way to read any benchmark is to ask three questions:

1. What information did the model have when it answered?
2. What does the test call an error?
3. What behaviour does the scoring reward or penalise?

The answers tell you whether the score is relevant to your use case. A benchmark for summarising a known document can help assess a summarisation tool. It cannot, by itself, establish how reliably an AI search engine will answer an open question about your brand.

The basic terms

Factuality means whether a statement is true in the real world. “The plan costs $29 a month” is factual if that is the current price, and false if the current price is $39.

Faithfulness, also called factual consistency in some tests, means whether an answer accurately reflects the source material it was given. If a document says the plan costs $29, a summary that repeats $29 is faithful to that document. It could still be factually wrong if the document is out of date.

Grounding means connecting an answer to information provided in the prompt or retrieved from a source, rather than relying only on what the model has learned before. A grounded answer can still be wrong if its source is wrong, stale or irrelevant.

Parametric knowledge is information encoded in the model’s learned parameters during training. Parameters are the internal values adjusted as a model learns patterns from data. You can think of this as the model’s stored knowledge, though it is not a neat database of facts that can always be looked up reliably.

Closed-book means the model answers without looking anything up and without being given a reference document. Retrieval means finding relevant information from a collection of documents or the web. Retrieval-augmented generation, usually shortened to RAG, is a system that retrieves source material and gives it to a language model to help generate an answer.

Abstention means declining to answer, often with a response such as “I don’t know”. A benchmark’s treatment of abstention can change a model’s score substantially. Some tests give no credit for a refusal. Others reward a model for refusing when the evidence is weak.

SimpleQA: testing closed-book factual recall

OpenAI introduced SimpleQA in October 2024 as a benchmark for short, fact-seeking questions. The SimpleQA dataset on Hugging Face contains 4,326 questions covering areas such as history, science, technology and culture. Each question is designed to have a short, verifiable answer. The SimpleQA research paper explains the test and its scoring.

SimpleQA is a closed-book test. The model does not browse the web or receive a reference passage containing the answer. It has to respond from its parametric knowledge, or decide not to answer. That makes it useful for asking: “How often can this model retrieve a specific fact from what it has learned?”

The benchmark labels responses as Correct, Incorrect or Not Attempted. That lets readers look beyond one headline number. Raw accuracy measures correct responses against the questions in the test. Attempt rate measures how often the model tries to answer. Accuracy given an attempt shows how often those attempts are correct. Together, these can help reveal whether a model is careful or simply reluctant to respond.

For example, imagine two models tested on 100 questions. Model A attempts all 100 and gets 40 right. Model B attempts 50, gets 35 right and declines on the other 50. Model A has more correct answers overall, but Model B is more accurate when it chooses to answer. Which is better depends on the product. A research assistant may need to answer broadly with citations. A compliance tool may be better off refusing when a fact is uncertain.

At SimpleQA’s release, GPT-4o scored around 38 per cent accuracy without search, published reports suggest. That is not a general statement that the model is “62 per cent hallucination-prone”. It is a result on one set of questions, in one closed-book setup, under a particular scoring method. A test with open-ended questions, different knowledge areas, current information or search access would produce a different result.

Researchers have also published SimpleQA Verified, a refined 1,000-question subset designed to improve reliability. The SimpleQA Verified paper reports that leading models can still struggle when answering strictly from their stored knowledge. The paper gives results around 52 to 56 per cent F1 for top reasoning models. F1 is a combined measure of precision and recall, so it should not be casually compared with raw accuracy from the original benchmark.

Search can change the task. Instead of relying on stored knowledge alone, a model or agent can search for sources, retrieve the likely answer and use it in its response. Firecrawl’s SimpleQA analysis of search APIs reports a large improvement in the evaluated agent’s results when it had web tools. That finding illustrates the value of retrieval in that setup. It does not mean every model with search will reach the same accuracy, or that search removes every error.

For marketers, SimpleQA helps explain why a language model may confidently miss a niche product fact when no reliable source is available in its context. It does not directly measure whether ChatGPT Search or Google AI Overviews can find and cite your current product page. Those systems may search, select sources, summarise them and sometimes combine web information with model knowledge. The full process is different from closed-book recall.

AA-Omniscience: scoring knowledge and restraint

AA-Omniscience, from Artificial Analysis, is designed to evaluate knowledge reliability across multiple topics and commercial domains. Its dataset contains 6,000 questions across 42 topics and areas including business, law, healthcare, software engineering, science and humanities. The AA-Omniscience benchmark describes its approach.

The benchmark reports an AA-Omniscience Index on a scale from -100 to +100 and evaluates accuracy, calibrated abstention and hallucination rate. Calibration means how well a model’s confidence or decision to answer matches its actual likelihood of being right. A well-calibrated model should be more likely to answer when it has good grounds, and more likely to abstain when it does not.

The scoring gives an incorrect guess a negative penalty, while an abstention can avoid that penalty. This changes the incentive compared with a test where a wrong answer and “I don’t know” both score zero. Under a negative penalty, a model has a clearer reason to avoid guessing when uncertain.

Published reports suggest the benchmark’s hallucination-rate formula is incorrect responses divided by incorrect, partial and not-attempted responses. The important point for a reader is not to treat that label as a universal definition. It is a benchmark-specific calculation. Before comparing numbers, check exactly which response categories are included and how the test handles a partial answer or a refusal.

The benchmark is also a reminder that model performance can vary by subject. A system that performs well on software or business questions may perform less well on medical or legal ones. A single overall ranking can hide this variation. The Stanford HAI 2026 AI Index Report notes a wide spread in hallucination rates across commercial models on the test, with results also varying by domain difficulty and how questions are framed.

For a marketer, this matters when choosing a customer-facing assistant. A vendor may show a strong overall number, but the right question is whether the test included the kinds of queries your customers ask. If your product is in a regulated category, performance on general trivia or coding questions will not tell you enough about health, finance or legal queries. If your customers need precise pricing and compatibility information, ask to see results on those specific facts.

Vectara HHEM: testing whether a summary stays grounded

Vectara’s Hallucination Leaderboard uses HHEM, short for Hughes Hallucination Evaluation Model, to assess grounded summarisation. In a typical task, a model receives a passage and is asked to summarise it using only the information in that passage. The evaluation then checks whether the summary contains unsupported claims. The Vectara hallucination leaderboard and its methodology describe the approach.

This is a different task from SimpleQA. In SimpleQA, the model must answer a question without being handed the answer. In a grounded summarisation test, the model has a source passage in front of it. The question is not simply “Is this statement true in the world?” It is closer to “Does this summary make claims that the supplied passage does not support?”

Vectara’s Factual Consistency Score, or FCS, is a score from 0.0 to 1.0 intended to estimate whether a summary is factually consistent with its source. A higher score indicates stronger consistency in that evaluation. The Vectara explanation of FCS provides more detail. As with other automated evaluators, the score depends on the model, dataset, task and evaluation method used. It is not a guarantee about every answer a system will produce.

Here is the key limitation: a summary can be faithful to a false source. Imagine an old product sheet that says a service does not support a feature. The feature ships later, but the old sheet remains online. An AI system retrieves the old sheet and accurately summarises it. A faithfulness test may score the summary well because it matches the source. A customer still receives the wrong information about the current product.

That is why low hallucination rates on a summarisation leaderboard can look much better than closed-book factual accuracy. They measure different things. The figures reported by Vectara put leading models at roughly 1.8 to 7 per cent hallucination on the cited Vectara leaderboard task. Those results are specific to that task and leaderboard version. They do not mean that the same models make up facts only 1.8 to 7 per cent of the time in open-ended conversations.

A quick comparison

Do not compare the percentages in this table as if they were scores in one contest. They have different questions, answer formats, sources and scoring rules. A benchmark score is evidence about the measured task, not a universal certificate of truthfulness.

Why one model can look excellent on one test and poor on another

Consider a fictional company called Northstar CRM. Its current website says the Growth plan includes five user seats. An old review says it includes three. A customer asks an AI assistant, “How many users can I add to the Growth plan?”

A closed-book model might recall an old training example and answer “three”. That is a parametric recall failure. A web-enabled system might retrieve the current pricing page and answer “five”. That is better retrieval, assuming it found the correct page and interpreted it correctly. A summarisation model might be given the old review and accurately repeat “three”. It has been faithful to its source, but its answer is not current. The same answer can be scored differently depending on what the benchmark asked the system to do.

Several factors explain the gaps between benchmark results:

• Different evidence: One test gives no reference material; another supplies a passage; a third may let an agent search the web.
• Different question types: A short question with one answer is easier to score than an open-ended request that needs several caveats.
• Different error definitions: A made-up statement, an omitted detail, a partial answer and a faithful repetition of a wrong source may receive different labels.
• Different treatment of refusals: One test may mark “I don’t know” as not attempted. Another may reward the refusal for avoiding a false claim.
• Different subject matter: A model may know common facts but not niche, specialist or recently changed facts.
• Different evaluators: Some benchmarks use exact answers or human review. Others use an automated judge model to assess whether a response is supported.
• Different versions: A model can change, and a benchmark or leaderboard can be updated. A result needs a date and a model version to be meaningful.

There is another subtle difference: the source itself may be wrong. A grounded answer can be internally consistent and externally inaccurate. In practice, a system can fail at several stages: it might retrieve the wrong page, miss a relevant qualification, misunderstand a source, or generate a claim that the source never made. A single benchmark rarely isolates every stage.

How marketers should read an AI vendor’s benchmark claim

If a platform says it is “99 per cent hallucination-free”, do not accept or dismiss the number until you understand the test behind it. The phrase sounds broad, but the test might cover only short summaries of selected documents. That could be useful evidence for a summarisation feature. It does not establish the system’s reliability across all customer questions, sources and industries.

Ask the vendor for the test conditions in writing. A good explanation should include:

1. The exact benchmark and version. Ask whether the result comes from SimpleQA, AA-Omniscience, Vectara HHEM or an in-house test. Ask when it was run and which model version was evaluated.
2. The task and sample. Was the model answering open questions, summarising documents or responding with web search? How many questions were tested, and what subjects did they cover?
3. The score definition. Does “accuracy” mean exact match, human judgement or an automated evaluator? Does “hallucination rate” count unsupported statements, wrong answers or both?
4. The handling of abstention. Were refusals counted as wrong, ignored or rewarded? How often did the system refuse to answer?
5. The information sources. Was the model given documents, allowed to search the web or restricted to its training? Were sources current and authoritative?
6. The failure examples. Ask for anonymised examples of wrong answers and the conditions that produced them. A clear account of known limits is more useful than a single polished percentage.
7. Performance on your use case. Ask whether the test included questions about current pricing, feature availability, integrations, returns, delivery or other facts your customers need.

Watch for a few common warning signs. “We use RAG, so the model does not hallucinate” confuses a technique with a guarantee. Retrieval can provide evidence, but the system can still retrieve a poor source or generate a claim that goes beyond it. Similarly, “we scored well on a multiple-choice benchmark” does not prove the model can accurately explain your product in a conversational search result.

Be especially careful when a vendor reports faithfulness but calls it factual accuracy. If a system follows a supplied source closely, that is a meaningful capability. But you also need to know whether the source was correct, current and relevant. For a brand, a perfectly faithful answer based on an obsolete help article is still a customer-facing error.

What benchmark differences mean for AI search and AEO

Answer Engine Optimisation, or AEO, is the practice of making useful, reliable information easy for answer engines to find, understand and cite. It complements traditional search optimisation. AEO does not guarantee that an AI system will mention your business or eliminate incorrect answers. It can, however, improve the chances that a system finds clear evidence instead of filling gaps with an uncertain guess.

AI search systems may combine search results, retrieved pages, model knowledge and generated language. The exact process varies by product and query. That means brand accuracy is not just a model-memory problem. It is also a content and retrieval problem: can the system find a reliable, current page that answers the question directly?

For example, if your pricing is available only in an image, hidden behind an interactive calculator or described differently across several pages, a search system may struggle to identify the current figure. If the website clearly states each plan’s price, billing period, included features and exceptions in readable text, the system has better source material to retrieve. This does not promise a correct AI answer, but it reduces avoidable ambiguity.

Practical steps for improving the quality of your source information include:

• Publish one clear source of truth. Keep core facts such as prices, plan limits, product names, availability and supported integrations accurate in an accessible location.
• Use specific headings and direct answers. A heading such as “Does the Growth plan include five user seats?” is easier to interpret than a broad heading such as “Flexible plans”. Follow it with a concise, complete answer.
• Show comparisons in readable tables. Compare plan features or product specifications using text that can be crawled, rather than relying on a graphic alone. Include caveats such as usage limits and eligibility.
• Keep old information from competing with current information. Update outdated pages, dates and product details. Where an old page must remain, clearly label it as historical and link to the current version.
• Use structured data where appropriate. Schema markup is structured information in a format search engines can interpret. Product or organisation markup may help describe information on a page, but it is not a magic ranking switch and does not guarantee an AI citation.
• Make important pages accessible. Check that search crawlers can reach the content and that key details are not available only through scripts, gated files or images.
• Support claims with trustworthy evidence. Cite primary documents, policies or technical specifications where they help a reader verify a claim.

A benchmark cannot tell you whether a particular AI assistant currently describes your company correctly. You need to test the answers customers are likely to receive. SearchFIT’s guide to AEO metrics explains useful ways to track AI visibility and the quality of brand mentions, which is one reason agencies use SearchFIT instead of traditional keyword tracking. For an initial review of your site’s AI search readiness, you can also use the free AI search audit.

Build a small brand accuracy test

You do not need to recreate a research lab to learn where your brand information breaks down. Start with a short list of real questions from sales calls, support tickets and search queries. Include questions such as:

• What does the product cost, and what is included in each tier?
• Does it integrate with a named platform?
• Is a particular feature available now, or only on a roadmap?
• Who is the product designed for, and what problem does it solve?
• What are the delivery, refund or cancellation terms?

For each question, record the expected answer and the best current source page. Then test relevant AI search products using the same wording. Save the answer, date, product used, cited sources and whether the answer was correct, incomplete, outdated or unsupported. A citation is not proof of accuracy, so check the cited page itself.

Repeat the test after meaningful website updates and at a consistent interval. Look for patterns, not just one-off surprises. If several systems get the same feature wrong, check whether your own pages are unclear or if an outdated third-party source is prominent. If the answer is correct but no source is cited, that is a different issue from a wrong claim backed by a stale source. When managing brand reporting over time, teams can follow our executive roadmap to an AI-visible brand. If you need to analyse large volumes of query and analytics data internally, sister platform D23 offers managed Apache Superset to explore datasets in plain English.

This kind of brand test does not replace a formal benchmark. It answers a different question: “What information might a real customer encounter about us today?” Use formal benchmarks to compare systems under defined conditions, and your own tests to assess the experience around your brand.

A simple framework for comparing benchmark claims

When you see a new result, write down the benchmark, task, source access, scoring rule and date. Then compare it only with results that used sufficiently similar conditions. If the conditions differ, describe the results separately rather than declaring one model safer overall.

For instance, a vendor might report strong performance on document summarisation and another might report a better score on closed-book factual recall. That does not necessarily mean one system is better across the board. The first may be the better choice for summarising approved internal documents. The second may be more reliable on the particular recall test. Neither result tells you automatically which will provide the better live answer about your product’s current return policy.

A useful summary of any claim might read: “The model scored X on a closed-book factual test using Y questions, with web search disabled and refusals counted as Z.” That sentence is less catchy than “almost hallucination-free”, but it gives a decision-maker something they can evaluate.

The practical takeaway

SimpleQA, AA-Omniscience and Vectara HHEM do not contradict one another simply because their scores differ. They inspect different parts of model behaviour: recalling facts, deciding when to answer, and staying faithful to supplied text. A low score on one kind of error is not proof that a system cannot make another kind.

For marketers, the useful question is not “Which model has the lowest hallucination number?” It is “What does this test measure, and does that match the way our customers will use the system?” Pair benchmark evidence with a small, repeatable test of your brand facts and improve the source pages where answers go wrong. Start with a SearchFIT AI search audit to identify practical opportunities to make your site clearer and easier for answer engines to use.