Gemini 4 Carbon Hallucination Rate: What Benchmarks Show

The reported gemini 4 carbon hallucination rate is about 11% on Artificial Analysis’s AA-Omniscience v1.1 benchmark. That is higher than Gemini 4 Argon’s reported 4%, but Carbon also answered more questions correctly, giving it a stronger overall score on this test. The result is a measure of closed-book factual recall, not a guarantee about how Gemini will behave when it searches the web or summarises a source.

Google announced Gemini 4 Carbon on 28 October 2026, according to third-party reporting. Its benchmark results matter to marketers because factual errors and abstentions can affect whether AI systems state, omit or misrepresent a brand’s details. For practical ways to make brand information clearer to AI systems, see SearchFIT’s guide to content strategies for AI search visibility.

What happened with Gemini 4 Carbon

Artificial Analysis’s evaluation reports that Gemini 4 Carbon scored 58% accuracy and an 11% hallucination rate on AA-Omniscience v1.1. The benchmark’s Omniscience Index gives Carbon a net score of 54, placing it first among the models shown in the published comparison. These figures describe performance on a particular test, not an overall score for every kind of AI task.

The comparison with Gemini 4 Argon is especially useful. Argon recorded 48% accuracy, a 4% to 4.3% hallucination rate, and a net Omniscience Index score of 46. Carbon’s hallucination rate is higher, but its accuracy is also ten percentage points higher. The pattern suggests Carbon attempts more questions and gets more right, while accepting a greater risk of giving a wrong answer.

The headline comparison with other frontier models is striking, but should be read just as carefully. The figures supplied for the same test put Claude Opus 5.5 at about 66% accuracy and around 50% hallucination, while OpenAI GPT-6 Astra is reported at about 62.6% accuracy and a 50% to 51.3% hallucination rate. On those numbers, Carbon has the lowest reported hallucination rate among this group, though it does not have the highest accuracy.

The primary source for the benchmark is Artificial Analysis’s AA-Omniscience evaluation. Its Gemini 4 Carbon announcement and Intelligence Index analysis reports the model’s standing in the broader index. The important distinction is that a model can rank highly overall while still making errors on particular prompts, topics or real-world workflows.

Figures in this table are reported benchmark results from Artificial Analysis and related third-party coverage. Small differences in displayed rates may reflect rounding or the version of a result being reported. Check the live AA-Omniscience evaluation page for the current methodology and model entries before using these numbers in a comparison or presentation.

What the 11% hallucination rate measures

AA-Omniscience is a closed-book factual recall test. The model receives questions without a supplied source passage to consult. The test covers 3,000 questions across 42 topics and six broad domains: science, engineering and maths; software engineering; finance; health; humanities and social sciences; and law.

For each question, the model can give an answer, decline to answer, or make a claim that is judged incorrect. In the benchmark’s scoring, a correct answer earns credit, an abstention is neutral, and a hallucinated answer is penalised. In everyday language, “hallucination” means a plausible-sounding answer that is factually wrong. In this benchmark, it refers to incorrect assertions made in response to the test questions.

That distinction matters because the 11% figure is not the percentage of all possible Gemini responses that will be wrong. It is a result on a defined collection of questions, under defined test conditions. It does not tell you the probability that Gemini will invent a fact in every conversation, nor does it directly predict its error rate on your company’s product details, support tickets or industry questions.

The denominator also matters. A model that answers fewer questions can have a low hallucination rate because it abstains when uncertain. A model that attempts more questions has more opportunities to be correct and more opportunities to be wrong. The hallucination rate therefore needs to be read alongside accuracy and the model’s tendency to abstain. Looking only at one number can reward excessive caution or hide an increase in risky answers.

Artificial Analysis explains its benchmarking approach in its methodology documentation. For a reader comparing systems, the practical questions are: What did the benchmark ask? Was the model given sources? How were refusals treated? How were wrong answers scored? Which model version was tested? Without those details, a single percentage can sound more definitive than the test warrants.

Why Carbon’s rate is higher than Argon’s

Argon’s lower rate does not automatically mean it is the more useful model. Its 4% to 4.3% result sits alongside 48% accuracy and a net score of 46. Carbon’s approximately 11% rate sits alongside 58% accuracy and a net score of 54. In this evaluation, Carbon appears to trade some caution for more correct answers.

This is a familiar trade-off in factual systems. If an assistant says “I don’t know” whenever confidence is low, it may avoid many false claims but leave more questions unanswered. If it answers more often, it can provide more useful information, but some of those extra attempts will be wrong. The right balance depends on the task. A brainstorming assistant and a legal research assistant should not necessarily have the same tolerance for uncertainty.

The published values do not show exactly how often Carbon abstained on each topic or how confidence varied across questions. So the safe reading is that Carbon performed better on the benchmark’s overall balance of correct answers, abstentions and wrong answers, not that it is always better calibrated in every field. A marketer evaluating a model for a particular workflow should test representative queries, including questions where the right response is to say there is not enough information.

Accuracy, hallucination and the net score are different measures

Accuracy asks how often the model produced a correct answer under the test’s scoring rules. Hallucination rate tracks the share of responses classified as false assertions, according to the benchmark’s definition. The net Omniscience Index combines the outcome types into a single score, with a range described as minus 100 to plus 100.

These measures answer different questions. Accuracy is useful for understanding how much of the test the model got right. Hallucination rate focuses attention on incorrect assertions. The net score captures the balance between useful answers and errors, while accounting for abstention. None replaces the others.

For example, Carbon’s 58% accuracy does not mean that all its other responses were hallucinations. Some may be abstentions, and some may be classified in other ways under the benchmark’s scoring. Likewise, a reported 11% hallucination rate does not mean that Carbon is correct on the remaining 89% of questions. Some of the remainder may be questions it declined to answer. That is why the three figures should be presented together.

What the comparison does and does not show

Based on the reported AA-Omniscience results, Carbon has a lower hallucination rate than the named competitors in this comparison, but a lower accuracy score than Claude Opus 5.5. It has the highest net score in the comparison table. That makes the headline more specific than “Carbon is the most accurate model” or “Carbon never hallucinates”. Neither statement follows from these figures.

The results also sit alongside other evaluations. Artificial Analysis reports an Intelligence Index v5 score in the range of 59 to 60 for Carbon, compared with 50 for Argon, 58 to 59 for Claude Opus 5.5 and 57 for GPT-6 Astra. Vals reports a general Vals Index score of 70.45% and a first-place result in its legal research benchmark. These results indicate that ranking depends on the task and the benchmark. They should not be folded into the AA-Omniscience score as if they measured the same thing.

A benchmark is a controlled comparison, not a field study of every production deployment. A leaderboard result can help narrow a shortlist, but it cannot tell a team whether the model is reliable enough for its own use. That requires testing with the actual sources, prompts, user needs and consequences involved.

Closed-book recall is not grounded summarisation

AA-Omniscience tests what the model can answer without being given source text. This is sometimes called parametric recall: the model’s answer draws on information encoded during training or other available model knowledge. It is different from a grounded task, where the system is asked to work from documents supplied in the prompt or retrieved for it.

Grounded summarisation is often the more relevant comparison for a company assistant that retrieves product documentation, help-centre articles or policy pages. In that setup, the model should base its response on the retrieved material and avoid adding unsupported claims. The system can still fail: retrieval may find the wrong passage, the source may be incomplete, or the model may misread it. But the task is different from recalling facts without sources.

Vectara’s Hallucination Evaluation Model and leaderboard is an example of work focused on factual consistency in generated summaries. Its results should not be directly compared with AA-Omniscience’s closed-book rate. The inputs, task and scoring differ. A low rate on a summarisation test does not prove that a model will reliably answer a question about a fact absent from the supplied documents.

This distinction is central to evaluating search and answer engines. A model may be strong at using a clear, current source and weaker at recalling obscure facts without one. Conversely, it may know a fact but fail to retrieve the relevant page or correctly attribute it. For a brand, “the model knows us” is not a single capability. It includes discoverability of information, retrieval, interpretation, answer construction and citation.

When the announcement happened and who it affects

Third-party reports say Google announced Gemini 4 Carbon on 28 October 2026. Reporting from Tom’s Hardware and VentureBeat describes a limited initial release, with access focused on security teams and enterprise partners through a pre-release oversight programme. The reports say broader developer access is expected after initial guardrail evaluations. Availability can change, so verify access status with current provider information before planning a deployment.

That limited access means the immediate audience is not every marketer or developer looking to switch models. The people most directly affected are enterprise teams assessing AI for security work, research, document analysis and other higher-stakes tasks; developers planning future integrations; and AI search teams watching how model capabilities may affect generated answers. For most brands, the result is a signal to update evaluation plans, not a reason to assume Carbon is already serving every Gemini answer.

Search marketers should also separate benchmark performance from product distribution. The AA-Omniscience result does not establish which model is used for a particular Google AI Overview, Gemini conversation or other product feature. Nor does it show that a stronger recall score will automatically increase citations to a company’s pages. The model’s access to current web sources, product routing and retrieval system can all affect what a user sees.

For agencies, the news is useful as a prompt to review how client information is represented in public sources. For B2B SaaS and ecommerce teams, it is a reminder that accurate, consistent facts matter even when a model is better at deciding when not to guess. Clear pages can help systems retrieve the right details; conflicting details can still cause omission or confusion.

What marketers should do now

Treat the benchmark as a reason to strengthen your own evidence and monitoring, not as a reason to optimise for one model’s leaderboard. Marketers cannot directly control model parameters or benchmark scores. They can make brand information easier to find, verify and quote, and they can test whether AI systems are representing it correctly.

1. Find your important factual claims

Start with facts that could change a recommendation or purchasing decision. These may include product capabilities, supported integrations, pricing, eligibility, locations served, availability, policies, technical limits and named customer segments. Put them in a simple inventory with the current approved answer and the page that supports it.

Prioritise claims that appear in several places. If your pricing page says one thing and a product page or older blog post says another, the model may encounter competing information. Resolve the inconsistency at the source. Do not try to compensate for contradictory pages by adding more content that repeats both versions.

For product and service pages, write explicit answers to likely customer questions. A page should make clear what a feature does, who it is for, what it requires and what it does not do. Specific wording helps both human readers and systems that need to extract a clear fact. Avoid making a claim broader than the evidence supports.

2. Create a reliable source of truth

Choose the canonical page for each core fact. Keep details current, use consistent names for products and features, and link related information where a reader needs it. If a product is available only in certain regions or plans, state the limits alongside the claim rather than burying them in a separate document.

Structured data can help describe page content to search systems, but it does not replace clear visible copy. Mark up facts that genuinely appear on the page, and keep the structured version aligned with the page itself. Avoid adding schema solely to imply a capability or rating that users cannot verify.

This work is useful whether an answer engine relies on a model’s recalled knowledge, retrieved pages or a mix of both. A stronger source of truth reduces ambiguity for people as well as machines. It also gives your team something concrete to check when an AI system gets a detail wrong.

3. Test prompts, not just model names

Build a small test set from real customer questions, sales calls, support tickets and search queries. Include direct questions such as “What does the product integrate with?” and comparison questions such as “Which option supports this workflow?” Add questions where the correct response should be cautious, such as asking about an unannounced feature or an unsupported guarantee.

Run the same prompts across the systems that matter to your audience, and record the date, product or model where visible, prompt, response, source links and whether the answer is accurate. Repeat tests over time. The purpose is not to produce a universal accuracy percentage from a handful of prompts. It is to identify recurring gaps, incorrect claims, missing citations and changes in visibility.

A practical review can label each answer as accurate, partly accurate, incorrect, unsupported, or no answer. Keep “not mentioned” separate from “wrong”. This reflects the benchmark lesson: abstention and hallucination are different outcomes, and a brand can be absent without being misrepresented.

4. Check whether answers have evidence

When an AI system cites sources, inspect the cited page. Does the page support the exact claim? Is the relevant detail easy to locate? Is the page current? If the model gives a correct statement but attributes it to a weak or outdated source, that is still a quality issue worth investigating.

When the system provides no citations, do not assume its answer came from your website. It may rely on other published material or its learned knowledge. Your test log should distinguish what you observed from what you can establish. Avoid claiming that a particular page caused an answer unless you have evidence for that connection.

For teams monitoring AI mentions and citations across answer engines, SearchFIT’s resources and AI search visibility articles can help shape a repeatable measurement process. The goal is to connect observations to pages and questions that your team can improve, rather than treating a single model response as a complete picture of visibility.

5. Match the model and safeguards to the risk

An 11% benchmark hallucination rate is not an acceptable-error threshold for every business task. If an incorrect response could affect legal, financial, health or safety decisions, use stronger controls than a general-purpose model score. These may include retrieval from approved sources, citations, restricted answer scope, human review and a clear escalation path when information is missing.

For lower-risk uses, such as drafting or summarising internal notes, teams may accept a different balance between response coverage and caution. Even there, define what must be checked before publication or action. A model that scores well on one benchmark can still fail on a particular document, prompt or niche subject.

Test workflows end to end. The model is only one component. Consider how information is retrieved, whether access permissions are respected, how prompts are constructed, what happens when no relevant source is found and how users can report an error. Re-test after a model or retrieval change, because results can shift when any part of the setup changes.

6. Keep benchmark claims precise in your reporting

If you share the Carbon result with clients or colleagues, name the benchmark and the model version. Say that Artificial Analysis reported about an 11% hallucination rate on AA-Omniscience v1.1, with 58% accuracy and a net score of 54. Avoid shortening that to “Gemini hallucinates 11% of the time” without explaining the test conditions.

When you compare models, use figures from the same test and note any rounding or version differences. Do not compare a closed-book factual recall rate with a grounded summarisation rate as if they were equivalent. Include abstentions and accuracy where available. This helps decision-makers see the trade-off rather than overreact to a single headline number.

For agencies, that precision is part of sound client advice. For in-house teams, it prevents a benchmark from becoming an unsupported procurement promise. Your own test results should likewise include the tested prompts and date, not just a score detached from its conditions.

What to watch next

The next useful signals are broader access, updated benchmark entries and results from independent tests on tasks that reflect real deployments. Watch whether the reported performance holds across different domains, whether Carbon’s abstention behaviour is documented in more detail, and how it performs when supplied with sources rather than asked to recall facts without them.

Also watch for changes in model access and deployment. The reported initial restrictions mean the benchmark result may arrive before broad developer experience. A score can help teams plan, but practical selection depends on availability, latency, cost, data controls, integration requirements and performance on the intended task. The information available in the third-party reports is not enough to settle those questions for every organisation.

For AI search visibility, the more immediate lesson is not that one model will now cite more brands. The benchmark measures factual recall, not brand citation share or website rankings. It does, however, underline why precise public information is valuable: a model that is willing to abstain may leave a poorly documented detail out, while a model that answers more often can still state an incorrect one.

Use the news to check whether your organisation has a clear, current and consistent source for its key facts. Then monitor what answer engines say, keep evidence for each observation and update content where a real ambiguity or error exists. To compare SearchFIT with other approaches to AI visibility and content operations, see SearchFIT vs agencies.