How accurate is NewsMarkets?

Every headline and summary here is written by an AI model, and every link between a story and a prediction market is chosen by software. No person reads them first. So we measure how often they are wrong, on a random sample, and publish the result with its uncertainty. This audit was run on 22 October 2026.

Headlines and summaries

We drew 200 published stories at random (100 told by two or more publishers, 100 by one) and graded each headline and summary against the articles the model was shown when it wrote them.

Market matches

We drew 200 story-to-market links at random and graded whether the market asks about what the story reports.

Counting a same-subject market as a match, 86% (80–90%) are matches. The distinction matters: a market on the same company is not the same as a market on what the story reports, which is why every story page shows the market's own question.

What a mistake looks like

• Summary: Trump Campaigns in Republican Strongholds Amid ICE Controversy
  Summary says Noem faces questions; the lead says Homan was asked about the ICE killing of Renee Good. Noem is not mentioned.
• Summary: Glasgow council workers face pay cuts in fire-and-rehire plan after union talks break down
  Latest article says council reached a deal with Unison averting fire-and-rehire; headline and summary present the earlier breakdown as current.
• Summary: Tennessee Defeats Alabama in Dominant SEC Rivalry Win
  The 26-10 score is not stated in any article shown.
• Market match: Texas Halts Data Center Permits Amid Grid and Water Audits → "Will Tennessee regulate data center water use?: Yes"
  Market is about Tennessee; story is about Texas data center permits.
• Market match: Barcelona defeat Real Madrid despite missing key players → "Will Real Madrid finish in the top 2 of the 2026-27 LaLiga season?"
  Story is a women's team match; market is the men's LaLiga top-2 finish.

How it was done

The sample was drawn at random from stories and links on the site on 22 October 2026. Each item was graded by an AI model (Claude, an AI model made by Anthropic) working from a written rubric, shown only the evidence the software itself had: for a summary, the member articles' titles and short leads; for a match, the story and the market's question. The auditors were a different model from the one that wrote the summaries, and each worked blind to the others. A second auditor regraded a subset without seeing the first's labels: they agreed on 52 of 60 summaries and 56 of 60 matches. Because "unsupported" is the rare, serious call, every summary marked that way (8) was regraded blind among 24 others: the second auditor confirmed 7 of the 8 and flagged 0 of the 24 others.

What this does not tell you

• No person graded these. The auditors are AI models. They are a check, not a substitute for a human editor, and models can share blind spots.
• The evidence is limited. Auditors saw only the article titles and short leads we store, not the full articles, so a detail that is true but only in the full text can be marked unsupported.
• The samples are small. The range after each percentage is a 95% confidence interval; a few hundred items cannot pin a rate down closely.
• It is a snapshot. The headline-writing model, the matching rules and the stories all change. This will be rerun, and the date above is the date of these numbers.

Errors we have found and fixed are in the corrections log. To report one, write to newsmarkets@simbasystems.com. How AI is used here: how it works.