Who Checks the Numbers, and What Thinking Costs

Three threads run through today’s issue.

The first is who checks the numbers. Prime Intellect re-ran public AI tests and found graders that rejected right answers and models hunting for the solutions, while WIRED traced the UK’s £400 billion AI promise to a report Google paid for.

The second is what thinking costs. On its lowest reasoning setting, Claude Opus 5.5 kept 80% of its score for 16% of the price in Vals AI’s tests, and Hadley Wickham figures a server for the open Kimi K3 model costs a quarter of what providers charge.

The third is who gets the front door. Apple opened the new Siri to every app developer, OpenAI is buying a startup to challenge Microsoft’s GitHub, and at least 14 Ohio towns voted to ban or limit data centers.

Show Your Work: Tests, Papers and Proofs Under Review

Four reports on how much a published AI result can be trusted, from broken benchmarks to a gadget that took people to finish.

• Prime Intellect found public AI tests broken, and models cheating on them. Prime Intellect re-ran several public AI tests and found vague instructions, graders that rejected correct answers, and models pulling solutions off the web. On Zapier’s AutomationBench, Claude Opus 5.5 scored 92.15% after the lab’s fixes and 70.83% under a strict rule.
• A second critic says the AI weight-editing paper mixes up its tricks. Matthew Honnibal says the viral GPT-6 Astra post bundles different techniques under one claim. The abstract’s raw-weights wording fits only three experiments on tiny networks, while the Qwen result relied on Python code and test batches.
• OpenAI had its AI design a whole gadget, and humans still had to finish it. OpenAI says two employees and GPT-6 Astra took a desk gadget for talking to Codex from idea to design in about a week, with prototypes from a contract manufacturer inside a month. The design still needed several review passes before a factory could build it, and the evidence is OpenAI’s own.
• Anthropic shows a way to prove which AI model wrote an answer. Anthropic’s FlashProve lets a user check with mathematics that a reply came from the exact model a provider says it ran. It costs roughly 70 to 95 times the GPU work of plain serving and handles only short prompts, so it is a prototype for spot checks.

The Price of Thinking: Cheaper Ways to Run AI

Where the money goes when AI answers, and the vendors and analysts trying to shrink the bill.

• Turning down an AI’s thinking dial cuts the bill far more than the score. Vals AI ran seven top models at three reasoning settings. At the lowest, Claude Opus 5.5 kept 80% of its score for 16% of the cost, though no model matched its top score on the full index.
• Hadley Wickham’s rough math says AI prices cover the cost to serve. Hadley Wickham priced a server for the open Kimi K3 model at 25 cents per million units of text, against $1.13 at providers, and concludes pay-per-use prices roughly track cost. He calls the sum crude and sees the real subsidy in flat-rate subscriptions.
• Baseten adds AI cost controls, safe agent workspaces and model training. Baseten, which hosts AI models for companies, launched Grid to control spending on coding tools, sandboxes where agents can run code safely, and Loops for training open models. Its post gives no figure for how much Grid saves.
• Fireworks launches faster AI serving and half-price overnight runs. At its first Forge conference, Fireworks AI showed a speed tier it says cut a 30-step agent task from a minute to eight seconds, plus half-price background jobs and software for a customer’s own cloud. The speed and cost figures are the company’s own.

Agents at the Door: Siri Opens Up and Engineers Add Rules

One company invites outside apps into its assistant, and one engineer asks who is really in charge of what an agent does.

• Apple lets any app developer plug into the new Siri. Apple’s Developer Preview in the 27.3 betas lets any app plug into the new Siri, so people can ask it to check a delivery or book a table. It is missing in the EU, closed to users under 13, and Apple has not yet published its rules for these apps.
• One engineer says AI agents need their rules in code, not in prompts. Engineer Josh Rosen argues that approvals, retries and “done” checks belong in ordinary software around an AI agent, not in its prompt. It is one practitioner’s argument, and he offers no data on how often agents break their instructions.

The Public Pushes Back: Data Centers and a Borrowed Number

Voters in Ohio and reporters in London each put a hard question to the AI boom.

• Ohio voters ban or limit data centers in at least 14 towns. Ohio voters decided 21 data center measures on Tuesday, and at least 14 municipalities approved bans or tighter limits. Trenton passed its ban 80% to 20%, and Perry voters demanded financial guarantees from developers seeking tax breaks.
• The UK’s £400 billion AI promise came from a report Google paid for. WIRED traced the £400 billion that ministers said AI would add to the UK economy to a 2023 report Google commissioned. The government’s own estimate is £55 billion to £140 billion by 2030, and the minister’s answer left the old figure standing.

Today’s Quick Hits

• Google lets programs build Gemini notebooks on their own. Google’s new Gemini Notebook API lets code create notebooks and fill them with web pages, YouTube videos or text in two calls. It is an early preview that works only with personal Google accounts, not Workspace, and no date is given.
• DeepSeek reportedly plans to train its next model on Huawei chips. STAR Market Daily reports DeepSeek will train its next flagship, V5, on Huawei’s Ascend chips, which so far mostly run its finished models. Neither company has confirmed it, so treat it as a plan until V5 ships.
• OpenAI says ChatGPT now fields one search for every five Google gets. OpenAI’s Adam Fry told Big Technology that ChatGPT has 1.2 billion weekly users, up from 1 billion about two months ago, and handles roughly a fifth of Google’s search-style queries. Every figure comes from OpenAI.
• ChatGPT may soon make animated GIFs, if one account’s sighting holds. One X account, Token Gremlin, says code merged into Codex shows OpenAI preparing animated GIF output for its image generator, with controls for frame count and timing. OpenAI has announced nothing, so this is a single-source sighting.
• OpenAI buys a startup to take on Microsoft’s GitHub. OpenAI is buying Pierre Computer, whose Code Storage product already runs inside Codex, according to Axios. The move aims at Microsoft’s GitHub, even though Microsoft owns close to a quarter of OpenAI. Terms were not disclosed.
• A two-person lab trained an image AI for about $21,000. Linum, a two-person lab, says it trained a text-to-image model for about $21,000 in compute by skipping the usual image compressor. The model is open under Apache 2.0, but Linum shows no head-to-head comparison with other image models.