I Ran an Automated Checker on My AI-Assisted Posts for a Month: 3 Real Errors, 9 False AlarmsA small fact checker built from one reference file and 21 rules. What it caught, what it got wrong, and why nobody looked at its alarms for three weeks

01Why I needed a checker

By late September this blog had 40 posts. The same model’s price appeared in several of them, and when a company changed its prices, only some of those posts got updated. For example, after one model stopped being sold, one post was updated with the new information while two others still showed the old price. Nobody can reread every post every time.

So I split the work. The correct value of each number lives in one file only. The machine reads every post daily and looks for numbers that differ from that file. A person looks only at the candidates it finds.

02How it works: one reference file, 21 rules, every morning at 7

The reference file. For each model I recorded its name and other spellings, the input and output price per million tokens, the release date, whether the weights are public, and the source and date I checked each value. It now holds 25 models and 17 events. An entry looks roughly like this.

"gpt-6-astra": {
  "aliases": ["GPT-6 Astra"],
  "open_weights": false,
  "api_price": { "input": [10], "output": [50], "cached_input": [1.0] },
  "cached_source": "OpenAI model docs", "cached_checked": "2026-10-24"
}

The rules. Beyond numbers, some kinds of sentences go wrong again and again. I turned those into 21 rules. A few of them:

• Claude Code’s /clear and /compact tidy up the conversation context. They do not reset the account’s usage limits.
• A token price one-fifth as high does not mean the same job costs one-fifth as much. Token counts and retries differ.
• A confidence score from an AI is a signal, not permission to move money or touch accounts.
• Absolute phrasing such as “there is no risk” is flagged so it can be rewritten as “designed to reduce the risk.”

Running it. GitHub Actions runs the checker every morning at 7 a.m. Toronto time. If it finds problems, it opens an issue on GitHub and saves the results as a table. The top of that table always carries the same sentence: “Zero findings only means nothing broke the registered rules and reference values. It does not mean there are no errors the rules don’t cover.”

03I started with sentences that were wrong on purpose

Before running the checker on real posts, I wrote 17 deliberately wrong sentences and 14 correct ones. It has to catch every wrong sentence and let every correct one pass. If it fails that test, its results on real posts can’t be trusted.

The first bug turned up right there. When a Korean particle came straight after an English model name (as in “Sonnet 5는”, “Sonnet 5 is” with the Korean topic marker attached), the checker could not find the name. Python’s regular-expression word boundary treats Korean letters as word characters, so it did not see a boundary between “5” and “는”. Testing in English alone would have missed it.

There was a funny moment too. The first design for this checker came from a proposal that appeared to have been written by another AI tool, and that proposal itself contained three factual errors. It credited one model’s training cost to a different model, got another model’s release date wrong by about three months, and suggested a data format that would not actually work. Fact-checking was needed from the moment I started building the fact checker.

04First run: 3 real errors in 40 posts

I ran it on all 40 published posts for the first time on the night of September 25. It found three things.

• An outdated price. The price of DeepSeek V4-Flash, which was no longer sold, was still in two open-weight posts (Korean and English). Another post on the same blog already had the updated information, so the posts contradicted each other.
• A number without a source. Kimi K3’s output price had no verified source.
• An overly absolute claim. One sentence said that if you only let an agent compare options, there is no payment risk.

All three were fixed that night. I rechecked the prices in the official pricing docs and rewrote the absolute claim with a narrower scope. Rereading 40 posts by hand would have taken hours.

05Nine false alarms and three weeks of neglect

From September 30 the checker started failing every day, with nine findings. A failure email arrived each morning, but busy days followed one another and nobody opened them. On October 13 I tightened the site’s security settings, and that stopped the checker from reading post bodies at all. That, too, was buried among the failure emails. (The broader story of these alarms is in my month of letting an AI agent run the blog.)

On October 24 I checked the nine findings against the original sources one by one. In all nine, the post’s numbers were right. Here is what went wrong.

• The reference file was incomplete. It had no field for discounted cached-input prices (for example, $0.25 per million tokens for Claude Fable 5.1 cache reads) or for an evaluator’s cost per task (for example, $3.76). The checker treated every number missing from the file as a mismatch.
• It attached numbers to the wrong model. A paragraph about Grok 4.7 contained a number for GPT-6.1 Sol, the model it was compared with, and the checker read it as Grok 4.7’s. The checker made exactly the mistake AI drafts make.
• It misread the context. In a scrolling ticker of stats, it read the phrase “public models” as a release announcement and “Open-weight gap” as a claim that the weights were public.

I made four fixes. I added cached-input prices, service-tier prices and cost-per-task fields to the reference file, each with a source. A new model name now ends the previous model’s number segment. Ticker items are read one by one. Negative sentences (“can’t be guaranteed”) no longer trigger the absolute-claim rule. For the security block, the checker now reads posts from their public pages. The test suite grew to 51 tests.

Run again after the fixes, it read 66 posts in about 50 seconds, and the findings dropped from 29 to 5. None of the five is classed as an error.

06A person reviews the last five

The five that remain show exactly where the checker’s job ends.

• One needs checking. A sentence saying three models were “tied for first.” That is true on one leaderboard by one measure, so a person has to see whether the sentence makes that scope clear.
• One needed fixing. A phrase saying people “cannot access” a review environment. It repeated a company’s design description far too absolutely, so I rewrote it to attribute the claim to the company.
• Three stay as they are. A sentence quoting a company’s own slogan in quotation marks, a sentence that used the words “cannot be accessed” while describing a server, and a table-of-contents title, “Things it never does.” They trip a rule but are not claims.

Tighter rules could cut these down further. But I would rather keep the list small enough for a person to judge in five minutes than chase it down to zero.

077 lessons from building it

1. The reference file is the checker’s skill. A number missing from the file shows up as wrong even when it is right. When posts start using a new kind of number (cached prices, cost per task), add the field to the file first.
2. Test with wrong sentences first. Include problems that only appear in real posts, such as Korean particles, in the test sentences too.
3. Triage alarms the same day. Is it the post or the checker? Leave false alarms for even a few days and real alarms get buried among them.
4. Grade the findings. Error, needs checking, needs a source, style. When everything is the same red light, nothing feels urgent.
5. Zero findings is not innocence. The checker finds only what its rules describe. Say so at the top of every report.
6. When you change site settings, check the checker. This time a single security change blinded it.
7. Build a rule checker before an AI editor. The same day, I also built an editor that lets AI rewrite posts, but I have never run it on a real post. A rule checker can explain its results, so I could see right away what was flagged and why.

A blog written with AI does not need a grand system. Collect the ten numbers you use most in one file and check that every post agrees with it. That is enough to start. If you are curious about what an AI agent should check before it declares a job done, read the story of the AI agent harness as well.