An AI Reviewer Scored Four AI-Written Posts: What It Caught and Got Wrong
You asked an AI to write a post, it looks fine, and you still don't trust it enough to hit publish. I had the same problem with four posts on one day, so I gave them to a second AI whose only job was to find what was wrong. It caught four mistakes that would have embarrassed me. It also got two things wrong itself.
Key Takeaways
- On November 15, 2026, a reviewer agent scored four AI-written drafts between 71 and 77 out of 100. After one round of fixes each scored 81 or 82. None reached the 90 the rubric asks for. The two rounds were scored with slightly different instructions, so read the gain as rough.
- The score was the least useful part. The useful part was a short list of "do not publish until this is fixed" items, each with a line number.
- The reviewer caught one answer that implied the opposite of the official rule, and one headline that called a price "official" when nobody had seen the official page.
- The reviewer was wrong twice and the writer was wrong once. Each time, the answer came from going back to the source, and twice that meant me opening it.
- The reviewer could not open links. I checked those separately: 52 links, all working.
How did I set up the test?
I used two separate AI agents in Claude Code: one wrote, one reviewed, and the reviewer never saw the writer's instructions. Claude Code is Anthropic's AI tool that works with files on your computer. An "agent" here is just a separate AI session with its own instructions.
The drafts were four Korean posts for my sister blog. They leaned on official material where the writer could get it: an exam notice, a city press release, a law database. The two-agent habit comes from an earlier setup I wrote about in how I make Claude Code and Codex share one worklog.
| Item | What I used |
|---|---|
| Writers | Four Claude Code subagents, one per post. Each produced the draft, a source list and a fact list. |
| Reviewer | The blog-reviewer agent from the open-source claude-blog plugin, version 2.3.1. It scores five categories that add up to 100. |
| Pass mark | 90 or higher with zero blocking issues. That is the plugin's own rule. |
| Rounds | Two. Review, fix, review again. |
| What the reviewer could read | The draft, the source list, the fact list and, for one post, the official PDF. It had no web access. |
| Not measured | I did not record which model version or plan tier ran each agent. I did not test any other reviewer tool. |
One flaw in my setup: in round one I forgot to hand the reviewer the plugin's full scoring guide, so it used its built-in rubric. I fixed that in round two. So the two rounds are not a like-for-like comparison, and I can't say how much of the gain came from that.
What scores did the four posts get?
All four posts scored between 71 and 77 in the first review and 81 or 82 after one round of fixes. None reached the 90 pass mark.
| Post topic | Round 1 | Round 2 |
|---|---|---|
| A national exam cheating case and what the law says | 73 | 82 |
| What you cannot bring into the college entrance exam | 77 | 81 |
| A baseball stadium closing and its replacement schedule | 76 | 82 |
| A fact-check of a car price headline | 71 | 81 |
One caution: the first review ran without the plugin's full scoring guide. Part of the change may come from the scoring, not the fixes.
The reviewer named two things I could not fix in the text: each post had only one internal link, and none had images. In the plugin's rubric those are worth 7 points out of 100, so they explain part of the gap, not all of it.
The fastest review took about two and a half minutes. The slowest took about five and a half.
What did the reviewer catch that I would have missed?
The reviewer caught four errors a reader might have acted on, and each came with a line number.
- An answer that pointed the wrong way. One FAQ answer read as if switching your phone off, putting it in your bag and handing in the bag was fine. The official notice says banned items go to the supervisor separately. A student who believed my post could have had the exam voided.
- A headline that said "official" without the official page. The writer could not open the company's site and took the price from news reports. The headline still called it the official price.
- A conclusion with no source behind it. One post said a target year "has not changed since 2023". The 2023 press release in my own source list gave no completion year.
- A sentence that came only from a search summary. In its revision note the writer admitted it had never opened an article containing it. I deleted it.
The drafts shared smaller problems too. Three of the four had no links to their sources, and the fourth only listed them at the bottom. Two had openings that sounded more certain than their fact lists. All four were signed with a pen name and nothing behind it.
Where did the reviewer get it wrong?
The reviewer was wrong twice, and both times it sounded just as sure as when it was right.
| Who was wrong | What it claimed | What the source showed |
|---|---|---|
| Reviewer | It "remembered" that a law says a committee shall be set up, and told the writer to fix the wording. | The official law text says the committee may be set up. The writer had it right. |
| Reviewer | It marked two details as unsourced because they were not in my fact list. | Both details were in the newspaper article I had cited. The reviewer could only read local files, so it never saw the article. |
| Writer | It told me a sentence the reviewer quoted "does not exist in the PDF". | The sentence was on page 2. The page was a picture, so a text search could not see it. I opened the page and read it. |
That third one matters most to me. Two AI agents disagreed about what a document said, and one of them had searched the full text. The only fix was a person opening page 2.
Who is a second AI reviewer good for?
A second AI reviewer is worth it if a wrong date, rule or price could cost your reader something. It also needs a source list next to each draft.
- It suits you if your drafts cite documents you can save as files, and you can spare time for a review and one round of fixes. My reviews ran about three to five minutes each. I did not time the fixes.
- It will frustrate you if you want a number that says "safe to publish". Mine never said that.
- It does not replace opening the source. All three disagreements in my test were settled that way.
I only tested one reviewer tool, so I can't tell you it's the best one. I can tell you what a reviewer needs from you to be useful.
What should you hand a reviewer so it can do its job?
Give the reviewer three things: the draft, the list of sources and the facts you took from each one. Then tell it to point at lines, not give opinions.
I didn't use this exact wording in my test. It's what I'd ask for now. I ran everything in Claude Code and haven't tried this in ChatGPT or Copilot. Change the bracketed parts.
Review this draft as someone who did not write it. Files: [draft], [source list with URLs], [fact list: each number, date and rule with its source]. 1. For every number, date and rule in the draft, find it in the fact list. List the ones you cannot find. 2. List anything a reader should not see until it is fixed. Give the line and a replacement sentence. 3. Tell me where the opening or the title sounds more certain than the fact list. 4. If you rely on your own memory for a fact, say so. I will check those myself. You cannot open links. Do not say a link works.
Then do two things yourself. Run a link check, because the reviewer cannot. And when the writer and the reviewer disagree, open the source before you believe either.
What should you ask for before you publish an AI draft?
Ask for the fact list first. Take one AI-written draft you were about to publish and have the AI list every number, date and rule, each with the page it came from. If the AI cannot produce that list, you have found your first problem before any reviewer runs.
Frequently Asked Questions
Can the same AI write and review in one chat?
I did not test that. I used separate agents so the reviewer would not see the writer's instructions or reasoning. If you try a single chat, treat the result with more doubt, because the reviewer already "knows" what the writer meant.
Why did no post reach 90?
The rubric gives points for several internal links and for images, and a small blog with few posts cannot earn them. After the second round, two of the four reports said that with every remaining fix the posts might reach the low 90s, with almost no margin. I did not run a third round to find out.
Did the fixes after round two get reviewed again?
No. The reviews were eating into my daily usage allowance, so I fixed the remaining accuracy items directly and checked the live pages myself. A third review round would have been cleaner.
Does this work for posts in English?
The four drafts in this test were Korean. The reviewer's rubric and reports mixed English and Korean without trouble, but I have not run the same before-and-after test on English drafts yet.
Tested November 15, 2026, in Claude Code with the claude-blog plugin 2.3.1. This post was drafted with the help of AI writing tools, based on my own notes from that session.
JH-A, Charrette AI Room editor (About, Editorial Policy)
Comments
Post a Comment