The News Reviewer — A Quality Gate Before the PR

📁 Computer

Part of the AI Software House series.

In short: Before opening the PR, one reviewer call compares the English article and both Chinese translations with the source. It reports specific factual, wording, or language-register problems, and the pipeline reruns only the stage responsible.


What can go wrong

By this point the pipeline produced three versions of every story. A human still had the final decision, but too much review time was going into mechanical checks: comparing version numbers, spotting a stray Simplified character, or noticing Cantonese phrasing in the Taiwan version.

The failures I had seen were repetitive enough to automate:

Hallucinated facts. The writer called fetch_url and got the article, but the LLM still occasionally invents a version number ("version 4.2" when the source says "version 4.1"), misattributes a quote, or embellishes a statistic. These are hard to catch on a quick PR review.

LLM artefacts. Awkward phrasing that's grammatically correct but obviously machine-generated. A headline that doesn't match the article body. The writer's analytical commentary accidentally included in the published text.

Simplified Chinese in Traditional Chinese outputs. This is the most reproducible failure mode. The translator is instructed to use Traditional characters, but certain words appear almost exclusively in Simplified form in the LLM's training data: 国 instead of 國, 软 instead of 軟, 网 instead of 網. They slip through, especially in compound words.

Register contamination. The zh-hk translation uses Mandarin vocabulary instead of Cantonese (現在 instead of 而家, 很多 instead of 好多). The zh-tw translation contains Cantonese particles (係、嘅、喺) that don't belong in a formal broadsheet register.


Pipeline position

The reviewer runs after both translation stages and before the PR stage:

stages:
  - discuss_news_analysis
  - news_writer
  - discuss_news_draft
  - news_editor
  - translate_cantonese
  - translate_zh_traditional
  - news_reviewer            # ← all three outputs available here
  - news_article_pr

The position is important. After both translation stages, one call can inspect all three files. Putting it earlier would require another reviewer just for translation errors.

The reviewer also fetches the source article directly — the source_url from the English article's YAML frontmatter — to compare facts at review time, not just at write time. By the time the reviewer runs, the writer's fetch_url call may have been hours or a pipeline restart ago.


One call, multiple check domains

The NewsReviewerAgent runs a single LLM call with the full English article, both translations, and the fetched source content in the prompt. The role prompt (roles/news_reviewer.md) defines four check domains:

English — fact plausibility: version numbers, dates, product names match the source; no invented quotes or statistics; no agent commentary in the article body.

English — wording QA: no awkward phrasing; headline matches content; article stays on topic.

zh-hk — Written Cantonese: all characters are Traditional (no Simplified: 国→國, 软→軟, 网→網); uses Cantonese vocabulary and particles (係, 唔係, 喺, 咁, 嘅, 咗); no Mainland Mandarin-only patterns.

zh-tw — Formal Traditional Chinese: all characters are Traditional; uses Taiwanese Mandarin vocabulary (軟體 not 软件, 影片 not 视频, 網路 not 网络); no Cantonese colloquialisms; no Mainland vocabulary patterns.

The output format is structured for machine parsing:

VERDICT: PASS | NEEDS_REVISION
ISSUES:
- [FACT] Version number "4.2" not found in source — source says "4.1"
- [ZH_HK] Simplified character found: "软" should be "軟"
- [ZH_TW] Mainland vocabulary: "软件" should be "軟體"
CONFIDENCE: high | medium | low

A low-confidence PASS still continues to the human review. I do not want an uncertain machine judgement to block the article; only an explicit NEEDS_REVISION starts a retry.


Retry logic

The reviewer distinguishes between English issues and translation-only issues because the fix paths are different.

English issues ([FACT] or [WORDING] in the issues list) cascade. If the English article has a wrong fact, both translations inherited that error. The full English-to-PR chain reruns:

news_reviewer (NEEDS_REVISION, English issues)
  → news_editor  (reviewer notes injected into prompt)
  → translate_cantonese   (redo)
  → translate_zh_traditional  (redo)
  → news_reviewer

Translation-only issues are isolated. A Simplified character in zh-hk doesn't require re-editing the English article:

news_reviewer (NEEDS_REVISION, ZH_HK only)
  → translate_cantonese  (reviewer notes injected)
  → news_reviewer

The reviewer notes are injected into the retried stage's prompt — same pattern as editorial notes into the writer. The translator sees exactly what was wrong with the previous translation and is told to fix it.

Maximum retries default to 2 per loop (English loop and each translation loop independently). After max retries, the article proceeds to PR regardless. The reviewer is a quality signal, not a blocker. An article that failed review twice still gets human eyes on it before merging.


The zh-hk check in detail

The Simplified character check is the most mechanical and the most reliable. The reviewer is instructed to flag specific character substitutions: 国→國, 软→軟, 网→網, 爱→愛, 时→時, 来→來, 这→這. These are the characters most likely to appear in Simplified form in CJK text generated by LLMs with heavy Simplified training data.

The Cantonese vocabulary check is less mechanical. The reviewer looks for the presence of characteristic Cantonese particles — 係, 唔係, 喺, 嘅, 咗, 囉, 咩 — as positive signals that the text is actually written in Cantonese register. Absence of these markers in a zh-hk article is a negative signal even if no Mandarin vocabulary is explicitly found.

A zh-hk article can use only Traditional characters and still read like translated Putonghua. That is why the reviewer checks vocabulary and particles as well as the character set.


The zh-tw check in detail

The Taiwanese vocabulary check is a list of known regional variants where the Mainland/Simplified form is wrong for Taiwan context:

  • 軟體 (Taiwan) vs 软件 (Mainland) — software
  • 網路 (Taiwan) vs 网络 (Mainland) — network
  • 影片 (Taiwan) vs 视频 (Mainland) — video
  • 程式 (Taiwan) vs 程序 (Mainland, also ambiguous) — program

The reviewer also checks the inverse of the zh-hk test: a zh-tw article should not contain Cantonese particles. If 係 or 嘅 appear in zh-tw output, that's a register contamination error — Cantonese from the translator's training data bleeding into a formal Mandarin context.


What the reviewer doesn't do

The reviewer reports problems but never rewrites the article.

I tried to keep the role narrow. The writer and editor already own the copy; the reviewer only decides whether one of them needs another attempt.

It also means the verdict format stays simple enough to be machine-parsed reliably. A verdict and a list of issues is easy to parse. An edited article that might or might not have fixed all the issues is harder to reason about.

If the reviewer can't parse its own output, the safe default applies: treat it as PASS, log a warning, continue to PR. The reviewer never blocks the pipeline on its own failure.


Error handling

Source fetch failure is graceful: if source_url is missing or the fetch returns an error, the fact check is skipped and only wording + character checks run. This prevents a network failure from blocking the entire quality gate.

Unparseable reviewer output: treat as PASS, log warning. Same principle as verdict parsing everywhere else in the pipeline — fail-open.

Reviewer agent error (timeout, LLM failure): log warning, skip reviewer, continue to PR. An article without a reviewer pass is better than a blocked pipeline.


The complete pipeline

stages:
  - news_triage
  - discuss_news_analysis
  - news_writer
  - discuss_news_draft
  - news_editor
  - translate_cantonese
  - translate_zh_traditional
  - news_reviewer
  - news_article_pr

Nine stages from story intake to PR. The triage gate means the majority of incoming stories never reach the writer. The reviewer gate means the minority that do reach the PR are checked for the most common failure modes. The human reviewer on the PR is approving content quality, not hunting for Simplified characters.

At this stage the whole path exists: intake, research, writing, two translations, automated checking, and finally a human merge. The remaining work is mostly learning which rules catch real problems and which ones only create noise.


Related reading: Batch Intake Triage