Batch Intake Triage β€” Evaluating Stories Together, Not One at a Time

πŸ“ Computer

Part of the AI Software House series.

In short: Per-story triage could say whether one article was acceptable, but not whether it was the best story waiting. Batch triage holds incoming items briefly, lets editors compare them together, and keeps the tracker-specific work behind a TrackerAdapter.


The isolation problem

editorial triage's triage solved the worst noise problem, but every discussion still saw only one story. The editors could ask β€œis this publishable?” They could not ask β€œis this worth doing before the other five stories waiting today?”

That difference showed up quickly. A middling story could pass on a quiet day even though the same editors would drop it immediately beside several stronger stories.

It also meant paying for a separate discussion for every item. To compare stories properly, I needed to let a small queue form first.


The TrackerAdapter abstraction

GitHub Issues are the current inbox, but I did not want batch triage itself to become GitHub code. I expect to use the same idea for other intake systems later, possibly JIRA.

@dataclass
class TriageItem:
    id: str           # issue number, ticket ID, etc.
    title: str
    body: str         # full content
    url: str
    created_at: datetime
    metadata: dict    # tracker-specific extras

class TrackerAdapter(ABC):
    @abstractmethod
    def list_pending(self) -> list[TriageItem]: ...

    @abstractmethod
    def approve(self, item: TriageItem, notes: str) -> None: ...

    @abstractmethod
    def skip(self, item: TriageItem, reason: str) -> None: ...

    @abstractmethod
    def is_approved(self, item_id: str) -> tuple[bool, str]: ...

TriageItem is the tracker-agnostic record. Everything the triage logic needs is in the dataclass; the metadata dict carries tracker-specific extras (GitHub labels, Jira status fields) without polluting the interface.

The GitHubTrackerAdapter maps this interface to GitHub Issues: list_pending() queries issues with the triage-pending label, approve() removes that label and adds triage-approved + the pipeline trigger label, skip() posts a comment and closes the issue. The is_approved() method is the bridge to the per-story triage fast-pass β€” more on that shortly.

A future JiraTrackerAdapter implements the same four methods using JQL and Jira transitions. The intake_triage.py script never changes.


Four triggers

The batch triage script is cron-runnable. It needs to decide when to actually fire. Four triggers are evaluated in order, and any one is sufficient:

1. --run flag (manual) β€” unconditional. Useful for testing or when you want to drain the pending queue immediately.

2. min_count β€” fire when β‰₯ N items are pending. Default is 5. Waits until there's a meaningful batch to compare.

3. max_age_hours β€” fire when the oldest pending item has been waiting β‰₯ N hours. Default is 6. Prevents stories from sitting in limbo too long even if volume is low.

4. schedule β€” fire if the cron expression matched within the last 65 minutes. Default is "0 9 * * *" (9am daily). The 65-minute window handles the case where the cron job runs slightly after the scheduled time.

def _should_fire(cfg, items, force=False, _now=None) -> bool:
    if not items:
        return False
    if force:
        return True
    if trigger.min_count is not None and len(items) >= trigger.min_count:
        return True
    if trigger.max_age_hours is not None:
        oldest = min(items, key=lambda x: x.created_at)
        age = (now - oldest.created_at).total_seconds() / 3600
        if age >= trigger.max_age_hours:
            return True
    if trigger.schedule:
        cron = croniter(trigger.schedule, now - timedelta(minutes=65))
        if cron.get_next(datetime) <= now:
            return True
    return False

With an empty queue, the script exits before creating a discussion or making any LLM call.


The batch context and discussion

When a trigger fires, items are sorted oldest-first, sliced to max_batch_size (default 10), and formatted into a single context block:

## Pending Items for Editorial Review
Triage scope: Focus areas: AI, software development tools, cybersecurity...
Item count: 5

--- ITEM 1 ---
Title: Apple releases iOS 18.5 with new AI features
URL: https://github.com/owner/ai-it-press/issues/42
Summary: Apple has released iOS 18.5 today, introducing on-device inference...

--- ITEM 2 ---
...

Body is truncated to body_preview_chars (default 300) per item. The editors see enough to evaluate each story, not so much that the context window fills before item 5.

The discussion (discussions/intake-triage.yaml) uses the same participants as per-story triage β€” editorial director, audience specialist, news editor β€” but the prompt context now shows all items simultaneously. Editors can explicitly compare: "Item 3 covers the same ground as Item 1 but with less depth β€” SKIP Item 3."

The moderator's final synthesis message emits per-item verdicts:

ITEM 1: PUBLISH
NOTES: Focus on the on-device inference angle for HK enterprise users.

ITEM 2: SKIP
NOTES: Acquisition too US-centric, no HK/Asia angle.

ITEM 3: PUBLISH
NOTES: Lead with privacy implications for Cantonese readers.

Parsed by _parse_batch_verdicts(text, item_count) into a list of (verdict, notes) tuples. Two regex passes: one looking for the full ITEM N: VERDICT\nNOTES: ... block, one fallback looking for bare ITEM N: VERDICT lines. Any item the parser can't match defaults to ("PUBLISH", "") β€” fail-open again, same principle as per-story triage.


What happens to max_batch_size overflow

If 14 items are pending and max_batch_size is 10, the oldest 10 are processed. The remaining 4 stay in triage-pending and will be included in the next run. Items don't expire or get auto-approved β€” they wait.

This keeps one busy period from producing an enormous prompt. If the remaining items are already old, max_age_hours may trigger another batch almost immediately, which is fineβ€”the point is to avoid leaving stale stories behind.


The fast-pass integration

Batch triage and per-story triage coexist. When the main pipeline processes a story that was already batch-triaged, the _stage_news_triage() method checks first:

def _stage_news_triage(self, result: PipelineResult) -> None:
    if self._intake_triage_approved(result):
        log.info("news_triage: batch-approved, fast-pass")
        return  # editorial_verdict/notes already set

    # Original per-story triage (unchanged)
    ...

def _intake_triage_approved(self, result: PipelineResult) -> bool:
    adapter = self._get_tracker_adapter()   # None if intake_triage disabled
    if not adapter:
        return False
    approved, notes = adapter.is_approved(str(result.issue_number))
    if approved:
        result.editorial_verdict = "PUBLISH"
        result.editorial_notes   = notes
    return approved

The is_approved() method checks the issue's labels for triage-approved, then fetches the most recent [INTAKE TRIAGE] comment to extract the editorial notes. The notes are the same ones the batch editors wrote β€” they carry forward to the writer just like per-story triage notes do.

When intake_triage.enabled: false (the default), _get_tracker_adapter() returns None, the fast-pass never fires, and per-story triage runs exactly as before. Zero impact on existing pipelines.


Human override

I did not add a separate override system. A person can add triage-approved in GitHub, and the ordinary fast-pass handles it. Manual and automated decisions use the same state.

Same logic for skipping: a human closes the issue manually. The pipeline watcher only picks up open issues, so a closed issue is never processed regardless of its labels.


The CLI

# Normal cron run β€” respects all trigger conditions
python intake_triage.py

# Manual trigger β€” fires regardless of count/age/schedule
python intake_triage.py --run

# Dry run β€” shows what would happen, no write/state-changing API calls
python intake_triage.py --dry-run

The --dry-run flag is the most useful for debugging. It logs the full batch context that would be sent to the discussion agents β€” you can see exactly what the editors will see before paying for the LLM calls.


Scope beyond news

The TrackerAdapter abstraction and the intake_triage.py script were designed with a second use case in mind from the start: client project intake for the ai-software-house build pipeline. Future client requests arriving by email (or API, or form) get issues created with a triage-pending label. A partner team β€” different roles, different scope description, same infrastructure β€” discusses the batch and approves or rejects each request before the expensive build pipeline runs.

That future use case is why I kept trigger logic, batch formatting, and verdict parsing separate from the editorial roles and labels. The news desk is one configuration of the intake system, not the intake system itself.


Related reading: Editorial Triage