In the infrastructure and memory review, I listed semantic memory search as the first thing I needed to build.

I was wrong. It had already been built. It simply had not worked for some time.

The memory tool hiding in plain sight

rag-mcp is the RAG server this project already runs for codebase search. Its docstring lists four tools:

"""RAG MCP Server — FastMCP server exposing search tools over Streamable HTTP.

Tools:
    search_codebase(query, top_k)  — search indexed source code
    search_memory(query, top_k)    — search past pipeline runs
    search_docs(query, top_k)      — search indexed documentation
    search_standards(query, top_k) — search coding standards and design guidelines
"""

I had read past search_memory more than once without really noticing it. It is a real tool, backed by the same pgvector store as code search, and orchestrator.py already gives it to the Junior and Senior Engineer agents when MCP is enabled. My configuration already enabled it.

So an agent could already call search_memory("rate limiting") mid-task and get back embedded, semantically-ranked past runs. That's the exact upgrade the infrastructure and memory review proposed as new infrastructure.

Except it couldn't, because the query would crash

rag-mcp/indexer.py is what's supposed to populate search_memory's index — read memory_store.py's SQLite file, embed each run, upsert into pgvector. Here's what it actually did:

def index_memory(db_path: str, embedder: Embedder) -> None:
    conn = sqlite3.connect(db_path)
    cur = conn.execute(
        "SELECT id, prd, design, summary, created_at FROM runs WHERE summary IS NOT NULL"
    )

memory_store.py's runs table has no prd or design columns. It never has, in the current schema — only id, repo, run_id, created_at, summary, tags, mode, tier, period_label, consolidated. The indexer expected an older or different schema; the mismatch alone does not establish when those columns changed. Run this against today's memory.db and it throws sqlite3.OperationalError: no such column: prd before it embeds a single row.

And even with that fixed, nothing called it. index_memory() was reachable from exactly one place: the CLI (python indexer.py --source memory --db ...). No cron entry, no hook in orchestrator.py's _finish(), no call in watcher.py. Every pipeline run writes a new row to memory.db and the vector index never hears about it unless someone remembers to re-run the indexer by hand.

So the tool existed, but its index could neither build successfully nor stay current. It was a switch connected to a dead battery.

Repairing the path

The repair had three small parts.

1. Read the schema that actually exists.

def index_memory(db_path: str, embedder: Embedder, incremental: bool = True) -> None:
    conn = sqlite3.connect(db_path)
    where = "WHERE summary IS NOT NULL AND summary != ''"
    if incremental:
        where += " AND indexed=0"
    cur = conn.execute(
        f"SELECT id, repo, tier, summary, created_at FROM runs {where} ORDER BY id ASC"
    )

Just summary — that's the only prose field the current schema actually has. repo and tier move into chunk metadata instead of being separate embedded fields.

2. Make it incremental, so it's cheap enough to call on every run.

The old version re-embedded the entire table on every invocation — fine for a one-off CLI backfill, wasteful for anything automatic. memory_store.py gets a new indexed column, added the same way tier/period_label/consolidated were added before it — a migration check in _init_schema(), not a breaking change to the table:

for col, dflt, col_type in [
    ("tier", "'run'", "TEXT"),
    ("period_label", "''", "TEXT"),
    ("consolidated", "0", "INTEGER"),
    ("indexed", "0", "INTEGER"),
]:
    if col not in existing:
        self._conn.execute(f"ALTER TABLE runs ADD COLUMN {col} {col_type} DEFAULT {dflt}")

Plus two small methods so anything in-process — orchestrator, tests, diagnostics — has a clean way to ask the same question the indexer asks over raw SQL:

def unindexed(self, limit: int = 500) -> list[dict]:
    """Rows not yet embedded into pgvector, oldest first."""
    rows = self._conn.execute(
        """SELECT id, repo, tier, summary, created_at FROM runs
           WHERE indexed=0 AND summary IS NOT NULL AND summary != ''
           ORDER BY id ASC LIMIT ?""",
        (limit,),
    ).fetchall()
    return [dict(zip(("id", "repo", "tier", "summary", "created_at"), r)) for r in rows]

def mark_indexed(self, ids: list[int]) -> None:
    if not ids:
        return
    with self._lock:
        placeholders = ",".join("?" * len(ids))
        with self._conn:
            self._conn.execute(f"UPDATE runs SET indexed=1 WHERE id IN ({placeholders})", ids)

rag-mcp/indexer.py still talks to the SQLite file directly rather than importing memory_store.py — it's a separately deployed service with its own requirements and its own container, and that boundary is deliberate. So the indexer keeps its own raw-SQL version of the same idea, plus a defensive ALTER TABLE check for the case where it runs against a memory.db written before this fix ever landed. memory_store.py stays the schema's single source of truth either way.

3. Actually call it.

repo_context.py already had the exact pattern needed, for a different indexer path:

class RepoAutoIndexer:
    """Downloads a GitHub repo and indexes it into the RAG codebase collection.

    Uses the rag-mcp/indexer.py subprocess so the RAG server is always
    the single source of truth for embeddings.
    """
    def _run_indexer(self, path: str) -> None:
        proc = subprocess.run(
            [sys.executable, self.indexer_script, "--source", "codebase", "--path", path, "--clean"],
            check=False, timeout=300, capture_output=True, text=True,
        )
        if proc.returncode != 0:
            log.warning("RAG indexer exited %d: %s", proc.returncode, proc.stderr[:200])

Same shape, pointed at --source memory instead, called from _finish() right after self.memory.save(...):

def _index_memory_for_search(self) -> None:
    """Run rag-mcp/indexer.py --source memory so search_memory (MCP) stays current."""
    script = Path("rag-mcp/indexer.py")
    if not script.exists():
        return  # RAG indexer not available — skip silently
    try:
        proc = subprocess.run(
            [sys.executable, str(script), "--source", "memory",
             "--db", str(self.workspace_dir / "memory.db")],
            check=False, timeout=300, capture_output=True, text=True,
        )
        if proc.returncode != 0:
            log.warning("Memory indexer exited %d: %s", proc.returncode, proc.stderr[:200])
    except Exception as exc:
        log.warning("Memory indexer failed to run: %s", exc)

Non-fatal by design, same as the memory-bank write step next to it in _finish() — a Postgres or Ollama outage in rag-mcp should never fail a pipeline run. And because indexing is now incremental, this call processes one new row on the typical run instead of re-embedding months of history every time.

What this actually closes

This was not the new vector-memory feature I described in the infrastructure and memory review. It was an old feature with three broken connections: the wrong columns, no incremental marker, and no automatic call after saving memory.

The more worrying discovery was that index_memory() had no test at all. That is how the schema drift stayed invisible. I added six tests covering the schema, empty summaries, two-pass incremental indexing, full backfill, embedder failure, and migration from the older schema.

What I learned from missing it

the infrastructure and memory review said the pieces were already deployed for another purpose. The reality was more embarrassing: they had already been assembled for this exact purpose, and I had mistaken a broken integration for a missing feature.

The right response was not a rewrite. I made the indexer read today's schema, made each run cheap enough to invoke automatically, and added the tests that should have caught the drift much earlier.

Part of the AI Software House series.