[feature] Retrieval-quality eval harness (recall@k + MRR) so retrieval changes can produce the evidence CONTRIBUTING requires

[CryptoJones]

Disclosure: I maintain omind, a separate
Apache-2.0 project solving an adjacent problem (durable agent memory in a plain-Markdown
Obsidian vault). I read claude-obsidian's source before filing and I'm linking omind's
implementation as a worked reference — not proposing you adopt its code.

Problem

claude-obsidian is unusually disciplined about not overclaiming retrieval quality.
scripts/contextual-prefix.py says it plainly:

Anthropic reported a 35-49% retrieval-failure reduction in its upstream experiments;
this local implementation requires evaluation on each vault and makes no fixed gain claim.

The README repeats it ("Model-based retrieval falls back to deterministic BM25 when the
embedding or reranking stage cannot be trusted"), and CONTRIBUTING makes it a rule: "Do
not add capability, performance, or competitor claims without current, traceable
evidence."

The problem is that the product ships no way to produce that evidence. The retrieval
stack has at least four independently tunable stages —

• prefix tier (anthropic-api / claude-cli / synthetic / skipped),
• chunking (2,000-char target, 4,000 hard split, 200 overlap),
• BM25 (k1=1.5, b=0.75),
• rerank (cosine on a selected Ollama model, or no-op),

— and no harness that answers "did that change help?" for any of them.
tests/test_retrieve.py and test_bm25_index.py verify correctness (right shape, right
ordering given fixed scores), which is a different question from quality. The one piece
of tooling that sounds relevant, scripts/benchmark-runner.py, isn't in the public tree at
all — tests/test_benchmark_tools.py guards on TOOLS_PRESENT and skips. And nothing in
the repo mentions recall, MRR, nDCG, or any ranking metric.

The practical consequence: a maintainer cannot tell whether the synthetic prefix tier is
worth its complexity, whether the rerank stage earns the Ollama dependency, or whether a
chunk-size change regressed anything. --explain shows per-stage diagnostics for one
query, which is a debugging tool, not a measurement.

This is the one gap where I think the project's own stated values most clearly want the
feature.

Proposed solution

A labelled query set plus a metrics runner.

1. A version-controlled labelled set — a JSON file mapping queries to the page(s) that
should be retrieved:

{
  "schema_version": 1,
  "queries": [
    {"query": "how does the transaction lock recover", "expect": ["wiki/concepts/Transactions.md"]},
    {"query": "why is egress gated by default",        "expect": ["wiki/concepts/Egress.md"]}
  ]
}

Ship a small fixture set against tests/fixtures/, and let users keep their own against a
real vault — vault-specific evaluation is exactly what contextual-prefix.py says is
required.

2. claude-obsidian.py bench --quality --vault PATH [--queries FILE] reporting
recall@1, recall@5, and MRR, plus a count of skipped targets (labelled pages that no longer
exist — otherwise vault drift silently deflates the score) and the worst individual misses,
which is where the diagnostic value actually is.

3. --strategy to hold everything constant but one stage, so --no-rerank,
--no-llm, and a chunk-size override become measurable rather than matters of taste.

omind's equivalent is run_quality in
src/omind/bench.py
(omind bench --quality), which reports exactly that set — recall@1, recall@5, MRR,
skipped targets, worst misses — against a version-controlled labelled set on the live
vault. It's how we justified keeping some retrieval stages and dropping others;
docs/retrieval.md
records the measured before/after.

Alternatives considered

• Keep it private. benchmark-runner.py presumably exists in the dev tree. But then
  contributors can't produce the evidence CONTRIBUTING demands of them, and the burden of
  proving a retrieval change lands entirely on the maintainer.
• Golden top-k assertions in the test suite. Brittle in the wrong way — they fail on
  any scoring tweak without saying whether quality moved up or down. Metrics degrade
  gracefully; golden orderings don't.
• nDCG with graded relevance. Better metric, much worse labelling ergonomics. Binary
  relevance with recall@k and MRR is the right first cut.

Scope

• A new script (scripts/<name>) — or a bench verb on the existing CLI
• A new skill / agent
• Change to existing skill
• Change to plugin manifest / hooks / setup scripts
• Documentation only — a short "how to evaluate retrieval on your vault" section

Compatibility

• Behavior change for existing v1.x vaults? No — read-only measurement, no vault writes.
• New opt-in (bin/setup-*.sh)? No.
• New dependency? No — recall@k and MRR are a few lines of arithmetic over the JSON
  retrieve.py already returns. Deliberately no numpy, no scikit.

Testing

Fully hermetic; this is one of the easier things to test offline:

• a fixture vault + fixture labelled set with a known answer — hand-computed recall@1,
  recall@5, and MRR, asserted exactly;
• a labelled entry pointing at a deleted page counts as skipped, not as a miss (the
  distinction that keeps the metric honest as a vault drifts);
• an empty/malformed labelled file fails with a usage error, not a divide-by-zero;
• --no-rerank runs without Ollama and still reports metrics, using the existing no-op
  rerank path;
• exit 10 when no index exists, matching retrieve.py's established contract.

Because BM25 scoring is deterministic, the metric assertions can be exact rather than
tolerance-based — no flakiness.

Additional context

• omind's quality harness: run_quality in src/omind/bench.py
• Its retrieval design notes and measured numbers: docs/retrieval.md
• Anthropic's contextual retrieval writeup, for the upstream methodology: https://www.anthropic.com/news/contextual-retrieval