How we evaluate

aizk keeps three kinds of evidence apart and never lets one borrow the authority of another. This page assumes you know roughly how find runs and that you can open src/eval/.

The three levels

  level 1   the test suite               proves a contract holds
            tests/, src/eval/            Hypothesis properties, backend matrices,
                                         one real migration on a throwaway database
               │
               ├─ the frozen ablation gate
               │  tests/benchmark/       proves a change did not regress a fixed
               │                         question set, under a paired statistical test
               │
  level 2   the production bench         measures the memory one deployment
            uv run --no-sync aizk-eval bench    actually holds, on one day, on one machine
               │
  level 3   an external benchmark        imports somebody else's corpus into an
            uv run --no-sync aizk-eval groupmem isolated scope and answers their questions

Each level answers a different question, so each one is allowed a different sentence.

Level one, the suite

The suite proves contracts. It uses Hypothesis for the algebraic and authorization properties, parametrized cases for backend matrices, and a migration test that creates a disposable PostgreSQL database, upgrades it to head, writes a real scoped document and child chunk, and then checks that the child insert policy demands both the parent document ID and the exact scope set. The RLS verifier walks every scoped table and policy in the catalog, and Alembic autogenerate must return an empty revision against the current models.

What that buys is certainty about behavior. What it cannot buy is any claim about answer quality, because none of it asks a model anything. Testing owns the suite in detail.

The gate between level one and level two

Sitting inside the suite is one benchmark that behaves like a measurement. The benchmark pytest marker runs the frozen retrieval ablation and writes reports. The gate variant runs the same arms and compares them against the blessed baseline in tests/benchmark/data/retrieval_baseline.csv. Both point at the aizk_verify database and both require --eval-mode explicitly, so a benchmark can never run by accident inside the ordinary gate.

The comparison is per query rather than per average. Every arm, stratum and metric pair gets a seeded paired permutation p-value, Holm correction is applied across the whole family, and a failure needs both Holm significance and a paired Cohen’s dz of at least 0.2. Run noise alone therefore cannot fail a build. The baseline also carries retrieval_baseline.meta.json with the corpus SHA-256, the judge model, k and the arm list, and a run whose metadata differs is rejected as a usage error rather than silently compared.

Two honest limits

The committed question corpus has only two placeholder questions, so the gate guards the plumbing and statistics rather than retrieval quality. It becomes a quality gate only after a representative corpus is frozen with uv run --no-sync aizk-eval freeze. A baseline is meaningful only against the same corpus, so reblessing with --force-regen is a deliberate act.

Level two, the production bench

uv run --no-sync aizk-eval bench samples the memory a deployment already holds, turns each sample into a probe with the LLM, and scores the maximal plan that production find always uses. That is a real corpus, which is exactly its strength and exactly its limit. It tells you whether this deployment got better or worse between two commits. It says nothing you can put next to another system’s published number, because nobody else has your corpus. Retrieval results holds what it has measured.

Level three, an external benchmark

Only a level three run compares aizk with anything. It brings its own conversations, imports them into a deterministic isolated scope in a separate database, and answers the released questions through the same write, find, answer and judge path a person would use. That is the only level whose score means the same thing to a stranger. AIZK has no published external benchmark score. External benchmarks explains why and what would have to be true first.

A number without its conditions is not evidence

This is enforced in code rather than asked for in a review. BenchmarkReport.publishable is computed from five conditions at once, the complete corpus, unsampled questions, a solvability-filtered domain, the reference protocol, and zero operational failures. Any limit flag flips the rendered scorecard from publishable to diagnostic, and the report prints the agent model and the judge model every time.

ExtractionReport refuses the same shortcut from the other direction. A model turn that fails schema validation or times out is kept as a failed case with its error string rather than scored as a wrong prediction, and the rendered line always carries failed= beside the F1. A benchmark that quietly drops its failures reports the quality of the subset that happened to work, which is not the quality of anything.

Why every result carries its conditions

Every measurement is a function of its corpus, model, hardware, and database configuration. Averaging incompatible runs into one headline figure would produce a number that describes no real system.

An extraction latency on a 31B checkpoint with a dedicated RTX 3090 is not comparable to the same number on 12B. A find p50 over a hundred thousand chunks is not the same measurement as one over two thousand. A retrieval hit rate on the vault is not one on planted synthetic ground truth. Averaging those would produce a number that describes no run that ever happened, and worse, would strip exactly the metadata a reader needs to decide whether it applies to them.

So a publishable result carries the complete fixture, hardware class, software configuration, models, and raw report. It also says what it proves and what it does not. Private regression runs stay private unless their corpus can be replaced with a reproducible public fixture.

Next

• The eval CLI is every command and the question it answers.
• Retrieval results has the production bench and publication rules.
• Extraction and models has the graph writer and model selection.
• External benchmarks has GroupMemBench and the claims we avoid.