Assay

Assay is an experiment harness for comparing changes to agent systems. It fixes the study inputs and execution plan, records each attempt and evaluation, and produces reports with enough provenance to inspect how the result was obtained.

From study to inspectable result

A study binds subjects, arm configurations, evaluator identities, and relevant versions. Execution follows an immutable, content-addressed plan through worker and evaluator adapters. Outputs, evaluations, accounting, and failures remain inspectable rather than disappearing into a final aggregate.

Paired reports distinguish measured effects, missing data, correctness, and cost coverage. Unknown cost is not measured zero. In coding studies, structural helper reuse and functional correctness are separate measurements: reusing a helper does not by itself establish that the resulting program works.

Verification and its boundary

Exported bundles include the immutable reference closure and can be verified offline. Verification checks record bindings and recomputes reports relative to the supplied plan and bundle. It does not prove producer identity or that unreported real-world attempts never happened.

What to inspect

The repository explains study materialization, plan compilation, execution, reporting, and offline verification. The related coding-model study compares clean and inconsistent conditions, then examines helper visibility and model choice while evaluating reuse separately from correctness.