Executive summary (v1.1)

• Panels today: GIAB core smoke, multi-platform smoke, and CHM13 perf stress (Illumina/HiFi/ONT). All are gated for correctness and performance in CI on every push/PR.
• Truth/reference: GIAB v4.2.1 (GRCh38) for accuracy; CHM13v2.0 for perf-only stress. Seeds + env.json captured for reproducibility across hardware.
• Contract is versioned (schema 1.0, contract v1.1) with gold snapshots in repo. CI refuses merges if accuracy or perf regress beyond thresholds.
• Badge: every run prints Verified by OGN Benchmark Contract v1.1 ✓ in CLI footers; UI surfaces should show the same pill with hover text describing truth sets and panels.

OGX engineering bench (sibling contract)

• Scope: chr20 OGX micro-bundle + overrides; validates Omega chain override handling, OGX ingest perf, and OGC manifest stamping.
• Outputs: metrics/ingest JSON, TSV summary, and OGC manifests per profile.
• Gates: override correctness, schema validation, and ingest perf thresholds (h2d throughput + wall time) in .github/workflows/ogx-ogc.yml.
• Goal: keep OGX packaging + mapper ingest as rigorously checked as GIAB accuracy/perf so both contracts evolve together.

Next panel (flashy, v1.2 candidate): HG002 multi-platform consistency

• Purpose: prove cross-platform stability (Illumina 2x150, PacBio HiFi, ONT R10.4.1) on HG002.
• Metric: genotype concordance between platforms in GIAB confident regions; plus per-platform recall/precision.
• Artifacts to add (same playbook as GIAB/OGX):
  • entries in benchmarks/configs/benchmarks.yaml for hg002-{illumina,hifi,ont} consistency panel
  • thresholds benchmarks/thresholds_hg002_multiplatform.json (per-platform recall/precision + cross-platform concordance)
  • gold TSV/JSON under benchmarks/gold/hg002_multiplatform_v1/
  • CI job running the smoke-sized slice and comparing to gold
  • public doc blurb + badge line once numbers are locked

OGN Benchmark Contract v1.1

• Contract version: v1.1 (schema 1.0)
• Panels covered:
  • GIAB core smoke (chr20 slice)
  • Multi-platform smoke (chr20 slice)
  • CHM13 stress perf (perf-only gate)
• Spec files:
  • Accuracy schema: benchmarks/metrics_spec.json
  • Performance schema: benchmarks/metrics_perf_spec.json
  • Accuracy thresholds: benchmarks/thresholds_giab_core.json, benchmarks/thresholds_multiplatform_smoke.json
  • Performance thresholds: benchmarks/thresholds_perf_smoke.json, benchmarks/thresholds_perf_multiplatform_smoke.json, benchmarks/thresholds_perf_chm13_stress.json
• Gold snapshots: benchmarks/gold/giab_core_v1/, benchmarks/gold/multi_platform_v1/, benchmarks/gold/chm13_stress_v1/
• CI gate: .github/workflows/bench_smoke.yml runs collector + comparator against gold on every push/PR (accuracy and perf). CHM13 perf gate is part of release checks.

Claim (current gold)

Reproducing locally

1. Build manifest: python benchmarks/make_manifest.py --manifest /tmp/manifest.csv --summary-json /tmp/manifest.json
2. Smoke gate: python tools/ogn_bench.py bench smoke
3. (Optional full run) bash benchmarks/run_suite.sh --panels giab-core
4. Collect + compare smoke outputs: see .github/workflows/bench_smoke.yml for the exact commands.

Reporting

Badge spec (internal + UI)

• Text: Verified by OGN Benchmark Contract v1.1 ✓
• Hover: “Accuracy and performance validated against GIAB v4.2.1 (GRCh38), HG002 multi-platform smoke, CHM13 perf stress. Deterministic seeds + env.json captured.”
• Source: contract_badge block emitted in <sample>_metrics.json and in collected summary JSON.
• Placement: CLI footers (already), Helix/Studio status pills, any metrics viewer.