OGN Architecture Overview

1. Engine core

• OGN/ – C++/CUDA libraries that implement the GPU‑first genomics engine:
  • common/ – shared utilities and CUDA guards.
  • io/ – FASTA/Q/CRAM ingest, pinned host buffers, S3 staging helpers.
  • fm_index/, fm2_index/ – FM index construction and queries.
  • mapper/ – ORBIT/seed‑and‑extend mapper, WFA/DPX paths, CPU fallback.
  • sw/ – Smith–Waterman kernels and host wrappers.
  • pairhmm/ – log‑space PairHMM kernels (with optional DPX acceleration).
  • variant/ – variant calling stages and orchestration.
• include/OGN/ – public headers that mirror the OGN/ layout; this is the stable C++ include surface for downstream applications.
• include/ogn/ – higher‑level C++ APIs (ogn_run_api.hpp, ogn_variant_runner_api.hpp, schema.hpp, etc.) and schema types shared with tools and SDKs.

2. Host scheduling and pipelines

• apps/ – host binaries:
  • ogn_run – end‑to‑end mapping + variant calling runner for local use.
  • ogn_variant_runner – pipeline entrypoint used by orchestration layers.
  • ogn_profile_exporter – Prometheus‑style metrics exporter for profiles.
• pipelines/ – YAML/JSON pipeline definitions that describe how to map from inputs (reads + reference) to outputs (VCF, metrics, artifacts).
• profiles/ – small JSON descriptors for named profiles (e.g., illumina_wgs), used by the Python CLI and jobs API.
• schema/ and schemas/ – JSON/Protobuf schema definitions and generated code for profile, provenance, and metrics payloads.

3. Data bundles and OGX

• data/ogx/ – OmegaAlign (OGX) bundles; each bundle is a directory with a manifest.json and pre‑built indices suitable for GPU ingest.
• benchmarks/ – benchmark configurations and regression profiles:
  • benchmarks/runs_hg002_wgs_ogn/ – HG002 WGS benchmark outputs.
  • benchmarks/regression/ (planned) – small golden regression packs used for CI perf/accuracy checks.
• pipelines/ + profiles/ + OGX bundle paths together form the contract “given reads + reference + profile → VCF + metrics”.

4. Verification and benchmarks

• tests/ – CTest/C++ tests plus Python tests:
  • Unit tests for core kernels (FM index, SW, PairHMM, mapper).
  • Integration tests such as Smoke_RunAndVerify and AlignmentTraceback.
  • CLI tests for ogn_cli and schema/provenance checks.
• bench/ and tools/ – benchmarking harnesses:
  • ogn_bench_* binaries for FM, SW, WFA, PairHMM, mapper.
  • tools/perf_guard.py and bench/perf_baseline.json for CI perf gating.
  • GIAB tooling (tools/run_giab_validation.py, tools/collect_giab_metrics.py) for HG002/HG005/HG007 validation.
• CI workflows:
  • ogn-engine-core – golden linux‑release build + tests (CPU and GPU).
  • ogn-bench-regression – perf regression guard versus baseline.
  • ogn-giab-hg002-chr20 and ogn-giab-wgs – GIAB correctness checks.

5. SDKs and integration points

• OGN Core Kit (ogn-core-kit) – open adoption surface: ogn CLI, ogn-runner, Job Spec v1, SDKs, and workflow adapters.
• openapi/ – OpenAPI spec for the gateway; maps the core “reads + reference + profile → VCF + metrics” contract to HTTP.

6. What is considered stable

• Stable, versioned surface:
  • C++ headers under include/OGN/ and include/ogn/.
  • JSON/Protobuf schemas under schema/ and schemas/.
  • The Python/Rust SDK APIs (ogn_sdk) plus the public CLI/runner shipped via OGN Core Kit.
• Internal / subject to change:
  • Most code under OGN/ and apps/ that is not explicitly documented as public.
  • Experimental scripts under scripts/dev/, benchmarking harnesses, and ad‑hoc tooling under tools/.