mlkem-sync/.trellis/spec/test_framework/structure.md

Test Framework Structure

Directory Layout

test_framework/
├── run_all.py              # CLI entry: --module, --case, --list, --quick
├── config.json              # Verilator path, clock period, timeouts
├── lib/
│   ├── test_runner.py      # Discovery, compile, run, compare pipeline
│   ├── sim_controller.py   # Verilator compiles/run wrapper
│   ├── vector_gen.py       # Base class for vector generators
│   ├── result_checker.py   # Hex-file comparison
│   └── reporter.py         # Terminal + HTML output
├── modules/
│   └── <module>/
│       ├── test_plan.json  # Module test definition
│       ├── gen_vectors.py  # Vector generator (subclass of VectorGenerator)
│       └── vectors/        # Generated hex files (auto-cleaned)
└── reports/
    ├── latest/
    └── history/

test_plan.json Schema

{
  "module": "<name>",
  "rtl_top": "sync_rtl/<path>/<top>.v",
  "rtl_deps": ["sync_rtl/common/<dep>.v"],
  "tb_cpp": "sync_rtl/<path>/TB/tb_<top>.cpp",
  "simulator": "verilator",
  "timeout_s": 30,
  "cases": [{
    "id": "<case_id>",
    "description": "<what this tests>",
    "params": {},
    "num_vectors": 50,
    "tolerance": "bit_exact"
  }]
}

• rtl_top: top module basename (without .v) is used as Verilator --top-module
• rtl_deps: listed before rtl_top in Verilator command line
• tolerance: "bit_exact" uses result_checker.py; otherwise delegates to gen_vectors.compare_results()

Vector Generator Contract

Each module's gen_vectors.py must:

1. Subclass VectorGenerator from test_framework.lib.vector_gen
2. Implement generate_one(params) -> dict returning {"input": {...}, "expected": {...}}
3. Override write_hex_file(vectors, filepath) for input format
4. Override write_expected_file(vectors, filepath) for expected format

The framework auto-discovers the generator class by scanning for subclasses of VectorGenerator.

Adding a New Module

1. Create test_framework/modules/<name>/test_plan.json
2. Create test_framework/modules/<name>/gen_vectors.py
3. Create sync_rtl/<name>/<name>_sync.v (RTL)
4. Create sync_rtl/<name>/TB/tb_<name>.cpp (Verilator C++ TB)
5. Run python3 test_framework/run_all.py --list to verify discovery
6. Run python3 test_framework/run_all.py --module <name> to test