mlkem_top now instantiates comp_decomp_sync (E5), so the shared
xsim_run.tcl must compile it (+ its pipeline_reg dep) before mlkem_top.
This unbreaks 'run_tb.sh top' which failed elaboration with
'Module comp_decomp_sync not found'. run_enc.sh simplified to reuse the
tcl's now-complete compile list (drops its duplicate leaf block).
ST_ENC_C1: per-coeff Compress_du via comp_decomp_sync (mode 0) then
LSB-first byte packing into ct_bram. 5-phase micro-seq reads u[cp_poly]
from bank_se (rel K+poly), feeds the compressor (1-cyc pipe), appends du
bits to cp_buf, and drains whole bytes. Each poly = 256*du bits (whole
bytes) so the bit buffer empties at every poly boundary.
ST_ENC_U now advances to ST_ENC_C1 (was ST_DONE).
TB: verify_e5 compares ct_bram[0..c1_bytes-1] to the KAT.ct prefix via
the dbg_ct tap. run_enc.sh: encaps TB runner (compiles comp_decomp_sync
which the KeyGen tcl omits).
Verified K=2/3/4 c1 == KAT.ct prefix (640/960/1408 B; K=4 du=11
cross-byte path), K=2 cases 0-2.
ST_ENC_U computes u[i] per row in 3 sub-phases reusing shared u_pmul + u_ntt:
sub0 MAC : sum_j A_hat[j][i] o y_hat[j] (TRANSPOSE: slot=j*K+i) -> NTT-domain
psum in bank_t rel slot UPSUM=1 (e2 in slot 0), init 0 at j==0
sub1 INTT: INTT(psum) mode=1 (built-in x3303) in place in bank_t[UPSUM]
sub2 ADD : u[i][w] = psum[w] + e1[i][w] mod Q -> bank_se rel (K+i), over e1
y_hat (bank_se 0..K-1) preserved for V. ntt_core mode + input muxed for the
INTT sub-phase; bank_a/se/t read+write ports extended for all 3 sub-phases.
Fixed a duplicate 'assign bse_we' (stale + new both present -> ADD writes
X-dropped); collapsed to one. Verified (K=2 c0) u[0..1] == ml-kem-r golden
(transpose + INTT + e1 all correct); E0/E1/E3 pass, E2 trimmed to e2 (e1
consumed into u, transitively checked by E4); K=3/4 no timeout.
ST_ENC_N reuses the ST_N forward-NTT datapath (mode=0, read-ahead load,
in-place writeback to bank_se). Slot count parameterized: KeyGen 2K (s,e),
Encaps K (y only; e1/e2 stay time-domain) via n_slot_max. bse rd/wr muxes
and the C->N arming extended for ST_ENC_N.
Verified (K=2 c0) y_hat[0..1] == ml-kem-r golden; E0/E1/E2 still pass
(E2 y check dropped since NTT overwrites y in place -> covered by E3);
K=3/4 run through E3 without timeout.
ST_ENC_C reuses the ST_C CBD datapath with Encaps muxes: seed=r (not sigma),
eta=eta1 for y[0..K-1] then eta2 for e1[0..K-1]/e2, nonce 0..2K. 2K+1 polys
(vs KeyGen 2K). y/e1 -> bank_se rel slots 0..2K-1; e2 -> bank_t rel slot 0
(free during C/N/U since TDEC is deferred to V-prep so the 28-slot banks hold
all of A+y_hat+e1+e2 at K=4 without resizing).
Bring-up golden via ml-kem-r dump_encaps_full (working-tree example):
vectors/encgold/ec_k2_c0_{y,e1,e2}.hex. Verified (K=2 c0) y[0..1],e1[0..1],e2
== ml-kem-r; A_hat (E1) and ss (E0) still pass; K=3/4 no timeout.
ST_ENC_LOAD: stream rho (32B) from ek_bram[384k..] into rho_r (read-ahead,
1-cyc bram latency). ST_ENC_A: regenerate A_hat via SampleNTT into bank_a
(reuses ST_A datapath, gated on st==ST_A||ST_ENC_A). ST_ENC_TDEC:
byteDecode12 ek[i*384..] -> t_hat[i] into bank_t (5-cycle micro-phase per
3-byte/2-coeff triple; bt write port muxed with ST_M).
Verified (K=2 c0) A_hat (1024 coeffs) + t_hat (512) == KeyGen golden via
dbg_coeff_o; E0 ss==KAT.ss still passes all K/cases (no timeout).
Extend mlkem_top with a runtime op_i select (0=KeyGen, 1=Encaps) and the
first Encaps stages, reusing the shared keccak_core and the ST_H multi-block
SHA3-256 machinery:
ST_ENC_H: H(ek) over preloaded ek_bram (same FSM as KeyGen ST_H)
ST_ENC_G: (K,r) = G(m||H(ek)) via new 64-byte single-block SHA3-512
- sha3_top_shared: add mode=2'b11 = SHA3-512 over a full 512-bit message
(g512_pad). Standalone tb_sha3_g512 confirms it byte-exact.
- mlkem_top: new ports op_i, msg_i, ek_in_{we,addr,byte} (ek preload), ss_o,
dbg_ct_*, dbg_r_o/dbg_hek_o. st widened 4->5 bits; ST_ENC_* states added.
Renamed message port to msg_i to avoid collision with ST_M counter m_i.
- TB tb_mlkem_enc_katK + gen_encaps_vectors.py (per-byte ek/m/ct/ss vectors).
Verified ss==KAT.ss for K=2/3/4, cases 0-2 (all PASS). KeyGen unaffected
(K=2 c0 still ek==pk, dk==sk byte-exact).
Cross-checks the hardware KAT vectors against the ml-kem-r Rust reference's
NIST .rsp files, accounting for the d/z byte-order convention: the hardware
hashes G(reverse(d)||K), so its vector files store d/z reversed vs the NIST
.rsp. The script applies that reversal for d/z and compares ek==pk, dk==sk
directly; 11/11 cases consistent.
ml-kem-r path via $ML_KEM_R (default ~/Dev/ml-kem-r); friendly errors on
missing files. Documents the convention that tripped up logging.rs (which
pasted hardware-order d into the literal-order Rust reference).
The KAT TB now prints, for each run: the d and z seeds (32-byte hex, MSB-first
== vector file order) before start, and the full ek/dk byte strings read back
from the DUT (32 bytes/line, offset-prefixed) after the byte-compare. Inputs
verified to match the kat_*_d/z.hex files exactly; dk[768..] echoes ek[0..] as
expected (dk_pke || ek || H(ek) || z). Pass/fail logic unchanged.
Final bank promoted to sd_bram (the busiest: 5 read sites, 2 write sites).
Read port phase-muxed: ST_N load / ST_M load (pm_b s_hat[j]) vs accumulate
(e_hat, selected by m_loading) / ST_E dk-half / dbg. Write port combinational:
ST_C CBD store vs ST_N NTT writeback (disjoint states). All explicit consumer
read registers (n_rd_data, pm_b_rd, m_eacc_rd, e_se_rd) collapsed into the
sd_bram internal read register; m_acc_src and e_rd_coeff now select between
two registered sd_bram outputs (same 1-cycle latency).
mlkem_top now contains ZERO behavioural RAM arrays: all coefficient storage is
3 sd_bram banks (a/se/t) + ek/dkp byte buffers = 5 sd_bram instances total,
each inferring BRAM (ASIC: compiled SRAM). 11/11 KAT PASS, byte-exact.
Second bank promoted to sd_bram. Read port muxed by phase (ST_M acc t_hat
addr / ST_E ek-half byteEncode addr / dbg). The shared mux-registers that fed
both bank_se and bank_t were split: the bank_se half stays a manual reg
(m_eacc_rd / e_se_rd), the bank_t half uses sd_bram's internal read register
(bt_rd_data) -- same 1-cycle latency, so the j-select (m_jq) and ST_E coeff
select (e_rd_coeff) just pick between the two registered outputs. RMW safe:
acc read addr (m_oidx+1) leads write addr (m_oidx), no same-cycle alias.
Write port combinational. 11/11 KAT PASS incl. K=3/4 deep accumulate.
First of 3 banks promoted from async reg array to an sd_bram. Read port
muxed by phase (ST_M load drives pm_a_full, else dbg index); sd_bram's
internal rd_addr_r replaces the explicit pm_a_rd reg (identical 1-cyc
latency). Write port driven combinationally so the posedge write matches the
old nonblocking reg-array timing. 11/11 KAT PASS, byte-exact.
Final per-consumer step of stage 2b. ST_E read two coeffs of a pair from one
bank combinationally; now serialized through a single registered read port
(e_rd_data_r) over a 4-cycle micro-phase e_ph: ph0 fetch c0, ph1 write b0 +
fetch c1, ph2 write b1, ph3 write b2. Matches sd_bram timing; +512 cyc on K=2
(one extra fetch per pair-stream). All polymem read consumers (ST_N, ST_M
load, ST_M acc, ST_E) are now registered-read. 11/11 KAT PASS, byte-exact.
mlkem_top instantiates sd_bram (ek/dk byte buffers) but create_project.tcl
was missing it -- elaboration would fail. Add sync_rtl/storage/sd_bram.v to
match xsim_run.tcl exactly. Also clarify the run flow: runtime=all means
launch_simulation runs to the TB $finish; a manual 'run all' afterwards trips
the 120ms watchdog and prints a spurious timeout.
Verified: vivado -mode batch builds the project (0 errors) and the simulation
prints 'K=2 CASE 0 PASS: ek==pk, dk==sk'.
The fast top runner was hiding xsim stdout in /tmp and only echoing PASS/FAIL,
which buried the TB $display diagnostics needed when a case fails. Now xsim
output is tee'd to both terminal and log, with the PASS/FAIL summary appended.
./run_tb.sh top [K] [CASE] now elaborates only the requested K and runs only
the requested CASE(s), parsing PASS/FAIL + file-not-found per run. Avoids the
full 3-snapshot, 11-case sweep when iterating. 'top' with no extra args and
all other modules keep their original full-tcl behaviour.
Third per-consumer step of stage 2b, the read-modify-write one. m_acc_src is
now a registered read (m_acc_rd) whose address leads by pm_vo (m_oidx+1 when
consuming) so the value lands exactly on the pm_vo cycle. Same-index RMW on
bank_t is read-old: the running-t_hat source was written by the previous term
and is long settled; read addr (m_oidx+1) leads write addr (m_oidx).
11/11 KAT PASS incl. K=3/4 j>0 paths, byte-exact.
Second per-consumer step of stage 2b. m_ld becomes a read-ahead pointer; both
poly_mul load reads (bank_a A_hat, bank_se s_hat) are registered into
pm_a_rd/pm_b_rd and fed one cycle later (pm_valid delayed 1 cyc), same pattern
as ST_N. 11/11 KAT PASS, byte-exact.
First per-consumer step of stage 2b. n_ridx becomes a read-ahead pointer
leading the consume index by 1; bank_se read is registered into n_rd_data and
fed to ntt_core one cycle later (n_valid delayed 1 cyc). Cores hold ready_o
high through LOAD so a fixed 1-cycle skew suffices. Matches sd_bram registered
timing for this read port. 11/11 KAT PASS, byte-exact.
Replace the single async-read polymem[0:28*256-1] with 3 polynomial-indexed
banks (bank_a A_hat / bank_se s_hat||e_hat / bank_t t_hat), addressed by
abs_slot - base_slot. Still async-read here -- a pure refactor that validates
bank sizing and base-relative addressing with zero timing change before
stage 2b converts them to registered sd_bram + read-ahead pipelines.
11/11 KAT PASS, byte-exact, 0 file-not-found.
KeyGen's Keccak consumers (G/H via sha3, A via SampleNTT, C via CBD) run in
disjoint top-FSM phases, so one keccak_core suffices. Add sha3_top_shared
(keccak_core externalised as kc_* ports, like the existing sample_*_shared
variants); mlkem_top now instantiates one keccak_core and a phase mux that
routes kc_state_i/kc_valid_i from the active consumer and gates kc_valid_o
per consumer (inactive samplers latch squeeze state unconditionally).
Cuts the KeyGen datapath from 4 keccak_core (1600-bit state + 24-round logic
each) to 1 -- the dominant ASIC area win. 11/11 KAT PASS (K=2 c0-4, K=3/4
c0-2), byte-exact, 0 file-not-found.
ST_G (single-block G) and ST_H (multi-block H(ek)) are disjoint FSM phases,
so one sha3_top serves both: mb_en and ready_i are phase-muxed, h_hash/h_vo
alias the shared core's outputs. Removes the dedicated u_sha3_h instance and
its keccak_core. 11/11 KAT PASS (K=2 c0-4, K=3/4 c0-2), byte-exact, 0 file-not-found.
Phase 1 of migrating mlkem_top's large arrays to inferable RAM. ek_mem
and dkp_mem reg arrays are replaced by two sd_bram instances (1R/1W,
registered read). Datapath changes to fit single-port-per-cycle BRAM:
- ST_E writes 1 byte/cycle (was 3): added e_byte sub-counter; ST_E
length ~3x (K=2 KeyGen 21403->22433 cyc, ~5%).
- ST_H ek read is now registered: assemble phase presents the address
one cycle ahead and writes back the byte that arrived (h_wb_* pipe),
h_byte runs 0..136 to flush the final byte. Pad bytes via h_padconst.
- dbg_byte_o/dbg_dk_o read combinationally off the BRAM registered
output (net 1-cycle latency, within the TB's 2-cycle read wait);
region decode for dk readback unchanged.
Add sd_bram.v to the top TB compile list. Verified byte-exact vs NIST
KAT: K=2 c0-4, K=3 c0-2, K=4 c0-2 -> 11/11 PASS, 0 file-not-found.
The old script referenced 5 non-existent files (keccak_arbiter,
sha3_chain_top_shared, tb_mlkem_top_xsim, tb_kg/en/de) and stale
vectors, so read_verilog/elaborate failed outright.
Rewrite to mirror the verified XSIM flow (sync_rtl/top/TB/xsim_run.tcl):
- load exactly the 14 sources mlkem_top depends on
- sim top = tb_mlkem_kg_katK_xsim, runtime K via generic KP, case via
-testplusarg CASE
- copy KAT vectors into the xsim working dir via xsim.compile.tcl.pre
(the only hook in 2019.2 that runs before $readmemh; an appended
-tclbatch runs after Vivado's own 'run all', too late)
- drop duplicate --relax (XSim adds it; passing again is an error)
Verified through the actual Vivado batch project flow:
K=2 CASE 0 -> PASS (21403 cyc), K=4 CASE 2 -> PASS (54059 cyc),
0 file-not-found warnings. gitignore the generated vivado_prj/.
Also rewrite README.md in Chinese documenting the mlkem_top workflow
and test flow.
mlkem_top now sizes storage for KMAX=4 (worst case) and selects the
active ML-KEM parameter set at start_i via the k_i input. All K-derived
quantities (eta1, slot bases, ek/dk byte counts, H(ek) block count, FSM
bounds) are computed at runtime from the captured k_r.
Verified byte-exact against NIST KAT for all three parameter sets:
K=2 (512) cases 0-4, K=3 (768) cases 0-2, K=4 (1024) cases 0-2
-> 11/11 PASS (ek==pk, dk==sk).
The parametric KAT TB (KP generic) builds filenames kat_k<K>_c<n>_*.hex, but
the K=2 vectors were committed earlier as kat_c<n>_*.hex (no k2 prefix), so
run_tb.sh top emitted 'file cannot be opened' warnings for K=2 and the data
read as X. Add kat_k2_* copies so all three parameter sets load cleanly.
./run_tb.sh top now: 0 file warnings, all 11 cases PASS (K=2:0..4, K=3:0..2,
K=4:0..2), exit 0.
./run_tb.sh top now compiles the full KeyGen datapath + tb_mlkem_kg_katN
and runs all 5 NIST KAT cases (ek==pk 800B, dk==sk 1632B byte-exact).
Also registers 'top' in ./run_tb.sh --list.
Add ST_H stage: second sha3_top (mb_en=1) computes H(ek) over 800B ek as 6
pre-padded SHA3-256 rate blocks. Per block: assemble 136 bytes (h_padbyte
applies 0x06...0x80 padding on final block) into h_block_r, feed (hold valid
until mb_ready drops), wait permute; capture digest on last block into hek_r.
Full dk readback tap: dk = dk_pke(768) || ek(800) || H(ek)(32) || z(32) = 1632B.
End-to-end TB (tb_mlkem_kg_kat_xsim, no force/release): drive KAT count=0 d/z,
run full KeyGen FSM (IDLE->G->A->C->N->M->E->H->DONE), verify:
ek == KAT pk (800B) byte-exact
dk == KAT sk (1632B) byte-exact
Done in 21403 cycles. ML-KEM-512 KeyGen complete and KAT-verified.
Prior stage TBs (2c/2e/2f) still pass (no regression).
Add ST_N stage: forward NTT (ntt_core mode=0, no scaling) of s[0],s[1],
e[0],e[1] in place (slots S0,S1,E0,E1). Per slot: stream 256 coeffs into
ntt_core during LOAD, collect 256 outputs back to same slot. n_pending
waits for core IDLE between slots.
Verified vs ml-kem-r golden: 1024/1024 shat/ehat coeffs exact (17318 cyc).
Phase-1 (d1 output) lacked the 'need_more' guard that phase-2 (d2) had, so
when the 256th accepted coefficient was a d1 whose group then advanced, the
FSM could fire one extra valid_o after last_o for certain seeds (e.g. KAT
count=0 rho, seed i=0/j=1 emitted 257 pulses). In mlkem_top KeyGen this
trailing pulse leaked into the next poly's index 0, shifting the stream.
Fix: gate phase-1 d1 output with 'd1_acc_r && need_more' (mirrors phase-2).
Applied to both sample_ntt_sync and sample_ntt_sync_shared.
Standalone TB had a coverage blind spot (stopped reading at 256, never
checked valid_o stayed low after last_o). Added a regression assertion:
counts spurious post-last_o pulses and fails if any. Verified the assertion
catches the bug on a reverted-fix copy (3 spurious) and passes on the fix.
Verified:
- 40-seed audit (sync) + 24-seed audit (shared): all exactly 256 pulses,
last_o@256, zero post-last pulses.
- Verilator vs hashlib oracle: 1536/1536 (no real coeff dropped).
- Full framework regression: 4334/4334.
- mlkem_top KeyGen Stage 2c: 2048/2048 A_hat+s+e coeffs exact.
Adds 4 Verilator cases covering d_u=11 and d_v=5 (ML-KEM-1024), which the
RTL already supports (d is 5-bit, products fit 24 bits). comp_decomp now
120/120 vectors. Also ignore .omo/ session runtime cache and archive the
06-27-sha3-g-test-specific-input trellis task.
Verified all 10 modules pass both frameworks:
- Verilator: 4334/4334 vectors
- XSIM (Vivado 2019.2): all 11 testbenches green, separate committed vectors
Oracles independently cross-checked vs hashlib (G/H/J/PRF) and FIPS 203
Alg 7/8/9/10/12 (sample_ntt, sample_cbd, ntt, poly_mul).
sample_ntt was non-conformant: both RTL and the test reference re-ran
keccak_p after every 3-byte squeeze instead of consuming the full
1344-bit SHAKE-128 rate. Only coeff[0] matched a standard sampler, so
the generated A matrix would not interoperate with any compliant ML-KEM.
- sample_ntt_sync{,_shared}.v: walk all 56 groups of the rate block via
grp_ptr_r; re-permute only when the block is exhausted. Verified
256/256 against ml-kem-r Rust sample_ntt on two seeds, and 1536/1536
in the Verilator framework (runtime ~128x faster per poly).
- gen_vectors.py: use a self-contained hashlib.shake_128 oracle.
sha3 testbench fixes (all now self-check hash_o against verified vectors,
cross-checked with hashlib and ml-kem-r mlkem_G):
- tb_sha3_xsim_simple.v: test G/H/J modes, not just G.
- tb_keccak_core_xsim.v: correct the wrong EXPECTED_STATE constant
(RTL was correct; lane0 = 0xf1258f7940e1dde7 per FIPS 202).
- tb_sha3_xsim.v: read expected file and self-check per vector; add
vectors/g_basic_{input,expected}.hex (3 G / 2 H / 2 J).
Remove stale sha3_chain test (its RTL was deleted in 1cace51) and its
README references. Extend .gitignore for XSIM artifacts and result dumps.
RTL was feeding d bytes in reversed order to SHA3-512,
causing G(d) to produce wrong rho/sigma. Fix: add d_rev
wire with byte reversal, connect to sha3_chain_top_shared.
Vivado 2019.2 xsim does not support -cd flag. Instead:
- Generate copy_vectors_pre.tcl that copies hex files
- Pass via -tclbatch for pre-simulation execution
- can then find vectors relative to xsim work dir
