- Replace / and % operators in comp_decomp_sync with Barrett multiply-by-reciprocal
(dividend * 5039 >> 24) + correction step. Eliminates ~100 CARRY4 divider chain.
- Add include_dirs for sources_1 fileset so Windows Vivado synthesis finds defines.vh.
- Change CLK_PERIOD from 10.0 (100MHz) to 40.0 (50MHz) to reflect actual target.
Add (* use_dsp = "no" *) to all modules containing multiplication
operators to force LUT-based multiplication instead of DSP inference:
- barrett_mul.v (3 multipliers x 7 instances = 21 ops)
- comp_decomp_sync.v, sample_ntt_sync*.v, mlkem_top.v
Also fix duplicate wire declaration of ct_bytes_rt in mlkem_top.v.
Add tb_mlkem_two_inst_xsim: runs the hello_world protocol across TWO mlkem_top
instances, modeling the real two-party split:
* u_genenc: KeyGen THEN Encaps on one instance. KeyGen writes ek into its own
ek_bram and Encaps reuses it directly (no re-streaming) -> shared_key, ct.
* u_dec: Decaps on a separate instance, receiving dk + ct streamed over from
u_genenc via the input ports.
Verifies A.shared_key == B.recovered_key and 'hello world' round-trips. Output
matches the single-instance TB and the Rust reference (key=ced0c031a4bee34a...).
run_hello.sh gains a 'two' arg to select this TB; default stays single-instance.
Mirror of ml-kem-r examples/hello_world.rs on the mlkem_top DUT (ML-KEM-512):
1. Alice KeyGen(d=0x42.., z=0x77..) -> ek (800B), dk (1632B)
2. Bob Encaps(ek, m=0xDE..) -> shared_key, kem_ct (768B)
3. Bob XOR-encrypt "hello world"
4. Alice Decaps(dk, kem_ct) -> recovered_key
5. Alice XOR-decrypt -> "hello world"
The whole protocol runs on ONE DUT instance: ek/dk are read out of KeyGen via
the dbg taps and fed back into Encaps/Decaps through the streaming input ports,
just as the keys/ciphertext would cross the wire between Alice and Bob. Each
step prints its inputs and outputs.
Output is byte-identical to the Rust example: shared_key=ced0c031a4bee34a...,
encrypted=a6b5ac5dcb9e9425b9e3b8, decrypted="hello world", keys match.
run_hello.sh compiles the RTL + TB and runs it. Cycle counts (K=2): KeyGen
~22.9k, Encaps ~32.5k, Decaps ~50.8k.
Final FO step (FIPS 203 Alg 17 steps 9-11): compare c' to c and select the
shared secret with implicit rejection. Completes full ML-KEM Decaps.
- ST_DEC_CMP walks all ct bytes (no early-out: constant work, matching the
constant-time spec intent), reading c' (ct_bram) and c (c_in_bram) in
parallel, OR-ing every byte difference into cmp_neq; the final byte latches
dec_reject.
- ss_o = dec_reject ? kbar_r (K-bar) : ss_r (K'). KeyGen/Encaps leave
dec_reject=0 so ss_r passes through unchanged.
- ST_ENC_C2 terminal branches on op_r: Decaps -> ST_DEC_CMP, Encaps -> DONE.
The dec TB now runs end-to-end twice per case (dk loaded once):
- valid ct : ss_o == KAT ss (c'==c -> K'), dec_reject==0
- corrupt ct: ss_o == KAT ss_n (c'!=c -> K-bar=J(z||ct_n)), dec_reject==1
exercising both the accept and implicit-reject paths against the KAT's own
ct_n/ss_n reject vectors.
Verified: dec D7 K=2/3/4 all cases PASS (accept + reject); KeyGen + Encaps
unregressed. Full ML-KEM (KeyGen + Encaps + Decaps w/ implicit rejection) now
works in hardware across all three parameter sets.
Re-encryption step of the FO transform (FIPS 203 Alg 17 step 8), done by
reusing the ENTIRE Encaps E1-E7 pipeline rather than duplicating it:
- FSM: ST_DEC_J (D5) -> ST_ENC_LOAD, then the existing Encaps chain
LOAD->A->C->N->U->C1->TDEC->E2MV->V->C2 runs unchanged and writes c' to
ct_bram. The reuse preconditions are all in place: rho loads from ek_bram's
ek_pke region (same 384k offset Encaps uses; populated at D0 load via
dk_ld_ekpke), the CBD seed is r_r (r' from D5), and ek_pke is in ek_bram.
- D4 now packs the recovered message directly into m_r (dropping the separate
mprime_r register): Encaps V's mu reads m_r[idx] and dbg_mprime_o now aliases
m_r, so the re-encrypt sees m' with no extra plumbing.
- ST_ENC_LOAD arming generalized to fire when entered from ST_ENC_G (Encaps)
or ST_DEC_J (Decaps re-encrypt).
The re-encrypt overwrites bank_a/bank_se/bank_t, so the bank-based stage checks
(D1 v', D2 s_hat/u_hat, D3 w) are no longer valid at end-of-run. The dec TB now
verifies the surviving register/BRAM artifacts: dk parse (D0), m' (D4, in m_r),
K'/r'/K-bar (D5), and the 768/1088/1568-byte c' against golden (D6). Earlier
stages remain proven by their per-stage builds and transitively by c'.
Verified: dec D6 K=2/3/4 all cases PASS (c' == golden == valid ciphertext c);
KeyGen + Encaps unregressed.
FO transform hash derivations (FIPS 203 Alg 17 steps 6-7), reusing the
shared SHA3 core:
- ST_DEC_G: (K',r') = G(m'||h) via the single-block SHA3-512 path (mode 11,
dec_g_data = {hek_r, mprime_r}; h was captured into hek_r at D0 load).
K' -> ss_r (candidate shared secret, ss_o), r' -> r_r (PRF seed for D6).
- ST_DEC_J: K-bar = J(z||c) via the multi-block absorb port (mb_en=1),
modeled on the H(ek) machine: assemble 136-byte blocks, byte source is
z_r (g<32), c_in_bram (32<=g<msglen), or SHAKE256 pad (0x1F suffix, last
byte |=0x80 -- the only difference from H's 0x06). mb_* inputs muxed
between H and J by state. K-bar -> kbar_r (dbg_kbar_o).
- FSM: MENC -> G -> J -> DONE.
Bring-up note: c_in_bram read through cin_rd_addr_r (a register) plus the
registered BRAM is 2-cycle latency, but the assemble/writeback pipeline only
budgets 1 -- so the first c byte at the z->c boundary read X and poisoned the
whole sponge. Fixed by driving cin_rd_addr combinationally from dj_c_idx
during ST_DEC_J (dropping the register stage) so data lands the next cycle.
Verified: dec D5 K=2/3/4 all cases PASS (ct 768/1088/1568B -> 6/9/12 J blocks);
K' matches the KAT shared secret for valid ciphertexts; KeyGen + Encaps
unregressed.
Completes K-PKE.Decrypt (FIPS 203 Alg 15) in hardware: m' is recovered.
D3 (ST_DEC_W) reuses the Encaps V MAC/INTT machine (u_row tied to 0):
- MAC s_hat[j] (bank_a slot j*K) o u_hat[j] (bank_se rel j) -> psum bank_t[UPSUM]
-- identical addressing to Encaps V (t_hat[j] o y_hat[j]), so free reuse.
- INTT(psum) in place.
- SUB: w = v' - psum mod Q (negative -> +Q), written to bank_t[UPSUM].
To read v' and psum in parallel during SUB (one read port per bank), D1's v'
write was relocated from bank_t to bank_a slot DEC_VASLOT=1 (always free:
s_hat occupies j*K, slot 1 is unused for K>=2). This mirrors V-ADD reading
psum (bank_t) + e2 (bank_a) simultaneously.
D4 (ST_DEC_MENC): m' = byteEncode_1(Compress_1(w)). Compress_1(w)=1 iff
832 < w <= 2496 (Q=3329); bits packed LSB-first into mprime_r, exposed on
dbg_mprime_o (was a placeholder tied to m_r).
Added ST_DEC_W to the u_* machine muxes/sub-phases and the FSM chain
NTT->W->MENC->DONE. TB verify_d3 checks w (bank_t UPSUM); verify_d4 checks the
32-byte m' against golden (== the KAT-decrypted m == original message).
Verified: dec D1-D4 K=2/3/4 all cases PASS; KeyGen + Encaps unregressed.
K-PKE.Decrypt steps 2-3 (FIPS 203 Alg 15), both by reusing Encaps machines:
- ST_DEC_SDEC reuses the Encaps TDEC (byteDecode12) machine: only the byte
source changes (td_byte mux -> dkp_rd_data; dkp_rd_addr driven by td_ekaddr
in SDEC). Decodes dk_pke -> s_hat[j] into bank_a slot j*K, the same layout
t_hat uses, so the D3 MAC can read s_hat[j] with the existing addressing.
- ST_DEC_NTT reuses the forward-NTT machine (n_slot_max=k_r) to transform
u'[i] in place in bank_se rel slots 0..K-1 -> u_hat[i]. Added ST_DEC_NTT to
the bank_se read/write muxes and the NTT load/process/arm blocks alongside
ST_N/ST_ENC_N.
- FSM: DECOMP -> SDEC -> NTT -> DONE.
TB verify_d2 checks s_hat[i] (bank_a slot i*K) and u_hat[i] (bank_se rel i)
against golden. verify_d1 narrowed to v' only: D2's in-place NTT overwrites u'
in bank_se, so u' correctness is now proven transitively via u_hat==NTT(u').
Verified: dec D2 K=2/3/4 all cases PASS; KeyGen + Encaps unregressed.
K-PKE.Decrypt step 1 (FIPS 203 Alg 15): decode+decompress the ciphertext.
- comp_decomp_sync instance made mode-selectable: Encaps C1/C2 still compress
(mode 0), Decaps ST_DEC_DECOMP decompresses (mode 1) with d=du/dv.
- New ST_DEC_DECOMP state with an inline byteDecode_d walker (reverse of the
C1/C2 bit-packer): walks c_in_bram bytes, accumulates LSB-first into a bit
buffer, extracts d-bit symbols, feeds comp_decomp, writes each decompressed
coeff (mod q) to a bank.
c1 = K polys, d=du -> u'[i] in bank_se rel slot i (0..K-1)
c2 = 1 poly, d=dv -> v' in bank_t rel slot DEC_VSLOT=2 (avoids UPSUM=1)
- dbg_slot_i widened 4->6 bits so the TB can read v' (abs slot 26 at K=4).
- bse/bt write muxes gain the DECOMP writeback paths (dec_u_we / dec_v_we).
Verification: examples/dump_decaps.rs (ml-kem-r worktree) emits per-stage
golden (u'/v'/s_hat/u_hat/w/m') into vectors/decgold/. TB verify_d1 reads back
u'[i] and v' and compares all 256 coeffs each.
Bring-up note: dbg coeff readback latency is bank(1)+dbg_coeff_r(1); the TB's
rdcoeff initially waited 2 cyc and saw data shifted by one index -> fixed to 3.
Verified: dec D1 K=2/3/4 all cases PASS; KeyGen + Encaps unregressed.
Verilator is no longer used (all verification is via Vivado XSIM). Remove:
- 10 per-module tb_*.cpp Verilator testbenches
- the entire test_framework/ Verilator harness (lib/, run_all.py, config.json,
per-module test_plan.json/gen_vectors.py, golden vectors, reports)
- stale specs: verilator-conventions.md, test_framework/structure.md
(index.md updated to drop the Verilator entry)
Parallelize run_tb.sh K x case execution (modules stay serial):
- new run_xsim_jobs helper: compile+elaborate once (serial, populates the
shared xsim.dir), then run each (K,case) xsim in its own private workdir
with a COPY of xsim.dir (~1MB) so concurrent same-snapshot runs don't clobber
each other's runtime logs. Each workdir symlinks the repo sync_rtl tree so
the TB's repo-relative $readmemh vector paths resolve.
- top/enc/dec runners refactored to build a (snapshot:K:case) spec list and
hand it to run_xsim_jobs; ordered PASS/FAIL summary + per-job /tmp logs
preserved. Bare './run_tb.sh top' now also takes the parallel path.
Speedup (20 cores): top full sweep 2:11 -> 0:51 (~2.6x), ~320% CPU.
Verified: top (11) / enc (9) / dec (9) all PASS; missing-vector runs still
fail (file-not-found guard -> exit 1).
Scaffolding for ML-KEM Decaps (FIPS 203 Alg 18):
- op_i widened to 2-bit: 00=KeyGen, 01=Encaps, 10=Decaps (op_r too).
- New ST_DEC_LOAD state (D0: settles to DONE so load/parse is dbg-checkable).
- dk (=sk) streamed via dk_in_*; load logic routes each byte by region:
[0,384K)->dk_pke (dkp_bram), [384K,768K+32)->ek_pke (ek_bram),
[768K+32,+32)->H(ek) (hek_r), [768K+64,+32)->z (z_r). Routing uses the
LIVE k_i input, not start-captured k_r (dk is streamed before start_i).
- c (=ct) streamed via c_in_* into a SEPARATE c_in_bram, so the computed c'
(ct_bram) can later be compared against original c and J(z||c) can read c.
- New dbg taps: dbg_mprime_o/dbg_kbar_o/dbg_decz_o/dbg_dech_o.
TB: tb_mlkem_dec_katK_xsim verifies dk parse (H(ek), z, ek_pke/dk_pke BRAM
round-trip). gen_decaps_vectors.py emits dec_k{K}_c{N}_{dk,ct,ss,ctn,ssn}.hex
from the NIST KAT. run_tb.sh gains a 'dec' module (mirrors 'enc').
Regression fix: old KeyGen/Encaps TBs didn't connect the new input ports,
floating them to X and corrupting the ek/dkp write muxes -> tied off
dk_in_*/c_in_*/new dbg taps in both.
Verified: dec D0 K=2/3/4 PASS; KeyGen K=2 + Encaps K=2 unregressed.
run_tb.sh gains an 'enc' module that shares the 'top' KeyGen tcl compile
list (same RTL datapath) and swaps in tb_mlkem_enc_katK_xsim. Usage mirrors
'top': ./run_tb.sh enc [K] [CASE]; no args -> full sweep K=2/3/4 cases 0..2.
--list shows enc; per-case summary lines parse PASS (E7). run_enc.sh deleted.
tb_mlkem_enc_katK_xsim verify_e7 now dumps the hardware-produced ct (read
from ct_bram via the dbg_ct tap) on one line (byte 0 first), same format as
ml-kem-r's encaps_io example, so ss/ct can be eyeballed and diffed. On
mismatch it re-scans to print the first 8 differing byte positions.
Verified: ./run_tb.sh enc -> 9/9 PASS (E7) (ct==KAT.ct && ss==KAT.ss);
./run_tb.sh top 3 0 KeyGen unregressed.
Reuses the E5 bit-packer FSM for ST_ENC_C2, generalized over region:
- coeff source: cp_coeff_src = bank_t[UPSUM] (v) for C2, bank_se (u) for C1
- bit width: cp_d already = dv_rt for C2 else du_rt
- poly count: cp_poly_max = 1 (single v) for C2, K for C1
- ct write pointer cp_wa CONTINUES from c1_bytes into C2 (not reset), so c2
lands right after c1. c1 ends on a whole-byte/poly boundary (256*du/8
integral), so cp_buf/nbits are empty at the C1->C2 handoff.
FSM tail: ST_ENC_V -> ST_ENC_C2 -> ST_DONE.
TB: verify_e7 compares the full ct (c1||c2, CTB bytes) to KAT.ct byte-exact
via the dbg_ct tap. Combined with the E0 ss==KAT.ss check this is the full
end-to-end Encaps KAT (ct==KAT.ct && ss==KAT.ss).
Verified end-to-end for K=2/3/4, cases 0-2 (K2) / 0-1 (K3,K4):
ct==KAT.ct && ss==KAT.ss. ML-KEM Encaps complete. KeyGen unregressed.
Storage choreography (per plan risk notes):
- TDEC now decodes t_hat[j] into bank_a slot j*K (was bank_t). This makes
V's MAC reuse E4's u_aslot=u_j*K+u_row addressing with u_row=0, no mux
change. bank_t has no room for K=4 (t_hat would fill all 4 slots vs
psum's UPSUM slot), hence bank_a (16 slots, A_hat dead after E4).
- New ST_ENC_E2MV state relocates e2 (bank_t[0]) -> bank_a[1] so V-ADD
reads psum (bank_t) and e2 (bank_a) from different banks (no port
conflict). bank_se (y_hat + u) stays intact -> verify_e2/e3/e4 unaffected.
- V reuses the u_* MAC/INTT/ADD machine with u_row tied to 0 (u_row_max=1).
ADD computes psum + e2 + mu mod Q -> bank_t[UPSUM] in place;
mu[w] = m_r[w] ? 1665 : 0 (Decompress_1). FSM: C1->TDEC->E2MV->V->DONE.
Bug found+fixed during bring-up: e2 relocation was off-by-one (wrote
e2[i+1] into slot i) because em_we/em_widx were registered an extra cycle
past the bram read. Fixed: em_widx==em_ridx, write scheduled for the cycle
bt_rd_data presents e2[em_ridx].
TB: verify_e6 compares v (bank_t dbg slot 9, K=2) to ml-kem-r golden.
verify_e1 dropped (TDEC overwrites bank_a A_hat slots; A_hat transitively
verified by E4). Verified: K=2 E2/E3/E4/E6 == golden, E5 c1 == KAT prefix;
K=3/4 E0+E5 pass; KeyGen K=2 unregressed.
mlkem_top now instantiates comp_decomp_sync (E5), so the shared
xsim_run.tcl must compile it (+ its pipeline_reg dep) before mlkem_top.
This unbreaks 'run_tb.sh top' which failed elaboration with
'Module comp_decomp_sync not found'. run_enc.sh simplified to reuse the
tcl's now-complete compile list (drops its duplicate leaf block).
ST_ENC_C1: per-coeff Compress_du via comp_decomp_sync (mode 0) then
LSB-first byte packing into ct_bram. 5-phase micro-seq reads u[cp_poly]
from bank_se (rel K+poly), feeds the compressor (1-cyc pipe), appends du
bits to cp_buf, and drains whole bytes. Each poly = 256*du bits (whole
bytes) so the bit buffer empties at every poly boundary.
ST_ENC_U now advances to ST_ENC_C1 (was ST_DONE).
TB: verify_e5 compares ct_bram[0..c1_bytes-1] to the KAT.ct prefix via
the dbg_ct tap. run_enc.sh: encaps TB runner (compiles comp_decomp_sync
which the KeyGen tcl omits).
Verified K=2/3/4 c1 == KAT.ct prefix (640/960/1408 B; K=4 du=11
cross-byte path), K=2 cases 0-2.
ST_ENC_U computes u[i] per row in 3 sub-phases reusing shared u_pmul + u_ntt:
sub0 MAC : sum_j A_hat[j][i] o y_hat[j] (TRANSPOSE: slot=j*K+i) -> NTT-domain
psum in bank_t rel slot UPSUM=1 (e2 in slot 0), init 0 at j==0
sub1 INTT: INTT(psum) mode=1 (built-in x3303) in place in bank_t[UPSUM]
sub2 ADD : u[i][w] = psum[w] + e1[i][w] mod Q -> bank_se rel (K+i), over e1
y_hat (bank_se 0..K-1) preserved for V. ntt_core mode + input muxed for the
INTT sub-phase; bank_a/se/t read+write ports extended for all 3 sub-phases.
Fixed a duplicate 'assign bse_we' (stale + new both present -> ADD writes
X-dropped); collapsed to one. Verified (K=2 c0) u[0..1] == ml-kem-r golden
(transpose + INTT + e1 all correct); E0/E1/E3 pass, E2 trimmed to e2 (e1
consumed into u, transitively checked by E4); K=3/4 no timeout.
ST_ENC_N reuses the ST_N forward-NTT datapath (mode=0, read-ahead load,
in-place writeback to bank_se). Slot count parameterized: KeyGen 2K (s,e),
Encaps K (y only; e1/e2 stay time-domain) via n_slot_max. bse rd/wr muxes
and the C->N arming extended for ST_ENC_N.
Verified (K=2 c0) y_hat[0..1] == ml-kem-r golden; E0/E1/E2 still pass
(E2 y check dropped since NTT overwrites y in place -> covered by E3);
K=3/4 run through E3 without timeout.
ST_ENC_C reuses the ST_C CBD datapath with Encaps muxes: seed=r (not sigma),
eta=eta1 for y[0..K-1] then eta2 for e1[0..K-1]/e2, nonce 0..2K. 2K+1 polys
(vs KeyGen 2K). y/e1 -> bank_se rel slots 0..2K-1; e2 -> bank_t rel slot 0
(free during C/N/U since TDEC is deferred to V-prep so the 28-slot banks hold
all of A+y_hat+e1+e2 at K=4 without resizing).
Bring-up golden via ml-kem-r dump_encaps_full (working-tree example):
vectors/encgold/ec_k2_c0_{y,e1,e2}.hex. Verified (K=2 c0) y[0..1],e1[0..1],e2
== ml-kem-r; A_hat (E1) and ss (E0) still pass; K=3/4 no timeout.
ST_ENC_LOAD: stream rho (32B) from ek_bram[384k..] into rho_r (read-ahead,
1-cyc bram latency). ST_ENC_A: regenerate A_hat via SampleNTT into bank_a
(reuses ST_A datapath, gated on st==ST_A||ST_ENC_A). ST_ENC_TDEC:
byteDecode12 ek[i*384..] -> t_hat[i] into bank_t (5-cycle micro-phase per
3-byte/2-coeff triple; bt write port muxed with ST_M).
Verified (K=2 c0) A_hat (1024 coeffs) + t_hat (512) == KeyGen golden via
dbg_coeff_o; E0 ss==KAT.ss still passes all K/cases (no timeout).
Extend mlkem_top with a runtime op_i select (0=KeyGen, 1=Encaps) and the
first Encaps stages, reusing the shared keccak_core and the ST_H multi-block
SHA3-256 machinery:
ST_ENC_H: H(ek) over preloaded ek_bram (same FSM as KeyGen ST_H)
ST_ENC_G: (K,r) = G(m||H(ek)) via new 64-byte single-block SHA3-512
- sha3_top_shared: add mode=2'b11 = SHA3-512 over a full 512-bit message
(g512_pad). Standalone tb_sha3_g512 confirms it byte-exact.
- mlkem_top: new ports op_i, msg_i, ek_in_{we,addr,byte} (ek preload), ss_o,
dbg_ct_*, dbg_r_o/dbg_hek_o. st widened 4->5 bits; ST_ENC_* states added.
Renamed message port to msg_i to avoid collision with ST_M counter m_i.
- TB tb_mlkem_enc_katK + gen_encaps_vectors.py (per-byte ek/m/ct/ss vectors).
Verified ss==KAT.ss for K=2/3/4, cases 0-2 (all PASS). KeyGen unaffected
(K=2 c0 still ek==pk, dk==sk byte-exact).
Cross-checks the hardware KAT vectors against the ml-kem-r Rust reference's
NIST .rsp files, accounting for the d/z byte-order convention: the hardware
hashes G(reverse(d)||K), so its vector files store d/z reversed vs the NIST
.rsp. The script applies that reversal for d/z and compares ek==pk, dk==sk
directly; 11/11 cases consistent.
ml-kem-r path via $ML_KEM_R (default ~/Dev/ml-kem-r); friendly errors on
missing files. Documents the convention that tripped up logging.rs (which
pasted hardware-order d into the literal-order Rust reference).
The KAT TB now prints, for each run: the d and z seeds (32-byte hex, MSB-first
== vector file order) before start, and the full ek/dk byte strings read back
from the DUT (32 bytes/line, offset-prefixed) after the byte-compare. Inputs
verified to match the kat_*_d/z.hex files exactly; dk[768..] echoes ek[0..] as
expected (dk_pke || ek || H(ek) || z). Pass/fail logic unchanged.
Final bank promoted to sd_bram (the busiest: 5 read sites, 2 write sites).
Read port phase-muxed: ST_N load / ST_M load (pm_b s_hat[j]) vs accumulate
(e_hat, selected by m_loading) / ST_E dk-half / dbg. Write port combinational:
ST_C CBD store vs ST_N NTT writeback (disjoint states). All explicit consumer
read registers (n_rd_data, pm_b_rd, m_eacc_rd, e_se_rd) collapsed into the
sd_bram internal read register; m_acc_src and e_rd_coeff now select between
two registered sd_bram outputs (same 1-cycle latency).
mlkem_top now contains ZERO behavioural RAM arrays: all coefficient storage is
3 sd_bram banks (a/se/t) + ek/dkp byte buffers = 5 sd_bram instances total,
each inferring BRAM (ASIC: compiled SRAM). 11/11 KAT PASS, byte-exact.
Second bank promoted to sd_bram. Read port muxed by phase (ST_M acc t_hat
addr / ST_E ek-half byteEncode addr / dbg). The shared mux-registers that fed
both bank_se and bank_t were split: the bank_se half stays a manual reg
(m_eacc_rd / e_se_rd), the bank_t half uses sd_bram's internal read register
(bt_rd_data) -- same 1-cycle latency, so the j-select (m_jq) and ST_E coeff
select (e_rd_coeff) just pick between the two registered outputs. RMW safe:
acc read addr (m_oidx+1) leads write addr (m_oidx), no same-cycle alias.
Write port combinational. 11/11 KAT PASS incl. K=3/4 deep accumulate.
First of 3 banks promoted from async reg array to an sd_bram. Read port
muxed by phase (ST_M load drives pm_a_full, else dbg index); sd_bram's
internal rd_addr_r replaces the explicit pm_a_rd reg (identical 1-cyc
latency). Write port driven combinationally so the posedge write matches the
old nonblocking reg-array timing. 11/11 KAT PASS, byte-exact.
Final per-consumer step of stage 2b. ST_E read two coeffs of a pair from one
bank combinationally; now serialized through a single registered read port
(e_rd_data_r) over a 4-cycle micro-phase e_ph: ph0 fetch c0, ph1 write b0 +
fetch c1, ph2 write b1, ph3 write b2. Matches sd_bram timing; +512 cyc on K=2
(one extra fetch per pair-stream). All polymem read consumers (ST_N, ST_M
load, ST_M acc, ST_E) are now registered-read. 11/11 KAT PASS, byte-exact.
Third per-consumer step of stage 2b, the read-modify-write one. m_acc_src is
now a registered read (m_acc_rd) whose address leads by pm_vo (m_oidx+1 when
consuming) so the value lands exactly on the pm_vo cycle. Same-index RMW on
bank_t is read-old: the running-t_hat source was written by the previous term
and is long settled; read addr (m_oidx+1) leads write addr (m_oidx).
11/11 KAT PASS incl. K=3/4 j>0 paths, byte-exact.
Second per-consumer step of stage 2b. m_ld becomes a read-ahead pointer; both
poly_mul load reads (bank_a A_hat, bank_se s_hat) are registered into
pm_a_rd/pm_b_rd and fed one cycle later (pm_valid delayed 1 cyc), same pattern
as ST_N. 11/11 KAT PASS, byte-exact.
First per-consumer step of stage 2b. n_ridx becomes a read-ahead pointer
leading the consume index by 1; bank_se read is registered into n_rd_data and
fed to ntt_core one cycle later (n_valid delayed 1 cyc). Cores hold ready_o
high through LOAD so a fixed 1-cycle skew suffices. Matches sd_bram registered
timing for this read port. 11/11 KAT PASS, byte-exact.
Replace the single async-read polymem[0:28*256-1] with 3 polynomial-indexed
banks (bank_a A_hat / bank_se s_hat||e_hat / bank_t t_hat), addressed by
abs_slot - base_slot. Still async-read here -- a pure refactor that validates
bank sizing and base-relative addressing with zero timing change before
stage 2b converts them to registered sd_bram + read-ahead pipelines.
11/11 KAT PASS, byte-exact, 0 file-not-found.
KeyGen's Keccak consumers (G/H via sha3, A via SampleNTT, C via CBD) run in
disjoint top-FSM phases, so one keccak_core suffices. Add sha3_top_shared
(keccak_core externalised as kc_* ports, like the existing sample_*_shared
variants); mlkem_top now instantiates one keccak_core and a phase mux that
routes kc_state_i/kc_valid_i from the active consumer and gates kc_valid_o
per consumer (inactive samplers latch squeeze state unconditionally).
Cuts the KeyGen datapath from 4 keccak_core (1600-bit state + 24-round logic
each) to 1 -- the dominant ASIC area win. 11/11 KAT PASS (K=2 c0-4, K=3/4
c0-2), byte-exact, 0 file-not-found.
ST_G (single-block G) and ST_H (multi-block H(ek)) are disjoint FSM phases,
so one sha3_top serves both: mb_en and ready_i are phase-muxed, h_hash/h_vo
alias the shared core's outputs. Removes the dedicated u_sha3_h instance and
its keccak_core. 11/11 KAT PASS (K=2 c0-4, K=3/4 c0-2), byte-exact, 0 file-not-found.
Phase 1 of migrating mlkem_top's large arrays to inferable RAM. ek_mem
and dkp_mem reg arrays are replaced by two sd_bram instances (1R/1W,
registered read). Datapath changes to fit single-port-per-cycle BRAM:
- ST_E writes 1 byte/cycle (was 3): added e_byte sub-counter; ST_E
length ~3x (K=2 KeyGen 21403->22433 cyc, ~5%).
- ST_H ek read is now registered: assemble phase presents the address
one cycle ahead and writes back the byte that arrived (h_wb_* pipe),
h_byte runs 0..136 to flush the final byte. Pad bytes via h_padconst.
- dbg_byte_o/dbg_dk_o read combinationally off the BRAM registered
output (net 1-cycle latency, within the TB's 2-cycle read wait);
region decode for dk readback unchanged.
Add sd_bram.v to the top TB compile list. Verified byte-exact vs NIST
KAT: K=2 c0-4, K=3 c0-2, K=4 c0-2 -> 11/11 PASS, 0 file-not-found.
mlkem_top now sizes storage for KMAX=4 (worst case) and selects the
active ML-KEM parameter set at start_i via the k_i input. All K-derived
quantities (eta1, slot bases, ek/dk byte counts, H(ek) block count, FSM
bounds) are computed at runtime from the captured k_r.
Verified byte-exact against NIST KAT for all three parameter sets:
K=2 (512) cases 0-4, K=3 (768) cases 0-2, K=4 (1024) cases 0-2
-> 11/11 PASS (ek==pk, dk==sk).
The parametric KAT TB (KP generic) builds filenames kat_k<K>_c<n>_*.hex, but
the K=2 vectors were committed earlier as kat_c<n>_*.hex (no k2 prefix), so
run_tb.sh top emitted 'file cannot be opened' warnings for K=2 and the data
read as X. Add kat_k2_* copies so all three parameter sets load cleanly.
./run_tb.sh top now: 0 file warnings, all 11 cases PASS (K=2:0..4, K=3:0..2,
K=4:0..2), exit 0.
./run_tb.sh top now compiles the full KeyGen datapath + tb_mlkem_kg_katN
and runs all 5 NIST KAT cases (ek==pk 800B, dk==sk 1632B byte-exact).
Also registers 'top' in ./run_tb.sh --list.
Add ST_H stage: second sha3_top (mb_en=1) computes H(ek) over 800B ek as 6
pre-padded SHA3-256 rate blocks. Per block: assemble 136 bytes (h_padbyte
applies 0x06...0x80 padding on final block) into h_block_r, feed (hold valid
until mb_ready drops), wait permute; capture digest on last block into hek_r.
Full dk readback tap: dk = dk_pke(768) || ek(800) || H(ek)(32) || z(32) = 1632B.
End-to-end TB (tb_mlkem_kg_kat_xsim, no force/release): drive KAT count=0 d/z,
run full KeyGen FSM (IDLE->G->A->C->N->M->E->H->DONE), verify:
ek == KAT pk (800B) byte-exact
dk == KAT sk (1632B) byte-exact
Done in 21403 cycles. ML-KEM-512 KeyGen complete and KAT-verified.
Prior stage TBs (2c/2e/2f) still pass (no regression).
Add ST_N stage: forward NTT (ntt_core mode=0, no scaling) of s[0],s[1],
e[0],e[1] in place (slots S0,S1,E0,E1). Per slot: stream 256 coeffs into
ntt_core during LOAD, collect 256 outputs back to same slot. n_pending
waits for core IDLE between slots.
Verified vs ml-kem-r golden: 1024/1024 shat/ehat coeffs exact (17318 cyc).
Phase-1 (d1 output) lacked the 'need_more' guard that phase-2 (d2) had, so
when the 256th accepted coefficient was a d1 whose group then advanced, the
FSM could fire one extra valid_o after last_o for certain seeds (e.g. KAT
count=0 rho, seed i=0/j=1 emitted 257 pulses). In mlkem_top KeyGen this
trailing pulse leaked into the next poly's index 0, shifting the stream.
Fix: gate phase-1 d1 output with 'd1_acc_r && need_more' (mirrors phase-2).
Applied to both sample_ntt_sync and sample_ntt_sync_shared.
Standalone TB had a coverage blind spot (stopped reading at 256, never
checked valid_o stayed low after last_o). Added a regression assertion:
counts spurious post-last_o pulses and fails if any. Verified the assertion
catches the bug on a reverted-fix copy (3 spurious) and passes on the fix.
Verified:
- 40-seed audit (sync) + 24-seed audit (shared): all exactly 256 pulses,
last_o@256, zero post-last pulses.
- Verilator vs hashlib oracle: 1536/1536 (no real coeff dropped).
- Full framework regression: 4334/4334.
- mlkem_top KeyGen Stage 2c: 2048/2048 A_hat+s+e coeffs exact.
