At mlkem_clk=100 MHz the worst path (WNS -2.14 ns) was coeff_bram read ->
butterfly diff (b-a) -> barrett first multiply stage, all in one cycle. The
butterfly took its operands combinationally from the BRAM read (unlike
poly_mul's basecase, which already registers them). Add an input register
stage (a/b/zeta/mode/valid) in butterfly_unit_pipe so the BRAM read and the
first multiply land in separate cycles; feed the a/b/mode alignment delay
lines from the registered inputs. Butterfly latency 8->9; ntt_core writeback
address delay lines deepened 9->10 to match.
Verified byte-exact vs NIST KAT: NTT standalone TB 19/19 (fwd+inv), Encaps
9/9, Decaps 9/9, hello_world e2e — all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace poly_mul_sync's two 256x12 LUT register arrays (mem_A/mem_B, ~8k LUT)
with four sd_bram banks (A_even/A_odd/B_even/B_odd, 128x12). The base-case
reads mem[2k] and mem[2k+1] together, so an even/odd split gives each bank a
single read port. This removes the multi-port LUT mux and the long
BRAM->register-file route that was the mlkem_clk critical path.
sd_bram reads are synchronous (1-cycle); the existing 2-cycle issue cadence
absorbs it (address on the gap cycle, data on the issue cycle), with a
one-cycle priming gap at LOAD->RUN. Interface and behavior unchanged.
Verified byte-exact vs NIST KAT: poly_mul standalone TB 11/11, Encaps 9/9,
Decaps 9/9 (incl. reject + DECOMP), KeyGen, hello_world e2e — all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace ntt_core's 256x12 LUT register file (multi-port async read, ~18.5k
LUT) with coeff_bram_4way: 4 sd_bram banks, conflict-free banking function
bank(a)={a[0], ^a[7:1]}, local(a)=a[7:1]. The dual-butterfly's four accesses
{j, j+1, j+len, j+1+len} map to four distinct banks for every layer_len in
{2..128}, so each bank needs only 1 read + 1 write/cycle (single buffer, no
ping-pong). Trades ~13k LUT for +4 BRAM (device BRAM at ~6%).
Pipeline gains one stage for the synchronous BRAM read; writeback address
delay deepened to 9. coeff_out is driven combinationally from the BRAM read
during OUTPUT; unused read ports mirror port 0 outside compute so the read
crossbar arbitration is not polluted by stale addresses.
Verified byte-exact vs NIST KAT: NTT standalone TB 19/19 (fwd+inv), Encaps
9/9, Decaps 9/9 (incl. reject + DECOMP), KeyGen, hello_world e2e — all PASS.
Cycle counts unchanged (ENC_N +4 for BRAM latency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Issue 2 butterflies/cycle from the same NTT group (positions p,p+1).
ML-KEM's 7 layers have layer_len in {128..2} (always even, never 1),
so each pair shares one zeta and touches 4 distinct coefficient indices
— no intra-group RAW conflict, single zeta ROM. Compute per transform
~966->~490 cycles.
Verified: standalone ntt TB 19/19 (fwd+inv), full KAT Encaps 9/9 +
Decaps 9/9 + KeyGen + hello_world all PASS (all K).
Double-buffer poly_mul_sync so the next MAC term's 256-cycle load overlaps
the current term's accumulate. Decouple the top-level MAC load index (u_ldj)
from the accumulate index (u_j) in the shared ENC_U/ENC_V/DEC_W engine; load
reads bank_a/bank_se while accumulate reads bank_t (disjoint ports, no
conflict). Accumulate stays strictly serial (psum order-dependent); only the
load is hidden.
Verified byte-exact vs NIST KAT in ~/Dev/mlkem: Encaps 9/9, Decaps 9/9
(incl. implicit-reject + DECOMP), KeyGen all, hello_world end-to-end.
ENC_U 5585->5073, ENC_V 2793->2537 (K=2). Larger relative win at K=4
(K^2 terms). poly_mul contract unchanged (standalone TB 11/11).
Decouple the ENC_C1/C2 compress micro-sequence into a streaming
producer/collector instead of feeding one coeff and draining the whole
6-stage comp_decomp pipeline per coeff.
- comp_decomp_sync: ready_o = 1'b1 (fully pipelined, consumer never
backpressures since ready_i is tied high), so feeders can issue a new
coeff every cycle.
- mlkem_top ENC_C1/C2: producer feeds one coeff every 2 cycles
(<=5.5 bit/cyc, under the single-port ct drain of 8 bit/cyc so the bit
buffer stays bounded); a per-cycle collector appends compressed bits
and drains one ct byte/cycle.
Verified byte-exact against NIST KAT (Vivado 2019.2 XSIM) for KeyGen,
Encaps (9/9), Decaps (9/9 incl. reject path + DECOMP), hello_world e2e.
K=2 Encaps: ENC_C1 6273->1035, ENC_C2 2945->522, total 27851->20190.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous Barrett multiplication-based division was incorrect. Both
compress and decompress used the same wrong implementation, so they
happened to be consistent with each other, but the math was wrong.
Replace with synthesizer-native / and % operators. This is cleaner
and the synthesis tool handles integer division/remainder correctly
for these small operand widths (12-bit dividend, 12-bit divisor).
Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)
Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
The core's ek_rd_addr/dkp_rd_addr muxes give dbg_dk_idx_i priority over
dbg_byte_idx_i whenever it falls in the ek (768..1567) or dk_pke (<768)
region. The wrapper drove both dbg_byte_idx_i and dbg_dk_idx_i from the
same rdb_addr, so an ek readback of the rho tail (bytes 768..799) was
hijacked to read ek[0..31] instead — corrupting the public key fed to
Encaps and causing Decaps implicit rejection (full 32-byte ss mismatch).
Present rdb_addr on dbg_dk_idx_i only when rdb_sel==2 (full-dk); hold it
at 0xFFF otherwise so ek/dk_pke readbacks aren't corrupted. Verified by
KeyGen->Encaps->Decaps round trip (MLKEM_RESULT=PASS).
Wrap mlkem_top in an AXI4 burst slave (mlkem_axi_slave) mapped to the
freed FFT slot 6 at 0x1f400000 (virtual 0xbf400000). The core runs on
sys_clk/2 (25 MHz); the wrapper handles the clock-domain crossing:
start toggle->pulse, wait-stated byte-preload handshake, busy/done
sync, and direct readback taps (ek/dk/ct/ss). CPU drives it via a
register map (CTRL/STATUS/CONFIG, d/z/msg, preload + readback windows).
Wire into soc_top on the existing axiOut_6 nets (no crossbar change),
add mlkem_intr on intrpt[3]. Add sdk mlkem_test example running a
KeyGen->Encaps->Decaps self-consistency round trip.
