At 100 MHz the worst path (WNS -1.58 ns) was CBD sample: squeeze_buf[buf_ptr]
extraction -> CBD add/sub -> mod-Q -> bank_t/bank_se write, all combinational
in one cycle and routing-dominated (squeeze_buf and the banks place far apart).
Register the CBD write (data/addr/enable) so the bank write commits one cycle
later; the registered write takes priority in the bank_se/bank_t mux so the
1-cycle spillover into the next state uses the correct addr/data. No consumer
reads these coeffs until the following stage (>1 cycle later), so no RAW hazard.
Verified byte-exact vs NIST KAT: Encaps 9/9, Decaps 9/9, KeyGen 11/11,
hello_world e2e -- all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
At mlkem_clk=100 MHz the worst path (WNS -2.14 ns) was coeff_bram read ->
butterfly diff (b-a) -> barrett first multiply stage, all in one cycle. The
butterfly took its operands combinationally from the BRAM read (unlike
poly_mul's basecase, which already registers them). Add an input register
stage (a/b/zeta/mode/valid) in butterfly_unit_pipe so the BRAM read and the
first multiply land in separate cycles; feed the a/b/mode alignment delay
lines from the registered inputs. Butterfly latency 8->9; ntt_core writeback
address delay lines deepened 9->10 to match.
Verified byte-exact vs NIST KAT: NTT standalone TB 19/19 (fwd+inv), Encaps
9/9, Decaps 9/9, hello_world e2e — all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace poly_mul_sync's two 256x12 LUT register arrays (mem_A/mem_B, ~8k LUT)
with four sd_bram banks (A_even/A_odd/B_even/B_odd, 128x12). The base-case
reads mem[2k] and mem[2k+1] together, so an even/odd split gives each bank a
single read port. This removes the multi-port LUT mux and the long
BRAM->register-file route that was the mlkem_clk critical path.
sd_bram reads are synchronous (1-cycle); the existing 2-cycle issue cadence
absorbs it (address on the gap cycle, data on the issue cycle), with a
one-cycle priming gap at LOAD->RUN. Interface and behavior unchanged.
Verified byte-exact vs NIST KAT: poly_mul standalone TB 11/11, Encaps 9/9,
Decaps 9/9 (incl. reject + DECOMP), KeyGen, hello_world e2e — all PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace ntt_core's 256x12 LUT register file (multi-port async read, ~18.5k
LUT) with coeff_bram_4way: 4 sd_bram banks, conflict-free banking function
bank(a)={a[0], ^a[7:1]}, local(a)=a[7:1]. The dual-butterfly's four accesses
{j, j+1, j+len, j+1+len} map to four distinct banks for every layer_len in
{2..128}, so each bank needs only 1 read + 1 write/cycle (single buffer, no
ping-pong). Trades ~13k LUT for +4 BRAM (device BRAM at ~6%).
Pipeline gains one stage for the synchronous BRAM read; writeback address
delay deepened to 9. coeff_out is driven combinationally from the BRAM read
during OUTPUT; unused read ports mirror port 0 outside compute so the read
crossbar arbitration is not polluted by stale addresses.
Verified byte-exact vs NIST KAT: NTT standalone TB 19/19 (fwd+inv), Encaps
9/9, Decaps 9/9 (incl. reject + DECOMP), KeyGen, hello_world e2e — all PASS.
Cycle counts unchanged (ENC_N +4 for BRAM latency).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Issue 2 butterflies/cycle from the same NTT group (positions p,p+1).
ML-KEM's 7 layers have layer_len in {128..2} (always even, never 1),
so each pair shares one zeta and touches 4 distinct coefficient indices
— no intra-group RAW conflict, single zeta ROM. Compute per transform
~966->~490 cycles.
Verified: standalone ntt TB 19/19 (fwd+inv), full KAT Encaps 9/9 +
Decaps 9/9 + KeyGen + hello_world all PASS (all K).
Double-buffer poly_mul_sync so the next MAC term's 256-cycle load overlaps
the current term's accumulate. Decouple the top-level MAC load index (u_ldj)
from the accumulate index (u_j) in the shared ENC_U/ENC_V/DEC_W engine; load
reads bank_a/bank_se while accumulate reads bank_t (disjoint ports, no
conflict). Accumulate stays strictly serial (psum order-dependent); only the
load is hidden.
Verified byte-exact vs NIST KAT in ~/Dev/mlkem: Encaps 9/9, Decaps 9/9
(incl. implicit-reject + DECOMP), KeyGen all, hello_world end-to-end.
ENC_U 5585->5073, ENC_V 2793->2537 (K=2). Larger relative win at K=4
(K^2 terms). poly_mul contract unchanged (standalone TB 11/11).
Decouple the ENC_C1/C2 compress micro-sequence into a streaming
producer/collector instead of feeding one coeff and draining the whole
6-stage comp_decomp pipeline per coeff.
- comp_decomp_sync: ready_o = 1'b1 (fully pipelined, consumer never
backpressures since ready_i is tied high), so feeders can issue a new
coeff every cycle.
- mlkem_top ENC_C1/C2: producer feeds one coeff every 2 cycles
(<=5.5 bit/cyc, under the single-port ct drain of 8 bit/cyc so the bit
buffer stays bounded); a per-cycle collector appends compressed bits
and drains one ct byte/cycle.
Verified byte-exact against NIST KAT (Vivado 2019.2 XSIM) for KeyGen,
Encaps (9/9), Decaps (9/9 incl. reject path + DECOMP), hello_world e2e.
K=2 Encaps: ENC_C1 6273->1035, ENC_C2 2945->522, total 27851->20190.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
