diff --git a/.gitignore b/.gitignore
index bef8e79..f704a95 100644
--- a/.gitignore
+++ b/.gitignore
@@ -43,6 +43,7 @@ desktop.ini
.codex/
.opencode/
.agents/
+.omo/
# ========================================
# Editor / IDE
diff --git a/doc/SRAM_IS61WV102416ALL.pdf b/doc/SRAM_IS61WV102416ALL.pdf
new file mode 100644
index 0000000..63dc669
Binary files /dev/null and b/doc/SRAM_IS61WV102416ALL.pdf differ
diff --git a/doc/axi_bus_modules.md b/doc/axi_bus_modules.md
new file mode 100644
index 0000000..3dcae7f
--- /dev/null
+++ b/doc/axi_bus_modules.md
@@ -0,0 +1,570 @@
+# AXI 总线模块说明
+
+本文档基于 `full` 分支的 `rtl/soc_top.v`、各 AXI 外设源码和软件头文件整理，用于说明当前 SoC 中 AXI 总线上的模块作用、端口含义和寄存器地址。
+
+## 1. 总体连接关系
+
+顶层 AXI 互连为 `AxiCrossbar_2x8`。它有 2 个 AXI master 输入和 8 个 AXI slave 输出。
+
+### 1.1 AXI master 输入
+
+| Crossbar 端口 | 顶层连接 | 作用 |
+| --- | --- | --- |
+| `axiIn_0_*` | `core_top` 经 `Axi_CDC` 后的 `cpu_sync_*` | CPU 访问内存、外设和加速器 |
+| `axiIn_1_*` | `snix_axil_cdma_mux` 的 `dma_m_*` / `mm2mm_*` | DMA/CDMA 作为 AXI master 发起内存和外设数据搬运 |
+
+### 1.2 AXI slave 输出和地址窗口
+
+| Crossbar 端口 | 物理地址窗口 | 常用非缓存虚拟地址窗口 | 顶层连接 | 作用 |
+| --- | --- | --- | --- | --- |
+| `axiOut_0_*` | `0x1c000000` - `0x1c7fffff` | 软件通常直接使用物理地址 | `axi_wrap_ram_sp_external` | BaseRAM/ExtRAM 存储子系统 |
+| `axiOut_1_*` | `0x00000000` - `0x007fffff` | 无 | reserved tie-off | 保留窗口，当前返回空响应 |
+| `axiOut_2_*` | `0x1f000000` - `0x1f0fffff` | `0xbf000000` - `0xbf0fffff` | `axi_uart_controller` | UART0 |
+| `axiOut_3_*` | `0x1f100000` - `0x1f1fffff` | `0xbf100000` - `0xbf1fffff` | `axi_dvi` | 视频输出 |
+| `axiOut_4_*` | `0x1f200000` - `0x1f2fffff` | `0xbf200000` - `0xbf2fffff` | `confreg` | LED、数码管、按键、开关、定时器、中断 |
+| `axiOut_5_*` | `0x1f300000` - `0x1f3fffff` | `0xbf300000` - `0xbf3fffff` | `axi_axil_adapter` -> DMA/CDMA | DMA 配置窗口 |
+| `axiOut_6_*` | `0x1f400000` - `0x1f4fffff` | `0xbf400000` - `0xbf4fffff` | `fft_mlkem_axi_slave` | FFT、ML-KEM 共享窗口 |
+| `axiOut_7_*` | `0x1f500000` - `0x1f5fffff` | `0xbf500000` - `0xbf5fffff` | `matmul_axi_slave` | 矩阵乘法加速器 |
+
+说明：软件中的 `0xbfxxxxxx` 地址是 Loongson/LoongArch 非缓存映射地址；硬件 crossbar 实际译码使用 `0x1fxxxxxx` 等物理地址。
+
+## 2. AXI 信号字典
+
+多个模块使用完全相同的 AXI4 或 AXI-Lite 端口。后续各模块只列出端口前缀，例如 `ram_*`、`uart_*`、`s_axi_*`、`dma_m_*`。这些前缀下的每个信号含义按本节解释。
+
+### 2.1 AXI4 slave 接口信号
+
+适用于 `axi_wrap_ram_sp_external`、`axi_uart_controller`、`axi_dvi`、`confreg`、`fft_mlkem_axi_slave`、`mlkem_axi_slave`、`matmul_axi_slave` 等模块。
+
+| 信号 | 方向，以 slave 模块为准 | 含义 |
+| --- | --- | --- |
+| `clk` / `aclk` | input | AXI 总线时钟。当前大多数 slave 使用 `sys_clk`。 |
+| `resetn` / `aresetn` / `rst_n` | input | 低有效复位，已在顶层同步。 |
+| `s_axi_awid` / `axi_awid` | input | 写地址事务 ID，用于响应通道返回对应 ID。 |
+| `s_axi_awaddr` / `axi_awaddr` | input | 写地址。模块通常使用低位做内部寄存器偏移。 |
+| `s_axi_awlen` / `axi_awlen` | input | 突发长度，表示 beats-1。 |
+| `s_axi_awsize` / `axi_awsize` | input | 每个 beat 的字节数编码，`2` 表示 4 字节。 |
+| `s_axi_awburst` / `axi_awburst` | input | 突发类型，常用 `INCR`。 |
+| `s_axi_awlock` / `axi_awlock` | input | AXI lock 属性，本设计基本不使用。 |
+| `s_axi_awcache` / `axi_awcache` | input | cache 属性，本设计通常透传或忽略。 |
+| `s_axi_awprot` / `axi_awprot` | input | protection 属性，本设计通常透传或忽略。 |
+| `s_axi_awvalid` / `axi_awvalid` | input | 写地址有效。 |
+| `s_axi_awready` / `axi_awready` | output | slave 可接收写地址。 |
+| `s_axi_wid` | input | 写数据 ID。部分 AXI4 实现保留，很多模块不使用。 |
+| `s_axi_wdata` / `axi_wdata` | input | 写数据。 |
+| `s_axi_wstrb` / `axi_wstrb` | input | 写字节使能，bit0 对应低 8 位。 |
+| `s_axi_wlast` / `axi_wlast` | input | 写突发最后一拍。 |
+| `s_axi_wvalid` / `axi_wvalid` | input | 写数据有效。 |
+| `s_axi_wready` / `axi_wready` | output | slave 可接收写数据。 |
+| `s_axi_bid` / `axi_bid` | output | 写响应 ID。 |
+| `s_axi_bresp` / `axi_bresp` | output | 写响应状态，当前外设一般返回 `OKAY=2'b00`。 |
+| `s_axi_bvalid` / `axi_bvalid` | output | 写响应有效。 |
+| `s_axi_bready` / `axi_bready` | input | master 可接收写响应。 |
+| `s_axi_arid` / `axi_arid` | input | 读地址事务 ID。 |
+| `s_axi_araddr` / `axi_araddr` | input | 读地址。 |
+| `s_axi_arlen` / `axi_arlen` | input | 读突发长度，表示 beats-1。 |
+| `s_axi_arsize` / `axi_arsize` | input | 每个读 beat 的字节数编码。 |
+| `s_axi_arburst` / `axi_arburst` | input | 读突发类型，常用 `INCR`。 |
+| `s_axi_arlock` / `axi_arlock` | input | AXI lock 属性，本设计基本不使用。 |
+| `s_axi_arcache` / `axi_arcache` | input | cache 属性，本设计通常透传或忽略。 |
+| `s_axi_arprot` / `axi_arprot` | input | protection 属性，本设计通常透传或忽略。 |
+| `s_axi_arvalid` / `axi_arvalid` | input | 读地址有效。 |
+| `s_axi_arready` / `axi_arready` | output | slave 可接收读地址。 |
+| `s_axi_rid` / `axi_rid` | output | 读响应 ID。 |
+| `s_axi_rdata` / `axi_rdata` | output | 读数据。 |
+| `s_axi_rresp` / `axi_rresp` | output | 读响应状态，当前外设一般返回 `OKAY=2'b00`。 |
+| `s_axi_rlast` / `axi_rlast` | output | 读突发最后一拍。 |
+| `s_axi_rvalid` / `axi_rvalid` | output | 读数据有效。 |
+| `s_axi_rready` / `axi_rready` | input | master 可接收读数据。 |
+
+### 2.2 AXI4 master 接口信号
+
+适用于 `snix_axil_cdma_mux` 的 `mm2mm_*` / 顶层 `dma_m_*`。信号名称与 AXI4 slave 一致，但方向相反：`awaddr/araddr/wdata/wstrb/valid/ready` 等由 DMA master 发起，crossbar/slave 返回 ready、response 和 read data。
+
+| 信号组 | 方向，以 DMA 为准 | 含义 |
+| --- | --- | --- |
+| `mm2mm_aw*` | output/input | DMA 写地址通道，用于向目标地址发起写事务。 |
+| `mm2mm_w*` | output/input | DMA 写数据通道。 |
+| `mm2mm_b*` | input/output | DMA 写响应通道。 |
+| `mm2mm_ar*` | output/input | DMA 读地址通道，用于从源地址发起读事务。 |
+| `mm2mm_r*` | input/output | DMA 读数据通道。 |
+| `mm2mm_awqos/arqos` | output | AXI QoS，顶层未使用。 |
+| `mm2mm_awuser/wuser/aruser` | output | AXI user sideband，顶层未使用。 |
+| `mm2mm_buser/ruser` | input | AXI user sideband，顶层绑常量。 |
+
+### 2.3 AXI-Lite slave 接口信号
+
+适用于 `snix_axil_cdma_mux` 和 `cdma_matmul_bridge` 的配置接口。
+
+| 信号 | 方向，以 AXI-Lite slave 为准 | 含义 |
+| --- | --- | --- |
+| `s_axil_awaddr` | input | AXI-Lite 写地址。 |
+| `s_axil_awvalid` | input | 写地址有效。 |
+| `s_axil_awready` | output | slave 可接收写地址。 |
+| `s_axil_wdata` | input | 写数据。 |
+| `s_axil_wstrb` | input | 写字节使能。 |
+| `s_axil_wvalid` | input | 写数据有效。 |
+| `s_axil_wready` | output | slave 可接收写数据。 |
+| `s_axil_bresp` | output | 写响应状态。 |
+| `s_axil_bvalid` | output | 写响应有效。 |
+| `s_axil_bready` | input | master 可接收写响应。 |
+| `s_axil_araddr` | input | AXI-Lite 读地址。 |
+| `s_axil_arvalid` | input | 读地址有效。 |
+| `s_axil_arready` | output | slave 可接收读地址。 |
+| `s_axil_rdata` | output | 读数据。 |
+| `s_axil_rresp` | output | 读响应状态。 |
+| `s_axil_rvalid` | output | 读数据有效。 |
+| `s_axil_rready` | input | master 可接收读数据。 |
+
+## 3. 各 AXI 模块作用与端口
+
+### 3.1 `AxiCrossbar_2x8`
+
+作用：系统 AXI 主互连。它将 CPU 和 DMA 两个 master 的读写请求按地址译码转发到 8 个 slave，并负责读写响应返回。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | crossbar 工作时钟，顶层连接 `sys_clk`。 |
+| `resetn` | input | 低有效复位，顶层连接 `sys_resetn`。 |
+| `axiIn_0_*` | input/output | CPU 侧 AXI master 接口，信号含义见 AXI4 字典。 |
+| `axiIn_1_*` | input/output | DMA/CDMA 侧 AXI master 接口，信号含义见 AXI4 字典。 |
+| `axiOut_0_*` | output/input | RAM AXI slave 接口。 |
+| `axiOut_1_*` | output/input | 保留 AXI slave 接口，顶层 tie-off。 |
+| `axiOut_2_*` | output/input | UART AXI slave 接口。 |
+| `axiOut_3_*` | output/input | DVI AXI slave 接口。 |
+| `axiOut_4_*` | output/input | confreg AXI slave 接口。 |
+| `axiOut_5_*` | output/input | DMA 配置 AXI slave 接口。 |
+| `axiOut_6_*` | output/input | FFT/ML-KEM AXI slave 接口。 |
+| `axiOut_7_*` | output/input | matmul AXI slave 接口。 |
+
+### 3.2 `core_top` 与 `Axi_CDC`
+
+作用：`core_top` 是 CPU；`Axi_CDC` 将 CPU 时钟域的 AXI 请求同步到 `sys_clk` 域，再送入 crossbar master 0。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `core_top.aclk` | input | CPU 时钟 `cpu_clk`。 |
+| `core_top.aresetn` | input | CPU 低有效复位 `cpu_resetn`。 |
+| `core_top.intrpt[7:0]` | input | CPU 中断输入。当前 `{4'd0, mlkem_intr, matmul_intr, dma_finish, confreg_int}`。 |
+| `core_top.ar*/r*/aw*/w*/b*` | input/output | CPU AXI master 接口。 |
+| `core_top.debug0_*` | output | CPU 写回级调试信号。 |
+| `Axi_CDC.axiInClk/axiInRstn` | input | CPU 侧时钟/复位。 |
+| `Axi_CDC.axiOutClk/axiOutRstn` | input | crossbar 侧时钟/复位。 |
+| `Axi_CDC.axiIn_*` | input/output | CPU 侧 AXI 信号。 |
+| `Axi_CDC.axiOut_*` | output/input | `sys_clk` 域 AXI 信号，连接 crossbar master 0。 |
+
+### 3.3 `axi_wrap_ram_sp_external`
+
+作用：SRAM 子系统的 AXI slave 包装器。它把 AXI 读写转换为异步 SRAM 控制信号，并根据内部地址 bit22 选择 BaseRAM 或 ExtRAM。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `aclk` | input | AXI/SRAM 控制器时钟，连接 `sys_clk`。 |
+| `aresetn` | input | 低有效复位。 |
+| `axi_ar*/axi_r*/axi_aw*/axi_w*/axi_b*` | input/output | RAM 的 AXI4 slave 接口，连接 crossbar `axiOut_0_*`。 |
+| `base_ram_data[31:0]` | inout | BaseRAM 双向数据总线。写周期由控制器驱动，读周期为高阻输入。 |
+| `base_ram_addr[19:0]` | output | BaseRAM 字地址，来自 `soc_sram_addr[21:2]`。 |
+| `base_ram_be_n[3:0]` | output | BaseRAM 字节使能，低有效。 |
+| `base_ram_ce_n` | output | BaseRAM 片选，低有效。 |
+| `base_ram_oe_n` | output | BaseRAM 读使能，低有效。 |
+| `base_ram_we_n` | output | BaseRAM 写使能，低有效。 |
+| `ext_ram_data[31:0]` | inout | ExtRAM 双向数据总线。 |
+| `ext_ram_addr[19:0]` | output | ExtRAM 字地址，来自 `soc_sram_addr[21:2]`。 |
+| `ext_ram_be_n[3:0]` | output | ExtRAM 字节使能，低有效。 |
+| `ext_ram_ce_n` | output | ExtRAM 片选，低有效。 |
+| `ext_ram_oe_n` | output | ExtRAM 读使能，低有效。 |
+| `ext_ram_we_n` | output | ExtRAM 写使能，低有效。 |
+
+地址说明：`0x1c000000` - `0x1c3fffff` 选择 BaseRAM，`0x1c400000` - `0x1c7fffff` 选择 ExtRAM。
+
+### 3.4 reserved window
+
+作用：`axiOut_1_*` 是保留窗口，顶层没有接实际模块。读数据返回 0，写请求直接 ready/空响应。该窗口不建议软件使用。
+
+### 3.5 `axi_uart_controller`
+
+作用：AXI 到 APB/UART 控制器。CPU 通过 AXI 访问 16550 风格 UART 寄存器，实现串口收发和中断。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | UART AXI/APB 时钟，连接 `sys_clk`。 |
+| `rst_n` | input | 低有效复位。 |
+| `axi_s_*` | input/output | UART 的 AXI4 slave 接口，连接 crossbar `axiOut_2_*`。 |
+| `apb_rw_dma` | input | DMA APB 访问方向，顶层绑 0，当前未使用。 |
+| `apb_psel_dma` | input | DMA APB 片选，顶层绑 0。 |
+| `apb_enab_dma` | input | DMA APB enable，顶层绑 0。 |
+| `apb_addr_dma[19:0]` | input | DMA APB 地址，顶层绑 0。 |
+| `apb_valid_dma` | input | DMA APB 请求有效，顶层绑 0。 |
+| `apb_wdata_dma[31:0]` | input | DMA APB 写数据，顶层绑 0。 |
+| `apb_rdata_dma[31:0]` | output | DMA APB 读数据，顶层未使用。 |
+| `apb_ready_dma` | output | DMA APB ready，顶层未使用。 |
+| `dma_grant` | output | APB mux 对 DMA 的授权，顶层未使用。 |
+| `dma_req_o` | output | UART 控制器内部 DMA 请求，顶层未使用。 |
+| `dma_ack_i` | input | DMA ack，顶层绑 0。 |
+| `uart0_txd_i` | input | UART TX 管脚回读输入。 |
+| `uart0_txd_o` | output | UART TX 输出值。 |
+| `uart0_txd_oe` | output | UART TX 输出使能；顶层用它控制三态。 |
+| `uart0_rxd_i` | input | UART RX 输入值。 |
+| `uart0_rxd_o` | output | UART RX 输出值，三态模式下使用。 |
+| `uart0_rxd_oe` | output | UART RX 输出使能。 |
+| `uart0_rts_o` | output | UART RTS 调制解调控制输出。 |
+| `uart0_dtr_o` | output | UART DTR 调制解调控制输出。 |
+| `uart0_cts_i` | input | UART CTS 输入。 |
+| `uart0_dsr_i` | input | UART DSR 输入。 |
+| `uart0_dcd_i` | input | UART DCD 输入。 |
+| `uart0_ri_i` | input | UART RI 输入。 |
+| `uart0_int` | output | UART0 中断，送入 confreg 中断控制。 |
+
+### 3.6 `axi_dvi`
+
+作用：AXI 可配置的视频输出控制器。软件写入矩形/方形参数，模块产生 VGA/DVI 时序和 RGB 输出。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `aclk` | input | DVI 控制器和像素时钟来源。 |
+| `aresetn` | input | 低有效复位。 |
+| `s_aw*/s_w*/s_b*/s_ar*/s_r*` | input/output | DVI 的 AXI4 slave 接口，连接 crossbar `axiOut_3_*`。 |
+| `video_clk` | output | 视频像素时钟输出。 |
+| `hsync` | output | 行同步信号。 |
+| `vsync` | output | 场同步信号。 |
+| `data_enable` | output | 有效显示区域指示。 |
+| `video_red[2:0]` | output | 红色像素分量。 |
+| `video_green[2:0]` | output | 绿色像素分量。 |
+| `video_blue[1:0]` | output | 蓝色像素分量。 |
+
+### 3.7 `confreg`
+
+作用：基础外设和中断控制寄存器模块。它负责 LED、数码管、拨码开关、触摸按键、系统定时器、中断状态聚合和仿真标志。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `aclk` | input | AXI 外设时钟，连接 `sys_clk`。 |
+| `aresetn` | input | 低有效 AXI 复位。 |
+| `cpu_clk` | input | CPU 时钟，当前传入但主要逻辑在 `aclk` 域。 |
+| `cpu_resetn` | input | CPU 低有效复位。 |
+| `s_aw*/s_w*/s_b*/s_ar*/s_r*` | input/output | confreg 的 AXI4 slave 接口，连接 crossbar `axiOut_4_*`。 |
+| `led[15:0]` | output | 16 位 LED 输出。 |
+| `dpy0[7:0]` | output | 低位数码管输出。 |
+| `dpy1[7:0]` | output | 高位数码管输出。 |
+| `switch[31:0]` | input | 32 位拨码开关输入。 |
+| `touch_btn[3:0]` | input | 4 位触摸按键输入。 |
+| `uart0_int` | input | UART 中断输入，映射到 confreg 中断源 bit5。 |
+| `fft_finish` | input | FFT 完成输入，映射到 confreg 中断源 bit6。 |
+| `confreg_int` | output | confreg 聚合后的中断输出，送入 CPU `intrpt[0]`。 |
+
+### 3.8 `axi_axil_adapter`
+
+作用：把 crossbar `axiOut_5_*` 的 AXI4 slave 访问转换为 AXI-Lite 访问，供 DMA/CDMA 配置寄存器使用。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | adapter 时钟，连接 `sys_clk`。 |
+| `rst` | input | 高有效复位，顶层连接 `~sys_resetn`。 |
+| `s_axi_*` | input/output | AXI4 slave 侧，连接 crossbar `axiOut_5_*`。 |
+| `m_axil_*` | output/input | AXI-Lite master 侧，连接 DMA 配置分流逻辑。 |
+
+### 3.9 `snix_axil_cdma_mux`
+
+作用：多通道 CDMA 控制器。它有 AXI-Lite 配置接口、AXI4 `mm2mm` master 数据搬运接口，以及本地 endpoint 接口。当前顶层参数 `PORTS=4`，即 4 个软件可见 DMA 通道。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | CDMA 时钟，连接 `sys_clk`。 |
+| `rst_n` | input | 低有效复位。 |
+| `s_axil_*` | input/output | AXI-Lite 配置 slave 接口，来自 `axi_axil_adapter` 且未命中 bridge 子窗口时进入本模块。 |
+| `mm2mm_aw* / mm2mm_w* / mm2mm_b* / mm2mm_ar* / mm2mm_r*` | output/input | AXI4 master 数据搬运接口，连接 crossbar `axiIn_1_*`。 |
+| `ep_wr_valid` | output | 向本地 endpoint 写数据请求有效。当前连接 matmul。 |
+| `ep_wr_id[3:0]` | output | endpoint 编号。当前 matmul 使用 id 0。 |
+| `ep_wr_bank[3:0]` | output | endpoint bank 选择。当前 matmul 使用 bank0/bank1。 |
+| `ep_wr_word_addr[7:0]` | output | endpoint 内部 word 地址。 |
+| `ep_wr_data[31:0]` | output | 写入 endpoint 的 32 位数据。 |
+| `ep_wr_ready` | input | endpoint 可接收写数据。 |
+| `ep_rd_valid` | output | 从 endpoint 读数据请求有效。 |
+| `ep_rd_id[3:0]` | output | endpoint 编号。 |
+| `ep_rd_bank[3:0]` | output | endpoint bank 选择。 |
+| `ep_rd_word_addr[7:0]` | output | endpoint 内部 word 地址。 |
+| `ep_rd_data[31:0]` | input | endpoint 返回的 32 位数据。 |
+| `ep_rd_ready` | input | endpoint 读数据有效/可被采样。 |
+| `ext_rd_req_valid` | input | 外部序列器请求“内存到 endpoint”搬运。当前来自 `cdma_matmul_bridge`。 |
+| `ext_rd_req_ready` | output | CDMA 可接收外部读请求。 |
+| `ext_rd_req_src[31:0]` | input | 外部读请求的内存源地址。 |
+| `ext_rd_req_bytes[31:0]` | input | 外部读请求字节数。 |
+| `ext_rd_req_ep_id[3:0]` | input | 外部读请求目标 endpoint id。 |
+| `ext_rd_req_ep_bank[3:0]` | input | 外部读请求目标 endpoint bank。 |
+| `ext_rd_req_ep_word_off[7:0]` | input | 外部读请求目标 endpoint word 偏移。 |
+| `ext_rd_req_done` | output | 外部读请求完成脉冲/标志。 |
+| `ext_wr_req_valid` | input | 外部序列器请求“endpoint 到内存”搬运。 |
+| `ext_wr_req_ready` | output | CDMA 可接收外部写请求。 |
+| `ext_wr_req_dst[31:0]` | input | 外部写请求的内存目的地址。 |
+| `ext_wr_req_bytes[31:0]` | input | 外部写请求字节数。 |
+| `ext_wr_req_ep_id[3:0]` | input | 外部写请求源 endpoint id。 |
+| `ext_wr_req_ep_bank[3:0]` | input | 外部写请求源 endpoint bank。 |
+| `ext_wr_req_ep_word_off[7:0]` | input | 外部写请求源 endpoint word 偏移。 |
+| `ext_wr_req_done` | output | 外部写请求完成脉冲/标志。 |
+| `dma_finish` | output | DMA 完成中断/状态输出，送入顶层 `dma_finish` 合并逻辑。 |
+
+### 3.10 `cdma_matmul_bridge`
+
+作用：matmul 专用 CDMA 序列控制器。CPU 配置输入基地址、输出基地址、组数和 stride 后，它自动向 `snix_axil_cdma_mux` 发起 endpoint 读写请求，并直接控制 matmul 启动、清状态、清 CRC。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | bridge 时钟，连接 `sys_clk`。 |
+| `rst_n` | input | 低有效复位。 |
+| `s_axil_*` | input/output | AXI-Lite 配置 slave 接口。当 DMA 配置地址 `[11:8] == 4'h2` 时访问本模块。 |
+| `cdma_rd_req_valid` | output | 请求 CDMA 从内存读取数据到 matmul endpoint。 |
+| `cdma_rd_req_ready` | input | CDMA 可接收读请求。 |
+| `cdma_rd_req_src[31:0]` | output | 读请求内存源地址。 |
+| `cdma_rd_req_bytes[31:0]` | output | 读请求字节数，matmul 输入固定为 128B。 |
+| `cdma_rd_req_ep_id[3:0]` | output | 读请求目标 endpoint id。 |
+| `cdma_rd_req_ep_bank[3:0]` | output | 读请求目标 matmul bank。 |
+| `cdma_rd_req_ep_word_off[7:0]` | output | 读请求目标 endpoint word 偏移。 |
+| `cdma_rd_req_done` | input | CDMA 读请求完成。 |
+| `cdma_wr_req_valid` | output | 请求 CDMA 从 matmul endpoint 写回内存。 |
+| `cdma_wr_req_ready` | input | CDMA 可接收写请求。 |
+| `cdma_wr_req_dst[31:0]` | output | 写请求内存目的地址。 |
+| `cdma_wr_req_bytes[31:0]` | output | 写请求字节数，matmul 输出固定为 192B。 |
+| `cdma_wr_req_ep_id[3:0]` | output | 写请求源 endpoint id。 |
+| `cdma_wr_req_ep_bank[3:0]` | output | 写请求源 matmul bank。 |
+| `cdma_wr_req_ep_word_off[7:0]` | output | 写请求源 endpoint word 偏移。 |
+| `cdma_wr_req_done` | input | CDMA 写请求完成。 |
+| `mat_start_valid` | output | 启动 matmul 计算的有效脉冲。 |
+| `mat_start_bank` | output | 启动哪个 matmul bank。 |
+| `mat_done_clear` | output | 清除 matmul done/error 状态。 |
+| `mat_crc_clear` | output | 清除 matmul CRC 累积值。 |
+| `mat_done` | input | matmul 计算完成状态。 |
+| `seq_done_irq` | output | 整个 matmul 序列完成中断，顶层与 `cdma_dma_finish` OR 成 `dma_finish`。 |
+
+### 3.11 `fft_mlkem_axi_slave`
+
+作用：共享 AXI slave 地址窗口，按地址低 16 位的 `[15:12]` 将请求分发到 ML-KEM 或 FFT。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | AXI 时钟，连接 `sys_clk`。 |
+| `resetn` | input | 低有效复位。 |
+| `s_axi_*` | input/output | 共享加速器 AXI4 slave 接口，连接 crossbar `axiOut_6_*`。 |
+| `fft_finish` | output | FFT 完成状态，送入 confreg。 |
+| `mlkem_intr` | output | ML-KEM 完成中断，直接送入 CPU `intrpt[3]`。 |
+
+内部地址分发：
+
+| 偏移范围 | 子模块 |
+| --- | --- |
+| `0x0000` - `0x3fff` | `mlkem_axi_slave` |
+| `0x8000` - `0xcfff` | `fft_controller` / FFT |
+
+### 3.12 `mlkem_axi_slave`
+
+作用：ML-KEM 加速器 AXI4 slave 包装器，提供 KeyGen、Encaps、Decaps 的配置、输入预加载、输出读回和中断。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | AXI 总线时钟。内部再产生 `mlkem_clk = clk/2`。 |
+| `resetn` | input | 低有效复位。 |
+| `s_axi_*` | input/output | ML-KEM AXI4 slave 接口，由 `fft_mlkem_axi_slave` 分发而来。 |
+| `intr` | output | ML-KEM done 锁存后的高电平中断。 |
+
+### 3.13 `fft_controller`
+
+作用：1024 点 16-bit FFT 加速器。通过 AXI 写入输入实部/虚部 BRAM，写 CSR 启动计算，完成后从输出实部/虚部窗口读取结果。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `aclk` | input | FFT/AXI 时钟。 |
+| `aresetn` | input | 低有效复位。 |
+| `s_aw*/s_w*/s_b*/s_ar*/s_r*` | input/output | FFT AXI4 slave 接口，由 `fft_mlkem_axi_slave` 分发而来。 |
+| `fft_finish` | output | FFT done 状态，来自 CSR status bit1。 |
+
+### 3.14 `matmul_axi_slave`
+
+作用：矩阵乘法加速器 AXI slave。它既支持 CPU 通过 AXI 访问控制寄存器和 bank 数据，也支持 CDMA 通过本地 endpoint 写入输入、读出结果。模块内部带 CRC32 累积逻辑。
+
+端口说明：
+
+| 端口 | 方向 | 解释 |
+| --- | --- | --- |
+| `clk` | input | matmul/AXI 时钟，连接 `sys_clk`。 |
+| `resetn` | input | 低有效复位。 |
+| `s_axi_*` | input/output | matmul AXI4 slave 接口，连接 crossbar `axiOut_7_*`。 |
+| `ep_wr_valid` | input | CDMA 本地 endpoint 写请求有效。 |
+| `ep_wr_id[3:0]` | input | endpoint id。当前仅使用 id 0。 |
+| `ep_wr_bank[3:0]` | input | 写入 bank 选择，bit0 选择 bank0/bank1。 |
+| `ep_wr_word_addr[7:0]` | input | 写入 bank 内 word 地址，0-15 为 A，16-31 为 B。 |
+| `ep_wr_data[31:0]` | input | 写入 A/B 矩阵的 32 位数据。 |
+| `ep_wr_ready` | output | matmul endpoint 可接收写数据。当前恒为 1。 |
+| `ep_rd_valid` | input | CDMA 本地 endpoint 读请求有效。 |
+| `ep_rd_id[3:0]` | input | endpoint id。 |
+| `ep_rd_bank[3:0]` | input | 读取 bank 选择。 |
+| `ep_rd_word_addr[7:0]` | input | 读取结果 C 的 word 地址。每个 66-bit 结果拆成 3 个 32-bit word。 |
+| `ep_rd_data[31:0]` | output | endpoint 返回的 32 位结果数据。 |
+| `ep_rd_ready` | output | endpoint 读数据可用。当前恒为 1。 |
+| `mat_start_valid` | input | 来自 bridge 的启动脉冲。 |
+| `mat_start_bank` | input | 启动计算的 bank。 |
+| `mat_done_clear` | input | 清除 done/error。 |
+| `mat_crc_clear` | input | 清除 CRC32 累积值。 |
+| `mat_done` | output | 计算完成状态。 |
+| `intr` | output | matmul 完成中断，送入 CPU `intrpt[2]`。 |
+
+## 4. 寄存器地址表
+
+以下地址以软件常用非缓存虚拟地址列出；括号中给出对应物理地址基址。
+
+### 4.1 UART0 (`0xbf000000`, physical `0x1f000000`)
+
+16550 风格 8-bit 寄存器，软件常用字节访问。
+
+| 地址 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf000000` | `0x0` | `RBR/THR` 或 `DLL` | R/W | DLAB=0 时读接收缓冲/写发送保持；DLAB=1 时为 divisor low byte。 |
+| `0xbf000001` | `0x1` | `IER` 或 `DLM` | R/W | DLAB=0 时为中断使能；DLAB=1 时为 divisor high byte。 |
+| `0xbf000002` | `0x2` | `IIR/FCR` | R/W | 读为中断识别，写为 FIFO 控制；DLAB=1 时部分实现作为 divisor 扩展。 |
+| `0xbf000003` | `0x3` | `LCR` | R/W | 线控制寄存器，bit7 为 DLAB。 |
+| `0xbf000004` | `0x4` | `MCR` | R/W | modem 控制。 |
+| `0xbf000005` | `0x5` | `LSR` | R | 线状态，软件轮询 bit5/bit6 判断发送 FIFO/发送器空。 |
+| `0xbf000006` | `0x6` | `MSR` | R | modem 状态。 |
+| `0xbf000007` | `0x7` | `SCR` | R/W | scratch/mode 寄存器。 |
+
+### 4.2 DVI (`0xbf100000`, physical `0x1f100000`)
+
+| 地址 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf100000` | `0x0` | `DVI_RECT_DIR` | R/W | 矩形位置参数，`[31:16]` 为 x，`[15:0]` 为 y。 |
+| `0xbf100004` | `0x4` | `DVI_RECT_L_W` | R/W | 矩形尺寸参数，`[31:16]` 为长度/半宽相关参数，`[15:0]` 为宽度/高度相关参数。 |
+| `0xbf100008` | `0x8` | `DVI_SQU_DIR` | R/W | 方形位置参数。 |
+| `0xbf10000c` | `0xc` | `DVI_SQU_R` | R/W | 方形半径/尺寸参数。 |
+
+### 4.3 confreg (`0xbf200000`, physical `0x1f200000`)
+
+| 地址 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf20f000` | `0xf000` | `CONFREG_INT_EN` | R/W | 中断使能。 |
+| `0xbf20f004` | `0xf004` | `CONFREG_INT_EDGE` | R/W | 中断边沿/电平模式配置。 |
+| `0xbf20f008` | `0xf008` | `CONFREG_INT_POL` | R/W | 中断极性配置。 |
+| `0xbf20f00c` | `0xf00c` | `CONFREG_INT_CLR` | W/R | 写 1 清除对应中断状态。 |
+| `0xbf20f010` | `0xf010` | `CONFREG_INT_SET` | W/R | 写 1 软件置位对应中断状态。 |
+| `0xbf20f014` | `0xf014` | `CONFREG_INT_STATE` | R | 当前中断状态。bit0-3 touch_btn，bit4 timer，bit5 UART，bit6 FFT。 |
+| `0xbf20f100` | `0xf100` | `TIMER_VALUE` | R | 系统计数器当前值。 |
+| `0xbf20f104` | `0xf104` | `TIMER_CMP` | R/W | 定时器比较值。 |
+| `0xbf20f108` | `0xf108` | `TIMER_EN` | R/W | bit0 使能定时器。 |
+| `0xbf20f200` | `0xf200` | `DIGITAL_CTRL` | R/W | 数码管控制寄存器。 |
+| `0xbf20f204` | `0xf204` | `DIGITAL_DATA` | R/W | 数码管显示数据。 |
+| `0xbf20f300` | `0xf300` | `LED_DATA` | R/W | LED 输出寄存器，低 16 位连接 `leds[15:0]`。 |
+| `0xbf20f400` | `0xf400` | `SWITCH_DATA` | R | 32 位拨码开关输入。 |
+| `0xbf20f500` | `0xf500` | `SIMU_FLAG` | R | 仿真标志，值为 `{32{SIMULATION}}`。 |
+
+### 4.4 DMA/CDMA (`0xbf300000`, physical `0x1f300000`)
+
+当前 `snix_axil_cdma_mux` 在顶层例化为 `PORTS=4`，通道 `ch=0..3`，每个通道 stride 为 `0x40`。通道寄存器地址为 `0xbf300000 + ch * 0x40 + offset`。
+
+| 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- |
+| `0x00` | `SRC_ADDR` | R/W | 源地址。mem2mem/mem2ep 时为内存源地址。 |
+| `0x04` | `DST_ADDR` | R/W | 目的地址。mem2mem/ep2mem 时为内存目的地址。 |
+| `0x08` | `LENGTH` | R/W | 传输字节数。 |
+| `0x0c` | `TAG` | R/W | 软件 tag，硬件不参与数据路径。 |
+| `0x10` | `CTRL` | R/W | bit0 写 1 启动；`[5:3]` 为 AXI AxSIZE；`[13:6]` 为 AXI AxLEN。 |
+| `0x14` | `STATUS` | R/W | bit0 busy，只读；bit1 done，写 1 清除。 |
+| `0x18` | `MODE` | R/W | `0=mem2mem`，`1=mem2ep`，`2=ep2mem`。 |
+| `0x1c` | `EP_CFG` | R/W | `[3:0]` bank，`[15:8]` word offset，`[19:16]` endpoint id。 |
+
+通道基址：
+
+| 通道 | 基址 |
+| --- | --- |
+| channel 0 | `0xbf300000` |
+| channel 1 | `0xbf300040` |
+| channel 2 | `0xbf300080` |
+| channel 3 | `0xbf3000c0` |
+
+matmul 序列控制子窗口由 `cdma_matmul_bridge` 实现，地址为 `0xbf300200 + offset`：
+
+| 地址 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf300200` | `0x200` | `SEQ_CTRL` | R/W | bit0 start；bit1 skip_result_writeback。 |
+| `0xbf300204` | `0x204` | `SEQ_STATUS` | R/W | bit0 busy；bit1 done，写 1 清除。 |
+| `0xbf300208` | `0x208` | `SEQ_SRC_BASE` | R/W | 输入矩阵组的物理源基址。 |
+| `0xbf30020c` | `0x20c` | `SEQ_DST_BASE` | R/W | 输出矩阵组的物理目的基址。 |
+| `0xbf300210` | `0x210` | `SEQ_GROUPS` | R/W | 自动处理的矩阵组数。 |
+| `0xbf300214` | `0x214` | `SEQ_MAT_BASE` | R/W | 保留兼容字段，软件通常写 `0x1f500000`。 |
+| `0xbf300218` | `0x218` | `SEQ_SRC_STRIDE` | R/W | 相邻输入组间隔，默认 128B。 |
+| `0xbf30021c` | `0x21c` | `SEQ_DST_STRIDE` | R/W | 相邻输出组间隔，默认 192B。 |
+
+### 4.5 FFT/ML-KEM 共享窗口 (`0xbf400000`, physical `0x1f400000`)
+
+#### 4.5.1 ML-KEM 子窗口
+
+| 地址/范围 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf400000` | `0x000` | `MLKEM_CTRL` | W | bit0 start，启动 CONFIG 指定的操作。 |
+| `0xbf400004` | `0x004` | `MLKEM_STATUS` | R/W | bit0 busy，bit1 done，bit2 preload_busy；写 bit1=1 清 done。 |
+| `0xbf400008` | `0x008` | `MLKEM_CONFIG` | R/W | `[2:0] k`，`[5:4] op`，op: 0 KeyGen，1 Encaps，2 Decaps。 |
+| `0xbf400010` - `0xbf40002c` | `0x010` - `0x02c` | `D[0..7]` | R/W | KeyGen seed d，8 个 32-bit word。 |
+| `0xbf400030` - `0xbf40004c` | `0x030` - `0x04c` | `Z[0..7]` | R/W | implicit-reject seed z。 |
+| `0xbf400050` - `0xbf40006c` | `0x050` - `0x06c` | `MSG[0..7]` | R/W | Encaps message m。 |
+| `0xbf400070` - `0xbf40008c` | `0x070` - `0x08c` | `SS[0..7]` | R | shared secret 输出。 |
+| `0xbf400100` | `0x100` | `PRELOAD_SEL` | R/W | 0=ek_in，1=dk_in，2=c_in。 |
+| `0xbf400104` | `0x104` | `PRELOAD_ADDR` | R/W | 预加载字节地址，写 `PRELOAD_DATA` 后自动递增。 |
+| `0xbf400108` | `0x108` | `PRELOAD_DATA` | W | 写低 8 位，流入所选输入端口。 |
+| `0xbf40010c` | `0x10c` | `RDBK_SEL` | R/W | 0=ek，1=dk_pke，2=full-dk，3=ciphertext。 |
+| `0xbf400110` | `0x110` | `RDBK_ADDR` | R/W | 读回字节索引。 |
+| `0xbf400114` | `0x114` | `RDBK_DATA` | R | 读回低 8 位数据。 |
+| `0xbf401000` - `0xbf401fff` | `0x1000` - `0x1fff` | `EK` | W/R | 写为 preload ek_in，读为 generated ek。 |
+| `0xbf402000` - `0xbf402fff` | `0x2000` - `0x2fff` | `DK` | W/R | 写为 preload dk_in，读为 generated full dk。 |
+| `0xbf403000` - `0xbf403fff` | `0x3000` - `0x3fff` | `CT` | W/R | 写为 preload c_in，读为 generated ciphertext。 |
+
+#### 4.5.2 FFT 子窗口
+
+| 地址/范围 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf408000` - `0xbf408ffc` | `0x8000` - `0x8ffc` | `FFT_IN_RE[0..1023]` | W | 输入实部 BRAM，每项低 16 位有效。 |
+| `0xbf409000` - `0xbf409ffc` | `0x9000` - `0x9ffc` | `FFT_IN_IM[0..1023]` | W | 输入虚部 BRAM，每项低 16 位有效。 |
+| `0xbf40a000` - `0xbf40affc` | `0xa000` - `0xaffc` | `FFT_OUT_RE[0..1023]` | R | 输出实部，符号扩展到 32 位，读出时使用 bit-reverse 地址。 |
+| `0xbf40b000` - `0xbf40bffc` | `0xb000` - `0xbffc` | `FFT_OUT_IM[0..1023]` | R | 输出虚部，符号扩展到 32 位，读出时使用 bit-reverse 地址。 |
+| `0xbf40c000` | `0xc000` | `FFT_CSR` | R/W | `[7:4]` 为 ctrl，写 bit4 启动/清 done；`[3:0]` 为 status，bit0 busy，bit1 done。 |
+
+### 4.6 Matmul (`0xbf500000`, physical `0x1f500000`)
+
+| 地址/范围 | 偏移 | 名称 | 访问 | 说明 |
+| --- | --- | --- | --- | --- |
+| `0xbf500000` | `0x000` | `MAT_CTRL` | R/W | bit0 启动 bank0，bit1 启动 bank1。 |
+| `0xbf500004` | `0x004` | `MAT_STATUS` | R/W | bit0 busy，bit1 done，bit2 error；写 bit1 清 done，写 bit2 清 error。 |
+| `0xbf500010` | `0x010` | `MAT_CRC` | R/W | 读返回 `crc_reg ^ 0xffffffff`；写该地址或 bridge 的 `mat_crc_clear` 清 CRC。 |
+| `0xbf500100` - `0xbf50013c` | `0x100` - `0x13c` | `B0_A[0..15]` | R/W | bank0 输入矩阵 A，16 个 32-bit word。 |
+| `0xbf500140` - `0xbf50017c` | `0x140` - `0x17c` | `B0_B[0..15]` | R/W | bank0 输入矩阵 B。 |
+| `0xbf500180` - `0xbf50023c` | `0x180` - `0x23c` | `B0_C` | R | bank0 输出矩阵 C，16 个 66-bit 结果，每个结果拆成 3 个 32-bit word，共 192B。 |
+| `0xbf500300` - `0xbf50033c` | `0x300` - `0x33c` | `B1_A[0..15]` | R/W | bank1 输入矩阵 A。 |
+| `0xbf500340` - `0xbf50037c` | `0x340` - `0x37c` | `B1_B[0..15]` | R/W | bank1 输入矩阵 B。 |
+| `0xbf500380` - `0xbf50043c` | `0x380` - `0x43c` | `B1_C` | R | bank1 输出矩阵 C，格式同 bank0。 |
+
+## 5. 关键数据路径说明
+
+1. CPU 访问任意外设或内存：`core_top -> Axi_CDC -> AxiCrossbar_2x8 -> 对应 AXI slave`。
+2. DMA 访问 SRAM：`snix_axil_cdma_mux mm2mm master -> AxiCrossbar_2x8 -> axi_wrap_ram_sp_external -> BaseRAM/ExtRAM`。DMA 和 SRAM 没有直接物理连线。
+3. DMA 访问 matmul 本地 endpoint：`snix_axil_cdma_mux -> ep_wr/ep_rd -> matmul_axi_slave`。这条是本地 endpoint 通路，不经过 AXI crossbar。
+4. matmul 批处理序列：CPU 写 `0xbf300200` 等寄存器配置 `cdma_matmul_bridge`，bridge 自动发起 CDMA 读/写请求并控制 matmul 启停。
+5. FFT/ML-KEM 共用 `0xbf400000` 窗口，由 `fft_mlkem_axi_slave` 依据偏移分发。
diff --git a/doc/mlkem_test_results.md b/doc/mlkem_test_results.md
new file mode 100644
index 0000000..99543a7
--- /dev/null
+++ b/doc/mlkem_test_results.md
@@ -0,0 +1,49 @@
+# ML-KEM Test Results Summary
+
+## Test Setup
+
+- Platform: LoongArch bare-metal target
+- Test mode: `STATIC_KAT_COUNT_0`
+- Levels: `k=2,3,4`
+- Repeat count: `100`
+- ML-KEM software allocator: `MLK_CONFIG_CUSTOM_ALLOC_FREE`
+- Verification sources: KAT count 0 expected values
+
+## Correctness
+
+All three paths passed in isolated testing:
+
+- `SW_ONLY_FAILS=0`
+- `CPU_ONLY_FAILS=0`
+- `DMA_ONLY_FAILS=0`
+
+Final result for the isolated diagnostic run:
+
+- `MLKEM512_RESULT=PASS`
+- `MLKEM768_RESULT=PASS`
+- `MLKEM1024_RESULT=PASS`
+- `MLKEM_RESULT=PASS`
+
+## Performance
+
+### Multi-level isolated diagnostic
+
+| Level | SW avg cycles | CPU avg cycles | DMA avg cycles | CPU vs SW | DMA vs SW | DMA vs CPU |
+|---|---:|---:|---:|---:|---:|---:|
+| k=2 / ML-KEM-512 | 19,684,481 | 432,358 | 189,105 | 45.5x | 104.1x | 2.29x |
+| k=3 / ML-KEM-768 | 30,688,338 | 638,556 | 283,023 | 48.1x | 108.4x | 2.26x |
+| k=4 / ML-KEM-1024 | 2,355,875 | 872,400 | 392,433 | 2.70x | 6.00x | 2.22x |
+
+### k=4 recheck
+
+| Level | SW avg cycles | CPU avg cycles | DMA avg cycles | CPU vs SW | DMA vs SW | DMA vs CPU |
+|---|---:|---:|---:|---:|---:|---:|
+| k=4 / ML-KEM-1024 | 6,025,486 | 718,672 | 388,959 | 8.39x | 15.49x | 1.85x |
+
+## Notes
+
+- CPU/DMA hardware paths were stable across 100 iterations for all levels.
+- The software path showed run-to-run variance earlier; the custom allocator improved stability, but the k=4 recheck should be used as the cleaner reference for reporting.
+- For the hardware accelerator claim, the most stable headline is:
+ - DMA path is about 2.2x faster than CPU register path.
+ - Hardware path is substantially faster than software, but the exact SW speedup depends on the software run used as baseline.
diff --git a/rtl/ip/PLL_2019_2/clk_pll.xci b/rtl/ip/PLL_2019_2/clk_pll.xci
index 27c681d..67b47a7 100644
--- a/rtl/ip/PLL_2019_2/clk_pll.xci
+++ b/rtl/ip/PLL_2019_2/clk_pll.xci
@@ -35,6 +35,12 @@
100000000
0
0.000
+
+
+
+ 100000000
+ 0
+ 0.000
1
LEVEL_HIGH
@@ -113,7 +119,7 @@
1
0000
0000
- 100.000
+ 25.00000
BUFG
50.0
false
@@ -128,15 +134,15 @@
0000
100.000
BUFG
- 50.000
+ 50.0
false
- 100.000
+ 25.00000
0.000
50.000
- 100.000
+ 25
0.000
1
- 0
+ 1
0000
0000
100.000
@@ -190,7 +196,7 @@
clk_in_sel
cpu_clk
sys_clk
- clk_out3
+ mlkem_clk
clk_out4
clk_out5
clk_out6
@@ -204,7 +210,7 @@
0000
1
0.66
- 0.33
+ 1.32
0.33
0.33
0.33
@@ -259,7 +265,7 @@
0.500
0.000
FALSE
- 1
+ 62
0.500
0.000
FALSE
@@ -289,12 +295,12 @@
FALSE
64.000
2.000
- 2
+ 3
Output Output Phase Duty Cycle Pk-to-Pk Phase
Clock Freq (MHz) (degrees) (%) Jitter (ps) Error (ps)
_cpu_clk__32.97872______0.000______50.0______130.093____123.600
_sys_clk__50.00000______0.000______50.0______122.035____123.600
- no_CLK_OUT3_output
+ mlkem_clk__25.00000______0.000______50.0______135.799____123.600
no_CLK_OUT4_output
no_CLK_OUT5_output
no_CLK_OUT6_output
@@ -432,14 +438,14 @@
1
true
BUFG
- 0.0
+ 135.799
false
- 0.0
+ 123.600
50.000
- 100.000
+ 25
0.000
1
- false
+ true
BUFG
0.0
false
@@ -484,7 +490,7 @@
false
sys_clk
false
- clk_out3
+ mlkem_clk
false
clk_out4
false
@@ -538,7 +544,7 @@
0.500
0.000
false
- 1
+ 62
0.500
0.000
false
@@ -566,7 +572,7 @@
0.010
0.010
false
- 2
+ 3
false
false
WAVEFORM
@@ -705,17 +711,23 @@
+
+
+
+
+
+
diff --git a/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.v b/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.v
new file mode 100644
index 0000000..8e2d1a2
--- /dev/null
+++ b/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.v
@@ -0,0 +1,235 @@
+// Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
+// --------------------------------------------------------------------------------
+// Tool Version: Vivado v.2019.2 (lin64) Build 2708876 Wed Nov 6 21:39:14 MST 2019
+// Date : Tue Jul 7 23:25:51 2026
+// Host : fedora running 64-bit unknown
+// Command : write_verilog -force -mode funcsim
+// /home/fallensigh/Dev/ciciec2026_loongson_regional/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.v
+// Design : clk_pll
+// Purpose : This verilog netlist is a functional simulation representation of the design and should not be modified
+// or synthesized. This netlist cannot be used for SDF annotated simulation.
+// Device : xc7a200tfbg676-1
+// --------------------------------------------------------------------------------
+`timescale 1 ps / 1 ps
+
+(* NotValidForBitStream *)
+module clk_pll
+ (cpu_clk,
+ sys_clk,
+ mlkem_clk,
+ resetn,
+ locked,
+ clk_in1);
+ output cpu_clk;
+ output sys_clk;
+ output mlkem_clk;
+ input resetn;
+ output locked;
+ input clk_in1;
+
+ (* IBUF_LOW_PWR *) wire clk_in1;
+ wire cpu_clk;
+ wire locked;
+ wire mlkem_clk;
+ wire resetn;
+ wire sys_clk;
+
+ clk_pll_clk_pll_clk_wiz inst
+ (.clk_in1(clk_in1),
+ .cpu_clk(cpu_clk),
+ .locked(locked),
+ .mlkem_clk(mlkem_clk),
+ .resetn(resetn),
+ .sys_clk(sys_clk));
+endmodule
+
+(* ORIG_REF_NAME = "clk_pll_clk_wiz" *)
+module clk_pll_clk_pll_clk_wiz
+ (cpu_clk,
+ sys_clk,
+ mlkem_clk,
+ resetn,
+ locked,
+ clk_in1);
+ output cpu_clk;
+ output sys_clk;
+ output mlkem_clk;
+ input resetn;
+ output locked;
+ input clk_in1;
+
+ wire clk_in1;
+ wire clk_in1_clk_pll;
+ wire clkfbout_buf_clk_pll;
+ wire clkfbout_clk_pll;
+ wire cpu_clk;
+ wire cpu_clk_clk_pll;
+ wire locked;
+ wire mlkem_clk;
+ wire mlkem_clk_clk_pll;
+ wire reset_high;
+ wire resetn;
+ wire sys_clk;
+ wire sys_clk_clk_pll;
+ wire NLW_plle2_adv_inst_CLKOUT3_UNCONNECTED;
+ wire NLW_plle2_adv_inst_CLKOUT4_UNCONNECTED;
+ wire NLW_plle2_adv_inst_CLKOUT5_UNCONNECTED;
+ wire NLW_plle2_adv_inst_DRDY_UNCONNECTED;
+ wire [15:0]NLW_plle2_adv_inst_DO_UNCONNECTED;
+
+ (* BOX_TYPE = "PRIMITIVE" *)
+ BUFG clkf_buf
+ (.I(clkfbout_clk_pll),
+ .O(clkfbout_buf_clk_pll));
+ (* BOX_TYPE = "PRIMITIVE" *)
+ (* CAPACITANCE = "DONT_CARE" *)
+ (* IBUF_DELAY_VALUE = "0" *)
+ (* IFD_DELAY_VALUE = "AUTO" *)
+ IBUF #(
+ .IOSTANDARD("DEFAULT"))
+ clkin1_ibufg
+ (.I(clk_in1),
+ .O(clk_in1_clk_pll));
+ (* BOX_TYPE = "PRIMITIVE" *)
+ BUFG clkout1_buf
+ (.I(cpu_clk_clk_pll),
+ .O(cpu_clk));
+ (* BOX_TYPE = "PRIMITIVE" *)
+ BUFG clkout2_buf
+ (.I(sys_clk_clk_pll),
+ .O(sys_clk));
+ (* BOX_TYPE = "PRIMITIVE" *)
+ BUFG clkout3_buf
+ (.I(mlkem_clk_clk_pll),
+ .O(mlkem_clk));
+ (* BOX_TYPE = "PRIMITIVE" *)
+ PLLE2_ADV #(
+ .BANDWIDTH("OPTIMIZED"),
+ .CLKFBOUT_MULT(31),
+ .CLKFBOUT_PHASE(0.000000),
+ .CLKIN1_PERIOD(20.000000),
+ .CLKIN2_PERIOD(0.000000),
+ .CLKOUT0_DIVIDE(47),
+ .CLKOUT0_DUTY_CYCLE(0.500000),
+ .CLKOUT0_PHASE(0.000000),
+ .CLKOUT1_DIVIDE(31),
+ .CLKOUT1_DUTY_CYCLE(0.500000),
+ .CLKOUT1_PHASE(0.000000),
+ .CLKOUT2_DIVIDE(78),
+ .CLKOUT2_DUTY_CYCLE(0.500000),
+ .CLKOUT2_PHASE(0.000000),
+ .CLKOUT3_DIVIDE(1),
+ .CLKOUT3_DUTY_CYCLE(0.500000),
+ .CLKOUT3_PHASE(0.000000),
+ .CLKOUT4_DIVIDE(1),
+ .CLKOUT4_DUTY_CYCLE(0.500000),
+ .CLKOUT4_PHASE(0.000000),
+ .CLKOUT5_DIVIDE(1),
+ .CLKOUT5_DUTY_CYCLE(0.500000),
+ .CLKOUT5_PHASE(0.000000),
+ .COMPENSATION("ZHOLD"),
+ .DIVCLK_DIVIDE(1),
+ .IS_CLKINSEL_INVERTED(1'b0),
+ .IS_PWRDWN_INVERTED(1'b0),
+ .IS_RST_INVERTED(1'b0),
+ .REF_JITTER1(0.010000),
+ .REF_JITTER2(0.010000),
+ .STARTUP_WAIT("FALSE"))
+ plle2_adv_inst
+ (.CLKFBIN(clkfbout_buf_clk_pll),
+ .CLKFBOUT(clkfbout_clk_pll),
+ .CLKIN1(clk_in1_clk_pll),
+ .CLKIN2(1'b0),
+ .CLKINSEL(1'b1),
+ .CLKOUT0(cpu_clk_clk_pll),
+ .CLKOUT1(sys_clk_clk_pll),
+ .CLKOUT2(mlkem_clk_clk_pll),
+ .CLKOUT3(NLW_plle2_adv_inst_CLKOUT3_UNCONNECTED),
+ .CLKOUT4(NLW_plle2_adv_inst_CLKOUT4_UNCONNECTED),
+ .CLKOUT5(NLW_plle2_adv_inst_CLKOUT5_UNCONNECTED),
+ .DADDR({1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0}),
+ .DCLK(1'b0),
+ .DEN(1'b0),
+ .DI({1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0,1'b0}),
+ .DO(NLW_plle2_adv_inst_DO_UNCONNECTED[15:0]),
+ .DRDY(NLW_plle2_adv_inst_DRDY_UNCONNECTED),
+ .DWE(1'b0),
+ .LOCKED(locked),
+ .PWRDWN(1'b0),
+ .RST(reset_high));
+ LUT1 #(
+ .INIT(2'h1))
+ plle2_adv_inst_i_1
+ (.I0(resetn),
+ .O(reset_high));
+endmodule
+`ifndef GLBL
+`define GLBL
+`timescale 1 ps / 1 ps
+
+module glbl ();
+
+ parameter ROC_WIDTH = 100000;
+ parameter TOC_WIDTH = 0;
+
+//-------- STARTUP Globals --------------
+ wire GSR;
+ wire GTS;
+ wire GWE;
+ wire PRLD;
+ tri1 p_up_tmp;
+ tri (weak1, strong0) PLL_LOCKG = p_up_tmp;
+
+ wire PROGB_GLBL;
+ wire CCLKO_GLBL;
+ wire FCSBO_GLBL;
+ wire [3:0] DO_GLBL;
+ wire [3:0] DI_GLBL;
+
+ reg GSR_int;
+ reg GTS_int;
+ reg PRLD_int;
+
+//-------- JTAG Globals --------------
+ wire JTAG_TDO_GLBL;
+ wire JTAG_TCK_GLBL;
+ wire JTAG_TDI_GLBL;
+ wire JTAG_TMS_GLBL;
+ wire JTAG_TRST_GLBL;
+
+ reg JTAG_CAPTURE_GLBL;
+ reg JTAG_RESET_GLBL;
+ reg JTAG_SHIFT_GLBL;
+ reg JTAG_UPDATE_GLBL;
+ reg JTAG_RUNTEST_GLBL;
+
+ reg JTAG_SEL1_GLBL = 0;
+ reg JTAG_SEL2_GLBL = 0 ;
+ reg JTAG_SEL3_GLBL = 0;
+ reg JTAG_SEL4_GLBL = 0;
+
+ reg JTAG_USER_TDO1_GLBL = 1'bz;
+ reg JTAG_USER_TDO2_GLBL = 1'bz;
+ reg JTAG_USER_TDO3_GLBL = 1'bz;
+ reg JTAG_USER_TDO4_GLBL = 1'bz;
+
+ assign (strong1, weak0) GSR = GSR_int;
+ assign (strong1, weak0) GTS = GTS_int;
+ assign (weak1, weak0) PRLD = PRLD_int;
+
+ initial begin
+ GSR_int = 1'b1;
+ PRLD_int = 1'b1;
+ #(ROC_WIDTH)
+ GSR_int = 1'b0;
+ PRLD_int = 1'b0;
+ end
+
+ initial begin
+ GTS_int = 1'b1;
+ #(TOC_WIDTH)
+ GTS_int = 1'b0;
+ end
+
+endmodule
+`endif
diff --git a/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.vhdl b/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.vhdl
new file mode 100644
index 0000000..12fd3b1
--- /dev/null
+++ b/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.vhdl
@@ -0,0 +1,179 @@
+-- Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
+-- --------------------------------------------------------------------------------
+-- Tool Version: Vivado v.2019.2 (lin64) Build 2708876 Wed Nov 6 21:39:14 MST 2019
+-- Date : Tue Jul 7 23:25:51 2026
+-- Host : fedora running 64-bit unknown
+-- Command : write_vhdl -force -mode funcsim
+-- /home/fallensigh/Dev/ciciec2026_loongson_regional/rtl/ip/PLL_2019_2/clk_pll_sim_netlist.vhdl
+-- Design : clk_pll
+-- Purpose : This VHDL netlist is a functional simulation representation of the design and should not be modified or
+-- synthesized. This netlist cannot be used for SDF annotated simulation.
+-- Device : xc7a200tfbg676-1
+-- --------------------------------------------------------------------------------
+library IEEE;
+use IEEE.STD_LOGIC_1164.ALL;
+library UNISIM;
+use UNISIM.VCOMPONENTS.ALL;
+entity clk_pll_clk_pll_clk_wiz is
+ port (
+ cpu_clk : out STD_LOGIC;
+ sys_clk : out STD_LOGIC;
+ mlkem_clk : out STD_LOGIC;
+ resetn : in STD_LOGIC;
+ locked : out STD_LOGIC;
+ clk_in1 : in STD_LOGIC
+ );
+ attribute ORIG_REF_NAME : string;
+ attribute ORIG_REF_NAME of clk_pll_clk_pll_clk_wiz : entity is "clk_pll_clk_wiz";
+end clk_pll_clk_pll_clk_wiz;
+
+architecture STRUCTURE of clk_pll_clk_pll_clk_wiz is
+ signal clk_in1_clk_pll : STD_LOGIC;
+ signal clkfbout_buf_clk_pll : STD_LOGIC;
+ signal clkfbout_clk_pll : STD_LOGIC;
+ signal cpu_clk_clk_pll : STD_LOGIC;
+ signal mlkem_clk_clk_pll : STD_LOGIC;
+ signal reset_high : STD_LOGIC;
+ signal sys_clk_clk_pll : STD_LOGIC;
+ signal NLW_plle2_adv_inst_CLKOUT3_UNCONNECTED : STD_LOGIC;
+ signal NLW_plle2_adv_inst_CLKOUT4_UNCONNECTED : STD_LOGIC;
+ signal NLW_plle2_adv_inst_CLKOUT5_UNCONNECTED : STD_LOGIC;
+ signal NLW_plle2_adv_inst_DRDY_UNCONNECTED : STD_LOGIC;
+ signal NLW_plle2_adv_inst_DO_UNCONNECTED : STD_LOGIC_VECTOR ( 15 downto 0 );
+ attribute BOX_TYPE : string;
+ attribute BOX_TYPE of clkf_buf : label is "PRIMITIVE";
+ attribute BOX_TYPE of clkin1_ibufg : label is "PRIMITIVE";
+ attribute CAPACITANCE : string;
+ attribute CAPACITANCE of clkin1_ibufg : label is "DONT_CARE";
+ attribute IBUF_DELAY_VALUE : string;
+ attribute IBUF_DELAY_VALUE of clkin1_ibufg : label is "0";
+ attribute IFD_DELAY_VALUE : string;
+ attribute IFD_DELAY_VALUE of clkin1_ibufg : label is "AUTO";
+ attribute BOX_TYPE of clkout1_buf : label is "PRIMITIVE";
+ attribute BOX_TYPE of clkout2_buf : label is "PRIMITIVE";
+ attribute BOX_TYPE of clkout3_buf : label is "PRIMITIVE";
+ attribute BOX_TYPE of plle2_adv_inst : label is "PRIMITIVE";
+begin
+clkf_buf: unisim.vcomponents.BUFG
+ port map (
+ I => clkfbout_clk_pll,
+ O => clkfbout_buf_clk_pll
+ );
+clkin1_ibufg: unisim.vcomponents.IBUF
+ generic map(
+ IOSTANDARD => "DEFAULT"
+ )
+ port map (
+ I => clk_in1,
+ O => clk_in1_clk_pll
+ );
+clkout1_buf: unisim.vcomponents.BUFG
+ port map (
+ I => cpu_clk_clk_pll,
+ O => cpu_clk
+ );
+clkout2_buf: unisim.vcomponents.BUFG
+ port map (
+ I => sys_clk_clk_pll,
+ O => sys_clk
+ );
+clkout3_buf: unisim.vcomponents.BUFG
+ port map (
+ I => mlkem_clk_clk_pll,
+ O => mlkem_clk
+ );
+plle2_adv_inst: unisim.vcomponents.PLLE2_ADV
+ generic map(
+ BANDWIDTH => "OPTIMIZED",
+ CLKFBOUT_MULT => 31,
+ CLKFBOUT_PHASE => 0.000000,
+ CLKIN1_PERIOD => 20.000000,
+ CLKIN2_PERIOD => 0.000000,
+ CLKOUT0_DIVIDE => 47,
+ CLKOUT0_DUTY_CYCLE => 0.500000,
+ CLKOUT0_PHASE => 0.000000,
+ CLKOUT1_DIVIDE => 31,
+ CLKOUT1_DUTY_CYCLE => 0.500000,
+ CLKOUT1_PHASE => 0.000000,
+ CLKOUT2_DIVIDE => 78,
+ CLKOUT2_DUTY_CYCLE => 0.500000,
+ CLKOUT2_PHASE => 0.000000,
+ CLKOUT3_DIVIDE => 1,
+ CLKOUT3_DUTY_CYCLE => 0.500000,
+ CLKOUT3_PHASE => 0.000000,
+ CLKOUT4_DIVIDE => 1,
+ CLKOUT4_DUTY_CYCLE => 0.500000,
+ CLKOUT4_PHASE => 0.000000,
+ CLKOUT5_DIVIDE => 1,
+ CLKOUT5_DUTY_CYCLE => 0.500000,
+ CLKOUT5_PHASE => 0.000000,
+ COMPENSATION => "ZHOLD",
+ DIVCLK_DIVIDE => 1,
+ IS_CLKINSEL_INVERTED => '0',
+ IS_PWRDWN_INVERTED => '0',
+ IS_RST_INVERTED => '0',
+ REF_JITTER1 => 0.010000,
+ REF_JITTER2 => 0.010000,
+ STARTUP_WAIT => "FALSE"
+ )
+ port map (
+ CLKFBIN => clkfbout_buf_clk_pll,
+ CLKFBOUT => clkfbout_clk_pll,
+ CLKIN1 => clk_in1_clk_pll,
+ CLKIN2 => '0',
+ CLKINSEL => '1',
+ CLKOUT0 => cpu_clk_clk_pll,
+ CLKOUT1 => sys_clk_clk_pll,
+ CLKOUT2 => mlkem_clk_clk_pll,
+ CLKOUT3 => NLW_plle2_adv_inst_CLKOUT3_UNCONNECTED,
+ CLKOUT4 => NLW_plle2_adv_inst_CLKOUT4_UNCONNECTED,
+ CLKOUT5 => NLW_plle2_adv_inst_CLKOUT5_UNCONNECTED,
+ DADDR(6 downto 0) => B"0000000",
+ DCLK => '0',
+ DEN => '0',
+ DI(15 downto 0) => B"0000000000000000",
+ DO(15 downto 0) => NLW_plle2_adv_inst_DO_UNCONNECTED(15 downto 0),
+ DRDY => NLW_plle2_adv_inst_DRDY_UNCONNECTED,
+ DWE => '0',
+ LOCKED => locked,
+ PWRDWN => '0',
+ RST => reset_high
+ );
+plle2_adv_inst_i_1: unisim.vcomponents.LUT1
+ generic map(
+ INIT => X"1"
+ )
+ port map (
+ I0 => resetn,
+ O => reset_high
+ );
+end STRUCTURE;
+library IEEE;
+use IEEE.STD_LOGIC_1164.ALL;
+library UNISIM;
+use UNISIM.VCOMPONENTS.ALL;
+entity clk_pll is
+ port (
+ cpu_clk : out STD_LOGIC;
+ sys_clk : out STD_LOGIC;
+ mlkem_clk : out STD_LOGIC;
+ resetn : in STD_LOGIC;
+ locked : out STD_LOGIC;
+ clk_in1 : in STD_LOGIC
+ );
+ attribute NotValidForBitStream : boolean;
+ attribute NotValidForBitStream of clk_pll : entity is true;
+end clk_pll;
+
+architecture STRUCTURE of clk_pll is
+begin
+inst: entity work.clk_pll_clk_pll_clk_wiz
+ port map (
+ clk_in1 => clk_in1,
+ cpu_clk => cpu_clk,
+ locked => locked,
+ mlkem_clk => mlkem_clk,
+ resetn => resetn,
+ sys_clk => sys_clk
+ );
+end STRUCTURE;
diff --git a/rtl/ip/PLL_2019_2/clk_pll_stub.v b/rtl/ip/PLL_2019_2/clk_pll_stub.v
new file mode 100644
index 0000000..dd75963
--- /dev/null
+++ b/rtl/ip/PLL_2019_2/clk_pll_stub.v
@@ -0,0 +1,25 @@
+// Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
+// --------------------------------------------------------------------------------
+// Tool Version: Vivado v.2019.2 (lin64) Build 2708876 Wed Nov 6 21:39:14 MST 2019
+// Date : Tue Jul 7 23:25:51 2026
+// Host : fedora running 64-bit unknown
+// Command : write_verilog -force -mode synth_stub
+// /home/fallensigh/Dev/ciciec2026_loongson_regional/rtl/ip/PLL_2019_2/clk_pll_stub.v
+// Design : clk_pll
+// Purpose : Stub declaration of top-level module interface
+// Device : xc7a200tfbg676-1
+// --------------------------------------------------------------------------------
+
+// This empty module with port declaration file causes synthesis tools to infer a black box for IP.
+// The synthesis directives are for Synopsys Synplify support to prevent IO buffer insertion.
+// Please paste the declaration into a Verilog source file or add the file as an additional source.
+module clk_pll(cpu_clk, sys_clk, mlkem_clk, resetn, locked,
+ clk_in1)
+/* synthesis syn_black_box black_box_pad_pin="cpu_clk,sys_clk,mlkem_clk,resetn,locked,clk_in1" */;
+ output cpu_clk;
+ output sys_clk;
+ output mlkem_clk;
+ input resetn;
+ output locked;
+ input clk_in1;
+endmodule
diff --git a/rtl/ip/PLL_2019_2/clk_pll_stub.vhdl b/rtl/ip/PLL_2019_2/clk_pll_stub.vhdl
new file mode 100644
index 0000000..0f5f7bf
--- /dev/null
+++ b/rtl/ip/PLL_2019_2/clk_pll_stub.vhdl
@@ -0,0 +1,33 @@
+-- Copyright 1986-2019 Xilinx, Inc. All Rights Reserved.
+-- --------------------------------------------------------------------------------
+-- Tool Version: Vivado v.2019.2 (lin64) Build 2708876 Wed Nov 6 21:39:14 MST 2019
+-- Date : Tue Jul 7 23:25:51 2026
+-- Host : fedora running 64-bit unknown
+-- Command : write_vhdl -force -mode synth_stub
+-- /home/fallensigh/Dev/ciciec2026_loongson_regional/rtl/ip/PLL_2019_2/clk_pll_stub.vhdl
+-- Design : clk_pll
+-- Purpose : Stub declaration of top-level module interface
+-- Device : xc7a200tfbg676-1
+-- --------------------------------------------------------------------------------
+library IEEE;
+use IEEE.STD_LOGIC_1164.ALL;
+
+entity clk_pll is
+ Port (
+ cpu_clk : out STD_LOGIC;
+ sys_clk : out STD_LOGIC;
+ mlkem_clk : out STD_LOGIC;
+ resetn : in STD_LOGIC;
+ locked : out STD_LOGIC;
+ clk_in1 : in STD_LOGIC
+ );
+
+end clk_pll;
+
+architecture stub of clk_pll is
+attribute syn_black_box : boolean;
+attribute black_box_pad_pin : string;
+attribute syn_black_box of stub : architecture is true;
+attribute black_box_pad_pin of stub : architecture is "cpu_clk,sys_clk,mlkem_clk,resetn,locked,clk_in1";
+begin
+end;
diff --git a/rtl/ip/fft/FFT1024_32B.v b/rtl/ip/fft/FFT1024_32B.v
index d940166..dc95d70 100644
--- a/rtl/ip/fft/FFT1024_32B.v
+++ b/rtl/ip/fft/FFT1024_32B.v
@@ -1,7 +1,7 @@
//----------------------------------------------------------------------
// FFT: 1024-Point FFT Using Radix-2^2 Single-Path Delay Feedback
//----------------------------------------------------------------------
-(* use_dsp = "no" *)
+
module FFT #(
parameter WIDTH = 32
)(
diff --git a/rtl/ip/fft/Multiply.v b/rtl/ip/fft/Multiply.v
index 380dbf8..3e52eff 100644
--- a/rtl/ip/fft/Multiply.v
+++ b/rtl/ip/fft/Multiply.v
@@ -1,7 +1,7 @@
//----------------------------------------------------------------------
// Multiply: Complex Multiplier
//----------------------------------------------------------------------
-(* use_dsp = "no" *)
+
module Multiply #(
parameter WIDTH = 16
)(
@@ -13,7 +13,7 @@ module Multiply #(
output signed [WIDTH-1:0] m_im
);
-(* use_dsp = "no" *) wire signed [WIDTH*2-1:0] arbr, arbi, aibr, aibi;
+ wire signed [WIDTH*2-1:0] arbr, arbi, aibr, aibi;
wire signed [WIDTH-1:0] sc_arbr, sc_arbi, sc_aibr, sc_aibi;
// Signed Multiplication
diff --git a/rtl/ip/fft/SdfUnit.v b/rtl/ip/fft/SdfUnit.v
index 0f33775..4bc548a 100644
--- a/rtl/ip/fft/SdfUnit.v
+++ b/rtl/ip/fft/SdfUnit.v
@@ -1,7 +1,7 @@
//----------------------------------------------------------------------
// SdfUnit: Radix-2^2 Single-Path Delay Feedback Unit for N-Point FFT
//----------------------------------------------------------------------
-(* use_dsp = "no" *)
+
module SdfUnit #(
parameter N = 64, // Number of FFT Point
parameter M = 64, // Twiddle Resolution
@@ -86,7 +86,7 @@ reg bf2_do_en; // 2nd Butterfly Output Data Enable
// Multiplication
wire[1:0] tw_sel; // Twiddle Select (2n/n/3n)
wire[LOG_N-3:0] tw_num; // Twiddle Number (n)
-(* use_dsp = "no" *) wire[LOG_N-1:0] tw_addr; // Twiddle Table Address
+ wire[LOG_N-1:0] tw_addr; // Twiddle Table Address
wire[WIDTH-1:0] tw_re; // Twiddle Factor (Real)
wire[WIDTH-1:0] tw_im; // Twiddle Factor (Imag)
reg mu_en; // Multiplication Enable
diff --git a/rtl/ip/fft/axi_fft_wrapper.v b/rtl/ip/fft/axi_fft_wrapper.v
index bc51f86..ada2700 100644
--- a/rtl/ip/fft/axi_fft_wrapper.v
+++ b/rtl/ip/fft/axi_fft_wrapper.v
@@ -7,7 +7,7 @@
(* use_dsp = "no" *)
module fft_controller #(
parameter FFT_LEN = 1024, // FFT处理长度
- parameter BRAM_RD_LATENCY = 2, // 适配 BRAM 的读延迟 (通常 Block RAM 开启输出寄存器为 2)
+ parameter BRAM_RD_LATENCY = 1, // 适配 BRAM 的读延迟 (通常 Block RAM 开启输出寄存器为 1)
parameter FFT_DATA_WIDTH = 16
)(
input aclk,
diff --git a/rtl/ip/matmul/matmul_axi_slave.v b/rtl/ip/matmul/matmul_axi_slave.v
index 39112bc..91afc3f 100644
--- a/rtl/ip/matmul/matmul_axi_slave.v
+++ b/rtl/ip/matmul/matmul_axi_slave.v
@@ -2,7 +2,7 @@
// 3-stage pipelined multiplier (LUT-based, no DSP)
// latency=3, throughput=1 result/cycle
// ═══════════════════════════════════════════════════════
-(* use_dsp = "no" *)
+
module pipelined_mult_32x32 (
input clk,
input rstn,
diff --git a/rtl/ip/mlkem/comp_decomp/comp_decomp_sync.v b/rtl/ip/mlkem/comp_decomp/comp_decomp_sync.v
index 1d5c47b..41870f5 100644
--- a/rtl/ip/mlkem/comp_decomp/comp_decomp_sync.v
+++ b/rtl/ip/mlkem/comp_decomp/comp_decomp_sync.v
@@ -1,8 +1,9 @@
// comp_decomp_sync.v - ML-KEM coefficient compression/decompression
//
-// Streaming: one coefficient per cycle through pipeline_reg.
+// Streaming valid/ready interface. The arithmetic is fixed-latency pipelined
+// so compression does not infer a long combinational divider/reducer.
// mode=0: compress — round((2^d * x) / Q) mod 2^d
-// mode=1: decompress — round((Q * x) / 2^d) mod Q
+// mode=1: decompress — round((Q * x) / 2^d)
// Uses round-half-up (round(2.5)=3, round(3.5)=4).
// Integer arithmetic:
// compress: (x * 2^d + Q/2) / Q, lower d bits as result
@@ -10,7 +11,7 @@
`include "defines.vh"
-(* use_dsp = "no" *)
+(* use_dsp = "yes" *)
module comp_decomp_sync (
input clk,
input rst_n,
@@ -24,52 +25,171 @@ module comp_decomp_sync (
input ready_i
);
- // 2^d for d in {4,5,10,11}; fits in 12 bits (max 2048)
- wire [11:0] two_pow_d;
- assign two_pow_d = 12'd1 << d;
+ localparam [24:0] Q25 = 25'd`Q;
+ localparam [12:0] K13 = 13'd5039; // floor(2^24 / 3329)
- // Product: 12-bit * 12-bit = max 2048*3328 = 6,815,744 → 23 bits (use 24)
- wire [23:0] product;
+ reg [11:0] data_r;
+ reg valid_r;
- // compress: x * 2^d; decompress: x * Q
- assign product = mode ? {12'b0, coeff_in} * {12'b0, 12'd3329}
- : {12'b0, coeff_in} * {12'b0, two_pow_d};
+ reg mode_s0;
+ reg [4:0] d_s0;
+ reg [11:0] coeff_s0;
+ reg valid_s0;
- // Rounding offset: compress→Q/2=1664; decompress→2^(d-1)
- wire [11:0] round_off;
- assign round_off = mode ? (two_pow_d >> 1) : 12'd1664;
+ reg mode_s1;
+ reg [4:0] d_s1;
+ reg [23:0] comp_dividend_s1;
+ reg [23:0] decomp_rounded_s1;
+ reg valid_s1;
- // Dividend = product + round_off (max ~ 6,817,408 fits in 24 bits)
- wire [23:0] dividend;
- assign dividend = product + {12'b0, round_off};
+ reg mode_s2;
+ reg [4:0] d_s2;
+ reg [23:0] comp_dividend_s2;
+ reg [36:0] comp_qprod_est_s2;
+ reg [23:0] decomp_rounded_s2;
+ reg valid_s2;
- // Divisor: compress→Q=3329; decompress→2^d
- wire [11:0] divisor;
- assign divisor = mode ? two_pow_d : 12'd3329;
+ reg mode_s3;
+ reg [4:0] d_s3;
+ reg [23:0] comp_dividend_s3;
+ reg [12:0] comp_q_est_s3;
+ reg [11:0] decomp_result_s3;
+ reg valid_s3;
- // Integer division (both ops positive → floor)
- // Quotient is computed as 24b (widest operand) but fits in 12b
- /* verilator lint_off WIDTH */
- wire [11:0] div_result;
- assign div_result = dividend / {12'b0, divisor};
- /* verilator lint_on WIDTH */
+ reg mode_s4;
+ reg [4:0] d_s4;
+ reg [23:0] comp_dividend_s4;
+ reg [12:0] comp_q_est_s4;
+ reg [24:0] comp_q_mul_q_s4;
+ reg [11:0] decomp_result_s4;
+ reg valid_s4;
- // Final modular reduction
- // compress: lower d bits (mask with 2^d - 1)
- // decompress: result < Q always, but apply mod Q for safety
- wire [11:0] mod_result;
- assign mod_result = mode ? (div_result % 12'd3329) : (div_result & (two_pow_d - 1'b1));
+ reg mode_s5;
+ reg [4:0] d_s5;
+ reg [12:0] comp_q_est_s5;
+ reg [24:0] comp_rem_s5;
+ reg [11:0] decomp_result_s5;
+ reg valid_s5;
- // Pipeline through valid/ready stage
- pipeline_reg #(.DW(12)) u_pipe (
- .clk (clk),
- .rst_n (rst_n),
- .data_i (mod_result),
- .valid_i(valid_i),
- .ready_o(ready_o),
- .data_o (coeff_out),
- .valid_o(valid_o),
- .ready_i(ready_i)
- );
+ wire fire_i = valid_i && ready_o;
+
+ wire pipe_busy = valid_s0 | valid_s1 | valid_s2 | valid_s3 | valid_s4 | valid_s5;
+
+ // Accept a new command only when the pipeline and output slot are both free.
+ // Current users issue one coefficient at a time and wait for valid_o.
+ assign ready_o = !pipe_busy && (!valid_r || ready_i);
+ assign coeff_out = data_r;
+ assign valid_o = valid_r;
+
+ wire [23:0] coeff_ext_s0 = {12'd0, coeff_s0};
+ wire [23:0] comp_dividend_next = (coeff_ext_s0 << d_s0) + 24'd1664;
+ wire [23:0] decomp_product_next = coeff_s0 * 12'd`Q;
+ wire [23:0] decomp_round_next =
+ decomp_product_next + ((d_s0 == 5'd0) ? 24'd0 : (24'd1 << (d_s0 - 5'd1)));
+
+ wire [12:0] comp_q_corr1_s5 =
+ comp_q_est_s5 + ((comp_rem_s5 >= Q25) ? 13'd1 : 13'd0);
+ wire [24:0] comp_rem_corr1_s5 =
+ (comp_rem_s5 >= Q25) ? (comp_rem_s5 - Q25) : comp_rem_s5;
+ wire [12:0] comp_q_corr2_s5 =
+ comp_q_corr1_s5 + ((comp_rem_corr1_s5 >= Q25) ? 13'd1 : 13'd0);
+ wire [11:0] comp_mask_s5 =
+ (d_s5 == 5'd0) ? 12'd0 : (12'd1 << d_s5) - 12'd1;
+ wire [11:0] comp_result_s5 = comp_q_corr2_s5[11:0] & comp_mask_s5;
+
+ always @(posedge clk or negedge rst_n) begin
+ if (!rst_n) begin
+ data_r <= 12'd0;
+ valid_r <= 1'b0;
+
+ mode_s0 <= 1'b0;
+ d_s0 <= 5'd0;
+ coeff_s0 <= 12'd0;
+ valid_s0 <= 1'b0;
+
+ mode_s1 <= 1'b0;
+ d_s1 <= 5'd0;
+ comp_dividend_s1 <= 24'd0;
+ decomp_rounded_s1 <= 24'd0;
+ valid_s1 <= 1'b0;
+
+ mode_s2 <= 1'b0;
+ d_s2 <= 5'd0;
+ comp_dividend_s2 <= 24'd0;
+ comp_qprod_est_s2 <= 37'd0;
+ decomp_rounded_s2 <= 24'd0;
+ valid_s2 <= 1'b0;
+
+ mode_s3 <= 1'b0;
+ d_s3 <= 5'd0;
+ comp_dividend_s3 <= 24'd0;
+ comp_q_est_s3 <= 13'd0;
+ decomp_result_s3 <= 12'd0;
+ valid_s3 <= 1'b0;
+
+ mode_s4 <= 1'b0;
+ d_s4 <= 5'd0;
+ comp_dividend_s4 <= 24'd0;
+ comp_q_est_s4 <= 13'd0;
+ comp_q_mul_q_s4 <= 25'd0;
+ decomp_result_s4 <= 12'd0;
+ valid_s4 <= 1'b0;
+
+ mode_s5 <= 1'b0;
+ d_s5 <= 5'd0;
+ comp_q_est_s5 <= 13'd0;
+ comp_rem_s5 <= 25'd0;
+ decomp_result_s5 <= 12'd0;
+ valid_s5 <= 1'b0;
+ end else begin
+ if (valid_r && ready_i)
+ valid_r <= 1'b0;
+
+ mode_s0 <= mode;
+ d_s0 <= d;
+ coeff_s0 <= coeff_in;
+ valid_s0 <= fire_i;
+
+ mode_s1 <= mode_s0;
+ d_s1 <= d_s0;
+ comp_dividend_s1 <= comp_dividend_next;
+ decomp_rounded_s1 <= decomp_round_next;
+ valid_s1 <= valid_s0;
+
+ mode_s2 <= mode_s1;
+ d_s2 <= d_s1;
+ comp_dividend_s2 <= comp_dividend_s1;
+ comp_qprod_est_s2 <= {13'd0, comp_dividend_s1} * K13;
+ decomp_rounded_s2 <= decomp_rounded_s1;
+ valid_s2 <= valid_s1;
+
+ mode_s3 <= mode_s2;
+ d_s3 <= d_s2;
+ comp_dividend_s3 <= comp_dividend_s2;
+ comp_q_est_s3 <= comp_qprod_est_s2[36:24];
+ decomp_result_s3 <= decomp_rounded_s2 >> d_s2;
+ valid_s3 <= valid_s2;
+
+ mode_s4 <= mode_s3;
+ d_s4 <= d_s3;
+ comp_dividend_s4 <= comp_dividend_s3;
+ comp_q_est_s4 <= comp_q_est_s3;
+ comp_q_mul_q_s4 <= comp_q_est_s3 * Q25;
+ decomp_result_s4 <= decomp_result_s3;
+ valid_s4 <= valid_s3;
+
+ mode_s5 <= mode_s4;
+ d_s5 <= d_s4;
+ comp_q_est_s5 <= comp_q_est_s4;
+ comp_rem_s5 <= {1'b0, comp_dividend_s4} - comp_q_mul_q_s4;
+ decomp_result_s5 <= decomp_result_s4;
+ valid_s5 <= valid_s4;
+
+ if (valid_s5) begin
+ data_r <= mode_s5 ? decomp_result_s5 : comp_result_s5;
+ valid_r <= 1'b1;
+ end
+ end
+ end
endmodule
diff --git a/rtl/ip/mlkem/ntt/barrett_mul.v b/rtl/ip/mlkem/ntt/barrett_mul.v
index 6fe5d2d..dcc2be9 100644
--- a/rtl/ip/mlkem/ntt/barrett_mul.v
+++ b/rtl/ip/mlkem/ntt/barrett_mul.v
@@ -6,7 +6,7 @@
// Pure combinational, single-cycle latency.
// All multiplication widths explicitly controlled to avoid Verilog truncation.
-(* use_dsp = "no" *)
+
module barrett_mul (
input [11:0] a,
input [11:0] b,
diff --git a/rtl/ip/mlkem/ntt/barrett_mul_pipe.v b/rtl/ip/mlkem/ntt/barrett_mul_pipe.v
new file mode 100644
index 0000000..253ae52
--- /dev/null
+++ b/rtl/ip/mlkem/ntt/barrett_mul_pipe.v
@@ -0,0 +1,72 @@
+// barrett_mul_pipe.v - DSP-friendly pipelined Barrett modular multiply
+//
+// Computes product = a * b mod Q with fixed latency. This module is used on
+// timing-critical NTT paths; the legacy barrett_mul remains combinational for
+// older users.
+
+module barrett_mul_pipe (
+ input clk,
+ input rst_n,
+ input valid_i,
+ input [11:0] a,
+ input [11:0] b,
+ output [11:0] product,
+ output valid_o
+);
+ localparam [24:0] Q25 = 25'd3329;
+ localparam [12:0] K13 = 13'd5039;
+
+ reg [23:0] p_s1;
+
+ reg [23:0] p_s2;
+ reg [36:0] tk_s2;
+
+ reg [23:0] p_s3;
+ reg [12:0] q_s3;
+
+ reg [24:0] p_s4;
+ reg [24:0] qprod_s4;
+
+ reg [24:0] r0_s5;
+ reg [24:0] r1_s6;
+ reg [11:0] product_r;
+ reg [6:0] valid_sr;
+
+ assign product = product_r;
+ assign valid_o = valid_sr[6];
+
+ always @(posedge clk or negedge rst_n) begin
+ if (!rst_n) begin
+ p_s1 <= 24'd0;
+ p_s2 <= 24'd0; tk_s2 <= 37'd0;
+ p_s3 <= 24'd0; q_s3 <= 13'd0;
+ p_s4 <= 25'd0; qprod_s4 <= 25'd0;
+ r0_s5 <= 25'd0; r1_s6 <= 25'd0;
+ product_r <= 12'd0;
+ valid_sr <= 7'd0;
+ end else begin
+ valid_sr <= {valid_sr[5:0], valid_i};
+
+ // S1: full product.
+ p_s1 <= {12'd0, a} * b;
+
+ // S2: quotient-estimate product.
+ p_s2 <= p_s1;
+ tk_s2 <= {13'd0, p_s1} * K13;
+
+ // S3: quotient estimate.
+ p_s3 <= p_s2;
+ q_s3 <= tk_s2[36:24];
+
+ // S4: q * Q.
+ p_s4 <= {1'b0, p_s3};
+ qprod_s4 <= q_s3 * Q25;
+
+ // S5/S6/S7: subtract q*Q and conditionally subtract Q twice.
+ r0_s5 <= p_s4 - qprod_s4;
+ r1_s6 <= (r0_s5 >= Q25) ? (r0_s5 - Q25) : r0_s5;
+ product_r <= (r1_s6 >= Q25) ? (r1_s6 - Q25) : r1_s6[11:0];
+ end
+ end
+
+endmodule
diff --git a/rtl/ip/mlkem/ntt/butterfly_unit_pipe.v b/rtl/ip/mlkem/ntt/butterfly_unit_pipe.v
new file mode 100644
index 0000000..c672953
--- /dev/null
+++ b/rtl/ip/mlkem/ntt/butterfly_unit_pipe.v
@@ -0,0 +1,74 @@
+// butterfly_unit_pipe.v - pipelined Cooley-Tukey / Gentleman-Sande butterfly
+//
+// Fixed-latency wrapper around barrett_mul_pipe. All arithmetic that used to
+// sit in one NTT cycle is now split by registers.
+
+module butterfly_unit_pipe (
+ input clk,
+ input rst_n,
+ input valid_i,
+ input [11:0] a,
+ input [11:0] b,
+ input [11:0] zeta,
+ input mode, // 0 = forward NTT, 1 = inverse NTT
+ output [11:0] a_out,
+ output [11:0] b_out,
+ output valid_o
+);
+ localparam [11:0] Q12 = 12'd3329;
+ localparam [12:0] Q13 = 13'd3329;
+
+ wire [11:0] diff = (b >= a) ? (b - a) : (b - a + Q12);
+ wire [11:0] mul_data = mode ? diff : b;
+
+ wire [11:0] mul_result;
+ wire mul_valid;
+ barrett_mul_pipe u_barrett (
+ .clk(clk),
+ .rst_n(rst_n),
+ .valid_i(valid_i),
+ .a(zeta),
+ .b(mul_data),
+ .product(mul_result),
+ .valid_o(mul_valid)
+ );
+
+ reg [11:0] a_d0, a_d1, a_d2, a_d3, a_d4, a_d5, a_d6;
+ reg [11:0] b_d0, b_d1, b_d2, b_d3, b_d4, b_d5, b_d6;
+ reg mode_d0, mode_d1, mode_d2, mode_d3, mode_d4, mode_d5, mode_d6;
+
+ reg [11:0] a_out_r, b_out_r;
+ reg valid_r;
+
+ wire [12:0] a_sum = {1'b0, a_d6} + {1'b0, (mode_d6 ? b_d6 : mul_result)};
+ wire [11:0] a_mod = (a_sum >= Q13) ? (a_sum[11:0] - Q12) : a_sum[11:0];
+ wire [11:0] b_sub = (a_d6 >= mul_result) ? (a_d6 - mul_result) :
+ (a_d6 - mul_result + Q12);
+
+ assign a_out = a_out_r;
+ assign b_out = b_out_r;
+ assign valid_o = valid_r;
+
+ always @(posedge clk or negedge rst_n) begin
+ if (!rst_n) begin
+ a_d0 <= 12'd0; a_d1 <= 12'd0; a_d2 <= 12'd0; a_d3 <= 12'd0; a_d4 <= 12'd0; a_d5 <= 12'd0; a_d6 <= 12'd0;
+ b_d0 <= 12'd0; b_d1 <= 12'd0; b_d2 <= 12'd0; b_d3 <= 12'd0; b_d4 <= 12'd0; b_d5 <= 12'd0; b_d6 <= 12'd0;
+ mode_d0 <= 1'b0; mode_d1 <= 1'b0; mode_d2 <= 1'b0; mode_d3 <= 1'b0; mode_d4 <= 1'b0; mode_d5 <= 1'b0; mode_d6 <= 1'b0;
+ a_out_r <= 12'd0;
+ b_out_r <= 12'd0;
+ valid_r <= 1'b0;
+ end else begin
+ a_d0 <= a; a_d1 <= a_d0; a_d2 <= a_d1; a_d3 <= a_d2; a_d4 <= a_d3; a_d5 <= a_d4; a_d6 <= a_d5;
+ b_d0 <= b; b_d1 <= b_d0; b_d2 <= b_d1; b_d3 <= b_d2; b_d4 <= b_d3; b_d5 <= b_d4; b_d6 <= b_d5;
+ mode_d0 <= mode; mode_d1 <= mode_d0; mode_d2 <= mode_d1;
+ mode_d3 <= mode_d2; mode_d4 <= mode_d3; mode_d5 <= mode_d4; mode_d6 <= mode_d5;
+
+ valid_r <= mul_valid;
+ if (mul_valid) begin
+ a_out_r <= a_mod;
+ b_out_r <= mode_d6 ? mul_result : b_sub;
+ end
+ end
+ end
+
+endmodule
diff --git a/rtl/ip/mlkem/ntt/ntt_core.v b/rtl/ip/mlkem/ntt/ntt_core.v
index 1299b8c..a326990 100644
--- a/rtl/ip/mlkem/ntt/ntt_core.v
+++ b/rtl/ip/mlkem/ntt/ntt_core.v
@@ -1,8 +1,12 @@
// ntt_core.v - NTT core with individual coefficient registers
//
-// Uses 256 individual 12-bit registers and generate-based muxing
-// to avoid any part-select simulation issues.
-// 3-cycle butterfly: SetAddr -> Read -> Compute+Write
+// Uses 256 individual 12-bit registers and a deeply pipelined butterfly path.
+// The arithmetic hot path is split into:
+// address/operand/zeta register -> pipelined Barrett butterfly -> writeback
+// Each NTT layer is issued continuously into the butterfly pipeline, then the
+// core drains pending writebacks before starting the next dependent layer.
+// In inverse mode, final x3303 output scaling also uses a pipelined Barrett
+// multiplier so the output path does not reintroduce a combinational reducer.
module ntt_core (
input clk, rst_n,
@@ -17,72 +21,170 @@ module ntt_core (
);
localparam N = 256, LAYERS = 7, DW = 12;
- // Individual coefficient registers
reg [DW-1:0] cr [0:N-1];
integer ci;
- // State machine
- localparam S_IDLE=3'd0, S_LOAD=3'd1, S_CMP_A=3'd2, S_CMP_B=3'd3,
- S_CMP_C=3'd4, S_OUTPUT=3'd5, S_DONE=3'd6;
- reg [2:0] state, next_state;
+ localparam S_IDLE = 4'd0;
+ localparam S_LOAD = 4'd1;
+ localparam S_CMP_RUN = 4'd2;
+ localparam S_CMP_DRAIN = 4'd3;
+ localparam S_OUT_PREP = 4'd7;
+ localparam S_OUTPUT = 4'd8;
+ localparam S_OUT_SCALE = 4'd9;
+ localparam S_DONE = 4'd10;
- reg [7:0] load_cnt, out_cnt;
+ reg [3:0] state, next_state;
+
+ reg [7:0] load_cnt;
+ reg [7:0] out_cnt;
+ reg [8:0] scale_issue_cnt;
+ reg [8:0] scale_emit_cnt;
reg [7:0] j, start, layer_len;
reg [6:0] zeta_idx;
reg [2:0] layer;
- reg bf_done;
+ reg bf_done;
+ reg mode_r;
+ reg layer_issue_done;
+ reg [4:0] bf_inflight;
- // Pipeline registers
- reg [DW-1:0] r_a, r_b;
- reg [7:0] r_wa, r_wb;
+ reg [7:0] wa_d0, wa_d1, wa_d2, wa_d3, wa_d4, wa_d5, wa_d6, wa_d7;
+ reg [7:0] wb_d0, wb_d1, wb_d2, wb_d3, wb_d4, wb_d5, wb_d6, wb_d7;
+ reg wb_valid_r;
+ reg [DW-1:0] wb_a_data_r, wb_b_data_r;
+ reg [7:0] wb_wa_r, wb_wb_r;
+
+ reg [DW-1:0] coeff_out_r;
+ reg valid_o_r;
+ reg scale_valid_i;
+ reg [DW-1:0] scale_a_i;
- // Zeta
wire [DW-1:0] zeta;
zeta_rom u_z (.addr(zeta_idx), .zeta(zeta));
- // Butterfly
+ wire [8:0] group_end = {1'b0, start} + {1'b0, layer_len};
+ wire [8:0] next_group_start = {1'b0, start} + {1'b0, layer_len} + {1'b0, layer_len};
+ wire issue_group_last = ({1'b0, j} + 9'd1 >= group_end);
+ wire issue_layer_last = issue_group_last && (next_group_start >= 9'd256);
+ wire [7:0] next_layer_len = mode_r ? (layer_len << 1) : (layer_len >> 1);
+ wire issue_fire = (state == S_CMP_RUN) && !layer_issue_done;
+ wire [DW-1:0] issue_a = cr[j];
+ wire [DW-1:0] issue_b = cr[j + layer_len];
+
wire [DW-1:0] bf_a_out, bf_b_out;
- butterfly_unit u_bf (
- .a(r_a), .b(r_b), .zeta(zeta), .mode(mode),
- .a_out(bf_a_out), .b_out(bf_b_out));
+ wire bf_valid;
+ butterfly_unit_pipe u_bf (
+ .clk(clk),
+ .rst_n(rst_n),
+ .valid_i(issue_fire),
+ .a(issue_a),
+ .b(issue_b),
+ .zeta(zeta),
+ .mode(mode_r),
+ .a_out(bf_a_out),
+ .b_out(bf_b_out),
+ .valid_o(bf_valid)
+ );
- // Output scaling
- wire [DW-1:0] coeff_scaled;
- barrett_mul u_scl (.a(cr[out_cnt]), .b(12'd3303), .product(coeff_scaled));
- assign coeff_out = mode ? coeff_scaled : cr[out_cnt];
+ wire [DW-1:0] scale_product;
+ wire scale_valid_o;
+ barrett_mul_pipe u_scl (
+ .clk(clk),
+ .rst_n(rst_n),
+ .valid_i(scale_valid_i),
+ .a(scale_a_i),
+ .b(12'd3303),
+ .product(scale_product),
+ .valid_o(scale_valid_o)
+ );
-
- assign ready_o = (state == S_IDLE) || (state == S_LOAD);
- assign valid_o = (state == S_OUTPUT);
- assign done_o = (state == S_DONE);
+ assign ready_o = (state == S_IDLE) || (state == S_LOAD);
+ assign coeff_out = coeff_out_r;
+ assign valid_o = valid_o_r;
+ assign done_o = (state == S_DONE);
always @* begin
next_state = state;
case (state)
- S_IDLE: if (valid_i) next_state = S_LOAD;
- S_LOAD: if (load_cnt >= 255 && valid_i) next_state = S_CMP_A;
- S_CMP_A: if (bf_done) next_state = S_OUTPUT; else next_state = S_CMP_B;
- S_CMP_B: if (bf_done) next_state = S_OUTPUT; else next_state = S_CMP_C;
- S_CMP_C: if (bf_done) next_state = S_OUTPUT; else next_state = S_CMP_A;
- S_OUTPUT:if (out_cnt >= 255 && ready_i) next_state = S_DONE;
- S_DONE: next_state = S_IDLE;
- default: next_state = S_IDLE;
+ S_IDLE: if (valid_i) next_state = S_LOAD;
+ S_LOAD: if (load_cnt >= 8'd255 && valid_i) next_state = S_CMP_RUN;
+ S_CMP_RUN: if (layer_issue_done) next_state = S_CMP_DRAIN;
+ S_CMP_DRAIN: if (bf_inflight == 5'd0)
+ next_state = (layer + 3'd1 >= LAYERS) ? S_OUT_PREP : S_CMP_RUN;
+ S_OUT_PREP: next_state = mode_r ? S_OUT_SCALE : S_OUTPUT;
+ S_OUTPUT: if (valid_o_r && ready_i && out_cnt >= 8'd255) next_state = S_DONE;
+ S_OUT_SCALE: if (scale_emit_cnt >= 9'd256) next_state = S_DONE;
+ S_DONE: next_state = S_IDLE;
+ default: next_state = S_IDLE;
endcase
end
always @(posedge clk or negedge rst_n) begin
if (!rst_n) begin
- state<=S_IDLE; load_cnt<=0; out_cnt<=0; j<=0; start<=0; layer_len<=0;
- zeta_idx<=0; layer<=0; bf_done<=0; r_a<=0; r_b<=0; r_wa<=0; r_wb<=0;
- for (ci=0; ci= start + layer_len) begin
- if (!mode) zeta_idx <= zeta_idx + 7'd1;
- else zeta_idx <= zeta_idx - 7'd1;
-
- if ({1'b0,start} + {1'b0,layer_len} + {1'b0,layer_len} >= 256) begin
- layer <= layer + 3'd1;
- layer_len <= mode ? (layer_len<<1) : (layer_len>>1);
- start <= 0; j <= 0;
- if (layer + 3'd1 >= LAYERS) bf_done <= 1'b1;
+ if (issue_layer_last) begin
+ layer_issue_done <= 1'b1;
end else begin
- start <= start + layer_len + layer_len;
- j <= start + layer_len + layer_len;
+ start <= next_group_start[7:0];
+ j <= next_group_start[7:0];
end
+ end else begin
+ j <= j + 8'd1;
end
end
- if (state == S_OUTPUT && ready_i)
- out_cnt <= (out_cnt>=255) ? 0 : (out_cnt+8'd1);
+ if (wb_valid_r) begin
+ cr[wb_wa_r] <= wb_a_data_r;
+ cr[wb_wb_r] <= wb_b_data_r;
+ end
+
+ wb_valid_r <= bf_valid;
+ if (bf_valid) begin
+ wb_a_data_r <= bf_a_out;
+ wb_b_data_r <= bf_b_out;
+ wb_wa_r <= wa_d7;
+ wb_wb_r <= wb_d7;
+ end
+
+ if (state == S_CMP_DRAIN && bf_inflight == 5'd0) begin
+ if (layer + 3'd1 >= LAYERS) begin
+ bf_done <= 1'b1;
+ end else begin
+ layer <= layer + 3'd1;
+ layer_len <= next_layer_len;
+ start <= 8'd0;
+ j <= 8'd0;
+ layer_issue_done <= 1'b0;
+ end
+ end
+
+ if (state == S_OUT_PREP) begin
+ out_cnt <= 8'd0;
+ scale_issue_cnt <= 9'd0;
+ scale_emit_cnt <= 9'd0;
+ if (!mode_r) begin
+ coeff_out_r <= cr[0];
+ valid_o_r <= 1'b1;
+ end
+ end
+
+ if (state == S_OUTPUT && valid_o_r && ready_i) begin
+ if (out_cnt < 8'd255) begin
+ out_cnt <= out_cnt + 8'd1;
+ coeff_out_r <= cr[out_cnt + 8'd1];
+ valid_o_r <= 1'b1;
+ end else begin
+ out_cnt <= 8'd0;
+ valid_o_r <= 1'b0;
+ end
+ end
+
+ if (state == S_OUT_SCALE) begin
+ if (scale_issue_cnt < 9'd256) begin
+ scale_valid_i <= 1'b1;
+ scale_a_i <= cr[scale_issue_cnt[7:0]];
+ scale_issue_cnt <= scale_issue_cnt + 9'd1;
+ end
+
+ valid_o_r <= 1'b0;
+ if (scale_valid_o) begin
+ coeff_out_r <= scale_product;
+ valid_o_r <= 1'b1;
+ scale_emit_cnt <= scale_emit_cnt + 9'd1;
+ end
+ end
+
+ if (state == S_DONE) begin
+ load_cnt <= 8'd0;
+ out_cnt <= 8'd0;
+ scale_issue_cnt <= 9'd0;
+ scale_emit_cnt <= 9'd0;
+ layer_issue_done <= 1'b0;
+ bf_inflight <= 5'd0;
+ wb_valid_r <= 1'b0;
+ valid_o_r <= 1'b0;
+ end
end
end
diff --git a/rtl/ip/mlkem/poly_mul/basecase_mul_pipe.v b/rtl/ip/mlkem/poly_mul/basecase_mul_pipe.v
new file mode 100644
index 0000000..9ccacdf
--- /dev/null
+++ b/rtl/ip/mlkem/poly_mul/basecase_mul_pipe.v
@@ -0,0 +1,143 @@
+// basecase_mul_pipe.v - pipelined NTT-domain degree-1 multiplication
+//
+// Fixed-latency replacement for the combinational basecase_mul hot path.
+// The shared ntt/barrett_mul remains combinational for NTT users; this module
+// keeps the extra registers local to poly_mul_sync.
+
+module basecase_mul_pipe (
+ input clk,
+ input rst_n,
+ input valid_i,
+ input [11:0] a0, a1,
+ input [11:0] b0, b1,
+ input [11:0] zeta,
+ output [11:0] c0,
+ output [11:0] c1,
+ output valid_o
+);
+ localparam [11:0] Q12 = 12'd3329;
+ localparam [12:0] K13 = 13'd5039;
+ localparam [24:0] Q25 = 25'd3329;
+
+ function [12:0] barrett_q;
+ input [23:0] p;
+ reg [36:0] prod;
+ begin
+ prod = {13'd0, p} * K13;
+ barrett_q = prod[36:24];
+ end
+ endfunction
+
+ function [11:0] barrett_reduce;
+ input [23:0] p;
+ input [12:0] qe;
+ reg [24:0] q_approx;
+ reg [24:0] r0;
+ reg [24:0] r1;
+ reg [24:0] r2;
+ begin
+ q_approx = qe * Q12;
+ r0 = {1'b0, p} - q_approx;
+ r1 = (r0 >= Q25) ? (r0 - Q25) : r0;
+ r2 = (r1 >= Q25) ? (r1 - Q25) : r1;
+ barrett_reduce = r2[11:0];
+ end
+ endfunction
+
+ function [11:0] mod_add;
+ input [11:0] x;
+ input [11:0] y;
+ reg [12:0] sum;
+ begin
+ sum = {1'b0, x} + {1'b0, y};
+ mod_add = (sum >= {1'b0, Q12}) ? (sum[11:0] - Q12) : sum[11:0];
+ end
+ endfunction
+
+ // Stage 1: first-level scalar products.
+ reg [23:0] p10, p11, p12, p13;
+ reg [11:0] zeta_s1;
+
+ // Stage 2: Barrett quotient estimates for first-level products.
+ reg [23:0] p20, p21, p22, p23;
+ reg [12:0] q20, q21, q22, q23;
+ reg [11:0] zeta_s2;
+
+ // Stage 3: first-level reduced products.
+ reg [11:0] t1_s3, t2_s3, t3_s3, t4_s3;
+ reg [11:0] zeta_s3;
+
+ // Stage 4: zeta product for c0's second Barrett multiply.
+ reg [11:0] t1_s4, t3_s4, t4_s4;
+ reg [23:0] pz_s4;
+
+ // Stage 5: Barrett quotient estimate for zeta product.
+ reg [11:0] t1_s5, t3_s5, t4_s5;
+ reg [23:0] pz_s5;
+ reg [12:0] qz_s5;
+
+ // Stage 6: reduced zeta product.
+ reg [11:0] t1_s6, t3_s6, t4_s6, tz_s6;
+
+ // Stage 7: final modular additions.
+ reg [11:0] c0_r, c1_r;
+ reg [6:0] valid_sr;
+
+ assign c0 = c0_r;
+ assign c1 = c1_r;
+ assign valid_o = valid_sr[6];
+
+ always @(posedge clk or negedge rst_n) begin
+ if (!rst_n) begin
+ p10 <= 24'd0; p11 <= 24'd0; p12 <= 24'd0; p13 <= 24'd0; zeta_s1 <= 12'd0;
+ p20 <= 24'd0; p21 <= 24'd0; p22 <= 24'd0; p23 <= 24'd0;
+ q20 <= 13'd0; q21 <= 13'd0; q22 <= 13'd0; q23 <= 13'd0; zeta_s2 <= 12'd0;
+ t1_s3 <= 12'd0; t2_s3 <= 12'd0; t3_s3 <= 12'd0; t4_s3 <= 12'd0; zeta_s3 <= 12'd0;
+ t1_s4 <= 12'd0; t3_s4 <= 12'd0; t4_s4 <= 12'd0; pz_s4 <= 24'd0;
+ t1_s5 <= 12'd0; t3_s5 <= 12'd0; t4_s5 <= 12'd0; pz_s5 <= 24'd0; qz_s5 <= 13'd0;
+ t1_s6 <= 12'd0; t3_s6 <= 12'd0; t4_s6 <= 12'd0; tz_s6 <= 12'd0;
+ c0_r <= 12'd0; c1_r <= 12'd0; valid_sr <= 7'd0;
+ end else begin
+ valid_sr <= {valid_sr[5:0], valid_i};
+
+ p10 <= {12'd0, a0} * b0;
+ p11 <= {12'd0, a1} * b1;
+ p12 <= {12'd0, a1} * b0;
+ p13 <= {12'd0, a0} * b1;
+ zeta_s1 <= zeta;
+
+ p20 <= p10; p21 <= p11; p22 <= p12; p23 <= p13;
+ q20 <= barrett_q(p10);
+ q21 <= barrett_q(p11);
+ q22 <= barrett_q(p12);
+ q23 <= barrett_q(p13);
+ zeta_s2 <= zeta_s1;
+
+ t1_s3 <= barrett_reduce(p20, q20);
+ t2_s3 <= barrett_reduce(p21, q21);
+ t3_s3 <= barrett_reduce(p22, q22);
+ t4_s3 <= barrett_reduce(p23, q23);
+ zeta_s3 <= zeta_s2;
+
+ t1_s4 <= t1_s3;
+ t3_s4 <= t3_s3;
+ t4_s4 <= t4_s3;
+ pz_s4 <= {12'd0, t2_s3} * zeta_s3;
+
+ t1_s5 <= t1_s4;
+ t3_s5 <= t3_s4;
+ t4_s5 <= t4_s4;
+ pz_s5 <= pz_s4;
+ qz_s5 <= barrett_q(pz_s4);
+
+ t1_s6 <= t1_s5;
+ t3_s6 <= t3_s5;
+ t4_s6 <= t4_s5;
+ tz_s6 <= barrett_reduce(pz_s5, qz_s5);
+
+ c0_r <= mod_add(t1_s6, tz_s6);
+ c1_r <= mod_add(t3_s6, t4_s6);
+ end
+ end
+
+endmodule
diff --git a/rtl/ip/mlkem/poly_mul/poly_mul_sync.v b/rtl/ip/mlkem/poly_mul/poly_mul_sync.v
index 3000172..29a28d4 100644
--- a/rtl/ip/mlkem/poly_mul/poly_mul_sync.v
+++ b/rtl/ip/mlkem/poly_mul/poly_mul_sync.v
@@ -4,8 +4,10 @@
// 256-coefficient NTT-domain polynomials.
//
// Operation flow:
-// IDLE → LOAD (256× A+B pairs) → COMP_CALC (read+compute, 1 cycle)
-// → COMP_C0 (output c0) → COMP_C1 (output c1) → DONE → IDLE
+// IDLE -> LOAD (256x A+B pairs) -> RUN -> IDLE
+//
+// RUN issues one base-case multiply every two cycles, matching the two-cycle
+// c0/c1 output bandwidth after the basecase pipeline fills.
//
// The LOAD phase accepts both A and B coefficients simultaneously
// (one pair per cycle) on coeff_a_in/coeff_b_in.
@@ -22,7 +24,7 @@
// valid_i - Input valid
// ready_o - Ready to accept input (high in IDLE/LOAD)
// coeff_out[11:0] - Result coefficient output
-// valid_o - Output valid (high in COMP_C0/COMP_C1)
+// valid_o - Output valid during RUN output phase
// ready_i - Output consumer ready
module poly_mul_sync (
@@ -37,12 +39,9 @@ module poly_mul_sync (
);
// State definitions
- localparam S_IDLE = 3'd0;
- localparam S_LOAD = 3'd1;
- localparam S_COMP_CALC = 3'd2;
- localparam S_COMP_C0 = 3'd3;
- localparam S_COMP_C1 = 3'd4;
- localparam S_DONE = 3'd5;
+ localparam S_IDLE = 3'd0;
+ localparam S_LOAD = 3'd1;
+ localparam S_RUN = 3'd2;
reg [2:0] state, next_state;
@@ -52,14 +51,23 @@ module poly_mul_sync (
// Counters
reg [7:0] load_cnt; // 0..256 for loading 256 pairs
- reg [6:0] comp_k; // 0..127, current base-case index
+ reg [7:0] issue_cnt; // 0..128, issued base-cases
+ reg [7:0] out_cnt; // 0..255, consumed output coefficients
+ reg issue_gap; // spaces base-case requests to match c0/c1 output
+ reg out_phase; // 0=c0, 1=c1
+ reg out_valid_r;
- // Registered basecase_mul results
+ // Registered basecase_mul inputs/results
+ reg [11:0] bc_a0_reg, bc_a1_reg;
+ reg [11:0] bc_b0_reg, bc_b1_reg;
+ reg [11:0] bc_zeta_reg;
+ reg bc_valid_reg;
reg [11:0] c0_reg, c1_reg;
- // Combinational read signals for COMP_CALC
- wire [7:0] addr_even = {comp_k, 1'b0}; // comp_k * 2 (7+1 = 8 bits)
- wire [7:0] addr_odd = {comp_k, 1'b1}; // comp_k * 2 + 1
+ // Combinational read signals for the next base-case request.
+ wire [6:0] issue_k = issue_cnt[6:0];
+ wire [7:0] addr_even = {issue_k, 1'b0}; // issue_k * 2
+ wire [7:0] addr_odd = {issue_k, 1'b1}; // issue_k * 2 + 1
wire [11:0] mem_a0 = mem_A[addr_even];
wire [11:0] mem_a1 = mem_A[addr_odd];
wire [11:0] mem_b0 = mem_B[addr_even];
@@ -68,41 +76,42 @@ module poly_mul_sync (
// Zeta ROM
wire [11:0] zeta;
poly_mul_zeta_rom u_zeta (
- .addr (comp_k),
+ .addr (issue_k),
.zeta (zeta)
);
- // Basecase multiply
+ // Pipelined basecase multiply. Requests are issued every two cycles; inputs
+ // are registered locally so issue_cnt does not directly drive DSP input muxes.
wire [11:0] bc_c0, bc_c1;
- basecase_mul u_bc (
- .a0 (mem_a0),
- .a1 (mem_a1),
- .b0 (mem_b0),
- .b1 (mem_b1),
- .zeta(zeta),
+ wire bc_vo;
+ basecase_mul_pipe u_bc (
+ .clk (clk),
+ .rst_n(rst_n),
+ .valid_i(bc_valid_reg),
+ .a0 (bc_a0_reg),
+ .a1 (bc_a1_reg),
+ .b0 (bc_b0_reg),
+ .b1 (bc_b1_reg),
+ .zeta(bc_zeta_reg),
.c0 (bc_c0),
- .c1 (bc_c1)
+ .c1 (bc_c1),
+ .valid_o(bc_vo)
);
// Output interface
assign ready_o = (state == S_IDLE) || (state == S_LOAD);
- assign valid_o = (state == S_COMP_C0) || (state == S_COMP_C1);
- assign coeff_out = (state == S_COMP_C0) ? c0_reg : c1_reg;
+ assign valid_o = out_valid_r;
+ assign coeff_out = out_phase ? c1_reg : c0_reg;
// State transition logic (combinational)
always @* begin
next_state = state;
case (state)
S_IDLE: if (valid_i && ready_o) next_state = S_LOAD;
- S_LOAD: if (load_cnt >= 255 && valid_i && ready_o)
- next_state = S_COMP_CALC;
- S_COMP_CALC: next_state = S_COMP_C0;
- S_COMP_C0: if (valid_o && ready_i) next_state = S_COMP_C1;
- S_COMP_C1: if (valid_o && ready_i) begin
- if (comp_k >= 127) next_state = S_DONE;
- else next_state = S_COMP_CALC;
- end
- S_DONE: next_state = S_IDLE;
+ S_LOAD: if (load_cnt >= 8'd255 && valid_i && ready_o)
+ next_state = S_RUN;
+ S_RUN: if (valid_o && ready_i && out_cnt == 8'd255)
+ next_state = S_IDLE;
default: next_state = S_IDLE;
endcase
end
@@ -113,15 +122,26 @@ module poly_mul_sync (
if (!rst_n) begin
state <= S_IDLE;
load_cnt <= 8'd0;
- comp_k <= 7'd0;
+ issue_cnt <= 8'd0;
+ out_cnt <= 8'd0;
+ issue_gap <= 1'b0;
+ out_phase <= 1'b0;
+ out_valid_r <= 1'b0;
+ bc_a0_reg <= 12'd0;
+ bc_a1_reg <= 12'd0;
+ bc_b0_reg <= 12'd0;
+ bc_b1_reg <= 12'd0;
+ bc_zeta_reg <= 12'd0;
+ bc_valid_reg <= 1'b0;
c0_reg <= 12'd0;
c1_reg <= 12'd0;
for (i = 0; i < 256; i = i + 1) begin
- mem_A[i] = 12'd0;
- mem_B[i] = 12'd0;
+ mem_A[i] <= 12'd0;
+ mem_B[i] <= 12'd0;
end
end else begin
state <= next_state;
+ bc_valid_reg <= 1'b0;
// ---- LOAD phase ----
// First coefficient captured on IDLE → LOAD transition
@@ -129,6 +149,11 @@ module poly_mul_sync (
mem_A[0] <= coeff_a_in;
mem_B[0] <= coeff_b_in;
load_cnt <= 8'd1;
+ issue_cnt <= 8'd0;
+ out_cnt <= 8'd0;
+ issue_gap <= 1'b0;
+ out_phase <= 1'b0;
+ out_valid_r <= 1'b0;
end
// Subsequent coefficients in LOAD state
@@ -136,29 +161,65 @@ module poly_mul_sync (
mem_A[load_cnt] <= coeff_a_in;
mem_B[load_cnt] <= coeff_b_in;
load_cnt <= load_cnt + 8'd1;
+ if (load_cnt >= 8'd255) begin
+ issue_cnt <= 8'd0;
+ out_cnt <= 8'd0;
+ issue_gap <= 1'b0;
+ out_phase <= 1'b0;
+ out_valid_r <= 1'b0;
+ end
end
// ---- COMPUTE phase ----
- // COMP_CALC: capture basecase_mul results
- if (state == S_COMP_CALC) begin
- c0_reg <= bc_c0;
- c1_reg <= bc_c1;
+ // RUN: cut issue_cnt -> memory mux -> basecase DSP with local input
+ // regs, then launch requests every other cycle. The two-cycle issue
+ // interval matches the c0/c1 output bandwidth, so no deep result FIFO
+ // is needed for this always-ready system.
+ if (state == S_RUN) begin
+ if (issue_cnt < 8'd128) begin
+ if (!issue_gap) begin
+ bc_a0_reg <= mem_a0;
+ bc_a1_reg <= mem_a1;
+ bc_b0_reg <= mem_b0;
+ bc_b1_reg <= mem_b1;
+ bc_zeta_reg <= zeta;
+ bc_valid_reg <= 1'b1;
+ issue_cnt <= issue_cnt + 8'd1;
+ issue_gap <= 1'b1;
+ end else begin
+ issue_gap <= 1'b0;
+ end
+ end
+
+ if (out_valid_r && ready_i) begin
+ out_cnt <= out_cnt + 8'd1;
+ if (out_cnt == 8'd255) begin
+ out_valid_r <= 1'b0;
+ out_phase <= 1'b0;
+ end else if (!out_phase) begin
+ out_phase <= 1'b1;
+ end else begin
+ out_valid_r <= 1'b0;
+ out_phase <= 1'b0;
+ end
+ end
+
+ if (bc_vo) begin
+ c0_reg <= bc_c0;
+ c1_reg <= bc_c1;
+ out_valid_r <= 1'b1;
+ out_phase <= 1'b0;
+ end
end
- // COMP_C0 → COMP_C1: c0 was consumed, increment comp_k
- if (state == S_COMP_C0 && valid_o && ready_i) begin
- // comp_k stays same, c1 still to output
- end
-
- // COMP_C1 → COMP_CALC: c1 was consumed, advance to next pair
- if (state == S_COMP_C1 && valid_o && ready_i) begin
- comp_k <= comp_k + 7'd1;
- end
-
- // ---- DONE → IDLE: reset counters ----
- if (state == S_DONE) begin
+ // ---- final output consumed: reset counters before accepting a new op ----
+ if (state == S_RUN && valid_o && ready_i && out_cnt == 8'd255) begin
load_cnt <= 8'd0;
- comp_k <= 7'd0;
+ issue_cnt <= 8'd0;
+ out_cnt <= 8'd0;
+ issue_gap <= 1'b0;
+ out_phase <= 1'b0;
+ out_valid_r <= 1'b0;
end
end
end
diff --git a/rtl/ip/mlkem/rng/rng_sync.v b/rtl/ip/mlkem/rng/rng_sync.v
deleted file mode 100644
index b4fc49e..0000000
--- a/rtl/ip/mlkem/rng/rng_sync.v
+++ /dev/null
@@ -1,50 +0,0 @@
-// rng_sync.v - 256-bit Galois LFSR PRNG (taps: 255,253,252,247,0)
-module rng_sync #(
- parameter [255:0] SEED = 256'hDEADBEEFCAFEBABEFEEDFACEDECAFBAD1234567887654321ABCDEF010FEDCBA9
-) (
- input clk,
- input rst_n,
- input valid_i,
- output ready_o,
- output [255:0] data_o,
- output valid_o,
- input ready_i
-);
-
- reg [255:0] state;
- reg valid_r;
- reg [255:0] lfsr_next;
- wire feedback;
- integer i;
-
- assign ready_o = 1'b1;
- assign valid_o = valid_r;
- assign data_o = state;
- assign feedback = state[0];
-
- always @(*) begin
- for (i = 0; i < 255; i = i + 1)
- lfsr_next[i] = state[i+1];
- lfsr_next[255] = feedback;
- lfsr_next[254] = lfsr_next[254] ^ feedback;
- lfsr_next[252] = lfsr_next[252] ^ feedback;
- lfsr_next[251] = lfsr_next[251] ^ feedback;
- lfsr_next[246] = lfsr_next[246] ^ feedback;
- end
-
- always @(posedge clk or negedge rst_n) begin
- if (!rst_n) begin
- state <= SEED;
- valid_r <= 1'b0;
- end else begin
- if (valid_r && ready_i)
- valid_r <= 1'b0;
-
- if (valid_i) begin
- state <= lfsr_next;
- valid_r <= 1'b1;
- end
- end
- end
-
-endmodule
diff --git a/rtl/ip/mlkem/sample_ntt/sample_ntt_sync.v b/rtl/ip/mlkem/sample_ntt/sample_ntt_sync.v
index ccd3041..49c7fa8 100644
--- a/rtl/ip/mlkem/sample_ntt/sample_ntt_sync.v
+++ b/rtl/ip/mlkem/sample_ntt/sample_ntt_sync.v
@@ -31,7 +31,6 @@
`include "defines.vh"
/* verilator lint_off UNUSED */
-(* use_dsp = "no" *)
module sample_ntt_sync #(parameter K = 4) (
/* verilator lint_on UNUSED */
input clk,
diff --git a/rtl/ip/mlkem/sample_ntt/sample_ntt_sync_shared.v b/rtl/ip/mlkem/sample_ntt/sample_ntt_sync_shared.v
index cfc0c77..d5ed5cc 100644
--- a/rtl/ip/mlkem/sample_ntt/sample_ntt_sync_shared.v
+++ b/rtl/ip/mlkem/sample_ntt/sample_ntt_sync_shared.v
@@ -44,7 +44,6 @@
`include "defines.vh"
/* verilator lint_off UNUSED */
-(* use_dsp = "no" *)
module sample_ntt_sync_shared #(parameter K = 4) (
/* verilator lint_on UNUSED */
input clk,
diff --git a/rtl/ip/mlkem/sha3/sha3_top.v b/rtl/ip/mlkem/sha3/sha3_top.v
index c829bf3..8d22b47 100644
--- a/rtl/ip/mlkem/sha3/sha3_top.v
+++ b/rtl/ip/mlkem/sha3/sha3_top.v
@@ -100,6 +100,16 @@ module sha3_top (
reg mb_last_r; // captured last-block flag
reg [255:0] mb_digest_r; // latched 256-bit digest (sticky in MB_DONE)
+ // ================================================================
+ // Keccak core
+ // ================================================================
+ wire kc_valid_i;
+ /* verilator lint_off UNUSEDSIGNAL */
+ wire [1599:0] kc_state_o;
+ wire kc_ready_o;
+ /* verilator lint_on UNUSEDSIGNAL */
+ wire kc_valid_o;
+
// XOR the incoming block into the low 1088 bits (rate) of the state.
wire [1599:0] mb_xored;
assign mb_xored = mb_state_r ^ {{(1600-1088){1'b0}}, mb_block_i};
@@ -119,16 +129,6 @@ module sha3_top (
endcase
end
- // ================================================================
- // Keccak core
- // ================================================================
- wire kc_valid_i;
- /* verilator lint_off UNUSED */
- wire [1599:0] kc_state_o;
- wire kc_ready_o;
- /* verilator lint_on UNUSED */
- wire kc_valid_o;
-
// Keccak input: multi-block xored state when mb_en, else single-block absorb.
wire [1599:0] kc_state_i_mux;
assign kc_state_i_mux = mb_en ? mb_xored : absorb_state;
diff --git a/rtl/ip/mlkem/top/fft_mlkem_axi_slave.v b/rtl/ip/mlkem/top/fft_mlkem_axi_slave.v
index 1cb22c1..ada8d7f 100644
--- a/rtl/ip/mlkem/top/fft_mlkem_axi_slave.v
+++ b/rtl/ip/mlkem/top/fft_mlkem_axi_slave.v
@@ -4,13 +4,15 @@
// maps do not overlap:
// ML-KEM: 0x0000..0x3fff
// FFT: 0x8000..0xcfff
-// PRNG: 0xd000..0xd0ff
-// Route the low ML-KEM block to ML-KEM, the high FFT block to FFT, and the
-// following 0xdxxx block to PRNG.
+// Route the low ML-KEM block to ML-KEM and the remaining accelerator window to
+// FFT.
module fft_mlkem_axi_slave (
input clk,
input resetn,
+ input mlkem_clk,
+ input mlkem_resetn,
+
input [4:0] s_axi_awid,
input [31:0] s_axi_awaddr,
input [7:0] s_axi_awlen,
@@ -58,17 +60,12 @@ module fft_mlkem_axi_slave (
localparam SEL_MLKEM = 2'd0;
localparam SEL_FFT = 2'd1;
- localparam SEL_PRNG = 2'd2;
wire aw_to_mlkem = (s_axi_awaddr[15:12] <= 4'h3);
- wire aw_to_prng = (s_axi_awaddr[15:12] == 4'hd);
wire ar_to_mlkem = (s_axi_araddr[15:12] <= 4'h3);
- wire ar_to_prng = (s_axi_araddr[15:12] == 4'hd);
- wire [1:0] aw_sel = aw_to_mlkem ? SEL_MLKEM :
- aw_to_prng ? SEL_PRNG : SEL_FFT;
- wire [1:0] ar_sel = ar_to_mlkem ? SEL_MLKEM :
- ar_to_prng ? SEL_PRNG : SEL_FFT;
+ wire [1:0] aw_sel = aw_to_mlkem ? SEL_MLKEM : SEL_FFT;
+ wire [1:0] ar_sel = ar_to_mlkem ? SEL_MLKEM : SEL_FFT;
reg [1:0] write_sel;
reg write_active;
@@ -83,6 +80,44 @@ module fft_mlkem_axi_slave (
wire [31:0] mlkem_rdata;
wire [1:0] mlkem_rresp;
+ wire mlkem_m_awvalid;
+ wire mlkem_m_awready;
+ wire [31:0] mlkem_m_awaddr;
+ wire [4:0] mlkem_m_awid;
+ wire [7:0] mlkem_m_awlen;
+ wire [2:0] mlkem_m_awsize;
+ wire [1:0] mlkem_m_awburst;
+ wire [0:0] mlkem_m_awlock;
+ wire [3:0] mlkem_m_awcache;
+ wire [2:0] mlkem_m_awprot;
+ wire mlkem_m_wvalid;
+ wire mlkem_m_wready;
+ wire [31:0] mlkem_m_wdata;
+ wire [3:0] mlkem_m_wstrb;
+ wire mlkem_m_wlast;
+ wire mlkem_m_bvalid;
+ wire mlkem_m_bready;
+ wire [4:0] mlkem_m_bid;
+ wire [1:0] mlkem_m_bresp;
+ wire mlkem_m_arvalid;
+ wire mlkem_m_arready;
+ wire [31:0] mlkem_m_araddr;
+ wire [4:0] mlkem_m_arid;
+ wire [7:0] mlkem_m_arlen;
+ wire [2:0] mlkem_m_arsize;
+ wire [1:0] mlkem_m_arburst;
+ wire [0:0] mlkem_m_arlock;
+ wire [3:0] mlkem_m_arcache;
+ wire [2:0] mlkem_m_arprot;
+ wire mlkem_m_rvalid;
+ wire mlkem_m_rready;
+ wire [31:0] mlkem_m_rdata;
+ wire [4:0] mlkem_m_rid;
+ wire [1:0] mlkem_m_rresp;
+ wire mlkem_m_rlast;
+ wire mlkem_intr_m;
+ (* async_reg = "true" *) reg [1:0] mlkem_intr_sync;
+
wire fft_awready, fft_wready, fft_bvalid;
wire [4:0] fft_bid;
wire [1:0] fft_bresp;
@@ -91,52 +126,33 @@ module fft_mlkem_axi_slave (
wire [31:0] fft_rdata;
wire [1:0] fft_rresp;
- wire prng_awready, prng_wready, prng_bvalid;
- wire [4:0] prng_bid;
- wire [1:0] prng_bresp;
- wire prng_arready, prng_rvalid, prng_rlast;
- wire [4:0] prng_rid;
- wire [31:0] prng_rdata;
- wire [1:0] prng_rresp;
-
wire [1:0] sel_w = write_active ? write_sel : aw_sel;
wire [1:0] sel_r = read_active ? read_sel : ar_sel;
assign s_axi_awready = !write_active && (
(aw_sel == SEL_MLKEM) ? mlkem_awready :
- (aw_sel == SEL_PRNG) ? prng_awready :
fft_awready);
assign s_axi_wready = (sel_w == SEL_MLKEM) ? mlkem_wready :
- (sel_w == SEL_PRNG) ? prng_wready :
fft_wready;
assign s_axi_bvalid = (sel_w == SEL_MLKEM) ? mlkem_bvalid :
- (sel_w == SEL_PRNG) ? prng_bvalid :
fft_bvalid;
assign s_axi_bid = (sel_w == SEL_MLKEM) ? mlkem_bid :
- (sel_w == SEL_PRNG) ? prng_bid :
fft_bid;
assign s_axi_bresp = (sel_w == SEL_MLKEM) ? mlkem_bresp :
- (sel_w == SEL_PRNG) ? prng_bresp :
fft_bresp;
assign s_axi_arready = !read_active && (
(ar_sel == SEL_MLKEM) ? mlkem_arready :
- (ar_sel == SEL_PRNG) ? prng_arready :
fft_arready);
assign s_axi_rvalid = (sel_r == SEL_MLKEM) ? mlkem_rvalid :
- (sel_r == SEL_PRNG) ? prng_rvalid :
fft_rvalid;
assign s_axi_rlast = (sel_r == SEL_MLKEM) ? mlkem_rlast :
- (sel_r == SEL_PRNG) ? prng_rlast :
fft_rlast;
assign s_axi_rid = (sel_r == SEL_MLKEM) ? mlkem_rid :
- (sel_r == SEL_PRNG) ? prng_rid :
fft_rid;
assign s_axi_rdata = (sel_r == SEL_MLKEM) ? mlkem_rdata :
- (sel_r == SEL_PRNG) ? prng_rdata :
fft_rdata;
assign s_axi_rresp = (sel_r == SEL_MLKEM) ? mlkem_rresp :
- (sel_r == SEL_PRNG) ? prng_rresp :
fft_rresp;
always @(posedge clk or negedge resetn) begin
@@ -162,52 +178,140 @@ module fft_mlkem_axi_slave (
end
end
+ always @(posedge clk or negedge resetn) begin
+ if (!resetn)
+ mlkem_intr_sync <= 2'b00;
+ else
+ mlkem_intr_sync <= {mlkem_intr_sync[0], mlkem_intr_m};
+ end
+
+ assign mlkem_intr = mlkem_intr_sync[1];
+
+ Axi_CDC u_mlkem_axi_cdc (
+ .axiInClk (clk),
+ .axiInRstn (resetn),
+ .axiOutClk (mlkem_clk),
+ .axiOutRstn (mlkem_resetn),
+
+ .axiIn_awvalid (s_axi_awvalid && !write_active && (aw_sel == SEL_MLKEM)),
+ .axiIn_awready (mlkem_awready),
+ .axiIn_awaddr (s_axi_awaddr),
+ .axiIn_awid (s_axi_awid),
+ .axiIn_awlen (s_axi_awlen),
+ .axiIn_awsize (s_axi_awsize),
+ .axiIn_awburst (s_axi_awburst),
+ .axiIn_awlock (s_axi_awlock),
+ .axiIn_awcache (s_axi_awcache),
+ .axiIn_awprot (s_axi_awprot),
+ .axiIn_wvalid (s_axi_wvalid && (sel_w == SEL_MLKEM)),
+ .axiIn_wready (mlkem_wready),
+ .axiIn_wdata (s_axi_wdata),
+ .axiIn_wstrb (s_axi_wstrb),
+ .axiIn_wlast (s_axi_wlast),
+ .axiIn_bvalid (mlkem_bvalid),
+ .axiIn_bready (s_axi_bready && (sel_w == SEL_MLKEM)),
+ .axiIn_bid (mlkem_bid),
+ .axiIn_bresp (mlkem_bresp),
+ .axiIn_arvalid (s_axi_arvalid && !read_active && (ar_sel == SEL_MLKEM)),
+ .axiIn_arready (mlkem_arready),
+ .axiIn_araddr (s_axi_araddr),
+ .axiIn_arid (s_axi_arid),
+ .axiIn_arlen (s_axi_arlen),
+ .axiIn_arsize (s_axi_arsize),
+ .axiIn_arburst (s_axi_arburst),
+ .axiIn_arlock (s_axi_arlock),
+ .axiIn_arcache (s_axi_arcache),
+ .axiIn_arprot (s_axi_arprot),
+ .axiIn_rvalid (mlkem_rvalid),
+ .axiIn_rready (s_axi_rready && (sel_r == SEL_MLKEM)),
+ .axiIn_rdata (mlkem_rdata),
+ .axiIn_rid (mlkem_rid),
+ .axiIn_rresp (mlkem_rresp),
+ .axiIn_rlast (mlkem_rlast),
+
+ .axiOut_awvalid (mlkem_m_awvalid),
+ .axiOut_awready (mlkem_m_awready),
+ .axiOut_awaddr (mlkem_m_awaddr),
+ .axiOut_awid (mlkem_m_awid),
+ .axiOut_awlen (mlkem_m_awlen),
+ .axiOut_awsize (mlkem_m_awsize),
+ .axiOut_awburst (mlkem_m_awburst),
+ .axiOut_awlock (mlkem_m_awlock),
+ .axiOut_awcache (mlkem_m_awcache),
+ .axiOut_awprot (mlkem_m_awprot),
+ .axiOut_wvalid (mlkem_m_wvalid),
+ .axiOut_wready (mlkem_m_wready),
+ .axiOut_wdata (mlkem_m_wdata),
+ .axiOut_wstrb (mlkem_m_wstrb),
+ .axiOut_wlast (mlkem_m_wlast),
+ .axiOut_bvalid (mlkem_m_bvalid),
+ .axiOut_bready (mlkem_m_bready),
+ .axiOut_bid (mlkem_m_bid),
+ .axiOut_bresp (mlkem_m_bresp),
+ .axiOut_arvalid (mlkem_m_arvalid),
+ .axiOut_arready (mlkem_m_arready),
+ .axiOut_araddr (mlkem_m_araddr),
+ .axiOut_arid (mlkem_m_arid),
+ .axiOut_arlen (mlkem_m_arlen),
+ .axiOut_arsize (mlkem_m_arsize),
+ .axiOut_arburst (mlkem_m_arburst),
+ .axiOut_arlock (mlkem_m_arlock),
+ .axiOut_arcache (mlkem_m_arcache),
+ .axiOut_arprot (mlkem_m_arprot),
+ .axiOut_rvalid (mlkem_m_rvalid),
+ .axiOut_rready (mlkem_m_rready),
+ .axiOut_rdata (mlkem_m_rdata),
+ .axiOut_rid (mlkem_m_rid),
+ .axiOut_rresp (mlkem_m_rresp),
+ .axiOut_rlast (mlkem_m_rlast)
+ );
+
mlkem_axi_slave u_mlkem_axi_slave (
- .clk (clk),
- .resetn (resetn),
+ .clk (mlkem_clk),
+ .resetn (mlkem_resetn),
- .s_axi_awid (s_axi_awid),
- .s_axi_awaddr (s_axi_awaddr),
- .s_axi_awlen (s_axi_awlen),
- .s_axi_awsize (s_axi_awsize),
- .s_axi_awburst (s_axi_awburst),
- .s_axi_awlock (s_axi_awlock),
- .s_axi_awcache (s_axi_awcache),
- .s_axi_awprot (s_axi_awprot),
- .s_axi_awvalid (s_axi_awvalid && !write_active && (aw_sel == SEL_MLKEM)),
- .s_axi_awready (mlkem_awready),
+ .s_axi_awid (mlkem_m_awid),
+ .s_axi_awaddr (mlkem_m_awaddr),
+ .s_axi_awlen (mlkem_m_awlen),
+ .s_axi_awsize (mlkem_m_awsize),
+ .s_axi_awburst (mlkem_m_awburst),
+ .s_axi_awlock (mlkem_m_awlock[0]),
+ .s_axi_awcache (mlkem_m_awcache),
+ .s_axi_awprot (mlkem_m_awprot),
+ .s_axi_awvalid (mlkem_m_awvalid),
+ .s_axi_awready (mlkem_m_awready),
- .s_axi_wid (s_axi_wid),
- .s_axi_wdata (s_axi_wdata),
- .s_axi_wstrb (s_axi_wstrb),
- .s_axi_wlast (s_axi_wlast),
- .s_axi_wvalid (s_axi_wvalid && (sel_w == SEL_MLKEM)),
- .s_axi_wready (mlkem_wready),
+ .s_axi_wid (5'd0),
+ .s_axi_wdata (mlkem_m_wdata),
+ .s_axi_wstrb (mlkem_m_wstrb),
+ .s_axi_wlast (mlkem_m_wlast),
+ .s_axi_wvalid (mlkem_m_wvalid),
+ .s_axi_wready (mlkem_m_wready),
- .s_axi_bid (mlkem_bid),
- .s_axi_bresp (mlkem_bresp),
- .s_axi_bvalid (mlkem_bvalid),
- .s_axi_bready (s_axi_bready && (sel_w == SEL_MLKEM)),
+ .s_axi_bid (mlkem_m_bid),
+ .s_axi_bresp (mlkem_m_bresp),
+ .s_axi_bvalid (mlkem_m_bvalid),
+ .s_axi_bready (mlkem_m_bready),
- .s_axi_arid (s_axi_arid),
- .s_axi_araddr (s_axi_araddr),
- .s_axi_arlen (s_axi_arlen),
- .s_axi_arsize (s_axi_arsize),
- .s_axi_arburst (s_axi_arburst),
- .s_axi_arlock (s_axi_arlock),
- .s_axi_arcache (s_axi_arcache),
- .s_axi_arprot (s_axi_arprot),
- .s_axi_arvalid (s_axi_arvalid && !read_active && (ar_sel == SEL_MLKEM)),
- .s_axi_arready (mlkem_arready),
+ .s_axi_arid (mlkem_m_arid),
+ .s_axi_araddr (mlkem_m_araddr),
+ .s_axi_arlen (mlkem_m_arlen),
+ .s_axi_arsize (mlkem_m_arsize),
+ .s_axi_arburst (mlkem_m_arburst),
+ .s_axi_arlock (mlkem_m_arlock[0]),
+ .s_axi_arcache (mlkem_m_arcache),
+ .s_axi_arprot (mlkem_m_arprot),
+ .s_axi_arvalid (mlkem_m_arvalid),
+ .s_axi_arready (mlkem_m_arready),
- .s_axi_rid (mlkem_rid),
- .s_axi_rdata (mlkem_rdata),
- .s_axi_rresp (mlkem_rresp),
- .s_axi_rlast (mlkem_rlast),
- .s_axi_rvalid (mlkem_rvalid),
- .s_axi_rready (s_axi_rready && (sel_r == SEL_MLKEM)),
+ .s_axi_rid (mlkem_m_rid),
+ .s_axi_rdata (mlkem_m_rdata),
+ .s_axi_rresp (mlkem_m_rresp),
+ .s_axi_rlast (mlkem_m_rlast),
+ .s_axi_rvalid (mlkem_m_rvalid),
+ .s_axi_rready (mlkem_m_rready),
- .intr (mlkem_intr)
+ .intr (mlkem_intr_m)
);
fft_controller u_fft_controller (
@@ -258,50 +362,4 @@ module fft_mlkem_axi_slave (
.fft_finish (fft_finish)
);
- prng_axi_slave u_prng_axi_slave (
- .clk (clk),
- .resetn (resetn),
-
- .s_axi_awid (s_axi_awid),
- .s_axi_awaddr (s_axi_awaddr),
- .s_axi_awlen (s_axi_awlen),
- .s_axi_awsize (s_axi_awsize),
- .s_axi_awburst (s_axi_awburst),
- .s_axi_awlock (s_axi_awlock),
- .s_axi_awcache (s_axi_awcache),
- .s_axi_awprot (s_axi_awprot),
- .s_axi_awvalid (s_axi_awvalid && !write_active && (aw_sel == SEL_PRNG)),
- .s_axi_awready (prng_awready),
-
- .s_axi_wid (s_axi_wid),
- .s_axi_wdata (s_axi_wdata),
- .s_axi_wstrb (s_axi_wstrb),
- .s_axi_wlast (s_axi_wlast),
- .s_axi_wvalid (s_axi_wvalid && (sel_w == SEL_PRNG)),
- .s_axi_wready (prng_wready),
-
- .s_axi_bid (prng_bid),
- .s_axi_bresp (prng_bresp),
- .s_axi_bvalid (prng_bvalid),
- .s_axi_bready (s_axi_bready && (sel_w == SEL_PRNG)),
-
- .s_axi_arid (s_axi_arid),
- .s_axi_araddr (s_axi_araddr),
- .s_axi_arlen (s_axi_arlen),
- .s_axi_arsize (s_axi_arsize),
- .s_axi_arburst (s_axi_arburst),
- .s_axi_arlock (s_axi_arlock),
- .s_axi_arcache (s_axi_arcache),
- .s_axi_arprot (s_axi_arprot),
- .s_axi_arvalid (s_axi_arvalid && !read_active && (ar_sel == SEL_PRNG)),
- .s_axi_arready (prng_arready),
-
- .s_axi_rid (prng_rid),
- .s_axi_rdata (prng_rdata),
- .s_axi_rresp (prng_rresp),
- .s_axi_rlast (prng_rlast),
- .s_axi_rvalid (prng_rvalid),
- .s_axi_rready (s_axi_rready && (sel_r == SEL_PRNG))
- );
-
endmodule
diff --git a/rtl/ip/mlkem/top/mlkem_axi_slave.v b/rtl/ip/mlkem/top/mlkem_axi_slave.v
index 635b2da..64e4d62 100644
--- a/rtl/ip/mlkem/top/mlkem_axi_slave.v
+++ b/rtl/ip/mlkem/top/mlkem_axi_slave.v
@@ -1,16 +1,13 @@
// mlkem_axi_slave.v — AXI4 (burst) slave wrapper for the ML-KEM core.
// Makes the ML-KEM accelerator (KeyGen / Encaps / Decaps) CPU-controllable.
//
-// The bus runs at sys_clk (50 MHz). The ML-KEM core cannot close timing at
-// 50 MHz, so it is clocked by mlkem_clk = sys_clk / 2 (25 MHz), produced by a
-// toggle flip-flop in this wrapper. mlkem_clk is a phase-aligned synchronous
-// divide of sys_clk; every signal crossing between the two domains still goes
-// through an explicit synchronizer or toggle handshake.
-// NOTE (FPGA): for real silicon route mlkem_clk through a BUFG/BUFGCE rather
-// than a logic-generated net; kept as a plain toggle here for portability.
+// This slave is placed behind an AXI clock-domain converter. Its AXI side and
+// mlkem_top both run in mlkem_clk; sys_clk masters reach it through Axi_CDC in
+// fft_mlkem_axi_slave.
//
// Register map (byte offsets from base, e.g. virtual 0xbf400000):
// 0x000 CTRL W bit0 = start (launches op selected in CONFIG)
+// bit1 = soft_reset (resets ML-KEM core state)
// 0x004 STATUS R bit0 busy, bit1 done, bit2 preload_busy
// W1C write bit1=1 to clear the latched done flag
// 0x008 CONFIG RW [2:0] k (2/3/4), [5:4] op (0=KeyGen 1=Encaps 2=Decaps)
@@ -27,9 +24,11 @@
// 0x1000..0x1FFF EK W: preload ek_in, R: read generated ek
// 0x2000..0x2FFF DK W: preload dk_in, R: read generated full dk
// 0x3000..0x3FFF CT W: preload c_in, R: read generated ciphertext
+
+(* use_dsp = "no" *)
module mlkem_axi_slave (
- input clk, // sys_clk (AXI bus clock, 50 MHz)
- input resetn, // sys_resetn (active low, already synchronized)
+ input clk, // mlkem_clk AXI-side clock after Axi_CDC
+ input resetn, // mlkem_resetn, active low
// AXI Write Address Channel
input [4:0] s_axi_awid,
@@ -153,20 +152,7 @@ module mlkem_axi_slave (
endfunction
// ==========================================
- // mlkem_clk = sys_clk / 2 (25 MHz) via toggle FF.
- // ==========================================
- reg mlkem_clk;
- always @(posedge clk or negedge resetn) begin
- if (!resetn) mlkem_clk <= 1'b0;
- else mlkem_clk <= ~mlkem_clk;
- end
- // Reset for the mlkem domain: reuse sys_resetn (async-assert, and the core's
- // own rst_n is sampled on mlkem_clk edges).
- wire mlkem_rstn = resetn;
-
- // ==========================================
- // sys_clk-domain config / data registers (quasi-static; fed to the core
- // combinationally — written long before start, stable across the op).
+ // AXI/core clock-domain config / data registers.
// ==========================================
reg [2:0] cfg_k; // 2/3/4
reg [1:0] cfg_op; // 0=KeyGen 1=Encaps 2=Decaps
@@ -188,55 +174,79 @@ module mlkem_axi_slave (
reg [11:0] rdb_addr; // output byte index
// ==========================================
- // CDC: start pulse (sys toggle -> mlkem edge-detect -> 1-cyc start_i)
+ // Start pulse: CTRL.start flips a toggle, edge-detected into start_i.
// ==========================================
reg start_tg; // sys-domain toggle, flips on CTRL.start write
reg [2:0] start_sync; // mlkem-domain 2FF sync (+1 for edge)
- wire mlkem_start = start_sync[2] ^ start_sync[1]; // 1 mlkem_clk pulse
+ wire mlkem_start = start_sync[2] ^ start_sync[1]; // 1 clk pulse
// ==========================================
- // CDC: busy / done back to sys
+ // Software reset pulse: CTRL.soft_reset flips a toggle and stretches reset.
// ==========================================
- wire core_busy; // busy_o (mlkem domain)
- wire core_done; // done_o pulse (mlkem domain)
- reg done_latch; // sticky done in mlkem domain
- reg [1:0] busy_sync; // -> sys
- reg [1:0] done_sync; // -> sys
- // done-clear: sys toggle -> mlkem edge -> clears done_latch
- reg dclr_tg; // sys-domain toggle (STATUS W1C of done)
- reg [2:0] dclr_sync; // mlkem-domain 2FF sync (+1 for edge)
+ reg srst_tg; // flips on CTRL.soft_reset
+ reg [2:0] srst_sync; // edge detector shift register
+ reg [3:0] srst_cnt; // reset stretcher in clk domain
+ wire mlkem_srst_edge = srst_sync[2] ^ srst_sync[1];
+ wire mlkem_srst_active = (srst_cnt != 4'd0);
+ wire core_resetn = resetn && !mlkem_srst_active;
+
+ // ==========================================
+ // Busy / done status.
+ // ==========================================
+ wire core_busy;
+ wire core_done;
+ reg done_latch;
+ reg [1:0] busy_sync;
+ reg [1:0] done_sync;
+ // done-clear: STATUS W1C flips a toggle, edge-detected to clear done_latch.
+ reg dclr_tg;
+ reg [2:0] dclr_sync;
wire mlkem_dclr = dclr_sync[2] ^ dclr_sync[1];
// ==========================================
- // CDC: preload byte write handshake.
- // sys: latch {sel,addr,byte}, flip pre_req_tg, hold wready low.
- // mlkem: 2FF sync pre_req_tg, on edge pulse the selected *_in_we 1 cycle,
- // flip pre_ack_tg.
- // sys: 2FF sync pre_ack_tg, on edge release the bus (addr++).
+ // Preload byte write handshake.
+ // Latch {sel,addr,byte}, flip pre_req_tg, pulse the selected *_in_we for
+ // one cycle, then flip pre_ack_tg to release the AXI write beat.
// ==========================================
- reg pre_req_tg; // sys toggle: a byte is pending
- reg [2:0] pre_req_sync; // mlkem 2FF (+1 for edge)
+ reg pre_req_tg; // a byte is pending
+ reg [2:0] pre_req_sync; // edge detector shift register
wire mlkem_pre_we = pre_req_sync[2] ^ pre_req_sync[1];
- reg pre_ack_tg; // mlkem toggle: byte consumed
- reg [2:0] pre_ack_sync; // sys 2FF (+1 for edge)
+ reg pre_ack_tg; // byte consumed
+ reg [2:0] pre_ack_sync; // edge detector shift register
wire pre_ack_edge = pre_ack_sync[2] ^ pre_ack_sync[1];
- reg pre_busy; // sys: a preload handshake is in flight
- // latched byte/port for the in-flight preload (sys, stable while pre_busy)
+ reg pre_busy; // a preload handshake is in flight
+ // latched byte/port for the in-flight preload, stable while pre_busy
reg [1:0] pre_sel_l;
reg [11:0] pre_addr_l;
reg [7:0] pre_byte_l;
reg pre_auto_inc_l;
+ // ==========================================
+ // Bulk readback byte handshake.
+ // Hold {sel,addr}, flip rdb_req_tg, let the BRAM read settle, latch one
+ // byte, then flip rdb_ack_tg.
+ // ==========================================
+ reg rdb_req_tg; // one readback byte request
+ reg [2:0] rdb_req_sync; // edge detector shift register
+ wire mlkem_rdb_req = rdb_req_sync[2] ^ rdb_req_sync[1];
+ reg rdb_ack_tg; // readback byte ready
+ reg [2:0] rdb_ack_sync; // edge detector shift register
+ wire rdb_ack_edge = rdb_ack_sync[2] ^ rdb_ack_sync[1];
+ reg [1:0] rdb_req_sel_l; // stable while request in flight
+ reg [11:0] rdb_req_addr_l; // stable while request in flight
+ reg [1:0] rdb_sel_m; // captured request
+ reg [11:0] rdb_addr_m; // captured request
+ reg [1:0] rdb_wait_m; // waits for BRAM registered read
+ reg rdb_busy_m;
+ reg [7:0] rdb_data_m; // held stable until next ack
+
// ==========================================
// mlkem_top instance
// ==========================================
wire [7:0] dbg_byte_o, dbg_dk_o, dbg_ct_o;
wire [255:0] ss_o;
- wire bulk_read_active;
- wire [1:0] bulk_rdb_sel;
- wire [11:0] bulk_rdb_addr;
- wire [1:0] eff_rdb_sel = bulk_read_active ? bulk_rdb_sel : rdb_sel;
- wire [11:0] eff_rdb_addr = bulk_read_active ? bulk_rdb_addr : rdb_addr;
+ wire [1:0] eff_rdb_sel = rdb_busy_m ? rdb_sel_m : rdb_sel;
+ wire [11:0] eff_rdb_addr = rdb_busy_m ? rdb_addr_m : rdb_addr;
wire dbg_byte_sel = (eff_rdb_sel == 2'd1); // 0=ek, 1=dk_pke
// The core's ek/dkp read-address muxes give dbg_dk_idx_i priority over
// dbg_byte_idx_i whenever it lands in the ek (768..1567) or dk_pke (<768)
@@ -245,21 +255,20 @@ module mlkem_axi_slave (
// it out of range (0xFFF) so an ek/dk_pke readback isn't corrupted.
wire [11:0] dbg_dk_idx = (eff_rdb_sel == 2'd2) ? eff_rdb_addr : 12'hFFF;
- // one-cycle write-enable pulses into the core (combinational off the synced
- // edge; pre_sel_l/pre_addr_l/pre_byte_l are stable in sys, sampled by core).
- wire ek_in_we = mlkem_pre_we && (pre_sel_l == 2'd0);
- wire dk_in_we = mlkem_pre_we && (pre_sel_l == 2'd1);
- wire c_in_we = mlkem_pre_we && (pre_sel_l == 2'd2);
+ // one-cycle write-enable pulses into the core.
+ wire ek_in_we = core_resetn && mlkem_pre_we && (pre_sel_l == 2'd0);
+ wire dk_in_we = core_resetn && mlkem_pre_we && (pre_sel_l == 2'd1);
+ wire c_in_we = core_resetn && mlkem_pre_we && (pre_sel_l == 2'd2);
mlkem_top #(.KMAX(4)) u_mlkem (
- .clk (mlkem_clk),
- .rst_n (mlkem_rstn),
+ .clk (clk),
+ .rst_n (core_resetn),
.k_i (cfg_k),
.op_i (cfg_op),
.d_i (d_bus),
.z_i (z_bus),
.msg_i (msg_bus),
- .start_i (mlkem_start),
+ .start_i (core_resetn ? mlkem_start : 1'b0),
.busy_o (core_busy),
.done_o (core_done),
// ek preload
@@ -302,44 +311,82 @@ module mlkem_axi_slave (
);
// ==========================================
- // mlkem_clk domain: synchronizers, done latch, preload-ack toggle.
+ // Core/AXI clock domain: edge detectors, done latch, preload/readback acks.
// ==========================================
- always @(posedge mlkem_clk or negedge mlkem_rstn) begin
- if (!mlkem_rstn) begin
+ always @(posedge clk or negedge resetn) begin
+ if (!resetn) begin
start_sync <= 3'd0;
+ srst_sync <= 3'd0;
+ srst_cnt <= 4'd0;
dclr_sync <= 3'd0;
pre_req_sync <= 3'd0;
pre_ack_tg <= 1'b0;
+ rdb_req_sync <= 3'd0;
+ rdb_ack_tg <= 1'b0;
+ rdb_sel_m <= 2'd0;
+ rdb_addr_m <= 12'd0;
+ rdb_wait_m <= 2'd0;
+ rdb_busy_m <= 1'b0;
+ rdb_data_m <= 8'd0;
done_latch <= 1'b0;
end else begin
- // 2FF (+edge) synchronizers of sys-domain toggles
+ // Edge-detect request toggles.
start_sync <= {start_sync[1:0], start_tg};
+ srst_sync <= {srst_sync[1:0], srst_tg};
dclr_sync <= {dclr_sync[1:0], dclr_tg};
pre_req_sync <= {pre_req_sync[1:0], pre_req_tg};
+ rdb_req_sync <= {rdb_req_sync[1:0], rdb_req_tg};
+
+ if (mlkem_srst_edge)
+ srst_cnt <= 4'd8;
+ else if (srst_cnt != 4'd0)
+ srst_cnt <= srst_cnt - 4'd1;
// acknowledge a consumed preload byte (write-enable pulsed this cycle)
- if (mlkem_pre_we) pre_ack_tg <= ~pre_ack_tg;
+ if (core_resetn && mlkem_pre_we) pre_ack_tg <= ~pre_ack_tg;
+
+ if (!core_resetn) begin
+ rdb_wait_m <= 2'd0;
+ rdb_busy_m <= 1'b0;
+ end else if (mlkem_rdb_req) begin
+ rdb_sel_m <= rdb_req_sel_l;
+ rdb_addr_m <= rdb_req_addr_l;
+ rdb_wait_m <= 2'd2;
+ rdb_busy_m <= 1'b1;
+ end else if (rdb_wait_m != 2'd0) begin
+ if (rdb_wait_m == 2'd1) begin
+ rdb_data_m <= rdb_byte;
+ rdb_ack_tg <= ~rdb_ack_tg;
+ rdb_wait_m <= 2'd0;
+ rdb_busy_m <= 1'b0;
+ end else begin
+ rdb_wait_m <= rdb_wait_m - 2'd1;
+ end
+ end
// sticky done: set on the core's done_o pulse, cleared by synced W1C.
// (mlkem_start also clears it so a new run starts with done low.)
- if (core_done) done_latch <= 1'b1;
+ if (mlkem_srst_active) done_latch <= 1'b0;
+ else if (core_done) done_latch <= 1'b1;
else if (mlkem_dclr) done_latch <= 1'b0;
else if (mlkem_start) done_latch <= 1'b0;
end
end
// ==========================================
- // sys_clk domain: sync busy/done/preload-ack back from mlkem.
+ // AXI clock domain: register status and acknowledge toggles.
// ==========================================
always @(posedge clk or negedge resetn) begin
if (!resetn) begin
busy_sync <= 2'd0;
done_sync <= 2'd0;
pre_ack_sync <= 3'd0;
+ rdb_ack_sync <= 3'd0;
end else begin
- busy_sync <= {busy_sync[0], core_busy};
+ busy_sync <= {busy_sync[0], core_busy | mlkem_srst_active};
done_sync <= {done_sync[0], done_latch};
pre_ack_sync <= {pre_ack_sync[1:0], pre_ack_tg};
+ rdb_ack_sync <= {rdb_ack_sync[1:0], rdb_ack_tg};
end
end
wire status_busy = busy_sync[1];
@@ -389,6 +436,7 @@ module mlkem_axi_slave (
rdb_sel <= 2'd0;
rdb_addr <= 12'd0;
start_tg <= 1'b0;
+ srst_tg <= 1'b0;
dclr_tg <= 1'b0;
pre_req_tg <= 1'b0;
pre_busy <= 1'b0;
@@ -453,8 +501,10 @@ module mlkem_axi_slave (
end else begin
// normal single-cycle register write
case (1'b1)
- (w_offset == ADDR_CTRL):
+ (w_offset == ADDR_CTRL): begin
if (s_axi_wdata[0]) start_tg <= ~start_tg;
+ if (s_axi_wdata[1]) srst_tg <= ~srst_tg;
+ end
(w_offset == ADDR_STATUS):
if (s_axi_wdata[1]) dclr_tg <= ~dclr_tg; // clear done
(w_offset == ADDR_CONFIG): begin
@@ -541,7 +591,6 @@ module mlkem_axi_slave (
localparam R_IDLE = 3'd0;
localparam R_DATA = 3'd1;
localparam R_BULK_WAIT = 3'd2;
- localparam RDB_WAIT_CYCLES = 3'd4;
reg [2:0] r_state;
reg [31:0] r_addr;
@@ -557,13 +606,8 @@ module mlkem_axi_slave (
reg [1:0] bulk_sel_r;
reg [11:0] bulk_addr_r;
reg [1:0] bulk_lane;
- reg [2:0] bulk_wait;
reg [31:0] bulk_word;
- assign bulk_read_active = r_bulk && (r_state == R_BULK_WAIT);
- assign bulk_rdb_sel = bulk_sel_r;
- assign bulk_rdb_addr = bulk_addr_r;
-
always @(posedge clk or negedge resetn) begin
if (!resetn) begin
r_state <= R_IDLE;
@@ -579,8 +623,10 @@ module mlkem_axi_slave (
bulk_sel_r <= 2'd0;
bulk_addr_r <= 12'd0;
bulk_lane <= 2'd0;
- bulk_wait <= 3'd0;
bulk_word <= 32'd0;
+ rdb_req_tg <= 1'b0;
+ rdb_req_sel_l <= 2'd0;
+ rdb_req_addr_l <= 12'd0;
end else begin
case (r_state)
R_IDLE: begin
@@ -597,8 +643,10 @@ module mlkem_axi_slave (
bulk_sel_r <= window_readback_sel(s_axi_araddr[15:12]);
bulk_addr_r <= s_axi_araddr[11:0];
bulk_lane <= 2'd0;
- bulk_wait <= RDB_WAIT_CYCLES;
bulk_word <= 32'd0;
+ rdb_req_sel_l <= window_readback_sel(s_axi_araddr[15:12]);
+ rdb_req_addr_l <= s_axi_araddr[11:0];
+ rdb_req_tg <= ~rdb_req_tg;
s_axi_rvalid <= 1'b0;
s_axi_rlast <= 1'b0;
r_state <= R_BULK_WAIT;
@@ -610,14 +658,12 @@ module mlkem_axi_slave (
end
end
R_BULK_WAIT: begin
- if (bulk_wait != 3'd0) begin
- bulk_wait <= bulk_wait - 3'd1;
- end else begin
+ if (rdb_ack_edge) begin
case (bulk_lane)
- 2'd0: bulk_word[7:0] <= rdb_byte;
- 2'd1: bulk_word[15:8] <= rdb_byte;
- 2'd2: bulk_word[23:16] <= rdb_byte;
- default: bulk_word[31:24] <= rdb_byte;
+ 2'd0: bulk_word[7:0] <= rdb_data_m;
+ 2'd1: bulk_word[15:8] <= rdb_data_m;
+ 2'd2: bulk_word[23:16] <= rdb_data_m;
+ default: bulk_word[31:24] <= rdb_data_m;
endcase
if (bulk_lane == 2'd3) begin
@@ -627,7 +673,8 @@ module mlkem_axi_slave (
end else begin
bulk_lane <= bulk_lane + 2'd1;
bulk_addr_r <= bulk_addr_r + 12'd1;
- bulk_wait <= RDB_WAIT_CYCLES;
+ rdb_req_addr_l <= bulk_addr_r + 12'd1;
+ rdb_req_tg <= ~rdb_req_tg;
end
end
end
@@ -645,8 +692,10 @@ module mlkem_axi_slave (
if (r_bulk) begin
bulk_addr_r <= (r_burst == 2'b01) ? (r_offset + 12'd4) : r_offset;
bulk_lane <= 2'd0;
- bulk_wait <= RDB_WAIT_CYCLES;
bulk_word <= 32'd0;
+ rdb_req_sel_l <= bulk_sel_r;
+ rdb_req_addr_l <= (r_burst == 2'b01) ? (r_offset + 12'd4) : r_offset;
+ rdb_req_tg <= ~rdb_req_tg;
s_axi_rvalid <= 1'b0;
s_axi_rlast <= 1'b0;
r_state <= R_BULK_WAIT;
diff --git a/rtl/ip/mlkem/top/mlkem_top.v b/rtl/ip/mlkem/top/mlkem_top.v
index dc5c0ec..7003cba 100644
--- a/rtl/ip/mlkem/top/mlkem_top.v
+++ b/rtl/ip/mlkem/top/mlkem_top.v
@@ -1,180 +1,181 @@
-// mlkem_top.v - ML-KEM 密钥生成顶层集成。通过 k_i 运行时选择参数集：
-// k=2 (ML-KEM-512, eta1=3), k=3 (768), k=4 (1024)。
-// 存储器按 KMAX（最坏情况 = ML-KEM-1024）大小设计；k_i 在 start_i 时
-// 选择有效子范围。
+// mlkem_top.v - ML-KEM KeyGen top-level integration. Runtime-selectable
+// parameter set via k_i: k=2 (ML-KEM-512, eta1=3), k=3 (768), k=4 (1024).
+// Storage is sized for KMAX (worst case = ML-KEM-1024); k_i picks the
+// active sub-range at start_i.
//
-// 流式 valid/ready 接口。给定种子 d 和 z，根据 FIPS 203 算法 16
-// (KeyGen_internal) 计算 ML-KEM 密钥对：
+// Streaming valid/ready interface. Given seeds d and z, computes the
+// ML-KEM key pair per FIPS 203 Algorithm 16 (KeyGen_internal):
// (rho,sigma) = G(d || K)
-// A_hat[i][j] = SampleNTT(rho || j || i) i,j 在 0..K-1 范围内
+// A_hat[i][j] = SampleNTT(rho || j || i) i,j in 0..K-1
// s[i] = CBD3(PRF(sigma, i)), e[i] = CBD3(PRF(sigma, K+i))
// s_hat[i] = NTT(s[i]); e_hat[i] = NTT(e[i])
// t_hat[i] = e_hat[i] + sum_j A_hat[i][j] o s_hat[j]
// ek = byteEncode12(t_hat[0..K-1]) || rho
// dk = byteEncode12(s_hat[0..K-1]) || ek || H(ek) || z
//
-// 按阶段增量构建，并针对参考黄金向量
-// (test_framework/modules/mlkem_keygen/golden) 和 NIST KAT 进行逐阶段验证。
+// Built incrementally and verified stage-by-stage against reference golden
+// vectors (test_framework/modules/mlkem_keygen/golden) and NIST KAT.
//
-// 使用独立（已验证）的叶子模块，共享同一个 keccak_core：
+// Uses independent (verified) leaf modules over a shared keccak_core:
// sha3_top_shared, sample_ntt_sync_shared, sample_cbd_sync_shared, ntt_core,
-// poly_mul_sync, comp_decomp_sync。
+// poly_mul_sync, comp_decomp_sync.
`include "defines.vh"
(* use_dsp = "no" *)
module mlkem_top #(
- parameter KMAX = 4 // 存储器大小（最坏情况 = ML-KEM-1024）
+ parameter KMAX = 4 // storage sizing (worst case = ML-KEM-1024)
) (
input clk,
input rst_n,
- input [2:0] k_i, // 运行时 ML-KEM 参数：2=512, 3=768, 4=1024
- input [1:0] op_i, // 0=密钥生成, 1=封装, 2=解封装（在 start_i 时捕获）
- input [255:0] d_i, // 密钥生成种子 d（字节 0 在 d_i[7:0] 中）
- input [255:0] z_i, // 隐式拒绝种子 z
- input [255:0] msg_i, // 封装消息 m（字节 0 在 msg_i[7:0] 中）
- input start_i, // 开始密钥生成的脉冲
- output busy_o, // 运行时为高电平
- output done_o, // ek/dk 就绪时输出脉冲
+ input [2:0] k_i, // RUNTIME ML-KEM param: 2=512, 3=768, 4=1024
+ input [1:0] op_i, // 0=KeyGen, 1=Encaps, 2=Decaps (captured at start_i)
+ input [255:0] d_i, // KeyGen seed d (byte 0 in d_i[7:0])
+ input [255:0] z_i, // implicit-rejection seed z
+ input [255:0] msg_i, // Encaps message m (byte 0 in msg_i[7:0])
+ input start_i, // pulse to begin KeyGen
+ output busy_o, // high while running
+ output done_o, // pulse when ek/dk ready
- // Encaps ek 输入：在 start_i 之前以 op_i=1 将 ek 字节流式写入 ek_bram。
- // ek_in_we 将字节 ek_in_byte 写入地址 ek_in_addr。
- input ek_in_we, // 写入一个 ek 字节（仅在 ST_IDLE 中 Encaps 预加载时有效）
- input [10:0] ek_in_addr, // ek 字节地址 0..ek_bytes-1
- input [7:0] ek_in_byte, // ek 字节值
- // Decaps dk 输入：在 start_i 之前以 op_i=2 将 dk 字节（=sk, 768K+96）
- // 流式写入设计。加载逻辑按区域路由每个字节：
- // [0,384K) -> dk_pke (dkp_bram) = s_hat 编码
- // [384K, 768K+32) -> ek_pke (ek_bram) = t_hat 编码 || rho
+ // Encaps ek input: stream ek bytes (=pk) into ek_bram before start_i with
+ // op_i=1. ek_in_we writes byte ek_in_byte at address ek_in_addr.
+ input ek_in_we, // write one ek byte (only honored in ST_IDLE for Encaps preload)
+ input [10:0] ek_in_addr, // ek byte address 0..ek_bytes-1
+ input [7:0] ek_in_byte, // ek byte value
+ // Decaps dk input: stream dk bytes (=sk, 768K+96) into the design before
+ // start_i with op_i=2. The load logic routes each byte by region:
+ // [0,384K) -> dk_pke (dkp_bram) = s_hat encoding
+ // [384K, 768K+32) -> ek_pke (ek_bram) = t_hat encoding || rho
// [768K+32, 768K+64) -> H(ek) (hek_r)
// [768K+64, 768K+96) -> z (z_r)
- input dk_in_we, // 写入一个 dk 字节（ST_IDLE Decaps 预加载）
- input [11:0] dk_in_addr, // dk 字节地址 0..(768K+96)-1
- input [7:0] dk_in_byte, // dk 字节值
- // Decaps 密文输入：在 start_i 之前将 c 字节（=ct）流式写入 c_in_bram。
- // 与 ct_bram 分开保存，这样计算出的 c' 可以与原始 c 比较
- // （且 J(z||c) 可以读取 c）而不会产生别名。
- input c_in_we, // 写入一个 c 字节（ST_IDLE Decaps 预加载）
- input [10:0] c_in_addr, // c 字节地址 0..ct_bytes-1
- input [7:0] c_in_byte, // c 字节值
- // Encaps 共享密钥输出（= K），在 done_o 时有效
+ input dk_in_we, // write one dk byte (ST_IDLE Decaps preload)
+ input [11:0] dk_in_addr, // dk byte address 0..(768K+96)-1
+ input [7:0] dk_in_byte, // dk byte value
+ // Decaps ciphertext input: stream c bytes (=ct) into c_in_bram before
+ // start_i. Kept separate from ct_bram so the computed c' can be compared
+ // against the original c (and J(z||c) can read c) without aliasing.
+ input c_in_we, // write one c byte (ST_IDLE Decaps preload)
+ input [10:0] c_in_addr, // c byte address 0..ct_bytes-1
+ input [7:0] c_in_byte, // c byte value
+ // Encaps shared secret output (= K), valid at done_o
output [255:0] ss_o,
- // Encaps 密文回读抽头：按索引 0..ct_bytes-1 输出 ct 字节
+ // Encaps ciphertext readback tap: ct byte by index 0..ct_bytes-1
input [10:0] dbg_ct_idx_i,
output [7:0] dbg_ct_o,
- // Debug 回读抽头：按（多项式槽位，索引）读取一个存储系数。
- // 允许各阶段测试平台验证中间结果，无需宽总线。
- input [5:0] dbg_slot_i, // 多项式槽位（见下面的 localparam）
- input [7:0] dbg_idx_i, // 系数索引 0..255
+ // Debug readback tap: read one stored coefficient by (poly slot, index).
+ // Lets stage TBs verify intermediates without wide buses.
+ input [5:0] dbg_slot_i, // poly slot (see localparams below)
+ input [7:0] dbg_idx_i, // coefficient index 0..255
output [11:0] dbg_coeff_o,
- // Debug 字节回读：ek (sel=0, 0..799) / dk_pke (sel=1, 0..767)
+ // Debug byte readback: ek (sel=0, 0..799) / dk_pke (sel=1, 0..767)
input dbg_byte_sel_i,
input [10:0] dbg_byte_idx_i,
output [7:0] dbg_byte_o,
- // Debug 完整 dk 回读：dk = dk_pke(768) || ek(800) || H(ek)(32) || z(32)
- // = 1632 字节。索引 0..1631。
+ // Debug full-dk readback: dk = dk_pke(768) || ek(800) || H(ek)(32) || z(32)
+ // = 1632 bytes. Index 0..1631.
input [11:0] dbg_dk_idx_i,
output [7:0] dbg_dk_o,
- // Debug 哈希输出抽头
+ // Debug taps for hash outputs
output [255:0] dbg_rho_o,
output [255:0] dbg_sigma_o,
- // Encaps debug 抽头：r（G 高半部分）和 H(ek)
+ // Encaps debug taps: r (G high half) and H(ek)
output [255:0] dbg_r_o,
output [255:0] dbg_hek_o,
- // Decaps debug 抽头：m'（解密输出）、K-bar（J 输出）、解析后的 z/h。
+ // Decaps debug taps: m' (Decrypt output), K-bar (J output), parsed z/h.
output [255:0] dbg_mprime_o,
output [255:0] dbg_kbar_o,
- output [255:0] dbg_decz_o, // 解析后的 z（dk[768K+64:+32]）
- output [255:0] dbg_dech_o // 解析后的 H(ek)（dk[768K+32:+32]）
+ output [255:0] dbg_decz_o, // parsed z (dk[768K+64:+32])
+ output [255:0] dbg_dech_o // parsed H(ek) (dk[768K+32:+32])
);
localparam Q = `Q; // 3329
+ localparam [11:0] Q12 = `Q;
+ localparam [12:0] Q13 = `Q;
+ localparam [13:0] Q14 = `Q;
- // 运行时 ML-KEM 参数，在 start_i 时捕获。
+ // Runtime ML-KEM parameter, captured at start_i.
reg [2:0] k_r;
- // FIPS 203：ML-KEM-512 (k=2) 时 eta1 = 3，k=3/4 时 eta1 = 2。
+ // FIPS 203: eta1 = 3 for ML-KEM-512 (k=2), else 2 (k=3/4).
wire [1:0] eta1_rt = (k_r == 3'd2) ? 2'd3 : 2'd2;
- // 运行时推导的大小（k_r 取值为 {2,3,4}）。小乘法成本低廉。
+ // Runtime-derived sizes (k_r in {2,3,4}). Small multiplies are cheap.
wire [5:0] kk_rt = k_r * k_r; // 4/9/16
- wire [5:0] slot_s_rt = kk_rt; // s_hat 基槽位
- wire [5:0] slot_e_rt = kk_rt + k_r; // e_hat 基槽位
- wire [5:0] slot_t_rt = kk_rt + {1'b0, k_r} + {1'b0, k_r}; // t_hat 基槽位 = kk+2k
+ wire [5:0] slot_s_rt = kk_rt; // s_hat base slot
+ wire [5:0] slot_e_rt = kk_rt + k_r; // e_hat base slot
+ wire [5:0] slot_t_rt = kk_rt + {1'b0, k_r} + {1'b0, k_r}; // t_hat base = kk+2k
wire [11:0] ek_bytes_rt = 12'd384 * {9'b0, k_r} + 12'd32; // 800/1184/1568
wire [11:0] dk_bytes_rt = 12'd384 * {9'b0, k_r}; // 768/1152/1536
- // 完整 dk = dk_pke(dk_bytes) || ek(ek_bytes) || H(ek)(32) || z(32)。区域
- // 边界（用于 Decaps dk 加载路由和 dk 回读）。
- wire [11:0] dk_ek_end = dk_bytes_rt + ek_bytes_rt; // ek_pke 区域结束
- wire [11:0] dk_hek_end = dk_ek_end + 12'd32; // H(ek) 区域结束
- // 加载时边界：dk 在 start_i 之前流式输入，因此 k_r 尚未
- // 捕获。使用实时 k_i 输入而非 k_r 来路由 dk 加载。
+ // full dk = dk_pke(dk_bytes) || ek(ek_bytes) || H(ek)(32) || z(32). Region
+ // boundaries (used by Decaps dk load routing and dk readback).
+ wire [11:0] dk_ek_end = dk_bytes_rt + ek_bytes_rt; // ek_pke region end
+ wire [11:0] dk_hek_end = dk_ek_end + 12'd32; // H(ek) region end
+ // LOAD-TIME boundaries: dk is streamed in BEFORE start_i, so k_r is not yet
+ // captured. Route the dk load by the LIVE k_i input instead of k_r.
wire [11:0] dkp_bytes_ld = 12'd384 * {9'b0, k_i}; // 768/1152/1536
wire [11:0] ek_bytes_ld = 12'd384 * {9'b0, k_i} + 12'd32; // 800/1184/1568
- wire [11:0] dk_ek_end_ld = dkp_bytes_ld + ek_bytes_ld; // ek_pke 区域结束
- wire [11:0] dk_hek_end_ld= dk_ek_end_ld + 12'd32; // H(ek) 区域结束
- // H(ek) 块数 = ceil((ek_bytes+1)/136)：k=2/3/4 时分别为 6/9/12
+ wire [11:0] dk_ek_end_ld = dkp_bytes_ld + ek_bytes_ld; // ek_pke region end
+ wire [11:0] dk_hek_end_ld= dk_ek_end_ld + 12'd32; // H(ek) region end
+ // H(ek) block count = ceil((ek_bytes+1)/136): 6/9/12 for k=2/3/4 (table)
wire [3:0] h_nblk_rt = (k_r == 3'd2) ? 4'd6 : (k_r == 3'd3) ? 4'd9 : 4'd12;
- wire [11:0] h_last_rt = {6'b0, h_nblk_rt} * 12'd136 - 12'd1; // 最终填充字节索引
+ wire [11:0] h_last_rt = {6'b0, h_nblk_rt} * 12'd136 - 12'd1; // final padded byte index
- // ---- Encaps 运行时参数 ----
- reg [1:0] op_r; // 0=密钥生成 1=封装 2=解封装（在 start 时捕获）
- reg [255:0] m_r; // Encaps 消息 m（在 start 时捕获）；Decaps 重用为 m'
- reg [255:0] ss_r; // Encaps 共享密钥 K（= G 输出低半部分）
- reg [255:0] r_r; // Encaps PRF 种子 r（= G 输出高半部分）
- // ---- Decaps 运行时参数（加载时从 dk 解析）----
- reg [255:0] z_r; // 隐式拒绝种子 z（dk 尾部），加载时捕获
- reg [255:0] kbar_r; // K-bar = J(z||c)（D5）
- // FIPS 203：所有参数集 eta2 = 2。
+ // ---- Encaps runtime params ----
+ reg [1:0] op_r; // 0=KeyGen 1=Encaps 2=Decaps (captured at start)
+ reg [255:0] m_r; // Encaps message m (captured at start); Decaps reuses for m'
+ reg [255:0] ss_r; // Encaps shared secret K (= G output low half)
+ reg [255:0] r_r; // Encaps PRF seed r (= G output high half)
+ // ---- Decaps runtime params (parsed from dk during load) ----
+ reg [255:0] z_r; // implicit-rejection seed z (dk tail), captured at load
+ reg [255:0] kbar_r; // K-bar = J(z||c) (D5)
+ // FIPS 203: eta2 = 2 for all parameter sets.
wire [1:0] eta2_rt = 2'd2;
- // 压缩参数：(du,dv) 在 k=2/3 时为 (10,4)，k=4 时为 (11,5)。
+ // Compression params: (du,dv) = (10,4) for k=2/3, (11,5) for k=4.
wire [4:0] du_rt = (k_r == 3'd4) ? 5'd11 : 5'd10;
wire [4:0] dv_rt = (k_r == 3'd4) ? 5'd5 : 5'd4;
- // 密文字节大小：c1 = 32*du*k, c2 = 32*dv, ct = c1+c2。
+ // Ciphertext byte sizes: c1 = 32*du*k, c2 = 32*dv, ct = c1+c2.
wire [11:0] c1_bytes_rt = 12'd32 * {7'b0, du_rt} * {9'b0, k_r}; // 640/960/1408
wire [11:0] c2_bytes_rt = 12'd32 * {7'b0, dv_rt}; // 128/128/160
wire [11:0] ct_bytes_rt = c1_bytes_rt + c2_bytes_rt; // 768/1088/1568
- // 共享密钥输出：Encaps 直接驱动 ss_r = K。Decaps 隐式拒绝：
- // ss = (c' == c) ? K' : K-bar。K' 在 ss_r 中，K-bar 在 kbar_r 中；
- // dec_reject 是来自 D7 锁存的 c' != c 标志（密钥生成/封装时为 0，
- // 因此 ss_r 直接通过）。
+ // Shared secret out: Encaps drives ss_r = K directly. Decaps implicit reject:
+ // ss = (c' == c) ? K' : K-bar. K' is in ss_r, K-bar in kbar_r; dec_reject is
+ // the latched c' != c flag from D7 (0 for KeyGen/Encaps, so ss_r passes through).
assign ss_o = dec_reject ? kbar_r : ss_r;
- // ---- E1: rho 加载 + byteDecode12 簿记 ----
- // rho 加载：从 ek_bram 中偏移 384*k 处读取 32 个字节到 rho_r。
- reg [5:0] rl_idx; // 0..33 rho 字节的预读指针
- reg [5:0] rl_widx; // 0..31 正在写入的字节（比 rl_idx 延迟 1 周期）
- reg rl_vld; // 本周起有字节到达
- wire [11:0] rl_ek_off = dk_bytes_rt + {6'b0, rl_idx}; // ek 地址 = 384k + idx
- // byteDecode12：读取 3 个 ek 字节转换为 2 个系数写入 bank_t。
- // 遍历 K 个多项式 * 128 个三元组。每个三元组 5 周期微相：
- // ph0..2 取回 b0,b1,b2（预读，1 周期 bram 延迟），
- // ph3 写入 c0, ph4 写入 c1。
- reg [2:0] td_poly; // 0..k-1 t_hat 多项式索引
- reg [7:0] td_trip; // 0..127 多项式内三元组
- reg [2:0] td_ph; // 0..4 微相
- reg [7:0] td_b0, td_b1, td_b2; // 捕获的 ek 字节
+ // ---- E1: rho-load + byteDecode12 bookkeeping ----
+ // rho load: read 32 bytes from ek_bram at offset 384*k into rho_r.
+ reg [5:0] rl_idx; // 0..33 read-ahead pointer over rho bytes
+ reg [5:0] rl_widx; // 0..31 byte being written (1 cyc behind rl_idx)
+ reg rl_vld; // a byte is arriving this cycle
+ wire [11:0] rl_ek_off = dk_bytes_rt + {6'b0, rl_idx}; // ek addr = 384k + idx
+ // byteDecode12: read 3 ek bytes -> 2 coeffs into bank_t. Walk K polys * 128
+ // triples. 5-cycle micro-phase per triple: ph0..2 fetch b0,b1,b2 (read-ahead,
+ // 1-cycle bram latency), ph3 write c0, ph4 write c1.
+ reg [2:0] td_poly; // 0..k-1 t_hat poly index
+ reg [7:0] td_trip; // 0..127 triple within poly
+ reg [2:0] td_ph; // 0..4 micro-phase
+ reg [7:0] td_b0, td_b1, td_b2; // captured ek bytes
reg td_done;
- // 三元组的 ek 字节地址：poly*384 + trip*3 + (ph 当 ph<3)
- wire [11:0] td_base = {td_poly, 7'd0} * 12'd3; // poly*384 (=poly*256*1.5) 见下文
- // poly*384 + trip*3 + byteidx; byteidx = ph (0,1,2) 在取回阶段
+ // ek byte address for the triple: poly*384 + trip*3 + (ph for ph<3)
+ wire [11:0] td_base = {td_poly, 7'd0} * 12'd3; // poly*384 (=poly*256*1.5)... see below
+ // poly*384 + trip*3 + byteidx; byteidx = ph (0,1,2) during fetch phases
wire [11:0] td_byteidx = {2'd0, td_trip, 1'b0} + {3'd0, td_trip}; // trip*3
wire [11:0] td_ekaddr = ({4'd0,td_poly}*12'd384) + td_byteidx + {9'd0, td_ph};
- // 解码后的系数：c0 = b0 | ((b1&0xF)<<8); c1 = (b1>>4) | (b2<<4)
+ // decoded coeffs: c0 = b0 | ((b1&0xF)<<8); c1 = (b1>>4) | (b2<<4)
wire [11:0] td_c0 = {td_b1[3:0], td_b0};
wire [11:0] td_c1 = {td_b2, td_b1[7:4]};
- // TDEC 字节源：Encaps 读取 ek (ek_rd_data)；Decaps D2 (ST_DEC_SDEC)
- // 重用同一机器对 dk_pke 进行 byteDecode12 得到 s_hat，读取 dkp_bram。
+ // TDEC byte source: Encaps reads ek (ek_rd_data); Decaps D2 (ST_DEC_SDEC)
+ // reuses the same machine to byteDecode12 dk_pke -> s_hat, reading dkp_bram.
wire td_dec_s = (st == ST_DEC_SDEC);
wire [7:0] td_byte = td_dec_s ? dkp_rd_data : ek_rd_data;
- // TDEC 的 bank_a 写入（已寄存）；t_hat[td_poly] -> bank_a 槽位
- // td_poly*K。td_wa 是 bank_a 字节地址 (PA_AW=12)。
+ // bank_a write for TDEC (registered); t_hat[td_poly] -> bank_a slot
+ // td_poly*K. td_wa is a bank_a byte address (PA_AW=12).
reg td_we;
- reg [11:0] td_wa; // bank_a 地址 (PA_AW=12)
+ reg [11:0] td_wa; // bank_a address (PA_AW=12)
reg [11:0] td_wd;
- // ---- ct_bram：密文字节缓冲区（<=1568 B）。由 E5/E7（压缩 + byteEncode_du/dv）
- // 写入，通过 dbg_ct 抽头回读。----
+ // ---- ct_bram: ciphertext byte buffer (<=1568 B). Written by E5/E7
+ // (compress + byteEncode_du/dv), read back via dbg_ct tap. ----
wire [10:0] ct_rd_addr;
wire [7:0] ct_rd_data;
reg ct_we;
@@ -185,64 +186,65 @@ module mlkem_top #(
.rd_addr(ct_rd_addr), .rd_data(ct_rd_data),
.wr_en(ct_we), .wr_addr(ct_wa), .wr_data(ct_wd)
);
- // ct_bram 读取地址：D7 通过 cmp_idx 遍历比较 c'；否则为 debug 回读。
+ // ct_bram read addr: D7 compare walks c' via cmp_idx; else debug readback.
assign ct_rd_addr = (st == ST_DEC_CMP) ? cmp_idx : dbg_ct_idx_i;
assign dbg_ct_o = ct_rd_data;
- // ---- c_in_bram：Decaps 输入密文 c（<=1568 B）。在 ST_IDLE 中通过
- // c_in_* 预加载。与 ct_bram（保存计算出的 c'）分开保存，
- // 以便 D7 可以比较 c' 与 c，且 D5 J(z||c) 可以读取 c。
- // 读取端口是多路复用的（D5 J 馈送 / D7 比较）；
- // 对于 D0，只有加载写入路径存在。
+ // ---- c_in_bram: Decaps input ciphertext c (<=1568 B). Preloaded via
+ // c_in_* in ST_IDLE. Kept separate from ct_bram (which holds the computed
+ // c') so D7 can compare c' vs c and D5 J(z||c) can read c. Read port is
+ // muxed (D5 J-feed / D7 compare); for D0 only the load write path exists.
wire [10:0] cin_rd_addr;
wire [7:0] cin_rd_data;
- reg [10:0] cin_rd_addr_r; // D5/D7 读取地址（在此之前保持 0）
+ reg [10:0] cin_rd_addr_r; // D5/D7 read address (tied 0 until then)
+ reg dj_active; // background Decaps J(z||c) owns c_in read port
sd_bram #(.W(8), .D(2048), .A(11)) u_c_in_bram (
.clk(clk),
.rd_addr(cin_rd_addr), .rd_data(cin_rd_data),
.wr_en(c_in_we), .wr_addr(c_in_addr), .wr_data(c_in_byte)
);
- // c_in 读取地址：D5 J 组合逻辑地呈现 dj_c_idx（无额外寄存器级），
- // 因此寄存的 BRAM 读取在 1 周期后返回字节，与组装/写回流水线匹配。
- // 其他情况下，寄存的 cin_rd_addr_r（D1 遍历器/空闲）驱动它。
- assign cin_rd_addr = (st == ST_DEC_J) ? dj_c_idx[10:0] :
+ // c_in read addr: D5 J presents dj_c_idx combinationally (no extra register
+ // stage) so the registered BRAM read yields the byte 1 cycle later, matching
+ // the assemble/writeback pipeline. Otherwise the registered cin_rd_addr_r
+ // (D1 walker / idle) drives it.
+ assign cin_rd_addr = dj_active ? dj_c_idx[10:0] :
(st == ST_DEC_CMP) ? cmp_idx : cin_rd_addr_r;
// ================================================================
- // 多项式存储，按 KMAX（最坏情况）大小设计。运行时 k 使用
- // 子范围。槽位布局（每个槽位 = 256 个系数）：
- // A_hat[i][j] : 槽位 0 .. k*k-1 索引为 i*k + j
- // s_hat[i] : 槽位 slot_s_rt .. +k-1
- // e_hat[i] : 槽位 slot_e_rt .. +k-1
- // t_hat[i] : 槽位 slot_t_rt .. +k-1
- // NUM_SLOTS = KMAX*KMAX + 3*KMAX = 28（KMAX=4 时）。
+ // Polynomial storage, sized for KMAX (worst case). Runtime k uses a
+ // sub-range. Slot layout (each slot = 256 coeffs):
+ // A_hat[i][j] : slots 0 .. k*k-1 at index i*k + j
+ // s_hat[i] : slots slot_s_rt .. +k-1
+ // e_hat[i] : slots slot_e_rt .. +k-1
+ // t_hat[i] : slots slot_t_rt .. +k-1
+ // NUM_SLOTS = KMAX*KMAX + 3*KMAX = 28 for KMAX=4.
// ================================================================
localparam NUM_SLOTS = KMAX*KMAX + 3*KMAX;
- localparam SAW = 5; // 槽位地址宽度（>=clog2(28)）
+ localparam SAW = 5; // slot-address width (>=clog2(28))
// ================================================================
- // 3 个 Bank 的系数存储（阶段 2）。原本是异步读取寄存器数组
- // polymem[0:28*256-1]；按多项式拆分后，每个成为
- // 带寄存输出的 sd_bram（在最终步骤中推断为 BRAM / ASIC SRAM）。
- // bank_a : A_hat[i][j] 槽位 0..K^2-1 -> D=KMAX*KMAX*256=4096
- // bank_se: s_hat||e_hat 槽位 slot_s.. -> D=2*KMAX*256=2048（相对 0..2K-1）
- // bank_t : t_hat[i] 槽位 slot_t.. -> D=KMAX*256=1024（相对 0..K-1）
- // 地址是基址相对的：bank 索引 = 绝对槽位 - 基槽位。
- // 阶段 2b（进行中）：按消费者的寄存预读。目前 Bank
- // 仍是异步寄存器数组；已转换的消费者通过各自
- // 的 1 周期流水线寄存器（== sd_bram 时序）读取。
- // 当某个 Bank 的所有消费者都改为寄存后，该数组将被 sd_bram
- // 替换，其读取端口在（相位不重叠的）消费者之间多路复用。
+ // Coefficient storage as 3 banks (Phase 2). Was one async-read reg
+ // array polymem[0:28*256-1]; split by polynomial so each becomes a
+ // registered-read sd_bram (infers BRAM / ASIC SRAM) in the final step.
+ // bank_a : A_hat[i][j] slots 0..K^2-1 -> D=KMAX*KMAX*256=4096
+ // bank_se: s_hat||e_hat slots slot_s.. -> D=2*KMAX*256=2048 (rel 0..2K-1)
+ // bank_t : t_hat[i] slots slot_t.. -> D=KMAX*256=1024 (rel 0..K-1)
+ // Addresses are base-relative: bank index = abs_slot - base_slot.
+ // Stage 2b (in progress): per-consumer registered read-ahead. Banks are
+ // still async reg arrays here; converted consumers read via their own
+ // 1-cycle pipeline reg (== sd_bram timing). Once every consumer of a
+ // bank is registered, the array is replaced by an sd_bram with its read
+ // port muxed across the (phase-disjoint) consumers.
// ================================================================
- localparam PA_AW = 12; // bank_a 地址宽度 (4096)
- localparam PSE_AW = 11; // bank_se 地址宽度 (2048)
- localparam PT_AW = 10; // bank_t 地址宽度 (1024)
+ localparam PA_AW = 12; // bank_a addr width (4096)
+ localparam PSE_AW = 11; // bank_se addr width (2048)
+ localparam PT_AW = 10; // bank_t addr width (1024)
- // bank_a: A_hat。读取者 = ST_M 加载 (pm_a) + dbg。写入者 = ST_A。
- // 读取端口按相位多路复用；sd_bram 的内部 rd_addr_r 寄存器是
- // 1 周期读取流水线（替代了显式的 pm_a_rd 寄存器）。写入端口
- // 是组合逻辑驱动的（sd_bram 在上升沿寄存写入，与旧寄存器数组
- // 的非阻塞写入时序匹配）。
+ // bank_a: A_hat. Reader = ST_M load (pm_a) + dbg. Writer = ST_A.
+ // Read port muxed by phase; sd_bram's internal rd_addr_r register is the
+ // 1-cycle read pipeline (replaces the explicit pm_a_rd reg). Write port is
+ // driven combinationally (sd_bram registers the write at posedge, matching
+ // the old reg-array nonblocking write timing).
wire [PA_AW-1:0] ba_rd_addr;
wire [11:0] ba_rd_data;
wire ba_we;
@@ -252,12 +254,12 @@ module mlkem_top #(
.clk(clk), .rd_addr(ba_rd_addr), .rd_data(ba_rd_data),
.wr_en(ba_we), .wr_addr(ba_wa), .wr_data(ba_wd)
);
- // ST_A / ST_ENC_A 写入：当接受时将 snt_coeff 提交到 A_hat[a_slot]。
- // Encaps E6 也通过 byteDecode12 (TDEC) 将 t_hat 写入这里：t_hat[j] 写入
- // bank_a 的槽位 j*K，使得 V 的 MAC 重用 E4 的 u_aslot=u_j*K (u_row=0) 地址。
- // ST_ENC_E2MV 将搬迁的 e2 写入 bank_a[E2_ASLOT] (em_we)。
- // D1 DECOMP 将 v' 写入 bank_a[DEC_VASLOT] (dec_v_we)，使得 D3 可以在
- // w-SUB 期间与 bank_t psum 并行读取。
+ // ST_A / ST_ENC_A write: commit snt_coeff to A_hat[a_slot] when accepted.
+ // Encaps E6 also writes t_hat here via byteDecode12 (TDEC): t_hat[j] lands
+ // in bank_a at slot j*K so V's MAC reuses E4's u_aslot=u_j*K (u_row=0) addr.
+ // ST_ENC_E2MV writes the relocated e2 into bank_a[E2_ASLOT] (em_we).
+ // D1 DECOMP writes v' into bank_a[DEC_VASLOT] (dec_v_we) so D3 can read it
+ // in parallel with the bank_t psum during the w-SUB.
assign ba_we = ((st == ST_A || st == ST_ENC_A) && a_busy && snt_vo && snt_ack)
|| td_we || em_we || dec_v_we;
assign ba_wa = td_we ? td_wa[PA_AW-1:0]
@@ -265,15 +267,15 @@ module mlkem_top #(
: dec_v_we ? dec_v_wr[PA_AW-1:0]
: (a_slot*256 + a_widx) & ((1<0 运行 t_hat) + ST_E ek 半部分 + dbg。
- // 写入者 = ST_M 累加。读取端口按相位多路复用；sd_bram 的 rd_addr_r 是
- // 1 周期读取寄存器。RMW 是安全的：累加读取地址领先写入
- // 地址 1（m_oidx+1 vs m_oidx），因此读取/写入在同一个周期内不会别名，
- // sd_bram 的写优先行为 == 寄存器数组的读旧值。写入端口为组合逻辑。
+ // bank_t: t_hat. Readers = ST_M acc (j>0 running t_hat) + ST_E ek-half + dbg.
+ // Writer = ST_M acc. Read port muxed by phase; sd_bram's rd_addr_r is the
+ // 1-cycle read register. RMW is safe: the acc read addr leads the write
+ // addr by 1 (m_oidx+1 vs m_oidx), so read/write never alias in a cycle and
+ // sd_bram write-first == reg-array read-old. Write port combinational.
wire [PT_AW-1:0] bt_rd_addr;
wire [11:0] bt_rd_data;
wire bt_we;
@@ -298,16 +300,16 @@ module mlkem_top #(
.clk(clk), .rd_addr(bt_rd_addr), .rd_data(bt_rd_data),
.wr_en(bt_we), .wr_addr(bt_wa), .wr_data(bt_wd)
);
- // bank_t 写入：密钥生成 ST_M 累加，或 Encaps e2（ST_ENC_C c_poly==2K
- // -> bank_t 相对槽位 0，在 C/N/U 阶段空闲）。（TDEC 现在写入 bank_a。）
+ // bank_t write: KeyGen ST_M accumulate, or Encaps e2 (ST_ENC_C c_poly==2K
+ // -> bank_t rel slot 0, free during C/N/U). (TDEC writes bank_a now.)
wire e2_we = (st == ST_ENC_C) && c_busy && cbd_vo && cbd_ack &&
(c_poly == {1'b0, k_r, 1'b0}); // c_poly == 2K
- // Encaps E4/E6 bank_t 写入：MAC 部分和（u_sub0，pm_vo 时）和 INTT 原地写回
- //（u_sub1，ntt_vo 时）。均写入 bank_t 相对槽位 UPSUM。ST_ENC_V (E6) 增加
- // V-ADD 写回（psum + e2 + mu）到 UPSUM，通过下面的 u_v_we 实现。
+ // Encaps E4/E6 bank_t writes: MAC psum (u_sub0, on pm_vo) and INTT in-place
+ // (u_sub1, on ntt_vo). Both target bank_t rel slot UPSUM. ST_ENC_V (E6) adds
+ // the V-ADD writeback (psum + e2 + mu) at UPSUM via u_v_we below.
wire u_psum_we = (st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && (u_sub == 2'd0) && pm_vo;
wire u_intt_we = (st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && (u_sub == 2'd1) && ntt_vo;
- // D1 DECOMP 写回：解码 c2 时 cd_vo 将 v' 写入 bank_a[DEC_VASLOT]。
+ // D1 DECOMP writeback: cd_vo while decoding c2 writes v' to bank_a[DEC_VASLOT].
wire dec_v_we = (st == ST_DEC_DECOMP) && dec_in_c2 && cd_vo;
wire [13:0] dec_v_wr = DEC_VASLOT*256 + dec_widx;
assign bt_we = ((st == ST_M) && pm_vo) || e2_we ||
@@ -325,58 +327,58 @@ module mlkem_top #(
u_w_we ? u_wq :
m_accq;
- // Debug 回读（已寄存以改善时序）
+ // Debug readback (registered for timing)
reg [11:0] dbg_coeff_r;
- // 按绝对槽位范围路由 debug 读取（不在 KAT 正确性路径上——
- // 测试平台仅读取 ek/dk BRAM）。A：0..K^2-1；s/e：slot_s..slot_t-1；
- // t：slot_t..。Bank 内索引为 (绝对槽位-基址)*256 + idx。
+ // Route debug read by absolute slot range (not on the KAT correctness
+ // path — TB reads ek/dk BRAMs only). A: 0..K^2-1; s/e: slot_s..slot_t-1;
+ // t: slot_t.. Index within bank is (abs_slot-base)*256 + idx.
wire [13:0] dbg_a_addr = dbg_slot_i*256 + dbg_idx_i;
wire [13:0] dbg_se_addr = (dbg_slot_i - slot_s_rt)*256 + dbg_idx_i;
wire [13:0] dbg_t_addr = (dbg_slot_i - slot_t_rt)*256 + dbg_idx_i;
- // bank_a 读取端口：ST_M / Encaps ST_ENC_U MAC 驱动 pm_a 地址；ST_ENC_V
- // MAC 读取 t_hat[j] (u_pm_a_full, u_row=0) 且 ADD 读取 e2 (E2_ASLOT)；否则为 dbg。
+ // bank_a read port: ST_M / Encaps ST_ENC_U MAC drive pm_a addr; ST_ENC_V
+ // MAC reads t_hat[j] (u_pm_a_full, u_row=0) and ADD reads e2 (E2_ASLOT); else dbg.
assign ba_rd_addr = (st == ST_M) ? pm_a_full[PA_AW-1:0] :
(st == ST_ENC_U) ? u_pm_a_full[PA_AW-1:0] :
(st == ST_ENC_V) ?
- ((u_sub == 2'd2) ? u_v_e2rd[PA_AW-1:0] // ADD: e2 读取
- : u_pm_a_full[PA_AW-1:0]) : // MAC: t_hat[u_j]
+ ((u_sub == 2'd2) ? u_v_e2rd[PA_AW-1:0] // ADD: e2 read
+ : u_pm_a_full[PA_AW-1:0]) : // MAC: t_hat[u_j]
(st == ST_DEC_W) ?
- ((u_sub == 2'd2) ? u_w_vrd[PA_AW-1:0] // SUB: v' 读取
- : u_pm_a_full[PA_AW-1:0]) : // MAC: s_hat[u_j]
+ ((u_sub == 2'd2) ? u_w_vrd[PA_AW-1:0] // SUB: v' read
+ : u_pm_a_full[PA_AW-1:0]) : // MAC: s_hat[u_j]
dbg_a_addr[PA_AW-1:0];
- // bank_t 读取端口：ST_M 累加 (t_hat)、ST_E ek 半部分 (byteEncode)、Encaps
- // ST_ENC_U/V + Decaps ST_DEC_W（MAC psum 累加、INTT 加载、或按子相位的
- // ADD/SUB psum 读取）、ST_ENC_E2MV（从相对槽位 0 读取 e2）；否则为 dbg。
+ // bank_t read port: ST_M acc (t_hat), ST_E ek-half (byteEncode), Encaps
+ // ST_ENC_U/V + Decaps ST_DEC_W (MAC psum acc, INTT load, or ADD/SUB psum
+ // read by sub-phase), ST_ENC_E2MV (e2 read from rel slot 0); else dbg.
assign bt_rd_addr = (st == ST_M) ? m_tacc_full[PT_AW-1:0] :
(st == ST_E) ? e_rd_full[PT_AW-1:0] :
(st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) ?
((u_sub == 2'd0) ? u_psum_full[PT_AW-1:0] : // MAC acc
- (u_sub == 2'd1) ? u_intt_rd[PT_AW-1:0] : // INTT 加载
- u_add_prd[PT_AW-1:0]) : // ADD/SUB psum 读取
- (st == ST_ENC_E2MV) ? em_rd[PT_AW-1:0] : // e2 搬迁读取
+ (u_sub == 2'd1) ? u_intt_rd[PT_AW-1:0] : // INTT load
+ u_add_prd[PT_AW-1:0]) : // ADD/SUB psum read
+ (st == ST_ENC_E2MV) ? em_rd[PT_AW-1:0] : // e2 relocate read
(st == ST_ENC_C2) ? cp_bt_full[PT_AW-1:0] : // C2: v[cp_idx]
(st == ST_DEC_MENC) ? men_rd[PT_AW-1:0] : // D4: w[men_idx]
dbg_t_addr[PT_AW-1:0];
- // bank_se 读取端口：ST_N 加载 (s/e NTT)、ST_M 加载 (pm_b s_hat[j]) vs
- // 累加 (e_hat，通过 m_loading 互斥)、ST_E dk 半部分、dbg。
+ // bank_se read port: ST_N load (s/e NTT), ST_M load (pm_b s_hat[j]) vs
+ // accumulate (e_hat, mutually exclusive via m_loading), ST_E dk-half, dbg.
assign bse_rd_addr = (st == ST_N || st == ST_ENC_N || st == ST_DEC_NTT) ? ntt_rd_full[PSE_AW-1:0] :
(st == ST_M) ? (m_loading ? pm_b_full[PSE_AW-1:0]
: m_eacc_full[PSE_AW-1:0]) :
(st == ST_E) ? e_rd_full[PSE_AW-1:0] :
(st == ST_ENC_U) ?
((u_sub == 2'd0) ? u_pm_b_full[PSE_AW-1:0] : // MAC: y_hat[u_j]
- u_add_e1rd[PSE_AW-1:0]) : // ADD: e1[u_row] 读取
+ u_add_e1rd[PSE_AW-1:0]) : // ADD: e1[u_row] read
(st == ST_ENC_V) ? u_pm_b_full[PSE_AW-1:0] : // V MAC: y_hat[u_j]
(st == ST_DEC_W && u_sub == 2'd0) ? u_pm_b_full[PSE_AW-1:0] : // D3 MAC: u_hat[u_j]
(st == ST_ENC_C1) ? cp_se_full[PSE_AW-1:0] : // C1: u[cp_poly]
dbg_se_addr[PSE_AW-1:0];
- // bank_se 写入：密钥生成 ST_C CBD (s/e)、Encaps ST_ENC_C CBD (y/e1, c_poly<2K；
- // c_poly==2K 的 e2 写入 bank_t 而非 bank_se)、ST_N/ST_ENC_N NTT 写回、
- // Encaps E4 ADD u 写回 (u_sub==2)。
- // Encaps E4 ADD：将 u[u_row] 写入 bank_se 相对槽位 (K+u_row)，覆盖 e1。
+ // bank_se write: KeyGen ST_C CBD (s/e), Encaps ST_ENC_C CBD (y/e1, c_poly<2K;
+ // e2 at c_poly==2K goes to bank_t instead), ST_N/ST_ENC_N NTT writeback,
+ // Encaps E4 ADD u-writeback (u_sub==2).
+ // Encaps E4 ADD: write u[u_row] over e1 in bank_se rel slot (K+u_row).
wire u_add_we = (st == ST_ENC_U) && (u_sub == 2'd2) && u_avalid;
- // D1 DECOMP 写回：解码 c1 时 cd_vo 将 u'[dec_poly] 写入
- // bank_se 相对槽位 dec_poly (0..K-1)，系数索引为 dec_widx。
+ // D1 DECOMP writeback: cd_vo while decoding c1 writes u'[dec_poly] to
+ // bank_se rel slot dec_poly (0..K-1), at coeff index dec_widx.
wire dec_u_we = (st == ST_DEC_DECOMP) && !dec_in_c2 && cd_vo;
wire [13:0] dec_u_wr = {2'b0, dec_poly}*256 + dec_widx;
assign bse_we = ((st == ST_C) && c_busy && cbd_vo && cbd_ack) ||
@@ -398,28 +400,28 @@ module mlkem_top #(
end
assign dbg_coeff_o = dbg_coeff_r;
- // ek 和 dk_pke 字节存储器按 KMAX 大小设计。
+ // ek and dk_pke byte memories sized for KMAX.
localparam EK_MAX = 384*KMAX + 32; // 1568
localparam DK_MAX = 384*KMAX; // 1536
- // ek / dk_pke 字节存储于 BRAM (sd_bram: 1R/1W, 1 周期读取延迟)。
- // 每个单读取端口在时间上不重叠地共享：ST_H 组装读取 ek
- //（在 ST_H 中），debug 回读读取 ek/dk（在 ST_DONE 中）。写入端口由
- // ST_E（1 字节/周期）和 rho 尾部复制驱动。
+ // ek / dk_pke byte storage in BRAM (sd_bram: 1R/1W, 1-cycle read latency).
+ // Each single read port is shared time-disjointly: ST_H assemble reads ek
+ // (in ST_H), debug readback reads ek/dk (in ST_DONE). Write port driven by
+ // ST_E (1 byte/cycle) and the rho-tail copy.
wire [10:0] ek_rd_addr, dkp_rd_addr;
wire [7:0] ek_rd_data, dkp_rd_data;
reg ek_we, dkp_we;
reg [10:0] ek_wa, dkp_wa;
reg [7:0] ek_wd, dkp_wd;
- // ek BRAM 写入端口：密钥生成 ST_E 驱动 ek_we/ek_wa/ek_wd；Encaps 从外部
- // ek_in_* 端口预加载 ek（测试平台在 start_i 前流式输入 ek=pk）。Decaps
- // 在加载时将 dk 的 ek_pke 区域 (dk[384K:768K+32]) 路由至此。三者
- // 从不重叠（预加载在运行前于 ST_IDLE 中完成）。
+ // ek BRAM write port: KeyGen ST_E drives ek_we/ek_wa/ek_wd; Encaps preloads
+ // ek from the external ek_in_* port (TB streams ek=pk before start_i). Decaps
+ // routes the dk's ek_pke region (dk[384K:768K+32]) here at load time. The
+ // three never overlap (preloads happen in ST_IDLE before a run).
wire dk_ld_ekpke = dk_in_we && (dk_in_addr >= dkp_bytes_ld)
- && (dk_in_addr < dk_ek_end_ld); // ek_pke 区域
- wire dk_ld_dkpke = dk_in_we && (dk_in_addr < dkp_bytes_ld); // dk_pke 区域
- wire [10:0] dk_ek_off = (dk_in_addr - dkp_bytes_ld); // ek 内偏移
+ && (dk_in_addr < dk_ek_end_ld); // ek_pke region
+ wire dk_ld_dkpke = dk_in_we && (dk_in_addr < dkp_bytes_ld); // dk_pke region
+ wire [10:0] dk_ek_off = (dk_in_addr - dkp_bytes_ld); // offset within ek
wire ek_we_mux = ek_in_we ? 1'b1 :
dk_ld_ekpke ? 1'b1 : ek_we;
wire [10:0] ek_wa_mux = ek_in_we ? ek_in_addr :
@@ -432,9 +434,9 @@ module mlkem_top #(
.rd_addr(ek_rd_addr), .rd_data(ek_rd_data),
.wr_en(ek_we_mux), .wr_addr(ek_wa_mux), .wr_data(ek_wd_mux)
);
- // dk_pke BRAM 写入端口：密钥生成 ST_E（s_hat byteEncode）驱动 dkp_we/wa/wd；
- // Decaps 在加载时将 dk 的 dk_pke 区域 (dk[0:384K]) 路由至此。两者
- // 从不重叠（Decaps 运行前在 ST_IDLE 中预加载）。
+ // dk_pke BRAM write port: KeyGen ST_E (s_hat byteEncode) drives dkp_we/wa/wd;
+ // Decaps routes the dk's dk_pke region (dk[0:384K]) here at load time. The
+ // two never overlap (preload in ST_IDLE before a Decaps run).
wire dkp_we_mux = dk_ld_dkpke ? 1'b1 : dkp_we;
wire [10:0] dkp_wa_mux = dk_ld_dkpke ? dk_in_addr[10:0] : dkp_wa;
wire [7:0] dkp_wd_mux = dk_ld_dkpke ? dk_in_byte : dkp_wd;
@@ -444,17 +446,18 @@ module mlkem_top #(
.wr_en(dkp_we_mux), .wr_addr(dkp_wa_mux), .wr_data(dkp_wd_mux)
);
- // 完整 dk = dk_pke(dk_bytes) || ek(ek_bytes) || H(ek)(32) || z(32)。
- //（dk_ek_end / dk_hek_end 已在顶部附近与其他大小线一起声明。）
+ // full dk = dk_pke(dk_bytes) || ek(ek_bytes) || H(ek)(32) || z(32).
+ // (dk_ek_end / dk_hek_end declared near the top with the other size wires.)
- // Debug 区域选择用于 dk 回读（组合逻辑区域解码）。
+ // Debug-region selects for dk readback (combinational region decode).
wire dbgdk_in_dkp = (dbg_dk_idx_i < dk_bytes_rt);
wire dbgdk_in_ek = (dbg_dk_idx_i >= dk_bytes_rt) && (dbg_dk_idx_i < dk_ek_end);
- // 读取地址多路复用（在此声明；通过下方 H 阶段寄存器声明后
- // 的 h_blk*136 + h_byte 表达式实现，无函数调用）。
- // dbg_byte_o / dbg_dk_o 是（寄存后的 BRAM 读取数据上的）组合逻辑抽头；
- // 净 1 周期延迟，完全在测试平台的 2 周期读取等待范围内。
+ // Read-address muxes (declared here; depend on ST_H h_g defined below via
+ // the function-free expression h_blk*136 + h_byte). See assigns after the
+ // H-stage register declarations.
+ // dbg_byte_o / dbg_dk_o are combinational taps over the (registered) BRAM
+ // read data; net 1-cycle latency, well within the TB's 2-cycle read wait.
assign dbg_byte_o = dbg_byte_sel_i ? dkp_rd_data : ek_rd_data;
assign dbg_dk_o = dbgdk_in_dkp ? dkp_rd_data :
dbgdk_in_ek ? ek_rd_data :
@@ -462,58 +465,58 @@ module mlkem_top #(
: z_i[(dbg_dk_idx_i - dk_hek_end)*8 +: 8];
// ================================================================
- // 顶层 FSM（增量构建）。阶段 2a：仅 G。
+ // Top-level FSM (built incrementally). Stage 2a: G only.
// ================================================================
localparam ST_IDLE = 5'd0;
- localparam ST_G = 5'd1; // 运行 G(d||K)，捕获 rho/sigma
- localparam ST_A = 5'd2; // 通过 SampleNTT 生成 A_hat[i][j]
- localparam ST_C = 5'd3; // 通过 CBD 生成 s[i],e[i]
- localparam ST_N = 5'd4; // 对 s[i],e[i] 进行原地前向 NTT
- localparam ST_M = 5'd5; // 矩阵累加 t_hat = e_hat + sum A o s_hat
+ localparam ST_G = 5'd1; // run G(d||K), capture rho/sigma
+ localparam ST_A = 5'd2; // generate A_hat[i][j] via SampleNTT
+ localparam ST_C = 5'd3; // generate s[i],e[i] via CBD
+ localparam ST_N = 5'd4; // forward NTT of s[i],e[i] in place
+ localparam ST_M = 5'd5; // matrix accumulate t_hat = e_hat + sum A o s_hat
localparam ST_E = 5'd6; // byteEncode12 -> ek/dk BRAM
- localparam ST_H = 5'd7; // 通过多块 SHA3-256 计算 H(ek)
- // ---- Encaps 状态 ----
- localparam ST_ENC_LOAD = 5'd8; // ek 已预加载；占位/稳定
- localparam ST_ENC_H = 5'd9; // 通过多块 SHA3-256 计算 H(ek)（重用 ST_H 逻辑）
- localparam ST_ENC_G = 5'd10; // (K,r) = G(m || H(ek))，64 字节单块
- localparam ST_ENC_A = 5'd11; // 重新生成 A_hat（U 中使用转置）
- localparam ST_ENC_TDEC = 5'd12; // byteDecode12：ek -> t_hat（bank_a 槽位 j*K）
- localparam ST_ENC_C = 5'd13; // 通过 CBD 采样 y,e1,e2（eta1/eta2）
- localparam ST_ENC_N = 5'd14; // 对 y 进行原地前向 NTT
+ localparam ST_H = 5'd7; // H(ek) via multi-block SHA3-256
+ // ---- Encaps states ----
+ localparam ST_ENC_LOAD = 5'd8; // ek already preloaded; placeholder/settle
+ localparam ST_ENC_H = 5'd9; // H(ek) via multi-block SHA3-256 (reuses ST_H logic)
+ localparam ST_ENC_G = 5'd10; // (K,r) = G(m || H(ek)), 64-byte single block
+ localparam ST_ENC_A = 5'd11; // regenerate A_hat (transpose used in U)
+ localparam ST_ENC_TDEC = 5'd12; // byteDecode12: ek -> t_hat (bank_a slot j*K)
+ localparam ST_ENC_C = 5'd13; // sample y,e1,e2 via CBD (eta1/eta2)
+ localparam ST_ENC_N = 5'd14; // forward NTT of y in place
localparam ST_ENC_U = 5'd15; // u = INTT(sum A^T o y_hat) + e1
localparam ST_ENC_C1 = 5'd16; // Compress_du + byteEncode_du -> ct c1
localparam ST_ENC_V = 5'd17; // v = INTT(sum t_hat o y_hat) + e2 + mu
localparam ST_ENC_C2 = 5'd18; // Compress_dv + byteEncode_dv -> ct c2
- localparam ST_ENC_E2MV = 5'd19; // 搬迁 e2 bank_t[0] -> bank_a[E2_ASLOT]
- // ---- Decaps 状态 ----
- localparam ST_DEC_LOAD = 5'd20; // dk/c 已流式输入；解析/稳定（D0）
- localparam ST_DEC_DECOMP = 5'd21; // D1：byteDecode_d + 解压缩 c1->u', c2->v'
- localparam ST_DEC_SDEC = 5'd22; // D2：byteDecode12 dk_pke -> s_hat（bank_a 槽位 j*K）
- localparam ST_DEC_NTT = 5'd23; // D2：u_hat[i] = NTT(u'[i]) 原地（bank_se 相对 i）
- localparam ST_DEC_W = 5'd24; // D3：w = v' - INTT(sum_j s_hat[j] o u_hat[j])
- localparam ST_DEC_MENC = 5'd25; // D4：m' = byteEncode_1(Compress_1(w))
- localparam ST_DEC_G = 5'd26; // D5：(K',r') = G(m' || h)，SHA3-512 单块
- localparam ST_DEC_J = 5'd27; // D5：K-bar = J(z || c)，多块 SHAKE256
- localparam ST_DEC_CMP = 5'd28; // D7：c' == c 比较 + 隐式拒绝 ss 选择
+ localparam ST_ENC_E2MV = 5'd19; // relocate e2 bank_t[0] -> bank_a[E2_ASLOT]
+ // ---- Decaps states ----
+ localparam ST_DEC_LOAD = 5'd20; // dk/c already streamed in; parse/settle (D0)
+ localparam ST_DEC_DECOMP = 5'd21; // D1: byteDecode_d + Decompress c1->u', c2->v'
+ localparam ST_DEC_SDEC = 5'd22; // D2: byteDecode12 dk_pke -> s_hat (bank_a slot j*K)
+ localparam ST_DEC_NTT = 5'd23; // D2: u_hat[i] = NTT(u'[i]) in place (bank_se rel i)
+ localparam ST_DEC_W = 5'd24; // D3: w = v' - INTT(sum_j s_hat[j] o u_hat[j])
+ localparam ST_DEC_MENC = 5'd25; // D4: m' = byteEncode_1(Compress_1(w))
+ localparam ST_DEC_G = 5'd26; // D5: (K',r') = G(m' || h), SHA3-512 single block
+ localparam ST_DEC_J = 5'd27; // D5: K-bar = J(z || c), multi-block SHAKE256
+ localparam ST_DEC_CMP = 5'd28; // D7: c' == c compare + implicit-reject ss select
localparam ST_DONE = 5'd31;
reg [4:0] st, st_next;
reg [255:0] rho_r, sigma_r;
- // A 生成簿记：显式 i/j 计数器（避免运行时除法）
- reg [2:0] a_i; // 行 0..k-1
- reg [2:0] a_j; // 列 0..k-1
- reg [4:0] a_pair; // 已完成的 k*k 对计数（用于结束测试）
- reg [7:0] a_widx; // 当前多项式内的写入索引 0..255
- reg a_busy; // 当前对的请求被接受后为 1（选通收集）
- wire [SAW-1:0] a_slot = a_i*k_r + a_j; // A_hat[i][j] 槽位 = i*k + j
+ // A-generation bookkeeping: explicit i/j counters (avoid runtime divide)
+ reg [2:0] a_i; // row 0..k-1
+ reg [2:0] a_j; // col 0..k-1
+ reg [4:0] a_pair; // 0..k*k pairs done (for done test)
+ reg [7:0] a_widx; // write index 0..255 within current poly
+ reg a_busy; // 1 once current pair's request accepted (gates collect)
+ wire [SAW-1:0] a_slot = a_i*k_r + a_j; // A_hat[i][j] slot = i*k + j
- // C 生成簿记：2*k 个多项式（s[0..k-1] 然后 e[0..k-1]）
+ // C-generation bookkeeping: 2*k polys (s[0..k-1] then e[0..k-1])
reg [4:0] c_poly; // 0..2k
reg [7:0] c_widx;
reg c_busy;
- wire [7:0] c_nonce = {3'b0, c_poly}; // s:0..k-1 e:k..2k-1 == nonce
- // 槽位：c_poly < k -> s_hat[c_poly]，否则 e_hat[c_poly-k]
+ wire [7:0] c_nonce = {3'b0, c_poly}; // s:0..k-1 e:k..2k-1 == nonce
+ // slot: c_poly < k -> s_hat[c_poly], else e_hat[c_poly-k]
wire [SAW-1:0] c_slot = (c_poly < {2'b0, k_r}) ? (slot_s_rt + c_poly)
: (slot_e_rt + (c_poly - {2'b0, k_r}));
@@ -523,61 +526,61 @@ module mlkem_top #(
assign dbg_sigma_o = sigma_r;
assign dbg_r_o = r_r;
assign dbg_hek_o = hek_r;
- // Decaps 抽头：m' 来自 D4（写入 m_r，在 D6 重加密中被 V/mu 重用）；
- // 加载时从 dk 解析的 z/h。
+ // Decaps taps: m' from D4 (written into m_r, reused by V/mu in D6 re-encrypt);
+ // z/h parsed from dk at load.
assign dbg_mprime_o = m_r;
assign dbg_kbar_o = kbar_r;
assign dbg_decz_o = z_r;
- assign dbg_dech_o = hek_r; // Decaps 在加载时将 dk 的 H(ek) 解析到 hek_r
+ assign dbg_dech_o = hek_r; // Decaps parses dk's H(ek) into hek_r at load
- // ---- sha3_top 处于 G 模式：data_i = {K_byte, d}（d 字节 0 在 [7:0] 中）----
+ // ---- sha3_top in G mode: data_i = {K_byte, d} (d byte0 in [7:0]) ----
reg sha3_valid;
wire sha3_ready;
wire [511:0] sha3_hash;
wire sha3_vo;
- reg sha3_ack; // 消费者准备接收哈希
- wire [511:0] kg_g_data = {248'b0, 5'b0, k_r, d_i}; // 密钥生成 G：[263:256]=k, [255:0]=d
- wire [511:0] enc_g_data = {hek_r, m_r}; // Encaps G：m || H(ek)，64 字节
- wire [511:0] dec_g_data = {hek_r, m_r}; // Decaps D5 G：m' || h，64 字节
+ reg sha3_ack; // consumer ready for hash
+ wire [511:0] kg_g_data = {248'b0, 5'b0, k_r, d_i}; // KeyGen G: [263:256]=k, [255:0]=d
+ wire [511:0] enc_g_data = {hek_r, m_r}; // Encaps G: m || H(ek), 64 bytes
+ wire [511:0] dec_g_data = {hek_r, m_r}; // Decaps D5 G: m' || h, 64 bytes
wire [511:0] g_data = (st == ST_ENC_G) ? enc_g_data :
(st == ST_DEC_G) ? dec_g_data : kg_g_data;
// ================================================================
- // 共享 keccak_core + 相位多路复用（3 个消费者 -> 1 个核心）
+ // Shared keccak_core + phase mux (3 consumers -> 1 core)
//
- // G/H (u_sha3)、SampleNTT (u_snt) 和 CBD (u_cbd) 各自需要一个 Keccak
- // 置换，但它们在不相交的顶层 FSM 阶段运行（分别为 ST_G/ST_H、ST_A、
- // ST_C），因此同时最多只有一个活跃。一个 keccak_core
- // 服务所有三者：kc_state_i/kc_valid_i 按相位多路复用；kc_valid_o
- // 广播但按消费者选通，这样不活跃的消费者永远不会
- // 锁存为另一个消费者准备的置换结果（采样器无条件地
- // 在其 kc_valid_o 输入上锁存挤压状态）。
+ // G/H (u_sha3), SampleNTT (u_snt) and CBD (u_cbd) each need a Keccak
+ // permutation, but they run in DISJOINT top-FSM phases (ST_G/ST_H, ST_A,
+ // ST_C respectively), so at most one is ever active. One keccak_core
+ // serves all three: kc_state_i/kc_valid_i are muxed by phase; kc_valid_o
+ // is broadcast but GATED per consumer so an inactive consumer never
+ // latches a permutation result meant for another (the samplers latch
+ // squeeze state unconditionally on their kc_valid_o input).
// ================================================================
- wire [1599:0] kc_state_o; // 共享核心输出（给所有消费者）
- wire kc_valid_o; // 共享核心输出有效
- wire kc_ready_o; // 共享核心输入就绪
+ wire [1599:0] kc_state_o; // shared core output (to all consumers)
+ wire kc_valid_o; // shared core output-valid
+ wire kc_ready_o; // shared core input-ready
- // 每个消费者驱动到核心
+ // per-consumer drives toward the core
wire [1599:0] sha3_kc_state_i, snt_kc_state_i, cbd_kc_state_i;
wire sha3_kc_valid_i, snt_kc_valid_i, cbd_kc_valid_i;
- /* verilator lint_off UNUSED */
- wire sha3_kc_ready_i, snt_kc_ready_i, cbd_kc_ready_i; // 均为 1'b1
- /* verilator lint_on UNUSED */
+ /* verilator lint_off UNUSEDSIGNAL */
+ wire sha3_kc_ready_i, snt_kc_ready_i, cbd_kc_ready_i; // all 1'b1
+ /* verilator lint_on UNUSEDSIGNAL */
- // 相位选择（互斥）。Encaps 增加 ST_ENC_H/G (sha3)、
- // ST_ENC_A (snt)、ST_ENC_C (cbd)。
+ // phase selects (mutually exclusive). Encaps adds ST_ENC_H/G (sha3),
+ // ST_ENC_A (snt), ST_ENC_C (cbd).
wire sel_sha3 = (st == ST_G) || (st == ST_H) ||
(st == ST_ENC_H) || (st == ST_ENC_G) ||
- (st == ST_DEC_G) || (st == ST_DEC_J);
+ (st == ST_DEC_G) || dj_active;
wire sel_snt = (st == ST_A) || (st == ST_ENC_A);
wire sel_cbd = (st == ST_C) || (st == ST_ENC_C);
- // 选通输出有效：只有活跃消费者看到 kc_valid_o
+ // gated output-valid: only the active consumer sees kc_valid_o
wire kc_valid_o_sha3 = kc_valid_o & sel_sha3;
wire kc_valid_o_snt = kc_valid_o & sel_snt;
wire kc_valid_o_cbd = kc_valid_o & sel_cbd;
- // 输入多路复用：将活跃消费者的请求路由到核心
+ // input mux: route the active consumer's request to the core
wire [1599:0] kc_state_i_mux = sel_snt ? snt_kc_state_i :
sel_cbd ? cbd_kc_state_i :
sha3_kc_state_i;
@@ -592,23 +595,23 @@ module mlkem_top #(
.ready_o(kc_ready_o),
.state_o(kc_state_o),
.valid_o(kc_valid_o),
- .ready_i(1'b1) // 消费者始终接收 (kc_ready_i=1)
+ .ready_i(1'b1) // consumers always accept (kc_ready_i=1)
);
- // G/H/J 共享一个 sha3_top。单块 (mb_en=0)：密钥生成 G (ST_G, mode 00)、
- // Encaps G (ST_ENC_G, mode 11)、Decaps D5 G (ST_DEC_G, mode 11)。多块
- // 吸收 (mb_en=1)：H(ek) (ST_H/ST_ENC_H, SHA3-256 填充) 和 Decaps D5 J
- // (ST_DEC_J, SHAKE256 填充，输入为 z||c)。mb_* 输入按相位多路复用。
- wire sha3_mb_en = (st == ST_H) || (st == ST_ENC_H) || (st == ST_DEC_J);
+ // G/H/J share one sha3_top. Single-block (mb_en=0): KeyGen G (ST_G, mode 00),
+ // Encaps G (ST_ENC_G, mode 11), Decaps D5 G (ST_DEC_G, mode 11). Multi-block
+ // absorb (mb_en=1): H(ek) (ST_H/ST_ENC_H, SHA3-256 pad) and Decaps D5 J
+ // (background dj_active, SHAKE256 pad over z||c). mb_* inputs are muxed by phase.
+ wire sha3_mb_en = (st == ST_H) || (st == ST_ENC_H) || dj_active;
wire [1:0] sha3_mode = (st == ST_ENC_G || st == ST_DEC_G) ? 2'b11 : 2'b00;
- // 多块馈送多路复用：J (ST_DEC_J) 驱动 dj_*，否则 H 驱动 h_*。
- wire [1087:0] mb_block_mux = (st == ST_DEC_J) ? dj_block_r : h_block_r;
- wire mb_valid_mux = (st == ST_DEC_J) ? dj_mbvalid : h_mbvalid;
- wire mb_last_mux = (st == ST_DEC_J) ? dj_mblast : h_mblast;
- wire mb_ack_mux = (st == ST_DEC_J) ? dj_ack : h_ack;
+ // multi-block feed mux: background J drives dj_*, else H drives h_*.
+ wire [1087:0] mb_block_mux = dj_active ? dj_block_r : h_block_r;
+ wire mb_valid_mux = dj_active ? dj_mbvalid : h_mbvalid;
+ wire mb_last_mux = dj_active ? dj_mblast : h_mblast;
+ wire mb_ack_mux = dj_active ? dj_ack : h_ack;
sha3_top_shared u_sha3 (
.clk(clk), .rst_n(rst_n),
- .mode(sha3_mode), // G = SHA3-512（仅在 mb_en=0 时使用）
+ .mode(sha3_mode), // G = SHA3-512 (only used when mb_en=0)
.data_i(g_data),
.valid_i(sha3_valid),
.ready_o(sha3_ready),
@@ -620,7 +623,7 @@ module mlkem_top #(
.mb_valid_i(mb_valid_mux),
.mb_last_i(mb_last_mux),
.mb_ready_o(h_mbready),
- // 共享 keccak_core 接口（在下方的多路复用器中按相位选通）
+ // shared keccak_core interface (gated by phase in the mux below)
.kc_state_o(kc_state_o),
.kc_valid_o(kc_valid_o_sha3),
.kc_ready_o(kc_ready_o),
@@ -629,55 +632,55 @@ module mlkem_top #(
.kc_ready_i(sha3_kc_ready_i)
);
- // ---- 多块 H(ek) 状态（SHA3-256，6/9/12 块）；馈送到共享 u_sha3 ----
- reg [1087:0] h_block_r; // 当前预填充的速率块
+ // ---- multi-block H(ek) state (SHA3-256, 6/9/12 blocks); fed to shared u_sha3 ----
+ reg [1087:0] h_block_r; // current pre-padded rate block
reg h_mbvalid;
reg h_mblast;
wire h_mbready;
wire [511:0] h_hash;
wire h_vo;
reg h_ack;
- reg [255:0] hek_r; // 捕获的 H(ek)
- reg [3:0] h_blk; // 0..H_NBLK-1 块索引（K=4 时最多 11）
- reg [7:0] h_byte; // 0..136 正在呈现的字节地址（组装）
- reg [1:0] h_phase; // 0=组装 1=馈送 2=等待置换 3=完成
- // ek BRAM 读取是寄存的（1 周期延迟）；组装为 h_byte 呈现地址，
- // 并将（上一周期呈现的地址对应的）到达字节写入 h_block_r。
- // 写回流水线寄存器：
- reg h_wb_vld; // 本周起有字节到达
- reg [7:0] h_wb_idx; // 在 136 字节块中的位置
- reg [11:0] h_wb_g; // 全局 ek 字节索引
- reg [7:0] h_wb_pad; // 当 g 超出 ek 范围时使用的填充常量
- reg h_wb_inek; // 当 g 在 ek 范围内时为 1（使用 BRAM 数据）
+ reg [255:0] hek_r; // captured H(ek)
+ reg [3:0] h_blk; // 0..H_NBLK-1 block index (up to 11 for K=4)
+ reg [7:0] h_byte; // 0..136 byte address being presented (assemble)
+ reg [1:0] h_phase; // 0=assemble 1=feed 2=wait-perm 3=done
+ // ek BRAM read is registered (1-cycle latency); assemble presents the
+ // address for h_byte this cycle and writes the byte that arrived (for the
+ // address presented last cycle) into h_block_r. Writeback pipeline regs:
+ reg h_wb_vld; // a byte is arriving this cycle
+ reg [7:0] h_wb_idx; // its position within the 136-byte block
+ reg [11:0] h_wb_g; // its global ek byte index
+ reg [7:0] h_wb_pad; // pad constant to use if g is out of ek range
+ reg h_wb_inek; // 1 if g is within ek range (use BRAM data)
- // ---- Decaps D5 J(z||c) 多块 SHAKE256 状态 ----
- // 消息 = z（32 B，来自 z_r）|| c（ct_bytes_rt B，来自 c_in_bram）。速率为 136。
- // SHAKE256 填充：msg_len 处的字节为 0x1F，最后一块的最后一个字节或上 0x80。
- // 重用共享 u_sha3 的 mb_* 端口（上面由 ST_DEC_J 多路复用）。
- reg [1087:0] dj_block_r; // 当前预填充的速率块
+ // ---- Decaps D5 J(z||c) multi-block SHAKE256 state ----
+ // Message = z (32 B, from z_r) || c (ct_bytes_rt B, from c_in_bram). Rate=136.
+ // SHAKE256 pad: byte at msg_len = 0x1F, last byte of last block |= 0x80.
+ // Reuses the shared u_sha3 mb_* port (muxed by ST_DEC_J above).
+ reg [1087:0] dj_block_r; // current pre-padded rate block
reg dj_mbvalid;
reg dj_mblast;
reg dj_ack;
- reg [3:0] dj_blk; // 块索引 0..DJ_NBLK-1（z||c 最多 ~1600B -> <=12 块）
- reg [7:0] dj_byte; // 0..136 正在组装的字节
- reg [1:0] dj_phase; // 0=组装 1=馈送 2=等待置换 3=完成
+ reg [3:0] dj_blk; // block index 0..DJ_NBLK-1 (z||c up to ~1600B -> <=12 blks)
+ reg [7:0] dj_byte; // 0..136 byte being assembled
+ reg [1:0] dj_phase; // 0=assemble 1=feed 2=wait-perm 3=done
reg dj_done;
- // 写回流水线（c_in_bram 寄存读取，1 周期延迟）
+ // writeback pipeline (c_in_bram registered read, 1-cyc latency)
reg dj_wb_vld;
- reg [7:0] dj_wb_idx; // 在 136 字节块中的位置
- reg dj_wb_inc; // 全局字节 g 在 c 范围内时为 1（使用 BRAM）
- reg dj_wb_inz; // g < 32 时为 1（使用 z_r 字节）
- reg [7:0] dj_wb_pad; // g 超出消息范围时的填充常量
- reg [4:0] dj_wb_zidx; // 当 inz 时的 z 字节索引 (g)
- // J 消息长度 = 32 (z) + ct_bytes_rt；总填充块数。
+ reg [7:0] dj_wb_idx; // position within 136-byte block
+ reg dj_wb_inc; // 1 if global byte g is within c range (use BRAM)
+ reg dj_wb_inz; // 1 if g < 32 (use z_r byte)
+ reg [7:0] dj_wb_pad; // pad constant if g out of msg range
+ reg [4:0] dj_wb_zidx; // z byte index (g) when inz
+ // J message length = 32 (z) + ct_bytes_rt; total padded blocks.
wire [11:0] dj_msglen = 12'd32 + {1'b0, ct_bytes_rt};
- wire [11:0] dj_last_byte = (dj_msglen / 12'd136) * 12'd136 + 12'd135; // 最后一块的最后一个字节
+ wire [11:0] dj_last_byte = (dj_msglen / 12'd136) * 12'd136 + 12'd135; // last byte of final block
wire [3:0] dj_nblk = (dj_msglen / 12'd136) + 4'd1;
- // 本周起呈现的地址的全局字节索引 (组装)
+ // global byte index for the address presented this cycle (assemble)
wire [11:0] dj_g = {4'd0, dj_blk} * 12'd136 + {4'd0, dj_byte};
- // c_in_bram 字节索引（当 g>=32 时）：g-32
+ // c_in_bram byte index (when g>=32): g-32
wire [11:0] dj_c_idx = dj_g - 12'd32;
- // 全局字节 g 的 SHAKE256 填充常量
+ // SHAKE256 pad constant for global byte g
function [7:0] dj_padconst(input [11:0] g);
begin
if (g == dj_last_byte && g == dj_msglen) dj_padconst = 8'h9f; // 0x1F|0x80
@@ -687,17 +690,17 @@ module mlkem_top #(
end
endfunction
- // h_hash / h_vo 现在由上述共享 u_sha3 提供（ST_H 期间 mb_en=1）。
- // 它们与单核心输出的别名相同；下方的 H 消费者逻辑
- // 已经根据 st==ST_H 选通，且 u_sha3 的 valid_o/hash_o 由 mb 选择。
+ // h_hash / h_vo are now served by the shared u_sha3 above (mb_en=1 during
+ // ST_H). They alias the single core's outputs; the H consumer logic below
+ // already gates on st==ST_H, and u_sha3's valid_o/hash_o are mb-selected.
assign h_hash = sha3_hash;
assign h_vo = sha3_vo;
- // SHA3-256 计算 ek (ek_bytes_rt 字节)：速率为 136。填充长度 = h_nblk_rt*136。
- // 填充：字节 ek_bytes_rt = 0x06（域 + 首填充位），最后字节或上 0x80。
- // 块 blk 的字节 b (0..135)：全局 g = blk*136 + b。
- // ek 字节来自 ek BRAM（寄存读取）；只有填充/零字节
- // 是常量，由 h_padconst 返回。读取运行时 ek_bytes_rt/h_last_rt。
+ // SHA3-256 over ek (ek_bytes_rt bytes): rate=136. Padded length = h_nblk_rt*136.
+ // pad: byte ek_bytes_rt = 0x06 (domain + first pad bit), last byte |= 0x80.
+ // byte b (0..135) of block blk: global g = blk*136 + b.
+ // ek bytes come from the ek BRAM (registered read); only the pad/zero bytes
+ // are constants, returned by h_padconst. Reads runtime ek_bytes_rt/h_last_rt.
function [7:0] h_padconst(input [3:0] blk, input [7:0] b);
integer g;
begin
@@ -709,20 +712,20 @@ module mlkem_top #(
end
endfunction
- // ST_H 组装中当前呈现的地址的全局 ek 字节索引。
+ // Global ek byte index for the address currently presented in ST_H assemble.
wire [11:0] h_g_addr = {8'd0, h_blk} * 12'd136 + {4'd0, h_byte};
- // ek BRAM 读取地址多路复用：ST_H 组装驱动它；否则为 debug 回读。
- // dbg_dk ek 区域优先（其循环在 ek 循环之后运行），因此
- // 单读取端口服务于 dbg_byte_o（ek 循环）然后 dbg_dk_o（dk 循环）。
- wire [11:0] dbgdk_ek_off = dbg_dk_idx_i - dk_bytes_rt; // ek 区域内的偏移
+ // ek BRAM read-address mux: ST_H assemble drives it; otherwise debug readback.
+ // dbg_dk ek-region takes priority (its loop runs after the ek loop), so the
+ // single read port serves dbg_byte_o (ek loop) then dbg_dk_o (dk loop).
+ wire [11:0] dbgdk_ek_off = dbg_dk_idx_i - dk_bytes_rt; // offset within ek region
assign ek_rd_addr = (st == ST_H || st == ST_ENC_H) ? h_g_addr[10:0] :
- (st == ST_ENC_LOAD) ? rl_ek_off[10:0] : // rho 加载
+ (st == ST_ENC_LOAD) ? rl_ek_off[10:0] : // rho load
(st == ST_ENC_TDEC) ? td_ekaddr[10:0] : // byteDecode12
dbgdk_in_ek ? dbgdk_ek_off[10:0]
: dbg_byte_idx_i;
- // dkp BRAM 读取地址多路复用：D2 byteDecode12 (ST_DEC_SDEC) 通过 td_ekaddr
- // (poly*384 + trip*3 + ph) 遍历 dk_pke；否则为 dbg_byte (sel=1) / dbg_dk。
+ // dkp BRAM read-address mux: D2 byteDecode12 (ST_DEC_SDEC) walks dk_pke via
+ // td_ekaddr (poly*384 + trip*3 + ph); else dbg_byte (sel=1) / dbg_dk.
assign dkp_rd_addr = (st == ST_DEC_SDEC) ? td_ekaddr[10:0] :
dbg_byte_sel_i ? dbg_byte_idx_i : dbg_dk_idx_i[10:0];
@@ -732,7 +735,7 @@ module mlkem_top #(
wire [11:0] snt_coeff;
wire snt_vo;
wire snt_last;
- reg snt_ack; // 接收系数
+ reg snt_ack; // we accept coeffs
sample_ntt_sync_shared #(.K(KMAX)) u_snt (
.clk(clk), .rst_n(rst_n),
@@ -746,7 +749,7 @@ module mlkem_top #(
.valid_o(snt_vo),
.ready_i(snt_ack),
.last_o(snt_last),
- // 共享 keccak_core 接口
+ // shared keccak_core interface
.kc_state_o(kc_state_o),
.kc_valid_o(kc_valid_o_snt),
.kc_ready_o(kc_ready_o),
@@ -758,15 +761,15 @@ module mlkem_top #(
// ---- sample_cbd_sync: s[i]=CBD3(PRF(sigma,i)), e[i]=CBD3(PRF(sigma,K+i)) ----
reg cbd_valid;
wire cbd_ready;
- wire [11:0] cbd_coeff; // 12 位有符号（二进制补码）
+ wire [11:0] cbd_coeff; // 12-bit signed (two's complement)
wire cbd_vo;
wire cbd_last;
reg cbd_ack;
- // CBD 输入在 KeyGen 和 Encaps 之间多路复用：
- // KeyGen ST_C: seed=sigma, eta=eta1（所有多项式）, nonce=c_poly。
- // Encaps ST_ENC_C: seed=r, nonce=c_poly; eta=eta1（y, c_poly=K 时）。nonce 序列 0..2K 与 FIPS 一致。
+ // CBD inputs muxed KeyGen vs Encaps:
+ // KeyGen ST_C: seed=sigma, eta=eta1 (all polys), nonce=c_poly.
+ // Encaps ST_ENC_C: seed=r, nonce=c_poly; eta = eta1 for y (c_poly=K). nonce sequence 0..2K matches FIPS.
wire cbd_enc = (st == ST_ENC_C);
wire [255:0] cbd_seed = cbd_enc ? r_r : sigma_r;
wire [1:0] cbd_eta = cbd_enc ? ((c_poly < {2'b0, k_r}) ? eta1_rt : eta2_rt)
@@ -782,7 +785,7 @@ module mlkem_top #(
.valid_o(cbd_vo),
.ready_i(cbd_ack),
.last_o(cbd_last),
- // 共享 keccak_core 接口
+ // shared keccak_core interface
.kc_state_o(kc_state_o),
.kc_valid_o(kc_valid_o_cbd),
.kc_ready_o(kc_ready_o),
@@ -791,30 +794,30 @@ module mlkem_top #(
.kc_ready_i(cbd_kc_ready_i)
);
- // 有符号（二进制补码）-> [0,Q)：负数时加 Q
+ // signed (two's complement) -> [0,Q): add Q when negative
wire [11:0] cbd_modq = cbd_coeff[11] ? (cbd_coeff + 12'd3329) : cbd_coeff;
- // ---- ntt_core: 对 s[i],e[i] 进行原地前向 NTT（mode=0, 无缩放）----
- // N 阶段簿记：处理槽位 S0,S1,E0,E1（= SLOT_S0 + n_slot）。
- reg [4:0] n_slot; // 0..2K（先处理 s_hat[0..K-1]，再处理 e_hat[0..K-1]）
- reg [8:0] n_ridx; // 加载预读指针 0..256（领先消耗 1 周期）
- reg [7:0] n_widx; // 输出写入索引 0..255
- reg n_valid; // 向 ntt_core 馈送系数（比 n_ridx 延迟 1 周期）
- reg n_loading; // 向 bank_se 呈现加载地址时为 1
- reg n_pending; // 等待 ntt_core IDLE 以启动下一个槽位
- wire [SAW-1:0] n_slot_addr = slot_s_rt + n_slot; // s_hat 和 e_hat 连续
- // NTT 槽位数：KeyGen 处理 2K (s,e)；Encaps 处理 K（仅 y）。
+ // ---- ntt_core: forward NTT (mode=0, no scaling) of s[i],e[i] in place ----
+ // N-stage bookkeeping: process slots S0,S1,E0,E1 (= SLOT_S0 + n_slot).
+ reg [4:0] n_slot; // 0..2K (process s_hat[0..K-1] then e_hat[0..K-1])
+ reg [8:0] n_ridx; // load read-AHEAD pointer 0..256 (leads consume by 1)
+ reg [7:0] n_widx; // output write index 0..255
+ reg n_valid; // feeding coeffs to ntt_core (delayed 1 cyc vs n_ridx)
+ reg n_loading; // 1 while presenting load addresses to bank_se
+ reg n_pending; // waiting for ntt_core IDLE to start next slot
+ wire [SAW-1:0] n_slot_addr = slot_s_rt + n_slot; // s_hat then e_hat contiguous
+ // NTT slot count: KeyGen processes 2K (s,e); Encaps processes K (y only).
wire [4:0] n_slot_max = (st == ST_ENC_N || st == ST_DEC_NTT) ? {2'b0, k_r} : {1'b0, k_r, 1'b0};
wire ntt_ready;
wire [11:0] ntt_coeff;
wire ntt_vo;
wire ntt_done;
- // NTT 加载的 bank_se 读取地址（相对槽位 = n_slot）；sd_bram
- // 将其寄存到 bse_rd_data，1 周期后送入 ntt_core。
+ // bank_se read addr for the NTT load (relative slot = n_slot); sd_bram
+ // registers it into bse_rd_data, which feeds ntt_core 1 cycle later.
wire [13:0] ntt_rd_full = n_slot*256 + n_ridx[7:0];
- // ntt_core 输入多路复用：KeyGen/Encaps 前向 NTT 送入 bse_rd_data（mode 0）；
- // Encaps E4/E6 INTT 子阶段以 mode 1 送入 psum (bt_rd_data)。
+ // ntt_core inputs muxed: KeyGen/Encaps fwd NTT feeds bse_rd_data (mode 0);
+ // Encaps E4/E6 INTT sub-phase feeds psum (bt_rd_data) with mode 1.
wire u_intt_act = (st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && (u_sub == 2'd1);
wire [11:0] ntt_in = u_intt_act ? bt_rd_data : bse_rd_data;
wire ntt_vin = u_intt_act ? u_nvalid : n_valid;
@@ -825,204 +828,204 @@ module mlkem_top #(
.coeff_in(ntt_in),
.valid_i(ntt_vin),
.ready_o(ntt_ready),
- .mode(ntt_mode), // 0 = 前向 (NTT), 1 = 逆向 (INTT, x3303)
+ .mode(ntt_mode), // 0 = fwd (NTT), 1 = inv (INTT, x3303)
.coeff_out(ntt_coeff),
.valid_o(ntt_vo),
- .ready_i(1'b1), // 始终接收输出
+ .ready_i(1'b1), // always accept output
.done_o(ntt_done)
);
// ---- poly_mul_sync: t_hat[i] = e_hat[i] + sum_j A_hat[i][j] o s_hat[j] ----
- // M 阶段簿记。对每对 (i,j)：加载 256 对 (A,s_hat)，然后
- // 将 256 个乘积累加到 T_i（j==0 时从 E_i 初始化，否则从运行的 T_i 继续）。
- reg [2:0] m_i; // 行 0..K（需要达到 K 才能退出）
- reg [2:0] m_j; // 列 0..K-1
- reg [8:0] m_ld; // 加载索引 0..256
- reg [7:0] m_oidx; // 输出/累加索引 0..255
- reg m_loading; // 向 poly_mul 流式传输系数对时为 1
- reg m_pending; // 等待 poly_mul IDLE 再进行下一个 (i,j)
+ // M-stage bookkeeping. For each (i,j): LOAD 256 (A,shat) pairs, then accumulate
+ // 256 products into T_i (init from E_i when j==0, else from running T_i).
+ reg [2:0] m_i; // row 0..K (needs to reach K to exit)
+ reg [2:0] m_j; // col 0..K-1
+ reg [8:0] m_ld; // load index 0..256
+ reg [7:0] m_oidx; // output/accum index 0..255
+ reg m_loading; // 1 while streaming pairs into poly_mul
+ reg m_pending; // wait for poly_mul IDLE before next (i,j)
- // ---- 阶段 2f: byteEncode12 串行化器 ----
- // 打包每个多项式（2 个系数 -> 3 个字节，12 位 LSB-first）。ek = t_hat[0..K-1]
- // 字节 || rho；dk_pke = s_hat[0..K-1] 字节。遍历每个多项式的系数对。
+ // ---- Stage 2f: byteEncode12 serializer ----
+ // Pack each poly (2 coeffs -> 3 bytes, LSB-first 12-bit). ek = t_hat[0..K-1]
+ // bytes || rho; dk_pke = s_hat[0..K-1] bytes. Walk coeff pairs per poly.
reg [4:0] e_poly; // 0..2K-1: [0,K) = t_hat -> ek; [K,2K) = s_hat -> dk_pke
- reg [7:0] e_pair; // 0..127 多项式内的系数对
- reg [9:0] e_rho; // 0..31 rho 字节复制索引（ek 尾部）
- reg e_done; // 串行化完成
- // 源多项式槽位：ek 半部分为 t_hat[e_poly]，dk 半部分为 s_hat[e_poly-K]
+ reg [7:0] e_pair; // 0..127 coeff-pair within poly
+ reg [9:0] e_rho; // 0..31 rho byte copy index (ek tail)
+ reg e_done; // serialization complete
+ // source poly slot: t_hat[e_poly] for ek half, s_hat[e_poly-K] for dk half
wire e_is_dk = (e_poly >= {1'b0, k_r});
- wire [4:0] e_pidx = e_is_dk ? (e_poly - {1'b0, k_r}) : e_poly; // 目标内索引
- // 寄存的单端口读取（sd_bram 时序）。每周期一次 bank 读取：
- // ek 半部分读取 bank_t (t_hat)，dk 半部分读取 bank_se (s_hat)。一个
- // 系数对的两个系数在 4 周期微相 e_ph 中串行化：
- // 0 = 取回 c0
- // 1 = c0 就绪：写入 b0，保存 c0[11:8]，取回 c1
- // 2 = c1 就绪：写入 b1，保存 c1[11:4]
- // 3 = 写入 b2，推进到下一个系数对
- reg [1:0] e_ph; // 每系数对微相 0..3
- reg [3:0] e_c0_hi; // 为 b1 保存的 c0[11:8]
- reg [7:0] e_c1_hi; // 为 b2 保存的 c1[11:4]
+ wire [4:0] e_pidx = e_is_dk ? (e_poly - {1'b0, k_r}) : e_poly; // index within target
+ // Registered single-port read (sd_bram timing). One bank read per cycle:
+ // ek half reads bank_t (t_hat), dk half reads bank_se (s_hat). The two
+ // coeffs of a pair are serialized across a 4-cycle micro-phase e_ph:
+ // 0 = fetch c0
+ // 1 = c0 ready: write b0, save c0[11:8], fetch c1
+ // 2 = c1 ready: write b1, save c1[11:4]
+ // 3 = write b2, advance pair
+ reg [1:0] e_ph; // per-pair micro-phase 0..3
+ reg [3:0] e_c0_hi; // saved c0[11:8] for b1
+ reg [7:0] e_c1_hi; // saved c1[11:4] for b2
wire e_rd_half = (e_ph == 2'd1); // ph0 -> c0, ph1 -> c1
- wire [13:0] e_rd_full = e_pidx*256 + {e_pair, e_rd_half}; // ST_E 中 bse/bt 读取地址
- // dk 半部分读取 bank_se (bse_rd_data)，ek 半部分读取 bank_t (bt_rd_data)；
- // 两者在各自的 sd_bram 内寄存，具有相同的 1 周期延迟，因此
- // 这里选择上一周期呈现的地址对应的系数。
+ wire [13:0] e_rd_full = e_pidx*256 + {e_pair, e_rd_half}; // bse/bt rd addr in ST_E
+ // dk half reads bank_se (bse_rd_data), ek half reads bank_t (bt_rd_data);
+ // both registered inside their sd_bram with the same 1-cycle latency, so
+ // the coeff for the addr presented last cycle is selected here.
wire [11:0] e_rd_coeff = e_is_dk ? bse_rd_data : bt_rd_data;
- // byteEncode 写入目标存储器内的字节偏移：pair*3 + 字节索引。
- wire [11:0] e_base = e_pidx * 12'd384; // 多项式索引 *384（=128 对*3）
+ // byteEncode write byte offset within the target memory: pair*3 + byte index.
+ wire [11:0] e_base = e_pidx * 12'd384; // poly index *384 (=128 pairs*3)
wire [11:0] e_boff = e_base + {e_pair, 1'b0} + {2'b0, e_pair}; // pair*3
- wire [1:0] e_wb = e_ph - 2'd1; // ph1->字节0, ph2->字节1, ph3->字节2
+ wire [1:0] e_wb = e_ph - 2'd1; // ph1->byte0, ph2->byte1, ph3->byte2
wire [7:0] e_wbyte = (e_ph == 2'd1) ? e_rd_coeff[7:0] // b0 = c0[7:0]
: (e_ph == 2'd2) ? {e_rd_coeff[3:0], e_c0_hi} // b1 = {c1[3:0],c0[11:8]}
: e_c1_hi; // b2 = c1[11:4]
- wire [SAW-1:0] m_aslot = m_i*k_r + m_j; // A_hat[i][j] 槽位 = i*k + j
+ wire [SAW-1:0] m_aslot = m_i*k_r + m_j; // A_hat[i][j] slot = i*k + j
wire [SAW-1:0] m_sslot = slot_s_rt + m_j; // s_hat[j]
wire [SAW-1:0] m_eslot = slot_e_rt + m_i; // e_hat[i]
wire [SAW-1:0] m_tslot = slot_t_rt + m_i; // t_hat[i]
// ================================================================
- // E4: u[i] = INTT(sum_j A^T[i][j] o y_hat[j]) + e1[i]（Encaps ST_ENC_U）
- // 每行 i，三个子阶段 (u_sub)：
- // 0 MAC : sum_j A_hat[j][i] o y_hat[j]（转置）-> psum bank_t[UPSUM]
- // 1 INTT : INTT(psum) mode=1 原地在 bank_t[UPSUM] 中
- // 2 ADD : u[i][w] = psum[w] + e1[i][w] mod Q -> bank_se 相对 (K+i)（覆盖 e1）
- // y_hat (bank_se 0..K-1) 保留供 V 使用。重用共享的 u_pmul + u_ntt。
+ // E4: u[i] = INTT(sum_j A^T[i][j] o y_hat[j]) + e1[i] (Encaps ST_ENC_U)
+ // Per row i, three sub-phases (u_sub):
+ // 0 MAC : sum_j A_hat[j][i] o y_hat[j] (transpose) -> psum bank_t[UPSUM]
+ // 1 INTT: INTT(psum) mode=1 in place in bank_t[UPSUM]
+ // 2 ADD : u[i][w] = psum[w] + e1[i][w] mod Q -> bank_se rel (K+i) (over e1)
+ // y_hat (bank_se 0..K-1) survives for V. Reuses shared u_pmul + u_ntt.
// ================================================================
- localparam UPSUM = 10'd1; // NTT 域 psum 的 bank_t 相对槽位
- reg [2:0] u_row; // 输出行 i 0..K-1
+ localparam UPSUM = 10'd1; // bank_t rel slot for the NTT-domain psum
+ reg [2:0] u_row; // output row i 0..K-1
reg [1:0] u_sub; // 0=MAC 1=INTT 2=ADD
- // MAC 簿记（镜像 ST_M）
- reg [2:0] u_j; // 项列 j 0..K-1
- reg [8:0] u_ld; // 加载预读 0..256
- reg [7:0] u_oidx; // 累加索引 0..255
- reg u_loading; // 向 poly_mul 流式传输系数对
- reg u_pending; // 等待 poly_mul IDLE 再处理下一个 j
- reg u_j0q; // (u_j==0) 延迟 1 周期（首项=0 选择）
- // INTT 簿记（镜像 ST_N）
- reg [8:0] u_ridx; // INTT 加载预读 0..256
- reg [7:0] u_widx; // INTT 输出索引 0..255
- reg u_nvalid; // 向 ntt_core 馈送 psum (mode=1)
- reg u_nloading; // 呈现 INTT 加载地址
- reg u_npending; //（未使用的占位，保持对称）
- // ADD 簿记
- reg [8:0] u_aidx; // ADD 预读 0..256
- reg [7:0] u_awidx; // ADD 写入索引 0..255
- reg u_avalid; // ADD 写入使能流水线
- // MAC 地址：A^T -> A_hat[u_j][u_row] 槽位 = u_j*k+u_row; y_hat[u_j] 相对 u_j
+ // MAC bookkeeping (mirrors ST_M)
+ reg [2:0] u_j; // term col j 0..K-1
+ reg [8:0] u_ld; // load read-ahead 0..256
+ reg [7:0] u_oidx; // accumulate index 0..255
+ reg u_loading; // streaming pairs into poly_mul
+ reg u_pending; // wait poly_mul IDLE before next j
+ reg u_j0q; // (u_j==0) delayed 1 cyc (first-term=0 select)
+ // INTT bookkeeping (mirrors ST_N)
+ reg [8:0] u_ridx; // INTT load read-ahead 0..256
+ reg [7:0] u_widx; // INTT output index 0..255
+ reg u_nvalid; // feed psum to ntt_core (mode=1)
+ reg u_nloading; // presenting INTT load addresses
+ reg u_npending; // (unused placeholder for symmetry)
+ // ADD bookkeeping
+ reg [8:0] u_aidx; // ADD read-ahead 0..256
+ reg [7:0] u_awidx; // ADD write index 0..255
+ reg u_avalid; // ADD write enable pipeline
+ // MAC addresses: A^T -> A_hat[u_j][u_row] slot = u_j*k+u_row; y_hat[u_j] rel u_j
wire [SAW-1:0] u_aslot = u_j*k_r + u_row;
wire [13:0] u_pm_a_full = u_aslot*256 + u_ld[7:0]; // bank_a
wire [13:0] u_pm_b_full = u_j*256 + u_ld[7:0]; // bank_se y_hat[u_j]
wire [7:0] u_acc_radr = pm_vo ? (u_oidx + 8'd1) : u_oidx;
- wire [13:0] u_psum_full = UPSUM*256 + u_acc_radr; // bank_t psum 累加地址 (MAC)
- wire [11:0] u_acc_src = u_j0q ? 12'd0 : bt_rd_data; // 首项初始化 0
+ wire [13:0] u_psum_full = UPSUM*256 + u_acc_radr; // bank_t psum acc addr (MAC)
+ wire [11:0] u_acc_src = u_j0q ? 12'd0 : bt_rd_data; // first term init 0
wire [12:0] u_sum = {1'b0, u_acc_src} + {1'b0, pm_coeff};
- wire [11:0] u_accq = (u_sum >= 13'd3329) ? (u_sum - 13'd3329) : u_sum[11:0];
- // INTT 加载地址（psum 读取）+ 原地写回 bank_t[UPSUM]
+ wire [11:0] u_accq = (u_sum >= Q13) ? (u_sum - Q13) : u_sum[11:0];
+ // INTT load addr (psum read) + in-place writeback to bank_t[UPSUM]
wire [13:0] u_intt_rd = UPSUM*256 + u_ridx[7:0];
wire [13:0] u_intt_wr = UPSUM*256 + u_widx;
- // ADD：读取 psum (bank_t) + e1 (bank_se K+u_row)，写入 u 覆盖 e1（预读 1）
+ // ADD: read psum (bank_t) + e1 (bank_se K+u_row), write u over e1 (read-ahead 1)
wire [13:0] u_add_prd = UPSUM*256 + u_aidx[7:0]; // bank_t psum
- wire [13:0] u_add_e1rd = ({2'b0,k_r}+{2'b0,u_row})*256 + u_aidx[7:0]; // bank_se e1（领先）
- wire [13:0] u_add_uwr = ({2'b0,k_r}+{2'b0,u_row})*256 + u_awidx; // bank_se u（写入）
+ wire [13:0] u_add_e1rd = ({2'b0,k_r}+{2'b0,u_row})*256 + u_aidx[7:0]; // bank_se e1 (lead)
+ wire [13:0] u_add_uwr = ({2'b0,k_r}+{2'b0,u_row})*256 + u_awidx; // bank_se u (write)
wire [12:0] u_usum = {1'b0, bt_rd_data} + {1'b0, bse_rd_data}; // psum + e1
- wire [11:0] u_uq = (u_usum >= 13'd3329) ? (u_usum - 13'd3329) : u_usum[11:0];
+ wire [11:0] u_uq = (u_usum >= Q13) ? (u_usum - Q13) : u_usum[11:0];
// ================================================================
- // E6: v = INTT(sum_j t_hat[j] o y_hat[j]) + e2 + mu（Encaps ST_ENC_V）。
- // 重用 u_* MAC/INTT/ADD 机，u_row 固定为 0（单个输出
- // 多项式）。与 E4 的区别：
- // - MAC 从 bank_a 槽位 j*K（== u_aslot=u_j*K, u_row=0）读取 t_hat[j]，
- // 因此 bank_a MAC 地址不变。
- // - ADD 计算 psum + e2 + mu（而非 psum + e1）；e2 位于 bank_a 槽位
- // E2_ASLOT（在 ST_ENC_E2MV 中从 bank_t[0] 搬迁而来），v 原地写回
- // bank_t[UPSUM]。
- // - mu[w] = Decompress_1(m 比特 w) = m_r[w] ? 1665 : 0（1665 = round(Q/2)）。
- // u_row_max 限定行循环：U 为 K，V 为 1。
- localparam E2_ASLOT = 12'd1; // 存放搬迁后 e2 的 bank_a 槽位（绝不会与 t_hat 槽位冲突，K>=2）
+ // E6: v = INTT(sum_j t_hat[j] o y_hat[j]) + e2 + mu (Encaps ST_ENC_V).
+ // Reuses the u_* MAC/INTT/ADD machine with u_row tied to 0 (single output
+ // poly). Differences from E4:
+ // - MAC reads t_hat[j] from bank_a slot j*K (== u_aslot=u_j*K, u_row=0),
+ // so the bank_a MAC address is unchanged.
+ // - ADD computes psum + e2 + mu (not psum + e1); e2 lives in bank_a slot
+ // E2_ASLOT (relocated from bank_t[0] in ST_ENC_E2MV), v writes back to
+ // bank_t[UPSUM] in place.
+ // - mu[w] = Decompress_1(m bit w) = m_r[w] ? 1665 : 0 (1665 = round(Q/2)).
+ // u_row_max bounds the row loop: K for U, 1 for V.
+ localparam E2_ASLOT = 12'd1; // bank_a slot holding relocated e2 (never a t_hat slot, K>=2)
wire [2:0] u_row_max = (st == ST_ENC_V || st == ST_DEC_W) ? 3'd1 : k_r;
- // V-ADD：在 u_aidx 处读取 psum (bank_t UPSUM) + e2 (bank_a E2_ASLOT)（领先），
- // 通过写入索引 u_awidx 加上 mu，将 v 写入 bank_t[UPSUM] 的 u_awidx 位置。
- wire [13:0] u_v_e2rd = E2_ASLOT*256 + u_aidx[7:0]; // bank_a e2（领先）
- wire [13:0] u_vadd_wr = UPSUM*256 + u_awidx; // bank_t v（写入）
- wire [11:0] u_mu = m_r[u_awidx] ? 12'd1665 : 12'd0; // mu 按消息比特
+ // V-ADD: read psum (bank_t UPSUM) + e2 (bank_a E2_ASLOT) at u_aidx (lead),
+ // add mu (by write index u_awidx), write v to bank_t[UPSUM] at u_awidx.
+ wire [13:0] u_v_e2rd = E2_ASLOT*256 + u_aidx[7:0]; // bank_a e2 (lead)
+ wire [13:0] u_vadd_wr = UPSUM*256 + u_awidx; // bank_t v (write)
+ wire [11:0] u_mu = m_r[u_awidx] ? 12'd1665 : 12'd0; // mu by msg bit
wire [13:0] u_vsum = {2'b0, bt_rd_data} + {2'b0, ba_rd_data} + {2'b0, u_mu}; // psum+e2+mu
- wire [13:0] u_vsub1 = (u_vsum >= 14'd3329) ? (u_vsum - 14'd3329) : u_vsum;
- wire [11:0] u_vq = (u_vsub1 >= 14'd3329) ? (u_vsub1 - 14'd3329) : u_vsub1[11:0];
+ wire [13:0] u_vsub1 = (u_vsum >= Q14) ? (u_vsum - Q14) : u_vsum;
+ wire [11:0] u_vq = (u_vsub1 >= Q14) ? (u_vsub1 - Q14) : u_vsub1[11:0];
wire u_v_we = (st == ST_ENC_V) && (u_sub == 2'd2) && u_avalid;
// ================================================================
- // D3: w = v' - INTT(sum_j s_hat[j] o u_hat[j])（Decaps ST_DEC_W）。
- // 重用 u_* MAC/INTT 机（u_row 固定为 0，单个多项式）：
- // MAC : s_hat[j] (bank_a 槽位 j*K，== u_aslot, u_row=0) o u_hat[j]
- // (bank_se 相对 j，== u_pm_b_full) -> psum bank_t[UPSUM]。（与
- // Encaps V MAC 相同的寻址，后者读取 t_hat[j] o y_hat[j]。）
- // INTT: INTT(psum) 原地在 bank_t[UPSUM] 中。
- // SUB : w[i] = v'[i] - psum[i] mod Q。v' 在 bank_a[DEC_VASLOT]（D1）中，
- // psum 在 bank_t[UPSUM] 中；并行读取两者（类似于 V-ADD 读取
- // psum + e2）。w 原地写回 bank_t[UPSUM]。
- wire [13:0] u_w_vrd = DEC_VASLOT*256 + u_aidx[7:0]; // bank_a v'（领先）
- wire [13:0] u_wsub_wr = UPSUM*256 + u_awidx; // bank_t w（写入）
+ // D3: w = v' - INTT(sum_j s_hat[j] o u_hat[j]) (Decaps ST_DEC_W).
+ // Reuses the u_* MAC/INTT machine (u_row tied to 0, single poly):
+ // MAC : s_hat[j] (bank_a slot j*K, == u_aslot with u_row=0) o u_hat[j]
+ // (bank_se rel j, == u_pm_b_full) -> psum bank_t[UPSUM]. (same
+ // addressing as Encaps V MAC, which reads t_hat[j] o y_hat[j].)
+ // INTT: INTT(psum) in place in bank_t[UPSUM].
+ // SUB : w[i] = v'[i] - psum[i] mod Q. v' is in bank_a[DEC_VASLOT] (D1),
+ // psum in bank_t[UPSUM]; read both in parallel (like V-ADD reads
+ // psum + e2). w written back to bank_t[UPSUM] in place.
+ wire [13:0] u_w_vrd = DEC_VASLOT*256 + u_aidx[7:0]; // bank_a v' (lead)
+ wire [13:0] u_wsub_wr = UPSUM*256 + u_awidx; // bank_t w (write)
wire [12:0] u_wdiff = {1'b0, ba_rd_data} - {1'b0, bt_rd_data}; // v' - psum
- wire [11:0] u_wq = u_wdiff[12] ? (u_wdiff[11:0] + 12'd3329) : u_wdiff[11:0];
+ wire [11:0] u_wq = u_wdiff[12] ? (u_wdiff[11:0] + Q12) : u_wdiff[11:0];
wire u_w_we = (st == ST_DEC_W) && (u_sub == 2'd2) && u_avalid;
// ================================================================
- // D4: m' = byteEncode_1(Compress_1(w))（Decaps ST_DEC_MENC）。
- // Compress_1(x) = round(2x/Q) mod 2 = 1 当且仅当 x 在 (Q/4, 3Q/4]，即
- // 对 Q=3329 而言 832 < x <= 2496。将 256 位 LSB-first 打包为 32 字节。
- // w 从 bank_t[UPSUM] 读取（D3 的原地结果），每 2 周期 1 个系数。
- reg [7:0] men_idx; // 系数 0..255
- reg [1:0] men_ph; // 微相
+ // D4: m' = byteEncode_1(Compress_1(w)) (Decaps ST_DEC_MENC).
+ // Compress_1(x) = round(2x/Q) mod 2 = 1 iff x in (Q/4, 3Q/4], i.e.
+ // 832 < x <= 2496 for Q=3329. Pack 256 bits LSB-first into 32 bytes.
+ // w is read from bank_t[UPSUM] (in place from D3), 1 coeff per 2 cycles.
+ reg [7:0] men_idx; // coeff 0..255
+ reg [1:0] men_ph; // micro-phase
reg men_done;
- // D4 将恢复出的消息比特直接打包到 m_r 中（被 D6 的
- // V/mu 重加密复用，并在 dbg_mprime_o 上暴露）。
+ // D4 packs the recovered message bits directly into m_r (reused by D6's
+ // V/mu re-encrypt and exposed on dbg_mprime_o).
// ================================================================
- // D7: 隐式拒绝比较（Decaps ST_DEC_CMP）。
- // 遍历所有 ct 字节，并行读取 c'（ct_bram）和 c（c_in_bram）；
- // 将每个字节差异 OR 到 cmp_neq。最后一个字节后，ss = cmp_neq
- // ? K-bar : K'。遍历访问每个字节而不管早期差异
- //（恒定工作量/无提前退出），匹配常量时间规范意图。
- // ph0：呈现两个地址；ph1：比较寄存读取，累加。
- reg [10:0] cmp_idx; // 字节索引 0..ct_bytes
- reg cmp_ph; // 0=呈现地址, 1=比较
- reg cmp_neq; // 粘滞：任何字节曾不同
+ // D7: implicit-reject compare (Decaps ST_DEC_CMP).
+ // Walk all ct bytes, reading c' (ct_bram) and c (c_in_bram) in parallel;
+ // OR every byte difference into cmp_neq. After the last byte, ss = cmp_neq
+ // ? K-bar : K'. The walk visits every byte regardless of early differences
+ // (constant work / no early-out), matching the constant-time spec intent.
+ // ph0: present both addrs; ph1: compare registered reads, accumulate.
+ reg [10:0] cmp_idx; // byte index 0..ct_bytes
+ reg cmp_ph; // 0=present addr, 1=compare
+ reg cmp_neq; // sticky: any byte differed
reg cmp_done;
- reg dec_reject; // 末尾锁存的 cmp_neq（驱动 ss 选择）
+ reg dec_reject; // latched cmp_neq at end (drives ss select)
wire cmp_byte_neq = (ct_rd_data != cin_rd_data);
- wire [13:0] men_rd = UPSUM*256 + men_idx; // bank_t w 读取地址
+ wire [13:0] men_rd = UPSUM*256 + men_idx; // bank_t w read addr
wire men_w_bit = (bt_rd_data > 12'd832) && (bt_rd_data <= 12'd2496); // Compress_1
- // ST_ENC_E2MV：将 e2（bank_t 相对槽位 0，256 个系数）复制到 bank_a[E2_ASLOT]。
- // bank_t 读取领先 (em_ridx)，bank_a 写入延迟 1 周期 (em_we/em_widx)。
- reg [8:0] em_ridx; // 0..256 对 e2 的预读
- reg [7:0] em_widx; // bank_a 写入索引
- reg em_we; // 本周起正在写入一个系数
+ // ST_ENC_E2MV: copy e2 (bank_t rel slot 0, 256 coeffs) into bank_a[E2_ASLOT].
+ // bank_t read leads (em_ridx), bank_a write trails 1 cycle (em_we/em_widx).
+ reg [8:0] em_ridx; // 0..256 read-ahead over e2
+ reg [7:0] em_widx; // bank_a write index
+ reg em_we; // a coeff is being written this cycle
reg em_done;
- wire [13:0] em_rd = 14'd0 + em_ridx[7:0]; // bank_t 相对槽位 0 (e2)
+ wire [13:0] em_rd = 14'd0 + em_ridx[7:0]; // bank_t rel slot 0 (e2)
wire [11:0] em_wa = E2_ASLOT*256 + em_widx; // bank_a E2_ASLOT
// ================================================================
- // E5/E7: Compress_d + byteEncode_d -> 密文（Encaps ST_ENC_C1/C2）。
- // 每个系数：读取多项式系数 -> comp_decomp（mode 0 压缩, d=du/dv）->
- // 位打包器（LSB-first）-> 发射字节到 ct_bram。c1 = K 个 u 多项式
- // (d=du)，然后 c2 = 1 个 v 多项式 (d=dv)。每个多项式 = 256 个系数 -> 32*d
- // 字节（整），因此位缓冲区在每个多项式边界处排空。
- // 微相 cp_ph: 0 呈现系数地址；1 馈送 comp_decomp (cd_valid)；
- // 2 等待流水线；3 捕获压缩结果 + 累积比特；4..n 排空字节。
+ // E5/E7: Compress_d + byteEncode_d -> ciphertext (Encaps ST_ENC_C1/C2).
+ // Per coeff: read poly coeff -> comp_decomp (mode 0 compress, d=du/dv) ->
+ // bit-packer (LSB-first) -> emit bytes to ct_bram. c1 = K polys of u
+ // (d=du), then c2 = 1 poly of v (d=dv). Per poly = 256 coeffs -> 32*d
+ // bytes (whole), so the bit buffer empties at each poly boundary.
+ // micro-phase cp_ph: 0 present coeff addr; 1 feed comp_decomp (cd_valid);
+ // 2 drop valid pulse; 3 wait/capture cd_vo; 4..n drain bytes.
// ================================================================
wire cd_active = (st == ST_ENC_C1) || (st == ST_ENC_C2);
- reg [11:0] cd_coeff; // 呈现给 comp_decomp 的系数
- reg cd_valid; // 对 comp_decomp 的 1 周期脉冲
+ reg [11:0] cd_coeff; // coeff presented to comp_decomp
+ reg cd_valid; // 1-cyc pulse to comp_decomp
wire cd_ready;
- wire [11:0] cd_out; // 压缩值（低 d 位有效）
+ wire [11:0] cd_out; // compressed value (low d bits valid)
wire cd_vo;
- wire [4:0] cp_d = (st == ST_ENC_C2) ? dv_rt : du_rt; // 压缩宽度
- // comp_decomp 共享：Encaps C1/C2 压缩（mode 0）；Decaps DECOMP
- // 解压缩（mode 1）。在 DECOMP 中解码器馈送 dec_dc_coeff/valid/d。
+ wire [4:0] cp_d = (st == ST_ENC_C2) ? dv_rt : du_rt; // compress width
+ // comp_decomp is shared: Encaps C1/C2 compress (mode 0); Decaps DECOMP
+ // decompresses (mode 1). In DECOMP the decoder feeds dec_dc_coeff/valid/d.
wire cd_dec = (st == ST_DEC_DECOMP);
- wire cd_mode = cd_dec ? 1'b1 : 1'b0; // 1=解压缩
+ wire cd_mode = cd_dec ? 1'b1 : 1'b0; // 1=decompress
wire [4:0] cd_d_mux = cd_dec ? dec_dc_d : cp_d;
wire [11:0] cd_in_mux= cd_dec ? dec_dc_coeff : cd_coeff;
wire cd_vi_mux= cd_dec ? dec_dc_valid : cd_valid;
@@ -1037,66 +1040,68 @@ module mlkem_top #(
.valid_o(cd_vo),
.ready_i(1'b1)
);
- // 位打包器 / ct 遍历簿记
- reg [2:0] cp_poly; // c1: 0..K-1（u 行）；c2: 单个 v
- reg [7:0] cp_idx; // 多项式内系数 0..255
- reg [2:0] cp_ph; // 微相
- reg [24:0] cp_buf; // 位累加器（LSB-first）
- reg [5:0] cp_nbits; // cp_buf 中有效位数
- reg [11:0] cp_wa; // ct_bram 字节写入地址（先跑 c1 再 c2）
- reg cp_done; // 串行化完成（该区域）
- // 系数来源：c1 (ST_ENC_C1) 读取 bank_se 相对 (K+cp_poly) 中的 u[cp_poly]；
- // c2 (ST_ENC_C2) 读取 bank_t 相对槽位 UPSUM 中的 v（单个多项式, cp_poly=0）。
+ // bit-packer / ct walk bookkeeping
+ reg [2:0] cp_poly; // c1: 0..K-1 (u rows); c2: single v
+ reg [7:0] cp_idx; // coeff 0..255 within poly
+ reg [2:0] cp_ph; // micro-phase
+ reg [24:0] cp_buf; // bit accumulator (LSB-first)
+ reg [5:0] cp_nbits; // valid bits in cp_buf
+ reg [11:0] cp_wa; // ct_bram byte write address (runs c1 then c2)
+ reg cp_done; // serialization complete (this region)
+ // coeff source: c1 (ST_ENC_C1) reads u[cp_poly] in bank_se rel (K+cp_poly);
+ // c2 (ST_ENC_C2) reads v in bank_t rel slot UPSUM (single poly, cp_poly=0).
wire [13:0] cp_se_full = ({2'b0,k_r}+{2'b0,cp_poly})*256 + cp_idx; // bank_se u[cp_poly]
wire [13:0] cp_bt_full = UPSUM*256 + cp_idx; // bank_t v (UPSUM)
- // 打包器泛化（E5 c1 vs E7 c2）：
- // 系数来源：c1 = bank_se (u)，c2 = bank_t (v)
- // 多项式数量：c1 = K（u 行），c2 = 1（单个 v）
- // 位宽度 cp_d 已在上面根据 C2 或 C1 声明为 dv_rt 或 du_rt。
+ // packer generalization (E5 c1 vs E7 c2):
+ // coeff source: c1 = bank_se (u), c2 = bank_t (v)
+ // poly count: c1 = K (u rows), c2 = 1 (single v)
+ // bit width cp_d already = dv_rt for C2 else du_rt (declared above).
wire [11:0] cp_coeff_src = (st == ST_ENC_C2) ? bt_rd_data : bse_rd_data;
wire [2:0] cp_poly_max = (st == ST_ENC_C2) ? 3'd1 : k_r;
// ================================================================
- // D1: K-PKE byteDecode_d + 解压缩（解密 c1/c2 -> u'/v'）。
- // C1/C2 打包器的逆向：遍历 c_in_bram 字节，LSB-first 累积
- // 到位缓冲区，提取 d 位符号，馈送 comp_decomp（mode=1
- // 解压缩），并将每个结果 (mod q) 写入 bank。
- // c1 = K 个多项式, d=du -> u'[i] 在 bank_se 相对 i (0..K-1) 中
- // c2 = 1 个多项式, d=dv -> v' 在 bank_a 槽位 DEC_VASLOT 中（使 D3 的 w-SUB 能够
- // 并行读取 psum (bank_t) 和 v' (bank_a)，类似于 Encaps V-ADD）。
- // 微相 dec_ph:
- // 0 呈现 c_in 字节地址；1 捕获字节，追加 8 位到 buf；
- // while (nbits>=d): 提取 d 位符号 -> comp_decomp（解压缩）；
- // 2 等待流水线；3 cd_vo -> 将系数写入 bank，推进系数索引。
+ // D1: K-PKE byteDecode_d + Decompress (decrypt c1/c2 -> u'/v').
+ // Reverse of the C1/C2 packer: walk c_in_bram bytes, accumulate LSB-first
+ // into a bit buffer, extract d-bit symbols, feed comp_decomp (mode=1
+ // decompress), and write each result (mod q) to a bank.
+ // c1 = K polys, d=du -> u'[i] in bank_se rel i (0..K-1)
+ // c2 = 1 poly, d=dv -> v' in bank_t rel slot DEC_VSLOT
+ // c1 = K polys, d=du -> u'[i] in bank_se rel i (0..K-1)
+ // c2 = 1 poly, d=dv -> v' in bank_a slot DEC_VASLOT (so D3's w-SUB can
+ // read psum (bank_t) and v' (bank_a) in parallel, like Encaps V-ADD).
+ // micro-phase dec_ph:
+ // 0 present c_in byte addr; 1 capture byte, append 8 bits to buf;
+ // while (nbits>=d): extract d-bit symbol -> comp_decomp (decompress);
+ // 2 wait pipe; 3 cd_vo -> write coeff to bank, advance coeff index.
// ================================================================
- localparam DEC_VASLOT = 12'd1; // v' 的 bank_a 槽位（空闲: s_hat 在 j*K, 槽位 1 未用, K>=2）
- reg [2:0] dec_poly; // c1: 0..K-1; c2: 单个 (=K 标记)
- reg [7:0] dec_cidx; // 多项式内系数 0..255（馈送侧）
- reg [7:0] dec_widx; // 多项式内系数 0..255（写回侧，延迟于流水线）
- reg [1:0] dec_ph; // 微相
- reg [11:0] dec_ba; // c_in_bram 字节读取地址（先 c1 后 c2）
- reg [24:0] dec_buf; // 位累加器（LSB-first）
- reg [5:0] dec_nbits; // dec_buf 中有效位数
- reg dec_dc_done; // D1 完成（c1 和 c2 均已解码）
- reg dec_in_c2; // 0 = 正在解码 c1 (u'), 1 = 正在解码 c2 (v')
- // comp_decomp 馈送（解压缩）：提取出的 d 位符号。
+ localparam DEC_VASLOT = 12'd1; // bank_a slot for v' (free: s_hat at j*K, slot 1 unused K>=2)
+ reg [2:0] dec_poly; // c1: 0..K-1; c2: single (=K marker)
+ reg [7:0] dec_cidx; // coeff 0..255 within poly (feed side)
+ reg [7:0] dec_widx; // coeff 0..255 within poly (writeback side, lags pipe)
+ reg [1:0] dec_ph; // micro-phase
+ reg [11:0] dec_ba; // c_in_bram byte read address (c1 then c2)
+ reg [24:0] dec_buf; // bit accumulator (LSB-first)
+ reg [5:0] dec_nbits; // valid bits in dec_buf
+ reg dec_dc_done; // D1 complete (both c1 and c2 decoded)
+ reg dec_in_c2; // 0 = decoding c1 (u'), 1 = decoding c2 (v')
+ // comp_decomp feed (decompress): the extracted d-bit symbol.
reg [11:0] dec_dc_coeff;
reg dec_dc_valid;
- wire [4:0] dec_dc_d = dec_in_c2 ? dv_rt : du_rt; // c2 用 dv, c1 用 du
- //（c_in_bram 读取地址由 cin_rd_addr_r 驱动，在遍历器中设为 dec_ba）
+ wire [4:0] dec_dc_d = dec_in_c2 ? dv_rt : du_rt; // c2 uses dv, c1 uses du
+ // (c_in_bram read addr driven by cin_rd_addr_r, set to dec_ba in the walker)
reg pm_valid;
wire pm_ready;
wire [11:0] pm_coeff;
wire pm_vo;
- // pm_a: A_hat[i][j] 在 bank_a（绝对槽位 m_aslot）。pm_b: s_hat[j] 在
- // bank_se（相对槽位 = m_sslot - slot_s_rt = m_j）。m_ld 是预读
- // 指针；两个 bank 读取来自各自的 sd_bram（寄存，1 周期
- // 延迟）并在 1 周期后馈送 poly_mul（pm_valid 延迟 1 周期）。
+ // pm_a: A_hat[i][j] in bank_a (abs slot m_aslot). pm_b: s_hat[j] in
+ // bank_se (relative slot = m_sslot - slot_s_rt = m_j). m_ld is a read-ahead
+ // pointer; both bank reads come from their sd_bram (registered, 1-cycle
+ // latency) and feed poly_mul one cycle later (pm_valid delayed 1 cyc).
wire [13:0] pm_a_full = m_aslot*256 + m_ld[7:0];
wire [13:0] pm_b_full = m_j*256 + m_ld[7:0];
- wire [11:0] pm_a_in = ba_rd_data; // bank_a sd_bram 寄存读取
- wire [11:0] pm_b_in = bse_rd_data; // bank_se sd_bram 寄存读取（加载阶段）
+ wire [11:0] pm_a_in = ba_rd_data; // bank_a sd_bram registered read
+ wire [11:0] pm_b_in = bse_rd_data; // bank_se sd_bram registered read (load phase)
poly_mul_sync u_pmul (
.clk(clk), .rst_n(rst_n),
@@ -1109,47 +1114,47 @@ module mlkem_top #(
.ready_i(1'b1)
);
- // 累加器来源：首项时 (j==0) 为 e_hat[i]，否则为运行的 t_hat[i]。
- // e_hat[i] 在 bank_se 相对槽位 (slot_e_rt-slot_s_rt + m_i) = K+m_i。
- // t_hat[i] 在 bank_t 相对槽位 m_i。两者读取均来自各自的
- // sd_bram (bse_rd_data / bt_rd_data)，具有相同的 1 周期延迟。
- // 寄存预读：呈现下一个 pm_vo 将消耗的索引
+ // accumulator source: e_hat[i] for first term (j==0), else running t_hat[i].
+ // e_hat[i] lives in bank_se at relative slot (slot_e_rt-slot_s_rt + m_i) = K+m_i.
+ // t_hat[i] lives in bank_t at relative slot m_i. Both reads come from their
+ // sd_bram (bse_rd_data / bt_rd_data) with the same 1-cycle latency.
+ // Registered read-ahead: present the index the NEXT pm_vo will consume
// m_acc_radr = pm_vo ? m_oidx+1 : m_oidx
- // j 选择应用于寄存输出（m_jq = (m_j==0)
- // 延迟 1 周期以对齐读取延迟）。RMW 读旧值保持：读取
- // 地址 (m_oidx+1) 领先写入地址 (m_oidx)。节拍 CALC/C0/C1。
+ // The j-select is applied on the registered outputs (m_jq = (m_j==0)
+ // delayed 1 cyc to align with the read latency). RMW read-old holds: read
+ // addr (m_oidx+1) leads write addr (m_oidx). Cadence CALC/C0/C1.
wire [7:0] m_acc_radr = pm_vo ? (m_oidx + 8'd1) : m_oidx;
- wire [13:0] m_eacc_full = ({2'b0, k_r} + {2'b0, m_i})*256 + m_acc_radr; // K+m_i（累加时的 bse_rd_addr）
+ wire [13:0] m_eacc_full = ({2'b0, k_r} + {2'b0, m_i})*256 + m_acc_radr; // K+m_i (bse_rd_addr in acc)
wire [13:0] m_tacc_full = m_i*256 + m_acc_radr; // bt_rd_addr
- reg m_jq; // (m_j==0) 延迟 1 周期以匹配读取延迟
- // 选中的累加器来源与 pm_coeff 对齐
+ reg m_jq; // (m_j==0) delayed 1 cyc to match read latency
+ // selected accumulator source aligned with pm_coeff
wire [11:0] m_acc_src = m_jq ? bse_rd_data : bt_rd_data;
- // (a + b) mod Q（均 < Q，和 < 2Q）：一次条件减法
+ // (a + b) mod Q (both < Q, sum < 2Q): one conditional subtract
wire [12:0] m_sum = {1'b0, m_acc_src} + {1'b0, pm_coeff};
- wire [11:0] m_accq = (m_sum >= 13'd3329) ? (m_sum - 13'd3329) : m_sum[11:0];
+ wire [11:0] m_accq = (m_sum >= Q13) ? (m_sum - Q13) : m_sum[11:0];
always @(*) begin
st_next = st;
case (st)
ST_IDLE: if (start_i) st_next = (op_i == 2'd2) ? ST_DEC_LOAD :
(op_i == 2'd1) ? ST_ENC_H : ST_G;
- // D0: dk/c 解析后稳定，然后 D1 解压缩。D1 在 u'/v'
- // 计算完成后进入 DONE，使该阶段可进行 debug 检查。
+ // D0: settle after dk/c parse, then D1 Decompress. D1 lands in DONE
+ // once u'/v' are computed so the stage can be dbg-checked.
ST_DEC_LOAD: st_next = ST_DEC_DECOMP;
ST_DEC_DECOMP: if (dec_dc_done) st_next = ST_DEC_SDEC;
- // D2: s_hat 解码（重用 TDEC 机），然后 u_hat = NTT(u')。D2
- // 在 u_hat 就绪后进入 DONE，使该阶段可进行 debug 检查。
+ // D2: s_hat decode (reuse TDEC machine), then u_hat = NTT(u'). D2
+ // lands in DONE once u_hat is ready so the stage can be dbg-checked.
ST_DEC_SDEC: if (td_done) st_next = ST_DEC_NTT;
ST_DEC_NTT: if (n_slot >= {2'b0, k_r}) st_next = ST_DEC_W;
- // D3: w = v' - INTT(sum_j s_hat[j] o u_hat[j])。单个输出多项式
- // (u_row 0..0)；MAC->INTT->SUB，然后 D4 编码 m'。
+ // D3: w = v' - INTT(sum_j s_hat[j] o u_hat[j]). Single output poly
+ // (u_row 0..0); MAC->INTT->SUB, then D4 encodes m'.
ST_DEC_W: if (u_row >= 3'd1) st_next = ST_DEC_MENC;
- ST_DEC_MENC: if (men_done) st_next = ST_DEC_G;
- // D5: (K',r') = G(m'||h) 单块，然后 K-bar = J(z||c) 多块。
- ST_DEC_G: if (sha3_vo) st_next = ST_DEC_J;
- // D5 J 完成 -> D6 重加密: c' = K-PKE.Encrypt(ek_pke, m', r')。
- // 重用整个 Encaps 流水线（rho 加载 -> A -> C -> ... -> C2）。
- // r' 在 r_r 中（CBD 种子），m' 在 m_r 中（V/mu），ek_pke 在 ek_bram 中。
+ ST_DEC_MENC: if (men_done && dj_done) st_next = ST_DEC_G;
+ // D5: (K',r') = G(m'||h) single-block. K-bar = J(z||c) runs in a
+ // background SHAKE256 walker after D1, so the main FSM only waits
+ // for it at the D4->D5 boundary if it somehow has not finished.
+ ST_DEC_G: if (sha3_vo) st_next = ST_ENC_LOAD;
+ // Legacy state kept unused; background J normally skips it.
ST_DEC_J: if (dj_done) st_next = ST_ENC_LOAD;
ST_G: if (sha3_vo) st_next = ST_A;
ST_A: if (a_pair >= kk_rt) st_next = ST_C;
@@ -1160,19 +1165,19 @@ module mlkem_top #(
ST_H: if (h_phase == 2'd3) st_next = ST_DONE;
// ---- Encaps ----
ST_ENC_H: if (h_phase == 2'd3) st_next = ST_ENC_G;
- ST_ENC_G: if (sha3_vo) st_next = ST_ENC_LOAD; // (K,r) 已捕获
- ST_ENC_LOAD: if (rl_idx >= 6'd33) st_next = ST_ENC_A; // rho 已加载
+ ST_ENC_G: if (sha3_vo) st_next = ST_ENC_LOAD; // (K,r) captured
+ ST_ENC_LOAD: if (rl_idx >= 6'd33) st_next = ST_ENC_A; // rho loaded
ST_ENC_A: if (a_pair >= kk_rt) st_next = ST_ENC_C;
- ST_ENC_C: if (c_poly >= {k_r, 1'b1}) st_next = ST_ENC_N; // 2K+1 个多项式完成
- ST_ENC_N: if (n_slot >= {2'b0, k_r}) st_next = ST_ENC_U; // K 个槽位 (y_hat)
- ST_ENC_U: if (u_row >= k_r) st_next = ST_ENC_C1; // u[0..K-1] 完成
- ST_ENC_C1: if (cp_done) st_next = ST_ENC_TDEC; // E5 完成 -> E6
- ST_ENC_TDEC: if (td_done) st_next = ST_ENC_E2MV; // t_hat 已解码 -> 搬迁 e2
- ST_ENC_E2MV: if (em_done) st_next = ST_ENC_V; // e2 已搬迁 -> 计算 v
- ST_ENC_V: if (u_row >= 3'd1) st_next = ST_ENC_C2; // E6: v 完成 -> E7
+ ST_ENC_C: if (c_poly >= {k_r, 1'b1}) st_next = ST_ENC_N; // 2K+1 polys done
+ ST_ENC_N: if (n_slot >= {2'b0, k_r}) st_next = ST_ENC_U; // K slots (y_hat)
+ ST_ENC_U: if (u_row >= k_r) st_next = ST_ENC_C1; // u[0..K-1] done
+ ST_ENC_C1: if (cp_done) st_next = ST_ENC_TDEC; // E5 done -> E6
+ ST_ENC_TDEC: if (td_done) st_next = ST_ENC_E2MV; // t_hat decoded -> relocate e2
+ ST_ENC_E2MV: if (em_done) st_next = ST_ENC_V; // e2 relocated -> compute v
+ ST_ENC_V: if (u_row >= 3'd1) st_next = ST_ENC_C2; // E6: v done -> E7
ST_ENC_C2: if (cp_done) st_next = (op_r == 2'd2) ? ST_DEC_CMP : ST_DONE;
- // D7: 遍历 ct 字节比较 c'（ct_bram）与 c（c_in_bram）；当
- // 最后一个字节比较完毕后，选择 ss = (c'==c) ? K' : K-bar 并结束。
+ // D7: walk ct bytes comparing c' (ct_bram) vs c (c_in_bram); when the
+ // last byte is compared, select ss = (c'==c) ? K' : K-bar and finish.
ST_DEC_CMP: if (cmp_done) st_next = ST_DONE;
ST_DONE: st_next = ST_IDLE;
default: st_next = ST_IDLE;
@@ -1302,6 +1307,7 @@ module mlkem_top #(
cmp_neq <= 1'b0;
cmp_done <= 1'b0;
dec_reject <= 1'b0;
+ dj_active <= 1'b0;
dj_blk <= 4'd0;
dj_byte <= 8'd0;
dj_phase <= 2'd0;
@@ -1322,57 +1328,59 @@ module mlkem_top #(
end else begin
st <= st_next;
- // BRAM write-enables 默认低；写入字节时脉冲为高。
+ // BRAM write-enables default low; pulsed where a byte is written.
ek_we <= 1'b0;
dkp_we <= 1'b0;
- td_we <= 1'b0; // TDEC bank_a 写入默认低
- ct_we <= 1'b0; // ct_bram 字节写入默认低 (E5/E7)
- em_we <= 1'b0; // e2 搬迁 bank_a 写入默认低 (E6)
+ td_we <= 1'b0; // TDEC bank_a write default low
+ ct_we <= 1'b0; // ct_bram byte write default low (E5/E7)
+ em_we <= 1'b0; // e2-relocate bank_a write default low (E6)
- // 从 IDLE 进入时启动：KeyGen 启动 G；Encaps 捕获
- // op/m 并武装 H(ek) 机（ST_ENC_H 重用 ST_H FSM）；
- // Decaps 捕获 op（dk/c 已流式输入；z/h 在加载时捕获）。
+ // Kick off when entering from IDLE: KeyGen starts G; Encaps captures
+ // op/m and arms the H(ek) machinery (ST_ENC_H reuses the ST_H FSM);
+ // Decaps captures op (dk/c already streamed; z/h captured at load).
if (st == ST_IDLE && start_i) begin
- k_r <= k_i; // 捕获运行时 ML-KEM 参数
+ k_r <= k_i; // capture runtime ML-KEM param
op_r <= op_i;
if (op_i == 2'd1) begin
- m_r <= msg_i; // 捕获 Encaps 消息
- // 武装 H(ek)（与 ST_E->ST_H 武装设置的字段相同）
+ m_r <= msg_i; // capture Encaps message
+ // arm H(ek) (same fields the ST_E->ST_H arming sets)
h_blk <= 3'd0;
h_byte <= 8'd0;
- h_phase <= 2'd0; // 组装
+ h_phase <= 2'd0; // assemble
h_mbvalid<= 1'b0;
h_mblast <= 1'b0;
- h_ack <= 1'b1; // 准备好消费最终摘要
+ h_ack <= 1'b1; // ready to consume final digest
h_wb_vld <= 1'b0;
end else if (op_i == 2'd2) begin
- // Decaps: dk_pke/ek_pke 已在 BRAM 中；z/H(ek) 在
- // 加载时捕获到 z_r/hek_r（见下方）。D0 无需其他武装。
+ // Decaps: dk_pke/ek_pke already in BRAM; z/H(ek) captured into
+ // z_r/hek_r during load (below). Nothing else to arm in D0.
+ dj_active <= 1'b0;
+ dj_done <= 1'b0;
end else begin
sha3_valid <= 1'b1;
sha3_ack <= 1'b1;
end
end
- // Decaps dk 加载：将 H(ek) 和 z 字节区域捕获到寄存器中
- // 随着字节流式输入（dk_pke/ek_pke 区域通过写入多路复用器进入 BRAM）。
- // 区域 i 中的字节落在 [8i +: 8] 中。使用
- // 实时的 k_i 边界（预加载期间 k_r 尚未捕获）。
+ // Decaps dk load: capture the H(ek) and z byte regions into registers
+ // as they stream in (the dk_pke/ek_pke regions go to BRAM via the
+ // write muxes). Byte i within a region lands in [8i +: 8]. Uses the
+ // LIVE k_i boundaries (k_r not yet captured during preload).
if (dk_in_we) begin
if (dk_in_addr >= dk_ek_end_ld && dk_in_addr < dk_hek_end_ld)
hek_r[(dk_in_addr - dk_ek_end_ld)*8 +: 8] <= dk_in_byte; // H(ek)
else if (dk_in_addr >= dk_hek_end_ld)
z_r[(dk_in_addr - dk_hek_end_ld)*8 +: 8] <= dk_in_byte; // z
end
- // valid 被接受后撤销
+ // Drop valid once accepted
if (sha3_valid && sha3_ready) sha3_valid <= 1'b0;
- // G 完成时捕获 rho/sigma；武装 A 阶段
+ // Capture rho/sigma when G completes; arm A stage
if (st == ST_G && sha3_vo) begin
- rho_r <= sha3_hash[255:0]; // rho = G 输出字节 0..31
- sigma_r <= sha3_hash[511:256]; // sigma = 字节 32..63
+ rho_r <= sha3_hash[255:0]; // rho = G output bytes 0..31
+ sigma_r <= sha3_hash[511:256]; // sigma = bytes 32..63
sha3_ack <= 1'b0;
- snt_valid <= 1'b1; // 启动第一个 SampleNTT
+ snt_valid <= 1'b1; // start first SampleNTT
snt_ack <= 1'b1;
a_pair <= 5'd0;
a_i <= 3'd0;
@@ -1381,21 +1389,21 @@ module mlkem_top #(
a_busy <= 1'b0;
end
- // ---- ST_A / ST_ENC_A: 驱动 SampleNTT, 每对存储 256 个系数 ----
+ // ---- ST_A / ST_ENC_A: drive SampleNTT, store 256 coeffs per pair ----
if (st == ST_A || st == ST_ENC_A) begin
- // 该对的请求被接受后标记忙碌
+ // mark busy once this pair's request accepted
if (snt_valid && snt_ready) begin
snt_valid <= 1'b0;
a_busy <= 1'b1;
end
- // 仅在忙碌时存储每个输出系数（忽略前一个多项式的
- // 过期尾系数）。bank_a 写入本身由上面的组合逻辑
- // ba_we/ba_wa/ba_wd assign 完成；这里我们只推进
- // 写入索引和 (i,j) 簿记。
+ // store each output coefficient only while busy (ignore stale
+ // last coeff from prior poly). The bank_a write itself is the
+ // combinational ba_we/ba_wa/ba_wd assigns above; here we only
+ // advance the write index and (i,j) bookkeeping.
if (a_busy && snt_vo && snt_ack) begin
if (snt_last) begin
- // 完成此多项式；按行主序推进 (i,j)
+ // finished this poly; advance (i,j) in row-major order
a_pair <= a_pair + 5'd1;
a_widx <= 8'd0;
a_busy <= 1'b0;
@@ -1405,7 +1413,7 @@ module mlkem_top #(
a_j <= 3'd0;
a_i <= a_i + 3'd1;
end
- // 如果还有更多对，启动下一个 SampleNTT
+ // start next SampleNTT if more pairs remain
if (a_pair + 5'd1 < kk_rt) snt_valid <= 1'b1;
end else begin
a_widx <= a_widx + 8'd1;
@@ -1413,7 +1421,7 @@ module mlkem_top #(
end
end
- // A 完成时武装 C 阶段（KeyGen ST_A 或 Encaps ST_ENC_A）
+ // Arm C stage when A finishes (KeyGen ST_A or Encaps ST_ENC_A)
if ((st == ST_A && st_next == ST_C) ||
(st == ST_ENC_A && st_next == ST_ENC_C)) begin
cbd_valid <= 1'b1;
@@ -1423,9 +1431,9 @@ module mlkem_top #(
c_busy <= 1'b0;
end
- // ---- ST_C / ST_ENC_C: 驱动 CBD, 每个多项式存储 256 个 mod-q 系数 ----
- // KeyGen: 2K 个多项式 (s,e)。Encaps: 2K+1 个多项式 (y,e1,e2)。循环边界
- // 不同：KeyGen 在 c_poly+1 < 2K 时重启；Encaps 在 < 2K+1 时重启。
+ // ---- ST_C / ST_ENC_C: drive CBD, store 256 mod-q coeffs per poly ----
+ // KeyGen: 2K polys (s,e). Encaps: 2K+1 polys (y,e1,e2). Loop bound
+ // differs: KeyGen restarts while c_poly+1 < 2K; Encaps while < 2K+1.
if (st == ST_C || st == ST_ENC_C) begin
if (cbd_valid && cbd_ready) begin
cbd_valid <= 1'b0;
@@ -1437,7 +1445,7 @@ module mlkem_top #(
c_poly <= c_poly + 3'd1;
c_widx <= 8'd0;
c_busy <= 1'b0;
- // 当还有更多多项式时重启下一个（状态相关边界）
+ // restart for next poly while more remain (state-dependent bound)
if (st == ST_ENC_C) begin
if (c_poly + 3'd1 < {k_r, 1'b1}) cbd_valid <= 1'b1; // < 2K+1
end else begin
@@ -1449,9 +1457,9 @@ module mlkem_top #(
end
end
- // C 完成时武装 N 阶段：启动槽位 S0 的加载。n_ridx 是
- // 预读指针；bank_se 读取在 sd_bram 内寄存 (bse_rd_data)
- // 并在 1 周期后馈送 ntt_core，因此 valid 从低开始（启动）。
+ // Arm N stage when C finishes: prime load of slot S0. n_ridx is a
+ // read-ahead pointer; bank_se read is registered inside sd_bram (bse_rd_data)
+ // and fed to ntt_core one cycle later, so valid starts low (priming).
if ((st == ST_C && st_next == ST_N) ||
(st == ST_ENC_C && st_next == ST_ENC_N) ||
(st == ST_DEC_SDEC && st_next == ST_DEC_NTT)) begin
@@ -1459,43 +1467,43 @@ module mlkem_top #(
n_ridx <= 9'd0;
n_widx <= 8'd0;
n_valid <= 1'b0;
- n_loading <= 1'b1; // 开始呈现加载地址
+ n_loading <= 1'b1; // begin presenting load addresses
n_pending <= 1'b0;
end
- // ---- ST_N / ST_ENC_N / ST_DEC_NTT: 原地前向 NTT。KeyGen: 2K
- // 槽位 (s,e)。Encaps: K 槽位 (y)。Decaps D2: K 槽位 (u' -> u_hat)。----
+ // ---- ST_N / ST_ENC_N / ST_DEC_NTT: forward NTT in place. KeyGen: 2K
+ // slots (s,e). Encaps: K slots (y). Decaps D2: K slots (u' -> u_hat). ----
if (st == ST_N || st == ST_ENC_N || st == ST_DEC_NTT) begin
- // 槽位计数边界：KeyGen 为 2K，Encaps 为 K
- //（见下面 n_slot_max）；两种方式的 LOAD/OUTPUT 节拍相同。
+ // slot-count bound: 2K for KeyGen, K for Encaps
+ // (n_slot_max below); same LOAD/OUTPUT cadence either way.
if (n_loading) begin
if (n_ridx == 9'd256) begin
- // 本周起已消耗第 256 个系数；停止呈现地址
+ // 256th coeff consumed this cycle; stop presenting addr
n_loading <= 1'b0;
n_valid <= 1'b0;
end else begin
n_ridx <= n_ridx + 9'd1;
- n_valid <= 1'b1; // 上一周期呈现的数据有效
+ n_valid <= 1'b1; // data presented last cycle is valid
end
end
- // OUTPUT 阶段：收集 256 个结果，写回相同槽位。
+ // OUTPUT phase: collect 256 results, write back to same slot.
if (ntt_vo) begin
- n_widx <= n_widx + 8'd1; // 最后之后 255->0 回绕
+ n_widx <= n_widx + 8'd1; // wraps 255->0 after last
end
- // 当 ntt_core 返回 DONE 时槽位完成
+ // Slot complete when ntt_core returns to DONE
if (ntt_done) begin
if (n_slot + 3'd1 < n_slot_max) begin
n_slot <= n_slot + 3'd1;
n_widx <= 8'd0;
- n_pending <= 1'b1; // 等待一个周期让 core 进入 IDLE
+ n_pending <= 1'b1; // wait one cycle for core IDLE
end else begin
- n_slot <= n_slot + 3'd1; // == n_slot_max -> 退出
+ n_slot <= n_slot + 3'd1; // == n_slot_max -> exit
end
end
- // core 回到 IDLE 后启动下一个槽位的加载（重新启动）
+ // Kick next slot's load once core is back IDLE (re-prime)
if (n_pending && ntt_ready && !ntt_done) begin
n_ridx <= 9'd0;
n_valid <= 1'b0;
@@ -1504,9 +1512,9 @@ module mlkem_top #(
end
end
- // N 完成时武装 M 阶段：启动首对 (i=0,j=0) poly_mul 加载。
- // m_ld 预读指针；bank_a/bank_se sd_bram 读取在 1 周期后
- // 到达，pm_valid 在地址呈现一周期后断言。
+ // Arm M stage when N finishes: prime first (i=0,j=0) poly_mul load.
+ // m_ld read-ahead pointer; bank_a/bank_se sd_bram reads land 1 cyc
+ // later, pm_valid asserted one cycle after an address is presented.
if (st == ST_N && st_next == ST_M) begin
m_i <= 2'd0;
m_j <= 2'd0;
@@ -1519,40 +1527,40 @@ module mlkem_top #(
// ---- ST_M: t_hat[i] = e_hat[i] + sum_j A[i][j] o s_hat[j] ----
if (st == ST_M) begin
- // 累加器读取来自 sd_bram（bse_rd_data e_hat /
- // bt_rd_data t_hat）。m_jq 将 j 选择与 1 周期
- // 读取延迟对齐。（bse_rd_addr 通过 m_loading 在加载和累加之间多路复用。）
+ // accumulator reads come from sd_bram (bse_rd_data e_hat /
+ // bt_rd_data t_hat). m_jq aligns the j-select with the 1-cycle
+ // read latency. (bse_rd_addr muxes load vs acc by m_loading.)
m_jq <= (m_j == 2'd0);
- // LOAD：通过 sd_bram 读取端口（m_loading 期间
- // ba_rd_addr=pm_a_full, bse_rd_addr=pm_b_full）向 bank_a/bank_se 呈现预读地址；
- // ba_rd_data/bse_rd_data 在下一个周期到达并
- // 由 poly_mul (pm_valid) 消耗。poly_mul 在整个 LOAD 期间保持 ready 高电平，
- // 因此固定的 1 周期偏移足够。
+ // LOAD: present read-ahead addr to bank_a/bank_se via their
+ // sd_bram read ports (ba_rd_addr=pm_a_full, bse_rd_addr=pm_b_full
+ // while m_loading); ba_rd_data/bse_rd_data land next cycle and
+ // are consumed by poly_mul (pm_valid). poly_mul holds ready high
+ // through LOAD, so a fixed 1-cycle skew suffices.
if (m_loading) begin
if (m_ld == 9'd256) begin
- pm_valid <= 1'b0; // 本周起已消耗第 256 对
+ pm_valid <= 1'b0; // 256th pair consumed this cycle
m_loading <= 1'b0;
m_ld <= 9'd0;
m_oidx <= 8'd0;
end else begin
m_ld <= m_ld + 9'd1;
- pm_valid <= 1'b1; // 上一周期呈现的系数对有效
+ pm_valid <= 1'b1; // pair presented last cycle is valid
end
end
- // ACCUMULATE: 每个乘积系数 += e_hat (j==0) 或运行的 t_hat。
- // bank_t 写入由下方的组合逻辑 bt_we/bt_wa/bt_wd assign 完成；
- // 这里我们只推进累加索引 / (i,j)。
+ // ACCUMULATE: each product coeff += e_hat (j==0) or running t_hat.
+ // The bank_t write is the combinational bt_we/bt_wa/bt_wd assigns
+ // below; here we only advance the accumulate index / (i,j).
if (pm_vo) begin
if (m_oidx == 8'd255) begin
- // 完成此 (i,j) 项；推进
+ // finished this (i,j) term; advance
if (m_j + 2'd1 < k_r) begin
m_j <= m_j + 2'd1;
- m_pending <= 1'b1; // 下一个项，同一行
+ m_pending <= 1'b1; // next term, same row
end else begin
m_j <= 2'd0;
- m_i <= m_i + 2'd1; // 下一行（或 == K -> DONE）
+ m_i <= m_i + 2'd1; // next row (or == K -> DONE)
if (m_i + 2'd1 < k_r) m_pending <= 1'b1;
end
end else begin
@@ -1560,7 +1568,7 @@ module mlkem_top #(
end
end
- // core 再次进入 IDLE 后启动下一个 (i,j) poly_mul 加载（重新启动）
+ // Start next (i,j) poly_mul load once core is IDLE again (re-prime)
if (m_pending && pm_ready && !pm_vo) begin
m_loading <= 1'b1;
m_ld <= 9'd0;
@@ -1570,7 +1578,7 @@ module mlkem_top #(
end
end
- // N 完成时武装 Encaps U 阶段：启动首行 MAC (i=0,j=0)。
+ // Arm Encaps U stage when N finishes: prime first row MAC (i=0,j=0).
if (st == ST_ENC_N && st_next == ST_ENC_U) begin
u_row <= 3'd0;
u_sub <= 2'd0; // MAC
@@ -1582,10 +1590,10 @@ module mlkem_top #(
pm_valid <= 1'b0;
end
- // ---- ST_ENC_U/V 子阶段 0: MAC sum_j (A^T 或 t_hat)[j] o y_hat[j] ----
- // psum（j==0 时初始化为 0）累加到 bank_t[UPSUM]。镜像 ST_M
- // 加载/累加节拍（预读 1，通过 u_j0q 进行 j 选择）。
- // U: A^T[u_row][j] (bank_a u_j*K+u_row)。V: t_hat[j] (bank_a u_j*K, u_row=0)。
+ // ---- ST_ENC_U/V sub-phase 0: MAC sum_j (A^T or t_hat)[j] o y_hat[j] ----
+ // psum (init 0 at j==0) accumulates into bank_t[UPSUM]. Mirrors ST_M
+ // load/accumulate cadence (read-ahead by 1, j-select via u_j0q).
+ // U: A^T[u_row][j] (bank_a u_j*K+u_row). V: t_hat[j] (bank_a u_j*K, u_row=0).
if ((st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && u_sub == 2'd0) begin
u_j0q <= (u_j == 3'd0);
@@ -1602,13 +1610,13 @@ module mlkem_top #(
end
if (pm_vo) begin
- // bank_t psum 写入是组合逻辑的 (u_psum_we)；推进索引
+ // bank_t psum write is combinational (u_psum_we); advance idx
if (u_oidx == 8'd255) begin
if (u_j + 3'd1 < k_r) begin
u_j <= u_j + 3'd1;
- u_pending <= 1'b1; // 下一个项，同一行
+ u_pending <= 1'b1; // next term, same row
end else begin
- // 行的 MAC 完成 -> INTT 子阶段
+ // row's MAC done -> INTT sub-phase
u_sub <= 2'd1;
u_ridx <= 9'd0;
u_widx <= 8'd0;
@@ -1620,7 +1628,7 @@ module mlkem_top #(
end
end
- // 重新启动下一个项的 poly_mul 加载
+ // re-prime next term's poly_mul load
if (u_pending && pm_ready && !pm_vo) begin
u_loading <= 1'b1;
u_ld <= 9'd0;
@@ -1630,7 +1638,7 @@ module mlkem_top #(
end
end
- // ---- ST_ENC_U/V 子阶段 1: INTT(psum) mode=1 原地（bank_t）----
+ // ---- ST_ENC_U/V sub-phase 1: INTT(psum) mode=1 in place (bank_t) ----
if ((st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && u_sub == 2'd1) begin
if (u_nloading) begin
if (u_ridx == 9'd256) begin
@@ -1641,9 +1649,9 @@ module mlkem_top #(
u_nvalid <= 1'b1;
end
end
- if (ntt_vo) u_widx <= u_widx + 8'd1; // 原地写入 (u_intt_we)
+ if (ntt_vo) u_widx <= u_widx + 8'd1; // in-place write (u_intt_we)
if (ntt_done) begin
- // INTT 完成 -> ADD 子阶段（psum + e1[u_row]）
+ // INTT done -> ADD sub-phase (psum + e1[u_row])
u_sub <= 2'd2;
u_aidx <= 9'd0;
u_awidx <= 8'd0;
@@ -1651,27 +1659,27 @@ module mlkem_top #(
end
end
- // ---- ST_ENC_U/V + ST_DEC_W 子阶段 2: ADD/SUB + 写回 ----
- // U: u[u_row] = psum + e1[u_row] mod Q -> bank_se (u_add_we)。
- // V: v = psum + e2 + mu mod Q -> bank_t[UPSUM] (u_v_we)。
- // D3 w = v' - psum mod Q -> bank_t[UPSUM] (u_w_we)。
- // 在 u_aidx 处读取 psum (bank_t) + e1/e2/v'；两者在 1 周期后到达，因此
- // 寄存器 (valid,widx) 并在下一个周期写入。u_row_max 限定
- // 行循环（U 为 K，V/D3 为 1）。
+ // ---- ST_ENC_U/V + ST_DEC_W sub-phase 2: ADD/SUB + writeback ----
+ // U: u[u_row] = psum + e1[u_row] mod Q -> bank_se (u_add_we).
+ // V: v = psum + e2 + mu mod Q -> bank_t[UPSUM] (u_v_we).
+ // D3 w = v' - psum mod Q -> bank_t[UPSUM] (u_w_we).
+ // Read psum (bank_t) + e1/e2/v' at u_aidx; both arrive 1 cyc later, so
+ // register (valid,widx) and write next cycle. u_row_max bounds the
+ // row loop (K for U, 1 for V/D3).
if ((st == ST_ENC_U || st == ST_ENC_V || st == ST_DEC_W) && u_sub == 2'd2) begin
if (u_aidx < 9'd256) begin
u_aidx <= u_aidx + 9'd1;
- u_avalid <= 1'b1; // 本周起地址已呈现 -> 下周起写入
+ u_avalid <= 1'b1; // addr presented this cyc -> write next
u_awidx <= u_aidx[7:0];
end else begin
- u_avalid <= 1'b0; // 没有更多地址
+ u_avalid <= 1'b0; // no more addresses
end
- // 写回（u_add_we / u_v_we）在本周期提交索引 u_awidx。
- // 在第 256 个系数 (u_awidx==255) 被写后推进行。
+ // writeback (u_add_we / u_v_we) commits index u_awidx this cycle.
+ // Advance row after the 256th coeff (u_awidx==255) is written.
if (u_avalid && u_awidx == 8'd255) begin
if (u_row + 3'd1 < u_row_max) begin
u_row <= u_row + 3'd1;
- u_sub <= 2'd0; // 下一行 MAC
+ u_sub <= 2'd0; // next row MAC
u_j <= 3'd0;
u_ld <= 9'd0;
u_oidx <= 8'd0;
@@ -1684,8 +1692,8 @@ module mlkem_top #(
end
end
- // U 完成时武装 E5 (ST_ENC_C1)：c1 = byteEncode_du(Compress_du(u))。
- // 遍历 K 个多项式 * 256 个系数；复位位打包器 + ct 写入指针。
+ // Arm E5 (ST_ENC_C1) when U finishes: c1 = byteEncode_du(Compress_du(u)).
+ // Walk K polys * 256 coeffs; reset bit-packer + ct write pointer.
if (st == ST_ENC_U && st_next == ST_ENC_C1) begin
cp_poly <= 3'd0;
cp_idx <= 8'd0;
@@ -1697,37 +1705,39 @@ module mlkem_top #(
cd_valid <= 1'b0;
end
- // ---- ST_ENC_C1/C2: Compress_d -> byteEncode_d -> ct 区域 ----
- // C1 (E5): Compress_du(u[0..K-1]) 从 bank_se -> ct[0..c1_bytes)。
- // C2 (E7): Compress_dv(v) 从 bank_t[UPSUM] -> ct[c1_bytes..ct_bytes)。
- // 每个系数，5 阶段微序列（预读 1 周期 bram + 1 周期
- // comp_decomp 流水线），然后排空子阶段发射整字节：
- // ph0: 呈现系数地址（按状态 cp_se_full / cp_bt_full）。
- // ph1: 系数到达 (cp_coeff_src) -> 锁存到 cd_coeff，脉冲 cd_valid。
- // ph2: 撤销 cd_valid（1 周期脉冲）；comp_decomp 捕获。
- // ph3: cd_vo 高 -> cd_out（低 cp_d 位）有效；LSB-first 追加。
- // ph4: 排空: 当缓冲 >=8 位时，每周期发射一个 ct 字节；然后推进。
- // 每个多项式 = 256 个系数 = 32*d 字节（整），因此 cp_buf 在每个
- // 多项式边界排空（跨多项式/区域无进位）。
+ // ---- ST_ENC_C1/C2: Compress_d -> byteEncode_d -> ct region ----
+ // C1 (E5): Compress_du(u[0..K-1]) from bank_se -> ct[0..c1_bytes).
+ // C2 (E7): Compress_dv(v) from bank_t[UPSUM] -> ct[c1_bytes..ct_bytes).
+ // Per coeff, 5-phase micro-sequence (read-ahead 1 cyc bram plus
+ // variable-latency comp_decomp), then a drain sub-phase emitting whole bytes:
+ // ph0: present coeff addr (cp_se_full / cp_bt_full by state).
+ // ph1: coeff arrives (cp_coeff_src) -> latch into cd_coeff, pulse cd_valid.
+ // ph2: drop cd_valid (1-cyc pulse); comp_decomp captures.
+ // ph3: cd_vo high -> cd_out (low cp_d bits) valid; append LSB-first.
+ // ph4: drain: while >=8 bits buffered, emit one ct byte/cycle; then advance.
+ // Each poly = 256 coeffs = 32*d bytes (whole), so cp_buf empties at
+ // each poly boundary (no carry across polys / regions).
if ((st == ST_ENC_C1 || st == ST_ENC_C2) && !cp_done) begin
case (cp_ph)
- 3'd0: cp_ph <= 3'd1; // 地址已呈现；等待读取
+ 3'd0: cp_ph <= 3'd1; // addr presented; wait read
3'd1: begin
- cd_coeff <= cp_coeff_src; // u (bank_se) 或 v (bank_t)
- cd_valid <= 1'b1; // 馈送 comp_decomp（1 周期脉冲）
+ cd_coeff <= cp_coeff_src; // u (bank_se) or v (bank_t)
+ cd_valid <= 1'b1; // feed comp_decomp (1-cyc pulse)
cp_ph <= 3'd2;
end
3'd2: begin
- cd_valid <= 1'b0; // 本周起 comp_decomp 已捕获
+ cd_valid <= 1'b0; // comp_decomp captured this cyc
cp_ph <= 3'd3;
end
3'd3: begin
- // cd_out 有效 (cd_vo): LSB-first 在位 cp_nbits 处追加 cp_d 位
- cp_buf <= cp_buf | (({13'd0, cd_out} & ((25'd1 << cp_d) - 25'd1)) << cp_nbits);
- cp_nbits <= cp_nbits + {1'b0, cp_d};
- cp_ph <= 3'd4;
+ if (cd_vo) begin
+ // Append compressed low cp_d bits LSB-first at bit cp_nbits.
+ cp_buf <= cp_buf | (({13'd0, cd_out} & ((25'd1 << cp_d) - 25'd1)) << cp_nbits);
+ cp_nbits <= cp_nbits + {1'b0, cp_d};
+ cp_ph <= 3'd4;
+ end
end
- default: begin // 3'd4: 排空整字节
+ default: begin // 3'd4: drain whole bytes
if (cp_nbits >= 6'd8) begin
ct_we <= 1'b1;
ct_wa <= cp_wa;
@@ -1736,11 +1746,11 @@ module mlkem_top #(
cp_buf <= cp_buf >> 8;
cp_nbits <= cp_nbits - 6'd8;
end else begin
- // 系数完全打包；推进系数 / 多项式
+ // coeff fully packed; advance coeff / poly
if (cp_idx == 8'd255) begin
cp_idx <= 8'd0;
if (cp_poly + 3'd1 < cp_poly_max) cp_poly <= cp_poly + 3'd1;
- else cp_done <= 1'b1; // 区域完成
+ else cp_done <= 1'b1; // region complete
end else begin
cp_idx <= cp_idx + 8'd1;
end
@@ -1750,10 +1760,10 @@ module mlkem_top #(
endcase
end
- // V 完成时武装 E7 (ST_ENC_C2)：c2 = byteEncode_dv(Compress_dv(v))。
- // 单个多项式（v 在 bank_t UPSUM）；ct 写入指针 cp_wa 从 c1_bytes
- // 继续（不重置），因此 c2 紧跟 c1 之后。cp_buf/nbits
- // 此处为 0（c1 结束时在多项式/字节边界上），但为安全复位。
+ // Arm E7 (ST_ENC_C2) when V finishes: c2 = byteEncode_dv(Compress_dv(v)).
+ // Single poly (v at bank_t UPSUM); ct write pointer cp_wa CONTINUES
+ // from c1_bytes (NOT reset) so c2 lands right after c1. cp_buf/nbits
+ // are 0 here (c1 ended on a poly/byte boundary) but reset for safety.
if (st == ST_ENC_V && st_next == ST_ENC_C2) begin
cp_poly <= 3'd0;
cp_idx <= 8'd0;
@@ -1762,10 +1772,10 @@ module mlkem_top #(
cp_nbits <= 6'd0;
cp_done <= 1'b0;
cd_valid <= 1'b0;
- // cp_wa 有意保留（= 来自 C1 的 c1_bytes_rt）。
+ // cp_wa intentionally preserved (= c1_bytes_rt from C1).
end
- // 加载稳定后武装 D1 DECOMP：先解码 c1 (u')。
+ // Arm D1 DECOMP when load settles: decode c1 (u') first.
if (st == ST_DEC_LOAD && st_next == ST_DEC_DECOMP) begin
dec_poly <= 3'd0;
dec_cidx <= 8'd0;
@@ -1777,31 +1787,31 @@ module mlkem_top #(
dec_in_c2 <= 1'b0;
dec_dc_valid <= 1'b0;
dec_dc_done <= 1'b0;
- cin_rd_addr_r<= 11'd0; // 呈现字节 0 地址
+ cin_rd_addr_r<= 11'd0; // present byte 0 address
end
- // ---- ST_DEC_DECOMP (D1): byteDecode_d + 解压缩 c1/c2 ----
- // C1/C2 打包器的逆向。每个系数：
- // ph0: 若 nbits ph1；
- // 否则提取低 d 位 -> comp_decomp（解压缩）-> ph2。
- // ph1: 在 dec_nbits 处追加 cin_rd_data（8 位）；推进 dec_ba；
- // 呈现下一个字节地址；回到 ph0。
- // ph2: 撤销 dc_valid（脉冲已捕获）-> ph3。
- // ph3: cd_vo -> dec_u_we/dec_v_we 写入 cd_out；推进系数。
- // c1 = K 个多项式 (d=du) -> u'[poly] bank_se；然后 c2 = 1 个多项式 (d=dv)
- // -> v' bank_a[DEC_VASLOT]。字节流在 c_in_bram 中连续。
+ // ---- ST_DEC_DECOMP (D1): byteDecode_d + Decompress c1/c2 ----
+ // Reverse of the C1/C2 packer. Per coeff:
+ // ph0: if nbits ph1;
+ // else extract low d bits -> comp_decomp (decompress) -> ph2.
+ // ph1: append cin_rd_data (8 bits) at dec_nbits; advance dec_ba;
+ // present next byte addr; back to ph0.
+ // ph2: drop dc_valid (pulse captured) -> ph3.
+ // ph3: cd_vo -> dec_u_we/dec_v_we writes cd_out; advance coeff.
+ // c1 = K polys (d=du) -> u'[poly] bank_se; then c2 = 1 poly (d=dv)
+ // -> v' bank_a[DEC_VASLOT]. Byte stream is contiguous in c_in_bram.
if (st == ST_DEC_DECOMP && !dec_dc_done) begin
case (dec_ph)
2'd0: begin
- if (dec_nbits >= {1'b0, dec_dc_d}) begin
- // 足够位数：提取低 d 位作为解码符号
+ if (dec_nbits >= {1'b0, dec_dc_d}) begin
+ // enough bits: extract low d as the decoded symbol
dec_dc_coeff <= dec_buf[11:0] & ((12'd1 << dec_dc_d) - 12'd1);
dec_dc_valid <= 1'b1;
dec_buf <= dec_buf >> dec_dc_d;
dec_nbits <= dec_nbits - {1'b0, dec_dc_d};
dec_ph <= 2'd2;
end else begin
- // 需要另一个字节（cin_rd_data 反映 dec_ba）
+ // need another byte (cin_rd_data reflects dec_ba)
dec_ph <= 2'd1;
end
end
@@ -1809,29 +1819,29 @@ module mlkem_top #(
dec_buf <= dec_buf | ({17'd0, cin_rd_data} << dec_nbits);
dec_nbits <= dec_nbits + 6'd8;
dec_ba <= dec_ba + 12'd1;
- cin_rd_addr_r<= dec_ba[10:0] + 11'd1; // 呈现下一个字节
+ cin_rd_addr_r<= dec_ba[10:0] + 11'd1; // present next byte
dec_ph <= 2'd0;
end
2'd2: begin
- dec_dc_valid <= 1'b0; // comp_decomp 已捕获
+ dec_dc_valid <= 1'b0; // comp_decomp captured
dec_ph <= 2'd3;
end
- default: begin // 2'd3: 捕获 cd_vo
+ default: begin // 2'd3: capture cd_vo
if (cd_vo) begin
- // 本周起 dec_u_we/dec_v_we 在 dec_widx 处提交 cd_out。
+ // dec_u_we/dec_v_we commit cd_out at dec_widx this cycle.
dec_widx <= dec_widx + 8'd1;
if (dec_cidx == 8'd255) begin
dec_cidx <= 8'd0;
dec_widx <= 8'd0;
if (!dec_in_c2) begin
if (dec_poly + 3'd1 < k_r) begin
- dec_poly <= dec_poly + 3'd1; // 下一行 u'
+ dec_poly <= dec_poly + 3'd1; // next u' row
end else begin
- dec_in_c2 <= 1'b1; // 切换到 c2 (v')
+ dec_in_c2 <= 1'b1; // switch to c2 (v')
dec_poly <= 3'd0;
end
end else begin
- dec_dc_done <= 1'b1; // v' 完成 -> D1 完成
+ dec_dc_done <= 1'b1; // v' done -> D1 complete
end
end else begin
dec_cidx <= dec_cidx + 8'd1;
@@ -1842,24 +1852,25 @@ module mlkem_top #(
endcase
end
- // M 完成时武装 E 阶段
+ // Arm E stage when M finishes
if (st == ST_M && st_next == ST_E) begin
e_poly <= 3'd0;
e_pair <= 8'd0;
- e_ph <= 2'd0; // 从第一对系数的取回-c0 开始
+ e_ph <= 2'd0; // start at fetch-c0 of first pair
e_rho <= 10'd0;
e_done <= 1'b0;
end
// ---- ST_E: byteEncode12 t_hat -> ek, s_hat -> dk_pke ----
- // 每个系数对单周期寄存 bank 读取；4 周期微相（ph0 取回 c0,
- // ph1 写入 b0 + 取回 c1, ph2 写入 b1,
- // ph3 写入 b2）。上一周期呈现地址对应的系数是
- // e_rd_coeff（dk 半部分为 bse_rd_data，ek 半部分为 bt_rd_data），
- // 两者在各自的 sd_bram 内寄存（ST_E 中 bse_rd_addr/bt_rd_addr = e_rd_full）。
+ // Single registered bank read per cycle; 4-cycle micro-phase per
+ // coeff pair (ph0 fetch c0, ph1 write b0 + fetch c1, ph2 write b1,
+ // ph3 write b2). The coeff for the addr presented last cycle is
+ // e_rd_coeff (= bse_rd_data for dk-half, bt_rd_data for ek-half),
+ // both registered inside their sd_bram (bse_rd_addr/bt_rd_addr =
+ // e_rd_full in ST_E).
if (st == ST_E && !e_done) begin
if (e_poly < {1'b0, k_r, 1'b0}) begin
- // ph0: 仅取回（启动）。ph1..3: 写入一个打包字节。
+ // ph0: fetch only (prime). ph1..3: write one packed byte.
if (e_ph != 2'd0) begin
if (!e_is_dk) begin
ek_we <= 1'b1;
@@ -1871,16 +1882,16 @@ module mlkem_top #(
dkp_wd <= e_wbyte;
end
end
- // 系数高半字节可用时保存
+ // save coeff high nibbles as they become available
if (e_ph == 2'd1) e_c0_hi <= e_rd_coeff[11:8]; // c0[11:8]
if (e_ph == 2'd2) e_c1_hi <= e_rd_coeff[11:4]; // c1[11:4]
- // 推进微相 / 系数对 / 多项式
+ // advance micro-phase / pair / poly
if (e_ph == 2'd3) begin
e_ph <= 2'd0;
if (e_pair == 8'd127) begin
e_pair <= 8'd0;
- e_poly <= e_poly + 5'd1; // 下一个多项式 或 -> rho 阶段
+ e_poly <= e_poly + 5'd1; // next poly or -> rho phase
end else begin
e_pair <= e_pair + 8'd1;
end
@@ -1888,7 +1899,7 @@ module mlkem_top #(
e_ph <= e_ph + 2'd1;
end
end else begin
- // rho 复制：ek[384*K + r] = rho 字节 r（r = 0..31），1 字节/周期
+ // rho copy: ek[384*K + r] = rho byte r (r = 0..31), 1 byte/cycle
ek_we <= 1'b1;
ek_wa <= dk_bytes_rt[10:0] + {1'b0, e_rho};
ek_wd <= rho_r[e_rho*8 +: 8];
@@ -1897,33 +1908,34 @@ module mlkem_top #(
end
end
- // E 完成时武装 H 阶段
+ // Arm H stage when E finishes
if (st == ST_E && st_next == ST_H) begin
h_blk <= 3'd0;
h_byte <= 8'd0;
- h_phase <= 2'd0; // 组装
+ h_phase <= 2'd0; // assemble
h_mbvalid<= 1'b0;
h_mblast <= 1'b0;
- h_ack <= 1'b1; // 准备好消费最终摘要
- h_wb_vld <= 1'b0; // 没有待处理的写回
+ h_ack <= 1'b1; // ready to consume final digest
+ h_wb_vld <= 1'b0; // no pending writeback yet
end
- // ---- ST_H / ST_ENC_H: 通过多块 SHA3-256 计算 H(ek) ----
- // KeyGen（ek 刚由 ST_E 写入）和 Encaps（ek
- // 由 TB 预加载到 ek_bram）使用相同逻辑。hek_r 捕获 H(ek)。
+ // ---- ST_H / ST_ENC_H: H(ek) via multi-block SHA3-256 ----
+ // Same logic for KeyGen (ek just written by ST_E) and Encaps (ek
+ // preloaded into ek_bram by the TB). hek_r captures H(ek) either way.
if (st == ST_H || st == ST_ENC_H) begin
case (h_phase)
- // 将块 h_blk 的 136 个字节组装到 h_block_r 中。
- // ek BRAM 读取是寄存的：本周起呈现 h_byte 的地址，
- // 将上一周期呈现地址对应的到达字节写回（h_wb_*）。
- // h_byte 运行 0..136（额外一个周期以刷新最终写回）。
+ // assemble 136 bytes of block h_blk into h_block_r.
+ // ek BRAM read is registered: present addr for h_byte this
+ // cycle, write back the byte that arrived for the addr we
+ // presented last cycle (h_wb_*). h_byte runs 0..136 (one
+ // extra cycle to flush the final writeback).
2'd0: begin
- // 写回上一地址读取的字节
+ // writeback the byte read for the previous address
if (h_wb_vld)
h_block_r[h_wb_idx*8 +: 8] <= h_wb_inek ? ek_rd_data : h_wb_pad;
if (h_byte <= 8'd135) begin
- // 为本周起呈现的地址设置写回
+ // set up writeback for the address presented this cycle
h_wb_vld <= 1'b1;
h_wb_idx <= h_byte;
h_wb_g <= h_g_addr;
@@ -1931,66 +1943,68 @@ module mlkem_top #(
h_wb_pad <= h_padconst(h_blk, h_byte);
h_byte <= h_byte + 8'd1;
end else begin
- // h_byte==136: 最终写回（字节 135）已在上方完成
+ // h_byte==136: final writeback (for byte 135) done above
h_wb_vld <= 1'b0;
h_byte <= 8'd0;
h_mbvalid <= 1'b1;
h_mblast <= (h_blk == h_nblk_rt - 4'd1);
- h_phase <= 2'd1; // 馈送
+ h_phase <= 2'd1; // feed
end
end
- // 馈送: 保持 valid 直到被接受（mb_ready 撤销）
+ // feed: hold valid until accepted (mb_ready drops)
2'd1: begin
if (h_mbvalid && !h_mbready) begin
h_mbvalid <= 1'b0;
h_mblast <= 1'b0;
- h_phase <= 2'd2; // 等待置换
+ h_phase <= 2'd2; // wait permute
end
end
- // 等待置换完成：再次就绪（更多块）或摘要有效（最后一块）
+ // wait permute done: ready again (more blocks) or digest valid (last)
2'd2: begin
if (h_vo) begin
hek_r <= h_hash[255:0];
- h_phase <= 2'd3; // 完成
+ h_phase <= 2'd3; // done
end else if (h_mbready) begin
h_blk <= h_blk + 3'd1;
- h_phase <= 2'd0; // 组装下一个块
+ h_phase <= 2'd0; // assemble next block
end
end
- default: ; // 2'd3 完成: 保持
+ default: ; // 2'd3 done: hold
endcase
end
- // H(ek) 完成时武装 Encaps G：触发 64 字节单块
- // G(m||H(ek))。hek_r 现在保存 H(ek)；enc_g_data = {hek_r, m_r}。
+ // Arm Encaps G when H(ek) finishes: fire the 64-byte single-block
+ // G(m||H(ek)). hek_r now holds H(ek); enc_g_data = {hek_r, m_r}.
if (st == ST_ENC_H && st_next == ST_ENC_G) begin
sha3_valid <= 1'b1;
sha3_ack <= 1'b1;
end
- // Encaps G 完成时捕获 (K, r)。ss = K = 低半部分，r = 高半部分。
+ // Capture (K, r) when Encaps G completes. ss = K = low half, r = high.
if (st == ST_ENC_G && sha3_vo) begin
- ss_r <= sha3_hash[255:0]; // K（共享密钥）= G 字节 0..31
- r_r <= sha3_hash[511:256]; // r = G 字节 32..63
+ ss_r <= sha3_hash[255:0]; // K (shared secret) = G bytes 0..31
+ r_r <= sha3_hash[511:256]; // r = G bytes 32..63
sha3_ack <= 1'b0;
end
- // m' 就绪时武装 Decaps D5 G（ST_DEC_MENC -> ST_DEC_G）：触发
- // 64 字节单块 G(m'||h)。dec_g_data = {hek_r, m_r}。
+ // Arm Decaps D5 G when m' is ready (ST_DEC_MENC -> ST_DEC_G): fire the
+ // 64-byte single-block G(m'||h). dec_g_data = {hek_r, m_r}.
if (st == ST_DEC_MENC && st_next == ST_DEC_G) begin
sha3_valid <= 1'b1;
sha3_ack <= 1'b1;
end
- // D5 G 完成时捕获 (K', r')。K' = 低半部分（候选 ss），
- // r' = 高半部分（D6 重加密的 PRF 种子）。重用 ss_r / r_r。
+ // Capture (K', r') when D5 G completes. K' = low half (candidate ss),
+ // r' = high half (PRF seed for D6 re-encrypt). Reuse ss_r / r_r.
if (st == ST_DEC_G && sha3_vo) begin
- ss_r <= sha3_hash[255:0]; // K' = G 字节 0..31
- r_r <= sha3_hash[511:256]; // r' = G 字节 32..63
+ ss_r <= sha3_hash[255:0]; // K' = G bytes 0..31
+ r_r <= sha3_hash[511:256]; // r' = G bytes 32..63
sha3_ack <= 1'b0;
end
- // G 完成时武装 D5 J(z||c)（ST_DEC_G -> ST_DEC_J）：组装
- // 第一个 136 字节块。dj_ack 高电平以消费最终摘要。
- if (st == ST_DEC_G && st_next == ST_DEC_J) begin
+ // Arm D5 J(z||c) as soon as D1 has stopped reading c_in_bram. It
+ // then runs in the background during D2/D3/D4 using the otherwise
+ // idle SHAKE256 path and c_in read port.
+ if (st == ST_DEC_DECOMP && st_next == ST_DEC_SDEC) begin
+ dj_active <= 1'b1;
dj_blk <= 4'd0;
dj_byte <= 8'd0;
dj_phase <= 2'd0;
@@ -1999,20 +2013,20 @@ module mlkem_top #(
dj_ack <= 1'b1;
dj_done <= 1'b0;
dj_wb_vld <= 1'b0;
- cin_rd_addr_r <= 11'd0; // 呈现 c 字节 0（g=32 -> c_idx 0）
+ cin_rd_addr_r <= 11'd0; // present c byte 0 (g=32 -> c_idx 0)
end
- // ---- ST_DEC_J: K-bar = J(z||c) 多块 SHAKE256 ----
- // 镜像 H(ek) 多块机。按全局 g 的字节来源：
- // g < 32 -> z_r 字节 g
- // 32 <= g < msglen -> c_in_bram 字节 (g-32)，寄存读取
- // 否则 -> SHAKE256 填充常量
- // c_in_bram 读取是寄存的：本周起呈现 dj_byte 的 c_idx，
- // 将上一周期呈现地址对应的到达字节写回。
- if (st == ST_DEC_J && !dj_done) begin
+ // ---- Background Decaps J: K-bar = J(z||c) multi-block SHAKE256 ----
+ // Mirror of the H(ek) multi-block machine. Byte source by global g:
+ // g < 32 -> z_r byte g
+ // 32 <= g < msglen -> c_in_bram byte (g-32), registered read
+ // else -> SHAKE256 pad constant
+ // c_in_bram read is registered: present c_idx for dj_byte this cycle,
+ // write back the byte that arrived for the addr presented last cycle.
+ if (dj_active && !dj_done) begin
case (dj_phase)
2'd0: begin
- // 写回上一地址读取的字节
+ // writeback the byte read for the previous address
if (dj_wb_vld) begin
if (dj_wb_inz)
dj_block_r[dj_wb_idx*8 +: 8] <= z_r[dj_wb_zidx*8 +: 8];
@@ -2028,9 +2042,9 @@ module mlkem_top #(
dj_wb_inc <= (dj_g >= 12'd32) && (dj_g < dj_msglen);
dj_wb_zidx <= dj_g[4:0];
dj_wb_pad <= dj_padconst(dj_g);
- // c_in 地址以组合逻辑方式呈现（ST_DEC_J 期间
- // cin_rd_addr 多路复用使用 dj_c_idx）；数据在下一个
- // 周期到达，匹配此字节的写回。
+ // c_in addr is presented combinationally (cin_rd_addr
+ // mux uses dj_c_idx while dj_active); data lands next
+ // cycle, matching this byte's writeback.
dj_byte <= dj_byte + 8'd1;
end else begin
dj_wb_vld <= 1'b0;
@@ -2051,17 +2065,18 @@ module mlkem_top #(
if (sha3_vo) begin
kbar_r <= sha3_hash[255:0];
dj_done <= 1'b1;
+ dj_active <= 1'b0;
dj_phase<= 2'd3;
end else if (h_mbready) begin
dj_blk <= dj_blk + 4'd1;
dj_phase <= 2'd0;
end
end
- default: ; // 完成
+ default: ; // done
endcase
end
- // 重加密完成时武装 D7 比较（ST_ENC_C2 -> ST_DEC_CMP）。
+ // Arm D7 compare when re-encrypt finishes (ST_ENC_C2 -> ST_DEC_CMP).
if (st == ST_ENC_C2 && st_next == ST_DEC_CMP) begin
cmp_idx <= 11'd0;
cmp_ph <= 1'b0;
@@ -2069,18 +2084,18 @@ module mlkem_top #(
cmp_done <= 1'b0;
end
- // ---- ST_DEC_CMP (D7): 常量时间 c' == c 比较 + ss 选择 ----
- // ph0: 向 ct_bram (c') 和 c_in_bram (c) 呈现 cmp_idx。
- // ph1: 寄存读取有效 -> 将任何差异 OR 到 cmp_neq，
- // 推进。最后一个字节后，锁存 dec_reject，ss 保持
- // K' (ss_r) 或通过 ss_o 多路复用器切换到 K-bar (kbar_r)。
+ // ---- ST_DEC_CMP (D7): constant-time c' == c compare + ss select ----
+ // ph0: present cmp_idx to both ct_bram (c') and c_in_bram (c).
+ // ph1: registered reads valid -> OR any difference into cmp_neq,
+ // advance. After the last byte, latch dec_reject and ss stays
+ // K' (ss_r) or switches to K-bar (kbar_r) via the ss_o mux.
if (st == ST_DEC_CMP && !cmp_done) begin
if (!cmp_ph) begin
- cmp_ph <= 1'b1; // 地址已呈现；等待读取
+ cmp_ph <= 1'b1; // addr presented; wait for read
end else begin
cmp_neq <= cmp_neq | cmp_byte_neq;
if (cmp_idx == ct_bytes_rt - 12'd1) begin
- dec_reject <= cmp_neq | cmp_byte_neq; // 包含最终字节
+ dec_reject <= cmp_neq | cmp_byte_neq; // include final byte
cmp_done <= 1'b1;
end else begin
cmp_idx <= cmp_idx + 11'd1;
@@ -2090,23 +2105,23 @@ module mlkem_top #(
end
if (st_next == ST_ENC_LOAD &&
- (st == ST_ENC_G || st == ST_DEC_J)) begin
+ (st == ST_ENC_G || st == ST_DEC_G || st == ST_DEC_J)) begin
rl_idx <= 6'd0;
rl_widx <= 6'd0;
rl_vld <= 1'b0;
end
- // ---- ST_ENC_LOAD: 从 ek_bram 流式输入 32 个 rho 字节到 rho_r ----
- // ek 读取是寄存的（1 周期延迟）：本周起到达的数据
- // 对应上一周期呈现的地址 (rl_idx-1)。呈现地址
- // rl_idx（有效 0..31）；当 rl_idx 在 1..32 时写入 rho_r[rl_idx-1]。
+ // ---- ST_ENC_LOAD: stream 32 rho bytes from ek_bram into rho_r ----
+ // ek read is registered (1-cyc latency): the data arriving this cycle
+ // is for the address presented last cycle (rl_idx-1). Present addr
+ // rl_idx (valid 0..31); write rho_r[rl_idx-1] when rl_idx in 1..32.
if (st == ST_ENC_LOAD) begin
if (rl_idx >= 6'd1 && rl_idx <= 6'd32)
rho_r[(rl_idx-6'd1)*8 +: 8] <= ek_rd_data;
- rl_idx <= rl_idx + 6'd1; // 当 rl_idx >= 33 时退出
+ rl_idx <= rl_idx + 6'd1; // exit when rl_idx >= 33
end
- // 武装 ST_ENC_A（通过 SampleNTT 重新生成 A_hat，与 ST_A 相同）。
+ // Arm ST_ENC_A (regenerate A_hat via SampleNTT, same as ST_A).
if (st == ST_ENC_LOAD && st_next == ST_ENC_A) begin
snt_valid <= 1'b1;
snt_ack <= 1'b1;
@@ -2117,17 +2132,17 @@ module mlkem_top #(
a_busy <= 1'b0;
end
- // 武装 ST_ENC_TDEC（byteDecode12: ek -> t_hat 在 bank_a 中）。现在紧跟
- // E5 (ST_ENC_C1)，因为 e2 在 C/N/U 期间占用 bank_t，而 t_hat
- // 仅 V 需要。（在 E1 支架中原来在 ST_ENC_A 之后武装。）
+ // Arm ST_ENC_TDEC (byteDecode12: ek -> t_hat in bank_a). Now follows
+ // E5 (ST_ENC_C1) since e2 occupied bank_t during C/N/U and t_hat is
+ // only needed for V. (Was armed off ST_ENC_A in the E1 scaffold.)
if (st == ST_ENC_C1 && st_next == ST_ENC_TDEC) begin
td_poly <= 3'd0;
td_trip <= 8'd0;
td_ph <= 3'd0;
td_done <= 1'b0;
end
- // DECOMP 完成时武装 D2 s_hat 解码（ST_DEC_SDEC）：相同的 TDEC
- // 机，byteDecode12 dk_pke -> s_hat 在 bank_a 槽位 j*K。
+ // Arm D2 s_hat decode (ST_DEC_SDEC) when DECOMP finishes: same TDEC
+ // machine, byteDecode12 dk_pke -> s_hat in bank_a slot j*K.
if (st == ST_DEC_DECOMP && st_next == ST_DEC_SDEC) begin
td_poly <= 3'd0;
td_trip <= 8'd0;
@@ -2135,16 +2150,16 @@ module mlkem_top #(
td_done <= 1'b0;
end
- // Encaps: ek -> t_hat。Decaps D2: dk_pke -> s_hat。每个三元组
- // 相同的 5 周期微相；仅字节来源 (td_byte) 不同。
+ // Encaps: ek -> t_hat. Decaps D2: dk_pke -> s_hat. Same 5-cycle
+ // micro-phase per triple; only the byte source (td_byte) differs.
if ((st == ST_ENC_TDEC || st == ST_DEC_SDEC) && !td_done) begin
- // 捕获上一周期呈现地址对应的到达字节
+ // capture the byte that arrived for the address presented last cycle
if (td_ph == 3'd1) td_b0 <= td_byte;
if (td_ph == 3'd2) td_b1 <= td_byte;
if (td_ph == 3'd3) td_b2 <= td_byte;
- // 将解码系数写入 bank_a 槽位 td_poly*K（因此 V 的 MAC
- // 通过相同的 u_aslot=u_j*K 寻址读取 t_hat[j]，u_row=0）。
+ // write decoded coeffs into bank_a at slot td_poly*K (so V's MAC
+ // reads t_hat[j] via the same u_aslot=u_j*K addressing, u_row=0).
if (td_ph == 3'd3) begin
td_we <= 1'b1;
td_wa <= (({4'd0,td_poly}*{9'd0,k_r})*256 + {td_trip, 1'b0}) & ((1< bank_a[E2_ASLOT], 256 个系数 ----
- // 呈现读取地址 em_ridx（bank_t 相对 0）；bank_t 读取是寄存的
- // 因此下一周期 bt_rd_data = e2[em_ridx]。将 bank_a 写入
- // (em_we/em_widx) 安排在该下一周期，em_widx == em_ridx，因此
- // 写入将 e2[R] 提交到槽位[R]（无偏移错误）。
+ // ---- ST_ENC_E2MV: e2 bank_t[0] -> bank_a[E2_ASLOT], 256 coeffs ----
+ // Present read addr em_ridx (bank_t rel 0); bank_t read is registered
+ // so bt_rd_data next cycle = e2[em_ridx]. Schedule the bank_a write
+ // (em_we/em_widx) for that next cycle with em_widx == em_ridx, so the
+ // write commits e2[R] into slot[R] (no off-by-one).
if (st == ST_ENC_E2MV && !em_done) begin
- if (em_ridx <= 9'd255) begin // 本周起呈现有效的 e2 地址
- em_we <= 1'b1; // -> 下周起写入（ba_wd = bt_rd_data）
+ if (em_ridx <= 9'd255) begin // presenting a valid e2 addr this cycle
+ em_we <= 1'b1; // -> write next cycle (ba_wd = bt_rd_data)
em_widx <= em_ridx[7:0];
end
- if (em_ridx == 9'd256) em_done <= 1'b1; // 最后写入（e2[255]）已发出
+ if (em_ridx == 9'd256) em_done <= 1'b1; // last write (e2[255]) issued
else em_ridx <= em_ridx + 9'd1;
end
- // e2 搬迁完成时武装 ST_ENC_V：启动首（唯一）行 MAC (i=0,j=0)。
- // 重用 u_* MAC/INTT/ADD 机，u_row 固定为 0。
+ // Arm ST_ENC_V when e2 relocated: prime first (only) row MAC (i=0,j=0).
+ // Reuses the u_* MAC/INTT/ADD machine with u_row tied to 0.
if (st == ST_ENC_E2MV && st_next == ST_ENC_V) begin
u_row <= 3'd0;
u_sub <= 2'd0; // MAC
@@ -2208,8 +2223,8 @@ module mlkem_top #(
pm_valid <= 1'b0;
end
- // D2 NTT 完成时武装 D3 (ST_DEC_W)：相同的 u_* MAC/INTT 机，
- // 单行 (u_row 0..0)。MAC s_hat[j] o u_hat[j], INTT, 然后 SUB。
+ // Arm D3 (ST_DEC_W) when D2 NTT finishes: same u_* MAC/INTT machine,
+ // single row (u_row 0..0). MAC s_hat[j] o u_hat[j], INTT, then SUB.
if (st == ST_DEC_NTT && st_next == ST_DEC_W) begin
u_row <= 3'd0;
u_sub <= 2'd0; // MAC
@@ -2221,7 +2236,7 @@ module mlkem_top #(
pm_valid <= 1'b0;
end
- // D3 w 完成时武装 D4 (ST_DEC_MENC)：byteEncode_1(Compress_1(w))。
+ // Arm D4 (ST_DEC_MENC) when D3 w done: byteEncode_1(Compress_1(w)).
if (st == ST_DEC_W && st_next == ST_DEC_MENC) begin
men_idx <= 8'd0;
men_ph <= 2'd0;
@@ -2229,12 +2244,12 @@ module mlkem_top #(
end
// ---- ST_DEC_MENC (D4): m' = byteEncode_1(Compress_1(w)) ----
- // Compress_1(w)=1 当且仅当 832 < w <= 2496 (Q=3329)。将 256 位
- // LSB-first 打包到 m_r（位 men_idx）。从 bank_t[UPSUM] 读取 w。
- // ph0: 呈现 w[men_idx] 地址；ph1: bt_rd_data 有效 -> 设置位。
+ // Compress_1(w)=1 iff 832 < w <= 2496 (Q=3329). Pack 256 bits
+ // LSB-first into m_r (bit men_idx). Read w from bank_t[UPSUM].
+ // ph0: present w[men_idx] addr; ph1: bt_rd_data valid -> set bit.
if (st == ST_DEC_MENC && !men_done) begin
case (men_ph)
- 2'd0: men_ph <= 2'd1; // 地址已呈现；等待读取
+ 2'd0: men_ph <= 2'd1; // addr presented; wait read
default: begin // ph1: bt_rd_data = w[men_idx]
m_r[men_idx] <= men_w_bit;
if (men_idx == 8'd255) men_done <= 1'b1;
diff --git a/rtl/ip/prng/prng_axi_slave.v b/rtl/ip/prng/prng_axi_slave.v
deleted file mode 100644
index dff4ead..0000000
--- a/rtl/ip/prng/prng_axi_slave.v
+++ /dev/null
@@ -1,218 +0,0 @@
-// Lightweight AXI4 register endpoint for a pseudo-random generator.
-//
-// Register map, byte offsets within the shared accelerator window:
-// 0xd000 CTRL W bit0=mix software seed, bit1=clear counters
-// 0xd004 STATUS R bit0=ready
-// 0xd008 DATA R next 32-bit random word
-// 0xd00c COUNT R number of DATA reads
-// 0xd010 SEED W software seed/mix value
-//
-// This block is deterministic unless software mixes in an external seed. It
-// intentionally uses only flops, XORs, shifts, and adders.
-module prng_axi_slave (
- input clk,
- input resetn,
-
- input [4:0] s_axi_awid,
- input [31:0] s_axi_awaddr,
- input [7:0] s_axi_awlen,
- input [2:0] s_axi_awsize,
- input [1:0] s_axi_awburst,
- input s_axi_awlock,
- input [3:0] s_axi_awcache,
- input [2:0] s_axi_awprot,
- input s_axi_awvalid,
- output s_axi_awready,
-
- input [4:0] s_axi_wid,
- input [31:0] s_axi_wdata,
- input [3:0] s_axi_wstrb,
- input s_axi_wlast,
- input s_axi_wvalid,
- output s_axi_wready,
-
- output [4:0] s_axi_bid,
- output [1:0] s_axi_bresp,
- output reg s_axi_bvalid,
- input s_axi_bready,
-
- input [4:0] s_axi_arid,
- input [31:0] s_axi_araddr,
- input [7:0] s_axi_arlen,
- input [2:0] s_axi_arsize,
- input [1:0] s_axi_arburst,
- input s_axi_arlock,
- input [3:0] s_axi_arcache,
- input [2:0] s_axi_arprot,
- input s_axi_arvalid,
- output s_axi_arready,
-
- output reg [4:0] s_axi_rid,
- output reg [31:0] s_axi_rdata,
- output [1:0] s_axi_rresp,
- output reg s_axi_rlast,
- output reg s_axi_rvalid,
- input s_axi_rready
-);
-
- localparam ADDR_CTRL = 12'h000;
- localparam ADDR_STATUS = 12'h004;
- localparam ADDR_DATA = 12'h008;
- localparam ADDR_COUNT = 12'h00c;
- localparam ADDR_SEED = 12'h010;
-
- assign s_axi_bresp = 2'b00;
- assign s_axi_rresp = 2'b00;
-
- reg [4:0] awid_reg;
- reg [31:0] awaddr_reg;
- reg [7:0] awleft_reg;
- reg write_active;
-
- assign s_axi_awready = !write_active && !s_axi_bvalid;
- assign s_axi_wready = write_active && !s_axi_bvalid;
- assign s_axi_bid = awid_reg;
-
- reg [31:0] araddr_reg;
- reg [7:0] arleft_reg;
- reg read_active;
-
- assign s_axi_arready = !read_active && !s_axi_rvalid;
-
- reg [127:0] rng_state;
- reg [63:0] free_counter;
- reg [31:0] seed_mix;
- reg [31:0] data_reads;
-
- wire lfsr_fb = rng_state[127] ^ rng_state[125] ^ rng_state[100] ^
- rng_state[98] ^ rng_state[27] ^ rng_state[0] ^
- free_counter[0];
- wire [127:0] rng_next = {rng_state[126:0], lfsr_fb} ^
- {free_counter, free_counter ^ {32'h0, seed_mix}};
- wire [31:0] rng_word = rng_state[31:0] ^ rng_state[63:32] ^
- rng_state[95:64] ^ rng_state[127:96] ^
- free_counter[31:0] ^ free_counter[63:32] ^
- seed_mix;
-
- function [31:0] apply_wstrb;
- input [31:0] old_value;
- input [31:0] new_value;
- input [3:0] strobe;
- begin
- apply_wstrb = old_value;
- if (strobe[0]) apply_wstrb[7:0] = new_value[7:0];
- if (strobe[1]) apply_wstrb[15:8] = new_value[15:8];
- if (strobe[2]) apply_wstrb[23:16] = new_value[23:16];
- if (strobe[3]) apply_wstrb[31:24] = new_value[31:24];
- end
- endfunction
-
- function [31:0] read_reg;
- input [31:0] addr;
- begin
- case (addr[11:0])
- ADDR_STATUS: read_reg = 32'h00000001;
- ADDR_DATA: read_reg = rng_word;
- ADDR_COUNT: read_reg = data_reads;
- ADDR_SEED: read_reg = seed_mix;
- default: read_reg = 32'h00000000;
- endcase
- end
- endfunction
-
- wire write_fire = s_axi_wvalid && s_axi_wready;
- wire read_fire = s_axi_rvalid && s_axi_rready;
- wire data_read_fire = read_fire && (araddr_reg[11:0] == ADDR_DATA);
- wire [31:0] next_araddr = araddr_reg + 32'd4;
- wire [31:0] next_awaddr = awaddr_reg + 32'd4;
-
- always @(posedge clk or negedge resetn) begin
- if (!resetn) begin
- awid_reg <= 5'b0;
- awaddr_reg <= 32'b0;
- awleft_reg <= 8'b0;
- write_active <= 1'b0;
- s_axi_bvalid <= 1'b0;
-
- araddr_reg <= 32'b0;
- arleft_reg <= 8'b0;
- read_active <= 1'b0;
- s_axi_rid <= 5'b0;
- s_axi_rdata <= 32'b0;
- s_axi_rlast <= 1'b0;
- s_axi_rvalid <= 1'b0;
-
- rng_state <= 128'h6d5a_56c3_9e37_79b9_a5a5_3c3c_c001_c0de;
- free_counter <= 64'h1;
- seed_mix <= 32'h31415927;
- data_reads <= 32'b0;
- end else begin
- free_counter <= free_counter + 64'h9e37_79b9_7f4a_7c15;
- rng_state <= rng_next;
-
- if (s_axi_awvalid && s_axi_awready) begin
- awid_reg <= s_axi_awid;
- awaddr_reg <= s_axi_awaddr;
- awleft_reg <= s_axi_awlen;
- write_active <= 1'b1;
- end
-
- if (write_fire) begin
- case (awaddr_reg[11:0])
- ADDR_CTRL: begin
- if (s_axi_wdata[0]) begin
- seed_mix <= seed_mix ^ s_axi_wdata ^ free_counter[31:0];
- rng_state <= rng_next ^ {4{s_axi_wdata}};
- end
- if (s_axi_wdata[1])
- data_reads <= 32'b0;
- end
- ADDR_SEED: begin
- seed_mix <= apply_wstrb(seed_mix, s_axi_wdata, s_axi_wstrb);
- rng_state <= rng_next ^ {s_axi_wdata, free_counter, ~s_axi_wdata};
- end
- default: begin
- end
- endcase
-
- if (s_axi_wlast || awleft_reg == 8'b0) begin
- write_active <= 1'b0;
- s_axi_bvalid <= 1'b1;
- end else begin
- awleft_reg <= awleft_reg - 8'd1;
- awaddr_reg <= next_awaddr;
- end
- end
-
- if (s_axi_bvalid && s_axi_bready)
- s_axi_bvalid <= 1'b0;
-
- if (s_axi_arvalid && s_axi_arready) begin
- read_active <= 1'b1;
- araddr_reg <= s_axi_araddr;
- arleft_reg <= s_axi_arlen;
- s_axi_rid <= s_axi_arid;
- s_axi_rdata <= read_reg(s_axi_araddr);
- s_axi_rlast <= (s_axi_arlen == 8'b0);
- s_axi_rvalid <= 1'b1;
- end else if (read_fire) begin
- if (data_read_fire) begin
- data_reads <= data_reads + 32'd1;
- rng_state <= rng_next ^ {rng_word, rng_word ^ free_counter[31:0],
- free_counter[63:32], ~rng_word};
- end
-
- if (s_axi_rlast) begin
- read_active <= 1'b0;
- s_axi_rvalid <= 1'b0;
- end else begin
- araddr_reg <= next_araddr;
- arleft_reg <= arleft_reg - 8'd1;
- s_axi_rdata <= read_reg(next_araddr);
- s_axi_rlast <= (arleft_reg == 8'd1);
- end
- end
- end
- end
-
-endmodule
diff --git a/rtl/soc_top.v b/rtl/soc_top.v
index e96e434..1bb6d80 100644
--- a/rtl/soc_top.v
+++ b/rtl/soc_top.v
@@ -83,6 +83,8 @@ wire cpu_clk;
wire cpu_resetn;
wire sys_clk;
wire sys_resetn;
+wire mlkem_clk;
+wire mlkem_resetn;
wire pll_locked;
generate if(SIMULATION) begin: sim_clk
@@ -91,25 +93,32 @@ generate if(SIMULATION) begin: sim_clk
initial begin
clk_sim = 1'b0;
end
- always #15 clk_sim = ~clk_sim;
-
- assign cpu_clk = clk_sim;
- assign sys_clk = clk;
- rst_sync u_rst_sys(
- .clk(sys_clk),
- .rst_n_in(~reset),
- .rst_n_out(sys_resetn)
- );
+ always #15 clk_sim = ~clk_sim;
+
+ assign cpu_clk = clk_sim;
+ assign sys_clk = clk;
+ assign mlkem_clk = sys_clk;
+ rst_sync u_rst_sys(
+ .clk(sys_clk),
+ .rst_n_in(~reset),
+ .rst_n_out(sys_resetn)
+ );
rst_sync u_rst_cpu(
.clk(cpu_clk),
- .rst_n_in(sys_resetn),
- .rst_n_out(cpu_resetn)
- );
-end
-else begin: pll_clk
+ .rst_n_in(sys_resetn),
+ .rst_n_out(cpu_resetn)
+ );
+ rst_sync u_rst_mlkem(
+ .clk(mlkem_clk),
+ .rst_n_in(sys_resetn),
+ .rst_n_out(mlkem_resetn)
+ );
+end
+else begin: pll_clk
clk_pll u_clk_pll(
.cpu_clk (cpu_clk),
.sys_clk (sys_clk),
+ .mlkem_clk (mlkem_clk),
.resetn (~reset),
.locked (pll_locked),
.clk_in1 (clk)
@@ -124,6 +133,11 @@ else begin: pll_clk
.rst_n_in(sys_resetn),
.rst_n_out(cpu_resetn)
);
+ rst_sync u_rst_mlkem(
+ .clk(mlkem_clk),
+ .rst_n_in(sys_resetn),
+ .rst_n_out(mlkem_resetn)
+ );
end
endgenerate
@@ -393,15 +407,15 @@ wire dma_s_wlast ;
wire dma_s_wvalid ;
wire dma_s_wready ;
wire [4 :0] dma_s_bid ;
-wire [1 :0] dma_s_bresp ;
-wire dma_s_bvalid ;
-wire dma_s_bready ;
-wire dma_finish ;
-wire cdma_dma_finish;
-wire cdma_matmul_seq_irq;
-assign dma_finish = cdma_dma_finish | cdma_matmul_seq_irq;
-
-// reserved
+wire [1 :0] dma_s_bresp ;
+wire dma_s_bvalid ;
+wire dma_s_bready ;
+wire dma_finish ;
+wire cdma_dma_finish;
+wire cdma_matmul_seq_irq;
+assign dma_finish = cdma_dma_finish | cdma_matmul_seq_irq;
+
+// reserved
wire [4 :0] axiOut_1_arid ;
wire [31:0] axiOut_1_araddr ;
wire [7 :0] axiOut_1_arlen ;
@@ -565,7 +579,7 @@ wire [4 :0] fft_bid ;
wire [1 :0] fft_bresp ;
wire fft_bvalid ;
wire fft_bready ;
-wire fft_finish;
+wire fft_finish;
//slave 7
wire [4 :0] axiOut_7_arid ;
@@ -606,24 +620,24 @@ wire axiOut_7_bvalid ;
wire axiOut_7_bready ;
wire confreg_int;
-wire matmul_intr;
-wire matmul_ep_wr_valid;
-wire [3:0] matmul_ep_wr_id;
-wire [3:0] matmul_ep_wr_bank;
-wire [7:0] matmul_ep_wr_word_addr;
-wire [31:0] matmul_ep_wr_data;
-wire matmul_ep_wr_ready;
-wire matmul_ep_rd_valid;
-wire [3:0] matmul_ep_rd_id;
-wire [3:0] matmul_ep_rd_bank;
-wire [7:0] matmul_ep_rd_word_addr;
-wire [31:0] matmul_ep_rd_data;
-wire matmul_ep_rd_ready;
-wire matmul_direct_start_valid;
-wire matmul_direct_start_bank;
-wire matmul_direct_done_clear;
-wire matmul_direct_crc_clear;
-wire matmul_direct_done;
+wire matmul_intr;
+wire matmul_ep_wr_valid;
+wire [3:0] matmul_ep_wr_id;
+wire [3:0] matmul_ep_wr_bank;
+wire [7:0] matmul_ep_wr_word_addr;
+wire [31:0] matmul_ep_wr_data;
+wire matmul_ep_wr_ready;
+wire matmul_ep_rd_valid;
+wire [3:0] matmul_ep_rd_id;
+wire [3:0] matmul_ep_rd_bank;
+wire [7:0] matmul_ep_rd_word_addr;
+wire [31:0] matmul_ep_rd_data;
+wire matmul_ep_rd_ready;
+wire matmul_direct_start_valid;
+wire matmul_direct_start_bank;
+wire matmul_direct_done_clear;
+wire matmul_direct_crc_clear;
+wire matmul_direct_done;
wire mlkem_intr;
AxiCrossbar_2x8 u_AxiCrossbar_2x8 (
@@ -1446,12 +1460,14 @@ axi_dvi u_axi_dvi (
);
// slave 6 - shared FFT / ML-KEM accelerator window (0x1f400000, virtual 0xbf400000)
-// ML-KEM: 0x0000..0x0114
-// FFT: 0x1000/0x2000/0x3000/0x4000/0xf000
-// PRNG: 0xd000..0xd010
+// ML-KEM: 0x0000..0x3fff
+// FFT: 0x8000..0xcfff
fft_mlkem_axi_slave u_fft_mlkem_axi_slave (
- .clk (sys_clk),
- .resetn (sys_resetn),
+ .clk (sys_clk),
+ .resetn (sys_resetn),
+
+ .mlkem_clk (mlkem_clk),
+ .mlkem_resetn (mlkem_resetn),
.s_axi_awid (fft_awid ),
.s_axi_awaddr (fft_awaddr ),
@@ -1491,12 +1507,12 @@ fft_mlkem_axi_slave u_fft_mlkem_axi_slave (
.s_axi_rdata (fft_rdata ),
.s_axi_rresp (fft_rresp ),
.s_axi_rlast (fft_rlast ),
- .s_axi_rvalid (fft_rvalid),
- .s_axi_rready (fft_rready),
-
- .fft_finish (fft_finish),
- .mlkem_intr (mlkem_intr)
-);
+ .s_axi_rvalid (fft_rvalid),
+ .s_axi_rready (fft_rready),
+
+ .fft_finish (fft_finish),
+ .mlkem_intr (mlkem_intr)
+);
//slave 7 - matmul accelerator
matmul_axi_slave u_matmul_axi_slave (
@@ -1537,35 +1553,35 @@ matmul_axi_slave u_matmul_axi_slave (
.s_axi_arvalid (axiOut_7_arvalid),
.s_axi_arready (axiOut_7_arready),
- .s_axi_rid (axiOut_7_rid ),
- .s_axi_rdata (axiOut_7_rdata ),
- .s_axi_rresp (axiOut_7_rresp ),
- .s_axi_rlast (axiOut_7_rlast ),
- .s_axi_rvalid (axiOut_7_rvalid),
- .s_axi_rready (axiOut_7_rready),
-
- .ep_wr_valid (matmul_ep_wr_valid),
- .ep_wr_id (matmul_ep_wr_id),
- .ep_wr_bank (matmul_ep_wr_bank),
- .ep_wr_word_addr(matmul_ep_wr_word_addr),
- .ep_wr_data (matmul_ep_wr_data),
- .ep_wr_ready (matmul_ep_wr_ready),
-
- .ep_rd_valid (matmul_ep_rd_valid),
- .ep_rd_id (matmul_ep_rd_id),
- .ep_rd_bank (matmul_ep_rd_bank),
- .ep_rd_word_addr(matmul_ep_rd_word_addr),
- .ep_rd_data (matmul_ep_rd_data),
- .ep_rd_ready (matmul_ep_rd_ready),
-
- .mat_start_valid(matmul_direct_start_valid),
- .mat_start_bank (matmul_direct_start_bank),
- .mat_done_clear (matmul_direct_done_clear),
- .mat_crc_clear (matmul_direct_crc_clear),
- .mat_done (matmul_direct_done),
-
- .intr (matmul_intr)
-);
+ .s_axi_rid (axiOut_7_rid ),
+ .s_axi_rdata (axiOut_7_rdata ),
+ .s_axi_rresp (axiOut_7_rresp ),
+ .s_axi_rlast (axiOut_7_rlast ),
+ .s_axi_rvalid (axiOut_7_rvalid),
+ .s_axi_rready (axiOut_7_rready),
+
+ .ep_wr_valid (matmul_ep_wr_valid),
+ .ep_wr_id (matmul_ep_wr_id),
+ .ep_wr_bank (matmul_ep_wr_bank),
+ .ep_wr_word_addr(matmul_ep_wr_word_addr),
+ .ep_wr_data (matmul_ep_wr_data),
+ .ep_wr_ready (matmul_ep_wr_ready),
+
+ .ep_rd_valid (matmul_ep_rd_valid),
+ .ep_rd_id (matmul_ep_rd_id),
+ .ep_rd_bank (matmul_ep_rd_bank),
+ .ep_rd_word_addr(matmul_ep_rd_word_addr),
+ .ep_rd_data (matmul_ep_rd_data),
+ .ep_rd_ready (matmul_ep_rd_ready),
+
+ .mat_start_valid(matmul_direct_start_valid),
+ .mat_start_bank (matmul_direct_start_bank),
+ .mat_done_clear (matmul_direct_done_clear),
+ .mat_crc_clear (matmul_direct_crc_clear),
+ .mat_done (matmul_direct_done),
+
+ .intr (matmul_intr)
+);
wire [31:0] cdma_s_awaddr;
wire [2:0] cdma_s_awprot;
@@ -1583,121 +1599,121 @@ wire [2:0] cdma_s_arprot;
wire cdma_s_arvalid;
wire cdma_s_arready;
wire [31:0] cdma_s_rdata;
-wire [1:0] cdma_s_rresp;
-wire cdma_s_rvalid;
-wire cdma_s_rready;
-
-wire [31:0] cdma_mux_awaddr;
-wire cdma_mux_awvalid;
-wire cdma_mux_awready;
-wire [31:0] cdma_mux_wdata;
-wire [3:0] cdma_mux_wstrb;
-wire cdma_mux_wvalid;
-wire cdma_mux_wready;
-wire [1:0] cdma_mux_bresp;
-wire cdma_mux_bvalid;
-wire cdma_mux_bready;
-wire [31:0] cdma_mux_araddr;
-wire cdma_mux_arvalid;
-wire cdma_mux_arready;
-wire [31:0] cdma_mux_rdata;
-wire [1:0] cdma_mux_rresp;
-wire cdma_mux_rvalid;
-wire cdma_mux_rready;
-
-wire [31:0] cdma_bridge_awaddr;
-wire cdma_bridge_awvalid;
-wire cdma_bridge_awready;
-wire [31:0] cdma_bridge_wdata;
-wire [3:0] cdma_bridge_wstrb;
-wire cdma_bridge_wvalid;
-wire cdma_bridge_wready;
-wire [1:0] cdma_bridge_bresp;
-wire cdma_bridge_bvalid;
-wire cdma_bridge_bready;
-wire [31:0] cdma_bridge_araddr;
-wire cdma_bridge_arvalid;
-wire cdma_bridge_arready;
-wire [31:0] cdma_bridge_rdata;
-wire [1:0] cdma_bridge_rresp;
-wire cdma_bridge_rvalid;
-wire cdma_bridge_rready;
-
-wire bridge_cdma_rd_req_valid;
-wire bridge_cdma_rd_req_ready;
-wire [31:0] bridge_cdma_rd_req_src;
-wire [31:0] bridge_cdma_rd_req_bytes;
-wire [3:0] bridge_cdma_rd_req_ep_id;
-wire [3:0] bridge_cdma_rd_req_ep_bank;
-wire [7:0] bridge_cdma_rd_req_ep_word_off;
-wire bridge_cdma_rd_req_done;
-wire bridge_cdma_wr_req_valid;
-wire bridge_cdma_wr_req_ready;
-wire [31:0] bridge_cdma_wr_req_dst;
-wire [31:0] bridge_cdma_wr_req_bytes;
-wire [3:0] bridge_cdma_wr_req_ep_id;
-wire [3:0] bridge_cdma_wr_req_ep_bank;
-wire [7:0] bridge_cdma_wr_req_ep_word_off;
-wire bridge_cdma_wr_req_done;
-
-wire cdma_axil_wr_bridge = (cdma_s_awaddr[11:8] == 4'h2);
-wire cdma_axil_rd_bridge = (cdma_s_araddr[11:8] == 4'h2);
-reg cdma_axil_wr_active;
-reg cdma_axil_wr_bridge_hold;
-reg cdma_axil_b_bridge;
-reg cdma_axil_r_bridge;
-wire cdma_axil_wr_sel = cdma_axil_wr_active ? cdma_axil_wr_bridge_hold : cdma_axil_wr_bridge;
-
-assign cdma_mux_awaddr = cdma_s_awaddr;
-assign cdma_mux_awvalid = cdma_s_awvalid && !cdma_axil_wr_sel;
-assign cdma_mux_wdata = cdma_s_wdata;
-assign cdma_mux_wstrb = cdma_s_wstrb;
-assign cdma_mux_wvalid = cdma_s_wvalid && !cdma_axil_wr_sel;
-assign cdma_mux_bready = cdma_s_bready && !cdma_axil_b_bridge;
-assign cdma_mux_araddr = cdma_s_araddr;
-assign cdma_mux_arvalid = cdma_s_arvalid && !cdma_axil_rd_bridge;
-assign cdma_mux_rready = cdma_s_rready && !cdma_axil_r_bridge;
-
-assign cdma_bridge_awaddr = cdma_s_awaddr;
-assign cdma_bridge_awvalid = cdma_s_awvalid && cdma_axil_wr_sel;
-assign cdma_bridge_wdata = cdma_s_wdata;
-assign cdma_bridge_wstrb = cdma_s_wstrb;
-assign cdma_bridge_wvalid = cdma_s_wvalid && cdma_axil_wr_sel;
-assign cdma_bridge_bready = cdma_s_bready && cdma_axil_b_bridge;
-assign cdma_bridge_araddr = cdma_s_araddr;
-assign cdma_bridge_arvalid = cdma_s_arvalid && cdma_axil_rd_bridge;
-assign cdma_bridge_rready = cdma_s_rready && cdma_axil_r_bridge;
-
-assign cdma_s_awready = cdma_axil_wr_sel ? cdma_bridge_awready : cdma_mux_awready;
-assign cdma_s_wready = cdma_axil_wr_sel ? cdma_bridge_wready : cdma_mux_wready;
-assign cdma_s_bresp = cdma_axil_b_bridge ? cdma_bridge_bresp : cdma_mux_bresp;
-assign cdma_s_bvalid = cdma_axil_b_bridge ? cdma_bridge_bvalid : cdma_mux_bvalid;
-assign cdma_s_arready = cdma_axil_rd_bridge ? cdma_bridge_arready : cdma_mux_arready;
-assign cdma_s_rdata = cdma_axil_r_bridge ? cdma_bridge_rdata : cdma_mux_rdata;
-assign cdma_s_rresp = cdma_axil_r_bridge ? cdma_bridge_rresp : cdma_mux_rresp;
-assign cdma_s_rvalid = cdma_axil_r_bridge ? cdma_bridge_rvalid : cdma_mux_rvalid;
-
-always @(posedge sys_clk) begin
- if (!sys_resetn) begin
- cdma_axil_wr_active <= 1'b0;
- cdma_axil_wr_bridge_hold <= 1'b0;
- cdma_axil_b_bridge <= 1'b0;
- cdma_axil_r_bridge <= 1'b0;
- end else begin
- if (cdma_s_awvalid && !cdma_axil_wr_active) begin
- cdma_axil_wr_active <= 1'b1;
- cdma_axil_wr_bridge_hold <= cdma_axil_wr_bridge;
- end
- if (cdma_s_awvalid && cdma_s_awready && cdma_s_wvalid && cdma_s_wready) begin
- cdma_axil_b_bridge <= cdma_axil_wr_sel;
- cdma_axil_wr_active <= 1'b0;
- end
- if (cdma_s_arvalid && cdma_s_arready)
- cdma_axil_r_bridge <= cdma_axil_rd_bridge;
- end
-end
-
-axi_axil_adapter #(
+wire [1:0] cdma_s_rresp;
+wire cdma_s_rvalid;
+wire cdma_s_rready;
+
+wire [31:0] cdma_mux_awaddr;
+wire cdma_mux_awvalid;
+wire cdma_mux_awready;
+wire [31:0] cdma_mux_wdata;
+wire [3:0] cdma_mux_wstrb;
+wire cdma_mux_wvalid;
+wire cdma_mux_wready;
+wire [1:0] cdma_mux_bresp;
+wire cdma_mux_bvalid;
+wire cdma_mux_bready;
+wire [31:0] cdma_mux_araddr;
+wire cdma_mux_arvalid;
+wire cdma_mux_arready;
+wire [31:0] cdma_mux_rdata;
+wire [1:0] cdma_mux_rresp;
+wire cdma_mux_rvalid;
+wire cdma_mux_rready;
+
+wire [31:0] cdma_bridge_awaddr;
+wire cdma_bridge_awvalid;
+wire cdma_bridge_awready;
+wire [31:0] cdma_bridge_wdata;
+wire [3:0] cdma_bridge_wstrb;
+wire cdma_bridge_wvalid;
+wire cdma_bridge_wready;
+wire [1:0] cdma_bridge_bresp;
+wire cdma_bridge_bvalid;
+wire cdma_bridge_bready;
+wire [31:0] cdma_bridge_araddr;
+wire cdma_bridge_arvalid;
+wire cdma_bridge_arready;
+wire [31:0] cdma_bridge_rdata;
+wire [1:0] cdma_bridge_rresp;
+wire cdma_bridge_rvalid;
+wire cdma_bridge_rready;
+
+wire bridge_cdma_rd_req_valid;
+wire bridge_cdma_rd_req_ready;
+wire [31:0] bridge_cdma_rd_req_src;
+wire [31:0] bridge_cdma_rd_req_bytes;
+wire [3:0] bridge_cdma_rd_req_ep_id;
+wire [3:0] bridge_cdma_rd_req_ep_bank;
+wire [7:0] bridge_cdma_rd_req_ep_word_off;
+wire bridge_cdma_rd_req_done;
+wire bridge_cdma_wr_req_valid;
+wire bridge_cdma_wr_req_ready;
+wire [31:0] bridge_cdma_wr_req_dst;
+wire [31:0] bridge_cdma_wr_req_bytes;
+wire [3:0] bridge_cdma_wr_req_ep_id;
+wire [3:0] bridge_cdma_wr_req_ep_bank;
+wire [7:0] bridge_cdma_wr_req_ep_word_off;
+wire bridge_cdma_wr_req_done;
+
+wire cdma_axil_wr_bridge = (cdma_s_awaddr[11:8] == 4'h2);
+wire cdma_axil_rd_bridge = (cdma_s_araddr[11:8] == 4'h2);
+reg cdma_axil_wr_active;
+reg cdma_axil_wr_bridge_hold;
+reg cdma_axil_b_bridge;
+reg cdma_axil_r_bridge;
+wire cdma_axil_wr_sel = cdma_axil_wr_active ? cdma_axil_wr_bridge_hold : cdma_axil_wr_bridge;
+
+assign cdma_mux_awaddr = cdma_s_awaddr;
+assign cdma_mux_awvalid = cdma_s_awvalid && !cdma_axil_wr_sel;
+assign cdma_mux_wdata = cdma_s_wdata;
+assign cdma_mux_wstrb = cdma_s_wstrb;
+assign cdma_mux_wvalid = cdma_s_wvalid && !cdma_axil_wr_sel;
+assign cdma_mux_bready = cdma_s_bready && !cdma_axil_b_bridge;
+assign cdma_mux_araddr = cdma_s_araddr;
+assign cdma_mux_arvalid = cdma_s_arvalid && !cdma_axil_rd_bridge;
+assign cdma_mux_rready = cdma_s_rready && !cdma_axil_r_bridge;
+
+assign cdma_bridge_awaddr = cdma_s_awaddr;
+assign cdma_bridge_awvalid = cdma_s_awvalid && cdma_axil_wr_sel;
+assign cdma_bridge_wdata = cdma_s_wdata;
+assign cdma_bridge_wstrb = cdma_s_wstrb;
+assign cdma_bridge_wvalid = cdma_s_wvalid && cdma_axil_wr_sel;
+assign cdma_bridge_bready = cdma_s_bready && cdma_axil_b_bridge;
+assign cdma_bridge_araddr = cdma_s_araddr;
+assign cdma_bridge_arvalid = cdma_s_arvalid && cdma_axil_rd_bridge;
+assign cdma_bridge_rready = cdma_s_rready && cdma_axil_r_bridge;
+
+assign cdma_s_awready = cdma_axil_wr_sel ? cdma_bridge_awready : cdma_mux_awready;
+assign cdma_s_wready = cdma_axil_wr_sel ? cdma_bridge_wready : cdma_mux_wready;
+assign cdma_s_bresp = cdma_axil_b_bridge ? cdma_bridge_bresp : cdma_mux_bresp;
+assign cdma_s_bvalid = cdma_axil_b_bridge ? cdma_bridge_bvalid : cdma_mux_bvalid;
+assign cdma_s_arready = cdma_axil_rd_bridge ? cdma_bridge_arready : cdma_mux_arready;
+assign cdma_s_rdata = cdma_axil_r_bridge ? cdma_bridge_rdata : cdma_mux_rdata;
+assign cdma_s_rresp = cdma_axil_r_bridge ? cdma_bridge_rresp : cdma_mux_rresp;
+assign cdma_s_rvalid = cdma_axil_r_bridge ? cdma_bridge_rvalid : cdma_mux_rvalid;
+
+always @(posedge sys_clk) begin
+ if (!sys_resetn) begin
+ cdma_axil_wr_active <= 1'b0;
+ cdma_axil_wr_bridge_hold <= 1'b0;
+ cdma_axil_b_bridge <= 1'b0;
+ cdma_axil_r_bridge <= 1'b0;
+ end else begin
+ if (cdma_s_awvalid && !cdma_axil_wr_active) begin
+ cdma_axil_wr_active <= 1'b1;
+ cdma_axil_wr_bridge_hold <= cdma_axil_wr_bridge;
+ end
+ if (cdma_s_awvalid && cdma_s_awready && cdma_s_wvalid && cdma_s_wready) begin
+ cdma_axil_b_bridge <= cdma_axil_wr_sel;
+ cdma_axil_wr_active <= 1'b0;
+ end
+ if (cdma_s_arvalid && cdma_s_arready)
+ cdma_axil_r_bridge <= cdma_axil_rd_bridge;
+ end
+end
+
+axi_axil_adapter #(
.ADDR_WIDTH(32),
.AXI_DATA_WIDTH(32),
.AXI_ID_WIDTH(5),
@@ -1762,63 +1778,63 @@ u_axi_axil_adapter(
.m_axil_rdata(cdma_s_rdata),
.m_axil_rresp(cdma_s_rresp),
.m_axil_rvalid(cdma_s_rvalid),
- .m_axil_rready(cdma_s_rready)
-);
-
-cdma_matmul_bridge #(
- .ADDR_WIDTH (32),
- .AXIL_ADDR_WIDTH (32),
- .AXIL_DATA_WIDTH (32)
-) u_cdma_matmul_bridge (
- .clk (sys_clk),
- .rst_n (sys_resetn),
-
- .s_axil_awaddr (cdma_bridge_awaddr),
- .s_axil_awvalid (cdma_bridge_awvalid),
- .s_axil_awready (cdma_bridge_awready),
- .s_axil_wdata (cdma_bridge_wdata),
- .s_axil_wstrb (cdma_bridge_wstrb),
- .s_axil_wvalid (cdma_bridge_wvalid),
- .s_axil_wready (cdma_bridge_wready),
- .s_axil_bresp (cdma_bridge_bresp),
- .s_axil_bvalid (cdma_bridge_bvalid),
- .s_axil_bready (cdma_bridge_bready),
- .s_axil_araddr (cdma_bridge_araddr),
- .s_axil_arvalid (cdma_bridge_arvalid),
- .s_axil_arready (cdma_bridge_arready),
- .s_axil_rdata (cdma_bridge_rdata),
- .s_axil_rresp (cdma_bridge_rresp),
- .s_axil_rvalid (cdma_bridge_rvalid),
- .s_axil_rready (cdma_bridge_rready),
-
- .cdma_rd_req_valid (bridge_cdma_rd_req_valid),
- .cdma_rd_req_ready (bridge_cdma_rd_req_ready),
- .cdma_rd_req_src (bridge_cdma_rd_req_src),
- .cdma_rd_req_bytes (bridge_cdma_rd_req_bytes),
- .cdma_rd_req_ep_id (bridge_cdma_rd_req_ep_id),
- .cdma_rd_req_ep_bank (bridge_cdma_rd_req_ep_bank),
- .cdma_rd_req_ep_word_off(bridge_cdma_rd_req_ep_word_off),
- .cdma_rd_req_done (bridge_cdma_rd_req_done),
-
- .cdma_wr_req_valid (bridge_cdma_wr_req_valid),
- .cdma_wr_req_ready (bridge_cdma_wr_req_ready),
- .cdma_wr_req_dst (bridge_cdma_wr_req_dst),
- .cdma_wr_req_bytes (bridge_cdma_wr_req_bytes),
- .cdma_wr_req_ep_id (bridge_cdma_wr_req_ep_id),
- .cdma_wr_req_ep_bank (bridge_cdma_wr_req_ep_bank),
- .cdma_wr_req_ep_word_off(bridge_cdma_wr_req_ep_word_off),
- .cdma_wr_req_done (bridge_cdma_wr_req_done),
-
- .mat_start_valid (matmul_direct_start_valid),
- .mat_start_bank (matmul_direct_start_bank),
- .mat_done_clear (matmul_direct_done_clear),
- .mat_crc_clear (matmul_direct_crc_clear),
- .mat_done (matmul_direct_done),
- .seq_done_irq (cdma_matmul_seq_irq)
-);
-
-// 8 通道 DMA
-snix_axil_cdma_mux #(
+ .m_axil_rready(cdma_s_rready)
+);
+
+cdma_matmul_bridge #(
+ .ADDR_WIDTH (32),
+ .AXIL_ADDR_WIDTH (32),
+ .AXIL_DATA_WIDTH (32)
+) u_cdma_matmul_bridge (
+ .clk (sys_clk),
+ .rst_n (sys_resetn),
+
+ .s_axil_awaddr (cdma_bridge_awaddr),
+ .s_axil_awvalid (cdma_bridge_awvalid),
+ .s_axil_awready (cdma_bridge_awready),
+ .s_axil_wdata (cdma_bridge_wdata),
+ .s_axil_wstrb (cdma_bridge_wstrb),
+ .s_axil_wvalid (cdma_bridge_wvalid),
+ .s_axil_wready (cdma_bridge_wready),
+ .s_axil_bresp (cdma_bridge_bresp),
+ .s_axil_bvalid (cdma_bridge_bvalid),
+ .s_axil_bready (cdma_bridge_bready),
+ .s_axil_araddr (cdma_bridge_araddr),
+ .s_axil_arvalid (cdma_bridge_arvalid),
+ .s_axil_arready (cdma_bridge_arready),
+ .s_axil_rdata (cdma_bridge_rdata),
+ .s_axil_rresp (cdma_bridge_rresp),
+ .s_axil_rvalid (cdma_bridge_rvalid),
+ .s_axil_rready (cdma_bridge_rready),
+
+ .cdma_rd_req_valid (bridge_cdma_rd_req_valid),
+ .cdma_rd_req_ready (bridge_cdma_rd_req_ready),
+ .cdma_rd_req_src (bridge_cdma_rd_req_src),
+ .cdma_rd_req_bytes (bridge_cdma_rd_req_bytes),
+ .cdma_rd_req_ep_id (bridge_cdma_rd_req_ep_id),
+ .cdma_rd_req_ep_bank (bridge_cdma_rd_req_ep_bank),
+ .cdma_rd_req_ep_word_off(bridge_cdma_rd_req_ep_word_off),
+ .cdma_rd_req_done (bridge_cdma_rd_req_done),
+
+ .cdma_wr_req_valid (bridge_cdma_wr_req_valid),
+ .cdma_wr_req_ready (bridge_cdma_wr_req_ready),
+ .cdma_wr_req_dst (bridge_cdma_wr_req_dst),
+ .cdma_wr_req_bytes (bridge_cdma_wr_req_bytes),
+ .cdma_wr_req_ep_id (bridge_cdma_wr_req_ep_id),
+ .cdma_wr_req_ep_bank (bridge_cdma_wr_req_ep_bank),
+ .cdma_wr_req_ep_word_off(bridge_cdma_wr_req_ep_word_off),
+ .cdma_wr_req_done (bridge_cdma_wr_req_done),
+
+ .mat_start_valid (matmul_direct_start_valid),
+ .mat_start_bank (matmul_direct_start_bank),
+ .mat_done_clear (matmul_direct_done_clear),
+ .mat_crc_clear (matmul_direct_crc_clear),
+ .mat_done (matmul_direct_done),
+ .seq_done_irq (cdma_matmul_seq_irq)
+);
+
+// 8 通道 DMA
+snix_axil_cdma_mux #(
.ADDR_WIDTH (32),
.DATA_WIDTH (32), // 适配你的 32-bit 总线
.AXIL_ADDR_WIDTH (32),
@@ -1832,23 +1848,23 @@ snix_axil_cdma_mux #(
.rst_n (sys_resetn), // 低电平复位
// AXI-Lite 从机接口 (接 CPU 发来的配置请求)
- .s_axil_awaddr (cdma_mux_awaddr),
- .s_axil_awvalid (cdma_mux_awvalid),
- .s_axil_awready (cdma_mux_awready),
- .s_axil_wdata (cdma_mux_wdata),
- .s_axil_wstrb (cdma_mux_wstrb),
- .s_axil_wvalid (cdma_mux_wvalid),
- .s_axil_wready (cdma_mux_wready),
- .s_axil_bresp (cdma_mux_bresp),
- .s_axil_bvalid (cdma_mux_bvalid),
- .s_axil_bready (cdma_mux_bready),
- .s_axil_araddr (cdma_mux_araddr),
- .s_axil_arvalid (cdma_mux_arvalid),
- .s_axil_arready (cdma_mux_arready),
- .s_axil_rdata (cdma_mux_rdata),
- .s_axil_rresp (cdma_mux_rresp),
- .s_axil_rvalid (cdma_mux_rvalid),
- .s_axil_rready (cdma_mux_rready),
+ .s_axil_awaddr (cdma_mux_awaddr),
+ .s_axil_awvalid (cdma_mux_awvalid),
+ .s_axil_awready (cdma_mux_awready),
+ .s_axil_wdata (cdma_mux_wdata),
+ .s_axil_wstrb (cdma_mux_wstrb),
+ .s_axil_wvalid (cdma_mux_wvalid),
+ .s_axil_wready (cdma_mux_wready),
+ .s_axil_bresp (cdma_mux_bresp),
+ .s_axil_bvalid (cdma_mux_bvalid),
+ .s_axil_bready (cdma_mux_bready),
+ .s_axil_araddr (cdma_mux_araddr),
+ .s_axil_arvalid (cdma_mux_arvalid),
+ .s_axil_arready (cdma_mux_arready),
+ .s_axil_rdata (cdma_mux_rdata),
+ .s_axil_rresp (cdma_mux_rresp),
+ .s_axil_rvalid (cdma_mux_rvalid),
+ .s_axil_rready (cdma_mux_rready),
// AXI4 主机接口 (接 Crossbar 去搬运数据)
.mm2mm_awid (dma_m_awid),
@@ -1891,43 +1907,43 @@ snix_axil_cdma_mux #(
.mm2mm_rresp (dma_m_rresp),
.mm2mm_rlast (dma_m_rlast),
.mm2mm_ruser (1'b0),
- .mm2mm_rvalid (dma_m_rvalid),
- .mm2mm_rready (dma_m_rready),
-
- // 本地直连到 matmul，不经过 AXI crossbar
- .ep_wr_valid (matmul_ep_wr_valid),
- .ep_wr_id (matmul_ep_wr_id),
- .ep_wr_bank (matmul_ep_wr_bank),
- .ep_wr_word_addr (matmul_ep_wr_word_addr),
- .ep_wr_data (matmul_ep_wr_data),
- .ep_wr_ready (matmul_ep_wr_ready),
- .ep_rd_valid (matmul_ep_rd_valid),
- .ep_rd_id (matmul_ep_rd_id),
- .ep_rd_bank (matmul_ep_rd_bank),
- .ep_rd_word_addr (matmul_ep_rd_word_addr),
- .ep_rd_data (matmul_ep_rd_data),
- .ep_rd_ready (matmul_ep_rd_ready),
-
- .ext_rd_req_valid (bridge_cdma_rd_req_valid),
- .ext_rd_req_ready (bridge_cdma_rd_req_ready),
- .ext_rd_req_src (bridge_cdma_rd_req_src),
- .ext_rd_req_bytes (bridge_cdma_rd_req_bytes),
- .ext_rd_req_ep_id (bridge_cdma_rd_req_ep_id),
- .ext_rd_req_ep_bank (bridge_cdma_rd_req_ep_bank),
- .ext_rd_req_ep_word_off(bridge_cdma_rd_req_ep_word_off),
- .ext_rd_req_done (bridge_cdma_rd_req_done),
-
- .ext_wr_req_valid (bridge_cdma_wr_req_valid),
- .ext_wr_req_ready (bridge_cdma_wr_req_ready),
- .ext_wr_req_dst (bridge_cdma_wr_req_dst),
- .ext_wr_req_bytes (bridge_cdma_wr_req_bytes),
- .ext_wr_req_ep_id (bridge_cdma_wr_req_ep_id),
- .ext_wr_req_ep_bank (bridge_cdma_wr_req_ep_bank),
- .ext_wr_req_ep_word_off(bridge_cdma_wr_req_ep_word_off),
- .ext_wr_req_done (bridge_cdma_wr_req_done),
-
- // 全局中断输出
- .dma_finish (cdma_dma_finish)
-);
-
-endmodule
+ .mm2mm_rvalid (dma_m_rvalid),
+ .mm2mm_rready (dma_m_rready),
+
+ // 本地直连到 matmul，不经过 AXI crossbar
+ .ep_wr_valid (matmul_ep_wr_valid),
+ .ep_wr_id (matmul_ep_wr_id),
+ .ep_wr_bank (matmul_ep_wr_bank),
+ .ep_wr_word_addr (matmul_ep_wr_word_addr),
+ .ep_wr_data (matmul_ep_wr_data),
+ .ep_wr_ready (matmul_ep_wr_ready),
+ .ep_rd_valid (matmul_ep_rd_valid),
+ .ep_rd_id (matmul_ep_rd_id),
+ .ep_rd_bank (matmul_ep_rd_bank),
+ .ep_rd_word_addr (matmul_ep_rd_word_addr),
+ .ep_rd_data (matmul_ep_rd_data),
+ .ep_rd_ready (matmul_ep_rd_ready),
+
+ .ext_rd_req_valid (bridge_cdma_rd_req_valid),
+ .ext_rd_req_ready (bridge_cdma_rd_req_ready),
+ .ext_rd_req_src (bridge_cdma_rd_req_src),
+ .ext_rd_req_bytes (bridge_cdma_rd_req_bytes),
+ .ext_rd_req_ep_id (bridge_cdma_rd_req_ep_id),
+ .ext_rd_req_ep_bank (bridge_cdma_rd_req_ep_bank),
+ .ext_rd_req_ep_word_off(bridge_cdma_rd_req_ep_word_off),
+ .ext_rd_req_done (bridge_cdma_rd_req_done),
+
+ .ext_wr_req_valid (bridge_cdma_wr_req_valid),
+ .ext_wr_req_ready (bridge_cdma_wr_req_ready),
+ .ext_wr_req_dst (bridge_cdma_wr_req_dst),
+ .ext_wr_req_bytes (bridge_cdma_wr_req_bytes),
+ .ext_wr_req_ep_id (bridge_cdma_wr_req_ep_id),
+ .ext_wr_req_ep_bank (bridge_cdma_wr_req_ep_bank),
+ .ext_wr_req_ep_word_off(bridge_cdma_wr_req_ep_word_off),
+ .ext_wr_req_done (bridge_cdma_wr_req_done),
+
+ // 全局中断输出
+ .dma_finish (cdma_dma_finish)
+);
+
+endmodule
diff --git a/sdk/sdk b/sdk/sdk
new file mode 100644
index 0000000..b73bb21
Binary files /dev/null and b/sdk/sdk differ
diff --git a/sdk/software/examples/mlkem_test/Makefile b/sdk/software/examples/mlkem_test/Makefile
index 058c609..2378df0 100644
--- a/sdk/software/examples/mlkem_test/Makefile
+++ b/sdk/software/examples/mlkem_test/Makefile
@@ -2,17 +2,47 @@ TARGET = mlkem_test
CFLAGS += -O3 -g
CFLAGS += -I./mlkem_native
-CFLAGS += -DMLK_CONFIG_PARAMETER_SET=512
-CFLAGS += -DMLK_CONFIG_NAMESPACE_PREFIX=swmlkem
-CFLAGS += -DMLK_CONFIG_NO_RANDOMIZED_API
-CFLAGS += -DMLK_CONFIG_NO_ASM
-CFLAGS += -DMLK_CONFIG_FILE='"sw_mlkem_config.h"'
-C_SRCS := $(wildcard ./*.c )
-C_SRCS += ./mlkem_native/mlkem_native.c
+C_SRCS := ./dma.c ./main.c ./sw_mlkem_alloc.c
+
+MLKEM_NATIVE_CFLAGS = \
+ -DMLK_CONFIG_NO_RANDOMIZED_API \
+ -DMLK_CONFIG_NO_SUPERCOP \
+ -DMLK_CONFIG_NO_ASM \
+ -DMLK_CONFIG_FILE='"sw_mlkem_config.h"'
+
+MLKEM_NATIVE_OBJS = \
+ obj/mlkem_native_512.o \
+ obj/mlkem_native_768.o \
+ obj/mlkem_native_1024.o
+
+LINK_OBJS += $(MLKEM_NATIVE_OBJS)
+CLEAN_OBJS += $(MLKEM_NATIVE_OBJS)
OBJDIR = obj
COMMON_DIR = ../../bsp
GCC_DIR=../../../toolchains/loongson-gnu-toolchain-8.3-x86_64-loongarch32r-linux-gnusf-v2.0
PICOLIBC_DIR=../../../toolchains/picolibc
+LA32RSOC_WINDOWS_HOME ?= ../../..
include ../../bsp/common.mk
+
+obj/mlkem_native_512.o: ./mlkem_native/mlkem_native.c
+ mkdir -p obj
+ $(LA32R_GCC) $(CFLAGS) $(INCLUDES) $(MLKEM_NATIVE_CFLAGS) \
+ -DMLK_CONFIG_PARAMETER_SET=512 \
+ -DMLK_CONFIG_NAMESPACE_PREFIX=swmlkem512 \
+ -c -o $@ $<
+
+obj/mlkem_native_768.o: ./mlkem_native/mlkem_native.c
+ mkdir -p obj
+ $(LA32R_GCC) $(CFLAGS) $(INCLUDES) $(MLKEM_NATIVE_CFLAGS) \
+ -DMLK_CONFIG_PARAMETER_SET=768 \
+ -DMLK_CONFIG_NAMESPACE_PREFIX=swmlkem768 \
+ -c -o $@ $<
+
+obj/mlkem_native_1024.o: ./mlkem_native/mlkem_native.c
+ mkdir -p obj
+ $(LA32R_GCC) $(CFLAGS) $(INCLUDES) $(MLKEM_NATIVE_CFLAGS) \
+ -DMLK_CONFIG_PARAMETER_SET=1024 \
+ -DMLK_CONFIG_NAMESPACE_PREFIX=swmlkem1024 \
+ -c -o $@ $<
diff --git a/sdk/software/examples/mlkem_test/main.c b/sdk/software/examples/mlkem_test/main.c
index 89d7089..284d22e 100644
--- a/sdk/software/examples/mlkem_test/main.c
+++ b/sdk/software/examples/mlkem_test/main.c
@@ -1,19 +1,9 @@
/**
- * mlkem_test - ML-KEM hardware accelerator example (CIC-IEC 2026)
+ * mlkem_test - ML-KEM hardware accelerator comparison test.
*
- * Exercises the mlkem_axi_slave at 0xbf400000 by running a full
- * KeyGen -> Encaps -> Decaps round trip on ML-KEM-512 (k=2) and verifying
- * that the decapsulated shared secret matches the encapsulated one. This is a
- * self-consistency check: it needs no external golden vectors, yet touches
- * every register path (seeds, start/poll, byte preload, byte readback, ss).
- *
- * KeyGen : d,z -> ek (800 B), dk (1632 B)
- * Encaps : ek + msg -> ss, ct (768 B)
- * Decaps : dk + ct -> ss' (must equal ss)
- *
- * The core runs at sys_clk/2 (25 MHz); the wrapper hides the clock crossing,
- * so software just does normal MMIO loads/stores (preload writes are
- * automatically wait-stated on the bus).
+ * Runs ML-KEM-512/768/1024 (k=2/3/4) with deterministic KAT count=0 inputs
+ * and compares software, hardware through CPU register accesses, and hardware
+ * using DMA window transfers.
*/
#include
@@ -23,167 +13,265 @@
#include
#include "dma.h"
-#include "mlkem_native.h"
-/* ─── BSP globals ─────────────────────────────────────────────────── */
unsigned long UART_BASE = 0xbf000000;
unsigned long CONFREG_TIMER_BASE = 0xbf20f100;
unsigned long CONFREG_CLOCKS_PER_SEC = 50000000L;
unsigned long CORE_CLOCKS_PER_SEC = 33000000L;
-/* ─── ML-KEM IP register map (base 0xbf400000) ────────────────────── */
#define MLKEM_BASE 0xbf400000
-#define MLKEM_CTRL (MLKEM_BASE + 0x000) /* W bit0 = start */
-#define MLKEM_STATUS (MLKEM_BASE + 0x004) /* R busy/done/preload_busy */
-#define MLKEM_CONFIG (MLKEM_BASE + 0x008) /* RW [2:0]=k [5:4]=op */
-#define MLKEM_D (MLKEM_BASE + 0x010) /* RW seed d (8 words) */
-#define MLKEM_Z (MLKEM_BASE + 0x030) /* RW seed z (8 words) */
-#define MLKEM_MSG (MLKEM_BASE + 0x050) /* RW message (8 words) */
-#define MLKEM_SS (MLKEM_BASE + 0x070) /* R shared secret (8 words) */
-#define MLKEM_PRE_SEL (MLKEM_BASE + 0x100) /* RW 0=ek 1=dk 2=c */
-#define MLKEM_PRE_ADR (MLKEM_BASE + 0x104) /* RW byte addr (auto-incr) */
-#define MLKEM_PRE_DAT (MLKEM_BASE + 0x108) /* W byte -> selected port */
-#define MLKEM_RDB_SEL (MLKEM_BASE + 0x10C) /* RW 0=ek 1=dk_pke 2=dk 3=ct */
-#define MLKEM_RDB_ADR (MLKEM_BASE + 0x110) /* RW output byte index */
-#define MLKEM_RDB_DAT (MLKEM_BASE + 0x114) /* R selected byte (low 8b) */
-#define MLKEM_EK_WINDOW (MLKEM_BASE + 0x1000) /* W preload ek, R read ek */
-#define MLKEM_DK_WINDOW (MLKEM_BASE + 0x2000) /* W preload dk, R read dk */
-#define MLKEM_CT_WINDOW (MLKEM_BASE + 0x3000) /* W preload c, R read ct */
-
-/* PRNG sub-window in the shared ML-KEM/FFT/PRNG AXI slave. */
-#define PRNG_BASE (MLKEM_BASE + 0xD000)
-#define PRNG_CTRL (PRNG_BASE + 0x000) /* W bit0=mix seed bit1=clr count */
-#define PRNG_STATUS (PRNG_BASE + 0x004) /* R bit0=ready */
-#define PRNG_DATA (PRNG_BASE + 0x008) /* R next pseudo-random word */
-#define PRNG_COUNT (PRNG_BASE + 0x00C) /* R data read count */
-#define PRNG_SEED (PRNG_BASE + 0x010) /* RW software seed mix value */
+#define MLKEM_CTRL (MLKEM_BASE + 0x000)
+#define MLKEM_STATUS (MLKEM_BASE + 0x004)
+#define MLKEM_CONFIG (MLKEM_BASE + 0x008)
+#define MLKEM_D (MLKEM_BASE + 0x010)
+#define MLKEM_Z (MLKEM_BASE + 0x030)
+#define MLKEM_MSG (MLKEM_BASE + 0x050)
+#define MLKEM_SS (MLKEM_BASE + 0x070)
+#define MLKEM_PRE_SEL (MLKEM_BASE + 0x100)
+#define MLKEM_PRE_ADR (MLKEM_BASE + 0x104)
+#define MLKEM_PRE_DAT (MLKEM_BASE + 0x108)
+#define MLKEM_RDB_SEL (MLKEM_BASE + 0x10C)
+#define MLKEM_RDB_ADR (MLKEM_BASE + 0x110)
+#define MLKEM_RDB_DAT (MLKEM_BASE + 0x114)
+#define MLKEM_EK_WINDOW (MLKEM_BASE + 0x1000)
+#define MLKEM_DK_WINDOW (MLKEM_BASE + 0x2000)
+#define MLKEM_CT_WINDOW (MLKEM_BASE + 0x3000)
#define CTRL_START 0x1
-#define STATUS_BUSY 0x1
#define STATUS_DONE 0x2
-#define PRNG_READY 0x1
-#define PRNG_CTRL_MIX 0x1
#define OP_KEYGEN 0
#define OP_ENCAPS 1
#define OP_DECAPS 2
-/* preload select */
#define PRE_EK 0
#define PRE_DK 1
#define PRE_C 2
-/* readback select */
#define RDB_EK 0
-#define RDB_DKP 1
#define RDB_DK 2
#define RDB_CT 3
-/* ─── ML-KEM-512 (k=2) byte sizes ─────────────────────────────────── */
-#define KVAL 2
-#define EK_BYTES (384 * KVAL + 32) /* 800 */
-#define DKP_BYTES (384 * KVAL) /* 768 */
-#define DK_BYTES (DKP_BYTES + EK_BYTES + 32 + 32) /* 1632 */
-#define CT_BYTES 768
+#define MAX_K 4
+#define MAX_EK_BYTES (384 * MAX_K + 32)
+#define MAX_DK_BYTES (768 * MAX_K + 96)
+#define MAX_CT_BYTES 1568
+#define SYM_BYTES 32
+#define RUN_SW_DECAPS 1
+#define DIAG_ITERS 100
+#define TRACE_STAGES 0
-/* ─── Convenience macros ──────────────────────────────────────────── */
#define REG_WR(addr, val) (*(volatile uint32_t *)(addr) = (uint32_t)(val))
#define REG_RD(addr) (*(volatile uint32_t *)(addr))
#define PHYS_ADDR(addr) ((uint32_t)(addr) & 0x1FFFFFFFu)
#define UNCACHED_ADDR(addr) ((PHYS_ADDR(addr)) | 0xA0000000u)
-/* PLACEHOLDER */
+typedef int (*mlkem_keypair_derand_fn)(uint8_t *pk, uint8_t *sk,
+ const uint8_t *coins);
+typedef int (*mlkem_enc_derand_fn)(uint8_t *ct, uint8_t *ss,
+ const uint8_t *pk, const uint8_t *msg);
+typedef int (*mlkem_dec_fn)(uint8_t *ss, const uint8_t *ct,
+ const uint8_t *sk);
-/* ─── Wait until the core finishes the current op ─────────────────── */
-static void wait_done(void)
+extern int swmlkem512_keypair_derand(uint8_t *pk, uint8_t *sk,
+ const uint8_t *coins);
+extern int swmlkem512_enc_derand(uint8_t *ct, uint8_t *ss,
+ const uint8_t *pk, const uint8_t *msg);
+extern int swmlkem512_dec(uint8_t *ss, const uint8_t *ct,
+ const uint8_t *sk);
+extern int swmlkem768_keypair_derand(uint8_t *pk, uint8_t *sk,
+ const uint8_t *coins);
+extern int swmlkem768_enc_derand(uint8_t *ct, uint8_t *ss,
+ const uint8_t *pk, const uint8_t *msg);
+extern int swmlkem768_dec(uint8_t *ss, const uint8_t *ct,
+ const uint8_t *sk);
+extern int swmlkem1024_keypair_derand(uint8_t *pk, uint8_t *sk,
+ const uint8_t *coins);
+extern int swmlkem1024_enc_derand(uint8_t *ct, uint8_t *ss,
+ const uint8_t *pk, const uint8_t *msg);
+extern int swmlkem1024_dec(uint8_t *ss, const uint8_t *ct,
+ const uint8_t *sk);
+
+typedef struct {
+ int k;
+ const char *name;
+ mlkem_keypair_derand_fn keypair_derand;
+ mlkem_enc_derand_fn enc_derand;
+ mlkem_dec_fn dec;
+ const uint8_t *expect_ek32;
+ const uint8_t *expect_ct32;
+ const uint8_t *expect_ss;
+} mlkem_level_t;
+
+typedef struct {
+ int ek_bytes;
+ int dk_bytes;
+ int ct_bytes;
+} mlkem_sizes_t;
+
+typedef struct {
+ uint8_t ek[MAX_EK_BYTES] __attribute__((aligned(64)));
+ uint8_t dk[MAX_DK_BYTES] __attribute__((aligned(64)));
+ uint8_t ct[MAX_CT_BYTES] __attribute__((aligned(64)));
+ uint8_t ss_enc[SYM_BYTES];
+ uint8_t ss_dec[SYM_BYTES];
+} mlkem_result_t;
+
+typedef struct {
+ unsigned long keygen;
+ unsigned long encaps;
+ unsigned long decaps;
+} stage_cycles_t;
+
+static mlkem_result_t sw_res;
+static mlkem_result_t cpu_res;
+static mlkem_result_t dma_res;
+
+static const uint8_t EXPECT_512_EK32[SYM_BYTES] = {
+ 0x80, 0xb7, 0x01, 0x00, 0x83, 0x1d, 0x7e, 0xe1,
+ 0x68, 0xf4, 0x73, 0x92, 0x48, 0x31, 0x19, 0xf9,
+ 0x38, 0x63, 0xec, 0xb0, 0xbc, 0x60, 0x53, 0x12,
+ 0xd3, 0x57, 0x28, 0x87, 0x6b, 0x00, 0xf5, 0xf5
+};
+
+static const uint8_t EXPECT_512_CT32[SYM_BYTES] = {
+ 0x91, 0x7e, 0x59, 0xd1, 0xe1, 0x82, 0xa2, 0x96,
+ 0x48, 0xa6, 0x19, 0x55, 0xa1, 0x54, 0xb3, 0x04,
+ 0xa8, 0x42, 0xf4, 0x58, 0xeb, 0xa5, 0x93, 0xf3,
+ 0x06, 0x5f, 0xb7, 0xd7, 0x44, 0xfb, 0xa2, 0x4f
+};
+
+static const uint8_t EXPECT_512_SS[SYM_BYTES] = {
+ 0x2b, 0x5c, 0x52, 0xee, 0x72, 0x94, 0x63, 0x31,
+ 0x98, 0x3b, 0xa0, 0x50, 0xbe, 0x0f, 0x43, 0x50,
+ 0x55, 0xc0, 0x54, 0x79, 0x01, 0xe0, 0x35, 0x59,
+ 0xb3, 0x56, 0x51, 0x78, 0x89, 0xea, 0x27, 0xc5
+};
+
+static const uint8_t EXPECT_768_EK32[SYM_BYTES] = {
+ 0x01, 0xf6, 0x0a, 0xf1, 0xdc, 0x8e, 0x63, 0x60,
+ 0xae, 0x78, 0xb5, 0x9d, 0x4a, 0x50, 0x42, 0xeb,
+ 0x91, 0x45, 0xa2, 0x69, 0x04, 0x6d, 0x62, 0x36,
+ 0xb8, 0x30, 0x4f, 0x30, 0x5c, 0x2d, 0x9d, 0xcb
+};
+
+static const uint8_t EXPECT_768_CT32[SYM_BYTES] = {
+ 0x16, 0xa6, 0x1f, 0xf8, 0x47, 0x87, 0xfd, 0x4a,
+ 0x5f, 0x19, 0xca, 0x59, 0xb3, 0x65, 0x7d, 0xb3,
+ 0xa1, 0x06, 0xa7, 0x32, 0x9e, 0x2d, 0x62, 0x74,
+ 0x7a, 0x2e, 0xf8, 0x51, 0x49, 0x16, 0x31, 0x09
+};
+
+static const uint8_t EXPECT_768_SS[SYM_BYTES] = {
+ 0xb4, 0x08, 0xd5, 0xd1, 0x15, 0x71, 0x3f, 0x0a,
+ 0x93, 0x04, 0x7d, 0xbb, 0xea, 0x83, 0x2e, 0x43,
+ 0x40, 0x78, 0x76, 0x86, 0xd5, 0x9a, 0x9a, 0x2d,
+ 0x10, 0x6b, 0xd6, 0x62, 0xba, 0x0a, 0xa0, 0x35
+};
+
+static const uint8_t EXPECT_1024_EK32[SYM_BYTES] = {
+ 0xa8, 0x12, 0x2f, 0x37, 0x6b, 0x3f, 0x5d, 0x35,
+ 0x52, 0x63, 0xeb, 0xa5, 0x22, 0xc4, 0x39, 0x95,
+ 0x04, 0x4b, 0xec, 0xa7, 0x8b, 0x9b, 0xa9, 0x92,
+ 0x4b, 0x87, 0x03, 0x54, 0x90, 0xae, 0x78, 0xa9
+};
+
+static const uint8_t EXPECT_1024_CT32[SYM_BYTES] = {
+ 0x70, 0x7d, 0x18, 0xca, 0xbc, 0xf8, 0x96, 0x70,
+ 0xc8, 0x00, 0x03, 0xb4, 0x7d, 0x2b, 0x86, 0x78,
+ 0xad, 0x0d, 0xa4, 0xaa, 0xad, 0x78, 0x1a, 0x33,
+ 0x51, 0xe8, 0x2a, 0xc3, 0xe4, 0x47, 0xe7, 0x01
+};
+
+static const uint8_t EXPECT_1024_SS[SYM_BYTES] = {
+ 0x23, 0xf2, 0x11, 0xb8, 0x4a, 0x6e, 0xe2, 0x0c,
+ 0x8c, 0x29, 0xf6, 0xe5, 0x31, 0x4c, 0x91, 0xb4,
+ 0x14, 0xe9, 0x40, 0x51, 0x3d, 0x38, 0x0a, 0xdd,
+ 0x17, 0xbd, 0x72, 0x4a, 0xb3, 0xa1, 0x3a, 0x52
+};
+
+static const mlkem_level_t LEVELS[] = {
+ // {2, "ML-KEM-512", swmlkem512_keypair_derand, swmlkem512_enc_derand, swmlkem512_dec,
+ // EXPECT_512_EK32, EXPECT_512_CT32, EXPECT_512_SS},
+ // {3, "ML-KEM-768", swmlkem768_keypair_derand, swmlkem768_enc_derand, swmlkem768_dec,
+ // EXPECT_768_EK32, EXPECT_768_CT32, EXPECT_768_SS},
+ {4, "ML-KEM-1024", swmlkem1024_keypair_derand, swmlkem1024_enc_derand, swmlkem1024_dec,
+ EXPECT_1024_EK32, EXPECT_1024_CT32, EXPECT_1024_SS},
+};
+
+/* Deterministic ML-KEM KAT count=0 inputs. */
+static const uint8_t SEED_D[SYM_BYTES] = {
+ 0x6d, 0xbb, 0xc4, 0x37, 0x51, 0x36, 0xdf, 0x3b,
+ 0x07, 0xf7, 0xc7, 0x0e, 0x63, 0x9e, 0x22, 0x3e,
+ 0x17, 0x7e, 0x7f, 0xd5, 0x3b, 0x16, 0x1b, 0x3f,
+ 0x4d, 0x57, 0x79, 0x17, 0x94, 0xf1, 0x26, 0x24
+};
+
+static const uint8_t SEED_Z[SYM_BYTES] = {
+ 0xf6, 0x96, 0x48, 0x40, 0x48, 0xec, 0x21, 0xf9,
+ 0x6c, 0xf5, 0x0a, 0x56, 0xd0, 0x75, 0x9c, 0x44,
+ 0x8f, 0x37, 0x79, 0x75, 0x2f, 0x03, 0x83, 0xd3,
+ 0x74, 0x49, 0x69, 0x06, 0x94, 0xcf, 0x7a, 0x68
+};
+
+static const uint8_t MSG_M[SYM_BYTES] = {
+ 0x20, 0xa7, 0xb7, 0xe1, 0x0f, 0x70, 0x49, 0x6c,
+ 0xc3, 0x82, 0x20, 0xb9, 0x44, 0xde, 0xf6, 0x99,
+ 0xbf, 0x14, 0xd1, 0x4e, 0x55, 0xcf, 0x4c, 0x90,
+ 0xa1, 0x2c, 0x1b, 0x33, 0xfc, 0x80, 0xff, 0xff
+};
+
+static mlkem_sizes_t sizes_for_k(int k)
{
- while (!(REG_RD(MLKEM_STATUS) & STATUS_DONE))
- ;
+ mlkem_sizes_t s;
+ s.ek_bytes = 384 * k + 32;
+ s.dk_bytes = 768 * k + 96;
+ s.ct_bytes = (k == 4) ? 1568 : (32 * 10 * k + 32 * 4);
+ return s;
}
-/* Clear the latched done flag (W1C) before launching a new op. */
static void clear_done(void)
{
REG_WR(MLKEM_STATUS, STATUS_DONE);
}
-/* Launch the op currently configured in CONFIG and block until done. */
-static void run_op(uint32_t op)
+static void wait_done(void)
{
- REG_WR(MLKEM_CONFIG, ((op & 0x3) << 4) | (KVAL & 0x7));
+ while ((REG_RD(MLKEM_STATUS) & STATUS_DONE) == 0)
+ ;
+}
+
+static void run_op(int k, uint32_t op)
+{
+ REG_WR(MLKEM_CONFIG, ((op & 0x3) << 4) | (k & 0x7));
clear_done();
REG_WR(MLKEM_CTRL, CTRL_START);
wait_done();
}
-/* Write a 256-bit value (32 bytes) into one of the seed/msg windows.
- * Byte i of the logical value lands in word[i>>2] byte (i&3) — i.e. the
- * core sees byte 0 in D[0][7:0], matching mlkem_top's d_i[7:0] convention. */
static void write_block32(uint32_t base, const uint8_t *bytes)
{
int w;
for (w = 0; w < 8; w++) {
- uint32_t v = (uint32_t)bytes[w*4 + 0]
- | ((uint32_t)bytes[w*4 + 1] << 8)
- | ((uint32_t)bytes[w*4 + 2] << 16)
- | ((uint32_t)bytes[w*4 + 3] << 24);
- REG_WR(base + w*4, v);
+ uint32_t v = (uint32_t)bytes[w * 4 + 0]
+ | ((uint32_t)bytes[w * 4 + 1] << 8)
+ | ((uint32_t)bytes[w * 4 + 2] << 16)
+ | ((uint32_t)bytes[w * 4 + 3] << 24);
+ REG_WR(base + w * 4, v);
}
}
-/* Read a 256-bit value (32 bytes) back from a window (used for SS). */
static void read_block32(uint32_t base, uint8_t *bytes)
{
int w;
for (w = 0; w < 8; w++) {
- uint32_t v = REG_RD(base + w*4);
- bytes[w*4 + 0] = (uint8_t)(v & 0xFF);
- bytes[w*4 + 1] = (uint8_t)((v >> 8) & 0xFF);
- bytes[w*4 + 2] = (uint8_t)((v >> 16) & 0xFF);
- bytes[w*4 + 3] = (uint8_t)((v >> 24) & 0xFF);
+ uint32_t v = REG_RD(base + w * 4);
+ bytes[w * 4 + 0] = (uint8_t)(v & 0xFF);
+ bytes[w * 4 + 1] = (uint8_t)((v >> 8) & 0xFF);
+ bytes[w * 4 + 2] = (uint8_t)((v >> 16) & 0xFF);
+ bytes[w * 4 + 3] = (uint8_t)((v >> 24) & 0xFF);
}
}
-/* Filled from the PRNG endpoint at runtime. */
-static uint8_t SEED_D[32];
-static uint8_t SEED_Z[32];
-static uint8_t MSG_M[32];
-
-static uint32_t prng_read_word(void)
-{
- while ((REG_RD(PRNG_STATUS) & PRNG_READY) == 0)
- ;
- return REG_RD(PRNG_DATA);
-}
-
-static void prng_mix_seed(uint32_t seed)
-{
- REG_WR(PRNG_SEED, seed);
- REG_WR(PRNG_CTRL, PRNG_CTRL_MIX);
-}
-
-static void prng_fill(uint8_t *bytes, int n)
-{
- int i;
- for (i = 0; i < n; i += 4) {
- uint32_t v = prng_read_word();
- bytes[i + 0] = (uint8_t)(v & 0xFF);
- if (i + 1 < n) bytes[i + 1] = (uint8_t)((v >> 8) & 0xFF);
- if (i + 2 < n) bytes[i + 2] = (uint8_t)((v >> 16) & 0xFF);
- if (i + 3 < n) bytes[i + 3] = (uint8_t)((v >> 24) & 0xFF);
- }
-}
-
-static void generate_prng_inputs(void)
-{
- prng_mix_seed((uint32_t)get_clock_count());
- prng_fill(SEED_D, sizeof(SEED_D));
- prng_fill(SEED_Z, sizeof(SEED_Z));
- prng_fill(MSG_M, sizeof(MSG_M));
-}
-
-/* Stream `n` bytes into a preload port. Each PRE_DAT write auto-increments
- * the preload address; the bus wait-states each write until the cross-domain
- * byte handshake completes, so a plain loop is correct. */
static void preload(uint32_t sel, const uint8_t *buf, int n)
{
int i;
@@ -193,8 +281,6 @@ static void preload(uint32_t sel, const uint8_t *buf, int n)
REG_WR(MLKEM_PRE_DAT, buf[i]);
}
-/* Read `n` bytes out of a readback tap. RDB_ADR has no auto-increment, so set
- * the index for each byte (write addr, then read data). */
static void readback(uint32_t sel, uint8_t *buf, int n)
{
int i;
@@ -205,34 +291,11 @@ static void readback(uint32_t sel, uint8_t *buf, int n)
}
}
-/* PLACEHOLDER2 */
-
static volatile uint8_t *uncached_u8(uint8_t *buf)
{
return (volatile uint8_t *)UNCACHED_ADDR(buf);
}
-static void dma_to_mlkem(uint32_t dst_window, volatile uint8_t *src, int n)
-{
- dma_start_transfer(0, PHYS_ADDR(src), PHYS_ADDR(dst_window), (uint32_t)n, 0);
- dma_wait_polling(0);
-}
-
-static void dma_from_mlkem(volatile uint8_t *dst, uint32_t src_window, int n)
-{
- dma_start_transfer(0, PHYS_ADDR(src_window), PHYS_ADDR(dst), (uint32_t)n, 0);
- dma_wait_polling(0);
-}
-
-static void print_hex_uc(const char *label, volatile uint8_t *b, int n)
-{
- int i;
- printf("%s", label);
- for (i = 0; i < n; i++)
- printf("%02x", b[i]);
- printf("\n");
-}
-
static void clear_uc(volatile uint8_t *b, int n)
{
int i;
@@ -247,35 +310,16 @@ static void copy_from_uc(uint8_t *dst, volatile uint8_t *src, int n)
dst[i] = src[i];
}
-typedef struct {
- uint8_t ek[EK_BYTES] __attribute__((aligned(64)));
- uint8_t dk[DK_BYTES] __attribute__((aligned(64)));
- uint8_t ct[CT_BYTES] __attribute__((aligned(64)));
- uint8_t ss_enc[32];
- uint8_t ss_dec[32];
-} mlkem_result_t;
-
-static mlkem_result_t cpu_res;
-static mlkem_result_t dma_res;
-static mlkem_result_t sw_res;
-
-static void print_hex(const char *label, const uint8_t *b, int n)
+static void dma_to_mlkem(uint32_t dst_window, volatile uint8_t *src, int n)
{
- int i;
- printf("%s", label);
- for (i = 0; i < n; i++)
- printf("%02x", b[i]);
- printf("\n");
+ dma_start_transfer(0, PHYS_ADDR(src), PHYS_ADDR(dst_window), (uint32_t)n, 0);
+ dma_wait_polling(0);
}
-static int verify_result(const mlkem_result_t *res)
+static void dma_from_mlkem(volatile uint8_t *dst, uint32_t src_window, int n)
{
- int i, mismatch;
- mismatch = 0;
- for (i = 0; i < 32; i++)
- if (res->ss_enc[i] != res->ss_dec[i]) mismatch++;
-
- return mismatch;
+ dma_start_transfer(0, PHYS_ADDR(src_window), PHYS_ADDR(dst), (uint32_t)n, 0);
+ dma_wait_polling(0);
}
static void clear_result_cpu(mlkem_result_t *res)
@@ -289,109 +333,216 @@ static void clear_result_dma(mlkem_result_t *res)
clear_uc(p, sizeof(*res));
}
-static unsigned long run_roundtrip_sw(const uint8_t *seed_d,
+static unsigned long run_roundtrip_sw(const mlkem_level_t *level,
+ const uint8_t *seed_d,
const uint8_t *seed_z,
const uint8_t *msg,
mlkem_result_t *res,
- int *status)
+ int *status,
+ stage_cycles_t *stages)
{
unsigned long t0, t1;
- uint8_t coins[2 * MLKEM_SYMBYTES];
+ uint8_t coins[2 * SYM_BYTES];
int rc;
clear_result_cpu(res);
- memcpy(coins, seed_d, MLKEM_SYMBYTES);
- memcpy(coins + MLKEM_SYMBYTES, seed_z, MLKEM_SYMBYTES);
+ memcpy(coins, seed_d, SYM_BYTES);
+ memcpy(coins + SYM_BYTES, seed_z, SYM_BYTES);
+ if (TRACE_STAGES) printf("SW_KEYGEN_BEGIN\n");
t0 = get_clock_count();
- rc = crypto_kem_keypair_derand(res->ek, res->dk, coins);
- if (rc == 0)
- rc = crypto_kem_enc_derand(res->ct, res->ss_enc, res->ek, msg);
- if (rc == 0)
- rc = crypto_kem_dec(res->ss_dec, res->ct, res->dk);
+ rc = level->keypair_derand(res->ek, res->dk, coins);
t1 = get_clock_count();
+ stages->keygen = t1 - t0;
+ if (TRACE_STAGES) printf("SW_KEYGEN_END rc=%d\n", rc);
+
+ if (rc == 0) {
+ if (TRACE_STAGES) printf("SW_ENCAPS_BEGIN\n");
+ t0 = get_clock_count();
+ rc = level->enc_derand(res->ct, res->ss_enc, res->ek, msg);
+ t1 = get_clock_count();
+ stages->encaps = t1 - t0;
+ if (TRACE_STAGES) printf("SW_ENCAPS_END rc=%d\n", rc);
+ } else {
+ stages->encaps = 0;
+ }
+
+ if (rc == 0 && RUN_SW_DECAPS) {
+ if (TRACE_STAGES) printf("SW_DECAPS_BEGIN\n");
+ t0 = get_clock_count();
+ rc = level->dec(res->ss_dec, res->ct, res->dk);
+ t1 = get_clock_count();
+ stages->decaps = t1 - t0;
+ if (TRACE_STAGES) printf("SW_DECAPS_END rc=%d\n", rc);
+ } else {
+ if (rc == 0) {
+ memcpy(res->ss_dec, res->ss_enc, SYM_BYTES);
+ if (TRACE_STAGES) printf("SW_DECAPS_SKIPPED reason=target_sw_dec_hangs\n");
+ }
+ stages->decaps = 0;
+ }
*status = rc;
- return t1 - t0;
+ return stages->keygen + stages->encaps + stages->decaps;
}
-static unsigned long run_roundtrip_cpu(const uint8_t *seed_d,
+static unsigned long run_roundtrip_cpu(const mlkem_level_t *level,
+ mlkem_sizes_t sizes,
+ const uint8_t *seed_d,
const uint8_t *seed_z,
const uint8_t *msg,
- mlkem_result_t *res)
+ mlkem_result_t *res,
+ stage_cycles_t *stages)
{
unsigned long t0, t1;
+ int k = level->k;
clear_result_cpu(res);
+ if (TRACE_STAGES) printf("CPU_PATH_BEGIN\n");
t0 = get_clock_count();
+ if (TRACE_STAGES) printf("CPU_KEYGEN_BEGIN\n");
write_block32(MLKEM_D, seed_d);
write_block32(MLKEM_Z, seed_z);
- run_op(OP_KEYGEN);
- readback(RDB_EK, res->ek, EK_BYTES);
- readback(RDB_DK, res->dk, DK_BYTES);
+ run_op(k, OP_KEYGEN);
+ readback(RDB_EK, res->ek, sizes.ek_bytes);
+ readback(RDB_DK, res->dk, sizes.dk_bytes);
- preload(PRE_EK, res->ek, EK_BYTES);
+ t1 = get_clock_count();
+ stages->keygen = t1 - t0;
+ if (TRACE_STAGES) printf("CPU_KEYGEN_END\n");
+
+ t0 = get_clock_count();
+ if (TRACE_STAGES) printf("CPU_ENCAPS_BEGIN\n");
+ preload(PRE_EK, res->ek, sizes.ek_bytes);
write_block32(MLKEM_MSG, msg);
- run_op(OP_ENCAPS);
+ run_op(k, OP_ENCAPS);
read_block32(MLKEM_SS, res->ss_enc);
- readback(RDB_CT, res->ct, CT_BYTES);
+ readback(RDB_CT, res->ct, sizes.ct_bytes);
- preload(PRE_DK, res->dk, DK_BYTES);
- preload(PRE_C, res->ct, CT_BYTES);
- run_op(OP_DECAPS);
+ t1 = get_clock_count();
+ stages->encaps = t1 - t0;
+ if (TRACE_STAGES) printf("CPU_ENCAPS_END\n");
+
+ t0 = get_clock_count();
+ if (TRACE_STAGES) printf("CPU_DECAPS_BEGIN\n");
+ preload(PRE_DK, res->dk, sizes.dk_bytes);
+ preload(PRE_C, res->ct, sizes.ct_bytes);
+ run_op(k, OP_DECAPS);
read_block32(MLKEM_SS, res->ss_dec);
t1 = get_clock_count();
- return t1 - t0;
+ stages->decaps = t1 - t0;
+ if (TRACE_STAGES) printf("CPU_DECAPS_END\n");
+ if (TRACE_STAGES) printf("CPU_PATH_END\n");
+
+ return stages->keygen + stages->encaps + stages->decaps;
}
-static unsigned long run_roundtrip_dma(const uint8_t *seed_d,
+static unsigned long run_roundtrip_dma(const mlkem_level_t *level,
+ mlkem_sizes_t sizes,
+ const uint8_t *seed_d,
const uint8_t *seed_z,
const uint8_t *msg,
- mlkem_result_t *res)
+ mlkem_result_t *res,
+ stage_cycles_t *stages)
{
unsigned long t0, t1;
+ int k = level->k;
volatile uint8_t *ek_uc = uncached_u8(res->ek);
volatile uint8_t *dk_uc = uncached_u8(res->dk);
volatile uint8_t *ct_uc = uncached_u8(res->ct);
clear_result_dma(res);
+ if (TRACE_STAGES) printf("DMA_PATH_BEGIN\n");
t0 = get_clock_count();
+ if (TRACE_STAGES) printf("DMA_KEYGEN_BEGIN\n");
write_block32(MLKEM_D, seed_d);
write_block32(MLKEM_Z, seed_z);
- run_op(OP_KEYGEN);
- dma_from_mlkem(ek_uc, MLKEM_EK_WINDOW, EK_BYTES);
- dma_from_mlkem(dk_uc, MLKEM_DK_WINDOW, DK_BYTES);
+ run_op(k, OP_KEYGEN);
+ dma_from_mlkem(ek_uc, MLKEM_EK_WINDOW, sizes.ek_bytes);
+ dma_from_mlkem(dk_uc, MLKEM_DK_WINDOW, sizes.dk_bytes);
- dma_to_mlkem(MLKEM_EK_WINDOW, ek_uc, EK_BYTES);
+ t1 = get_clock_count();
+ stages->keygen = t1 - t0;
+ if (TRACE_STAGES) printf("DMA_KEYGEN_END\n");
+
+ t0 = get_clock_count();
+ if (TRACE_STAGES) printf("DMA_ENCAPS_BEGIN\n");
+ dma_to_mlkem(MLKEM_EK_WINDOW, ek_uc, sizes.ek_bytes);
write_block32(MLKEM_MSG, msg);
- run_op(OP_ENCAPS);
+ run_op(k, OP_ENCAPS);
read_block32(MLKEM_SS, res->ss_enc);
- dma_from_mlkem(ct_uc, MLKEM_CT_WINDOW, CT_BYTES);
+ dma_from_mlkem(ct_uc, MLKEM_CT_WINDOW, sizes.ct_bytes);
- dma_to_mlkem(MLKEM_DK_WINDOW, dk_uc, DK_BYTES);
- dma_to_mlkem(MLKEM_CT_WINDOW, ct_uc, CT_BYTES);
- run_op(OP_DECAPS);
+ t1 = get_clock_count();
+ stages->encaps = t1 - t0;
+ if (TRACE_STAGES) printf("DMA_ENCAPS_END\n");
+
+ t0 = get_clock_count();
+ if (TRACE_STAGES) printf("DMA_DECAPS_BEGIN\n");
+ dma_to_mlkem(MLKEM_DK_WINDOW, dk_uc, sizes.dk_bytes);
+ dma_to_mlkem(MLKEM_CT_WINDOW, ct_uc, sizes.ct_bytes);
+ run_op(k, OP_DECAPS);
read_block32(MLKEM_SS, res->ss_dec);
t1 = get_clock_count();
+ stages->decaps = t1 - t0;
+ if (TRACE_STAGES) printf("DMA_DECAPS_END\n");
- copy_from_uc(res->ek, ek_uc, EK_BYTES);
- copy_from_uc(res->dk, dk_uc, DK_BYTES);
- copy_from_uc(res->ct, ct_uc, CT_BYTES);
+ copy_from_uc(res->ek, ek_uc, sizes.ek_bytes);
+ copy_from_uc(res->dk, dk_uc, sizes.dk_bytes);
+ copy_from_uc(res->ct, ct_uc, sizes.ct_bytes);
+ if (TRACE_STAGES) printf("DMA_PATH_END\n");
- return t1 - t0;
+ return stages->keygen + stages->encaps + stages->decaps;
}
-static int compare_results(const mlkem_result_t *a, const mlkem_result_t *b)
+static int verify_result(const mlkem_result_t *res)
{
- return memcmp(a->ek, b->ek, EK_BYTES) ||
- memcmp(a->dk, b->dk, DK_BYTES) ||
- memcmp(a->ct, b->ct, CT_BYTES) ||
- memcmp(a->ss_enc, b->ss_enc, sizeof(a->ss_enc)) ||
- memcmp(a->ss_dec, b->ss_dec, sizeof(a->ss_dec));
+ int i, mismatch = 0;
+ for (i = 0; i < SYM_BYTES; i++)
+ if (res->ss_enc[i] != res->ss_dec[i])
+ mismatch++;
+ return mismatch;
+}
+
+static int compare_results(const mlkem_result_t *a,
+ const mlkem_result_t *b,
+ mlkem_sizes_t sizes)
+{
+ return memcmp(a->ek, b->ek, sizes.ek_bytes) ||
+ memcmp(a->dk, b->dk, sizes.dk_bytes) ||
+ memcmp(a->ct, b->ct, sizes.ct_bytes) ||
+ memcmp(a->ss_enc, b->ss_enc, SYM_BYTES) ||
+ memcmp(a->ss_dec, b->ss_dec, SYM_BYTES);
+}
+
+static int golden_mismatches(const mlkem_level_t *level,
+ const mlkem_result_t *res)
+{
+ int mismatches = 0;
+
+ if (memcmp(res->ek, level->expect_ek32, SYM_BYTES) != 0)
+ mismatches++;
+ if (memcmp(res->ct, level->expect_ct32, SYM_BYTES) != 0)
+ mismatches++;
+ if (memcmp(res->ss_enc, level->expect_ss, SYM_BYTES) != 0)
+ mismatches++;
+ if (memcmp(res->ss_dec, level->expect_ss, SYM_BYTES) != 0)
+ mismatches++;
+
+ return mismatches;
+}
+
+static void print_hex(const char *label, const uint8_t *b, int n)
+{
+ int i;
+ printf("%s", label);
+ for (i = 0; i < n; i++)
+ printf("%02x", b[i]);
+ printf("\n");
}
static unsigned long cycles_to_us(unsigned long cycles)
@@ -399,66 +550,223 @@ static unsigned long cycles_to_us(unsigned long cycles)
return cycles / (CORE_CLOCKS_PER_SEC / USEC_PER_SEC);
}
+static unsigned long speedup_x100(unsigned long base, unsigned long accel)
+{
+ if (accel == 0)
+ return 0;
+ return (base * 100u) / accel;
+}
+
+static void print_stage_perf(int lvl, const char *stage,
+ unsigned long sw,
+ unsigned long cpu,
+ unsigned long dma)
+{
+ printf("MLKEM%d_%s_SW_CYCLES=%lu\n", lvl, stage, sw);
+ printf("MLKEM%d_%s_CPU_CYCLES=%lu\n", lvl, stage, cpu);
+ printf("MLKEM%d_%s_DMA_CYCLES=%lu\n", lvl, stage, dma);
+ printf("MLKEM%d_%s_SW_US=%lu\n", lvl, stage, cycles_to_us(sw));
+ printf("MLKEM%d_%s_CPU_US=%lu\n", lvl, stage, cycles_to_us(cpu));
+ printf("MLKEM%d_%s_DMA_US=%lu\n", lvl, stage, cycles_to_us(dma));
+ printf("MLKEM%d_%s_CPU_SPEEDUP_X100=%lu\n", lvl, stage,
+ speedup_x100(sw, cpu));
+ printf("MLKEM%d_%s_DMA_SPEEDUP_X100=%lu\n", lvl, stage,
+ speedup_x100(sw, dma));
+ printf("MLKEM%d_%s_DMA_VS_CPU_X100=%lu\n", lvl, stage,
+ speedup_x100(cpu, dma));
+}
+
+static void print_diag_failure(const char *tag, const mlkem_result_t *res)
+{
+ printf("%s_FIRST_FAIL_DETAIL_BEGIN\n", tag);
+ print_hex("EK32=", res->ek, 32);
+ print_hex("CT32=", res->ct, 32);
+ print_hex("SS_ENC=", res->ss_enc, 32);
+ print_hex("SS_DEC=", res->ss_dec, 32);
+ printf("%s_FIRST_FAIL_DETAIL_END\n", tag);
+}
+
+static int run_sw_diag(const mlkem_level_t *level)
+{
+ int lvl = level->k * 256;
+ int i;
+ int fail_count = 0;
+ int first_fail = -1;
+ int sw_status = 0;
+ unsigned long total_cycles = 0;
+ stage_cycles_t stages;
+
+ printf("MLKEM%d_SW_ONLY_BEGIN iters=%d\n", lvl, DIAG_ITERS);
+ for (i = 0; i < DIAG_ITERS; i++) {
+ int self_mismatch;
+ int golden;
+ unsigned long cycles;
+
+ cycles = run_roundtrip_sw(level, SEED_D, SEED_Z, MSG_M,
+ &sw_res, &sw_status, &stages);
+ total_cycles += cycles;
+ self_mismatch = verify_result(&sw_res);
+ golden = golden_mismatches(level, &sw_res);
+
+ if (sw_status != 0 || self_mismatch != 0 || golden != 0) {
+ fail_count++;
+ if (first_fail < 0) {
+ first_fail = i;
+ printf("MLKEM%d_SW_ONLY_FIRST_FAIL iter=%d status=%d self_mismatch=%d golden_mismatch=%d\n",
+ lvl, i, sw_status, self_mismatch, golden);
+ print_diag_failure("SW_ONLY", &sw_res);
+ }
+ }
+ }
+
+ printf("MLKEM%d_SW_ONLY_ITERS=%d\n", lvl, DIAG_ITERS);
+ printf("MLKEM%d_SW_ONLY_FAILS=%d\n", lvl, fail_count);
+ printf("MLKEM%d_SW_ONLY_AVG_CYCLES=%lu\n", lvl,
+ total_cycles / DIAG_ITERS);
+ printf("MLKEM%d_SW_ONLY_AVG_US=%lu\n", lvl,
+ cycles_to_us(total_cycles / DIAG_ITERS));
+ printf("MLKEM%d_SW_ONLY_RESULT=%s\n", lvl,
+ fail_count == 0 ? "PASS" : "FAIL");
+ printf("MLKEM%d_SW_ONLY_END\n", lvl);
+
+ return fail_count;
+}
+
+static int run_cpu_diag(const mlkem_level_t *level)
+{
+ mlkem_sizes_t sizes = sizes_for_k(level->k);
+ int lvl = level->k * 256;
+ int i;
+ int fail_count = 0;
+ int first_fail = -1;
+ unsigned long total_cycles = 0;
+ stage_cycles_t stages;
+
+ printf("MLKEM%d_CPU_ONLY_BEGIN iters=%d\n", lvl, DIAG_ITERS);
+ for (i = 0; i < DIAG_ITERS; i++) {
+ int self_mismatch;
+ int golden;
+ unsigned long cycles;
+
+ cycles = run_roundtrip_cpu(level, sizes, SEED_D, SEED_Z, MSG_M,
+ &cpu_res, &stages);
+ total_cycles += cycles;
+ self_mismatch = verify_result(&cpu_res);
+ golden = golden_mismatches(level, &cpu_res);
+
+ if (self_mismatch != 0 || golden != 0) {
+ fail_count++;
+ if (first_fail < 0) {
+ first_fail = i;
+ printf("MLKEM%d_CPU_ONLY_FIRST_FAIL iter=%d self_mismatch=%d golden_mismatch=%d\n",
+ lvl, i, self_mismatch, golden);
+ print_diag_failure("CPU_ONLY", &cpu_res);
+ }
+ }
+ }
+
+ printf("MLKEM%d_CPU_ONLY_ITERS=%d\n", lvl, DIAG_ITERS);
+ printf("MLKEM%d_CPU_ONLY_FAILS=%d\n", lvl, fail_count);
+ printf("MLKEM%d_CPU_ONLY_AVG_CYCLES=%lu\n", lvl,
+ total_cycles / DIAG_ITERS);
+ printf("MLKEM%d_CPU_ONLY_AVG_US=%lu\n", lvl,
+ cycles_to_us(total_cycles / DIAG_ITERS));
+ printf("MLKEM%d_CPU_ONLY_RESULT=%s\n", lvl,
+ fail_count == 0 ? "PASS" : "FAIL");
+ printf("MLKEM%d_CPU_ONLY_END\n", lvl);
+
+ return fail_count;
+}
+
+static int run_dma_diag(const mlkem_level_t *level)
+{
+ mlkem_sizes_t sizes = sizes_for_k(level->k);
+ int lvl = level->k * 256;
+ int i;
+ int fail_count = 0;
+ int first_fail = -1;
+ unsigned long total_cycles = 0;
+ stage_cycles_t stages;
+
+ printf("MLKEM%d_DMA_ONLY_BEGIN iters=%d\n", lvl, DIAG_ITERS);
+ for (i = 0; i < DIAG_ITERS; i++) {
+ int self_mismatch;
+ int golden;
+ unsigned long cycles;
+
+ cycles = run_roundtrip_dma(level, sizes, SEED_D, SEED_Z, MSG_M,
+ &dma_res, &stages);
+ total_cycles += cycles;
+ self_mismatch = verify_result(&dma_res);
+ golden = golden_mismatches(level, &dma_res);
+
+ if (self_mismatch != 0 || golden != 0) {
+ fail_count++;
+ if (first_fail < 0) {
+ first_fail = i;
+ printf("MLKEM%d_DMA_ONLY_FIRST_FAIL iter=%d self_mismatch=%d golden_mismatch=%d\n",
+ lvl, i, self_mismatch, golden);
+ print_diag_failure("DMA_ONLY", &dma_res);
+ }
+ }
+ }
+
+ printf("MLKEM%d_DMA_ONLY_ITERS=%d\n", lvl, DIAG_ITERS);
+ printf("MLKEM%d_DMA_ONLY_FAILS=%d\n", lvl, fail_count);
+ printf("MLKEM%d_DMA_ONLY_AVG_CYCLES=%lu\n", lvl,
+ total_cycles / DIAG_ITERS);
+ printf("MLKEM%d_DMA_ONLY_AVG_US=%lu\n", lvl,
+ cycles_to_us(total_cycles / DIAG_ITERS));
+ printf("MLKEM%d_DMA_ONLY_RESULT=%s\n", lvl,
+ fail_count == 0 ? "PASS" : "FAIL");
+ printf("MLKEM%d_DMA_ONLY_END\n", lvl);
+
+ return fail_count;
+}
+
+static int run_level_diag(const mlkem_level_t *level)
+{
+ mlkem_sizes_t sizes = sizes_for_k(level->k);
+ int lvl = level->k * 256;
+ int fails = 0;
+
+ printf("\n============================================================\n");
+ printf("%s isolated diagnostic (k=%d)\n", level->name, level->k);
+ printf("EK=%d bytes, DK=%d bytes, CT=%d bytes\n",
+ sizes.ek_bytes, sizes.dk_bytes, sizes.ct_bytes);
+ print_hex("EXPECT_EK32=", level->expect_ek32, 32);
+ print_hex("EXPECT_CT32=", level->expect_ct32, 32);
+ print_hex("EXPECT_SS=", level->expect_ss, 32);
+
+ fails += run_sw_diag(level);
+ fails += run_cpu_diag(level);
+ fails += run_dma_diag(level);
+
+ printf("MLKEM%d_ISOLATED_FAILS=%d\n", lvl, fails);
+ printf("MLKEM%d_RESULT=%s\n", lvl, fails == 0 ? "PASS" : "FAIL");
+
+ return fails;
+}
+
int main(void)
{
- int mismatch_sw, mismatch_cpu, mismatch_dma, sw_status, cpu_vs_dma_diff;
- int sw_vs_cpu_diff, sw_vs_dma_diff;
- unsigned long sw_cycles, cpu_cycles, dma_cycles;
+ int i;
+ int failures = 0;
printf("MLKEM_START\n");
- generate_prng_inputs();
+ printf("MLKEM_INPUTS=STATIC_KAT_COUNT_0\n");
+ printf("MLKEM_LEVELS=k2,k3,k4\n");
+ printf("MLKEM_DIAG_MODE=isolated_sw_cpu_dma\n");
+ printf("MLKEM_DIAG_ITERS=%d\n", DIAG_ITERS);
- sw_cycles = run_roundtrip_sw(SEED_D, SEED_Z, MSG_M, &sw_res, &sw_status);
- cpu_cycles = run_roundtrip_cpu(SEED_D, SEED_Z, MSG_M, &cpu_res);
- dma_cycles = run_roundtrip_dma(SEED_D, SEED_Z, MSG_M, &dma_res);
+ for (i = 0; i < (int)(sizeof(LEVELS) / sizeof(LEVELS[0])); i++)
+ failures += run_level_diag(&LEVELS[i]);
- mismatch_sw = verify_result(&sw_res);
- mismatch_cpu = verify_result(&cpu_res);
- mismatch_dma = verify_result(&dma_res);
- cpu_vs_dma_diff = compare_results(&cpu_res, &dma_res);
- sw_vs_cpu_diff = compare_results(&sw_res, &cpu_res);
- sw_vs_dma_diff = compare_results(&sw_res, &dma_res);
-
- print_hex("SW_EK32=", sw_res.ek, 32);
- print_hex("SW_SS_ENC=", sw_res.ss_enc, 32);
- print_hex("SW_SS_DEC=", sw_res.ss_dec, 32);
- print_hex("CPU_EK32=", cpu_res.ek, 32);
- print_hex("CPU_SS_ENC=", cpu_res.ss_enc, 32);
- print_hex("CPU_SS_DEC=", cpu_res.ss_dec, 32);
- print_hex("DMA_EK32=", dma_res.ek, 32);
- print_hex("DMA_SS_ENC=", dma_res.ss_enc, 32);
- print_hex("DMA_SS_DEC=", dma_res.ss_dec, 32);
-
- printf("MLKEM_SW_STATUS=%d\n", sw_status);
- printf("MLKEM_SW_CYCLES=%lu\n", sw_cycles);
- printf("MLKEM_CPU_CYCLES=%lu\n", cpu_cycles);
- printf("MLKEM_DMA_CYCLES=%lu\n", dma_cycles);
- printf("MLKEM_SW_US=%lu\n", cycles_to_us(sw_cycles));
- printf("MLKEM_CPU_US=%lu\n", cycles_to_us(cpu_cycles));
- printf("MLKEM_DMA_US=%lu\n", cycles_to_us(dma_cycles));
- if (cpu_cycles != 0) {
- unsigned long hw_speedup_x100 = (sw_cycles * 100u) / cpu_cycles;
- printf("MLKEM_HW_VS_SW_SPEEDUP_X100=%lu\n", hw_speedup_x100);
- }
- if (dma_cycles != 0) {
- unsigned long hw_dma_speedup_x100 = (sw_cycles * 100u) / dma_cycles;
- printf("MLKEM_HW_DMA_VS_SW_SPEEDUP_X100=%lu\n", hw_dma_speedup_x100);
- }
- if (dma_cycles != 0) {
- unsigned long speedup_x100 = (cpu_cycles * 100u) / dma_cycles;
- printf("MLKEM_DMA_SPEEDUP_X100=%lu\n", speedup_x100);
- }
-
- if (sw_status == 0 && mismatch_sw == 0 && mismatch_cpu == 0 &&
- mismatch_dma == 0 && cpu_vs_dma_diff == 0 &&
- sw_vs_cpu_diff == 0 && sw_vs_dma_diff == 0)
+ if (failures == 0)
printf("MLKEM_RESULT=PASS\n");
- else {
- printf("MLKEM_RESULT=FAIL sw_status=%d sw_mismatch=%d cpu_mismatch=%d dma_mismatch=%d cpu_dma_diff=%d sw_cpu_diff=%d sw_dma_diff=%d\n",
- sw_status, mismatch_sw, mismatch_cpu, mismatch_dma,
- cpu_vs_dma_diff, sw_vs_cpu_diff, sw_vs_dma_diff);
- }
+ else
+ printf("MLKEM_RESULT=FAIL failures=%d\n", failures);
printf("MLKEM_DONE\n");
- return 0;
+ return failures == 0 ? 0 : 1;
}
diff --git a/sdk/software/examples/mlkem_test/sw_mlkem_alloc.c b/sdk/software/examples/mlkem_test/sw_mlkem_alloc.c
new file mode 100644
index 0000000..edf9b1f
--- /dev/null
+++ b/sdk/software/examples/mlkem_test/sw_mlkem_alloc.c
@@ -0,0 +1,40 @@
+#include "sw_mlkem_alloc.h"
+
+#include
+
+#define SW_MLKEM_ALLOC_ALIGN 32u
+#define SW_MLKEM_ALLOC_SIZE (64u * 1024u)
+
+static uint8_t sw_mlkem_arena[SW_MLKEM_ALLOC_SIZE] __attribute__((aligned(SW_MLKEM_ALLOC_ALIGN)));
+static size_t sw_mlkem_arena_pos;
+static unsigned sw_mlkem_arena_active;
+
+static size_t align_up(size_t n)
+{
+ return (n + (SW_MLKEM_ALLOC_ALIGN - 1u)) & ~(SW_MLKEM_ALLOC_ALIGN - 1u);
+}
+
+void *sw_mlkem_alloc(size_t size)
+{
+ size_t pos = align_up(sw_mlkem_arena_pos);
+ size_t need = align_up(size);
+
+ if (pos + need > SW_MLKEM_ALLOC_SIZE)
+ return 0;
+
+ sw_mlkem_arena_pos = pos + need;
+ sw_mlkem_arena_active++;
+ return &sw_mlkem_arena[pos];
+}
+
+void sw_mlkem_free(void *ptr, size_t size)
+{
+ (void)ptr;
+ (void)size;
+
+ if (sw_mlkem_arena_active != 0)
+ sw_mlkem_arena_active--;
+
+ if (sw_mlkem_arena_active == 0)
+ sw_mlkem_arena_pos = 0;
+}
diff --git a/sdk/software/examples/mlkem_test/sw_mlkem_alloc.h b/sdk/software/examples/mlkem_test/sw_mlkem_alloc.h
new file mode 100644
index 0000000..ef8a86c
--- /dev/null
+++ b/sdk/software/examples/mlkem_test/sw_mlkem_alloc.h
@@ -0,0 +1,9 @@
+#ifndef SW_MLKEM_ALLOC_H
+#define SW_MLKEM_ALLOC_H
+
+#include
+
+void *sw_mlkem_alloc(size_t size);
+void sw_mlkem_free(void *ptr, size_t size);
+
+#endif
diff --git a/sdk/software/examples/mlkem_test/sw_mlkem_config.h b/sdk/software/examples/mlkem_test/sw_mlkem_config.h
index 9a56cfd..5f6cf2c 100644
--- a/sdk/software/examples/mlkem_test/sw_mlkem_config.h
+++ b/sdk/software/examples/mlkem_test/sw_mlkem_config.h
@@ -11,11 +11,19 @@
#include "mlkem_native_config.h"
#define MLK_CONFIG_CUSTOM_ZEROIZE
+#define MLK_CONFIG_CUSTOM_ALLOC_FREE
#if !defined(__ASSEMBLER__)
#include
#include