test: add board FFT validation and VCD capture

2026-07-15 04:13:15 +08:00
parent 2359e7976c
commit d45646342b
10 changed files with 198 additions and 23 deletions
--- a/doc/reports/system/.gitignore
+++ b/doc/reports/system/.gitignore
@@ -7,3 +7,4 @@ system_design_report.synctex.gz
 system_design_report.toc
 system_design_report.xdv
 tools/__pycache__/
+evidence/**/*.vcd
--- a/doc/reports/system/REPORT_RERUN_CHECKLIST.md
+++ b/doc/reports/system/REPORT_RERUN_CHECKLIST.md
@@ -42,10 +42,35 @@ cd doc/reports/system
 ./tools/run_report_tests.sh soc
 ```

-软件集合固定为 `asm`、`fft`、`fft_dma`、`mlkem_test` 和 `mlkem_hello`。`asm`
-需要官方 ExtRAM 测试数据和专用 matmul 流程，因此 `soc` 只编译它，不用通用
-`tb_top` 仿真；其余四项编译后依次运行 SoC 仿真。每次构建都会覆盖
-`sdk/axi_ram.mif`，因此不要并行执行这些用例。构建日志位于 `evidence/build/`。
+需要保存 VCD 时使用可选开关。默认只记录 `tb_top` 的时钟、复位、UART 和提交 PC，
+文件写入 `evidence/soc/<用例>.vcd`。记录内部模块时用 `VCD_SCOPE` 指定层次；长时间
+软件测试若递归记录整个 SoC，可能产生数 GB 文件：
+
+```bash
+SAVE_VCD=1 \
+VCD_SCOPE='/tb_top/u_soc_top/u_fft_mlkem_axi_slave/u_mlkem_axi_slave/*' \
+./tools/run_report_tests.sh soc
+```
+
+启用 VCD 后 elaboration 使用 `debug_level=typical`，并从 0 ns 开始记录，因此包含
+上电复位和复位释放过程。只需要短窗口诊断时可额外设置 `SIM_RUN_TIME=20us`；该选项
+会在软件完成前退出，不能作为功能回归通过证据。VCD 属于大体积临时证据，默认被
+Git 忽略。
+
+软件集合固定为 `asm`、`fft`、`fft_dma`、`mlkem_test` 和 `mlkem_hello`。默认 `soc`
+会编译全部五项，但只对 `mlkem_test` 和 `mlkem_hello` 运行通用 `tb_top` 仿真。
+`asm` 需要官方 ExtRAM 数据和专用 matmul 流程；`fft` 和 `fft_dma` 包含耗时很长的
+软件 FFT 参考计算，因此默认只做编译检查。每次构建都会覆盖 `sdk/axi_ram.mif`，不要
+并行执行这些用例。构建日志位于 `evidence/build/`。
+
+需要更新 FFT 当前提交的功能或性能证据时，再单独执行长仿真：
+
+```bash
+./tools/run_report_tests.sh soc-fft
+```
+
+没有执行 `soc-fft` 且没有生成当前提交的板级日志时，报告中的 FFT 数值只能标记为
+历史基线，不能声明为当前提交的功能回归结果。
 脚本会设置 `CCACHE_DISABLE=1`，避免 Vivado 2019.2 在编译 XSIM 自动生成的 C 文件时
 因 ccache 目录不可写而只报告不透明的 `XSIM 43-3409`。
 同时会将 `/lib64` 加入 XSIM 子进程库路径，避免旧版 Vivado clang 混用不匹配的
@@ -56,11 +81,51 @@ cd doc/reports/system
 | 日志 | 必须出现 | 不得出现 |
 |---|---|---|
 | `evidence/build/{asm,fft,fft_dma,mlkem_test,mlkem_hello}.log` | 编译、链接和 objcopy 成功 | `Error` |
-| `evidence/soc/fft.log` | Hardware/Software FFT Time；更新后的程序应输出 `Max Abs Error` | 仿真超时、X/Z 扩散 |
-| `evidence/soc/fft_dma.log` | Hardware/Software FFT Time、`Max Abs Error` | DMA 死锁、仿真超时 |
 | `evidence/soc/mlkem_test.log` | `MLKEM_RESULT=PASS`、`MLKEM_DONE` | 任一 `FAILS` 非 0 |
 | `evidence/soc/mlkem_hello.log` | `MLKEM_HELLO_RESULT=PASS`、`MLKEM_HELLO_DONE` | `RESULT=FAIL` |

+若执行了可选的 `soc-fft`，再检查：
+
+| 日志 | 必须出现 | 不得出现 |
+|---|---|---|
+| `evidence/soc/fft.log` | Hardware/Software FFT Time；更新后的程序应输出 `Max Abs Error` | 仿真超时、X/Z 扩散 |
+| `evidence/soc/fft_dma.log` | Hardware/Software FFT Time、`Max Abs Error` | DMA 死锁、仿真超时 |
+
+### FFT 板级验证（推荐）
+
+FFT 的软件浮点参考在 RTL 仿真中耗时很长，默认使用板级测试作为当前提交的功能与性能
+证据。使用同一提交生成的 bitstream，分别下载 `fft` 和 `fft_dma` 的程序镜像，在
+115200 baud 下保存完整 UART 输出：
+
+```text
+doc/reports/system/evidence/board/fft.log
+doc/reports/system/evidence/board/fft_dma.log
+```
+
+`fft.log` 必须出现：
+
+```text
+FFT_MAX_ABS_ERROR_RE=<不大于8>
+FFT_MAX_ABS_ERROR_IM=<不大于8>
+FFT_ERROR_LIMIT=8
+FFT_RESULT=PASS
+FFT_DONE
+```
+
+`fft_dma.log` 必须出现：
+
+```text
+FFT_DMA_MAX_ABS_ERROR_RE=<不大于8>
+FFT_DMA_MAX_ABS_ERROR_IM=<不大于8>
+FFT_DMA_ERROR_LIMIT=8
+FFT_DMA_RESULT=PASS
+FFT_DMA_DONE
+```
+
+两项日志还必须保留 Hardware/Software FFT Time 和 Speedup Ratio。每项至少冷启动运行
+3 次，确认结果和最大误差稳定；DMA 用例还要确认没有轮询死锁。板级日志存在时，
+`update_report_metrics.py` 会优先于 XSIM 日志读取它们。
+
 ML-KEM 的正确计数是 3 个固定 KAT 向量、900 次执行，不是 360 个不同 KAT。若要在
 报告中声称覆盖隐式拒绝，必须另外增加“篡改 ciphertext 后共享密钥不等且流程按规范
 完成”的负向测试；当前脚本不提供该证据。
--- a/doc/reports/system/evidence/README.md
+++ b/doc/reports/system/evidence/README.md
@@ -5,8 +5,9 @@ Generated logs and Vivado reports used by `system_design_report.tex` live here:
 - `soc/`: UART output from SoC simulations
 - `build/`: software build logs for the five selected examples
 - `vivado/`: timing, utilization, CDC, DRC, route and power reports
- `board/`: official board evaluation output
+- `board/`: official matmul evaluation plus FFT/FFT-DMA board UART logs

 Run `../tools/run_report_tests.sh refresh` after replacing evidence. Large generated
-artifacts such as `.wdb`, `.jou`, `.str`, implementation checkpoints and bitstreams do
-not belong in this directory.
+artifacts such as `.vcd`, `.wdb`, `.jou`, `.str`, implementation checkpoints and
+bitstreams do not belong in Git. SoC VCD files may be generated here temporarily with
+`SAVE_VCD=1`; archive them separately when they are needed as report evidence.
--- a/doc/reports/system/system_design_report.pdf
+++ b/doc/reports/system/system_design_report.pdf
--- a/doc/reports/system/system_design_report.tex
+++ b/doc/reports/system/system_design_report.tex
@@ -271,8 +271,8 @@ DRC、DSP 与资源；板级测试确认真实 PLL、SRAM、串口和复位行
 \midrule
 Verilator lint & 无 Error；集成顶层可达性检查通过 & \path{evidence/lint.log} \\
 指定软件构建 & asm、fft、fft\_dma、mlkem\_test、mlkem\_hello 均成功链接 & \path{evidence/build/*.log} \\
-FFT CPU 路径 & 1024 点完成；记录端到端 ns 与逐点最大误差 & \path{evidence/soc/fft.log} \\
-FFT DMA 路径 & 四次搬运无死锁；最大误差和性能均输出 & \path{evidence/soc/fft_dma.log} \\
+FFT CPU 路径 & 1024 点完成；记录端到端 ns 与逐点最大误差 & \path{evidence/board/fft.log} \\
+FFT DMA 路径 & 四次搬运无死锁；最大误差和性能均输出 & \path{evidence/board/fft_dma.log} \\
 ML-KEM 三路径 & k=2/3/4、SW/CPU/DMA 各 100 次，FAILS=0 & \path{evidence/soc/mlkem_test.log} \\
 ML-KEM 示例 & 共享密钥一致且明文恢复，输出 RESULT=PASS & \path{evidence/soc/mlkem_hello.log} \\
 完整实现 & setup/hold WNS$\ge0$，unconstrained=0，DSP=0，DRC 无关键错误 & \path{evidence/vivado/*.rpt} \\
--- a/doc/reports/system/tools/run_report_tests.sh
+++ b/doc/reports/system/tools/run_report_tests.sh
@@ -50,11 +50,16 @@ Commands:
  prepare       Create evidence directories and Vivado project if absent
  lint          Run the project-aware Verilator lint check
  build         Build asm, fft, fft_dma, mlkem_test and mlkem_hello
-  soc           Build all five; simulate all except board-only asm
+  soc           Build all five; simulate mlkem_test and mlkem_hello
+  soc-fft       Explicitly run the long fft and fft_dma simulations
  implement     Run Vivado implementation and collect PPA/CDC/DRC reports
  refresh       Parse collected evidence, regenerate figures and build the PDF
  quick         Run lint and refresh only; does not run Vivado implementation

+Set SAVE_VCD=1 for SoC simulations to write evidence/soc/<example>.vcd.
+Set VCD_SCOPE to a Vivado hierarchy glob to limit waveform size.
+Set SIM_RUN_TIME only for bounded waveform diagnostics; regressions use 'all'.
+
 Board-only tests (matmul official 5000-group timing and reset behavior) are
 listed in REPORT_RERUN_CHECKLIST.md and are intentionally not automated here.
 EOF
@@ -95,21 +100,31 @@ build() {

 run_soc_case() {
  local example="$1"
+  local vcd_path="$EVIDENCE_DIR/soc/$example.vcd"
  require_vivado
  build_example "$example"
-  (cd "$ROOT/fpga" && "$VIVADO_BIN" -mode batch \
+  if [[ "${SAVE_VCD:-0}" != "0" ]]; then
+    echo "VCD output: $vcd_path (scope: ${VCD_SCOPE:-top-level control signals})"
+  fi
+  (cd "$ROOT/fpga" && SOC_VCD_PATH="$vcd_path" "$VIVADO_BIN" -mode batch \
    -source "$SCRIPT_DIR/run_soc_sim.tcl") 2>&1 | tee "$EVIDENCE_DIR/soc/$example.log"
 }

 soc() {
  prepare
  build_example asm
-  run_soc_case fft
-  run_soc_case fft_dma
+  build_example fft
+  build_example fft_dma
  run_soc_case mlkem_test
  run_soc_case mlkem_hello
 }

+soc_fft() {
+  prepare
+  run_soc_case fft
+  run_soc_case fft_dma
+}
+
 implement() {
  prepare
  require_vivado
@@ -130,6 +145,7 @@ case "${1:-}" in
  lint) lint ;;
  build) build ;;
  soc) soc ;;
+  soc-fft) soc_fft ;;
  implement) implement ;;
  refresh) refresh ;;
  quick) lint; refresh ;;
--- a/doc/reports/system/tools/run_soc_sim.tcl
+++ b/doc/reports/system/tools/run_soc_sim.tcl
@@ -5,6 +5,17 @@ set script_dir [file dirname [file normalize [info script]]]
 set repo_root [file normalize [file join $script_dir ../../../..]]
 set xpr [file join $repo_root fpga project Loongson_Soc.xpr]

+set save_vcd 0
+if {[info exists ::env(SAVE_VCD)]} {
+    set value [string tolower [string trim $::env(SAVE_VCD)]]
+    set save_vcd [expr {$value ni {"" "0" "false" "no" "off"}}]
+}
+
+set sim_run_time "all"
+if {[info exists ::env(SIM_RUN_TIME)] && $::env(SIM_RUN_TIME) ne ""} {
+    set sim_run_time $::env(SIM_RUN_TIME)
+}
+
 # xelab inherits this when compiling generated C files with the host gcc.
 set ::env(CCACHE_DISABLE) 1

@@ -31,11 +42,58 @@ set simset [get_filesets sim_1]
 set_property top tb_top $simset
 set_property incremental false $simset
 set_property xsim.elaborate.mt_level off $simset
-set_property xsim.elaborate.debug_level off $simset
+set_property xsim.elaborate.debug_level [expr {$save_vcd ? "typical" : "off"}] $simset
 set_property xsim.simulate.log_all_signals false $simset
+if {$save_vcd} {
+    # launch_simulation normally runs 1000 ns before returning. Start at 0 ns
+    # so the VCD includes power-on reset and reset release.
+    set_property xsim.simulate.runtime 0ns $simset
+}
 update_compile_order -fileset sim_1
 launch_simulation
-run all
+
+if {$save_vcd} {
+    if {![info exists ::env(SOC_VCD_PATH)] || $::env(SOC_VCD_PATH) eq ""} {
+        puts stderr "ERROR: SAVE_VCD requires SOC_VCD_PATH"
+        close_sim
+        close_project
+        exit 4
+    }
+
+    set vcd_path [file normalize $::env(SOC_VCD_PATH)]
+    set vcd_patterns [list \
+        /tb_top/clk \
+        /tb_top/reset \
+        /tb_top/UART_TX \
+        /tb_top/uart_wen \
+        /tb_top/uart_data \
+        /tb_top/debug_wb_pc]
+    if {[info exists ::env(VCD_SCOPE)] && $::env(VCD_SCOPE) ne ""} {
+        set vcd_patterns [list $::env(VCD_SCOPE)]
+        set vcd_objects [get_objects -quiet -r {*}$vcd_patterns]
+    } else {
+        # Keep the default trace small. Recursive collection is only enabled
+        # when the caller explicitly supplies VCD_SCOPE.
+        set vcd_objects [get_objects -quiet {*}$vcd_patterns]
+    }
+    if {[llength $vcd_objects] == 0} {
+        puts stderr "ERROR: VCD scope matched no simulation objects: $vcd_patterns"
+        close_sim
+        close_project
+        exit 5
+    }
+
+    file mkdir [file dirname $vcd_path]
+    puts "Recording VCD: $vcd_path (scope: $vcd_patterns)"
+    open_vcd $vcd_path
+    log_vcd $vcd_objects
+}
+
+puts "Running simulation for: $sim_run_time"
+run $sim_run_time
+if {$save_vcd} {
+    close_vcd
+}
 close_sim
 close_project
 exit 0
--- a/doc/reports/system/tools/update_report_metrics.py
+++ b/doc/reports/system/tools/update_report_metrics.py
@@ -144,6 +144,7 @@ def update_matmul(metrics: str, log: str) -> str:
 def main() -> None:
    metrics = METRICS.read_text(encoding="utf-8")
    vivado = REPORT_DIR / "evidence" / "vivado"
+    board = REPORT_DIR / "evidence" / "board"
    legacy = ROOT / "fpga" / "vivado_reports_nodsp"
    metrics = update_timing(metrics, read_first(vivado / "timing_summary_full.rpt", legacy / "timing_summary_full.rpt"))
    metrics = update_util(metrics, read_first(vivado / "utilization.rpt", legacy / "utilization.rpt"))
@@ -153,10 +154,10 @@ def main() -> None:
    metrics = update_mlkem(metrics, read_first(soc / "mlkem_test.log", ROOT / "sdk/software/examples/mlkem_test/result.txt"))
    metrics = update_fft(
        metrics,
-        read_first(soc / "fft.log", ROOT / "sdk/software/examples/fft/result.txt"),
-        read_first(soc / "fft_dma.log", ROOT / "sdk/software/examples/fft_dma/result.txt"),
+        read_first(board / "fft.log", soc / "fft.log", ROOT / "sdk/software/examples/fft/result.txt"),
+        read_first(board / "fft_dma.log", soc / "fft_dma.log", ROOT / "sdk/software/examples/fft_dma/result.txt"),
    )
-    metrics = update_matmul(metrics, read_first(REPORT_DIR / "evidence" / "board" / "matmul_official.log"))
+    metrics = update_matmul(metrics, read_first(board / "matmul_official.log"))

    try:
        revision = subprocess.check_output(