Add a Verilator simulation flow for the mesh - #98
Open
FrancescoConti wants to merge 4 commits into
Open
Conversation
Simulates the 4x4 mesh with Verilator alongside the existing QuestaSim flow. Independent of it: own targets, own build directory, no vsim-scripts/build-hw step. make verilate build the model make verilate-run test=<name> build + run make clean-verilate remove verilator/build Hierarchical Verilation ----------------------- Flattening the tile 16 times does not build in reasonable time or memory, so the tile is Verilated once into a library and instantiated 16 times, via a hier_block metacomment on a magia_tile_hier wrapper. Consequences: - magia_hier_params.v is empty but required. Verilator only substitutes a prebuilt library when --hierarchical-params-file is given; without it the tile is inlined and the result is a flat model that still builds, links the unused library, and looks fine. verilate-check-hierarchy inspects the built parent for exactly that. - A hierarchy block is opaque to dotted paths from the parent, so the testbench can no longer reach into a tile. magia_tile drives a packed observe_o struct (AXI write channel for prints, ID/EX/WB instructions) that the VIP reads instead. Verilator-only, behind `ifdef VERILATOR. - Parent and child must agree on where simulated time lives. Verilator defines VL_TIME_CONTEXT only as a side effect of --main and only for the parent, so this flow forces it on both; the mismatch segfaults in VlDelayScheduler at time 0. Waveforms --------- The model is always built with --trace-fst; dumping is opt-in per run with VERILATOR_FST=<file>, so a run that asks for nothing pays only a larger binary. There is no way to build without tracing: the non-tracing model diverges from the tracing one on tests that do real memory traffic, for reasons not yet understood, and nothing about that failure announces itself. target/verilator/src/magia_main.cpp owns dumping, replacing the main Verilator generates. A $dumpvars in the testbench runs at time zero, before the first eval() has constructed the hier_block children, so their trace callbacks are never registered and the dump silently contains nothing below i_magia_tile. The main evaluates once, then connects and opens the trace. It also constructs the model as "TOP": the parent looks its children up by scope name, and the empty name Verilator's own main passes matches nothing. Fixes found by this flow ------------------------ - The tile was never reset. Its flops run on a gated sys_clk whose enable is itself reset to 0, so no clock edge arrives while reset is asserted; and in a 2-state simulator rst_n powers up at 0, so clk_rst_gen driving it to 0 at time 0 is not a falling edge either. Neither event `always_ff @(posedge clk_i or negedge rst_ni)` needs ever occurred. Questa is unaffected because rst_n is X until time 0 and X -> 0 is an edge. Only flops with a non-zero reset value showed a symptom: obi_atop_resolver's credit counter came up at 0, held amo_available low, and deadlocked the first L1 access of any program. The Verilator VIP now holds test_mode during reset so the gate passes the clock. - NoC address maps ended at 32'h100000000, which does not fit 32 bits. - local_interconnect sized FILTER_WRITE_R_VALID from N_HWPE alone rather than the widest of N_HWPE/N_DMA/N_CORE. Layout ------ target/verilator/src/ model sources: main, hier_block control, wrapper, VIP verilator/scripts/ file-list filter and the two checkers verilator/verilator.mk the flow; verilator/build/ its output Limits ------ mesh_dv=1 and CV32E40P only. VERILATOR_THREADS>1 is experimental: 8 halved a test, 4 segfaults at time zero. Nothing bounds a hung run. Roughly 1 us of simulated time per wall-clock second.
clk_rst_gen drives rst_no to 1'b0 from an initial block. In a simulator whose
signals power up at 0 (e.g. Verilator) that is not a falling edge, so no
always_ff @(posedge clk_i or negedge rst_ni)
in the design is ever reset. A four-state simulator works thanks to
its X -> 0 transition at time 0.
The Verilator VIP worked around the symptom by holding test_mode high while
rst_n was low, which forces test_en_i on every tc_clk_gating in the tile and
un-gates its clocks during reset. Because the gate enable is a latch that can
only re-evaluate on a low clock phase, one posedge still escaped after reset had
already been released. This caused problems with the Spatz i-cache, and likely
other subsystems that did not work correctly in Verilator simulation (e.g.,
the long-running mesh_mm_test is currently looking fixed by this commit)
This commit generates clock and reset in the VIP instead, with the same waveform
clk_rst_gen produced:
- rst_n starts high, falls, is held for RST_CYCLES, then goes high for good,
so the asynchronous reset edge exists in any simulator;
- the clock is held low across the initial high phase and starts only once
reset is already asserted, so no flop is clocked while its state is unknown
(clocking X's there trips assertions in the FlooNoC chimneys at time 0);
- the two transitions never land in the same time step, so reset and clock
cannot race.
wait_for_reset() now waits for the falling edge first; rst_n is high at power-up
and a bare @(posedge rst_n) would return at time 0, before reset had happened.
Applied to both mesh testbenches, since this is a portability defect rather than
a Verilator one, and the test_mode workaround is dropped.
hello_mesh_spatz_test now passes on both: 16/16 tiles start and complete the
Spatz task, exit code 0, and the two simulators' output matches message for
message. hello_mesh and inter_l1_test also pass under Verilator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Enables simulation of the MAGIA mesh with Verilator alongside the existing QuestaSim flow, but independent of it. Verilation enables also easier AI-assisted debugging in MAGIA tiles.
Hierarchical Verilation
Flattening the tile 16 times does not build in reasonable time or memory, so the tile is Verilated once into a library and instantiated 16 times, via a
hier_blockmetacomment on a Verilator-specificmagia_tile_hierwrapper. Consequences:magia_hier_params.vis empty but required. Verilator only substitutes a prebuilt library when--hierarchical-params-fileis given; without it the tile is inlined and the result is a flat model that still builds, links the unused library, and looks fine.verilate-check-hierarchyinspects the built parent to avoid this situation.magia_tiledrives a packedobserve_ostruct (AXI write channel for prints, ID/EX/WB instructions) that the VIP reads instead.VL_TIME_CONTEXTonly as a side effect of--mainand only for the parent, so this flow forces it on both.Waveforms
The model is always built with
--trace-fst; dumping is opt-in per run withVERILATOR_FST=<file>, so a run that asks for nothing pays only a larger binary. There is currently no way to build without tracing: the non-tracing model diverges from the tracing one on tests that do real memory traffic, for reasons not yet understood, and nothing about that failure announces itself. This was tested before fixing the reset bug (see below): it may be that a non-tracing version of Verilator is possible again.target/verilator/src/magia_main.cppowns dumping, replacing themainVerilator generates: this enables correctly dumpting the signals in the hierarchical blocks.Non-Verilator specific issues fixed
rst_n <- 0and thenrst_n <- 1. In 2-level logic, with implicit initialization to 0, this does not work.32'h100000000, which does not fit 32 bits.FILTER_WRITE_R_VALIDfromN_HWPEalone rather than the widest ofN_HWPE/N_DMA/N_CORE. This is tolerated by QuestaSim, but prevents Verilation.Layout
Limits
Tested on
mesh_dv=1andCV32E40Ponly. VERILATOR_THREADS>1 is experimental: 4 threads seem like a reasonable compromise between speedup and efficiency.