diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000000..1f4bd3bdec --- /dev/null +++ b/.dockerignore @@ -0,0 +1,13 @@ +.git +.github +build +build-femu +build-docker +roms +*.qcow2 +*.img +*.iso +*.log +docker-data +images +guest diff --git a/.gitignore b/.gitignore index b24a3d5a36..f1f16cc001 100644 --- a/.gitignore +++ b/.gitignore @@ -133,3 +133,15 @@ trace-ust-all.c build-femu/ build/ *.md +# The blanket *.md above came in with the QEMU 10.1.0 upgrade and silently +# drops documentation. The harness's own docs are the part a reader needs +# most, so they are excepted rather than force-added one at a time. +!moe-harness/**/*.md +# and the documentation at the root, which is the first thing a reader opens. +# README.md predates the rule above and is tracked; without this the next +# document beside it would be dropped without a word, as RUNNING.md was. +!/*.md +# What the container writes: counter CSVs, console logs, the QMP socket. The +# compose default puts it beside the checkout, so it needs naming here. +/docker-data/ +subprojects/.wraplock diff --git a/README.md b/README.md index d3ea9c168e..532a2f4de2 100644 --- a/README.md +++ b/README.md @@ -163,6 +163,65 @@ OpenChannel needs a host that speaks it. LightNVM was removed from Linux in --- +## Running it in a container + +Commands, with their real output, are in [RUNNING.md](RUNNING.md). The short +version and the reasoning follow. + + +`femu-scripts/femu-docker.sh` is the path below done in a container. Two +separate things push it there. + +**The host cannot run the path below.** `pkgdep.sh` installs packages, and +`run-blackbox.sh` launches QEMU under `sudo` because FEMU pins its memory +backend, which needs `RLIMIT_MEMLOCK` raised past the device size. A host +allowing the usual 64 MiB cannot start a 64 GiB device at all, and raising it +needs root. The container gets `IPC_LOCK` and an unlimited memlock without the +host granting root to anyone. + +**Inside the container that argument stops applying, and a different one +starts.** The container runs as root, so `sudo` would be moot there -- it is not +even installed. What makes `run-blackbox.sh` unusable in it is that the SSD +layout is written into the file: `pgs_per_blk=256`, `luns_per_ch=8`, `nchs=8`, +`ssd_size=12288`, a fixed `u20s.qcow2`. It reads no environment, so there is no +way to hand it this fork's geometry, a different guest disk, a payload disk or +a counter path per run. `docker/femu-run` is that same script with those +constants lifted out into environment variables; the steps it performs are +unchanged. + +The steps map one to one: + +| this README | container | +|---|---| +| `sudo ./pkgdep.sh` | `docker/Dockerfile`, builder stage | +| `./femu-compile.sh` | `docker/Dockerfile`, builder stage | +| `./qemu-system-x86_64 -device femu,help` | `femu-docker.sh verify` | +| `./run-blackbox.sh` | `femu-docker.sh run` | + +```bash +./femu-scripts/femu-docker.sh build # dependencies and compile, inside +./femu-scripts/femu-docker.sh verify # did the femu device register +./femu-scripts/femu-docker.sh image # guest overlay + cloud-init seed +./femu-scripts/femu-docker.sh run # start the device, boot the guest +./femu-scripts/femu-docker.sh ssh # a shell in the guest +./femu-scripts/femu-docker.sh stop +``` + +One difference worth knowing: this README has you build a VM image by hand and +reuse it. `image` cuts a copy-on-write overlay per instance instead and writes +a cloud-init seed that authorises your ssh key, so the base image is never +written and a broken guest is one file to delete. The cloud image ships no +password, so without that seed there is no way in. + +The device is the same on both paths -- geometry, cell type and the read-energy +coefficients come from `compose.yaml`, whose defaults are the configuration the +measurements in this repository were taken on. `femu-docker.sh status` prints +what the instance would get. + +Running an actual measurement is a different entry point, because it needs a +payload image, a compiled trace and the placement checks: see +[moe-harness/README.md](moe-harness/README.md). + ## Installation ### Build FEMU diff --git a/RUNNING.md b/RUNNING.md new file mode 100644 index 0000000000..eeccd0a0da --- /dev/null +++ b/RUNNING.md @@ -0,0 +1,146 @@ +# Running this FEMU + +Every command below was run on the measurement host and its output is what is +quoted. Two entry points, because they answer different questions. + +| you want | go to | +|---|---| +| a VM with this project's SSD attached, to poke at | [A device](#a-device) | +| the placement measurement, end to end | [A measurement](#a-measurement) | + +The device is the same either way, and no geometry argument is needed for it: +`compose.yaml` defaults to the configuration the measurements were taken on -- +64 GiB over 2 channels x 4 LUNs, 512 pages per block, QLC, `op_pcent=7`. + +## Why a container + +The [README](README.md)'s path builds on the host and launches with +`run-blackbox.sh`. That does not work here. `pkgdep.sh` installs packages, and +FEMU pins its memory backend, which needs `RLIMIT_MEMLOCK` raised past the +device size -- this host allows 64 MiB against 64 GiB. Both need root, and +there is no passwordless sudo. + +The container is given `IPC_LOCK` and an unlimited memlock, so it pins without +the host granting root to anyone. The steps are the README's: + +| README | here | +|---|---| +| `sudo ./pkgdep.sh` | `docker/Dockerfile`, builder stage | +| `./femu-compile.sh` | `docker/Dockerfile`, builder stage | +| `./qemu-system-x86_64 -device femu,help` | `femu-docker.sh verify` | +| `./run-blackbox.sh` | `femu-docker.sh run` | + +Inside the container that argument no longer applies -- it runs as root -- and +`run-blackbox.sh` is usable there too: it now reads the same environment +variables. What it could not do before was take any configuration at all; the +layout was written into the file. + +## A device + +```bash +cd /data/kwkim02/MoE_FEMU + +export FEMU_GUEST_DIR=/data/kwkim02/images # where guest disks live +export FEMU_DATA_DIR=$PWD/docker-data # container's /data +export FEMU_INSTANCE=demo # names disk, seed, container +``` + +**Build.** Dependencies and compile happen inside; the host gets nothing. + +```bash +./femu-scripts/femu-docker.sh build +``` + +**Check the device registered.** + +```bash +./femu-scripts/femu-docker.sh verify +# femu options: +# acl= ... blks_per_pl= ... nand_cell_type= ... +``` + +**Make this instance's guest disk.** A copy-on-write overlay plus a cloud-init +seed carrying your ssh key. The base image is never written, and the cloud +image ships no password, so without the seed there is no way in. + +```bash +./femu-scripts/femu-docker.sh image +# overlay /data/kwkim02/images/femu-root-demo.qcow2 +# seed /data/kwkim02/images/seed-demo.iso +``` + +**Start it.** Holds the terminal; Ctrl-C detaches and leaves the container up. + +```bash +./femu-scripts/femu-docker.sh run +# FEMU mode=bbssd, NAND cell type=4, image=/guest/femu-root-demo.qcow2 +# Guest SSH is forwarded to container port 2222 +``` + +**Get in.** Boot takes three to four minutes. The `femu login:` prompt appears +well before you can log in -- cloud-init installs the key after it. Wait for +`Cloud-init ... finished` on the console. + +```bash +./femu-scripts/femu-docker.sh ssh +# femu@femu:~$ lsblk -dno NAME,SIZE /dev/nvme0n1 +# nvme0n1 59.8G +``` + +59.8 G rather than 64 is `op_pcent=7`: the over-provisioning every layout here +is planned against. + +**Stop.** + +```bash +./femu-scripts/femu-docker.sh stop +``` + +**What am I about to get.** + +```bash +./femu-scripts/femu-docker.sh status +# cell=4 size=65536MB 2ch x 4LUN 512pg/blk 1024blk/pl opts=op_pcent=7 +``` + +## A measurement + +A device with nothing on it measures nothing. A run also needs a payload image, +a compiled trace and the placement checks, which is a different entry point: + +```bash +FEMU_PROJECT_ROOT=/data/kwkim02/MoE_SSD \ + bash /data/kwkim02/MoE_FEMU/moe-harness/exp/moe_bcq/femu_run/run_device.sh \ + DEVICE_TAG IMAGE_BASENAME IMAGE_PAGES SPECFILE +``` + +`FEMU_PROJECT_ROOT` is where the data lives. The harness resolves its own code +from where it sits and the data from there, so the two need not be together -- +the images, payload packages and records are tens of gigabytes and are +distributed separately from this repository. + +It boots a device, fills it, asserts the placement landed, replays each trace +in the spec file, and writes `groups.jsonl.gz` and `replay.csv` per run under +`/exp/moe_bcq/femu_run/records//`. + +See [moe-harness/README.md](moe-harness/README.md) for what a spec file is, what +the fill contract is, and why a matching read-back hash does not mean the +placement is right. + +## If something goes wrong + +**`no python3 with pycdlib`** — the seed builder needs it. `pip install --user +pycdlib`, or set `PYTHON` to an interpreter that has it. + +**`kex_exchange_identification: Connection closed`** — the guest is up but +cloud-init has not installed the key yet. Wait for `Cloud-init ... finished`. + +**`already exists: .../femu-root-demo.qcow2`** — `image` refuses to overwrite an +instance's disk. Delete it to start over, or use another `FEMU_INSTANCE`. + +**A device that is not 59.8 G** — something is overriding the compose defaults. +`femu-docker.sh status` prints what the instance would actually get. + +**Port 2222 already bound** — another instance is running. `docker ps`, then +`FEMU_INSTANCE= ./femu-scripts/femu-docker.sh stop`, or set +`FEMU_SSH_PORT` for this one. diff --git a/compose.yaml b/compose.yaml new file mode 100644 index 0000000000..4fb1218c1f --- /dev/null +++ b/compose.yaml @@ -0,0 +1,86 @@ +services: + femu: + build: + context: . + dockerfile: docker/Dockerfile + args: + BUILD_JOBS: ${FEMU_BUILD_JOBS:-8} + image: ${FEMU_DOCKER_IMAGE:-femu-qlc:latest} + container_name: ${FEMU_CONTAINER_NAME:-femu} + devices: + - /dev/kvm:/dev/kvm + cap_add: + - IPC_LOCK + ulimits: + memlock: + soft: -1 + hard: -1 + volumes: + - ${FEMU_GUEST_DIR:-../images}:/guest + - ${FEMU_DATA_DIR:-./docker-data}:/data + ports: + - "${FEMU_SSH_PORT:-2222}:2222" + environment: + FEMU_MODE: ${FEMU_MODE:-bbssd} + FEMU_IMAGE: /guest/${FEMU_IMAGE_NAME:-u20s.qcow2} + FEMU_KERNEL: ${FEMU_KERNEL:-} + FEMU_INITRD: ${FEMU_INITRD:-} + FEMU_KERNEL_APPEND: ${FEMU_KERNEL_APPEND:-root=LABEL=rootfs console=ttyS0} + FEMU_MEMORY: ${FEMU_MEMORY:-8G} + FEMU_CPUS: ${FEMU_CPUS:-6} + FEMU_GUEST_SSH_PORT: 2222 + FEMU_NAND_CELL_TYPE: ${FEMU_NAND_CELL_TYPE:-4} + # The QLC-aligned expert placement geometry: 64 GiB over 2 channels x 4 + # LUNs, 512 pages per block. These are defaults, not a suggestion -- they + # are the device the measurements in this repository were taken on, and + # a run that leaves them alone reproduces it. + # + # 512 pages per block is the part that matters most. The QLC pairing table + # only covers pages 0..495 upstream, so at the old 256-page default every + # page class is correct and the bug this fork fixes (nand.c, rows - 1) + # cannot appear. Same source, different physics, silently. + # + # op_pcent=7 is over-provisioning: it sets the exposed namespace to 59.8 G + # of the 64 GiB raw, which is the capacity every layout here is planned + # against. + FEMU_SSD_SIZE_MB: ${FEMU_SSD_SIZE_MB:-65536} + FEMU_SECTORS_PER_PAGE: ${FEMU_SECTORS_PER_PAGE:-32} + FEMU_PAGES_PER_BLOCK: ${FEMU_PAGES_PER_BLOCK:-512} + FEMU_BLOCKS_PER_PLANE: ${FEMU_BLOCKS_PER_PLANE:-1024} + FEMU_PLANES_PER_LUN: ${FEMU_PLANES_PER_LUN:-1} + FEMU_LUNS_PER_CHANNEL: ${FEMU_LUNS_PER_CHANNEL:-4} + FEMU_CHANNELS: ${FEMU_CHANNELS:-2} + FEMU_QLC_STATS_PATH: ${FEMU_QLC_STATS_PATH:-/data/qlc_counts.csv} + # 에너지 계수 (milli-pJ/bit). peripheral + array 분해 모델: + # peripheral = P_fix + rho_p * t_R(c), P_fix=1.273 rho_p=0.668 + # array = E_fix + (n(c)-1) * E_sense, E_fix=0.551 E_sense=0.494 + # n(c) = 1/2/4/8 은 class 별 sensing 횟수 추정값 (인용값 아님) + FEMU_E_READ_C0: ${FEMU_E_READ_C0:-33821} + FEMU_E_READ_C1: ${FEMU_E_READ_C1:-53220} + FEMU_E_READ_C2: ${FEMU_E_READ_C2:-93219} + FEMU_E_READ_C3: ${FEMU_E_READ_C3:-157653} + FEMU_E_ARRAY_C0: ${FEMU_E_ARRAY_C0:-551} + FEMU_E_ARRAY_C1: ${FEMU_E_ARRAY_C1:-1045} + FEMU_E_ARRAY_C2: ${FEMU_E_ARRAY_C2:-2033} + FEMU_E_ARRAY_C3: ${FEMU_E_ARRAY_C3:-4009} + FEMU_E_XFER: ${FEMU_E_XFER:-28100} + FEMU_STATS_FLUSH_MS: ${FEMU_STATS_FLUSH_MS:-0} + FEMU_IMAGE_FORMAT: ${FEMU_IMAGE_FORMAT:-qcow2} + FEMU_EXTRA_DEVICE_OPTS: ${FEMU_EXTRA_DEVICE_OPTS:-op_pcent=7} + # Extra guest disks, ';'-separated QEMU -drive specs. A replay payload + # is far too large for a cloud-init seed and the guest has no network, + # so it comes in as a read-only disk and is copied in from inside. + FEMU_EXTRA_DRIVES: ${FEMU_EXTRA_DRIVES:-} + # FEMU_EXP_LOG switches the [EXP] log on; FEMU_SECRET is the marker + # string it looks for. Together they make FEMU report lpn -> PPA for every + # page whose content carries the marker, which is the only way to read the + # real address mapping instead of inferring it from read latency. + FEMU_EXP_LOG: ${FEMU_EXP_LOG:-} + FEMU_SECRET: ${FEMU_SECRET:-} + FEMU_DUMP_LPN: ${FEMU_DUMP_LPN:-} + FEMU_QMP_SOCKET: /data/qmp.sock + FEMU_ALLOW_UNPINNED: ${FEMU_ALLOW_UNPINNED:-} + command: ["${FEMU_MODE:-bbssd}"] + stdin_open: true + tty: true + stop_grace_period: 30s diff --git a/docker/Dockerfile b/docker/Dockerfile new file mode 100644 index 0000000000..1e72080406 --- /dev/null +++ b/docker/Dockerfile @@ -0,0 +1,72 @@ +FROM ubuntu:24.04 AS builder + +ARG DEBIAN_FRONTEND=noninteractive +ARG BUILD_JOBS=8 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + bash \ + build-essential \ + ca-certificates \ + git \ + libaio-dev \ + libdw-dev \ + libfdt-dev \ + libglib2.0-dev \ + libnuma-dev \ + libpixman-1-dev \ + libslirp-dev \ + ninja-build \ + pkg-config \ + python3 \ + python3-venv \ + zlib1g-dev \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /src/femu +COPY . . + +# FEMU carries a compatible Meson wheel in python/wheels. QEMU may still fetch +# pinned source subprojects that are not populated in the source checkout. +RUN mkdir -p build-docker \ + && cd build-docker \ + && ../configure \ + --enable-kvm \ + --enable-slirp \ + --target-list=x86_64-softmmu \ + --disable-docs \ + --disable-gtk \ + --disable-sdl \ + --disable-werror \ + --enable-strip \ + --prefix=/opt/femu \ + && ninja -j "${BUILD_JOBS}" \ + && ninja install \ + && /opt/femu/bin/qemu-system-x86_64 --version + +FROM ubuntu:24.04 AS runtime + +ARG DEBIAN_FRONTEND=noninteractive + +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + bash \ + libaio1t64 \ + libdw1t64 \ + libfdt1 \ + libglib2.0-0t64 \ + libnuma1 \ + libpixman-1-0 \ + libslirp0 \ + zlib1g \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /opt/femu /opt/femu + +COPY docker/femu-run /usr/local/bin/femu-run +RUN chmod 0755 /usr/local/bin/femu-run \ + && mkdir -p /images /data + +ENV PATH="/opt/femu/bin:${PATH}" +ENTRYPOINT ["/usr/local/bin/femu-run"] +CMD ["help"] diff --git a/docker/femu-run b/docker/femu-run new file mode 100755 index 0000000000..0085130e62 --- /dev/null +++ b/docker/femu-run @@ -0,0 +1,193 @@ +#!/usr/bin/env bash +set -euo pipefail + +usage() { + cat <<'EOF' +Usage: femu-run MODE [additional QEMU options] + +MODE: + bbssd Run a BlackBox SSD (default cell type: QLC) + zns Run a Zoned Namespace SSD (default flash type: QLC) + shell Open a shell in the container + version Print the FEMU/QEMU version + help Show this message + +Required runtime input: + FEMU_IMAGE=/guest/u20s.qcow2 + +Common environment variables: + FEMU_KERNEL=/guest/kernel Optional external guest kernel + FEMU_INITRD=/guest/initrd Optional initrd used with FEMU_KERNEL + FEMU_KERNEL_APPEND=... Kernel command line + FEMU_MEMORY=4G Guest RAM + FEMU_CPUS=4 Guest vCPUs + FEMU_NAND_CELL_TYPE=4 1=SLC, 2=MLC, 3=TLC, 4=QLC + FEMU_SSD_SIZE_MB=16384 Emulated SSD capacity + FEMU_QLC_STATS_PATH=... Write page-class read counters as CSV on shutdown + FEMU_E_READ_C0..C3=... Read energy per page class, milli-pJ/bit + (default 33821/53220/93219/157653) + FEMU_E_ARRAY_C0..C3=... Array share of that coefficient; the rest is + peripheral, which scales with read latency + FEMU_E_XFER=28100 Channel transfer energy, milli-pJ/bit + FEMU_STATS_FLUSH_MS=0 Rewrite the CSV every N ms while running (0 = on exit only) + FEMU_GUEST_SSH_PORT=2222 Port inside the container forwarded to guest :22 + FEMU_EXTRA_DEVICE_OPTS=... Extra comma-separated femu device properties + FEMU_EXTRA_DRIVES=... Extra guest disks, one QEMU -drive spec each, + separated by ';'. The payload image a replay + experiment loads onto the emulated SSD is far too + large for a cloud-init seed, so it is attached as a + read-only disk and copied in from inside the guest. + FEMU_ALLOW_TCG=1 Permit slow TCG fallback when /dev/kvm is absent +EOF +} + +mode="${1:-${FEMU_MODE:-bbssd}}" +if [[ $# -gt 0 ]]; then + shift +fi + +case "${mode}" in + help|-h|--help) + usage + exit 0 + ;; + version) + exec qemu-system-x86_64 --version + ;; + shell|bash) + exec /bin/bash "$@" + ;; + bbssd|zns) + ;; + *) + echo "femu-run: unsupported mode '${mode}'" >&2 + usage >&2 + exit 2 + ;; +esac + +femu_image="${FEMU_IMAGE:-/guest/u20s.qcow2}" +if [[ ! -f "${femu_image}" ]]; then + echo "femu-run: guest image not found: ${femu_image}" >&2 + echo "Mount the host guest directory at /guest or set FEMU_IMAGE." >&2 + exit 1 +fi + +boot_args=() +femu_kernel="${FEMU_KERNEL:-}" +femu_initrd="${FEMU_INITRD:-}" +if [[ -n "${femu_kernel}" ]]; then + if [[ ! -f "${femu_kernel}" ]]; then + echo "femu-run: guest kernel not found: ${femu_kernel}" >&2 + exit 1 + fi + boot_args+=(-kernel "${femu_kernel}") + if [[ -n "${femu_initrd}" ]]; then + if [[ ! -f "${femu_initrd}" ]]; then + echo "femu-run: guest initrd not found: ${femu_initrd}" >&2 + exit 1 + fi + boot_args+=(-initrd "${femu_initrd}") + fi + boot_args+=(-append "${FEMU_KERNEL_APPEND:-root=LABEL=rootfs console=ttyS0}") +fi + +accel_args=() +if [[ -c /dev/kvm && -r /dev/kvm && -w /dev/kvm ]]; then + accel_args=(-enable-kvm -cpu host) +elif [[ "${FEMU_ALLOW_TCG:-0}" == "1" ]]; then + echo "femu-run: warning: /dev/kvm unavailable; using slow TCG emulation" >&2 + accel_args=(-accel tcg -cpu max) +else + echo "femu-run: /dev/kvm is unavailable or not accessible" >&2 + echo "Start the container with: --device /dev/kvm" >&2 + echo "Set FEMU_ALLOW_TCG=1 only if slow software emulation is acceptable." >&2 + exit 1 +fi + +cell_type="${FEMU_NAND_CELL_TYPE:-4}" +if [[ ! "${cell_type}" =~ ^[1-4]$ ]]; then + echo "femu-run: FEMU_NAND_CELL_TYPE must be 1, 2, 3, or 4" >&2 + exit 2 +fi + +ssd_size_mb="${FEMU_SSD_SIZE_MB:-16384}" +extra_device_opts="${FEMU_EXTRA_DEVICE_OPTS:-}" + +if [[ "${mode}" == "bbssd" ]]; then + device="femu,devsz_mb=${ssd_size_mb},namespaces=1,femu_mode=1" + device+=",secsz=${FEMU_SECTOR_SIZE:-512}" + device+=",secs_per_pg=${FEMU_SECTORS_PER_PAGE:-8}" + device+=",pgs_per_blk=${FEMU_PAGES_PER_BLOCK:-256}" + device+=",blks_per_pl=${FEMU_BLOCKS_PER_PLANE:-256}" + device+=",pls_per_lun=${FEMU_PLANES_PER_LUN:-1}" + device+=",luns_per_ch=${FEMU_LUNS_PER_CHANNEL:-8}" + device+=",nchs=${FEMU_CHANNELS:-8}" + device+=",gc_thres_pcent=${FEMU_GC_THRESHOLD:-75}" + device+=",gc_thres_pcent_high=${FEMU_GC_THRESHOLD_HIGH:-95}" + device+=",nand_cell_type=${cell_type}" + device+=",e_read_c0_mpj=${FEMU_E_READ_C0:-34000}" + device+=",e_read_c1_mpj=${FEMU_E_READ_C1:-54000}" + device+=",e_read_c2_mpj=${FEMU_E_READ_C2:-95400}" + device+=",e_read_c3_mpj=${FEMU_E_READ_C3:-161700}" + device+=",e_array_c0_mpj=${FEMU_E_ARRAY_C0:-551}" + device+=",e_array_c1_mpj=${FEMU_E_ARRAY_C1:-1045}" + device+=",e_array_c2_mpj=${FEMU_E_ARRAY_C2:-2033}" + device+=",e_array_c3_mpj=${FEMU_E_ARRAY_C3:-4009}" + device+=",e_xfer_mpj=${FEMU_E_XFER:-28100}" + device+=",stats_flush_ms=${FEMU_STATS_FLUSH_MS:-0}" +else + # The supplied FEMU ZNS model supports SLC, TLC, and QLC (not MLC). + if [[ "${cell_type}" == "2" ]]; then + echo "femu-run: the ZNS model does not support MLC (cell type 2)" >&2 + exit 2 + fi + device="femu,devsz_mb=${ssd_size_mb},namespaces=1,femu_mode=3" + device+=",zns_num_ch=${FEMU_ZNS_CHANNELS:-8}" + device+=",zns_num_lun=${FEMU_ZNS_LUNS_PER_CHANNEL:-4}" + device+=",zns_num_plane=${FEMU_ZNS_PLANES_PER_LUN:-2}" + device+=",zns_num_blk=${FEMU_ZNS_BLOCKS_PER_PLANE:-32}" + device+=",zns_flash_type=${cell_type}" +fi + +if [[ -n "${extra_device_opts}" ]]; then + device+=",${extra_device_opts#,}" +fi + +memory="${FEMU_MEMORY:-4G}" +cpus="${FEMU_CPUS:-4}" +ssh_port="${FEMU_GUEST_SSH_PORT:-2222}" +qmp_socket="${FEMU_QMP_SOCKET:-/data/qmp.sock}" +image_format="${FEMU_IMAGE_FORMAT:-qcow2}" + +# Extra disks, if any. Split on ';' rather than whitespace: a -drive spec is a +# comma-separated list that may itself contain paths with spaces. +extra_drive_args=() +if [[ -n "${FEMU_EXTRA_DRIVES:-}" ]]; then + # `read` returns non-zero on a final line with no newline, which would end + # the loop before that last spec is used, so terminate the stream explicitly. + while IFS= read -r spec; do + [[ -z "${spec}" ]] && continue + extra_drive_args+=(-drive "${spec}") + done < <(printf '%s\n' "${FEMU_EXTRA_DRIVES}" | tr ';' '\n') +fi + +echo "FEMU mode=${mode}, NAND cell type=${cell_type}, image=${femu_image}" +echo "Guest SSH is forwarded to container port ${ssh_port}" + +exec qemu-system-x86_64 \ + -name "FEMU-${mode^^}-VM" \ + "${accel_args[@]}" \ + -smp "${cpus}" \ + -m "${memory}" \ + "${boot_args[@]}" \ + -device virtio-scsi-pci,id=scsi0 \ + -device scsi-hd,drive=hd0 \ + -drive "file=${femu_image},if=none,aio=native,cache=none,format=${image_format},id=hd0" \ + -device "${device}" \ + "${extra_drive_args[@]}" \ + -netdev "user,id=net0,hostfwd=tcp::${ssh_port}-:22" \ + -device virtio-net-pci,netdev=net0 \ + -nographic \ + -qmp "unix:${qmp_socket},server=on,wait=off" \ + "$@" diff --git a/hw/femu/backend/dram.c b/hw/femu/backend/dram.c index 1c3c5bae37..e655293aee 100644 --- a/hw/femu/backend/dram.c +++ b/hw/femu/backend/dram.c @@ -65,13 +65,41 @@ int init_dram_backend(SsdDramBackend **mbe, int64_t nbytes) /* * Pinning keeps page faults out of the emulated latency, but it needs * RLIMIT_MEMLOCK to cover the backend, which an unprivileged run rarely - * has. Say what that costs and carry on rather than refuse to start: the - * device works either way, only its timing is then subject to faults. + * has. Upstream warns and carries on; this fork refuses instead, because + * the emulated latency is the measurement, and a fault landing inside one + * is indistinguishable from the NAND time it is supposed to be reporting. + * FEMU_ALLOW_UNPINNED=1 opts back into upstream's behaviour. */ if (mlock(b->logical_space, nbytes) == -1) { - femu_err("cannot pin the %" PRId64 " MiB memory backend (%s); " - "latencies may jitter until RLIMIT_MEMLOCK allows it\n", - nbytes / MiB, strerror(errno)); + /* + * Pinning keeps the backing store out of swap so a page fault cannot show + * up inside an emulated NAND access. It needs RLIMIT_MEMLOCK >= the device + * size, which an unprivileged user often cannot raise (the hard limit is + * commonly 64 MB and only root can lift it). + * + * FEMU_ALLOW_UNPINNED=1 downgrades the failure to a warning. Only set it + * on a host with no memory pressure -- check that `vmstat` reports si/so + * at 0 and that free RAM comfortably exceeds the device size. Swap that is + * merely *occupied* by stale pages is fine; swap that is *active* is not, + * because a fault during an emulated access lands directly in the measured + * latency. Default behaviour is unchanged. + */ + const char *allow_unpinned = getenv("FEMU_ALLOW_UNPINNED"); + if (allow_unpinned && allow_unpinned[0] && + strcmp(allow_unpinned, "0") != 0) { + femu_err("WARNING: memory backend is NOT pinned (mlock: %s).\n", + strerror(errno)); + femu_err("WARNING: FEMU_ALLOW_UNPINNED=1 is set, continuing anyway. " + "Latency measurements are only trustworthy while the host " + "is not swapping.\n"); + } else { + femu_err("Failed to pin the memory backend to the host DRAM\n"); + femu_err("Raise RLIMIT_MEMLOCK (ulimit -l) to at least %" PRId64 + " MB, or set FEMU_ALLOW_UNPINNED=1 to continue unpinned.\n", + nbytes >> 20); + g_free(b->logical_space); + abort(); + } } return 0; diff --git a/hw/femu/bbssd/bb.c b/hw/femu/bbssd/bb.c index 06acc25883..27aa216a64 100644 --- a/hw/femu/bbssd/bb.c +++ b/hw/femu/bbssd/bb.c @@ -135,6 +135,8 @@ static void bb_init(FemuCtrl *n, NvmeNamespace *ns, Error **errp) * only whichever namespace brought its mode up first, and using it would leave * every other FTL-backed namespace on the previous setting. */ +static void bb_flush_stats(FemuCtrl *n); + static void bb_flip_apply(FemuCtrl *n, int64_t cdw10) { bool resume; @@ -221,6 +223,37 @@ static void bb_flip(FemuCtrl *n, NvmeCmd *cmd) femu_log("%s,Reset tt_late_ios/tt_ios,%ld/%ld\n", n->devname, late, tt); break; } + case FEMU_RESET_QLC: { + /* + * Zero the physical-read meters so what follows is attributable to the + * workload alone. Stored data, the mapping table and the page layout are + * untouched. qlc_first_read_ns goes too, so elapsed time restarts at the + * next counted read rather than at one from the fill. + */ + uint64_t before = 0; + for (int i = 0; i < n->num_namespaces; i++) { + struct ssd *ssd = n->namespaces[i].ssd; + if (!ssd) { + continue; + } + for (int c = 0; c < 4; c++) { + before += __atomic_load_n(&ssd->qlc_read_pages[c], __ATOMIC_RELAXED); + __atomic_store_n(&ssd->qlc_read_pages[c], 0, __ATOMIC_RELAXED); + __atomic_store_n(&ssd->qlc_read_bytes[c], 0, __ATOMIC_RELAXED); + __atomic_store_n(&ssd->qlc_read_active_ns[c], 0, __ATOMIC_RELAXED); + } + __atomic_store_n(&ssd->qlc_first_read_ns, 0, __ATOMIC_RELAXED); + } + /* Logged, not discarded: the pre-workload total is itself a measurement + * of what boot and fill cost, and it is the only record of it. */ + femu_log("%s,QLC counters reset, discarded %" PRIu64 " pages\n", + n->devname, before); + break; + } + case FEMU_SNAP_QLC: + bb_flush_stats(n); + femu_log("%s,QLC counters snapshotted\n", n->devname); + break; case FEMU_ENABLE_LOG: n->print_log = true; femu_log("%s,Log print [Enabled]!\n", n->devname); @@ -234,6 +267,144 @@ static void bb_flip(FemuCtrl *n, NvmeCmd *cmd) } } +/* Snapshot physical QLC activity without freeing state; process-exit notifiers + * use this path because PCI device teardown is not guaranteed at VM shutdown. */ +static void bb_flush_stats(FemuCtrl *n) +{ + uint64_t pages[4] = {0}; + uint64_t bytes[4] = {0}; + uint64_t active_ns[4] = {0}; + uint64_t first_ns = 0; + uint64_t wall_ns = 0; + int n_luns = 0; + const char *stats_path = getenv("FEMU_QLC_STATS_PATH"); + FILE *stats = NULL; + uint64_t t; + int i; + + for (i = 0; i < n->num_namespaces; i++) { + struct ssd *ssd = n->namespaces[i].ssd; + + if (ssd) { + int page_class; + + for (page_class = 0; page_class < 4; page_class++) { + pages[page_class] += __atomic_load_n( + &ssd->qlc_read_pages[page_class], __ATOMIC_RELAXED); + bytes[page_class] += __atomic_load_n( + &ssd->qlc_read_bytes[page_class], __ATOMIC_RELAXED); + active_ns[page_class] += __atomic_load_n( + &ssd->qlc_read_active_ns[page_class], __ATOMIC_RELAXED); + } + + if (!n_luns) { + n_luns = ssd->sp.nchs * ssd->sp.luns_per_ch * ssd->sp.pls_per_lun; + } + + t = __atomic_load_n(&ssd->qlc_first_read_ns, __ATOMIC_RELAXED); + if (t && (!first_ns || t < first_ns)) { + first_ns = t; + } + } + } + + /* + * Elapsed time since the first counted read. Sum(t_active) is per-LUN service + * time added up, so it runs ahead of this by roughly the LUN parallelism; the + * controller is a single resource and has to be charged against elapsed time. + */ + if (first_ns) { + uint64_t now = qemu_clock_get_ns(QEMU_CLOCK_REALTIME); + + wall_ns = now > first_ns ? now - first_ns : 0; + } + + if (stats_path && stats_path[0]) { + stats = fopen(stats_path, "w"); + if (!stats) { + femu_log("QLC stats: cannot open %s: %s\n", + stats_path, strerror(errno)); + } + } + + /* + * Energy columns are Sum(count x cited coefficient), the same arithmetic the + * offline accounting does; the coefficients used are written into the file so + * the columns stay auditable. Coefficients are milli-pJ/bit, byte counts are + * exact, so uJ = bytes * 8 * mpj / 1e9. Sweeps (channel, controller, idle, + * scenario codes) stay offline in experiments/energy_account.py. + */ + if (stats) { + double e_nand_uj[4], e_xfer_uj[4], e_array_uj[4], e_periph_uj[4]; + double nand_total = 0, xfer_total = 0; + double array_total = 0, periph_total = 0; + + for (i = 0; i < 4; i++) { + double bits = (double)bytes[i] * 8.0; + /* + * Peripheral is the remainder rather than its own coefficient, so + * the two halves always add back to the total the offline + * accounting uses. An array share above the total would make it + * negative, which is a misconfiguration, not a measurement. + */ + uint32_t array_mpj = n->e_array_mpj[i] <= n->e_read_mpj[i] + ? n->e_array_mpj[i] : n->e_read_mpj[i]; + + e_nand_uj[i] = bits * n->e_read_mpj[i] / 1e9; + e_array_uj[i] = bits * array_mpj / 1e9; + e_periph_uj[i] = e_nand_uj[i] - e_array_uj[i]; + e_xfer_uj[i] = bits * n->e_xfer_mpj / 1e9; + nand_total += e_nand_uj[i]; + array_total += e_array_uj[i]; + periph_total += e_periph_uj[i]; + xfer_total += e_xfer_uj[i]; + + if (n->e_array_mpj[i] > n->e_read_mpj[i]) { + femu_log("QLC energy: class %d array coefficient %u exceeds the " + "read total %u; clamped\n", + i, n->e_array_mpj[i], n->e_read_mpj[i]); + } + } + + fprintf(stats, "# coeff_mpj_per_bit: c0=%u c1=%u c2=%u c3=%u xfer=%u\n", + n->e_read_mpj[0], n->e_read_mpj[1], n->e_read_mpj[2], + n->e_read_mpj[3], n->e_xfer_mpj); + fprintf(stats, "# array_mpj_per_bit: c0=%u c1=%u c2=%u c3=%u " + "(peripheral is the remainder of each read coefficient)\n", + n->e_array_mpj[0], n->e_array_mpj[1], n->e_array_mpj[2], + n->e_array_mpj[3]); + fprintf(stats, "# nand_cell_type=%u e_nand_uj_total=%.3f " + "e_periph_uj_total=%.3f e_array_uj_total=%.3f " + "e_xfer_uj_total=%.3f\n", + n->nand_cell_type, nand_total, periph_total, array_total, + xfer_total); + fprintf(stats, "# t_wall_us=%.3f t_active_sum_us=%.3f luns=%d\n", + wall_ns / 1000.0, + (active_ns[0] + active_ns[1] + active_ns[2] + active_ns[3]) + / 1000.0, n_luns); + fprintf(stats, "# t_wall is elapsed since the first counted read (the " + "observation window). t_active_sum is per-LUN service time " + "added up, so device busy time is about t_active_sum/luns; the " + "controller belongs on that, not on either raw number.\n"); + fprintf(stats, "page_class,n_read,bytes_read,t_active_us," + "e_nand_uj,e_periph_uj,e_array_uj,e_xfer_uj\n"); + for (i = 0; i < 4; i++) { + fprintf(stats, "%d,%" PRIu64 ",%" PRIu64 ",%.3f,%.3f,%.3f,%.3f,%.3f\n", + i, pages[i], bytes[i], active_ns[i] / 1000.0, + e_nand_uj[i], e_periph_uj[i], e_array_uj[i], e_xfer_uj[i]); + } + fclose(stats); + } else { + for (i = 0; i < 4; i++) { + femu_log("QLC_READ_STATS,class=%d,reads=%" PRIu64 + ",bytes=%" PRIu64 ",active_ns=%" PRIu64 + ",e_nand_uj=%.3f\n", + i, pages[i], bytes[i], active_ns[i], + (double)bytes[i] * 8.0 * n->e_read_mpj[i] / 1e9); + } + } +} + /* * Release what the namespace's FTL still holds. Reached for the mode the * controller itself runs; a namespace running bbssd underneath a controller of @@ -250,6 +421,7 @@ static void bb_exit(FemuCtrl *n) { int i; + bb_flush_stats(n); for (i = 0; i < n->num_namespaces; i++) { NvmeNamespace *ns = &n->namespaces[i]; @@ -300,6 +472,7 @@ int nvme_register_bbssd(FemuCtrl *n) .state = NULL, .init = bb_init, .exit = bb_exit, + .stats_flush = bb_flush_stats, .rw_check_req = NULL, .admin_cmd = bb_admin_cmd, .io_cmd = bb_io_cmd, @@ -308,4 +481,3 @@ int nvme_register_bbssd(FemuCtrl *n) return 0; } - diff --git a/hw/femu/bbssd/ftl-media.c b/hw/femu/bbssd/ftl-media.c index f173f5b11c..87efc8e522 100644 --- a/hw/femu/bbssd/ftl-media.c +++ b/hw/femu/bbssd/ftl-media.c @@ -213,6 +213,30 @@ uint64_t ssd_advance_status(struct ssd *ssd, struct ppa *ppa, } loc = bb_decode_loc(ssd, ppa, stime); + + /* + * Count physical QLC page reads at the NAND boundary. This includes host, + * mapping-table and GC reads, which is the correct boundary for NAND-core + * energy. Queueing time is deliberately excluded from active_ns; it records + * the raw PACA array latency selected for this page class. + */ + if (op == NAND_MEDIA_READ && loc.flash_type == QLC && loc.page_type < 4) { + uint64_t page_bytes = (uint64_t)ssd->sp.secsz * ssd->sp.secs_per_pg; + uint64_t active_ns = + ssd->media.cfg.timing.rd_table_ns[loc.flash_type][loc.page_type]; + + if (!__atomic_load_n(&ssd->qlc_first_read_ns, __ATOMIC_RELAXED)) { + __atomic_store_n(&ssd->qlc_first_read_ns, + qemu_clock_get_ns(QEMU_CLOCK_REALTIME), + __ATOMIC_RELAXED); + } + __atomic_fetch_add(&ssd->qlc_read_pages[loc.page_type], 1, + __ATOMIC_RELAXED); + __atomic_fetch_add(&ssd->qlc_read_bytes[loc.page_type], page_bytes, + __ATOMIC_RELAXED); + __atomic_fetch_add(&ssd->qlc_read_active_ns[loc.page_type], active_ns, + __ATOMIC_RELAXED); + } return nand_media_op(&ssd->media, &loc, op, stime).latency_ns; } diff --git a/hw/femu/bbssd/ftl.h b/hw/femu/bbssd/ftl.h index 094873368e..db835c1369 100644 --- a/hw/femu/bbssd/ftl.h +++ b/hw/femu/bbssd/ftl.h @@ -55,6 +55,20 @@ enum { FEMU_RESET_ACCT = 5, FEMU_ENABLE_LOG = 6, FEMU_DISABLE_LOG = 7, + + /* + * The QLC read counters start at zero when the device is created, so a + * dump at process exit covers everything the device ever did: the guest + * probing it at boot, the catalog fill, FTL and GC traffic, the workload, + * and shutdown. Nothing in that total distinguishes the part under study. + * + * FEMU_RESET_QLC zeroes them without touching stored data, the LBA-to-PPA + * map or the page layout -- it resets the meter, not the drive. Issue it + * after the fill and before the workload; FEMU_SNAP_QLC writes the counters + * out at a chosen instant rather than waiting for teardown. + */ + FEMU_RESET_QLC = 8, + FEMU_SNAP_QLC = 9, }; @@ -520,6 +534,20 @@ struct ssd { uint64_t nand_write_pages; /* user pages programmed into NAND */ uint64_t gc_write_pages; /* pages the device relocated itself */ + /* + * QLC read-energy accounting. FEMU records physical media activity only; + * cited energy coefficients are applied offline so model assumptions stay + * explicit and replaceable. Index is QLC page class 0..3. + */ + uint64_t qlc_read_pages[4]; + uint64_t qlc_read_bytes[4]; + uint64_t qlc_read_active_ns[4]; + /* First counted NAND read, QEMU_CLOCK_REALTIME ns; 0 until the first one. + * Sum(t_active) adds per-LUN service time, so it exceeds elapsed time by the + * LUN parallelism. The controller is one resource, so its energy has to be + * charged against elapsed time instead - this is what makes that available. */ + uint64_t qlc_first_read_ns; + /* * Wear: erases summed over every block, kept as a running total so the * average cycle count does not cost a walk of the geometry, and the diff --git a/hw/femu/femu.c b/hw/femu/femu.c index edf4ca2666..f78d2271f9 100644 --- a/hw/femu/femu.c +++ b/hw/femu/femu.c @@ -1,6 +1,8 @@ #include "qemu/osdep.h" #include "qemu/cutils.h" +#include "qemu/timer.h" #include "hw/qdev-properties.h" +#include "system/system.h" #include "./nvme.h" @@ -1463,6 +1465,29 @@ static void nvme_register_extensions_ns(FemuCtrl *n, NvmeNamespace *ns) n->ext_ops = saved_ops; } +static void femu_flush_extension_stats(FemuCtrl *n); + +static void femu_process_exit_notify(Notifier *notifier, void *data) +{ + FemuCtrl *n = container_of(notifier, FemuCtrl, process_exit_notifier); + + femu_flush_extension_stats(n); +} + +/* + * Periodic snapshot so a long run can be watched while it is still going. + * Re-arms itself; the snapshot path does not free or reset any state, so the + * counters keep accumulating and the file is simply rewritten each tick. + */ +static void femu_stats_timer_cb(void *opaque) +{ + FemuCtrl *n = opaque; + + femu_flush_extension_stats(n); + timer_mod(n->stats_timer, + qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + n->stats_flush_ms); +} + /* * Give back what realize has taken. QEMU does not call the exit callback for a * device that never realized, and a device_add that fails validation is an @@ -1630,6 +1655,20 @@ static void femu_realize(PCIDevice *pci_dev, Error **errp) n, QEMU_THREAD_JOINABLE); n->ftl_thread_running = true; } + + /* PCI exit is not guaranteed on whole-process shutdown. Keep experiment + * counters observable on normal guest poweroff and QMP quit as well. */ + n->process_exit_notifier.notify = femu_process_exit_notify; + qemu_add_exit_notifier(&n->process_exit_notifier); + n->process_exit_notifier_registered = true; + + if (n->stats_flush_ms) { + n->stats_timer = timer_new_ms(QEMU_CLOCK_REALTIME, + femu_stats_timer_cb, n); + timer_mod(n->stats_timer, + qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + n->stats_flush_ms); + femu_log("QLC stats snapshot every %u ms\n", n->stats_flush_ms); + } } /* @@ -1755,6 +1794,38 @@ static void femu_exit_extensions(FemuCtrl *n) } } +/* Run each distinct mode's lightweight stats snapshot without freeing state. */ +static void femu_flush_extension_stats(FemuCtrl *n) +{ + void (*seen[FEMU_NR_MODES])(struct FemuCtrl *); + int nseen = 0, i, j; + + if (n->ext_ops.stats_flush) { + seen[nseen++] = n->ext_ops.stats_flush; + } + + for (i = 0; n->namespaces && i < n->num_namespaces; i++) { + void (*flush)(struct FemuCtrl *) = + n->namespaces[i].ext_ops.stats_flush; + + if (!flush) { + continue; + } + for (j = 0; j < nseen; j++) { + if (seen[j] == flush) { + break; + } + } + if (j == nseen && nseen < (int)ARRAY_SIZE(seen)) { + seen[nseen++] = flush; + } + } + + for (j = 0; j < nseen; j++) { + seen[j](n); + } +} + static void femu_exit(PCIDevice *pci_dev) { FemuCtrl *n = FEMU(pci_dev); @@ -1772,6 +1843,14 @@ static void femu_exit(PCIDevice *pci_dev) */ femu_stop_pollers(n); femu_stop_ftl_thread(n); + if (n->stats_timer) { + timer_free(n->stats_timer); + n->stats_timer = NULL; + } + if (n->process_exit_notifier_registered) { + qemu_remove_exit_notifier(&n->process_exit_notifier); + n->process_exit_notifier_registered = false; + } femu_exit_extensions(n); nvme_clear_ctrl(n, true); @@ -1942,6 +2021,33 @@ static const Property femu_props[] = { DEFINE_PROP_STRING("mapping", FemuCtrl, bb_params.mapping_scheme), DEFINE_PROP_UINT32("mapping_cache_mb", FemuCtrl, mapping_cache_mb, 0), DEFINE_PROP_UINT8("nand_cell_type", FemuCtrl, nand_cell_type, 0), + /* QLC page-class read energy, milli-pJ/bit. docs/25 SS5.1 primary profile. */ + /* + * Read energy per class, milli-pJ/bit, as peripheral + array: + * peripheral = P_fix + rho_p * t_R(c), P_fix = 1.273, rho_p = 0.668 + * array = E_fix + (n(c) - 1) * E_sense + * with t_R(c) the measured class read latencies this device already models. + * The peripheral term therefore tracks read latency and, at 97-98% of the + * total, dominates it; see e_array_c*_mpj for the other half. + */ + DEFINE_PROP_UINT32("e_read_c0_mpj", FemuCtrl, e_read_mpj[0], 33821), + DEFINE_PROP_UINT32("e_read_c1_mpj", FemuCtrl, e_read_mpj[1], 53220), + DEFINE_PROP_UINT32("e_read_c2_mpj", FemuCtrl, e_read_mpj[2], 93219), + DEFINE_PROP_UINT32("e_read_c3_mpj", FemuCtrl, e_read_mpj[3], 157653), + /* + * Array share of each read coefficient, milli-pJ/bit: + * E_fix + (n(c) - 1) * E_sense, E_fix = 0.551, E_sense = 0.494 pJ/bit, + * n(c) = 1 / 2 / 4 / 8 senses. + * n(c) is an estimate of per-class sensing complexity, not a cited figure; + * the read latencies it sits beside are measured. Peripheral energy is the + * remainder, e_read_mpj - e_array_mpj. + */ + DEFINE_PROP_UINT32("e_array_c0_mpj", FemuCtrl, e_array_mpj[0], 551), + DEFINE_PROP_UINT32("e_array_c1_mpj", FemuCtrl, e_array_mpj[1], 1045), + DEFINE_PROP_UINT32("e_array_c2_mpj", FemuCtrl, e_array_mpj[2], 2033), + DEFINE_PROP_UINT32("e_array_c3_mpj", FemuCtrl, e_array_mpj[3], 4009), + DEFINE_PROP_UINT32("e_xfer_mpj", FemuCtrl, e_xfer_mpj, 28100), + DEFINE_PROP_UINT32("stats_flush_ms", FemuCtrl, stats_flush_ms, 0), DEFINE_PROP_UINT32("pe_cycles_rated", FemuCtrl, pe_cycles_rated, 0), DEFINE_PROP_INT32("cell_pages", FemuCtrl, bb_params.cell_pages, 0), DEFINE_PROP_INT32("pgtype_lat", FemuCtrl, bb_params.pgtype_lat, 0), diff --git a/hw/femu/nand/nand.c b/hw/femu/nand/nand.c index 1aeeba7398..7298661ce1 100644 --- a/hw/femu/nand/nand.c +++ b/hw/femu/nand/nand.c @@ -134,7 +134,13 @@ static void init_qlc_page_pairing(FemuCtrl *n) for (i = 0; i < sizeof(centerup)/sizeof(centerup[0]); i++) qlc_tbl[centerup[i]] = QLC_UPPER_CENTER_PAGE; - for (i = 0; i < rows - 3; i++) { + /* + * The cycle starts at page 8, so it needs (rows - 1) iterations to reach + * the last page; rows - 3 stops at index 495 and leaves pages 496..511 at + * their zero-initialised value, which reads as QLC_LOWER_PAGE. That is + * invisible while pgs_per_blk <= 496, and wrong above it. + */ + for (i = 0; i < rows - 1; i++) { for (j = 0; j < page_per_row; j += 2) { int idx = 8 + (i * page_per_row) + j; qlc_tbl[idx] = qlc_tbl[idx+1] = lpflag; diff --git a/hw/femu/nand/nand.h b/hw/femu/nand/nand.h index ac3ace8314..976e40e7ea 100644 --- a/hw/femu/nand/nand.h +++ b/hw/femu/nand/nand.h @@ -43,26 +43,28 @@ /* * QLC NAND latency numbers in nanoseconds * - * Read Latency is extrapolated from TLC drives based on Micron FMS'19 - * presentation: "Component-Level Characterization of 3D TLC, QLC, and - * Low-Latency NAND" + * Measured on Intel 96-layer 3D QLC gen3 (1024Q3D3A): + * Q. Chen et al., "PACA: A Page Type Aware Read Cache Scheme in QLC + * Flash-based SSDs", ICCD 2022, Fig. 2 and Table I. * - * Write Latency is increased similar to read latencies, but may be higher in - * practice. + * PACA reports one average program latency rather than a value for each page + * class. Keep the four program entries equal so only measured page-class + * variation is represented. */ -#define QLC_LOWER_PAGE_READ_LATENCY_NS (TLC_LOWER_PAGE_READ_LATENCY_NS * 1.05) -#define QLC_CENTER_LOWER_PAGE_READ_LATENCY_NS (TLC_CENTER_PAGE_READ_LATENCY_NS * 1.1) -#define QLC_CENTER_UPPER_PAGE_READ_LATENCY_NS (TLC_UPPER_PAGE_READ_LATENCY_NS * 1.2) -#define QLC_UPPER_PAGE_READ_LATENCY_NS (TLC_UPPER_PAGE_READ_LATENCY_NS * 1.6) +#define QLC_LOWER_PAGE_READ_LATENCY_NS (47900) +#define QLC_CENTER_LOWER_PAGE_READ_LATENCY_NS (76200) +#define QLC_CENTER_UPPER_PAGE_READ_LATENCY_NS (134600) +#define QLC_UPPER_PAGE_READ_LATENCY_NS (228100) -#define QLC_LOWER_PAGE_WRITE_LATENCY_NS (TLC_LOWER_PAGE_WRITE_LATENCY_NS * 1.05) -#define QLC_CENTER_LOWER_PAGE_WRITE_LATENCY_NS (TLC_CENTER_PAGE_WRITE_LATENCY_NS * 1.1) -#define QLC_CENTER_UPPER_PAGE_WRITE_LATENCY_NS (TLC_UPPER_PAGE_WRITE_LATENCY_NS * 1.2) -#define QLC_UPPER_PAGE_WRITE_LATENCY_NS (TLC_UPPER_PAGE_WRITE_LATENCY_NS * 1.6) +#define QLC_LOWER_PAGE_WRITE_LATENCY_NS (1860000) +#define QLC_CENTER_LOWER_PAGE_WRITE_LATENCY_NS (1860000) +#define QLC_CENTER_UPPER_PAGE_WRITE_LATENCY_NS (1860000) +#define QLC_UPPER_PAGE_WRITE_LATENCY_NS (1860000) -#define QLC_CHNL_PAGE_TRANSFER_LATENCY_NS (52433) -#define QLC_BLOCK_ERASE_LATENCY_NS (3000000) +/* 16 KiB at 800 MT/s. Sweep 40960/20480/13653 ns for 400/800/1200 MT/s. */ +#define QLC_CHNL_PAGE_TRANSFER_LATENCY_NS (20480) +#define QLC_BLOCK_ERASE_LATENCY_NS (6340000) enum { SLC_PAGE = 0, @@ -151,4 +153,3 @@ uint32_t get_rated_pe_cycles(int flash_type); int init_nand_flash(void *opaque); #endif - diff --git a/hw/femu/nand/test/README.md b/hw/femu/nand/test/README.md new file mode 100644 index 0000000000..6933d908a5 --- /dev/null +++ b/hw/femu/nand/test/README.md @@ -0,0 +1,107 @@ +# NAND host tests + +Both tests build and run outside QEMU — no guest, no KVM, no root. Run them from +the FEMU checkout root. + +## test_channel — the media timing model + +`nand_media_op()` is pure timing arithmetic over a caller-supplied timeline. + +```bash +mkdir -p /tmp/nandtest/qemu +printf '#include \n#include \n#include \n#include \n#include \n' \ + > /tmp/nandtest/qemu/osdep.h +gcc -c -I/tmp/nandtest -Ihw/femu/nand -o /tmp/nandtest/nand-media.o hw/femu/nand/nand-media.c +gcc -I/tmp/nandtest -Ihw/femu/nand -o /tmp/nandtest/t \ + hw/femu/nand/test/test_channel.c /tmp/nandtest/nand-media.o -lm +/tmp/nandtest/t +``` + +## test_pairing — the QLC page-class table + +`init_qlc_page_pairing()` is static and its translation unit pulls in QEMU, so the +function text is extracted from `nand.c` instead of being copied into the test. +Re-extract on every build; a stale `.inc` would test nothing. + +```bash +mkdir -p /tmp/nandtest +sed -n '/^static void init_qlc_page_pairing/,/^}/p' hw/femu/nand/nand.c \ + > /tmp/nandtest/pairing_extract.inc +gcc -std=c11 -Wall -Wextra -Werror -I/tmp/nandtest -o /tmp/nandtest/tp \ + hw/femu/nand/test/test_pairing.c +/tmp/nandtest/tp +``` + +Expected output is `class page counts: 132 128 126 126` and `test_pairing: ok`. + +The table is what every physical-layout experiment predicts against, and a +mismatch is silent — the mapper still emits a plan and the device still serves the +reads, only the latency is wrong. Upstream's loop bound was `rows - 3`, which +stops at page index 495 and leaves 496..511 at the zero-initialised value, a +valid-looking `QLC_LOWER_PAGE`. That is invisible at `pgs_per_blk <= 496` — which +is why the earlier 256-page runs were unaffected — and wrong at the 512 pages the +QLC-aligned expert layout requires. Against the unpatched source this test reports +`FAIL pg 496` and counts `128 124 122 122`. + +## test_channel: STAGED assertions are stale, deliberately + +Since the merge of upstream's channel rework, `test_channel` reports six +failures. They are not a regression in anything this fork measures, and they +are not to be "fixed" by adjusting the numbers until it passes. + +Case [1]'s OFF assertions still pass, and OFF is what bbssd selects here: every +channel knob (`pg_xfer_lat`, `ch_xfer_lat`, `cmd_addr_lat`, `status_lat`) +defaults to 0 and the run environment sets none of them, so +`cfg.policy.channel_mode` resolves to `NAND_CH_OFF`. An OFF-mode replay of 4,096 +reads over this geometry (2 ch x 4 LUN) completes at the same nanosecond on both +sides of the merge, so the measurements taken before it stand. + +Cases [2]-[4] run `NAND_CH_STAGED`, and upstream changed what that model does. +It used to serialise the LUNs on a channel completely -- case [3] asserted +exactly that, calling it out as unlike real NAND -- and case [4] therefore saw +the same 1.219x page-mapping gain at every LUN count. Sensing now overlaps +across LUNs and only data-out is serialised on the bus, so the gain varies with +LUN count as the analytical model always assumed it should. + +So the old expectations encoded a defect. Rewriting them is a decision about +what the staged model ought to do, not a mechanical update, and nothing here +uses the staged model yet. Deferred until something does: at that point rebuild +the expectations from the intended physics, not from whatever the code prints. + +## What the timing tests establish + +**bbssd never enables the channel stage.** `ftl-media.c` copies `pg_xfer_lat` into +`cfg.timing.page_xfer_ns` and then sets `cfg.policy.channel_mode = NAND_CH_OFF` +unconditionally. `nand-media.c` reads `page_xfer_ns` only under `NAND_CH_STAGED`, +so passing `pg_xfer_lat=...` on the command line today changes nothing. +`NAND_CH_STAGED` is dead code: both call sites (`bbssd/ftl-media.c`, +`zns/zftl.c`) set `NAND_CH_OFF`. + +**Turning it on is not behaviour-preserving, even with a zero bus.** With every +bus phase at 0, `NAND_CH_STAGED` still differs from `NAND_CH_OFF`, because the +channel timeline is advanced to each op's data-out time and the next op's +command phase is clamped to it. Any `channel_model` option must therefore default +to `off`. + +**The staged model serialises the channel across LUNs.** Reservations are made in +op-submission order, so an op's command phase waits for the *previous* op's +data-out even when the two are on different LUNs. Measured with two LUNs on one +channel, both reads issued at t=0, slow page (228.1 us array): + +| bus transfer | LUN0 done | LUN1 done | if sensing overlapped | +|---|---:|---:|---:| +| 0 us | 228.1 us | 456.2 us | 228.1 us | +| 52.4 us | 280.5 us | 561.1 us | 280.5 us | + +Real NAND issues LUN1's command while LUN0 senses; the bus is needed only for the +command and the data burst. This model holds the channel from command through +data-out, so **adding LUNs per channel buys nothing** and per-op cost is always +`array + transfer`. It describes a controller that does not pipeline. + +The consequence for page-mapping studies: the gain is `(mean_array + xfer) / +(aware_array + xfer)` at every LUN count — 1.219x for the 2-tier unified-LRU +traffic mix at 1.8 GB — rather than falling toward 1.0 as the channel saturates. +That flat 1.219x is a property of the model, not of the device. Anything claiming +a LUN-count dependence needs the reservation order fixed first (event-driven +issue, or a separate command-phase timeline), with these tests extended to cover +it. diff --git a/hw/femu/nand/test/test_channel.c b/hw/femu/nand/test/test_channel.c new file mode 100644 index 0000000000..5fcd3afcb6 --- /dev/null +++ b/hw/femu/nand/test/test_channel.c @@ -0,0 +1,175 @@ +/* Host unit tests for the NAND media timing model's channel stage. + * + * bbssd hardcodes channel_mode = NAND_CH_OFF (ftl-media.c:129), so pg_xfer_lat is + * copied into the config and never read. Before enabling NAND_CH_STAGED we need + * to know (a) that OFF is unchanged, and (b) that STAGED does what the analytical + * channel model assumed. Neither needs QEMU, a guest, or KVM: nand_media_op() is + * pure timing arithmetic over a caller-supplied timeline. + */ +#include +#include +#include +#include +#include +#include "nand-media.h" + +#define QLC 4 +static const int64_t QLC_RD[4] = {47900, 76200, 134600, 228100}; /* measured */ +static const int64_t XFER = 52433; /* QLC_CHNL_PAGE_TRANSFER_LATENCY_NS */ +#define MAXCH 8 +#define MAXLUN 16 + +typedef struct { uint64_t ch[MAXCH]; uint64_t lun[MAXCH][MAXLUN]; uint64_t pl[MAXCH][MAXLUN]; } State; +static State ST; +static uint64_t *t_ch(void *o, uint32_t c) { (void)o; return &ST.ch[c]; } +static uint64_t *t_lun(void *o, const NandLoc *l) { (void)o; return &ST.lun[l->ch][l->lun]; } +static uint64_t *t_pl(void *o, const NandLoc *l) { (void)o; return &ST.pl[l->ch][l->lun]; } +static const NandTimelineOps OPS = { .ch_avail=t_ch, .lun_avail=t_lun, .plane_avail=t_pl }; + +static void setup(NandMedia *m, uint32_t nch, uint32_t luns, NandChannelMode mode, bool bus) +{ + NandMediaConfig c; + memset(&c, 0, sizeof c); + memset(&ST, 0, sizeof ST); + c.nchs = nch; c.luns_per_ch = luns; c.planes_per_lun = 1; + for (int p = 0; p < 4; p++) c.timing.rd_table_ns[QLC][p] = QLC_RD[p]; + c.timing.wr_table_ns[QLC][0] = 1000000; + c.timing.er_table_ns[QLC] = 3000000; + if (bus) c.timing.page_xfer_ns = XFER; /* cmd_addr / status left 0 */ + c.policy.use_flat_timing = false; + c.policy.array_gate = NAND_GATE_LUN_ONLY; + c.policy.channel_mode = mode; + c.timeline = &OPS; c.timeline_opaque = NULL; + nand_media_init(m, &c); +} + +static uint64_t rd(NandMedia *m, uint32_t ch, uint32_t lun, int ptype, uint64_t at) +{ + NandLoc l; memset(&l, 0, sizeof l); + l.ch = ch; l.lun = lun; l.flash_type = QLC; l.page_type = ptype; + return nand_media_op(m, &l, NAND_MEDIA_READ, at).done_ns; +} + +static int fails; +static void ck(int ok, const char *what, const char *detail) +{ + printf(" [%s] %s%s%s\n", ok ? "PASS" : "FAIL", what, + detail && *detail ? " — " : "", detail ? detail : ""); + if (!ok) fails++; +} + +/* ---- 1. OFF must be untouched, and STAGED with a zero bus must equal it ---- */ +static void t_compat(void) +{ + printf("[1] 하위호환: OFF, 그리고 bus=0 인 STAGED\n"); + uint64_t off[64], staged0[64]; + NandMedia m; + setup(&m, 1, 4, NAND_CH_OFF, false); + for (int i = 0; i < 64; i++) off[i] = rd(&m, 0, i % 4, i % 4, 0); + setup(&m, 1, 4, NAND_CH_STAGED, false); + for (int i = 0; i < 64; i++) staged0[i] = rd(&m, 0, i % 4, i % 4, 0); + /* Documents the opposite of what one would hope: enabling the channel stage + * changes timing even with every bus phase at zero, because the channel + * timeline is advanced to each op's data-out and the next command is clamped + * to it. This is why a channel_model option has to default to off. */ + ck(memcmp(off, staged0, sizeof off) != 0, + "STAGED 는 bus=0 이어도 OFF 와 다름", + "channel_model 옵션은 반드시 off 를 기본값으로 해야 함"); + + /* OFF must ignore the bus entirely, even when pg_xfer_lat is set */ + uint64_t offbus[64]; + setup(&m, 1, 4, NAND_CH_OFF, true); + for (int i = 0; i < 64; i++) offbus[i] = rd(&m, 0, i % 4, i % 4, 0); + ck(!memcmp(off, offbus, sizeof off), "OFF 는 pg_xfer_lat 를 무시", + "오늘의 동작 — 값을 줘도 타이밍에 반영 안 됨"); +} + +/* ---- 2. one LUN: array read then data-out, serialised ---- */ +static void t_single(void) +{ + printf("[2] LUN 1개: array read 후 data-out 직렬화\n"); + NandMedia m; char b[160]; + setup(&m, 1, 1, NAND_CH_STAGED, true); + uint64_t d = rd(&m, 0, 0, 0, 0); + snprintf(b, sizeof b, "관측 %.1f us = array %.1f + xfer %.1f", + d/1000.0, QLC_RD[0]/1000.0, XFER/1000.0); + ck(d == (uint64_t)(QLC_RD[0] + XFER), "1회 읽기 = array + xfer", b); + + setup(&m, 1, 1, NAND_CH_STAGED, true); + uint64_t a = rd(&m, 0, 0, 0, 0), c = rd(&m, 0, 0, 0, 0); + snprintf(b, sizeof b, "1번째 %.1f us, 2번째 %.1f us", a/1000.0, c/1000.0); + ck(c >= a + QLC_RD[0], "같은 LUN 연속 읽기는 array 시간만큼 직렬화", b); +} + +/* ---- 3. two LUNs on one channel: sensing overlaps, data-out does not ---- */ +static void t_overlap(void) +{ + printf("[3] 한 채널의 LUN 2개: sensing 겹침, data-out 직렬화\n"); + NandMedia m; char b[160]; + setup(&m, 1, 2, NAND_CH_STAGED, true); + uint64_t a = rd(&m, 0, 0, 3, 0); /* slow page on LUN0 */ + uint64_t c = rd(&m, 0, 1, 3, 0); /* slow page on LUN1, same instant */ + snprintf(b, sizeof b, "LUN0 %.1f us, LUN1 %.1f us (직렬이면 %.1f)", + a/1000.0, c/1000.0, (QLC_RD[3]*2 + XFER*2)/1000.0); + /* Physical NAND would overlap here. This model does not: reservations are made + * in submission order, so LUN1's command waits for LUN0's data-out. */ + ck(c == (uint64_t)(QLC_RD[3] * 2 + XFER * 2), + "두 LUN 이 완전히 직렬화됨 (실제 NAND 와 다름)", b); + snprintf(b, sizeof b, "두 완료 간격 %.1f us, xfer %.1f us", (c-a)/1000.0, XFER/1000.0); + ck(c - a >= (uint64_t)XFER, "data-out 은 채널에서 직렬화", b); +} + +/* ---- 4. saturation: does the channel erase the page-type advantage? ---- */ +static void t_saturation(void) +{ + printf("[4] 포화: LUN/채널 수에 따라 page mapping 이득이 남는가\n"); + /* traffic shares by plane index, 2-tier unified-lru @1.8GB */ + const double w[4] = {0.381, 0.381, 0.119, 0.119}; + const int N = 4000; + /* The analytical channel model predicted 1.346 / 1.161 / 1.000 / 1.000 as LUNs + * per channel grow, assuming extra LUNs overlap sensing with another LUN's + * data burst. This model never overlaps them, so the gain is flat at + * (mean_array + xfer) / (aware_array + xfer). */ + const double pred[] = {1.219, 1.219, 1.219, 1.219}; + printf(" %-8s %10s %10s %8s %s\n", "LUN/ch", "oblivious", "aware", "gain", "모델 예상"); + int pi = 0; + for (uint32_t luns = 1; luns <= 8; luns *= 2, pi++) { + uint64_t mk[2]; + for (int arm = 0; arm < 2; arm++) { + NandMedia m; setup(&m, 1, luns, NAND_CH_STAGED, true); + uint64_t last = 0; int k = 0; + for (int i = 0; i < N; i++) { + /* pick a plane index by traffic share */ + double u = (double)(i % 1000) / 1000.0, acc = 0; int plane = 3; + for (int j = 0; j < 4; j++) { acc += w[j]; if (u < acc) { plane = j; break; } } + /* oblivious: traffic spread evenly over the four page classes. + aware: plane index maps one-to-one onto page class. */ + int ptype = arm == 0 ? (k++ % 4) : plane; + uint64_t d = rd(&m, 0, i % luns, ptype, 0); + if (d > last) last = d; + } + mk[arm] = last; + } + double gain = (double)mk[0] / (double)mk[1]; + char note[64]; + snprintf(note, sizeof note, "%.3fx", pred[pi]); + printf(" %-8u %9.2fms %9.2fms %7.3fx %s\n", + luns, mk[0]/1e6, mk[1]/1e6, gain, note); + if (fabs(gain - pred[pi]) > 0.05) { + printf(" ^ 예측에서 벗어남\n"); fails++; + } + } +} + +int main(void) +{ + printf("NAND media 채널 스테이지 단위 테스트 (QEMU/게스트/KVM 불필요)\n"); + printf("QLC read %.1f/%.1f/%.1f/%.1f us, channel page xfer %.2f us\n\n", + QLC_RD[0]/1000.0, QLC_RD[1]/1000.0, QLC_RD[2]/1000.0, QLC_RD[3]/1000.0, XFER/1000.0); + t_compat(); printf("\n"); + t_single(); printf("\n"); + t_overlap(); printf("\n"); + t_saturation(); printf("\n"); + printf(fails ? "실패 %d 건\n" : "전부 통과\n", fails); + return fails ? 1 : 0; +} diff --git a/hw/femu/nand/test/test_pairing.c b/hw/femu/nand/test/test_pairing.c new file mode 100644 index 0000000000..ab499bd64d --- /dev/null +++ b/hw/femu/nand/test/test_pairing.c @@ -0,0 +1,83 @@ +/* Host unit test for the QLC page-pairing table. + * + * The expert plane-major layout assigns bit-plane Bn to QLC class n-1, so every + * predicted address depends on qlc_tbl matching the device exactly. A mismatch + * is silent: the mapper still emits a plan, the device still serves the reads, + * and only the latency is wrong. Checking the table before booting is far + * cheaper than reading it back out of a FEMU WRITE log. + * + * init_qlc_page_pairing() is static and its translation unit pulls in QEMU, so + * the function text is extracted from nand.c at build time (see the Makefile + * rule) rather than copied here, which would let the two drift apart. + */ +#include +#include +#include +#include + +typedef struct FemuCtrl FemuCtrl; +#include "../nand.h" + +int slc_tbl[MAX_SUPPORTED_PAGES_PER_BLOCK]; +int mlc_tbl[MAX_SUPPORTED_PAGES_PER_BLOCK]; +int tlc_tbl[MAX_SUPPORTED_PAGES_PER_BLOCK]; +int qlc_tbl[MAX_SUPPORTED_PAGES_PER_BLOCK]; +struct NandFlashTiming nand_flash_timing; + +/* Upstream's own style: size_t/int comparisons and the unused FemuCtrl argument. + * Our test code stays under -Wall -Wextra -Werror; the extracted text does not. */ +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wsign-compare" +#pragma GCC diagnostic ignored "-Wunused-parameter" +#include "pairing_extract.inc" +#pragma GCC diagnostic pop + +static int fails; + +static void check(int cond, const char *what) +{ + if (!cond) { printf("FAIL %s\n", what); fails++; } +} + +int main(void) +{ + int pg, counts[4] = {0}; + + memset(qlc_tbl, -1, sizeof(qlc_tbl)); + init_qlc_page_pairing(NULL); + + /* Prologue: the shadow-programming sequence leaves pages 0..7 special. */ + for (pg = 0; pg < 6; pg++) + check(qlc_tbl[pg] == QLC_LOWER_PAGE, "prologue pg 0..5 is class 0"); + for (pg = 6; pg < 8; pg++) + check(qlc_tbl[pg] == QLC_LOWER_CENTER_PAGE, "prologue pg 6..7 is class 1"); + + /* From page 8 the cycle is 0 0 1 1 2 2 3 3, to the last page of the block. */ + for (pg = 8; pg < MAX_SUPPORTED_PAGES_PER_BLOCK; pg++) { + int want = ((pg - 8) % 8) / 2; + if (qlc_tbl[pg] != want) { + printf("FAIL pg %d: class %d, expected %d\n", pg, qlc_tbl[pg], want); + fails++; + break; + } + } + + /* No page may keep the -1 poison: rows-3 used to leave 496..511 untouched, + * where the zero-initialised global reads as a valid QLC_LOWER_PAGE. */ + for (pg = 0; pg < MAX_SUPPORTED_PAGES_PER_BLOCK; pg++) { + if (qlc_tbl[pg] < 0 || qlc_tbl[pg] > 3) { + printf("FAIL pg %d never assigned (%d)\n", pg, qlc_tbl[pg]); + fails++; + break; + } + counts[qlc_tbl[pg]]++; + } + + printf("class page counts: %d %d %d %d\n", + counts[0], counts[1], counts[2], counts[3]); + check(counts[0] == 132 && counts[1] == 128 && + counts[2] == 126 && counts[3] == 126, "class counts for 512 pages"); + + printf(fails ? "test_pairing: %d failure(s)\n" : "test_pairing: ok\n", fails); + return fails != 0; +} diff --git a/hw/femu/nvme.h b/hw/femu/nvme.h index 021efe77d2..a903beacb1 100644 --- a/hw/femu/nvme.h +++ b/hw/femu/nvme.h @@ -6,6 +6,7 @@ #include "qemu/units.h" #include "qemu/cutils.h" #include "qemu/memalign.h" +#include "qemu/notify.h" #include "hw/pci/msix.h" #include "hw/pci/msi.h" #include "hw/virtio/vhost.h" @@ -1536,6 +1537,7 @@ typedef struct FemuExtCtrlOps { void *state; void (*init)(struct FemuCtrl *, NvmeNamespace *, Error **); void (*exit)(struct FemuCtrl *); + void (*stats_flush)(struct FemuCtrl *); uint16_t (*rw_check_req)(struct FemuCtrl *, NvmeCmd *, NvmeRequest *); int (*start_ctrl)(struct FemuCtrl *); uint16_t (*admin_cmd)(struct FemuCtrl *, NvmeCmd *); @@ -1848,6 +1850,8 @@ typedef struct FemuCtrl { /* Coperd: OC2.0 FIXME */ NvmeParams params; FemuExtCtrlOps ext_ops; + Notifier process_exit_notifier; + bool process_exit_notifier_registered; /* * Controller-wide CSD state. Kept out of ext_ops.state because that slot is * shared with whatever other mode a namespace runs: CSD claimed it @@ -1956,6 +1960,27 @@ typedef struct FemuCtrl { uint32_t read_cache_mb; /* bbssd DRAM read cache size (0 = off) */ uint32_t mapping_cache_mb; /* bbssd DFTL translation cache size (0 = off) */ uint8_t nand_cell_type; /* bbssd NAND cell type: 0=off(flat), 1 SLC..4 QLC */ + + /* + * Read-energy coefficients for the QLC page-class accounting, in + * milli-pJ/bit (34000 = 34.0 pJ/bit). Integers because QEMU device + * properties carry no floating point. Defaults are the cited profile: + * MCFlash energy density 0.709 pJ/bit/us x PACA measured tR. + * The emitted CSV records the coefficients actually used, so the + * energy columns stay auditable and reproducible offline. + */ + uint32_t e_read_mpj[4]; + /* + * The array half of the read coefficient: base sensing plus (n-1) extra + * senses for a class that needs n of them. The rest of e_read_mpj is + * peripheral, which scales with the class read latency. Splitting the two + * is what shows that the peripheral term dominates -- reporting only the + * total leaves that as an offline assertion instead of a measurement. + */ + uint32_t e_array_mpj[4]; + uint32_t e_xfer_mpj; /* channel transfer, milli-pJ/bit */ + uint32_t stats_flush_ms; /* periodic stats snapshot; 0 = on exit only */ + QEMUTimer *stats_timer; /* program/erase cycles the media is rated for; 0 takes the cell type's */ uint32_t pe_cycles_rated; uint32_t nand_bad_blocks; /* bbssd factory bad blocks reported via SMART; 0 = none */ diff --git a/hw/femu/scripts/femu-docker.sh b/hw/femu/scripts/femu-docker.sh new file mode 100755 index 0000000000..b2c78e6b6d --- /dev/null +++ b/hw/femu/scripts/femu-docker.sh @@ -0,0 +1,153 @@ +#!/usr/bin/env bash +# The README's build-and-run path, done in a container. +# +# The README tells you to install dependencies with sudo, compile into +# build-femu/, and launch with run-blackbox.sh. That path does not run on the +# host this fork is measured on: there is no passwordless sudo for pkgdep.sh, +# and FEMU pins its memory backend, which needs RLIMIT_MEMLOCK raised past the +# device size -- this host allows 64 MiB against 64 GiB. +# +# Inside the container neither of those bites: it runs as root and compose +# gives it IPC_LOCK with an unlimited memlock. run-blackbox.sh still is not +# what runs there, for an unrelated reason -- it writes the SSD layout into +# itself (pgs_per_blk=256, luns_per_ch=8, nchs=8, ssd_size=12288, a fixed +# u20s.qcow2) and reads no environment, so a run cannot be given this fork's +# geometry, its guest disk, its payload disk or its counter path. femu-run is +# that script with those constants lifted into environment variables. +# +# The steps themselves are unchanged: +# +# README here +# sudo ./pkgdep.sh docker/Dockerfile, builder stage +# ./femu-compile.sh docker/Dockerfile, builder stage +# ./run-blackbox.sh docker/femu-run bbssd, via compose +# +# The device is the same either way. Geometry, cell type and the energy +# coefficients come from compose.yaml, whose defaults are the configuration the +# measurements in this repository were taken on. +set -uo pipefail + +HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +REPO=$(cd "$HERE/.." && pwd) + +# Where guest disks live. The README has you build a VM image by hand and keep +# it beside the build; this is that directory. +IMAGES=${FEMU_GUEST_DIR:-$HOME/images} +# What the container sees as /data: the counter CSV and the console log land here. +DATA=${FEMU_DATA_DIR:-$REPO/docker-data} +BASE=${FEMU_BASE_IMAGE:-jammy-server-cloudimg-amd64.img} +TAG=${FEMU_INSTANCE:-femu} +SSH_PORT=${FEMU_SSH_PORT:-2222} +SSH_PUBKEY=${FEMU_SSH_PUBKEY:-$HOME/.ssh/id_rsa.pub} + +compose() { + FEMU_GUEST_DIR=$IMAGES FEMU_DATA_DIR=$DATA \ + FEMU_CONTAINER_NAME=femu-$TAG FEMU_SSH_PORT=$SSH_PORT \ + FEMU_IMAGE_NAME=femu-root-$TAG.qcow2 \ + FEMU_QLC_STATS_PATH=/data/${TAG}_qlc.csv \ + FEMU_EXTRA_DRIVES="${FEMU_EXTRA_DRIVES:-file=/guest/seed-$TAG.iso,if=virtio,format=raw,readonly=on}" \ + docker compose -f "$REPO/compose.yaml" "$@" +} + +usage() { + cat <<'EOF' +Usage: femu-docker.sh COMMAND + + build Build the FEMU image. Dependencies and compile both happen inside, + so nothing is installed on the host. + verify Ask the built binary whether the femu device registered. + image Create this instance's guest disk and cloud-init seed. The overlay + is copy-on-write over the base image, which is never written. + run Start the device and boot the guest. Ctrl-C detaches; the container + keeps running. + ssh Open a shell in the guest. + stop Stop the container and remove it. + status What is running, and the device this instance would get. + +Environment: + FEMU_INSTANCE name for this instance's disk, seed and container (femu) + FEMU_GUEST_DIR where guest disks live ($HOME/images) + FEMU_DATA_DIR what the container sees as /data (/docker-data) + FEMU_BASE_IMAGE base image the overlay is cut from + (jammy-server-cloudimg-amd64.img) + FEMU_SSH_PORT host port forwarded to the guest's sshd (2222) + FEMU_SSH_PUBKEY key authorised in the guest (~/.ssh/id_rsa.pub) + +Running an actual measurement is a different entry point: it needs a payload +image, a compiled trace and the placement checks. See moe-harness/README.md. +EOF +} + +case "${1:-help}" in +help|-h|--help) usage ;; + +build) + echo "== building (dependencies and compile are inside the image) ==" + compose build femu + ;; + +verify) + echo "== does the femu device register? ==" + compose run --rm --entrypoint qemu-system-x86_64 femu -device femu,help 2>&1 | + head -20 + ;; + +image) + mkdir -p "$IMAGES" "$DATA" + [ -f "$IMAGES/$BASE" ] || { + echo "no base image at $IMAGES/$BASE" + echo "download an Ubuntu 22.04 cloud image there, or set FEMU_BASE_IMAGE" + exit 1 + } + [ -f "$SSH_PUBKEY" ] || { echo "no ssh public key at $SSH_PUBKEY"; exit 1; } + ovl=$IMAGES/femu-root-$TAG.qcow2 + seed=$IMAGES/seed-$TAG.iso + # Never write the base image: a run gets its own overlay, so a broken guest + # is one file to delete rather than a re-download. + [ -e "$ovl" ] && { echo "already exists: $ovl (delete it to start over)"; exit 1; } + qemu-img create -f qcow2 -F qcow2 -b "$BASE" "$ovl" 32G >/dev/null + echo " overlay $ovl" + # The cloud image ships no password, so without a seed carrying a key there + # is no way in. No --run: this seed only authorises the key. + for c in python3 /usr/bin/python3 python3.8; do + command -v "$c" >/dev/null 2>&1 || continue + "$c" -c 'import pycdlib' 2>/dev/null && { PY=$c; break; } + done + [ -n "${PY:-}" ] || { echo "no python3 with pycdlib; pip install --user pycdlib"; exit 1; } + (cd "$REPO/moe-harness/exp/gating_nand/femu" && + "$PY" make_seed.py -o "$seed" --tag "${TAG^^}" \ + --instance-id "femu-$TAG" --ssh-key "$SSH_PUBKEY") >/dev/null || exit 1 + echo " seed $seed" + ;; + +run) + [ -f "$IMAGES/femu-root-$TAG.qcow2" ] || { + echo "no guest disk for instance '$TAG'; run: $0 image"; exit 1; } + mkdir -p "$DATA" + echo "== starting femu-$TAG (Ctrl-C detaches, container keeps running) ==" + compose up femu + ;; + +ssh) + exec ssh -p "$SSH_PORT" -o StrictHostKeyChecking=no \ + -o UserKnownHostsFile=/dev/null femu@127.0.0.1 "${@:2}" + ;; + +stop) + compose down + ;; + +status) + echo "== containers ==" + docker ps --filter "name=femu-$TAG" --format ' {{.Names}} {{.Status}}' || true + echo "== the device this instance gets ==" + compose run --rm --entrypoint sh femu -c 'echo " \ +cell=$FEMU_NAND_CELL_TYPE size=${FEMU_SSD_SIZE_MB}MB \ +${FEMU_CHANNELS}ch x ${FEMU_LUNS_PER_CHANNEL}LUN \ +${FEMU_PAGES_PER_BLOCK}pg/blk ${FEMU_BLOCKS_PER_PLANE}blk/pl \ +opts=$FEMU_EXTRA_DEVICE_OPTS"' 2>/dev/null | tail -1 + ;; + +*) + echo "unknown command: $1"; echo; usage; exit 1 ;; +esac diff --git a/hw/femu/scripts/run-blackbox.sh b/hw/femu/scripts/run-blackbox.sh index c6184fe939..741a828cbb 100755 --- a/hw/femu/scripts/run-blackbox.sh +++ b/hw/femu/scripts/run-blackbox.sh @@ -1,31 +1,46 @@ #!/bin/bash # Huaicheng Li # Run FEMU as a black-box SSD (FTL managed by the device) +# +# Every setting below can be overridden from the environment, and each default +# is the value this script used when they were constants. Set nothing and the +# command line is what it always was. +# +# The reason for the change: the layout was written into the file, so running +# two devices, or running one under a harness, meant editing the script or +# keeping a forked copy per configuration. The names match the ones the +# container path uses, so a run is described the same way whichever way it is +# started. # image directory -IMGDIR=$HOME/images +IMGDIR=${FEMU_GUEST_DIR:-$HOME/images} # Virtual machine disk image -OSIMGF=$IMGDIR/u20s.qcow2 +OSIMGF=${FEMU_IMAGE:-$IMGDIR/${FEMU_IMAGE_NAME:-u20s.qcow2}} # Configurable SSD Controller layout parameters (must be power of 2) -secsz=512 # sector size in bytes -secs_per_pg=8 # number of sectors in a flash page -pgs_per_blk=256 # number of pages per flash block -blks_per_pl=256 # number of blocks per plane -pls_per_lun=1 # planes per LUN -luns_per_ch=8 # number of chips per channel -nchs=8 # number of channels -ssd_size=12288 # in megabytes, if you change the above layout parameters, make sure you manually recalculate the ssd size and modify it here, please consider a default 25% overprovisioning ratio. +secsz=${FEMU_SECTOR_SIZE:-512} # sector size in bytes +secs_per_pg=${FEMU_SECTORS_PER_PAGE:-8} # number of sectors in a flash page +pgs_per_blk=${FEMU_PAGES_PER_BLOCK:-256} # number of pages per flash block +blks_per_pl=${FEMU_BLOCKS_PER_PLANE:-256} # number of blocks per plane +pls_per_lun=${FEMU_PLANES_PER_LUN:-1} # planes per LUN +luns_per_ch=${FEMU_LUNS_PER_CHANNEL:-8} # number of chips per channel +nchs=${FEMU_CHANNELS:-8} # number of channels +ssd_size=${FEMU_SSD_SIZE_MB:-12288} # in megabytes, if you change the above layout parameters, make sure you manually recalculate the ssd size and modify it here, please consider a default 25% overprovisioning ratio. # Latency in nanoseconds -pg_rd_lat=40000 # page read latency -pg_wr_lat=200000 # page write latency -blk_er_lat=2000000 # block erase latency -ch_xfer_lat=0 # channel transfer time, ignored for now +pg_rd_lat=${FEMU_PAGE_READ_LATENCY:-40000} # page read latency +pg_wr_lat=${FEMU_PAGE_WRITE_LATENCY:-200000} # page write latency +blk_er_lat=${FEMU_BLOCK_ERASE_LATENCY:-2000000} # block erase latency +ch_xfer_lat=${FEMU_CHANNEL_TRANSFER_LATENCY:-0} # channel transfer time, ignored for now # GC Threshold (1-100) -gc_thres_pcent=75 -gc_thres_pcent_high=95 +gc_thres_pcent=${FEMU_GC_THRESHOLD:-75} +gc_thres_pcent_high=${FEMU_GC_THRESHOLD_HIGH:-95} + +# Guest resources and the port forwarded to its sshd +vm_cpus=${FEMU_CPUS:-4} +vm_memory=${FEMU_MEMORY:-4G} +ssh_port=${FEMU_GUEST_SSH_PORT:-8080} #----------------------------------------------------------------------- @@ -48,8 +63,38 @@ FEMU_OPTIONS=${FEMU_OPTIONS}",ch_xfer_lat=${ch_xfer_lat}" FEMU_OPTIONS=${FEMU_OPTIONS}",gc_thres_pcent=${gc_thres_pcent}" FEMU_OPTIONS=${FEMU_OPTIONS}",gc_thres_pcent_high=${gc_thres_pcent_high}" +# Appended only when asked for, so an unset environment reproduces the command +# line this script produced before any of this was configurable. The flat +# pg_rd_lat above is what a cell type replaces: set FEMU_NAND_CELL_TYPE and the +# device reads its per-page-class table instead. +[ -n "${FEMU_NAND_CELL_TYPE:-}" ] && + FEMU_OPTIONS=${FEMU_OPTIONS}",nand_cell_type=${FEMU_NAND_CELL_TYPE}" +for c in 0 1 2 3; do + v=FEMU_E_READ_C$c + [ -n "${!v:-}" ] && FEMU_OPTIONS=${FEMU_OPTIONS}",e_read_c${c}_mpj=${!v}" + v=FEMU_E_ARRAY_C$c + [ -n "${!v:-}" ] && FEMU_OPTIONS=${FEMU_OPTIONS}",e_array_c${c}_mpj=${!v}" +done +[ -n "${FEMU_E_XFER:-}" ] && FEMU_OPTIONS=${FEMU_OPTIONS}",e_xfer_mpj=${FEMU_E_XFER}" +[ -n "${FEMU_STATS_FLUSH_MS:-}" ] && + FEMU_OPTIONS=${FEMU_OPTIONS}",stats_flush_ms=${FEMU_STATS_FLUSH_MS}" +# Anything else the device takes, comma separated: op_pcent=7, buffer_size=... +[ -n "${FEMU_EXTRA_DEVICE_OPTS:-}" ] && + FEMU_OPTIONS=${FEMU_OPTIONS}",${FEMU_EXTRA_DEVICE_OPTS}" + echo ${FEMU_OPTIONS} +# Extra read-only guest disks, ';' separated -drive specs. A replay payload is +# far too large for a cloud-init seed, so it arrives as a disk the guest copies +# onto the emulated SSD from inside. +EXTRA_DRIVES=() +if [ -n "${FEMU_EXTRA_DRIVES:-}" ]; then + IFS=';' read -r -a specs <<< "${FEMU_EXTRA_DRIVES}" + for spec in "${specs[@]}"; do + [ -n "$spec" ] && EXTRA_DRIVES+=(-drive "$spec") + done +fi + if [[ ! -e "$OSIMGF" ]]; then echo "" echo "VM disk image couldn't be found ..." @@ -59,20 +104,34 @@ if [[ ! -e "$OSIMGF" ]]; then exit fi -sudo FEMU_EXP_LOG=${FEMU_EXP_LOG} \ +# sudo is how an ordinary user reaches KVM and raises RLIMIT_MEMLOCK for the +# pinned memory backend. In a container the process is already root and sudo is +# often not installed, so asking for it there fails for want of a binary. +SUDO=sudo +[ "$(id -u)" = 0 ] && SUDO= +# Built in build-femu/ by femu-compile.sh, which is where this script is run +# from; a packaged build puts it on PATH instead. +QEMU=${FEMU_QEMU_BIN:-./qemu-system-x86_64} +[ -x "$QEMU" ] || QEMU=$(command -v qemu-system-x86_64) || { + echo "qemu-system-x86_64 not found; build it or set FEMU_QEMU_BIN"; exit 1; } + +$SUDO FEMU_EXP_LOG=${FEMU_EXP_LOG} \ FEMU_SECRET=${FEMU_SECRET} \ FEMU_DUMP_LPN=${FEMU_DUMP_LPN} \ - ./qemu-system-x86_64 \ + FEMU_QLC_STATS_PATH=${FEMU_QLC_STATS_PATH} \ + FEMU_ALLOW_UNPINNED=${FEMU_ALLOW_UNPINNED} \ + "$QEMU" \ -name "FEMU-BBSSD-VM" \ -enable-kvm \ -cpu host \ - -smp 4 \ - -m 4G \ + -smp ${vm_cpus} \ + -m ${vm_memory} \ -device virtio-scsi-pci,id=scsi0 \ -device scsi-hd,drive=hd0 \ -drive file=$OSIMGF,if=none,aio=native,cache=none,format=qcow2,id=hd0 \ + "${EXTRA_DRIVES[@]}" \ ${FEMU_OPTIONS} \ - -net user,hostfwd=tcp::8080-:22 \ + -net user,hostfwd=tcp::${ssh_port}-:22 \ -net nic,model=virtio \ -nographic \ -qmp unix:./qmp-sock,server,nowait 2>&1 | tee log diff --git a/moe-harness/README.md b/moe-harness/README.md new file mode 100644 index 0000000000..6fc61d578a --- /dev/null +++ b/moe-harness/README.md @@ -0,0 +1,104 @@ +# QLC placement measurement harness + +What drives the experiment: it boots a FEMU device from this checkout, fills it +with an image whose byte layout decides which QLC page class every bit-plane +lands on, proves the placement landed with the device's own counters, and +replays a recorded MoE inference trace against it. + +Only the FEMU-host half is here. The GPU host's analysis and plotting +(`compose_e2e.py`, `make_figures.py`, `timed_replay.py`, and the quantiser +tooling) stays there, because it needs the model weights and a GPU. + +## Layout + + exp/moe_bcq/femu_run/ run the experiment + run_device.sh one device: boot, fill, then N replays + drive_multi.sh confusion check once, then a replay per trace + run_policy.sh older path: one device, one replay + drive_run.sh its driver + run_sweep.sh, preflight.sh sweep wrapper and its guard + guest_replay.sh runs inside the guest: fill and read-back + class_confusion.c reads one class back and counts what the + device says it read -- the placement check + probe_map.c, mark_write.c LPN -> PPA probes, for diagnosing a fill + verify_fill_256.py checks the queue satisfies the fill contract + audit_handoff.py re-derives a binary's totals from the JSONL + + exp/moe_bcq/femu_handoff/packages/ + replay_v1.c the guest replayer, QD=32 O_DIRECT AIO + REPLAYER_V1.md the binary format it consumes + + exp/gating_nand/femu/make_seed.py cloud-init seed carrying the binaries + scripts/femu_compose.sh compose wrapper + scripts/build_replay.sh builds the guest replayer + runs/femu/run01.env per-run environment template + +## Where the rest lives + +Three things are kept apart by who owns them, because the copies that used to +exist on both hosts drifted -- one pair of mapper copies ended up 64 lines +apart, and an edit to run_policy.sh was reverted twice by a sync. + +| | | +|---|---| +| this repository | the harness, `replay_v1.c`, the emulator | +| `MoE_Trace` | the mapper, `bundle.py`, the rest of the trace tooling | +| neither, they are data | payload, layouts, `replay_qd32.bin`, images | + +The split has a consequence worth stating plainly: **building an image needs +`MoE_Trace`.** `qlc_aligned_mapper.py materialize` turns a layout and a payload +into the image this harness fills a device from, and it is not in this +repository. Clone `MoE_Trace` alongside, or have the image built where that +tooling already is and shipped as data. + +Nothing here calls it, so the harness runs without it once an image exists. + +## What it needs that is not here + +The payload (`planes.bin`, `scales.bin`, about 15 GB for the two models), the +collected traces, and the layouts built from them. They are data, not code, and +are distributed separately. A guest image is also needed: an Ubuntu 22.04 cloud +image, with a per-run qcow2 overlay -- 20.04 will not do, because replay_v1 +needs a kernel new enough for the AIO path. + +## Running one device + + bash exp/moe_bcq/femu_run/run_device.sh DEVICE_TAG IMAGE_BASENAME IMAGE_PAGES SPECFILE + +`IMAGE_BASENAME` is an image under `runs/femu/images/`, `IMAGE_PAGES` its page +count from the layout summary, and `SPECFILE` a list of ` ` +lines, one replay each. All of them share the device, which is the point: the +placement is a property of the image, so every trace that shares the placement +has to be replayed on the same fill. Re-filling per trace would re-run +out-of-place allocation and land the pages on different physical classes. + +The device geometry comes from the compose defaults in the parent checkout +(64 GiB, 2 channels x 4 LUNs, 512 pages per block, `op_pcent=7`) and needs no +argument. `runs/femu/run01.env` restates them and adds the per-run pieces: the +QLC counter path, the payload disk, the container name. + +## Two things that will waste a day if you skip them + +**Fill in 256 KiB writes.** This queue's `max_segments` is 127, so a larger +O_DIRECT write splits at 508 KiB, which falls in the middle of the 32nd 16 KiB +flash page. That page then belongs to both fragments and is programmed twice; +out-of-place update spends an extra physical page, and every later page shifts +one slot along. `guest_replay.sh` checks the queue before filling and dies if +the contract does not hold. This is configuration-dependent, which is why the +PPA check below stays in the procedure. + +**A matching read-back hash does not mean the placement is right.** Three early +runs passed SHA-256 read-back and were still wrong: correct data says nothing +about which cell holds it. `drive_multi.sh` therefore reads every class back +across the whole image and asserts the counter diagonal before any replay. A +partial range would miss exactly the pages a drifted fill puts out of place. + +## Checking a result + +Each replay writes `groups.jsonl.gz` (per-group timing, the only record of the +prefill/decode split) and `replay.csv` (the QLC counter snapshot for that +replay) into `records//`. `replay_v1` resets the counters when it +starts and snapshots them when it ends, so consecutive replays on one device +report independently -- verified by comparing every run's per-class page counts +against the counts its own compiled binary implies, which agreed exactly across +24 runs. diff --git a/moe-harness/exp/gating_nand/femu/make_seed.py b/moe-harness/exp/gating_nand/femu/make_seed.py new file mode 100644 index 0000000000..39dd8073ad --- /dev/null +++ b/moe-harness/exp/gating_nand/femu/make_seed.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Build a cloud-init seed ISO that runs a program in the guest and exits. + +FEMU's own scripts expect a VM you SSH into over slirp. This host has no libslirp, +so FEMU is built without it and the guest has no network at all: nothing can be +installed and nothing can be copied in after boot. Everything the run needs -- the +binary, its shared library, the trace -- is embedded here, gzip+base64, and the +results come back over the serial console. + + python3 make_seed.py -o ~/images/seed.iso \ + --file /usr/local/bin/replay=../../../build/guest/replay:0755 \ + --file /usr/local/lib/liburing.so.2=$CONDA/lib/liburing.so.2.14 \ + --file /root/objects.csv= \ + --file /root/stream.csv= \ + --run "LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/replay /dev/nvme0n1 0 /root/objects.csv /root/stream.csv" +""" + +import argparse +import base64 +import gzip +import io +import sys +from pathlib import Path + +import pycdlib + +HEAD = """#cloud-config +password: femu +chpasswd: {{ expire: False }} +ssh_pwauth: true +users: + - name: femu + plain_text_passwd: femu + lock_passwd: false + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash +{keys}write_files: +{files}runcmd: + - [ sh, -c, "echo '==={tag}-START===' > /dev/ttyS0" ] +{cmds} - [ sh, -c, "echo '==={tag}-DONE===' > /dev/ttyS0" ] +""" + + +def parse_args(): + p = argparse.ArgumentParser() + p.add_argument("-o", "--output", type=Path, required=True) + p.add_argument("--file", action="append", default=[], metavar="GUEST=HOST[:MODE]", + help="embed HOST at GUEST, default mode 0644") + p.add_argument("--run", action="append", default=[], + help="shell command; stdout and stderr go to the serial console") + p.add_argument("--ssh-key", type=Path, default=None, + help="public key authorised for the femu user. The serial " + "console is one-shot: it runs what the seed says and " + "nothing more. A key turns each further question into " + "an ssh command instead of another boot and refill.") + p.add_argument("--tag", default="RUN", help="marker wrapping the output") + p.add_argument("--instance-id", default="femu-01") + return p.parse_args() + + +def main() -> int: + args = parse_args() + blocks = [] + for spec in args.file: + guest, _, rest = spec.partition("=") + host, _, mode = rest.partition(":") + data = gzip.compress(Path(host).read_bytes()) + blocks.append( + f" - path: {guest}\n" + f" permissions: '{mode or '0644'}'\n" + f" encoding: gz+b64\n" + f" content: {base64.b64encode(data).decode()}\n") + cmds = "".join(f' - [ sh, -c, "{c} > /dev/ttyS0 2>&1" ]\n' for c in args.run) + keys = "" + if args.ssh_key: + keys = " ssh_authorized_keys:\n - {}\n".format( + args.ssh_key.read_text().strip()) + user = HEAD.format(files="".join(blocks), cmds=cmds, tag=args.tag, + keys=keys).encode() + meta = f"instance-id: {args.instance_id}\nlocal-hostname: femu\n".encode() + + iso = pycdlib.PyCdlib() + iso.new(interchange_level=3, joliet=3, vol_ident="cidata", rock_ridge="1.09") + for data, path, rr, jol in ((user, "/USERDATA.;1", "user-data", "/user-data"), + (meta, "/METADATA.;1", "meta-data", "/meta-data")): + iso.add_fp(io.BytesIO(data), len(data), path, rr_name=rr, joliet_path=jol) + args.output.parent.mkdir(parents=True, exist_ok=True) + iso.write(str(args.output)) + iso.close() + print(f"{args.output} ({args.output.stat().st_size:,} bytes, " + f"{len(args.file)} files, {len(args.run)} commands)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/moe-harness/exp/moe_bcq/femu_handoff/packages/REPLAYER_V1.md b/moe-harness/exp/moe_bcq/femu_handoff/packages/REPLAYER_V1.md new file mode 100644 index 0000000000..d30d542054 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_handoff/packages/REPLAYER_V1.md @@ -0,0 +1,134 @@ +# Binary trace compiler와 FEMU replayer v1 — 2026-09-09 + +`mapped_reads.jsonl`은 사람이 주소와 cache 결과를 감사하는 기준 파일이고, `replay_qd32.bin`은 게스트가 재생하는 입력이다. Qwen JSONL 101.6 MB는 2.14 MB, DeepSeek JSONL 124.4 MB는 2.59 MB의 고정 길이 binary record로 줄었다. 두 형식은 binary header의 `mapped_reads_sha256`으로 묶인다. + +현재 binary는 실제 C 모델에서 수집한 **WikiText-2 pilot generation trace**를 2 GiB LRU와 QLC-aligned mapper에 통과시킨 결과다. 정식 WikiText-2 전체, GSM8K, MMLU trace는 아니다. + +## 포함된 파일 + +| 파일 | 역할 | +|---|---| +| `trace_compiler.py` | 검증된 layout의 mapped JSONL을 binary로 컴파일하거나 binary를 정적으로 검사 | +| `replay_v1.c` | Linux native AIO로 binary trace를 재생하는 게스트 프로그램 | +| `test_trace_replayer.py` | compiler, O_DIRECT, rolling QD, 빈 그룹 barrier를 검사하는 작은 통합 테스트 | +| `{model}/layouts/.../replay_qd32.bin` | QD=32 기본값을 담은 실제 pilot replay 입력 | +| `replay_qd32.bin.json` | 요청 index 표, 원본 hash, record 크기와 총계 | + +replayer는 `linux/aio_abi.h`의 syscall을 직접 사용하므로 `libaio`나 `liburing`에 링크하지 않는다. + +## Binary 형식 + +모든 정수는 little-endian이고 record에는 포인터나 가변 길이 문자열이 없다. + +| record | 크기 | 핵심 내용 | +|---|---:|---| +| header | 136 B | magic/version, sector·alignment, 기본 QD, group/command/byte 총계, extent map과 mapped JSONL SHA256 | +| group | 40 B | group ID, 이전 group dependency, request index, forward/layer/phase, cache reset, command 수 | +| command | 16 B | LBA, sector 수, QLC page class | + +group record 뒤에 그 group의 command record가 바로 온다. command가 0개인 group도 record를 남긴다. 따라서 DRAM hit로 SSD read가 없어진 레이어도 순서에서 사라지지 않는다. 문자열 `request_id`는 sidecar JSON의 `requests[]`가 `request_index`로 복원한다. + +Compiler는 다음을 실패 조건으로 둔다. + +- `extent_map.json`, `mapped_reads.jsonl`의 hash가 layout summary와 다름 +- `layout_validation.json`이 통과 상태가 아님 +- group ID·dependency·command ID가 연속 규칙과 다름 +- 4 KiB O_DIRECT 정렬, 512 B sector, 4 MiB 최대 command 규칙 위반 +- JSONL과 layout summary의 group·command·byte 총계 불일치 + +## 컴파일과 정적 검사 + +전달 폴더 `packages/`에서 실행한다. 제공된 output은 이미 있으므로 재생성할 때는 새 이름을 쓴다. + +```bash +python trace_compiler.py compile \ + qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun \ + --output qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun/replay_check.bin \ + --queue-depth 32 + +python trace_compiler.py inspect \ + qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun/replay_qd32.bin +``` + +게스트에서 replayer를 빌드하고 trace만 먼저 검사한다. + +```bash +cc -O2 -std=c11 -Wall -Wextra -Werror -o replay_v1 replay_v1.c + +./replay_v1 \ + --trace qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun/replay_qd32.bin \ + --dry-run +``` + +Qwen은 1,536 groups, 130,090 commands, 22,608,650,240 requested bytes이고 DeepSeek은 1,664 groups, 157,515 commands, 27,372,060,672 bytes여야 한다. + +## Image 적재와 실제 replay + +먼저 payload와 filler를 합친 image를 생성한다. 이 파일은 패키지에 미리 넣지 않았다. + +```bash +python qlc_aligned_mapper.py materialize qwen_C \ + qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun \ + --image qwen_C.img +``` + +요청한 FEMU geometry와 patched 512-row pairing으로 새 장치를 부팅한다. namespace를 마운트하지 않고 LBA 0부터 image 끝까지 한 번의 순차 stream으로 적재한다. + +```bash +sudo dd if=qwen_C.img of=/dev/nvme0n1 bs=256K \ + iflag=fullblock oflag=direct conv=fsync status=progress +``` + +2026-09-09 실장 검증에서 기존 `bs=4M` fill이 NAND page 중간에서 분할되어 같은 LPN을 두 번 프로그램하고 PPA 순서를 밀어내는 현상이 확인됐다. 따라서 fill 크기를 256 KiB로 수정했다. 관측된 guest는 4 KiB 메모리 page, `max_segments=127`이며 256 KiB는 page-aligned buffer에서 64개 메모리 page다. 이 설정에서 여유를 둔 크기이지 모든 장치에서 무분할을 보장하는 상수는 아니다. `getconf PAGESIZE`와 queue의 `max_segments`, `max_segment_size`, `max_sectors_kb`, `max_hw_sectors_kb`를 기록하고 실제 program 순서를 확인한다. `max_sectors_kb=4096`만으로 무분할을 판정하지 않는다. + +적재 전후에 다른 host write가 없어야 한다. 최종 device command도 NAND page 경계에서 나뉘고 각 LPN이 정확히 한 번씩 program되어야 mapper의 순차 배치 가정이 성립한다. WRITE log의 LPN→PPA class와 image read-back을 각각 확인한 다음 replay한다. byte hash 일치는 page class 일치를 보장하지 않는다. output 파일은 덮어쓰지 않으므로 run마다 새 디렉터리나 이름을 사용한다. + +```bash +sudo ./replay_v1 \ + --trace qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun/replay_qd32.bin \ + --device /dev/nvme0n1 --controller /dev/nvme0 \ + --qd 32 \ + --group-log run01.groups.jsonl \ + --summary run01.summary.json +``` + +기본 동작은 O_DIRECT다. `--buffered`는 작은 개발용 파일에만 쓰는 fallback이다. +`--skip-qlc-counters`는 stock NVMe나 unit test용이며 논문 측정에서는 쓰지 않는다. 기본 +실행은 vendor admin command `0xef/cdw10=8`로 적재 후 QLC counter를 초기화한다. + +각 group은 barrier이므로 phase가 바뀌는 시점에는 직전 phase의 I/O가 모두 완료되어 있다. +replayer는 이 경계와 replay 끝에서 `cdw10=10/11/12`를 보내 직전 physical-counter 차분을 +각각 prefill/decode/teacher-forced bank에 누적한 후, 마지막에 `cdw10=9`로 snapshot한다. +요청마다 `prefill→decode`가 반복돼도 각 bank에 합산된다. + +최종 CSV의 기존 total 열 뒤에는 phase별 `n_read`, `bytes_read`, `e_nand_uj`가 추가된다. +모든 class에서 `total = prefill + decode + teacher_forced`가 성립해야 한다. 명령 중 하나라도 +실패하면 run을 실패 처리한다. plain FEMU가 알 수 없는 selector를 성공으로 돌려줄 수 있으므로 +실행 스크립트에서도 phase 열과 위 closure를 검사해야 한다. + +## 제출과 timestamp 의미 + +그룹 사이에는 strict barrier가 있다. 이전 그룹의 모든 command가 완료된 뒤 다음 그룹으로 이동한다. 그룹 안에서는 outstanding read를 최대 QD까지 채우고, 하나 이상 완료될 때마다 다시 채우는 rolling QD를 쓴다. QD=32는 동시에 완료되는 수가 아니라 host가 아직 completion을 받지 않은 command의 상한이다. + +`group-log`의 시간은 `CLOCK_MONOTONIC_RAW`이며 replay 시작을 0으로 둔다. + +| 필드 | 의미 | +|---|---| +| `group_ready_ns` | command record를 읽어 준비했고 이전 barrier도 끝난 시점 | +| `first_submit_ns` | 첫 `io_submit` 호출 직전 | +| `last_submit_ns` | 마지막 `io_submit`가 반환된 직후 | +| `last_complete_ns` | 마지막 completion을 `io_getevents`에서 관찰한 시점 | +| `group_io_ns` | 첫 submit 직전부터 마지막 completion 관찰까지 | +| `peak_outstanding` | 실제 host-side 최대 outstanding command 수 | + +빈 그룹은 submit 시점이 `null`이고 `group_io_ns=0`이다. `summary`의 `sum_group_io_ns`를 SSD service 구간 합으로 사용한다. `wall_ns_including_log_overhead`에는 binary 해석과 group JSON 기록 비용도 들어간다. + +`--io-log`는 command별 submit과 completion 관찰 시점을 남기는 진단 옵션이다. 한 번의 `io_getevents`로 여러 completion을 받으면 같은 관찰 timestamp가 기록되며 실제 device completion 순간과는 다를 수 있다. 이 로그 자체도 timing을 교란하므로 주 측정에는 group log만 사용한다. + +group log는 제공된 pilot 전체가 메모리 buffer에 머물도록 4 MiB buffering한 뒤 replay 후 flush한다. 그래도 결과 파일은 측정 대상 namespace가 아닌 root disk나 `/dev/shm`에 둔다. target namespace에 로그를 쓰면 순차 적재 계약과 QLC counter가 모두 오염된다. + +## 검증 범위 + +로컬 fixture에서 C11 `-Werror` 빌드, binary dry-run, 실제 O_DIRECT native-AIO read, QD=2 rolling 제출, 빈 그룹 barrier, 총 command/byte 보존을 검사했다. 실제 Qwen과 DeepSeek binary도 Python inspector와 C preflight를 모두 통과했다. + +FEMU에서는 먼저 class가 알려진 작은 LBA 집합을 fio와 replayer에서 QD=1/4/32로 각각 읽어 bytes, command 수, group makespan, QLC counter가 맞는지 교차 확인해야 한다. 현재 패키지에는 **FEMU에서 얻은 latency 결과가 아직 없다.** diff --git a/moe-harness/exp/moe_bcq/femu_handoff/packages/replay_v1.c b/moe-harness/exp/moe_bcq/femu_handoff/packages/replay_v1.c new file mode 100644 index 0000000000..d974efdb24 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_handoff/packages/replay_v1.c @@ -0,0 +1,467 @@ +/* Replay a compiled routed-MoE trace against a preconditioned FEMU namespace. + * + * Groups are strict barriers. Within a group, variable-size O_DIRECT reads are + * kept at a rolling queue depth. The program does not fill the device: use the + * QLC layout materializer and a fresh sequential fill before running it. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define VERSION 1u +#define FEMU_ADM_FLIP 0xef +#define FEMU_RESET_QLC 8 +#define FEMU_SNAP_QLC 9 +#define FEMU_ACCUM_QLC_PREFILL 10 +#define FEMU_ACCUM_QLC_DECODE 11 +#define FEMU_ACCUM_QLC_TEACHER_FORCED 12 + +static const unsigned char MAGIC[8] = {'M','B','Q','R','P','L','1','\0'}; + +struct __attribute__((packed)) trace_header { + unsigned char magic[8]; + uint32_t version, header_bytes, sector_bytes, direct_alignment; + uint32_t default_qd, max_command_bytes, group_record_bytes, command_record_bytes; + uint64_t group_count, command_count, total_bytes, max_end_byte; + unsigned char extent_map_sha256[32], mapped_reads_sha256[32]; +}; + +struct __attribute__((packed)) group_record { + uint64_t group_id; + int64_t release_after_group_id; + uint64_t forward_id; + int32_t layer; + uint8_t phase, cache_reset; + uint16_t reserved; + uint32_t command_count, request_index; +}; + +struct __attribute__((packed)) command_record { + uint64_t lba_start; + uint32_t sector_count; + uint8_t page_class; + uint8_t reserved[3]; +}; + +_Static_assert(sizeof(struct trace_header) == 136, "trace header size"); +_Static_assert(sizeof(struct group_record) == 40, "group record size"); +_Static_assert(sizeof(struct command_record) == 16, "command record size"); + +struct slot { + struct iocb cb; + void *buffer; + struct command_record command; + uint32_t local_index; + uint64_t submit_ns; + bool active; +}; + +static void usage(const char *program) +{ + fprintf(stderr, + "usage:\n" + " %s --trace TRACE --dry-run\n" + " %s --trace TRACE --device /dev/nvme0n1 --controller /dev/nvme0\n" + " --group-log GROUPS.jsonl --summary RUN.json [--io-log IOS.jsonl]\n" + " [--qd 32] [--skip-qlc-counters] [--buffered]\n", + program, program); +} + +static void fail(const char *message) +{ + if (errno) fprintf(stderr, "fatal: %s: %s\n", message, strerror(errno)); + else fprintf(stderr, "fatal: %s\n", message); + exit(1); +} + +static void read_exact(FILE *stream, void *buffer, size_t size, const char *what) +{ + if (fread(buffer, 1, size, stream) != size) { + errno = 0; + fprintf(stderr, "fatal: truncated %s\n", what); + exit(1); + } +} + +static uint64_t now_ns(void) +{ + struct timespec value; + if (clock_gettime(CLOCK_MONOTONIC_RAW, &value)) fail("clock_gettime"); + return (uint64_t)value.tv_sec * 1000000000ull + (uint64_t)value.tv_nsec; +} + +static const char *phase_name(uint8_t phase) +{ + static const char *names[] = {"prefill", "decode", "teacher_forced"}; + return phase < 3 ? names[phase] : "invalid"; +} + +static int qlc_accum_command(uint8_t phase) +{ + static const int commands[] = { + FEMU_ACCUM_QLC_PREFILL, + FEMU_ACCUM_QLC_DECODE, + FEMU_ACCUM_QLC_TEACHER_FORCED, + }; + if (phase >= sizeof commands / sizeof commands[0]) { + errno = 0; + fail("invalid phase for QLC accounting"); + } + return commands[phase]; +} + +static FILE *open_exclusive(const char *path) +{ + int fd = open(path, O_WRONLY | O_CREAT | O_EXCL, 0644); + if (fd < 0) fail(path); + FILE *stream = fdopen(fd, "w"); + if (!stream) fail("fdopen"); + return stream; +} + +static int aio_setup_sys(unsigned entries, aio_context_t *context) +{ + return (int)syscall(__NR_io_setup, entries, context); +} +static int aio_submit_sys(aio_context_t context, long nr, struct iocb **iocbs) +{ + return (int)syscall(__NR_io_submit, context, nr, iocbs); +} +static int aio_getevents_sys(aio_context_t context, long min_nr, long nr, + struct io_event *events) +{ + return (int)syscall(__NR_io_getevents, context, min_nr, nr, events, NULL); +} +static int aio_destroy_sys(aio_context_t context) +{ + return (int)syscall(__NR_io_destroy, context); +} + +static void hash_hex(const unsigned char hash[32], char output[65]) +{ + static const char digits[] = "0123456789abcdef"; + for (int i = 0; i < 32; i++) { + output[2*i] = digits[hash[i] >> 4]; + output[2*i+1] = digits[hash[i] & 15]; + } + output[64] = 0; +} + +static int femu_flip(const char *controller, uint32_t selector) +{ + int fd = open(controller, O_RDONLY); + if (fd < 0) return -1; + struct nvme_admin_cmd command; + memset(&command, 0, sizeof command); + command.opcode = FEMU_ADM_FLIP; + command.cdw10 = selector; + int result = ioctl(fd, NVME_IOCTL_ADMIN_CMD, &command); + int saved = errno; + close(fd); + errno = saved; + return result; +} + +static struct trace_header read_header(FILE *trace) +{ + struct trace_header header; + read_exact(trace, &header, sizeof header, "trace header"); + uint16_t endian = 1; + if (*(unsigned char *)&endian != 1) { errno = 0; fail("little-endian host required"); } + if (memcmp(header.magic, MAGIC, 8) || header.version != VERSION + || header.header_bytes != sizeof header + || header.group_record_bytes != sizeof(struct group_record) + || header.command_record_bytes != sizeof(struct command_record) + || header.sector_bytes != 512 || header.direct_alignment < 512 + || !header.default_qd || !header.max_command_bytes) { + errno = 0; fail("binary trace header mismatch"); + } + return header; +} + +static uint32_t preflight(FILE *trace, const struct trace_header *header) +{ + uint64_t commands = 0, bytes = 0, max_end = 0; + int64_t previous = -1; + uint32_t max_group = 0; + for (uint64_t expected = 0; expected < header->group_count; expected++) { + struct group_record group; + read_exact(trace, &group, sizeof group, "group record"); + if (group.group_id != expected || group.phase > 2 || group.reserved + || (group.cache_reset && group.release_after_group_id != -1) + || (!group.cache_reset && group.release_after_group_id != previous)) { + errno = 0; fail("group ordering/dependency mismatch"); + } + if (group.command_count > max_group) max_group = group.command_count; + for (uint32_t index = 0; index < group.command_count; index++) { + struct command_record command; + read_exact(trace, &command, sizeof command, "command record"); + if (command.lba_start > UINT64_MAX / header->sector_bytes) { + errno = 0; fail("command offset overflow"); + } + uint64_t size = (uint64_t)command.sector_count * header->sector_bytes; + uint64_t offset = command.lba_start * header->sector_bytes; + if (!command.sector_count || command.page_class > 3 + || size > header->max_command_bytes + || offset % header->direct_alignment || size % header->direct_alignment + || command.reserved[0] || command.reserved[1] || command.reserved[2]) { + errno = 0; fail("invalid command record"); + } + if (offset > UINT64_MAX - size || bytes > UINT64_MAX - size) { + errno = 0; fail("command total overflow"); + } + commands++; + bytes += size; + if (offset + size > max_end) max_end = offset + size; + } + previous = (int64_t)group.group_id; + } + if (fgetc(trace) != EOF || commands != header->command_count + || bytes != header->total_bytes || max_end != header->max_end_byte) { + errno = 0; fail("trace length/totals mismatch"); + } + return max_group; +} + +static int free_slot(struct slot *slots, uint32_t qd) +{ + for (uint32_t i = 0; i < qd; i++) if (!slots[i].active) return (int)i; + return -1; +} + +int main(int argc, char **argv) +{ + const char *trace_path = NULL, *device = NULL, *controller = NULL; + const char *group_path = NULL, *io_path = NULL, *summary_path = NULL; + uint32_t qd_override = 0; + bool dry_run = false, skip_counters = false, buffered = false; + for (int i = 1; i < argc; i++) { + if (!strcmp(argv[i], "--trace") && ++i < argc) trace_path = argv[i]; + else if (!strcmp(argv[i], "--device") && ++i < argc) device = argv[i]; + else if (!strcmp(argv[i], "--controller") && ++i < argc) controller = argv[i]; + else if (!strcmp(argv[i], "--group-log") && ++i < argc) group_path = argv[i]; + else if (!strcmp(argv[i], "--io-log") && ++i < argc) io_path = argv[i]; + else if (!strcmp(argv[i], "--summary") && ++i < argc) summary_path = argv[i]; + else if (!strcmp(argv[i], "--qd") && ++i < argc) qd_override = (uint32_t)strtoul(argv[i], NULL, 10); + else if (!strcmp(argv[i], "--dry-run")) dry_run = true; + else if (!strcmp(argv[i], "--skip-qlc-counters")) skip_counters = true; + else if (!strcmp(argv[i], "--buffered")) buffered = true; + else { usage(argv[0]); return 2; } + } + if (!trace_path || (!dry_run && (!device || !group_path || !summary_path + || (!skip_counters && !controller)))) { + usage(argv[0]); return 2; + } + FILE *trace = fopen(trace_path, "rb"); + if (!trace) fail(trace_path); + struct trace_header header = read_header(trace); + uint32_t max_group = preflight(trace, &header); + uint32_t qd = qd_override ? qd_override : header.default_qd; + if (!qd || qd > 4096) { errno = 0; fail("queue depth must be in [1, 4096]"); } + char layout_hash[65], mapped_hash[65]; + hash_hex(header.extent_map_sha256, layout_hash); + hash_hex(header.mapped_reads_sha256, mapped_hash); + if (dry_run) { + printf("trace valid: groups=%" PRIu64 " commands=%" PRIu64 + " bytes=%" PRIu64 " max_group=%u qd=%u max_end=%" PRIu64 "\n", + header.group_count, header.command_count, header.total_bytes, + max_group, qd, header.max_end_byte); + fclose(trace); return 0; + } + + int flags = O_RDONLY | (buffered ? 0 : O_DIRECT); + int device_fd = open(device, flags); + if (device_fd < 0) fail(device); + uint64_t device_bytes = 0; + if (ioctl(device_fd, BLKGETSIZE64, &device_bytes)) { + struct stat st; + if (fstat(device_fd, &st) || !S_ISREG(st.st_mode)) fail("BLKGETSIZE64/fstat"); + device_bytes = (uint64_t)st.st_size; + } + if (header.max_end_byte > device_bytes) { errno = 0; fail("trace exceeds device size"); } + + struct slot *slots = calloc(qd, sizeof *slots); + struct io_event *events = calloc(qd, sizeof *events); + if (!slots || !events) fail("allocate queue state"); + for (uint32_t i = 0; i < qd; i++) { + int rc = posix_memalign(&slots[i].buffer, header.direct_alignment, + header.max_command_bytes); + if (rc) { errno = rc; fail("posix_memalign"); } + } + aio_context_t context = 0; + if (aio_setup_sys(qd, &context)) fail("io_setup"); + FILE *group_log = open_exclusive(group_path); + FILE *io_log = io_path ? open_exclusive(io_path) : NULL; + FILE *run_summary = open_exclusive(summary_path); + /* The complete group log is below 1 MiB for the supplied traces. Keep it + * out of the timed groups and flush after replay. Per-I/O logging remains + * a diagnostic mode whose overhead is explicitly visible. */ + if (setvbuf(group_log, NULL, _IOFBF, 4u << 20)) fail("setvbuf group log"); + if (setvbuf(run_summary, NULL, _IOFBF, 4096)) fail("setvbuf summary"); + + if (!skip_counters) { + if (!controller || femu_flip(controller, FEMU_RESET_QLC)) + fail("FEMU QLC counter reset"); + } + rewind(trace); + (void)read_header(trace); + uint64_t origin = now_ns(), sum_group_io = 0; + uint64_t observed_commands = 0, observed_bytes = 0; + uint8_t counter_phase = UINT8_MAX; + for (uint64_t expected = 0; expected < header.group_count; expected++) { + struct group_record group; + read_exact(trace, &group, sizeof group, "group record during replay"); + /* Every preceding group is complete here. Attribute its physical NAND + * counter delta before the next phase starts; requests may alternate + * prefill/decode multiple times in one compiled trace. */ + if (!skip_counters && counter_phase != UINT8_MAX + && group.phase != counter_phase) { + if (femu_flip(controller, qlc_accum_command(counter_phase))) + fail("FEMU QLC phase accumulation"); + } + counter_phase = group.phase; + struct command_record *commands = NULL; + if (group.command_count) { + commands = malloc((size_t)group.command_count * sizeof *commands); + if (!commands) fail("allocate group commands"); + read_exact(trace, commands, (size_t)group.command_count * sizeof *commands, + "group commands during replay"); + } + uint64_t ready = now_ns(), first_submit = 0, last_submit = 0, last_complete = ready; + uint64_t group_requested_bytes = 0; + for (uint32_t index = 0; index < group.command_count; index++) + group_requested_bytes += + (uint64_t)commands[index].sector_count * header.sector_bytes; + uint32_t next = 0, completed = 0, in_flight = 0, peak = 0; + while (completed < group.command_count) { + while (next < group.command_count && in_flight < qd) { + int index = free_slot(slots, qd); + if (index < 0) { errno = 0; fail("queue bookkeeping"); } + struct slot *slot = &slots[index]; + slot->command = commands[next]; + slot->local_index = next; + memset(&slot->cb, 0, sizeof slot->cb); + slot->cb.aio_data = (uint64_t)index + 1; + slot->cb.aio_lio_opcode = IOCB_CMD_PREAD; + slot->cb.aio_fildes = (uint32_t)device_fd; + slot->cb.aio_buf = (uint64_t)(uintptr_t)slot->buffer; + slot->cb.aio_nbytes = (uint64_t)slot->command.sector_count * header.sector_bytes; + slot->cb.aio_offset = (int64_t)(slot->command.lba_start * header.sector_bytes); + struct iocb *pointer = &slot->cb; + slot->submit_ns = now_ns(); + int submitted; + do submitted = aio_submit_sys(context, 1, &pointer); while (submitted < 0 && errno == EINTR); + if (submitted != 1) fail("io_submit"); + slot->active = true; + if (!first_submit) first_submit = slot->submit_ns; + last_submit = now_ns(); + next++; in_flight++; + if (in_flight > peak) peak = in_flight; + } + int count; + do count = aio_getevents_sys(context, 1, qd, events); while (count < 0 && errno == EINTR); + if (count <= 0) fail("io_getevents"); + uint64_t observed = now_ns(); + for (int i = 0; i < count; i++) { + if (!events[i].data || events[i].data > qd) { errno = 0; fail("invalid AIO user data"); } + struct slot *slot = &slots[events[i].data - 1]; + uint64_t expected_bytes = (uint64_t)slot->command.sector_count * header.sector_bytes; + if (!slot->active || events[i].res != (int64_t)expected_bytes || events[i].res2) { + errno = events[i].res < 0 ? (int)-events[i].res : 0; + fail("asynchronous read completion"); + } + if (io_log) fprintf(io_log, + "{\"group_id\":%" PRIu64 ",\"command_index\":%u," + "\"lba_start\":%" PRIu64 ",\"sector_count\":%u,\"page_class\":%u," + "\"submit_ns\":%" PRIu64 ",\"completion_observed_ns\":%" PRIu64 "," + "\"observed_latency_ns\":%" PRIu64 "}\n", + group.group_id, slot->local_index, slot->command.lba_start, + slot->command.sector_count, slot->command.page_class, + slot->submit_ns-origin, observed-origin, observed-slot->submit_ns); + slot->active = false; + in_flight--; completed++; + observed_commands++; + observed_bytes += expected_bytes; + } + last_complete = observed; + } + uint64_t group_io = first_submit ? last_complete - first_submit : 0; + sum_group_io += group_io; + if (group.command_count) { + fprintf(group_log, + "{\"group_id\":%" PRIu64 ",\"request_index\":%u,\"forward_id\":%" PRIu64 + ",\"layer\":%d,\"phase\":\"%s\",\"cache_reset\":%s," + "\"release_after_group_id\":%" PRId64 ",\"command_count\":%u," + "\"requested_bytes\":%" PRIu64 ",\"peak_outstanding\":%u," + "\"group_ready_ns\":%" PRIu64 ",\"first_submit_ns\":%" PRIu64 "," + "\"last_submit_ns\":%" PRIu64 ",\"last_complete_ns\":%" PRIu64 "," + "\"group_io_ns\":%" PRIu64 "}\n", + group.group_id, group.request_index, group.forward_id, group.layer, + phase_name(group.phase), group.cache_reset ? "true" : "false", + group.release_after_group_id, group.command_count, + group_requested_bytes, + peak, ready-origin, first_submit-origin, last_submit-origin, + last_complete-origin, group_io); + } else { + fprintf(group_log, + "{\"group_id\":%" PRIu64 ",\"request_index\":%u,\"forward_id\":%" PRIu64 + ",\"layer\":%d,\"phase\":\"%s\",\"cache_reset\":%s," + "\"release_after_group_id\":%" PRId64 ",\"command_count\":0," + "\"requested_bytes\":0,\"peak_outstanding\":0," + "\"group_ready_ns\":%" PRIu64 ",\"first_submit_ns\":null," + "\"last_submit_ns\":null,\"last_complete_ns\":%" PRIu64 "," + "\"group_io_ns\":0}\n", + group.group_id, group.request_index, group.forward_id, group.layer, + phase_name(group.phase), group.cache_reset ? "true" : "false", + group.release_after_group_id, ready-origin, ready-origin); + } + free(commands); + } + uint64_t wall = now_ns() - origin; + if (observed_commands != header.command_count || observed_bytes != header.total_bytes) { + errno = 0; fail("replay totals mismatch"); + } + fflush(group_log); if (io_log) fflush(io_log); + if (!skip_counters) { + if (counter_phase != UINT8_MAX + && femu_flip(controller, qlc_accum_command(counter_phase))) + fail("FEMU final QLC phase accumulation"); + if (femu_flip(controller, FEMU_SNAP_QLC)) + fail("FEMU QLC counter snapshot"); + } + fprintf(run_summary, + "{\n \"schema\": \"moe-bcq-replay-result-v1\",\n" + " \"queue_depth\": %u,\n \"direct_io\": %s,\n" + " \"qlc_counters\": \"%s\",\n" + " \"groups\": %" PRIu64 ",\n \"commands\": %" PRIu64 ",\n" + " \"requested_bytes\": %" PRIu64 ",\n" + " \"wall_ns_including_log_overhead\": %" PRIu64 ",\n" + " \"sum_group_io_ns\": %" PRIu64 ",\n" + " \"extent_map_sha256\": \"%s\",\n" + " \"mapped_reads_sha256\": \"%s\"\n}\n", + qd, buffered ? "false" : "true", + skip_counters ? "skipped" : "reset_phase_accumulated_and_snapshotted", + header.group_count, + observed_commands, observed_bytes, wall, sum_group_io, layout_hash, mapped_hash); + fflush(run_summary); + + fclose(run_summary); if (io_log) fclose(io_log); fclose(group_log); + if (aio_destroy_sys(context)) fail("io_destroy"); + for (uint32_t i = 0; i < qd; i++) free(slots[i].buffer); + free(events); free(slots); close(device_fd); fclose(trace); + return 0; +} diff --git a/moe-harness/exp/moe_bcq/femu_run/audit_handoff.py b/moe-harness/exp/moe_bcq/femu_run/audit_handoff.py new file mode 100644 index 0000000000..77b3cc4e6e --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/audit_handoff.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Read-only audit of the supplied Qwen binary and saved FEMU evidence. + +Run from the repository root. This does not boot FEMU or touch a device. +""" +import collections +import csv +import hashlib +import io +import json +from pathlib import Path +import re +import struct + + +def digest(path): + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def audit(root): + layout = root / 'exp/moe_bcq/femu_handoff/packages/qwen_C/layouts/qlc_aligned_epm_aif_2ch4lun' + runs = root / 'runs/femu' + binary = layout / 'replay_qd32.bin' + header = struct.Struct('<8s8I4Q32s32s') + group = struct.Struct(' ch=(\d+) lun=(\d+) pl=(\d+) blk=(\d+) pg=(\d+)') + rows = [tuple(map(int, m.groups())) for m in pattern.finditer(marker.read_text())] + # Absolute PPA equality is meaningful for the first fresh-device prefix. + prefix = [r for r in rows if r[0] < 4096] + prefix_mismatches = sum((ch, lun, pl, blk, pg) != + (lpn % 2, lpn // 2 % 4, 0, lpn // 4096, lpn // 8 % 512) + for lpn, ch, lun, pl, blk, pg in prefix) + assert len(prefix) == 4096 and prefix_mismatches == 0 + # Split into continuous LPN runs; these are observations, not inferred + # command boundaries. Later probes deliberately start at different LBAs. + sequences = [] + for row in rows: + if not sequences or row[0] not in (sequences[-1][-1][0], sequences[-1][-1][0] + 1): + sequences.append([]) + sequences[-1].append(row) + phys = lambda r: r[4] * 4096 + r[5] * 8 + r[2] * 2 + r[1] + sequence_reports = [] + for seq in sequences: + counts = collections.Counter(r[0] for r in seq) + extra = len(seq) - len(counts) + advance_extra = phys(seq[-1]) - phys(seq[0]) - (seq[-1][0] - seq[0][0]) + assert advance_extra == extra + sequence_reports.append(dict(first_lpn=seq[0][0], last_lpn=seq[-1][0], + write_records=len(seq), duplicate_programs=extra, + extra_physical_slots=advance_extra, + duplicate_lpn_mod32=dict(collections.Counter(k % 32 for k, v in counts.items() if v > 1)))) + + saved_counters = {} + for path in sorted(runs.glob('*qlc_counts.csv*')): + values = [0] * 4 + for row in csv.DictReader(io.StringIO(''.join( + line for line in path.read_text().splitlines(True) if not line.startswith('#')))): + values[int(row['page_class'])] += int(row['n_read']) + saved_counters[path.name] = dict(sha256=digest(path), pages=values, + total=sum(values), matches_replay_prediction=values == pages) + + return dict(binary=dict(sha256=digest(binary), groups=h[9], commands=commands, + host_read_bytes=byte_count, expected_nand_page_reads=pages, + total_nand_page_reads=sum(pages), incorrect_floor_counts=floor_pages, + command_size_histogram=dict(sorted(sizes.items())), max_command_bytes=max(sizes), + mapper_class_mismatches=class_mismatches, binary_matches_mapped_jsonl=True), + marker_log=dict(sha256=digest(marker), fresh_prefix_records=len(prefix), + fresh_prefix_ppa_mismatches=prefix_mismatches, sequences=sequence_reports), + saved_counters=saved_counters, + limitations=['No new FEMU run; audit uses saved files only.', + 'WRITE markers show program allocation, not NVMe command boundaries or Linux segment counts.', + '256 KiB fill still requires queue-limit and actual PPA verification on each configuration.']) + + +if __name__ == '__main__': + print(json.dumps(audit(Path.cwd()), indent=2)) diff --git a/moe-harness/exp/moe_bcq/femu_run/class_confusion.c b/moe-harness/exp/moe_bcq/femu_run/class_confusion.c new file mode 100644 index 0000000000..a5d14dbb2d --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/class_confusion.c @@ -0,0 +1,85 @@ +/* Read a chosen set of LPNs so FEMU's own per-class counters report which + * physical page classes they actually landed on. + * + * The latency probe infers a class from a measured time and can be fooled by + * host jitter. The device counts every physical page read by class itself, so + * reading only the LPNs a layout model calls class k turns the counter vector + * into that model's confusion row exactly, with no timing inference at all. + * + * Counters are reset here and snapshotted at the end, so the vector covers this + * program's reads and nothing else. The snapshot is written by FEMU to the path + * it was started with; this program cannot see it. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include + +#define PGSZ (16 * 1024) +#define PGS_PER_BLK 512 +#define LPN_PER_PAGE_INDEX 8 /* nchs * luns_per_ch, the modelled stride */ +#define FEMU_FLIP_OPCODE 0xef +#define FEMU_RESET_QLC 8 +#define FEMU_SNAP_QLC 9 + +static int femu_flip(const char *ctrl, int selector) +{ + int fd = open(ctrl, O_RDONLY); + if (fd < 0) { perror(ctrl); return -1; } + struct nvme_admin_cmd cmd; + memset(&cmd, 0, sizeof cmd); + cmd.opcode = FEMU_FLIP_OPCODE; + cmd.cdw10 = selector; + int rc = ioctl(fd, NVME_IOCTL_ADMIN_CMD, &cmd); + close(fd); + return rc; +} + +/* mirrors init_qlc_page_pairing() with the rows-1 fix */ +static int page_class(long pg) +{ + if (pg < 6) return 0; + if (pg < 8) return 1; + return (int)(((pg - 8) % 8) / 2); +} + +int main(int argc, char **argv) +{ + if (argc < 5) { + fprintf(stderr, "usage: %s DEV CTRL WANT_CLASS LPN_LIMIT\n" + " reads every LPN below LPN_LIMIT the model calls WANT_CLASS\n", + argv[0]); + return 2; + } + const char *dev = argv[1], *ctrl = argv[2]; + int want = atoi(argv[3]); + long limit = atol(argv[4]); + + int fd = open(dev, O_RDONLY | O_DIRECT); + if (fd < 0) { perror(dev); return 1; } + void *buf; + if (posix_memalign(&buf, 4096, PGSZ)) { perror("memalign"); return 1; } + + if (femu_flip(ctrl, FEMU_RESET_QLC)) { + fprintf(stderr, "fatal: QLC counter reset failed\n"); + return 1; + } + + long n = 0; + for (long lpn = 0; lpn < limit; lpn++) { + if (page_class((lpn / LPN_PER_PAGE_INDEX) % PGS_PER_BLK) != want) continue; + if (pread(fd, buf, PGSZ, (off_t)lpn * PGSZ) != PGSZ) { perror("pread"); return 1; } + n++; + } + + if (femu_flip(ctrl, FEMU_SNAP_QLC)) { + fprintf(stderr, "fatal: QLC counter snapshot failed\n"); + return 1; + } + printf("CONFUSION want_class=%d lpn_limit=%ld reads=%ld\n", want, limit, n); + return 0; +} diff --git a/moe-harness/exp/moe_bcq/femu_run/drive_multi.sh b/moe-harness/exp/moe_bcq/femu_run/drive_multi.sh new file mode 100755 index 0000000000..7637607350 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/drive_multi.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# Replay several traces against one already-filled device. +# +# A placement is a property of the image, not of the trace: re-filling the +# device for every trace would re-run out-of-place allocation and land the pages +# on different physical classes, so the fill has to happen once and every trace +# that shares that placement has to be replayed on it. replay_v1 resets the QLC +# counters when it starts and snapshots them when it ends, so consecutive +# replays on one device still report independent per-class totals. +# +# Two things from drive_run.sh carry over because they are what made earlier +# runs unrecoverable: FEMU rewrites one stats file per snapshot and the guest +# cannot read it, so the host copies it between steps; and replay_v1 refuses to +# overwrite its outputs, so every replay needs unique paths or it exits without +# running and leaves the previous counters in place, which reads as success. +set -uo pipefail + +TAG=${1:?usage: drive_multi.sh DEVICE_TAG SPECFILE} +SPEC=${2:?usage: drive_multi.sh DEVICE_TAG SPECFILE} +# Everything this script touches is data -- counters, records, replay binaries +# -- so it follows the project root, not where the script itself lives. +ROOT=${FEMU_PROJECT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)} +CSV=$ROOT/runs/femu/${TAG}_qlc.csv +RECORDS=$ROOT/exp/moe_bcq/femu_run/records +# -n on every ssh that is not being fed a file: without it ssh inherits the +# loop's stdin and swallows the rest of the spec, so the first replay runs and +# the loop then reads EOF and exits reporting success. SSHIN is the one that +# does take stdin, for streaming a binary in. +SSHOPT="-p ${SSH_PORT:-2222} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10" +SSH="ssh -n $SSHOPT femu@127.0.0.1" +SSHIN="ssh $SSHOPT femu@127.0.0.1" + +echo "== device $TAG ==" +echo " queue: $($SSH 'cat /sys/block/nvme0n1/queue/max_segments /sys/block/nvme0n1/queue/max_sectors_kb' 2>/dev/null | tr '\n' ' ')" + +# ---- placement check, once per device ------------------------------------- +# Asserted, not printed. A drifted fill still produces a plausible counter row, +# and the whole reason the earlier runs went unnoticed is that a human had to +# spot the difference. The whole image, not a prefix: the pages past a partial +# range are exactly the ones that check would miss. +IMAGE_PAGES=${IMAGE_PAGES:?IMAGE_PAGES must be set (Qwen 471040, DeepSeek 544768)} +CONF=$ROOT/runs/femu/$TAG +mkdir -p "$CONF" +echo " class c0 c1 c2 c3" +for c in 0 1 2 3; do + before=$(stat -c %Y.%N "$CSV" 2>/dev/null || echo none) + $SSH "sudo /usr/local/bin/class_confusion /dev/nvme0n1 /dev/nvme0 $c $IMAGE_PAGES" \ + > "$CONF/confusion_c$c.stdout" 2>"$CONF/confusion_c$c.stderr" || { + echo " confusion_c$c FAILED"; tail -3 "$CONF/confusion_c$c.stderr" | sed 's/^/ /'; exit 1; } + for _ in $(seq 40); do + [ -f "$CSV" ] && [ "$(stat -c %Y.%N "$CSV" 2>/dev/null)" != "$before" ] && break + sleep 0.5 + done + cp "$CSV" "$CONF/confusion_c$c.csv" || exit 1 + read -r -a got <<<"$(grep -v '^#' "$CONF/confusion_c$c.csv" | grep -v page_class | cut -d, -f2 | tr '\n' ' ')" + printf ' confusion_c%-3s' "$c"; printf '%9d ' "${got[@]:0:4}"; printf '\n' + for k in 0 1 2 3; do + if { [ "$k" = "$c" ] && [ "${got[$k]}" -eq 0 ]; } || + { [ "$k" != "$c" ] && [ "${got[$k]}" -ne 0 ]; }; then + echo " FAIL class $c leaked into class $k -- the fill did not land as planned" + echo " counters: ${got[*]}"; exit 1 + fi + done +done +echo " confusion diagonal over $IMAGE_PAGES pages: clean" + +# ---- one replay per spec line --------------------------------------------- +rc=0 +mapfile -t SPEC_LINES < "$SPEC" +for LINE in "${SPEC_LINES[@]}"; do + read -r NAME BIN <<<"$LINE" + [ -n "${NAME:-}" ] || continue + case $NAME in \#*) continue;; esac + OUT=$RECORDS/$NAME + if [ -f "$OUT/replay.csv" ] && [ -f "$OUT/groups.jsonl.gz" ]; then + echo " -- $NAME already collected, skipping" + continue + fi + [ -f "$ROOT/$BIN" ] || { echo " -- $NAME MISSING binary $BIN"; rc=1; continue; } + mkdir -p "$OUT" + echo " -- $NAME ($(du -h "$ROOT/$BIN" | cut -f1))" + + # Fresh guest-side paths each time: replay_v1 opens its outputs O_EXCL, so a + # reused path makes it exit without replaying while the old counters stay + # put -- indistinguishable from a successful repeat unless rc is checked. + $SSH "rm -f /home/femu/current.bin /dev/shm/$NAME.*" >/dev/null 2>&1 + if ! $SSHIN "cat > /home/femu/current.bin" < "$ROOT/$BIN"; then + echo " binary transfer FAILED"; rc=1; continue + fi + want=$(stat -c %s "$ROOT/$BIN"); got=$($SSH "stat -c %s /home/femu/current.bin" 2>/dev/null) + [ "$want" = "$got" ] || { echo " binary truncated in transit ($got of $want)"; rc=1; continue; } + + before=$(stat -c %Y.%N "$CSV" 2>/dev/null || echo none) + if ! $SSH "sudo /usr/local/bin/replay_v1 --trace /home/femu/current.bin --device /dev/nvme0n1 \ + --controller /dev/nvme0 --qd 32 --group-log /dev/shm/$NAME.groups.jsonl \ + --summary /dev/shm/$NAME.summary.json && cat /dev/shm/$NAME.summary.json" \ + > "$OUT/replay.stdout" 2>"$OUT/replay.stderr"; then + echo " replay FAILED"; tail -3 "$OUT/replay.stderr" | sed 's/^/ /'; rc=1 + $SSH "rm -f /home/femu/current.bin /dev/shm/$NAME.*" >/dev/null 2>&1 + continue + fi + for _ in $(seq 60); do + [ -f "$CSV" ] && [ "$(stat -c %Y.%N "$CSV" 2>/dev/null)" != "$before" ] && break + sleep 0.5 + done + cp "$CSV" "$OUT/replay.csv" || { echo " counters not captured"; rc=1; } + if $SSH "sudo gzip -c /dev/shm/$NAME.groups.jsonl" > "$OUT/groups.jsonl.gz" 2>/dev/null && + gzip -t "$OUT/groups.jsonl.gz" 2>/dev/null; then + echo " group log $(zcat "$OUT/groups.jsonl.gz" | wc -l) lines" + else + echo " WARNING group log not retrieved -- per-phase timing unavailable"; rc=1 + fi + python3 - "$OUT" <<'PY' +import json, sys +from pathlib import Path +out = Path(sys.argv[1]) +s = json.loads((out/'replay.stdout').read_text()) +cls = [int(l.split(',')[1]) for l in (out/'replay.csv').read_text().splitlines() + if l[:1].isdigit()] +print(f" groups {s['groups']:,} commands {s['commands']:,} " + f"{s['requested_bytes']/2**30:.1f} GiB io {s['sum_group_io_ns']/1e9:.1f}s " + f"pages {sum(cls):,}") +PY + # tmpfs is small and the gsm8k logs are tens of MB; the record is on the host now. + $SSH "rm -f /home/femu/current.bin /dev/shm/$NAME.*" >/dev/null 2>&1 +done + +echo "== device $TAG done (rc=$rc) ==" +exit $rc diff --git a/moe-harness/exp/moe_bcq/femu_run/drive_run.sh b/moe-harness/exp/moe_bcq/femu_run/drive_run.sh new file mode 100755 index 0000000000..a66eecc9a7 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/drive_run.sh @@ -0,0 +1,120 @@ +#!/bin/bash +# Drive one measurement run from the host, preserving a counter snapshot per step. +# +# FEMU rewrites the same stats file on every snapshot and the guest cannot see +# that file, so a step's counters survive only if the host copies them before the +# next step runs. The previous run lost its replay-only snapshot exactly this way, +# leaving a reported result that could not be re-checked. +# +# Every guest command's exit status is checked. replay_v1 refuses to overwrite an +# existing output file, so a repeated run with the same paths exits non-zero and +# leaves the counters untouched -- which reads as a successful repeat unless the +# status is examined. +set -uo pipefail + +RUN=${1:?usage: drive_run.sh RUN_TAG} +# Derive the checkout root from this script rather than naming it, so the same +# script drives a run on whichever machine it was copied to. +ROOT=${FEMU_PROJECT_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)} +CSV=$ROOT/runs/femu/${RUN}_qlc.csv +OUT=$ROOT/runs/femu/$RUN +SSH="ssh -p ${SSH_PORT:-2222} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 femu@127.0.0.1" + +mkdir -p "$OUT" + +# Wait for FEMU to rewrite the stats file, then keep it under a step-specific name. +keep() { + local label=$1 before=$2 + for _ in $(seq 40); do + [ -f "$CSV" ] && [ "$(stat -c %Y.%N "$CSV" 2>/dev/null)" != "$before" ] && break + sleep 0.5 + done + cp "$CSV" "$OUT/$label.csv" || return 1 + printf ' %-14s ' "$label" + grep -v '^#' "$OUT/$label.csv" | tail -4 | awk -F, '{printf "%9d ", $2} END {print ""}' +} + +step() { # step LABEL COMMAND... + local label=$1; shift + local before; before=$(stat -c %Y.%N "$CSV" 2>/dev/null || echo none) + if ! $SSH "$@" > "$OUT/$label.stdout" 2>"$OUT/$label.stderr"; then + echo " $label FAILED (rc=$?); see $OUT/$label.stderr" + tail -3 "$OUT/$label.stderr" | sed 's/^/ /' + return 1 + fi + keep "$label" "$before" +} + +echo "== $RUN ==" +echo " queue: $($SSH 'cat /sys/block/nvme0n1/queue/max_segments /sys/block/nvme0n1/queue/max_sectors_kb' 2>/dev/null | tr '\n' ' ')" +echo " class c0 c1 c2 c3" +# Assert the diagonal rather than print it. A drifted fill still produces a +# plausible-looking counter row, and the whole reason the earlier runs went +# unnoticed for so long is that a human had to spot the difference. Reading the +# whole image, not a prefix: pages past the checked range are exactly the ones a +# partial check would miss. +IMAGE_PAGES=${IMAGE_PAGES:-471040} +for c in 0 1 2 3; do + step "confusion_c$c" "sudo /usr/local/bin/class_confusion /dev/nvme0n1 /dev/nvme0 $c $IMAGE_PAGES" || exit 1 + read -r -a got <<<"$(grep -v '^#' "$OUT/confusion_c$c.csv" | grep -v page_class | cut -d, -f2 | tr '\n' ' ')" + for k in 0 1 2 3; do + if { [ "$k" = "$c" ] && [ "${got[$k]}" -eq 0 ]; } || + { [ "$k" != "$c" ] && [ "${got[$k]}" -ne 0 ]; }; then + echo " FAIL class $c leaked into class $k — the fill did not land as planned" + echo " counters: ${got[*]}" + exit 1 + fi + done +done +echo " confusion diagonal over $IMAGE_PAGES pages: clean" +step replay "sudo /usr/local/bin/replay_v1 --trace /root/replay.bin --device /dev/nvme0n1 \ + --controller /dev/nvme0 --qd 32 --group-log /dev/shm/$RUN.groups.jsonl \ + --summary /dev/shm/$RUN.summary.json && cat /dev/shm/$RUN.summary.json" || exit 1 +# Phase-aware counters are optional. The replayer announces each phase change +# with an admin FLIP (selectors 10-12), but FEMU's bb_flip only implements 8 and +# 9, so those announcements currently fall through to default and no phase +# columns are produced. That is not a reason to throw the run away: NAND energy +# is exactly linear in per-class page count, and compose_e2e.py splits it on the +# host from the page_class each compiled command carries, checking that its +# per-class totals equal the ones below. If a rebuilt FEMU does emit the +# columns, they are checked here and become a second, independent split. +python3 - "$OUT/replay.csv" <<'PY' || exit 1 +import csv +import sys + +with open(sys.argv[1], encoding="utf-8") as source: + rows = list(csv.DictReader(line for line in source if not line.startswith("#"))) +if len(rows) != 4: + raise SystemExit(f"expected 4 page-class rows, found {len(rows)}") +required = { + "prefill_n_read", "prefill_bytes_read", + "decode_n_read", "decode_bytes_read", + "teacher_forced_n_read", "teacher_forced_bytes_read", +} +if not required.issubset(rows[0]): + print(" phase counters: absent (FEMU bb_flip lacks selectors 10-12); " + "the prefill/decode split comes from the compiled commands on the host") + raise SystemExit(0) +for row in rows: + for suffix in ("n_read", "bytes_read"): + total = int(row[suffix]) + phases = sum(int(row[f"{phase}_{suffix}"]) + for phase in ("prefill", "decode", "teacher_forced")) + if total != phases: + raise SystemExit( + f"class {row['page_class']} {suffix}: total {total} != phase sum {phases}") +if not any(int(row["prefill_n_read"]) for row in rows): + raise SystemExit("prefill physical-read counter is empty") +if not any(int(row["decode_n_read"]) for row in rows): + raise SystemExit("decode physical-read counter is empty") +print(" phase counter closure: total = prefill + decode + teacher-forced") +PY +# The group log carries per-phase timing and is the only record of it; the guest +# writes it to tmpfs, so it dies with the container unless it is pulled here. +if $SSH "sudo gzip -c /dev/shm/$RUN.groups.jsonl" > "$OUT/groups.jsonl.gz" 2>/dev/null && + gunzip -f -k "$OUT/groups.jsonl.gz" 2>/dev/null; then + echo " group log $(wc -l < "$OUT/groups.jsonl") lines" +else + echo " WARNING group log not retrieved; per-phase timing will be unavailable" +fi +echo " saved under $OUT/" diff --git a/moe-harness/exp/moe_bcq/femu_run/guest_bringup.sh b/moe-harness/exp/moe_bcq/femu_run/guest_bringup.sh new file mode 100755 index 0000000000..f710b54ed1 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/guest_bringup.sh @@ -0,0 +1,19 @@ +#!/bin/bash +# Bring the guest to a usable state and stop. Everything else runs over SSH. +# +# The serial console is one-shot: cloud-init runs what the seed says and then the +# machine just sits there. Putting an experiment in the seed therefore costs a +# boot and a device fill per question. This script only prepares the device and +# gets out of the way. +set -uo pipefail +exec 2>&1 +Q=/sys/block/nvme0n1/queue +echo "[guest] kernel $(uname -r)" +echo "[guest] nvme $(lsblk -dno NAME,SIZE /dev/nvme0n1 2>/dev/null)" +cat "$Q/max_hw_sectors_kb" > "$Q/max_sectors_kb" 2>/dev/null || true +echo none > "$Q/scheduler" 2>/dev/null || true +echo "[guest] max_sectors_kb $(cat $Q/max_sectors_kb), scheduler $(cat $Q/scheduler)" +for d in /dev/vd?; do + [ -b "$d" ] && echo "[guest] virtio $d $(blockdev --getsize64 "$d") bytes" +done +echo "[guest] READY, device NOT filled; drive the rest over ssh" diff --git a/moe-harness/exp/moe_bcq/femu_run/guest_replay.sh b/moe-harness/exp/moe_bcq/femu_run/guest_replay.sh new file mode 100755 index 0000000000..5221733f54 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/guest_replay.sh @@ -0,0 +1,91 @@ +#!/bin/bash +# Guest side of one QLC-aligned replay: load the payload, replay the trace, +# and hand everything back over the serial console. +# +# The payload arrives as a read-only virtio disk. The root disk is virtio-scsi (/dev/sda), +# so the only virtio-blk disks are the cloud-init seed and the payload, and the +# payload is by far the larger -- picked by size rather than by enumeration +# order, which is not something to bet a 7 GiB dd on. +set -euo pipefail +exec 2>&1 + +SRC="" +best=0 +for d in /dev/vd?; do + [ -b "$d" ] || continue + sz=$(blockdev --getsize64 "$d" 2>/dev/null || echo 0) + echo "[guest] virtio disk $d $sz bytes" + if [ "$sz" -gt "$best" ]; then best=$sz; SRC=$d; fi +done +DEV=/dev/nvme0n1 +CTRL=/dev/nvme0 +TRACE=/root/replay.bin +OUT=/dev/shm + +say() { echo "[guest] $*"; } + +say "kernel $(uname -r)" +say "nvme: $(lsblk -dno NAME,SIZE /dev/nvme0n1 2>/dev/null || echo MISSING)" +[ -b "$DEV" ] || { say "FATAL no $DEV"; exit 1; } +[ -n "$SRC" ] && [ -b "$SRC" ] || { say "FATAL no payload disk found"; lsblk; exit 1; } +say "payload disk is $SRC" + +# 4 MiB is the compiler's cap; current Qwen replay commands are <=256 KiB. +# max_sectors_kb is only one split limit. Record segment limits as well; +# a userspace AIO command count need not equal the NVMe command count. +Q=/sys/block/nvme0n1/queue +say "max_hw_sectors_kb=$(cat $Q/max_hw_sectors_kb) max_sectors_kb=$(cat $Q/max_sectors_kb)" +cat "$Q/max_hw_sectors_kb" > "$Q/max_sectors_kb" 2>/dev/null || true +echo none > "$Q/scheduler" 2>/dev/null || true +say "max_sectors_kb now $(cat $Q/max_sectors_kb), scheduler $(cat $Q/scheduler)" +say "memory_page_bytes=$(getconf PAGESIZE) max_segments=$(cat $Q/max_segments) max_segment_size=$(cat $Q/max_segment_size)" +if [ "$(cat $Q/max_sectors_kb)" -lt 256 ]; then + say "WARNING below the 256 KiB largest replay command; the device will see more commands than the trace has" +fi + +IMG_BYTES=$(blockdev --getsize64 "$SRC") +say "payload disk $IMG_BYTES bytes" +[ "$IMG_BYTES" -le "$(blockdev --getsize64 "$DEV")" ] || { say "FATAL payload exceeds namespace"; exit 1; } +[ "$((IMG_BYTES % 16384))" -eq 0 ] || { say "FATAL image is not NAND-page aligned"; exit 1; } + +# Leave one memory-page segment for a potentially unaligned userspace buffer. +# Fail rather than silently returning to a fill size that can split mid-page. +MEM_PAGE=$(getconf PAGESIZE) +SEGMENTS=$(cat "$Q/max_segments") +SEG_SIZE=$(cat "$Q/max_segment_size") +MAX_KB=$(cat "$Q/max_sectors_kb") +HW_KB=$(cat "$Q/max_hw_sectors_kb") +[ "$SEG_SIZE" -ge "$MEM_PAGE" ] && [ "$SEGMENTS" -ge "$((262144 / MEM_PAGE + 1))" ] && + [ "$MAX_KB" -ge 256 ] && [ "$HW_KB" -ge 256 ] || { + say "FATAL queue limits do not satisfy the 256 KiB fill contract"; exit 1; +} + +say "=== fill: sequential write from LBA 0 ===" +t0=$(date +%s.%N) +# bs is capped well under the queue's max_segments (127) on purpose. A larger +# O_DIRECT write is split by the block layer at a 127-segment = 508 KiB boundary, +# which falls in the middle of the 32nd 16 KiB flash page. That page then belongs +# to both fragments, so it is written twice, and out-of-place update spends an +# extra physical page -- shifting every later page one slot along and scrambling +# the QLC class the layout counted on. With the observed 4 KiB pages and +# max_segments=127, a page-aligned 256 KiB buffer needs at most 64 page segments. +# This is configuration-dependent: PPA verification is still required. +dd if="$SRC" of="$DEV" bs=256k iflag=fullblock oflag=direct conv=fsync status=none || { say "FATAL fill"; exit 1; } +t1=$(date +%s.%N) +say "fill done in $(echo "$t1 - $t0" | bc) s" + +say "=== read-back verification ===" +SRC_SHA=$(dd if="$SRC" bs=4M iflag=direct,fullblock status=none | sha256sum | cut -d' ' -f1) +DST_SHA=$(dd if="$DEV" bs=4M count="$IMG_BYTES" iflag=direct,fullblock,count_bytes status=none | sha256sum | cut -d' ' -f1) +say "payload sha256 $SRC_SHA" +say "device sha256 $DST_SHA" +[ "$SRC_SHA" = "$DST_SHA" ] && say "read-back OK" || { say "FATAL read-back mismatch"; exit 1; } + +# The replay's class prediction is only as good as the assumed LPN->PPA order. +# Read it back off the clock before trusting it: two windows, one at the start of +# the image and one deep inside it, so a pattern that drifts is visible. +# Stop here. Everything past the fill is driven over ssh instead, one step at a +# time, so the QLC counter file can be read on the host between steps -- the +# guest cannot see that file, and each snapshot overwrites the last. +say "device is filled and verified; driving the rest over ssh" +say "ALL DONE" diff --git a/moe-harness/exp/moe_bcq/femu_run/mark_write.c b/moe-harness/exp/moe_bcq/femu_run/mark_write.c new file mode 100644 index 0000000000..fb9700f8dd --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/mark_write.c @@ -0,0 +1,58 @@ +/* Write marker-bearing pages so FEMU logs the real LPN -> PPA for each one. + * + * Every attempt to predict the physical page class from the LPN has been a model + * of the write pointer, and the counters say the model is wrong. FEMU will name + * the physical address itself for any page whose content carries the marker + * string it was started with, so write that string into each page and read the + * mapping out of the emulator's log rather than inferring it. + * + * Write sequentially from LPN 0 on a fresh device: that is exactly the fill the + * layout assumes, so the logged addresses are the ones the layout would get. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include + +#define PGSZ (16 * 1024) + +int main(int argc, char **argv) +{ + if (argc < 4) { + fprintf(stderr, "usage: %s DEV MARKER COUNT [FIRST] [PAGES_PER_WRITE]\n" + " PAGES_PER_WRITE > 1 issues one multi-page command, the way a\n" + " bulk fill does, rather than a command per page.\n", argv[0]); + return 2; + } + const char *dev = argv[1], *marker = argv[2]; + long count = atol(argv[3]); + long first = argc > 4 ? atol(argv[4]) : 0; + long batch = argc > 5 ? atol(argv[5]) : 1; + if (batch < 1) batch = 1; + + int fd = open(dev, O_WRONLY | O_DIRECT); + if (fd < 0) { perror(dev); return 1; } + void *buf; + size_t span = (size_t)batch * PGSZ; + if (posix_memalign(&buf, 4096, span)) { perror("memalign"); return 1; } + + for (long i = 0; i < count; i += batch) { + long n = count - i < batch ? count - i : batch; + memset(buf, 0, span); + /* the marker plus the LPN, so a dump can be checked against the write */ + for (long j = 0; j < n; j++) + snprintf((char *)buf + (size_t)j * PGSZ, PGSZ, "%s lpn=%ld", + marker, first + i + j); + size_t want = (size_t)n * PGSZ; + if (pwrite(fd, buf, want, (off_t)(first + i) * PGSZ) != (ssize_t)want) { + perror("pwrite"); + return 1; + } + } + if (fsync(fd)) { perror("fsync"); return 1; } + printf("MARKWRITE first=%ld count=%ld pages_per_write=%ld marker=%s\n", + first, count, batch, marker); + return 0; +} diff --git a/moe-harness/exp/moe_bcq/femu_run/preflight.sh b/moe-harness/exp/moe_bcq/femu_run/preflight.sh new file mode 100755 index 0000000000..e27bdfe0e0 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/preflight.sh @@ -0,0 +1,45 @@ +#!/bin/bash +# Verify run_policy.sh is the patched version before committing to a run. +# +# On 2026-09-10 this file reverted, mid-sweep and silently, to its pre-patch +# state: byte-identical to the original and carrying the original mtime. What +# reverted it was never identified. Unpatched, two defaults come back and both +# fail quietly rather than loudly: +# +# BUNDLE -> qwen_C, so a DeepSeek run replays the Qwen binary against a +# DeepSeek image. Fill and read-back both still pass; neither +# check knows whose bytes it is looking at. +# IMAGE_PAGES -> 471040, so the last 73,728 pages of a DeepSeek image are +# never class-checked and confusion still reports "clean". +# +# So the guard cannot live inside run_policy.sh -- a revert takes it too. The +# canonical copy and this check sit outside the tree that reverted. +set -uo pipefail + +CANON=${FEMU_CANONICAL_DIR:-$HOME/.femu_canonical} +HARNESS=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) +TARGET=$HARNESS/exp/moe_bcq/femu_run/run_policy.sh + +[ -f "$CANON/SHA256SUMS" ] || { echo "preflight: no canonical copy at $CANON"; exit 1; } +want=$(awk '$2=="run_policy.sh"{print $1}' "$CANON/SHA256SUMS") +[ -n "$want" ] || { echo "preflight: SHA256SUMS has no run_policy.sh entry"; exit 1; } + +got=$(sha256sum "$TARGET" 2>/dev/null | cut -d' ' -f1) +if [ "$got" = "$want" ]; then + echo "preflight: run_policy.sh matches canonical (${want:0:12})" + exit 0 +fi + +echo "preflight: run_policy.sh DOES NOT match the canonical copy" +echo " expected ${want:0:12} got ${got:0:12}" +# Restoring is right only when the canonical copy is itself intact; otherwise a +# corrupted canonical would be copied over a good working file. +canon_now=$(sha256sum "$CANON/run_policy.sh" 2>/dev/null | cut -d' ' -f1) +if [ "$canon_now" != "$want" ]; then + echo " canonical copy is itself altered -- refusing to restore. Fix $CANON by hand." + exit 1 +fi +cp -p "$CANON/run_policy.sh" "$TARGET" +chmod +x "$TARGET" +echo " restored from $CANON/run_policy.sh" +sha256sum "$TARGET" | sed 's/^/ now /' diff --git a/moe-harness/exp/moe_bcq/femu_run/probe_map.c b/moe-harness/exp/moe_bcq/femu_run/probe_map.c new file mode 100644 index 0000000000..a06a253ec9 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/probe_map.c @@ -0,0 +1,104 @@ +/* Measure the LPN -> QLC page-class mapping the device actually uses. + * + * The layout planner predicts a class for every page it places, and the whole + * experiment rests on that prediction matching the device. Read latency is the + * one signal that reports the class directly: the four classes are 47.9, 76.2, + * 134.6 and 228.1 us apart, far enough to separate under any constant host + * overhead. So time single-page reads at known LPNs and read the class back off + * the clock instead of trusting a model of the write pointer. + * + * The device must already be filled: a read of an unmapped LPN never reaches the + * media, so it would time as class 0 and look like a mapping that starts fast. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include + +#define PGSZ (16 * 1024) +static const double EXPECT_US[4] = {47.9, 76.2, 134.6, 228.1}; + +static int cmp(const void *a, const void *b) +{ + double x = *(const double *)a, y = *(const double *)b; + return x < y ? -1 : x > y; +} + +static double now_us(void) +{ + struct timespec t; + clock_gettime(CLOCK_MONOTONIC_RAW, &t); + return t.tv_sec * 1e6 + t.tv_nsec / 1e3; +} + +int main(int argc, char **argv) +{ + const char *dev = argc > 1 ? argv[1] : "/dev/nvme0n1"; + long first = argc > 2 ? atol(argv[2]) : 0; + long count = argc > 3 ? atol(argv[3]) : 1024; + int reps = argc > 4 ? atoi(argv[4]) : 5; + + int fd = open(dev, O_RDONLY | O_DIRECT); + if (fd < 0) { perror(dev); return 1; } + void *buf; + if (posix_memalign(&buf, 4096, PGSZ)) { perror("memalign"); return 1; } + + double *med = malloc(count * sizeof *med); + double *s = malloc(reps * sizeof *s); + if (!med || !s) { fprintf(stderr, "oom\n"); return 1; } + + /* Sweep repetition-major rather than LPN-major: consecutive reads of the + * same page would sit behind one another on the same LUN and measure queue + * time as well as array time. */ + for (long i = 0; i < count; i++) med[i] = 0; + for (int r = 0; r < reps; r++) { + for (long i = 0; i < count; i++) { + off_t off = (off_t)(first + i) * PGSZ; + double t0 = now_us(); + if (pread(fd, buf, PGSZ, off) != PGSZ) { perror("pread"); return 1; } + double dt = now_us() - t0; + /* keep the running minimum: the cleanest estimate of array time */ + if (r == 0 || dt < med[i]) med[i] = dt; + } + } + + /* Calibrate the constant host overhead from the observed spread, then class + * each page by nearest expected latency. The offset is whatever makes the + * fastest pages land on class 0. */ + double *sorted = malloc(count * sizeof *sorted); + memcpy(sorted, med, count * sizeof *sorted); + qsort(sorted, count, sizeof *sorted, cmp); + double floor_us = sorted[count / 100]; /* 1st percentile */ + double offset = floor_us - EXPECT_US[0]; + + printf("PROBE dev=%s first=%ld count=%ld reps=%d\n", dev, first, count, reps); + printf("PROBE floor=%.1fus implied_host_offset=%.1fus\n", floor_us, offset); + printf("PROBE quartiles %.1f %.1f %.1f %.1f\n", + sorted[count / 8], sorted[count * 3 / 8], + sorted[count * 5 / 8], sorted[count * 7 / 8]); + + long hist[4] = {0}; + printf("CLASSMAP %ld ", first); + for (long i = 0; i < count; i++) { + int best = 0; + double bd = 1e18; + for (int c = 0; c < 4; c++) { + double d = med[i] - offset - EXPECT_US[c]; + if (d < 0) d = -d; + if (d < bd) { bd = d; best = c; } + } + hist[best]++; + putchar('0' + best); + if ((i + 1) % 128 == 0 && i + 1 < count) printf("\nCLASSMAP %ld ", first + i + 1); + } + printf("\nPROBE counts %ld %ld %ld %ld\n", hist[0], hist[1], hist[2], hist[3]); + + /* A few raw samples so the classification can be audited by hand. */ + printf("PROBE raw"); + for (long i = 0; i < 16 && i < count; i++) printf(" %ld:%.1f", first + i, med[i]); + printf("\n"); + return 0; +} diff --git a/moe-harness/exp/moe_bcq/femu_run/run_device.sh b/moe-harness/exp/moe_bcq/femu_run/run_device.sh new file mode 100755 index 0000000000..e877f7f691 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/run_device.sh @@ -0,0 +1,85 @@ +#!/bin/bash +# Boot one FEMU device, fill it from a given image, then replay a list of traces. +# +# This is deliberately NOT a patch to run_policy.sh. That file also exists on the +# GPU host and the trace delivery rsyncs the whole femu_run directory over, which +# silently reverted two edits to it mid-sweep on 2026-09-10. Files that exist +# only here survive that sync, so the multi-replay path lives in its own. +# +# One device per placement, several traces per device: the image decides which +# physical class each plane lands on, and every trace sharing that image shares +# that placement. Re-filling per trace would re-run out-of-place allocation. +set -uo pipefail + +TAG=${1:?usage: run_device.sh DEVICE_TAG IMAGE_BASENAME IMAGE_PAGES SPECFILE} +IMG_BASE=${2:?}; PAGES=${3:?}; SPEC=${4:?} +# Two roots, because the code and the data it works on need not live together. +# HARNESS is where these scripts are -- inside the FEMU checkout, tracked in +# git. ROOT is the project holding the images, the built guest binaries and the +# payload packages, which are tens of gigabytes and are distributed separately. +# They coincide when someone unpacks the data under the harness, which is what +# a fresh clone does, so the default keeps that case working with no argument. +HARNESS=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) +ROOT=${FEMU_PROJECT_ROOT:-$HARNESS} +IMAGES=${FEMU_GUEST_DIR:-$HOME/images} +SSH_PUBKEY=${FEMU_SSH_PUBKEY:-$HOME/.ssh/id_rsa.pub} +QEMU_IMG=${QEMU_IMG:-$(command -v qemu-img || echo "$HARNESS/../build/qemu-img")} +cd "$ROOT" + +# make_seed.py needs pycdlib, which is installed for the system interpreter and +# not for whatever python3 a shell resolves first. Fail here, not three lines +# into building the seed after the caller has committed to the run. +if [ -z "${PYTHON:-}" ]; then + for c in python3 /usr/bin/python3 python3.8; do + command -v "$c" >/dev/null 2>&1 || continue + "$c" -c 'import pycdlib' 2>/dev/null && { PYTHON=$c; break; } + done +fi +[ -n "${PYTHON:-}" ] || { echo " no python3 with pycdlib; pip install --user pycdlib"; exit 1; } + +IMG=/data/images/$IMG_BASE # container path; /data is runs/femu +HOST_IMG=$ROOT/runs/femu/images/$IMG_BASE +[ -f "$HOST_IMG" ] || { echo " no image at $HOST_IMG"; exit 1; } +[ -s "$SPEC" ] || { echo " empty or missing spec $SPEC"; exit 1; } +SEED=$IMAGES/seed-$TAG.iso +OVL=$IMAGES/femu-root-$TAG.qcow2 + +echo "=== device $TAG image $IMG_BASE pages $PAGES ($(grep -cve '^\s*$' "$SPEC") traces) ===" +rm -f "$SEED" "$OVL" "runs/femu/${TAG}_qlc.csv" +# No replay.bin in the seed: the binaries are streamed in per replay, and the +# gsm8k ones are ~100 MB each. +(cd "$HARNESS/exp/gating_nand/femu" && "$PYTHON" make_seed.py -o "$SEED" --tag "${TAG^^}" \ + --instance-id "femu-$TAG" --ssh-key "$SSH_PUBKEY" \ + --file /usr/local/bin/replay_v1=$ROOT/build/guest/replay_v1:0755 \ + --file /usr/local/bin/class_confusion=$ROOT/build/guest/class_confusion:0755 \ + --file /usr/local/bin/guest_replay.sh=$HARNESS/exp/moe_bcq/femu_run/guest_replay.sh:0755 \ + --run "/usr/local/bin/guest_replay.sh > /dev/ttyS0 2>&1") >/dev/null || exit 1 +"$QEMU_IMG" create -f qcow2 -F qcow2 -b jammy-server-cloudimg-amd64.img "$OVL" 32G >/dev/null + +sed -e "s|^FEMU_QLC_STATS_PATH=.*|FEMU_QLC_STATS_PATH=/data/${TAG}_qlc.csv|" \ + -e "s|^FEMU_IMAGE_NAME=.*|FEMU_IMAGE_NAME=$(basename "$OVL")|" \ + -e "s|^FEMU_CONTAINER_NAME=.*|FEMU_CONTAINER_NAME=femu-$TAG|" \ + -e "s|FEMU_EXTRA_DRIVES=.*|FEMU_EXTRA_DRIVES='file=/guest/$(basename "$SEED"),if=virtio,format=raw,readonly=on;file=$IMG,if=virtio,format=raw,readonly=on'|" \ + runs/femu/run01.env > "runs/femu/${TAG}.env" + +set -a; . "runs/femu/${TAG}.env"; set +a +if [ -z "${FEMU_SOURCE_DIR:-}" ]; then + # "$ROOT/.." is this harness living inside the FEMU checkout itself, which + # is how the published repository is laid out; the _deps forms are the + # original layout, where the harness was the outer project. + for c in "$HARNESS/.." "$ROOT/_deps/FEMU-MoE" "$ROOT/FEMU-MoE"; do + [ -f "$c/compose.yaml" ] && { FEMU_SOURCE_DIR=$c; break; } + done +fi +[ -n "${FEMU_SOURCE_DIR:-}" ] || { echo " no FEMU checkout with compose.yaml"; exit 1; } +export FEMU_SOURCE_DIR +nohup bash "$HARNESS/scripts/femu_compose.sh" up femu > "runs/femu/${TAG}.console.log" 2>&1 & +until grep -qa "ALL DONE\|FATAL" "runs/femu/${TAG}.console.log" 2>/dev/null; do sleep 10; done +grep -qa FATAL "runs/femu/${TAG}.console.log" && { echo " FATAL during fill"; exit 1; } +grep -a "fill done\|read-back OK" "runs/femu/${TAG}.console.log" | tr -d '\r' | sed 's/^femu[^|]*| / /' + +IMAGE_PAGES=$PAGES bash "$HARNESS/exp/moe_bcq/femu_run/drive_multi.sh" "$TAG" "$SPEC"; rc=$? +export FEMU_CONTAINER_NAME=femu-$TAG +bash "$HARNESS/scripts/femu_compose.sh" down >/dev/null 2>&1 +echo " device torn down (rc=$rc)" +exit $rc diff --git a/moe-harness/exp/moe_bcq/femu_run/run_policy.sh b/moe-harness/exp/moe_bcq/femu_run/run_policy.sh new file mode 100755 index 0000000000..00c2c91dc6 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/run_policy.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# One placement policy end to end: fresh device, 256 KiB fill, class check, replay. +# +# Each policy needs its own device. Refilling an existing one would land every +# page on a fresh PPA (out-of-place update), so the second fill would not be the +# layout the image describes. +set -uo pipefail +POL=${1:?usage: run_policy.sh POLICY} +# Same reason as drive_run.sh: the path is where the script is, not a constant. +HARNESS=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) +ROOT=${FEMU_PROJECT_ROOT:-$HARNESS} +# Guest images and the key authorised inside them are per-machine. +IMAGES=${FEMU_GUEST_DIR:-$HOME/images} +SSH_PUBKEY=${FEMU_SSH_PUBKEY:-$HOME/.ssh/id_rsa.pub} +# The patched build ships one; a system qemu-img works too and is preferred when +# present, since it does not depend on the checkout having been built yet. +QEMU_IMG=${QEMU_IMG:-$(command -v qemu-img || echo "$HARNESS/../build/qemu-img")} +cd "$ROOT" + +# One knob, not two. The run tag defaulted to something different from the +# layout prefix once, and the results then carried a name that did not say which +# layout produced them. +TAG=${TAG_PREFIX:-${LAYOUT_PREFIX:-pol}}_$POL +IMG=/data/images/${IMG_PREFIX:-qwen_C}_$POL.img +SEED=$IMAGES/seed-$TAG.iso +OVL=$IMAGES/femu-root-$TAG.qcow2 +LAYOUT=${LAYOUT_PREFIX:-qlc}_epm_aif_2ch4lun_$POL +BIN=exp/moe_bcq/femu_handoff/packages/qwen_C/layouts/$LAYOUT/replay_qd32.bin + +echo "=== $POL ===" +rm -f "$SEED" "$OVL" "runs/femu/${TAG}_qlc.csv" +(cd exp/gating_nand/femu && python3 make_seed.py -o "$SEED" --tag "${TAG^^}" \ + --instance-id "femu-$TAG" --ssh-key "$SSH_PUBKEY" \ + --file /usr/local/bin/replay_v1=$ROOT/build/guest/replay_v1:0755 \ + --file /usr/local/bin/class_confusion=$ROOT/build/guest/class_confusion:0755 \ + --file /root/replay.bin=$ROOT/$BIN \ + --file /usr/local/bin/guest_replay.sh=$HARNESS/exp/moe_bcq/femu_run/guest_replay.sh:0755 \ + --run "/usr/local/bin/guest_replay.sh > /dev/ttyS0 2>&1") >/dev/null || exit 1 +"$QEMU_IMG" create -f qcow2 -F qcow2 -b jammy-server-cloudimg-amd64.img "$OVL" 32G >/dev/null + +sed -e "s|^FEMU_QLC_STATS_PATH=.*|FEMU_QLC_STATS_PATH=/data/${TAG}_qlc.csv|" \ + -e "s|^FEMU_IMAGE_NAME=.*|FEMU_IMAGE_NAME=$(basename "$OVL")|" \ + -e "s|^FEMU_CONTAINER_NAME=.*|FEMU_CONTAINER_NAME=femu-$TAG|" \ + -e "s|FEMU_EXTRA_DRIVES=.*|FEMU_EXTRA_DRIVES='file=/guest/$(basename "$SEED"),if=virtio,format=raw,readonly=on;file=$IMG,if=virtio,format=raw,readonly=on'|" \ + runs/femu/run01.env > "runs/femu/${TAG}.env" + +set -a; . "runs/femu/${TAG}.env"; set +a +# setup_femu.sh puts the checkout under _deps; a working tree kept beside it is +# the older layout. Prefer whichever actually has the compose file rather than +# naming one, or the run stalls waiting for a container that was never started. +if [ -z "${FEMU_SOURCE_DIR:-}" ]; then + for c in "$HARNESS/.." "$ROOT/_deps/FEMU-MoE" "$ROOT/FEMU-MoE"; do + [ -f "$c/compose.yaml" ] && { FEMU_SOURCE_DIR=$c; break; } + done +fi +[ -n "${FEMU_SOURCE_DIR:-}" ] || { echo " no FEMU checkout with compose.yaml; run scripts/setup_femu.sh"; exit 1; } +export FEMU_SOURCE_DIR +nohup bash "$HARNESS/scripts/femu_compose.sh" up femu > "runs/femu/${TAG}.console.log" 2>&1 & +until grep -qa "ALL DONE\|FATAL" "runs/femu/${TAG}.console.log" 2>/dev/null; do sleep 10; done +grep -qa FATAL "runs/femu/${TAG}.console.log" && { echo " FATAL during fill"; exit 1; } +grep -a "fill done\|read-back OK" "runs/femu/${TAG}.console.log" | tr -d '\r' | sed 's/^femu[^|]*| / /' + +bash exp/moe_bcq/femu_run/drive_run.sh "$TAG" || exit 1 +export FEMU_CONTAINER_NAME=femu-$TAG +bash "$HARNESS/scripts/femu_compose.sh" down >/dev/null 2>&1 +echo " device torn down" diff --git a/moe-harness/exp/moe_bcq/femu_run/run_sweep.sh b/moe-harness/exp/moe_bcq/femu_run/run_sweep.sh new file mode 100755 index 0000000000..e0bbe49ed1 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/run_sweep.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Run one bundle's placement policies end to end, preflight first. +# +# Usage: run_sweep.sh BUNDLE LAYOUT_PREFIX IMG_PREFIX TAG_PREFIX [policy ...] +# run_sweep.sh deepseek_C gen256 ds256 ds256 +# run_sweep.sh qwen_C gen256 qw256 qw256 inverted +# +# The preflight call is the point of this wrapper: run_policy.sh reverted once +# between two policies of a live sweep, and the unpatched defaults fail quietly +# (see preflight.sh). Going through here means no sweep can start on a reverted +# script, and a revert mid-sweep is caught at the next policy. +set -uo pipefail +HARNESS=$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd) +ROOT=${FEMU_PROJECT_ROOT:-$HARNESS} +HERE=$HARNESS/exp/moe_bcq/femu_run + +BUNDLE=${1:?usage: run_sweep.sh BUNDLE LAYOUT_PREFIX IMG_PREFIX TAG_PREFIX [policy ...]} +LAYOUT_PREFIX=${2:?}; IMG_PREFIX=${3:?}; TAG_PREFIX=${4:?} +shift 4 +POLICIES=("$@"); [ ${#POLICIES[@]} -gt 0 ] || POLICIES=(aligned rotated inverted) + +cd "$ROOT" +rc=0 +for pol in "${POLICIES[@]}"; do + # Re-checked per policy, not once at the top: the observed revert landed + # between two policies of a running sweep. + bash "$HERE/preflight.sh" || { echo "!!!! preflight failed; not running $pol"; rc=1; break; } + echo "############ $(date +%T) $TAG_PREFIX $pol ############" + BUNDLE=$BUNDLE LAYOUT_PREFIX=$LAYOUT_PREFIX IMG_PREFIX=$IMG_PREFIX TAG_PREFIX=$TAG_PREFIX \ + bash "$HERE/run_policy.sh" "$pol" || { echo "!!!! $pol FAILED rc=$?"; rc=1; } +done +echo "############ $(date +%T) sweep done (rc=$rc) ############" +exit $rc diff --git a/moe-harness/exp/moe_bcq/femu_run/verify_fill_256.py b/moe-harness/exp/moe_bcq/femu_run/verify_fill_256.py new file mode 100644 index 0000000000..fed9635677 --- /dev/null +++ b/moe-harness/exp/moe_bcq/femu_run/verify_fill_256.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""Run the isolated, pre-provisioned verify256 VM; preserve each snapshot. + +Requires seed-verify256-20260910.iso and its private root overlay. Never uses +the existing femu-map VM. Results/output paths are exclusive. +""" +import csv +import hashlib +import io +import json +from pathlib import Path +import subprocess +import time + +ROOT = Path(__file__).resolve().parents[3] +OUT = ROOT / 'runs/femu/verify256_20260910' +NAME = 'femu-verify256-20260910' +IMAGE = 'sha256:a91a26ae90eb2f193e1d98b8b180f587fcb463c418a76a7f56360e1cfb9453b5' +EXPECTED = [420294, 420294, 315822, 237450] +SSH = ['ssh', '-o', 'BatchMode=yes', '-o', 'StrictHostKeyChecking=accept-new', + '-o', 'ConnectTimeout=5', '-p', '2223', 'femu@localhost'] + + +def run(args, timeout=120): + return subprocess.run(args, check=True, stdout=subprocess.PIPE, + stderr=subprocess.PIPE, timeout=timeout).stdout + + +def save(path, data): + with path.open('xb') as stream: + stream.write(data) + + +def remote(command, timeout=120): + return run(SSH + [command], timeout) + + +def counts(raw): + values = [0] * 4 + lines = ''.join(x for x in raw.decode().splitlines(True) if not x.startswith('#')) + for row in csv.DictReader(io.StringIO(lines)): + values[int(row['page_class'])] += int(row['n_read']) + return values + + +def snapshot(directory, stem): + # Must be called after successful guest completion and before another probe. + raw = (OUT / 'counter.live.csv').read_bytes() + save(directory / (stem + '.qlc.csv'), raw) + return counts(raw) + + +def verify_groups(records, summary): + assert len(records) == summary['groups'] == 1536 + assert sum(g['command_count'] for g in records) == summary['commands'] == 130090 + assert sum(g['requested_bytes'] for g in records) == summary['requested_bytes'] == 22608650240 + assert sum(g['group_io_ns'] for g in records) == summary['sum_group_io_ns'] + previous_complete = 0 + for index, g in enumerate(records): + assert g['group_id'] == index and g['group_ready_ns'] >= previous_complete + assert g['peak_outstanding'] <= 32 + if g['command_count']: + assert g['first_submit_ns'] >= g['group_ready_ns'] + assert g['last_complete_ns'] >= g['last_submit_ns'] + else: + assert g['first_submit_ns'] is None and g['group_io_ns'] == 0 + previous_complete = g['last_complete_ns'] + + +def main(): + env = dict(FEMU_IMAGE='/guest/femu-root-verify256-20260910.qcow2', + FEMU_MEMORY='8G', FEMU_CPUS='6', FEMU_NAND_CELL_TYPE='4', + FEMU_SSD_SIZE_MB='65536', FEMU_SECTORS_PER_PAGE='32', + FEMU_PAGES_PER_BLOCK='512', FEMU_BLOCKS_PER_PLANE='1024', + FEMU_PLANES_PER_LUN='1', FEMU_LUNS_PER_CHANNEL='4', FEMU_CHANNELS='2', + FEMU_EXTRA_DEVICE_OPTS='op_pcent=7', + FEMU_QLC_STATS_PATH='/data/verify256_20260910/counter.live.csv', + FEMU_QMP_SOCKET='/data/verify256_20260910/qmp.sock', + FEMU_EXTRA_DRIVES='file=/guest/seed-verify256-20260910.iso,if=virtio,format=raw,readonly=on;' + 'file=/data/images/qwen_C.img,if=virtio,format=raw,readonly=on') + save(OUT / 'configuration.json', json.dumps(dict(image=IMAGE, env=env, + fill_bytes=262144, expected_counts=EXPECTED), indent=2).encode()) + cmd = ['docker', 'run', '-d', '--name', NAME, '--device', '/dev/kvm', + '--cap-add', 'IPC_LOCK', '--ulimit', 'memlock=-1:-1', + '-p', '127.0.0.1:2223:2222', + '-v', '/data01/kwkim02/images:/guest', + '-v', str(ROOT / 'runs/femu') + ':/data'] + for key, value in env.items(): + cmd.extend(['-e', key + '=' + value]) + cmd.extend([IMAGE, 'bbssd']) + boot_reports = [] + launched = False + try: + for boot in (1, 2): + directory = OUT / f'boot{boot}' + directory.mkdir(exist_ok=False) + print(f'boot{boot}: starting fresh FEMU SSD', flush=True) + if boot == 1: + run(cmd) + launched = True + else: + run(['docker', 'start', NAME]) + for attempt in range(120): + try: + remote('sudo test -x /root/guest_fill.sh && sudo test -f /root/replay.bin', 10) + break + except (subprocess.CalledProcessError, subprocess.TimeoutExpired): + time.sleep(2) + else: + raise RuntimeError('guest SSH/cloud-init did not become ready') + save(directory / 'container.json', run(['docker', 'inspect', NAME])) + save(directory / 'guest_trace_sha256.txt', remote('sudo sha256sum /root/replay.bin')) + print(f'boot{boot}: sequential 256 KiB fill + complete read-back', flush=True) + fill = remote('sudo bash /root/guest_fill.sh', 1200) + save(directory / 'fill.log', fill) + expected_sha = json.loads((ROOT / 'runs/femu/images/qwen_C.img.json').read_text())['sha256'] + assert fill.count(expected_sha.encode()) == 2 and b'read-back OK' in fill + print(f'boot{boot}: read-back passed', flush=True) + reports = [] + for repeat in (1, 2, 3): + stem = f'replay{repeat}' + guest = f'/dev/shm/verify_boot{boot}_{stem}' + # Exclusive output paths and checked exit prevent stale success. + result = remote(f'sudo replay_v1 --trace /root/replay.bin --device /dev/nvme0n1 ' + f'--controller /dev/nvme0 --qd 32 --group-log {guest}.groups.jsonl ' + f'--summary {guest}.summary.json', 300) + save(directory / (stem + '.stdout'), result) + actual = snapshot(directory, stem) + summary_raw = remote(f'sudo cat {guest}.summary.json') + groups_raw = remote(f'sudo cat {guest}.groups.jsonl') + save(directory / (stem + '.summary.json'), summary_raw) + save(directory / (stem + '.groups.jsonl'), groups_raw) + summary = json.loads(summary_raw) + verify_groups([json.loads(line) for line in groups_raw.splitlines()], summary) + assert actual == EXPECTED, (boot, repeat, actual) + reports.append(dict(repeat=repeat, counts=actual, group_io_ns=summary['sum_group_io_ns'])) + print(f'boot{boot} {stem}: exact class counts, group_io={summary["sum_group_io_ns"]/1e9:.3f}s', flush=True) + matrix = [] + # Covers all 471040 LPNs in the image, including filler. + for klass in range(4): + result = remote(f'sudo class_confusion /dev/nvme0n1 /dev/nvme0 {klass} 471040', 180) + save(directory / f'class{klass}.stdout', result) + row = snapshot(directory, f'class{klass}') + n = int(result.decode().split('reads=')[1].strip()) + assert sum(row) == n and row[klass] == n, row + matrix.append(row) + print(f'boot{boot}: class {klass} full-image check passed ({n} pages)', flush=True) + boot_reports.append(dict(boot=boot, repeats=reports, full_image_confusion=matrix)) + run(['docker', 'stop', '-t', '15', NAME], 60) + save(directory / 'console.log', run(['docker', 'logs', NAME])) + report = dict(passed=True, boots=boot_reports, image_sha256=expected_sha) + save(OUT / 'verification.json', json.dumps(report, indent=2).encode()) + print('PASS: two fresh boots, six replays, two full-image class checks', flush=True) + finally: + if launched: + subprocess.run(['docker', 'stop', '-t', '10', NAME], stdout=subprocess.DEVNULL, timeout=30) + if not (OUT / 'console.final.log').exists(): + save(OUT / 'console.final.log', run(['docker', 'logs', NAME])) + + +if __name__ == '__main__': + main() diff --git a/moe-harness/runs/femu/run01.env b/moe-harness/runs/femu/run01.env new file mode 100644 index 0000000000..247e218f02 --- /dev/null +++ b/moe-harness/runs/femu/run01.env @@ -0,0 +1,17 @@ +FEMU_GUEST_DIR=/data/kwkim02/images +FEMU_DATA_DIR=/data/kwkim02/MoE_SSD/runs/femu +FEMU_IMAGE_NAME=placeholder.qcow2 +FEMU_MEMORY=8G +FEMU_CPUS=6 +FEMU_NAND_CELL_TYPE=4 +FEMU_SSD_SIZE_MB=65536 +FEMU_SECTORS_PER_PAGE=32 +FEMU_PAGES_PER_BLOCK=512 +FEMU_BLOCKS_PER_PLANE=1024 +FEMU_PLANES_PER_LUN=1 +FEMU_LUNS_PER_CHANNEL=4 +FEMU_CHANNELS=2 +FEMU_EXTRA_DEVICE_OPTS=op_pcent=7 +FEMU_QLC_STATS_PATH=/data/placeholder.csv +FEMU_EXTRA_DRIVES=placeholder +FEMU_CONTAINER_NAME=placeholder diff --git a/moe-harness/scripts/build_replay.sh b/moe-harness/scripts/build_replay.sh new file mode 100755 index 0000000000..7267b11a48 --- /dev/null +++ b/moe-harness/scripts/build_replay.sh @@ -0,0 +1,19 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +src="${repo_root}/exp/gating_nand/femu/replay.c" +out_dir="${repo_root}/build/guest" + +mkdir -p "${out_dir}" + +if ! pkg-config --exists liburing; then + echo "build_replay: liburing development files are missing" >&2 + echo "install liburing-dev, or activate the conda environment containing liburing" >&2 + exit 1 +fi + +read -r -a uring_flags <<< "$(pkg-config --cflags --libs liburing)" +cc -O2 -Wall -Wextra "${src}" -o "${out_dir}/replay" "${uring_flags[@]}" + +echo "build_replay: wrote ${out_dir}/replay" diff --git a/moe-harness/scripts/femu_compose.sh b/moe-harness/scripts/femu_compose.sh new file mode 100755 index 0000000000..805a807fae --- /dev/null +++ b/moe-harness/scripts/femu_compose.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +femu_root="${FEMU_SOURCE_DIR:-${repo_root}/_deps/FEMU-MoE}" + +if [[ ! -f "${femu_root}/compose.yaml" ]]; then + echo "femu_compose: ${femu_root}/compose.yaml is missing" >&2 + echo "run ${repo_root}/scripts/setup_femu.sh first" >&2 + exit 1 +fi + +export FEMU_GUEST_DIR="${FEMU_GUEST_DIR:-${repo_root}/images}" +export FEMU_DATA_DIR="${FEMU_DATA_DIR:-${repo_root}/runs/femu}" +mkdir -p "${FEMU_GUEST_DIR}" "${FEMU_DATA_DIR}" + +# Run from the FEMU source tree instead of relying on --project-directory. The +# latter is unavailable in old Compose installations and can be misparsed as a +# top-level Docker flag when the Compose CLI plugin is absent. +cd "${femu_root}" + +if docker compose version >/dev/null 2>&1; then + exec docker compose -f compose.yaml "$@" +fi + +if command -v docker-compose >/dev/null 2>&1; then + exec docker-compose -f compose.yaml "$@" +fi + +echo "femu_compose: Docker Compose is not installed" >&2 +echo "install the Docker Compose plugin, or the legacy docker-compose command" >&2 +exit 1