Moe qlc

.dockerignore

@@ -0,0 +1,13 @@
+.git
+.github
+build
+build-femu
+build-docker
+roms
+*.qcow2
+*.img
+*.iso
+*.log
+docker-data
+images
+guest

.gitignore

@@ -133,3 +133,15 @@ trace-ust-all.c
 build-femu/
 build/
 *.md
+# The blanket *.md above came in with the QEMU 10.1.0 upgrade and silently
+# drops documentation. The harness's own docs are the part a reader needs
+# most, so they are excepted rather than force-added one at a time.
+!moe-harness/**/*.md
+# and the documentation at the root, which is the first thing a reader opens.
+# README.md predates the rule above and is tracked; without this the next
+# document beside it would be dropped without a word, as RUNNING.md was.
+!/*.md
+# What the container writes: counter CSVs, console logs, the QMP socket. The
+# compose default puts it beside the checkout, so it needs naming here.
+/docker-data/
+subprojects/.wraplock

README.md

@@ -163,6 +163,65 @@ OpenChannel needs a host that speaks it. LightNVM was removed from Linux in
 
 ---
 
+## Running it in a container
+
+Commands, with their real output, are in [RUNNING.md](RUNNING.md). The short
+version and the reasoning follow.
+
+
+`femu-scripts/femu-docker.sh` is the path below done in a container. Two
+separate things push it there.
+
+**The host cannot run the path below.** `pkgdep.sh` installs packages, and
+`run-blackbox.sh` launches QEMU under `sudo` because FEMU pins its memory
+backend, which needs `RLIMIT_MEMLOCK` raised past the device size. A host
+allowing the usual 64 MiB cannot start a 64 GiB device at all, and raising it
+needs root. The container gets `IPC_LOCK` and an unlimited memlock without the
+host granting root to anyone.
+
+**Inside the container that argument stops applying, and a different one
+starts.** The container runs as root, so `sudo` would be moot there -- it is not
+even installed. What makes `run-blackbox.sh` unusable in it is that the SSD
+layout is written into the file: `pgs_per_blk=256`, `luns_per_ch=8`, `nchs=8`,
+`ssd_size=12288`, a fixed `u20s.qcow2`. It reads no environment, so there is no
+way to hand it this fork's geometry, a different guest disk, a payload disk or
+a counter path per run. `docker/femu-run` is that same script with those
+constants lifted out into environment variables; the steps it performs are
+unchanged.
+
+The steps map one to one:
+
+| this README | container |
+|---|---|
+| `sudo ./pkgdep.sh` | `docker/Dockerfile`, builder stage |
+| `./femu-compile.sh` | `docker/Dockerfile`, builder stage |
+| `./qemu-system-x86_64 -device femu,help` | `femu-docker.sh verify` |
+| `./run-blackbox.sh` | `femu-docker.sh run` |
+
+```bash
+./femu-scripts/femu-docker.sh build     # dependencies and compile, inside
+./femu-scripts/femu-docker.sh verify    # did the femu device register
+./femu-scripts/femu-docker.sh image     # guest overlay + cloud-init seed
+./femu-scripts/femu-docker.sh run       # start the device, boot the guest
+./femu-scripts/femu-docker.sh ssh       # a shell in the guest
+./femu-scripts/femu-docker.sh stop
+```
+
+One difference worth knowing: this README has you build a VM image by hand and
+reuse it. `image` cuts a copy-on-write overlay per instance instead and writes
+a cloud-init seed that authorises your ssh key, so the base image is never
+written and a broken guest is one file to delete. The cloud image ships no
+password, so without that seed there is no way in.
+
+The device is the same on both paths -- geometry, cell type and the read-energy
+coefficients come from `compose.yaml`, whose defaults are the configuration the
+measurements in this repository were taken on. `femu-docker.sh status` prints
+what the instance would get.
+
+Running an actual measurement is a different entry point, because it needs a
+payload image, a compiled trace and the placement checks: see
+[moe-harness/README.md](moe-harness/README.md).
+
 ## Installation
 
 ### Build FEMU

RUNNING.md

@@ -0,0 +1,146 @@
+# Running this FEMU
+
+Every command below was run on the measurement host and its output is what is
+quoted. Two entry points, because they answer different questions.
+
+| you want | go to |
+|---|---|
+| a VM with this project's SSD attached, to poke at | [A device](#a-device) |
+| the placement measurement, end to end | [A measurement](#a-measurement) |
+
+The device is the same either way, and no geometry argument is needed for it:
+`compose.yaml` defaults to the configuration the measurements were taken on --
+64 GiB over 2 channels x 4 LUNs, 512 pages per block, QLC, `op_pcent=7`.
+
+## Why a container
+
+The [README](README.md)'s path builds on the host and launches with
+`run-blackbox.sh`. That does not work here. `pkgdep.sh` installs packages, and
+FEMU pins its memory backend, which needs `RLIMIT_MEMLOCK` raised past the
+device size -- this host allows 64 MiB against 64 GiB. Both need root, and
+there is no passwordless sudo.
+
+The container is given `IPC_LOCK` and an unlimited memlock, so it pins without
+the host granting root to anyone. The steps are the README's:
+
+| README | here |
+|---|---|
+| `sudo ./pkgdep.sh` | `docker/Dockerfile`, builder stage |
+| `./femu-compile.sh` | `docker/Dockerfile`, builder stage |
+| `./qemu-system-x86_64 -device femu,help` | `femu-docker.sh verify` |
+| `./run-blackbox.sh` | `femu-docker.sh run` |
+
+Inside the container that argument no longer applies -- it runs as root -- and
+`run-blackbox.sh` is usable there too: it now reads the same environment
+variables. What it could not do before was take any configuration at all; the
+layout was written into the file.
+
+## A device
+
+```bash
+cd /data/kwkim02/MoE_FEMU
+
+export FEMU_GUEST_DIR=/data/kwkim02/images          # where guest disks live
+export FEMU_DATA_DIR=$PWD/docker-data               # container's /data
+export FEMU_INSTANCE=demo                           # names disk, seed, container
+```
+
+**Build.** Dependencies and compile happen inside; the host gets nothing.
+
+```bash
+./femu-scripts/femu-docker.sh build
+```
+
+**Check the device registered.**
+
+```bash
+./femu-scripts/femu-docker.sh verify
+#   femu options:
+#     acl=<uint8> ... blks_per_pl=<int32> ... nand_cell_type=<uint8> ...
+```
+
+**Make this instance's guest disk.** A copy-on-write overlay plus a cloud-init
+seed carrying your ssh key. The base image is never written, and the cloud
+image ships no password, so without the seed there is no way in.
+
+```bash
+./femu-scripts/femu-docker.sh image
+#   overlay /data/kwkim02/images/femu-root-demo.qcow2
+#   seed    /data/kwkim02/images/seed-demo.iso
+```
+
+**Start it.** Holds the terminal; Ctrl-C detaches and leaves the container up.
+
+```bash
+./femu-scripts/femu-docker.sh run
+#   FEMU mode=bbssd, NAND cell type=4, image=/guest/femu-root-demo.qcow2
+#   Guest SSH is forwarded to container port 2222
+```
+
+**Get in.** Boot takes three to four minutes. The `femu login:` prompt appears
+well before you can log in -- cloud-init installs the key after it. Wait for
+`Cloud-init ... finished` on the console.
+
+```bash
+./femu-scripts/femu-docker.sh ssh
+#   femu@femu:~$ lsblk -dno NAME,SIZE /dev/nvme0n1
+#   nvme0n1 59.8G
+```
+
+59.8 G rather than 64 is `op_pcent=7`: the over-provisioning every layout here
+is planned against.
+
+**Stop.**
+
+```bash
+./femu-scripts/femu-docker.sh stop
+```
+
+**What am I about to get.**
+
+```bash
+./femu-scripts/femu-docker.sh status
+#   cell=4 size=65536MB 2ch x 4LUN 512pg/blk 1024blk/pl opts=op_pcent=7
+```
+
+## A measurement
+
+A device with nothing on it measures nothing. A run also needs a payload image,
+a compiled trace and the placement checks, which is a different entry point:
+
+```bash
+FEMU_PROJECT_ROOT=/data/kwkim02/MoE_SSD \
+  bash /data/kwkim02/MoE_FEMU/moe-harness/exp/moe_bcq/femu_run/run_device.sh \
+       DEVICE_TAG IMAGE_BASENAME IMAGE_PAGES SPECFILE
+```
+
+`FEMU_PROJECT_ROOT` is where the data lives. The harness resolves its own code
+from where it sits and the data from there, so the two need not be together --
+the images, payload packages and records are tens of gigabytes and are
+distributed separately from this repository.
+
+It boots a device, fills it, asserts the placement landed, replays each trace
+in the spec file, and writes `groups.jsonl.gz` and `replay.csv` per run under
+`<data root>/exp/moe_bcq/femu_run/records/<run name>/`.
+
+See [moe-harness/README.md](moe-harness/README.md) for what a spec file is, what
+the fill contract is, and why a matching read-back hash does not mean the
+placement is right.
+
+## If something goes wrong
+
+**`no python3 with pycdlib`** — the seed builder needs it. `pip install --user
+pycdlib`, or set `PYTHON` to an interpreter that has it.
+
+**`kex_exchange_identification: Connection closed`** — the guest is up but
+cloud-init has not installed the key yet. Wait for `Cloud-init ... finished`.
+
+**`already exists: .../femu-root-demo.qcow2`** — `image` refuses to overwrite an
+instance's disk. Delete it to start over, or use another `FEMU_INSTANCE`.
+
+**A device that is not 59.8 G** — something is overriding the compose defaults.
+`femu-docker.sh status` prints what the instance would actually get.
+
+**Port 2222 already bound** — another instance is running. `docker ps`, then
+`FEMU_INSTANCE=<that one> ./femu-scripts/femu-docker.sh stop`, or set
+`FEMU_SSH_PORT` for this one.

compose.yaml

@@ -0,0 +1,86 @@
+services:
+  femu:
+    build:
+      context: .
+      dockerfile: docker/Dockerfile
+      args:
+        BUILD_JOBS: ${FEMU_BUILD_JOBS:-8}
+    image: ${FEMU_DOCKER_IMAGE:-femu-qlc:latest}
+    container_name: ${FEMU_CONTAINER_NAME:-femu}
+    devices:
+      - /dev/kvm:/dev/kvm
+    cap_add:
+      - IPC_LOCK
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+    volumes:
+      - ${FEMU_GUEST_DIR:-../images}:/guest
+      - ${FEMU_DATA_DIR:-./docker-data}:/data
+    ports:
+      - "${FEMU_SSH_PORT:-2222}:2222"
+    environment:
+      FEMU_MODE: ${FEMU_MODE:-bbssd}
+      FEMU_IMAGE: /guest/${FEMU_IMAGE_NAME:-u20s.qcow2}
+      FEMU_KERNEL: ${FEMU_KERNEL:-}
+      FEMU_INITRD: ${FEMU_INITRD:-}
+      FEMU_KERNEL_APPEND: ${FEMU_KERNEL_APPEND:-root=LABEL=rootfs console=ttyS0}
+      FEMU_MEMORY: ${FEMU_MEMORY:-8G}
+      FEMU_CPUS: ${FEMU_CPUS:-6}
+      FEMU_GUEST_SSH_PORT: 2222
+      FEMU_NAND_CELL_TYPE: ${FEMU_NAND_CELL_TYPE:-4}
+      # The QLC-aligned expert placement geometry: 64 GiB over 2 channels x 4
+      # LUNs, 512 pages per block. These are defaults, not a suggestion -- they
+      # are the device the measurements in this repository were taken on, and
+      # a run that leaves them alone reproduces it.
+      #
+      # 512 pages per block is the part that matters most. The QLC pairing table
+      # only covers pages 0..495 upstream, so at the old 256-page default every
+      # page class is correct and the bug this fork fixes (nand.c, rows - 1)
+      # cannot appear. Same source, different physics, silently.
+      #
+      # op_pcent=7 is over-provisioning: it sets the exposed namespace to 59.8 G
+      # of the 64 GiB raw, which is the capacity every layout here is planned
+      # against.
+      FEMU_SSD_SIZE_MB: ${FEMU_SSD_SIZE_MB:-65536}
+      FEMU_SECTORS_PER_PAGE: ${FEMU_SECTORS_PER_PAGE:-32}
+      FEMU_PAGES_PER_BLOCK: ${FEMU_PAGES_PER_BLOCK:-512}
+      FEMU_BLOCKS_PER_PLANE: ${FEMU_BLOCKS_PER_PLANE:-1024}
+      FEMU_PLANES_PER_LUN: ${FEMU_PLANES_PER_LUN:-1}
+      FEMU_LUNS_PER_CHANNEL: ${FEMU_LUNS_PER_CHANNEL:-4}
+      FEMU_CHANNELS: ${FEMU_CHANNELS:-2}
+      FEMU_QLC_STATS_PATH: ${FEMU_QLC_STATS_PATH:-/data/qlc_counts.csv}
+      # 에너지 계수 (milli-pJ/bit). peripheral + array 분해 모델:
+      #   peripheral = P_fix + rho_p * t_R(c),  P_fix=1.273  rho_p=0.668
+      #   array      = E_fix + (n(c)-1) * E_sense,  E_fix=0.551  E_sense=0.494
+      #   n(c) = 1/2/4/8 은 class 별 sensing 횟수 추정값 (인용값 아님)
+      FEMU_E_READ_C0: ${FEMU_E_READ_C0:-33821}
+      FEMU_E_READ_C1: ${FEMU_E_READ_C1:-53220}
+      FEMU_E_READ_C2: ${FEMU_E_READ_C2:-93219}
+      FEMU_E_READ_C3: ${FEMU_E_READ_C3:-157653}
+      FEMU_E_ARRAY_C0: ${FEMU_E_ARRAY_C0:-551}
+      FEMU_E_ARRAY_C1: ${FEMU_E_ARRAY_C1:-1045}
+      FEMU_E_ARRAY_C2: ${FEMU_E_ARRAY_C2:-2033}
+      FEMU_E_ARRAY_C3: ${FEMU_E_ARRAY_C3:-4009}
+      FEMU_E_XFER: ${FEMU_E_XFER:-28100}
+      FEMU_STATS_FLUSH_MS: ${FEMU_STATS_FLUSH_MS:-0}
+      FEMU_IMAGE_FORMAT: ${FEMU_IMAGE_FORMAT:-qcow2}
+      FEMU_EXTRA_DEVICE_OPTS: ${FEMU_EXTRA_DEVICE_OPTS:-op_pcent=7}
+      # Extra guest disks, ';'-separated QEMU -drive specs. A replay payload
+      # is far too large for a cloud-init seed and the guest has no network,
+      # so it comes in as a read-only disk and is copied in from inside.
+      FEMU_EXTRA_DRIVES: ${FEMU_EXTRA_DRIVES:-}
+      # FEMU_EXP_LOG switches the [EXP] log on; FEMU_SECRET is the marker
+      # string it looks for. Together they make FEMU report lpn -> PPA for every
+      # page whose content carries the marker, which is the only way to read the
+      # real address mapping instead of inferring it from read latency.
+      FEMU_EXP_LOG: ${FEMU_EXP_LOG:-}
+      FEMU_SECRET: ${FEMU_SECRET:-}
+      FEMU_DUMP_LPN: ${FEMU_DUMP_LPN:-}
+      FEMU_QMP_SOCKET: /data/qmp.sock
+      FEMU_ALLOW_UNPINNED: ${FEMU_ALLOW_UNPINNED:-}
+    command: ["${FEMU_MODE:-bbssd}"]
+    stdin_open: true
+    tty: true
+    stop_grace_period: 30s