test: design record on test fidelity + DD15 real-ddev migration loops - #87
Merged
Merged
Conversation
Why bugs pass unit + e2e and still hit real user terminals (stdin drains, rc-file hooks, isatty branches, signals), test double fidelity lessons from b14a198, patterns (behavioral doubles, real-binary tiers, stdin matrix, PTY harnesses), and a glossary.
test-environment-fidelity.md §4.2: the b14a198 stdin-drain bug lived in loops the real-binary suite never drove. New e2e-ddev section DD15 (two projects per loop — the bug only shows from the second on): - export loop, dev side: real 'ddev config' as dev (registry + dev-owned .ddev), real ddev-migrate export; starts fail by design (dev has no daemon) — asserts the loop REACHES both projects instead of breaking on the first failing child (loop survival). - import loop, opencode side (green path): fabricated dump dir in the real export layout, real ddev-migrate import; asserts both databases land (mark tables via export-db). Mutation-verified: stripping </dev/null from the import loop flips the checks to first-only import. Learned caveat (documented in code + both design records): a FAILING real ddev never reaches its stdin probing — the drain itself is only provable on the success path; the export part therefore covers loop survival, the unit DDEV_EAT_STDIN fake keeps the export call-site contract. e2e-ddev 70 (was 59); unit 23, lint green.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two related pieces, motivated by the recurring "green tests, broken production" pattern (most recently
b14a198: ddev consumed the ddev-migrate export loop's stdin — one database dump per run, only visible with real ddev on a real user's machine):1. Design record:
docs/design/test-environment-fidelity.mdReference on why kit bugs pass unit + e2e and still hit real terminals:
script -qec, pexpect, expect, tmux), session recordingIndexed in
docs/README.md.2. DD15: both ddev-migrate loops driven with REAL ddev (
tests/e2e/run-ddev.sh)Implements §4.2 of the record — the stdin bug lived in a loop the real-binary suite never drove. Two projects per loop (the bug only becomes visible from the second project on):
ddev configas the dev user (real registry, dev-owned.ddev), realddev-migrate export. Starts fail by design (dev owns no daemon in the container) — asserts the loop reaches both projects instead of breaking on the first failing child.ddev-migrate import; asserts both databases land (per-project mark tables verified viaexport-db).Mutation-verified: stripping
</dev/nullfrom the import loop's ddev call flips the checks to first-only import — the production failure mode.Documented caveat (code + both design records): a failing real ddev never reaches its stdin probing, so a fail-path scenario proves loop survival only — the drain itself needs the success path (import part), while the unit
DDEV_EAT_STDINfake keeps the export call-site contract locked.Verification
make e2e-ddev: 70 passed (was 59), 0 failed, 0 skipped — including a mutation run (2 expected FAILs) and a final green run after revertmake lint(ShellCheck),make check-versiongreenddev-e2e-test.md(status delta, DD0–DD15, 70 checks),test-environment-fidelity.md§4.2 (shipped note + caveat)