Skip to content

Latest commit

 

History

History
78 lines (52 loc) · 4.54 KB

checking-api-after-refactor.md

File metadata and controls

78 lines (52 loc) · 4.54 KB

Checking API behaviour after a refactor

Notes from moving our orders service off a hand-rolled data layer onto an ORM. The goal was "nothing changes for clients." That turned out to be harder to prove than to say.

The problem

The refactor touched about 60% of the service. Unit tests mostly got rewritten along with the code, so they couldn't tell us much. A test you rewrote alongside the thing it tests will happily agree with your new bugs.

What we actually needed to know was simpler: for the same request, does the API still return the same thing? That's a black-box question, so it needs black-box checks. If you want a short primer on what is functional testing and why it sits apart from unit tests, that link covers it. Below is what we did in practice.

Step 1: freeze the current behaviour first

Before touching any code, we captured responses from the old version for a fixed set of requests. Nothing fancy, just a script and a folder of JSON files.

# run against the old build, commit the output
while read -r path; do
  name=$(echo "$path" | tr '/?=&' '____')
  curl -s -H "Authorization: Bearer $TOKEN" \
    "http://localhost:8080$path" | jq -S . > "snapshots/$name.json"
done < requests.txt

jq -S sorts keys. Without it you get diffs just because the new serializer orders fields differently, and you'll waste an afternoon on noise. Ask me how I know.

requests.txt had about 120 paths. We picked them from access logs, not from the API docs, because the docs don't tell you which weird query params real clients actually send.

Step 2: diff, then decide

After the refactor, run the same script against the new build into a second folder and diff:

diff -r snapshots/ snapshots-new/ | less

First run: 31 files differed. Sorting those out was the real work.

  • 14 were timestamp precision. The old code returned seconds, the ORM returned milliseconds. Technically a breaking change. One mobile client parsed it with a strict format and would have crashed.
  • 9 were null vs missing field. Old code omitted empty fields, new code sent "discount": null. Mostly harmless, but one partner integration treated presence as "has a discount."
  • 5 were ordering. List endpoints without an explicit ORDER BY came back in a different order. The old order was an accident of the old query, but clients relied on it.
  • 3 were real bugs. One was a join that dropped orders with no shipping address.

Only 3 of 31 would have failed an assertion in the kind of functionality testing we already had. The rest were "correct" by our own spec and still would have broken someone.

Step 3: turn the snapshots into real tests

Diffing folders by hand is fine once. For the next few weeks of follow-up changes we wanted it in CI, so we turned the snapshots into parametrized tests that ignore volatile fields:

IGNORE = {"updated_at", "request_id", "trace_id"}

def strip(obj):
    if isinstance(obj, dict):
        return {k: strip(v) for k, v in obj.items() if k not in IGNORE}
    if isinstance(obj, list):
        return [strip(x) for x in obj]
    return obj

@pytest.mark.parametrize("path", load_paths("requests.txt"))
def test_response_matches_snapshot(client, path):
    expected = load_snapshot(path)
    actual = client.get(path).json()
    assert strip(actual) == strip(expected)

Keep the ignore list short and reviewed. Every field you add to it is a field you've stopped checking.

Things I'd do differently

  • Capture write endpoints too. We only snapshotted GETs and a bug in order cancellation got through. For writes you need a known starting state, which means a seeded database per run.
  • Snapshot with more than one user role. All our captures used one test account, so permission differences were invisible.
  • Don't keep snapshot tests forever. Once the refactor settled, about half of them were replaced with normal assertions. Snapshot tests are great at "did anything change" and bad at "is this correct," and after a while the second question matters more.

Related tools

You don't need anything special for this, which is part of why I like it. curl, jq and pytest got us through. There are tools that record real traffic and replay it as tests, and they're worth a look if you can't easily build a request list by hand. For a one-off refactor on a service with decent logs, the script approach was enough.

The one rule I'd keep: freeze behaviour before the first commit of the refactor, not after. Once the old build is gone, you've lost the only reliable reference you had.