Feature/calibration driver - #35
Open
Tinitto wants to merge 16 commits into
Open
Conversation
Reviews the draft against qpi-driver, qpi-ui and the SDKs, and fixes what did not survive contact with them: - Dispatch: the RFC pushed CalibrateDispatch onto "the driver's NNG PUSH socket". That socket is a local in runDriverDispatcher and is fed only by scheduler.FetchNextJob; no HTTP handler can reach it. Replaced with a calibration_requests queue collection the dispatcher polls, which inherits the job path's durability and requeue-on-send-failure. - Persistence: handleCryostatReading appends to the shared `events` log and creates no per-type collection, so calibration_results was a departure from the pattern it cited, not an instance of it. Deferred to the events log for v1; recorded the model/migration/config/rules work for when it is earned. - Operation blast radius: an operation is closed across all three SDKs and the server (RFC 0003 §13.1), with tests asserting it. Listed every required change, including AllEventTypes, which driver registration validates against. - Citations: 1109.6887 is PRA 2012, not PRL 2011 (that is 1009.3639); rows 15 and 16 both pointed at the interleaved-RB paper. - Resolved the pyproject contradiction, dropped the test-py-calibrate target (test-py runs one env per extra; tier-1 needs no scheduler), and renamed monitor_interval so a calibrate option stops naming another operation. Adds the Decisions rationale and rejected alternatives, a Security section (the device YAML is a trust boundary: a tuner writes what every job then reads), and an implementation-plan pointer, per the conventions in docs/rfcs/README.md.
Takes the reviewed RFC 0004 over the branch's original copy; the former is the latter plus the corrections in 9c92ac2.
Replaces the scaffold on feature/calibration-driver with working routines, per RFC 0004. What was there reported success in 22 microseconds having run nothing, and destroyed the device config doing it. Routines - One backend-agnostic routine set replaces 32 per-scheduler stub files. Both schedulers expose the same gate vocabulary, so a routine composed against a SchedulerBackend runs under either; two copies would be two things to keep in step. tuners/base/device.py absorbs the shapes that do differ — qcodes callables against pydantic fields, and motzoi against beta. - All 16 routines now build real swept schedules and compile against the dummy cluster. Previously Rabi was a single fixed Rxy(90) with no sweep, T1 was a bare Measure, apply() was `pass` everywhere, and the qblox tuner passed None where a compiler was expected. - Fine amplitude gained the pi/2 pre-rotation and two calibration points. Without the former the sign of the error is unrecoverable at integer n; without the latter only the product of contrast and error is. Correctness of failure - A missing or empty calibration.yml no longer means "calibrate nothing and report success": absent routines default to enabled, an unknown routine name is a startup error, and a run with nothing to do reports failed. - Fits raise instead of returning zeros, and every fitted device parameter is range-checked against the sweep that produced it. A fit that returns zeros on failure gets written to the device as though it were a measurement. - The Clifford group is generated by exact matrix composition and closed under inversion, replacing a table with 8 of 24 entries and a "dummy inverse". RB measures nothing if the recovery gate does not actually invert the sequence. Write-back (RFC 0004 §10) - Writes the schema load_quantum_device reads, via safe_dump, verified to read back before anything is replaced, with the previous file kept beside it. It was writing a compilation config through yaml.dump, producing python tags the loader's safe_load refuses — one calibration bricked the QPU. - A failed run no longer writes at all. Driver - CalibrationResult is emitted flat, matching CalibrationResultPayload. Nested under "results" it unmarshalled to an all-zero record without erroring. - check_fidelity returns a report, so the drift path no longer dies on AttributeError every tick; the thresholds are now actually compared and a drop queues a partial recalibration. - recalibrate() honours its qubits argument rather than running the full DAG. - A second calibration is refused rather than queued behind a multi-hour run. Also: discovery resolves a Tuner named by import path, and the shipped example config names routines that exist. Deletes generate_quantify_tuner.py, a codegen script with a hardcoded home directory that shipped inside the package. Tests: 359 passing, 99% coverage on the gated modules against a 96% floor.
…tion The operation existed in the Python SDK and the server but nowhere else, and nothing could trigger a calibration: CalibrateDispatchPayload was declared and never constructed, with no route, no queue and no dispatcher change. Dispatch (RFC 0004 §6.8) - A `calibration_requests` collection the driver's dispatcher polls, rather than a socket write from an HTTP handler — the PUSH socket is a local inside runDriverDispatcher and nothing else can reach it. Queueing through the database is also what makes a request survive a restart, requeue on a send error as a job does, and run when an offline driver reconnects. - POST /api/op/calibrate/dispatch, admin-only like the /api/op/* routes around it: a calibration takes a QPU out of service for hours and rewrites the parameters every later job runs against (RFC 0004 §10). - Calibrations are offered before jobs, or a busy QPU would never calibrate — the state calibration exists to get it out of. One at a time per driver. Result handling - handleCalibrationResult validates mode and status, as handleCryostatReading validates its readings, and records the QPU as well as the driver. A blank payload was previously stored as a blank row. - The report closes out the queued request it answers, so the dispatcher can offer the next one. Cross-language parity (RFC 0003 §8, RFC 0004 §6.1) - `calibrate` added to the Go and TypeScript SDK operation enums and event types. Their closed-set tests failed on the addition, which is what those tests are for; all three now assert the same three operations. Also: the catalog's drift option is `drift_check_interval`, since `monitor` is already an operation; a new test pins the catalog's -o keys to the ones the Python builder reads, because an option nothing reads is a startup error and the two lists are kept in step by hand (RFC 0003 §9). Drops the vestigial `calibration_data` field from the dashboard's QPU type — no collection has it.
Records the two design changes the implementation forced, with reasons: - One shared routine set rather than one per scheduler (§6.2). Both expose the same gate vocabulary, so the seams that carry the real differences are a SchedulerBackend and a device-parameter shim, not sixteen duplicated files. - calibration_results is a collection after all (§6.8), reversing the earlier deferral to the events log. The dashboard history list is a query over months rather than a tail of recent events, and a calibration report's retention is different in kind from a cryostat reading's. Also rewrites §6.3/§6.4 to the interfaces that exist (check_fidelity returns a report; routines declare "updates" and "benchmark", and analyse raises rather than returning zeros), marks tier 3 as not implemented, and adds §11 saying plainly what is built and what is not — above all that no routine here has been run against physical hardware. Adds the tuners doc page the mkdocs nav already pointed at, and drops the test-py-calibrate target, which ran an import check and was not in test-py. The calibration tests live in the existing per-extra targets: tier 1 needs no scheduler, tier 2 needs the one it tests.
Admin-only tab showing what the chip currently looks like and letting an operator act on it: - Measured fidelity per target, each against the threshold that governs it — two-qubit for an edge, one-qubit for a qubit. Holding a CZ to the 1Q threshold would make the whole grid cry wolf. Worst protocol wins per target, as the driver's own drift check does. - Current parameters, grouped by qubit rather than by routine, because "what does q0 look like now" is the question an operator actually has. Values are formatted by kind: GHz for frequencies, µs/ns for times. - History, with per-run status, duration and error counts. - A trigger modal defaulting to the drift check rather than the full run: it is the common case, it is cheap, and it changes nothing, whereas a mis-clicked full calibration costs hours of QPU time. A partial run with no targets is refused in the form as well as at the endpoint, since the driver would otherwise widen it to a full one. - An in-flight banner driven by calibration_requests. A calibration takes hours, so the queued request is the only thing that can say "in progress" during the gap before a report exists. Registration now offers the two tuner kinds, and the sidebar and hash router know the tab — the E2E spec caught the missing hash allow-list entry, which is what it is there for. make test-e2e-dashboard gains SPEC=<glob> to run one spec: eight seconds against four minutes when iterating on a single tab. E2E: 116 passing, including 6 new.
The page is symlinked into docs/driver/, so relative links resolve from there rather than from the package — mkdocs strict mode caught both. Uses absolute GitHub URLs, as the other double-duty READMEs do. Also rewords the RFC's note about there being no dedicated calibration test target, which check_docs.py read as a claim that one exists.
The merged docs described an endpoint and an access rule that do not exist, and gave two CLI examples that would fail on startup. Corrected against the code: - `POST /api/calibrate` is `POST /api/op/calibrate/dispatch`. - `calibration_results` is authenticated-read, not public read, and `calibration_requests` was missing from the collections table entirely. - Both tuner examples omitted `calibration_config`, which a tuner cannot start without — a calibration with nothing to run would report success having measured nothing, so its absence is deliberately a startup error. Adds the usage that was missing: how to queue a calibration from the server and what the three modes cost, how to turn on drift monitoring, a Tuner Backends section in the Python SDK README, and a calibration section in the operations runbook covering the queue, the `.prev` backup as the way back from a bad calibration, and why `eventsRetention` does not touch calibration reports. The changelog entry now covers what actually shipped — the dispatch endpoint and its queue, drift monitoring, the dashboard tab, the SDK operation enums and the write-back guarantees — and states plainly that none of it has been run against physical hardware. make test-docs passes: 137 claims checked, site builds under strict mode.
…er 3) Tier 3 was specified and never built. The gap it leaves is not cosmetic: tier 1 generates its data from the same analytic form the fit assumes — a decaying cosine in, a decaying cosine fitted — so it proves the optimiser converges but not that the model is right, and tier 2's dummy cluster returns no data at all. tests/simulation.py builds a transmon by diagonalising a real Cooper-pair-box Hamiltonian with scqubits and integrates the Lindblad master equation with qutip. Nothing is generated from a fitting model, so a routine that recovers the simulator's parameters has been tested against physics rather than itself. Covered: qubit spectroscopy against the true eigenspectrum, Rabi, T1, T2 echo, Ramsey, and randomized benchmarking against real Clifford unitaries. The RB test is the strongest in the suite. Sequences are composed from this package's own Clifford decomposition, evolved as unitaries, closed with the computed recovery gate and depolarised by a known amount; a control test proves the decay disappears when the recovery gate is removed, so a fit returning a plausible number on flat data cannot pass. Tolerances are set from measured accuracy, not by feel, because a loose one makes a test that passes without discriminating: fitting a Gaussian decay to this simulator's exponential relaxation still recovers T1 to within 7%, so the 15% I first wrote accepted the wrong physical model. The correct model recovers T1 to 0.01%, the Ramsey fringe to 0.004% and f01 to 3 kHz; each tolerance is set just above that. Verified by mutation: swapping the exponential decay for a Gaussian passes tier 1 and fails tier 3, which is the whole argument for the tier existing. Three modelling errors of mine that the tests caught, all fixed here: qutip 5 makes e_ops keyword-only; pulses built from a + a† leak into the second excited state, so they are now built on the 0-1 subspace as a calibrated pulse is; and shot noise that did not fall as 1/sqrt(N) made averaging decorative and left a 0.2% gate error unresolvable no matter how many sequences were run. Dependencies are a `sim` group rather than an extra — nothing shipped to a user needs them — and the tests skip when it is absent. `make test-py-sim` runs them, and it is in `make test-py`. Documents what the tier does not cover: the simulator supplies the acquisition rather than interpreting the compiled schedule, so a schedule that does not produce the physics its fit assumes would still pass. Hardware remains the only answer for that.
The tests so far took a calibration apart: the DAG with stub routines, the schedules against a dummy cluster, each fit against physics. Every one of them replaced the pieces either side of the one under test, so nothing covered the joins. Put it back together. The simulator behind a SchedulerBackend gives the routines a `run` to call, and behind the real Tuner gives a calibration everything it needs bar a scheduler — so `_execute_calibration`, the calibrate worker's own entry point, can be handed a job dict and driven through the DAG walk, the fits, the write-back and the report. It found two faults, and both are fixed here: fit_rb_decay bounded the model's amplitude to ±2. That suits a raw survival probability but not the rescaled signal the `rb` routine hands it: a decay that has not reached its asymptote by the deepest sequence has an amplitude far larger than the range observed, so the bound was met by pulling the decay rate down instead. The reported fidelity pinned near 0.98 for any chip better than about 3% error per Clifford — below the 0.999 default drift threshold, so a healthy chip would have recalibrated on every check, forever. Only the decay rate is bounded now; it is the only parameter with a physical range. Tuner.recalibrate never narrowed its routines. CalibrationDAG.partial_order was written and unit-tested but left unwired, so a partial recalibration ran the whole enabled graph over fewer targets — including the readout bring-up, which is most of the cost a partial run exists to avoid. It now runs RECALIBRATION_ROOTS and everything downstream. Also moves the simulator into tests/fixtures/ with the backends and tuner built on it, and corrects the RFC: tier 3 needs no scheduler extra, only the sim dependency group.
`InstrumentCoordinator.components` is a qcodes ManualParameter holding component
*names*. Both close() paths treated it as an iterable of component objects, so
each raised TypeError before releasing anything:
TypeError: 0 is not a list; Parameter: quantify_ic.components
QuantifyExecutor.close() is called by the QPU worker on shutdown (qpu.py:276)
and QuantifyTuner.close() by the calibrate worker (calibrate.py:362), so in both
cases a driver stopping left its instruments held against the next driver that
wanted them.
The suite did not catch it. test_tuner_routines' module fixture calls close() in
teardown, but a test earlier in the same module calls Instrument.close_all() —
so by teardown the coordinator is already gone and the loop never runs, which is
the one arrangement in which a broken shutdown looks fine. Both regression tests
therefore build their own live instance, and both fail against the old code with
the TypeError above.
Detaching by name is enough: close_all() closes the components themselves, so
looking each one up to close it individually only duplicated the next step.
quantify reaches its hardware through four calls — prepare, start, wait_done, retrieve_acquisition — and both QuantifyBackend.run and QuantifyExecutor._run_circuit go through exactly those. So SimulatedCoordinator answers them and drops in wherever a cluster does, via `is_simulated=True` on either the tuner or the executor. The difference from the dummy cluster is that this one reads the schedule it was given. A dummy returns nan+nanj regardless, so the calibration a tuner writes and the pulses an executor plays were never actually connected: an amp180 off by a factor of ten compiled, ran, and returned the same nothing. Here the compiled schedule's own G_amp is the amplitude that rotates the state, so a wrong calibration is a wrong population, measured. That closes the loop between the two operations, and tests/test_calibration_loop walks it with nothing faked in the middle: a QuantifyTuner calibrates the simulated chip, writes a real quantify.device.yml, and a QuantifyExecutor loads that file and runs circuits on the same chip. The fixture device starts 214 MHz off resonance, where an X gate does nothing — so an X that lands in |1> is evidence a number survived the fit, the write-back, the YAML, the loader and the compiler with its meaning intact. A negative control asserts the uncalibrated device still gets it wrong. Three things the loop had to get right, each of which looked like a broken routine until it was found: - Clock frequencies come from `resources` once, but SetClockFrequency retunes them mid-schedule, and that is how every spectroscopy routine sweeps. Without replaying the retunings the whole sweep sits at one point. - Idles and pulses are propagated by exponentiating the Liouvillian rather than by stepping an ODE. A Reset is a 200 us idle, and at a few hundred MHz of detuning that is ~1e5 radians of phase, which exhausts any step budget. - The drift Hamiltonian's detuning sign pairs with the drive's phase convention. Ramsey is the experiment that distinguishes them: with this sign a device set 5 MHz off is pulled back to within 5 Hz, with the other it "corrects" to 2 MHz off — twice the artificial detuning. Not modelled: two-qubit gates. A CZ is a flux pulse whose area sets the conditional phase, so a flux pulse raises rather than silently acting as an identity and making a Bell state look merely broken. The simulator moves from tests/ into the package, since a thing you can put where the hardware goes is not a test fixture. scqubits and qutip stay lazily imported and in the sim dependency group, so a base install is unaffected.
Both operations read it, so pointing a calibrate driver and a process driver at the same quantify.device.yml runs a whole node with no instruments: the tuner calibrates the simulated chip, and the QPU beside it compiles jobs against what the tuner wrote. Distinct from is_dummy, which is the vendor's dummy cluster: that compiles and runs and then returns nan for every acquisition, so nothing tells a correct calibration from a wrong one. qblox raises rather than accepting the flag. qblox-scheduler reaches its hardware through a HardwareAgent that compiles and runs in one call, not through quantify's four-call coordinator, so the simulator does not drop in there — accepting it and quietly falling back to a dummy cluster would return nan and look like a chip that answered.
A Lorentzian narrower than the spacing between setpoints did not appear in the data. Whatever the fit converged on came from the noise between the points, and it arrives with a small linewidth and a confident centre — the worst shape a wrong answer can take here, because that centre is written straight to the device as f01 and every gate afterwards is driven at it. Found while trying to make spectroscopy more precise: narrowing the line to 63 kHz while the sweep still stepped 5 MHz produced a frequency 377 MHz from the qubit, fitted tidily, reported as success. The precision attempt itself is not kept, and the reasoning is recorded in the routine because it is not obvious. A long square saturation tone is what textbook two-tone spectroscopy uses and it does resolve the line far better — 63 kHz against 13.6 MHz, measured. But this routine's job in the graph is to *locate* a qubit across hundreds of MHz, and a square tone's frequency response is a sinc: over a coarse sweep the Lorentzian fit catches a side lobe and lands 91 MHz out. The DRAG envelope is smooth and has no lobes to catch. Precision is not lost by keeping it, only delegated — `ramsey` runs after `rabi` and refines f01 to hertz, which is what the dependency order already said.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements RFC 0004: Calibration Tuners, bringing automated parameter calibration and drift tracking for superconducting transmon qubits to QPI.
It introduces the
calibrateoperation across the full stack—from Python SDK calibration tuners (quantify_tuner,qblox_tuner) and backend-agnostic experiment routines to server-side event handling, atomic device file write-backs, and an admin-facing dashboard UI for QPU calibration status and manual triggers.What's Changed
1. Operation & Event Infrastructure
calibrateas a first-class operation alongsideprocessandmonitor.CalibrateDispatchandCalibrationResultevent types.POST /api/op/calibrate/dispatchendpoint inqpi-uifor triggering full, partial, or fidelity-check calibrations.calibration_resultsandcalibration_requestsserver storage collections.2. Python SDK & Tuners Framework (
qpi-driver/py)CalibrateDriver: Worker subprocess architecture handling calibration runs without blocking the main event loop.Tuner&CalibrationDAG: Topological DAG execution of routines configured viacalibration.yml.quantify-schedulerandqblox-schedulervia a unifiedSchedulerBackendinterface.quantify.device.ymlwith automatic backup (.prev) and verification before swapping files. Fits outside expected parameter bounds or failed acquisitions abort writeback to prevent corrupting hardware state.drift_check_interval) triggering targeted partial recalibrations when gate fidelity drops below configured thresholds.3. Admin Dashboard UI (
qpi-ui)calibration_requests.4. Testing & Verification
test_fitting.py): Validates parameter recovery from noisy synthetic data and verifies error-handling when fits fail.test_tuner_routines.py): Verifies schedule compilation for all routines against Quantify and Qblox dummy clusters.make test-docsclaims pass strict MkDocs verification.Documentation
qpi-driver/py/qpi_driver/tuners/README.mdand linked it in MkDocs navigation (docs/driver/tuners.md).qpi-driver/py/calibration.example.yml.calibrateCLI examples, queue mechanics, and rollback procedures.How to Test