Skip to content

ci: pin the mssql test container to the 2022 line - #6227

Open
prql-bot wants to merge 2 commits into
mainfrom
fix/ci-32660412890
Open

ci: pin the mssql test container to the 2022 line#6227
prql-bot wants to merge 2 commits into
mainfrom
fix/ci-32660412890

Conversation

@prql-bot

Copy link
Copy Markdown
Collaborator

Problem

tests failed on main in run 32660412890 with Error: Timed out waiting for: tcp:1433 — the same test-rust (x86_64-unknown-linux-gnu, ubuntu-24.04, default,test-dbs-external,lsp) failure #6217 added the container dump for. That dump ran for the first time here, and it answers the question #6217 deliberately left open: mssql crashed, it wasn't slow and it wasn't starved for disk.

Docker Hub's mcr.microsoft.com/mssql/server was untagged, so it resolved to latest — which is now byte-identical to 2025-latest (both sha256:4bab24f3…) and labelled com.microsoft.version: 17.0.4075.5, i.e. SQL Server 2025. CI moved to a new major version when the tag rolled over, without a commit. Upstream has open reports of exactly this: microsoft/mssql-docker#954 (2025 RTM/CU1/CU2 crash at SQLPAL startup on hosts whose CPU topology fails an internal assertion, "2022-latest works fine on the same host") and #940 (2025-latest faults in sqlpal.dll; 2025-CTP2.0 fine). Host-dependence explains the shape here — intermittent, and a re-run on different hardware passes.

This is the third occurrence in three days, so it's past the point of being written off as a blip.

Evidence from the dump

The wait started at 19:14:23 and gave up at 19:15:23, but mssql was already dead by then — the container started at 19:12:10 and the crash timestamp is 19:12:12, two seconds in:

dbs-mssql-1  mcr.microsoft.com/mssql/server  "/opt/mssql/bin/laun…"  mssql  3 minutes ago  Exited (1) 3 minutes ago
mssql-1 |          file://package6/windows/system32/sqlpal.dll+0x000000000012260B
mssql-1 |          file:///windows/system32/ntdll.dll+0x00000000001066AF
mssql-1 |          file:///Windows/SYSTEM32/lsasrv.dll+0x00000000000AB7F9
mssql-1 |     Process: 10 - sqlservr
mssql-1 |  Build stamp: 30e43f7bebb02a8b9d405ff3f9f2c5c93cb26d20e0ee95b4522fac211a19fac7
mssql-1 | Distribution: Ubuntu 24.04.4 LTS
mssql-1 |   Timestamp: Sun Aug 23 19:12:12 2026
mssql-1 | Capturing a dump of 10
mssql-1 | Executing: /opt/mssql/bin/handle-crash.sh with parameters

The two hypotheses #6217 wanted to separate are both ruled out. Disk: /dev/root 145G 75G 70G 52% / — 70G free. Starvation/slowness: the other three containers were Up 3 minutes with their ports mapped, and mssql never reached the point of listening at all.

Occurrences, all tcp:1433, all on the same matrix leg, none in the ~3 months before:

Run When Branch
32428511588 2026-08-20 23:24 main (re-run passed; this is #6217's trigger)
32623545342 2026-08-23 06:40 ci/snap-toolchain-pin
32660412890 2026-08-23 19:10 main

Solution

Pin the mssql service to mcr.microsoft.com/mssql/server:2022-latest (16.0.4265.3, pushed 2026-07-08 — still actively updated). The other three services are untouched; none of them failed.

This is at the tag level rather than the digest level deliberately. 2022-latest still floats, so cumulative updates keep arriving; what it stops is the major-version jump happening silently, which is what the file's existing "these aren't tagged yet, since there's no dependabot support for docker-compose yet" comment left open. It matches the precedent already in the file for clickhouse.

A restart: policy was the other candidate — the containers come up ~2 minutes before the wait, so a crashed mssql would have time to retry. I didn't take it: it papers over the crash rather than avoiding it, and it makes an image that can't start look like one that's merely slow, which is the exact ambiguity #6217 was written to remove.

Testing

The mssql leg only runs in CI, so the substantive verification is this PR's own test-rust (x86_64-unknown-linux-gnu, ubuntu-24.04, default,test-dbs-external,lsp) job — it starts the pinned container and runs the full external-DB suite against it. Locally: yaml.safe_load parses the file with the *vol anchor still resolving on all four services, and prettier --check passes.

Nothing in the test code is version-specific — MsSqlTestRunner just connects on 1433 — and this restores the major version the suite ran against for years before the tag rolled over, so I'm not expecting result differences. If any surface, they'd show up on that job.

One thing the dump didn't give us, for the record: docker compose logs --tail=200 is per container, and mssql's crash handler emits several thousand lines of find: … Permission denied, so the tail window held only the crash report — the startup lines above it were cut. Enough to diagnose this, but the exception header itself never printed.


Automated fix for failed run

@prql-bot prql-bot left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the premise independently: mcr.microsoft.com/mssql/server:latest and :2025-latest are the same digest (sha256:4bab24f3…), while :2022-latest is sha256:ba4c8329… with com.microsoft.version: 16.0.4265.3, created 2026-07-08 — so the pin does what the description says and the 2022 line is still receiving updates. The compose file still parses with the *vol anchor resolving on all four services, 2022-latest is a single-platform amd64 manifest so the existing platform: linux/amd64 stays correct, and nothing in MsSqlTestRunner (BULK INSERT … FORMAT = 'CSV', Dialect::MsSql) is version-gated.

One fix on the comment, inline: mssql-docker#954 is closed (as a duplicate of #951, which is open and titled with the exact sosnumap.cpp Line=208 assertion, on an Ubuntu 24.04.4 LTS host — the same distribution string as our crash dump). Since this comment is the durable record of why the pin exists, a reader following it lands on a closed thread and may reasonably conclude the problem is fixed upstream.

The test-rust (x86_64-unknown-linux-gnu, …, test-dbs-external, …) leg is the real verification and is still in progress; I'll report its result on this thread.

Comment thread prqlc/prqlc/tests/integration/dbs/docker-compose.yaml Outdated
@prql-bot

Copy link
Copy Markdown
Collaborator Author

The verification leg came back green: test-rust (x86_64-unknown-linux-gnu, ubuntu-24.04, default,test-dbs-external,lsp) passed, so the pinned 2022-latest container starts and the full external-DB suite runs against it with no result differences. Every other gating check on 2197b05d is green too.

microsoft/mssql-docker#954 is closed as a duplicate of #951, which is the
open tracking issue for the SQLPAL startup assertion. Point the comment at
the live thread so a reader doesn't take "closed" for "fixed".
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant