ci: stop re-downloading SDK images after docker load

[satwiksps]

docker_image_from_buildcache() decides whether the local SDK image is current by comparing its image ID against the .id file on bincache. that .id comes from docker image inspect in CI, but docker load derives its own image ID on the consumer side and only matches when both daemons use the same storage driver. where they differ the check never passes, and run_sdk_container re-downloads the multi-GB tarball every time it creates a container.

The earlier version of this PR cached the loaded ID locally and returned early on it, which skipped the remote .id entirely and would have missed a rebuilt image, so that's gone.

The remote .id is now fetched on every run and stays what we compare against. after a load, <name>-<version>.id.local records the published ID we loaded from plus the ID our own daemon assigned to the image. It only counts as current when both still match, published ID unchanged means bincache hasn't rebuilt it, local ID unchanged means nothing replaced it locally. the direct image_id == remote_id comparison still runs first, so daemons where the IDs do line up are unaffected, and with no stamp file present the behaviour is what it is today. *.id.local is added gitignored since run_sdk_container runs from the repo root.

[zhijianli88]

But a docker save / docker load round-trip can produce a different image ID on the consumer's Docker daemon (e.g., due to different Docker versions or storage drivers between CI and the developer's machine)

Do you have any more concrete evidences?

<.id> is used to verify the image integrity, if the <.id> is untrustable, I'd like to choose another method

[John15321]

@zhijianli88 We are currently experiencing a lot of AI contributions, if you see anything like that just close it

[satwiksps]

@zhijianli88 @John15321 this is documented upstream behavior: moby/moby#51934.
few months ago I came across above docker issue and learnt this when a maintainer there said "The ID will be persistent only if you make sure that both hosts have the same storage driver."
IMO since flatcar CI and developer machines can have different backends, the .id from bincache won't always match the local image after docker load, even though the content is the same. I thought I could fix it in this way, there could be other way too like comparing layer digests which are consistent across storage backends ig...so yeah I gave the issue flatcar/Flatcar#2086 a go, my fix caches the locally loaded ID after docker load.

Also this PR was purely from my own knowledge, though could be wrong, I have not used any AI 🙂

[zhijianli88]

@zhijianli88 @John15321 this is documented upstream behavior: moby/moby#51934. few months ago I came across above docker issue and learnt this when a maintainer there said ["The ID will be persistent only if you make sure that both hosts have the same storage driver."]

Make sense.

However, the .id is used to ensure image integrity and your patch doesn't address this critical aspect

[satwiksps]

However, the .id is used to ensure image integrity and your patch doesn't address this critical aspect

My prev approach cached the loaded ID locally and returned early on it. I understood that once a local copy existed, the remote .id was never looked at again. Rebuilt image on bincache would have gone unnoticed

In my second approach the remote .id is fetched on every run again and stays what we compare against. After a load I record two values in -.id.local, the published ID we loaded from & the ID our own daemon gave the image. it only treated as current when both still match, so the published ID still has to agree (no silent rebuild slips through) and the local ID catches the image being replaced underneath us. image_id == remote_id still runs first, so nothing changes where the IDs already line up. Only assumption dropped is that docker load reproduces the ID docker image inspect printed in CI, which is the part that doesn't hold across storage drivers. I hope this covers the integrity side properly now