[VPDPersonal]

Experiment, not for merge: can the official Unity CLI replace game-ci/unity-test-runner in the Unity EditMode job of ci.yml, and does the job get simpler or faster?

Recommendation: stay on GameCI. The CLI job works on all 7 Unity versions with the Personal license, and the tests themselves run much faster on the bare runner. But the job is not simpler, and it is faster only with the Editor cached: 7 cached Editors need about 33 GB, and the repository cache holds 10 GB. Without the cache, the matrix takes 15:14 against 10:37 for GameCI. Details and a possible hybrid are below.

unity-cli-experiment.yml runs on this PR only. Its file name does not match the Unity plan regex of ci.yml, so the GameCI matrix skips here and the two never compete for Personal seats.

What the experimental job does

On a bare ubuntu-latest runner, without a container:

1. Installs the CLI with install.sh, pinned by UNITY_CLI_VERSION=1.0.0-beta.13. There is no stable channel yet.
2. Restores the installed Editor folder from actions/cache. On a miss it runs unity install <version> --yes --accept-eula and saves the folder at once. In this experiment only 6000.0.53f1 is saved.
3. Builds the throwaway project with scripts/make-unity-test-project.sh.
4. Takes the Personal seat with the Licensing Client bundled in the Editor: Unity.Licensing.Client --activate-all --include-personal --username --password. It succeeds only on a Seat ID: line and retries twice.
5. Runs xvfb-run unity test ci-unity-project --editor-path … --mode EditMode --report-format nunit,junit --timeout 1200 --format github -- -aspidFastToolsStrictSetup -logFile …. Exit 8 fails the job. Exit 6 gets one more attempt.
6. Returns the seat with Unity -returnlicense -username -password in an always() step, then checks unity license status.
7. Removes the account e-mail from the Editor logs and uploads the NUnit, JUnit and Editor logs.

No extra apt packages were needed for any 6000.x Editor. The unity ci init template installs 29 of them.

Timings

6000.0.53f1, per step

GameCI: main run 38788273451 (job 116605893901). CLI warm: 38816519929. CLI cold: 38819031692, attempt 2, with the cache entry deleted first.

Step	GameCI	CLI, cold	CLI, warm
Set up job + checkout	0:04	0:03	0:02
Install the CLI (install.sh)	—	0:13	0:12
Get the Editor	image pull 1:40	unity install 11:17 (download 7:48, unpack 3:20); 9:24 on an earlier cold run	cache restore 1:13 (4.5 GB at 83 MB/s ≈ 0:55, unpack 0:18)
Save the Editor cache	—	0:35	—
Activate the Personal seat	0:04	0:03	0:03
Editor start, import, compile, tests	2:19 (test run 106 s)	1:41 (test run 75 s)	1:52 (test run 79 s)
Return the seat	0:03	0:03	0:04
Upload	0:01	0:01	0:02
Job	4:19 (5:22 on run 38769879616)	14:00	3:31 (3:29 on run 38819031692)

All versions

GameCI: the same main run 38788273451. CLI: run 38819031692, cold for every version except the cached 6000.0.53f1.

Version	GameCI job	GameCI image pull	GameCI test run	CLI job	CLI download	CLI unpack	CLI test run	Installed size
6000.0.53f1	4:19	1:40	106 s	3:29 (warm)	—	—	79 s	7.9 GB
6000.1.17f1	6:48	2:46	176 s	11:35	6:28	3:00	63 s	7.9 GB
6000.2.15f1	5:00	1:49	133 s	15:13	9:19	3:30	82 s	8.0 GB
6000.3.26f1	10:19	1:21	448 s	7:22	1:19	3:33	77 s	8.2 GB
6000.4.12f1	10:36	1:19	454 s	11:36	5:46	3:10	81 s	7.8 GB
6000.5.11f1	10:14	1:27	428 s	11:35	7:15	2:30	51 s	8.6 GB
6000.6.5f1	9:36	1:25	391 s	6:17	0:22	3:19	79 s	8.8 GB
Matrix	10:37			15:14

What the numbers say:

• Getting the Editor. The Unity CDN download is the cold bottleneck, and its speed varies widely: 0:22 to 9:19 for archives of the same size. Unpacking takes another 2:30–3:30. A GameCI image pull takes 1:19–2:46.
• The tests themselves run 1.3–8.4× faster on the bare runner: 51–82 s against 106–454 s. On 6000.3–6000.6 this costs GameCI 5–6 minutes per job, so those jobs set the matrix time. The Editor logs of the two runs differ in the software renderer: llvmpipe on LLVM 15 in the unityci/editor image, LLVM 20 on the ubuntu-24.04 runner. That is the likely cause, not verified. This gain comes from running outside the GameCI container, not from the CLI.
• With the cache warm, a CLI job takes about 3:00–3:45 (CLI 0:12, restore ~1:10, tests 1:20–2:10). A fully cached matrix would take about 4 minutes against 10:37.

The cache

• One installed Editor is 7.8–8.8 GB, and the cache entry for 6000.0.53f1 is 4.5 GB. Of the 7.9 GB, Data/Resources/PackageManager is 4.5 GB: Editor 2.0 GB (bundled package tarballs), BuiltInPackages 1.4 GB, ProjectTemplates 1.0 GB. Removing ProjectTemplates would save about 1 GB per entry, since those tarballs are already compressed. The rest is needed to resolve the test project.
• All 7 versions need about 33 GB, or about 26 GB trimmed. The repository limit is 10 GB, and other caches use about 0.7 GB today. Over the limit, GitHub evicts entries in LRU order, so most runs would be cold. Storage above 10 GB is only available as a paid extension.
• Cache scope: a PR can restore only main's entries and its own. The entries would have to be saved on push to main, and a PR miss should not save, or every PR writes another 4.5 GB.
• Caching ~/.config/unityhub/downloads (the unity ci init default) would skip only the download. The 2:30–3:30 unpack would remain.

The Personal license

• Activation through the bundled Licensing Client takes 3–4 s, the same as GameCI.
• All 7 jobs activated one account at the same time and each got a seat: 4 with ASSIGN_SEAT and 3 with SameMachine, so runner machine ids sometimes repeat. The "assigned no seat" case that ci.yml retries did not occur.
• Return: every job handed its seat back, and unity license status showed "none active" afterwards. Neither CLI route can return a Personal seat:
  • unity license return --yes: "Some licenses, such as free Personal licenses, can't be returned."
  • Unity.Licensing.Client --return-ulf: "Ulf license file not found … (1404)". GameCI's log shows the same 1404, and GameCI then falls back to the Editor.
  • Only Unity -returnlicense -username -password works, which is what GameCI does too.

Simpler?

No. The YAML and bash we would own:

	GameCI job (ci.yml)	CLI job
Lines (non-comment)	76 (59)	152 (120), of which 59 lines of bash
Activation and return	inside the action	ours: undocumented client flags, Seat ID: check, retries, Editor -returnlicense with the password
Editor	Docker image per version	ours: install, cache key, save policy, 10 GB budget
Graphics device	the image wraps the Editor in xvfb-run	ours: xvfb-run. -nographics fails 68 tests with "No graphic device is available to initialize the view"
Test results check run	posted by the action	lost; it needs a separate JUnit reporter action
Failed tests vs. broken run	heuristic: was a results file written?	exit 8 vs. exit 6

What the CLI gives:

• exit 8 for failed tests and 6 for no verdict, so only infrastructure failures are retried;
• --report-format nunit,junit;
• --retries N, which reruns failing tests and reports flaky ones separately;
• --shard N/M (not needed for a 1–2 minute suite);
• --format github annotations;
• --timeout;
• no Docker image per version; unity install takes any version and module from the CDN.

What we would maintain ourselves:

• seat activation and return;
• retries and backoff;
• the cache and its budget;
• xvfb-run;
• removing the account from uploaded logs;
• a check-run reporter;
• the pinned CLI version.

The CLI is still a beta. install.sh is unpinned and fetched from unity.com, and it checks the binary against a checksum from the same CDN.

Findings along the way:

• unity install-path --set rejects a path that contains .. (exit 6, "This path is not valid.").
• unity ci init --provider github writes a workflow with a serial license, 29 apt packages and a downloads cache. Its own comment says unity license activate --personal does not work in CI.

Recommendation

Stay on GameCI for the required matrix:

1. Not simpler. It is twice the YAML, and the hardest parts move to us: seat activation and return with undocumented client behaviour, the cache, and xvfb-run.
2. Not faster when cold. The CDN download varies from 0:22 to 9:19, so the cold matrix takes 15:14 against 10:37.
3. Faster only when warm, at about 4 minutes. That needs about 33 GB of cache against a 10 GB limit.
4. The CLI is a beta.

The speed lever this experiment found is elsewhere. The EditMode tests of 6000.3–6000.6 take 6.5–7.5 minutes inside the GameCI container and 1–1.5 minutes on the bare runner. Two next steps follow from that:

• Find out why the tests are slow in the unityci/editor image (LLVM 15 llvmpipe is the first suspect).
• Try a hybrid: take the Editor from the GameCI image instead of the CDN (docker create + docker cp /opt/unity: a fast Docker Hub pull and no cache budget), then run it on the bare runner with unity test. This was not tested here.

Revisit a full switch when the CLI is stable, or when the cache budget can hold the Editors.

🤖 Generated with Claude Code