[!!!][FEATURE] Share test instances between identically configured test cases - #740
Open
bmack wants to merge 2 commits into
Open
[!!!][FEATURE] Share test instances between identically configured test cases#740bmack wants to merge 2 commits into
bmack wants to merge 2 commits into
Conversation
…st cases Functional test instances are identified by sha1(static::class), so every test case class provisions its own instance: its own directory tree, its own compiled dependency injection container and its own database schema. Across the TYPO3 core corpus that is 764 instances, while only 207 distinct instance configurations exist. Measured on ext:core, provisioning accounts for roughly a third of the suite's wall clock, and a cold bootstrap costs about 25 times a warm one because the container has to be compiled again. Instances are now identified by what actually shapes them - the extensions to load, the paths to link and provide, the configuration overrides, the folders to create and whether the database is initialised - so test cases configured identically share one instance. It is provisioned once, its container is compiled once and its schema is created once. Sharing an instance between test case classes exposes state that used to be private simply because every test case owned an instance. Six such channels were found and are addressed here: * The instance cache directory is shared, because the package dependent cache identifier derives from the project path. The whole "core" cache group is dropped when one test case class hands the instance to the next, so a cached TCA schema or a backend module registry written by one test case cannot be seen by another. The compiled container is deliberately kept: it depends only on the active package set, and keeping it is what makes sharing worthwhile. * Database snapshots were keyed on the instance and created on the first test of the instance rather than of the test case. They are now scoped to the test case class, so a test case that did not provision the instance no longer restores a snapshot it never created. * Test cases write to typo3conf/system/settings.php, for example through the "configuration:set" command. A pristine copy is kept at provisioning time and restored at a test case class boundary. * Test cases that override how the database is provisioned - the core schema test cases start from a database with no tables - rely on being the test case that provisions the instance. Such test cases now get their own instance family, keyed on the declaring class of initializeTestDatabase() and initializeTestDatabaseAndTruncateTables(). Breaking changes: * getInstanceIdentifier() and getInstancePath() are no longer static. They cannot be, because test cases may assign the configuration properties in their own setUp() before calling parent::setUp(), and a static method cannot see those assignments. Both are marked @internal, and getInstancePath() already documented that it may break at any time. Migration, for test cases that touch them: - Calling them: use $this->getInstanceIdentifier() and $this->getInstancePath() instead of self:: or static::. - Overriding them: drop the "static" keyword from the declaration. A subclass that still declares them static will fail to load. - Overriding getInstanceIdentifier() to pin an instance is usually the wrong hook now. Anything that shapes what the instance *contains* belongs in getInstanceConfiguration(), so that test cases configured the same way keep sharing an instance; an overridden identifier opts the test case out of sharing entirely. Keeping the methods static was considered, by computing the identifier in setUp() and handing it out through a static property. That preserves both static call sites and static overrides, but it puts mutable static state back into FunctionalTestCase - the same class of coupling as the six channels fixed above - and leaves a static signature whose result is only meaningful between setUp() and tearDown(). The honest break was preferred. Neither typo3/cms nor the testing framework itself calls these statically any more. * DatabaseSnapshot::initialize() gains an optional third argument naming the snapshot separately from the instance. Existing two argument calls behave as before. Validated against the full functional suite of TYPO3 core - all system extensions - comparing assertion counts and failing test sets, not just totals: DBMS tests assertions baseline shared delta sqlite 12392 72711 1341.6 s 847.8 s -36.8 % MariaDB 12427 72966 identical outcome, timing per notes Postgres 12388 72693 2109.8 s 1672.4 s -20.7 % Identical failure sets in every case. A controlled back to back measurement of ext:core on MariaDB, three runs in one job, shows 563.6 s against 450.0 s, a 20.2 per cent improvement with a 3.6 per cent spread between repeats.
The test instance directory and the test database are named after the instance identifier alone, so two functional test runs sharing a working copy address the same instance directory and the same database. A run that provisions an instance then deletes it while the other run is still using it, and both collapse. Reproduced on an unmodified checkout by running one test case twice at the same time: 41 errors and 10 failures in one run, 34 errors and 12 failures in the other, one shared instance directory. Setting TYPO3_TESTING_WORKER to a different value per run appends it to the instance identifier, so the runs get their own instance directory and their own database. The same two runs then pass with 70 tests each. The variable is unset by default and naming is then unchanged. It becomes part of database names, so the value is restricted to one to eight alphanumeric characters. An invalid value is rejected rather than sanitised: silently folding "1-a" and "1a" onto one instance would reintroduce exactly the collision this prevents. This is a prerequisite for running the suite with several workers, but it is useful on its own - it is what makes it possible to run two database backends, or one suite and one test file, against a single working copy.
bmack
force-pushed
the
layer0-share-test-instances
branch
from
August 10, 2026 21:21
544a014 to
c5369a3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Functional test instances are identified by
sha1(static::class), so every test case classprovisions its own instance: its own directory tree, its own compiled dependency injection
container and its own database schema. Across the TYPO3 core corpus that is 764 instances,
while only 207 distinct instance configurations exist.
This series keys instances on what actually shapes them — the extensions to load, the paths to
link and provide, the configuration overrides, the folders to create, whether the database is
initialised — so test cases configured identically share one instance. It is provisioned once,
its container is compiled once and its schema is created once.
Measurements
Layer 0 in isolation, full core corpus, measured 2026-07-31 against the then-current
main:A more recent full-corpus run (2026-08-04, with the composer-mode work of the follow-up PR also
applied, but in classic mode where that work is largely inert) measured 1379.2 s → 945.9 s,
−31.4 % on sqlite.
In every case the comparison was made on assertion counts and failing test sets, not just
totals — a suite that merely stays green while doing less work would otherwise look like a win.
Sharing state that used to be private
Sharing an instance between test case classes exposes state that was previously private simply
because every test case owned an instance. Six such channels were found and are addressed here;
the commit message lists them individually. The compiled dependency injection container is
deliberately not flushed: it depends only on the active package set, and keeping it is what
makes sharing worthwhile at all.
Breaking change
getInstanceIdentifier()andgetInstancePath()are no longer static — they cannot be, becausetest cases may assign the configuration properties in their own
setUp()before callingparent::setUp(). Both are@internal, andgetInstancePath()already documented that it maybreak at any time. The first commit carries a migration note for the three cases a downstream
test case can be in. Neither
typo3/cmsnor the testing framework itself calls these staticallyany more.
Keeping them static via a holder was considered and rejected; the reasoning is in the commit
message.
Follow-up
A second PR provisions functional test instances in composer mode, which builds on this one.
The TYPO3 core side of that work is prepared and waits on a release of this.
Verification
Each commit was verified green individually against the full TYPO3 core functional corpus, with
the upstream testing framework run as a control arm in the same job. On the current base both
commits produce results identical to upstream: 12473 tests, 72962 assertions, 0 errors, the same
6 pre-existing failures.