From e823075d3714e2aee06b39f0f0374aff720c4ebb Mon Sep 17 00:00:00 2001 From: Benni Mack Date: Fri, 31 Jul 2026 21:49:16 +0200 Subject: [PATCH 1/2] [!!!][FEATURE] Share test instances between identically configured test cases Functional test instances are identified by sha1(static::class), so every test case class provisions its own instance: its own directory tree, its own compiled dependency injection container and its own database schema. Across the TYPO3 core corpus that is 764 instances, while only 207 distinct instance configurations exist. Measured on ext:core, provisioning accounts for roughly a third of the suite's wall clock, and a cold bootstrap costs about 25 times a warm one because the container has to be compiled again. Instances are now identified by what actually shapes them - the extensions to load, the paths to link and provide, the configuration overrides, the folders to create and whether the database is initialised - so test cases configured identically share one instance. It is provisioned once, its container is compiled once and its schema is created once. Sharing an instance between test case classes exposes state that used to be private simply because every test case owned an instance. Six such channels were found and are addressed here: * The instance cache directory is shared, because the package dependent cache identifier derives from the project path. The whole "core" cache group is dropped when one test case class hands the instance to the next, so a cached TCA schema or a backend module registry written by one test case cannot be seen by another. The compiled container is deliberately kept: it depends only on the active package set, and keeping it is what makes sharing worthwhile. * Database snapshots were keyed on the instance and created on the first test of the instance rather than of the test case. They are now scoped to the test case class, so a test case that did not provision the instance no longer restores a snapshot it never created. * Test cases write to typo3conf/system/settings.php, for example through the "configuration:set" command. A pristine copy is kept at provisioning time and restored at a test case class boundary. * Test cases that override how the database is provisioned - the core schema test cases start from a database with no tables - rely on being the test case that provisions the instance. Such test cases now get their own instance family, keyed on the declaring class of initializeTestDatabase() and initializeTestDatabaseAndTruncateTables(). Breaking changes: * getInstanceIdentifier() and getInstancePath() are no longer static. They cannot be, because test cases may assign the configuration properties in their own setUp() before calling parent::setUp(), and a static method cannot see those assignments. Both are marked @internal, and getInstancePath() already documented that it may break at any time. Migration, for test cases that touch them: - Calling them: use $this->getInstanceIdentifier() and $this->getInstancePath() instead of self:: or static::. - Overriding them: drop the "static" keyword from the declaration. A subclass that still declares them static will fail to load. - Overriding getInstanceIdentifier() to pin an instance is usually the wrong hook now. Anything that shapes what the instance *contains* belongs in getInstanceConfiguration(), so that test cases configured the same way keep sharing an instance; an overridden identifier opts the test case out of sharing entirely. Keeping the methods static was considered, by computing the identifier in setUp() and handing it out through a static property. That preserves both static call sites and static overrides, but it puts mutable static state back into FunctionalTestCase - the same class of coupling as the six channels fixed above - and leaves a static signature whose result is only meaningful between setUp() and tearDown(). The honest break was preferred. Neither typo3/cms nor the testing framework itself calls these statically any more. * DatabaseSnapshot::initialize() gains an optional third argument naming the snapshot separately from the instance. Existing two argument calls behave as before. Validated against the full functional suite of TYPO3 core - all system extensions - comparing assertion counts and failing test sets, not just totals: DBMS tests assertions baseline shared delta sqlite 12392 72711 1341.6 s 847.8 s -36.8 % MariaDB 12427 72966 identical outcome, timing per notes Postgres 12388 72693 2109.8 s 1672.4 s -20.7 % Identical failure sets in every case. A controlled back to back measurement of ext:core on MariaDB, three runs in one job, shows 563.6 s against 450.0 s, a 20.2 per cent improvement with a 3.6 per cent spread between repeats. --- .../Snapshot/DatabaseSnapshot.php | 24 +- .../Core/Functional/FunctionalTestCase.php | 241 ++++++++++++++++-- 2 files changed, 237 insertions(+), 28 deletions(-) diff --git a/Classes/Core/Functional/Framework/DataHandling/Snapshot/DatabaseSnapshot.php b/Classes/Core/Functional/Framework/DataHandling/Snapshot/DatabaseSnapshot.php index 180d7568..df804e4e 100644 --- a/Classes/Core/Functional/Framework/DataHandling/Snapshot/DatabaseSnapshot.php +++ b/Classes/Core/Functional/Framework/DataHandling/Snapshot/DatabaseSnapshot.php @@ -38,9 +38,18 @@ class DatabaseSnapshot private static DatabaseSnapshot $instance; private array $inMemoryImport = []; - public static function initialize(string $sqliteDir, string $identifier): void + /** + * @param string $identifier Identifies the *instance* database file, which several + * test case classes may share. + * @param string|null $snapshotIdentifier Identifies the snapshot taken of it. Defaults + * to $identifier for backwards compatibility, but callers + * sharing one instance between test case classes must pass a + * per test case value, otherwise one test case class restores + * the snapshot another one created. + */ + public static function initialize(string $sqliteDir, string $identifier, ?string $snapshotIdentifier = null): void { - self::$instance = new self($sqliteDir, $identifier); + self::$instance = new self($sqliteDir, $identifier, $snapshotIdentifier ?? $identifier); } public static function instance(): self @@ -50,11 +59,12 @@ public static function instance(): self private function __construct( private readonly string $sqliteDir, - private readonly string $identifier + private readonly string $identifier, + private readonly string $snapshotIdentifier ) {} /** - * Create a new snapshot. This is called for the *first* test in a test case. + * Create a new snapshot. This is called for the *first* test of a test case class. */ public function create(DatabaseAccessor $accessor, Connection $connection): void { @@ -63,7 +73,7 @@ public function create(DatabaseAccessor $accessor, Connection $connection): void $connection->close(); copy( $this->sqliteDir . 'test_' . $this->identifier . '.sqlite', - $this->sqliteDir . 'test_' . $this->identifier . '.snapshot.sqlite' + $this->sqliteDir . 'test_' . $this->snapshotIdentifier . '.snapshot.sqlite' ); $this->inMemoryImport = [true]; } else { @@ -80,14 +90,14 @@ public function create(DatabaseAccessor $accessor, Connection $connection): void } /** - * Restore a snapshot. This is called for subsequent tests in a test case. + * Restore a snapshot. This is called for subsequent tests of a test case class. */ public function restore(DatabaseAccessor $accessor, Connection $connection): void { if ($connection->getDatabasePlatform() instanceof SQLitePlatform) { $connection->close(); copy( - $this->sqliteDir . 'test_' . $this->identifier . '.snapshot.sqlite', + $this->sqliteDir . 'test_' . $this->snapshotIdentifier . '.snapshot.sqlite', $this->sqliteDir . 'test_' . $this->identifier . '.sqlite' ); } else { diff --git a/Classes/Core/Functional/FunctionalTestCase.php b/Classes/Core/Functional/FunctionalTestCase.php index 1dbec1f4..7172216e 100644 --- a/Classes/Core/Functional/FunctionalTestCase.php +++ b/Classes/Core/Functional/FunctionalTestCase.php @@ -77,6 +77,39 @@ */ abstract class FunctionalTestCase extends BaseTestCase implements ContainerInterface { + /** + * Cache groups dropped when one test case class hands an instance over to the next. + * + * Test cases configured identically share an instance, and the package dependent + * cache identifier is derived from the project path, so they also address the same + * cache entries. Any entry a test case writes is therefore visible to every later + * test case sharing that instance. + * + * The whole "core" group is dropped rather than individual entries. Individual + * entries were tried first and are demonstrably fragile: TcaSchema alone was + * sufficient for two extensions but missed BackendModules, which a test writes + * deliberately and never cleans up. Any test may legitimately write a package + * dependent cache entry, so the safe default is to drop the group and let it be + * rebuilt. Measured cost is within noise. + * + * "di" is deliberately not listed: the compiled dependency injection container + * depends only on the active package set, and keeping it is what makes sharing an + * instance worthwhile at all. + * + * @var non-empty-string[] + */ + private const TEST_CASE_SCOPED_CACHE_GROUPS = [ + 'core', + ]; + + /** + * Instance configuration written during provisioning, and the pristine copy kept + * beside it so it can be restored when one test case class hands the instance to + * the next. + */ + private const SETTINGS_FILE = '/typo3conf/system/settings.php'; + private const SETTINGS_FILE_PRISTINE = '/typo3conf/system/settings.pristine.php'; + /** * Unique identifier for this test case. Location of the test * instance and database name depend on this. Calculated early in setUp() @@ -264,15 +297,43 @@ abstract class FunctionalTestCase extends BaseTestCase implements ContainerInter private ContainerInterface $container; /** - * These two internal variable track if the given test is the first test of - * that test case. This variable is set to current calling test case class. - * Consecutive tests then optimize and do not create a full - * database structure again but instead just truncate all tables which - * is much quicker. + * Instance identifiers provisioned in this PHP process. + * + * Instances are keyed on the test case *configuration*, not on the test case + * class, so several test case classes can share one instance. Provisioning + * therefore has to be tracked per identifier: the first test that needs a + * given instance creates it and its database schema, every following test - + * in the same class or in another one sharing the configuration - only + * truncates the tables, which is much quicker. + * + * @var array + */ + private static array $provisionedInstances = []; + + /** + * True when this test had to provision the instance, i.e. when it is the + * first test in this process using this instance configuration. */ - private static string $currentTestCaseClass = ''; private bool $isFirstTest = true; + /** + * True when this is the first test of *this test case class* in this process. + * + * Since instances are keyed on configuration rather than on the test case + * class, this is not the same question as $isFirstTest: several test case + * classes share one instance, so only the first of them provisions it while + * each of them still has a first test. Anything scoped to a test case class + * rather than to an instance - database snapshots, most notably - has to key + * off this. + */ + private bool $isFirstTestOfTestCase = true; + + /** + * Last test case class seen in this process, used by the cache isolation + * experiment to detect a test case class boundary. + */ + private static string $lastTestCaseClass = ''; + /** * Set up creates a test instance and database. * @@ -284,26 +345,60 @@ protected function setUp(): void self::markTestSkipped('Functional tests must be called through phpunit on CLI'); } - $this->identifier = static::getInstanceIdentifier(); - $this->instancePath = static::getInstancePath(); + $this->identifier = $this->getInstanceIdentifier(); + $this->instancePath = $this->getInstancePath(); putenv('TYPO3_PATH_ROOT=' . $this->instancePath); putenv('TYPO3_PATH_APP=' . $this->instancePath); $testbase = new Testbase(); $testbase->setTypo3TestingContext(); - // See if we're the first test of this test case. - $currentTestCaseClass = static::class; - if (self::$currentTestCaseClass !== $currentTestCaseClass) { - self::$currentTestCaseClass = $currentTestCaseClass; - } else { + // See if this instance has already been provisioned in this process. Note + // this is keyed on the instance identifier, not on the test case class: + // test cases sharing a configuration share the instance. + if (isset(self::$provisionedInstances[$this->identifier])) { $this->isFirstTest = false; + } else { + self::$provisionedInstances[$this->identifier] = true; + } + + // Independently of the above: is this the first test of this test case class? + $this->isFirstTestOfTestCase = self::$lastTestCaseClass !== static::class; + + // Database snapshots are scoped to a test case class, not to an instance: + // the snapshot is created by the first test of a test case and restored by + // its remaining tests. Re-initialise per test case class and give the + // snapshot file a per test case name, so that two test case classes sharing + // an instance cannot overwrite or restore each other's snapshot. + if ($this->isFirstTestOfTestCase) { + DatabaseSnapshot::initialize( + dirname($this->getInstancePath()) . '/functional-sqlite-dbs/', + $this->identifier, + $this->identifier . '-' . substr(sha1(static::class), 0, 7) + ); } // sqlite db path preparation $dbPathSqlite = dirname($this->instancePath) . '/functional-sqlite-dbs/test_' . $this->identifier . '.sqlite'; $dbPathSqliteEmpty = dirname($this->instancePath) . '/functional-sqlite-dbs/test_' . $this->identifier . '.empty.sqlite'; + // Test case classes configured identically share one instance, and therefore + // also share that instance's cache directory: the package dependent cache + // identifier is derived from the project path, which is the same for all of + // them. Cache entries that depend on more than the active package set must + // therefore be dropped when moving from one test case class to the next. + // + // In practice that is the TCA schema: test cases legitimately modify $GLOBALS['TCA'] + // and rebuild the schema from it, and the result must not survive into the next + // test case class. The compiled dependency injection container deliberately does + // *not* qualify - it depends only on the active package set, and keeping it is + // what makes sharing an instance worthwhile in the first place. + if (!$this->isFirstTest && $this->isFirstTestOfTestCase) { + $this->resetTestCaseScopedInstanceState(); + } + + self::$lastTestCaseClass = static::class; + if (!$this->isFirstTest) { // Reusing an existing instance. This typically happens for the second, third, ... test // in a test case, so environment is set up only once per test case. @@ -312,7 +407,6 @@ protected function setUp(): void $this->initializeTestDatabaseAndTruncateTables($testbase, $this->initializeDatabase, $dbPathSqlite, $dbPathSqliteEmpty); $testbase->loadExtensionTables(); } else { - DatabaseSnapshot::initialize(dirname($this->getInstancePath()) . '/functional-sqlite-dbs/', $this->identifier); $testbase->removeOldInstanceIfExists($this->instancePath); // Basic instance directory structure $testbase->createDirectory($this->instancePath . '/fileadmin'); @@ -419,6 +513,12 @@ protected function setUp(): void $localConfiguration['SYS']['caching']['cacheConfigurations']['pages']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\NullBackend'; $localConfiguration['SYS']['caching']['cacheConfigurations']['rootline']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\NullBackend'; $testbase->setUpLocalConfiguration($this->instancePath, $localConfiguration, $this->configurationToUseInTestInstance); + // Keep a pristine copy: test cases sharing this instance may write to the + // configuration, and the next one must not inherit that. + copy( + $this->instancePath . self::SETTINGS_FILE, + $this->instancePath . self::SETTINGS_FILE_PRISTINE + ); $testbase->setUpPackageStates( $this->instancePath, $defaultCoreExtensionsToLoad, @@ -1089,7 +1189,9 @@ protected function withDatabaseSnapshot(?callable $createCallback = null, ?calla $connection = $this->getConnectionPool()->getConnectionByName(ConnectionPool::DEFAULT_CONNECTION_NAME); $accessor = new DatabaseAccessor($connection); $snapshot = DatabaseSnapshot::instance(); - if ($this->isFirstTest) { + // Scoped to the test case class, not to the instance: several test case + // classes may share one instance, and each of them needs its own snapshot. + if ($this->isFirstTestOfTestCase) { if ($createCallback) { $createCallback(); } @@ -1103,14 +1205,112 @@ protected function withDatabaseSnapshot(?callable $createCallback = null, ?calla } /** - * Create a 7 char long hash of class name as identifier. + * Drops the state that belongs to a test case class rather than to the instance, + * when one test case class hands a shared instance over to the next. + */ + private function resetTestCaseScopedInstanceState(): void + { + $cacheRoot = $this->instancePath . '/typo3temp/var/cache'; + foreach (self::TEST_CASE_SCOPED_CACHE_GROUPS as $group) { + foreach (['code', 'data'] as $kind) { + $dir = $cacheRoot . '/' . $kind . '/' . $group; + if (is_dir($dir)) { + GeneralUtility::rmdir($dir, true); + } + } + } + // Tests may write to the instance configuration - the "configuration:set" + // command does, for instance - and that must not be visible to the next + // test case class sharing the instance. Restore the file provisioning wrote. + $settings = $this->instancePath . self::SETTINGS_FILE; + $pristine = $this->instancePath . self::SETTINGS_FILE_PRISTINE; + if (is_file($pristine)) { + copy($pristine, $settings); + } + } + + /** + * Identifier of the test instance, derived from the *configuration* of this + * test case rather than from its class name. + * + * Test cases that declare the same instance configuration therefore share + * one instance: it is provisioned once, its dependency injection container + * is compiled once and its database schema is created once, instead of once + * per test case class. + * + * This must not be static. Test cases may assign the configuration + * properties in their own setUp() before calling parent::setUp(), and those + * assignments have to be part of the identifier - a static method could not + * see them and would hand the test an instance built for a different + * extension set. * * @internal * @return non-empty-string */ - protected static function getInstanceIdentifier(): string + protected function getInstanceIdentifier(): string { - return substr(sha1(static::class), 0, 7); + return substr(sha1(serialize($this->getInstanceConfiguration())), 0, 10); + } + + /** + * Everything that shapes the content of a test instance, normalised so that + * two test cases configured equivalently produce the same identifier. + * + * Anything influencing instance content but missing here would let two + * genuinely different instances share an identifier, so keep this in sync + * with what setUp() passes to Testbase. + * + * @internal + * @return array + */ + protected function getInstanceConfiguration(): array + { + return [ + 'coreExtensionsToLoad' => self::normalizeInstanceConfigurationValue($this->coreExtensionsToLoad), + 'testExtensionsToLoad' => self::normalizeInstanceConfigurationValue($this->testExtensionsToLoad), + 'pathsToLinkInTestInstance' => self::normalizeInstanceConfigurationValue($this->pathsToLinkInTestInstance), + 'pathsToProvideInTestInstance' => self::normalizeInstanceConfigurationValue($this->pathsToProvideInTestInstance), + 'configurationToUseInTestInstance' => self::normalizeInstanceConfigurationValue($this->configurationToUseInTestInstance), + 'additionalFoldersToCreate' => self::normalizeInstanceConfigurationValue($this->additionalFoldersToCreate), + 'initializeDatabase' => $this->initializeDatabase, + 'databaseLifecycle' => $this->getDatabaseLifecycleIdentifier(), + ]; + } + + /** + * Test cases may override how the test database is set up and reset - the core + * schema test cases do, to start from a database with no tables at all. Such a + * test case relies on being the one that provisions the instance, because its + * override is only called on a first test. It must therefore not share an + * instance with a test case that provisions the database differently. + * + * Test cases overriding the same way still share with each other. + */ + private function getDatabaseLifecycleIdentifier(): string + { + $declaring = []; + foreach (['initializeTestDatabase', 'initializeTestDatabaseAndTruncateTables'] as $method) { + $declaring[] = (new \ReflectionMethod(static::class, $method))->getDeclaringClass()->getName(); + } + return implode('|', $declaring); + } + + /** + * Sorts lists by value and maps by key, recursively, so that declaration + * order does not influence the instance identifier. + */ + private static function normalizeInstanceConfigurationValue(mixed $value): mixed + { + if (!is_array($value)) { + return $value; + } + $value = array_map(self::normalizeInstanceConfigurationValue(...), $value); + if (array_is_list($value)) { + sort($value); + return $value; + } + ksort($value); + return $value; } /** @@ -1119,9 +1319,8 @@ protected static function getInstanceIdentifier(): string * are usually ways to avoid it. * @return non-empty-string */ - protected static function getInstancePath(): string + protected function getInstancePath(): string { - $identifier = static::getInstanceIdentifier(); - return ORIGINAL_ROOT . 'typo3temp/var/tests/functional-' . $identifier; + return ORIGINAL_ROOT . 'typo3temp/var/tests/functional-' . $this->getInstanceIdentifier(); } } From c5369a3f3035061e6ec5b9e7ca453d097d4978a2 Mon Sep 17 00:00:00 2001 From: Benni Mack Date: Sun, 2 Aug 2026 11:08:51 +0200 Subject: [PATCH 2/2] [FEATURE] Allow several test runs against one working copy The test instance directory and the test database are named after the instance identifier alone, so two functional test runs sharing a working copy address the same instance directory and the same database. A run that provisions an instance then deletes it while the other run is still using it, and both collapse. Reproduced on an unmodified checkout by running one test case twice at the same time: 41 errors and 10 failures in one run, 34 errors and 12 failures in the other, one shared instance directory. Setting TYPO3_TESTING_WORKER to a different value per run appends it to the instance identifier, so the runs get their own instance directory and their own database. The same two runs then pass with 70 tests each. The variable is unset by default and naming is then unchanged. It becomes part of database names, so the value is restricted to one to eight alphanumeric characters. An invalid value is rejected rather than sanitised: silently folding "1-a" and "1a" onto one instance would reintroduce exactly the collision this prevents. This is a prerequisite for running the suite with several workers, but it is useful on its own - it is what makes it possible to run two database backends, or one suite and one test file, against a single working copy. --- .../Core/Functional/FunctionalTestCase.php | 37 ++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/Classes/Core/Functional/FunctionalTestCase.php b/Classes/Core/Functional/FunctionalTestCase.php index 7172216e..8143ceff 100644 --- a/Classes/Core/Functional/FunctionalTestCase.php +++ b/Classes/Core/Functional/FunctionalTestCase.php @@ -1249,7 +1249,42 @@ private function resetTestCaseScopedInstanceState(): void */ protected function getInstanceIdentifier(): string { - return substr(sha1(serialize($this->getInstanceConfiguration())), 0, 10); + return substr(sha1(serialize($this->getInstanceConfiguration())), 0, 10) . self::getWorkerToken(); + } + + /** + * Token identifying the worker this test runs in, or an empty string. + * + * The test instance directory and the test database are named after the instance + * identifier alone, so two test runs sharing a working copy address the same + * instance directory and the same database - and a run that provisions an instance + * deletes it while the other run is still using it. Setting TYPO3_TESTING_WORKER to + * a different value per run keeps them apart, which is what makes it possible to run + * several suites, or several workers of one suite, against one working copy. + * + * Unset by default, in which case naming is unchanged. + * + * The token is part of database names, so it is deliberately restricted: it must be + * short and alphanumeric, and an invalid value is rejected rather than sanitised, so + * that two workers cannot be silently folded onto the same instance. + */ + private static function getWorkerToken(): string + { + $token = (string)getenv('TYPO3_TESTING_WORKER'); + if ($token === '') { + return ''; + } + if (preg_match('/^[a-zA-Z0-9]{1,8}$/', $token) !== 1) { + throw new \RuntimeException( + sprintf( + 'Environment variable TYPO3_TESTING_WORKER must be 1 to 8 alphanumeric' + . ' characters, "%s" given. It becomes part of test database names.', + $token + ), + 1754006400 + ); + } + return 'w' . $token; } /**