Repository navigation
Replace Crave with parallel test workflows - #5354
Conversation
- tests-core.yml: solr/core split into four package slices (cloud, search, handler, other) run as a parallel matrix; cloud runs with tests.jvms=3 - tests-solrj.yml (was solrj-test.yml): also solrj-jetty, solrj-zookeeper, solrj-streaming and api - tests-modules-search.yml and tests-modules-cloud.yml group the modules - tests-extraction.yml (was gradle-extraction-check.yml) - tests-misc.yml: test-framework, benchmark, ref-guide examples - New tests.exclude option (comma-separated --tests patterns to exclude), needed for the exclusion-based "other" core slice
|
Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).
|
… job
An include that differs from every base combination on a key is added
as a new combination, so {cloud, jvms: 3} joined {cloud, jvms: 2}
instead of replacing it and the cloud slice ran twice. With jvms gone
from the base axis the include only decorates the existing cloud job,
and the step defaults the other slices to 2.
CoreContainerShutdownTest waits out the production close timeouts and TestCpuAllowedLimit runs queries until the CPU limit trips: each holds one test JVM for about two minutes on every PR run, by design. Both keep running nightly.
The cloud slice set the critical path at 16.6 minutes against 12 for search and other, and the cloud package alone is the largest unit in solr/core. Its api and overseer sub-packages now run in the search slice, which had the most room, and request moves there from handler. Every slice runs with two test JVMs: the third JVM the cloud slice had was measured slower, not faster, and the include that added it ran the whole slice a second time.
| # Top-level packages under org.apache.solr for each named slice. | ||
| # The "other" slice runs everything not listed here. | ||
| SLICE_CLOUD: cloud | ||
| SLICE_SEARCH: search uninverting spelling highlight analysis legacy index parser | ||
| SLICE_HANDLER: handler cli response request servlet jersey rest security api | ||
|
|
There was a problem hiding this comment.
Thinking aloud: If I could re-organize our core module code, it'd be around: cli (soon to be separated!), cloud, server, node, core. And maybe others, but those are the big pillars in my mind. Ah well. I'm tempted to propose a big bang break-up for Solr 11... at least for the code to be in the right paths with the right packages... even if the surrounding modules aren't ready to be formed yet.
The SolrJ job was the shortest of the test jobs and solrj-zookeeper is the heaviest module in it, so move that module to the exclusion-based "other" slice of the core workflow, which has the most headroom. The cloud slice is the critical path, so it does not take on more work. The "other" slice passes no --tests, only a -Ptests.exclude of the packages owned by the three named slices, so a single gradle invocation can drive both test tasks without the filter becoming ambiguous. None of the excluded package prefixes match solrj-zookeeper's tests, which live in org.apache.solr.client.solrj.impl and org.apache.solr.common.cloud.
The core tests build on the SolrJ modules, so a change confined to solrj, solrj-jetty or solrj-streaming can break them without the core workflow running at all. Add those to the existing "built on these" group, and list solrj-zookeeper with solr/core since the workflow now tests it.
The handler slice was the second longest at 15.2 minutes while the exclusion-based "other" slice ran 12.0, so drop cli from the handler list and let it fall through to "other".
The modules it runs are the operational ones -- backup repositories, cross-dc, jwt-auth, opentelemetry -- and "cloud" invited confusion with SolrCloud and with the cloud slice of the core tests.
These modules are mostly unit tests that do not each start a cluster, so they take a third JVM on the runner better than the core slices do.
Drop the percentage from the core workflow's header, since it only has to say that solr/core dominates the suite, and put the extraction workflow's name above its comment so every workflow file opens with its name.
|
@dsmiley Thanks for review, I did a lot of fixes, replied and closed most threds. A few open, all test workflows <18m. Could perhaps investigate if we can optimize some tests in the core/cloud one in order to get it down to 15min. Perhpas move some to Is this |
| env: | ||
| # Top-level packages under org.apache.solr for each named slice. | ||
| # The "other" slice runs everything not listed here. | ||
| SLICE_CLOUD: cloud |
There was a problem hiding this comment.
Now that we're going to balance by time, the slice names have lost their meaning. Let's use what use suggested of SLICE1, SLICE2, ...
There was a problem hiding this comment.
I like the new name -- ops
Crave is going away, and the single
tests-via-crave.ymljob ran the whole test suite serially for 75% of PRs. This replaces it with per-moduletests-*workflows that run in parallel, each triggered only by its module's paths (plus its own file andgradle/libs.versions.toml), so a PR waits for the slowest job instead of the sum. Estimates are from per-class timings on Jenkins calibrated against real ubuntu-latest runs; trigger rates are from the 386 PRs merged in the last 13 weeks. Full analysis with charts: https://claude.ai/artifact/V8efRjyq6fejDdT8W2rgHwsolr/coresplit by Java package into four parallel matrix jobs: cloud, search, handler and other (exclusion-based, so a new package can never fall outside every slice), balanced from the measured timings of this PR's first run. Also triggered bysolr/test-frameworkandsolr/api.solrj-test.yml)gradle-extraction-check.yml)Adds a
tests.excludeoption (comma-separated--testspatterns to exclude) ingradle/testing/defaults-tests.gradle, used by the "other" core slice and by the cloud slice.After the first run. Measured test-step times were cloud 16.6 min, handler 15.2, other 12.0, search 11.6, and the other five workflows all within two minutes of their estimates. Three follow-up commits act on that:
includethat gave it a third JVM did not match the base combination and so added a fifth job. Thejvmsaxis is gone and every slice runs with two JVMs, since the 3-JVM job was measured slower (18.0 min) than the 2-JVM one, not faster.CoreContainerShutdownTestandTestCpuAllowedLimitare@Nightly: each holds a test JVM for about two minutes by design.cloud.apiandcloud.overseerrun in the search slice (the cloud slice excludes them), andrequestmoves from handler to search. Thecloudroot package alone is about 1180 s of suite time and sets the floor at roughly 11.5 min with two JVMs; the four slices should now land within about 3.5 min of each other. The same mechanism is the fallback if a slice drifts again.In theory, this is how it will play out:

This is an alternative to #5350 which is one serial test run