Skip to content

Replace Crave with parallel test workflows - #5354

Merged
janhoy merged 11 commits into
apache:mainfrom
janhoy:gh-parallel-tests
Nov 18, 2026
Merged

janhoy merged 11 commits into
apache:mainfrom
janhoy:gh-parallel-tests

Conversation

@janhoy

@janhoy janhoy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Crave is going away, and the single tests-via-crave.yml job ran the whole test suite serially for 75% of PRs. This replaces it with per-module tests-* workflows that run in parallel, each triggered only by its module's paths (plus its own file and gradle/libs.versions.toml), so a PR waits for the slowest job instead of the sum. Estimates are from per-class timings on Jenkins calibrated against real ubuntu-latest runs; trigger rates are from the 386 PRs merged in the last 13 weeks. Full analysis with charts: https://claude.ai/artifact/V8efRjyq6fejDdT8W2rgHw

Workflow Tests Est. time PRs triggering
tests-core solr/core split by Java package into four parallel matrix jobs: cloud, search, handler and other (exclusion-based, so a new package can never fall outside every slice), balanced from the measured timings of this PR's first run. Also triggered by solr/test-framework and solr/api. ~11.5 min 59%
tests-solrj (was solrj-test.yml) solrj, solrj-jetty, solrj-zookeeper, solrj-streaming, api ~9.5 min 30%
tests-modules-search ltr, sql, language-models, langid, analysis-extras, clustering, scripting, cuvs ~6.5 min 13%
tests-modules-cloud s3-repository, gcs-repository, jwt-auth, opentelemetry, cross-dc, cross-dc-manager ~6 min 16%
tests-extraction (was gradle-extraction-check.yml) extraction, needs Docker for TikaServer ~4.5 min 5%
tests-misc test-framework, benchmark, ref-guide example tests ~4.5 min 17%

Adds a tests.exclude option (comma-separated --tests patterns to exclude) in gradle/testing/defaults-tests.gradle, used by the "other" core slice and by the cloud slice.

After the first run. Measured test-step times were cloud 16.6 min, handler 15.2, other 12.0, search 11.6, and the other five workflows all within two minutes of their estimates. Three follow-up commits act on that:

  • The cloud slice ran twice: the matrix include that gave it a third JVM did not match the base combination and so added a fifth job. The jvms axis is gone and every slice runs with two JVMs, since the 3-JVM job was measured slower (18.0 min) than the 2-JVM one, not faster.
  • CoreContainerShutdownTest and TestCpuAllowedLimit are @Nightly: each holds a test JVM for about two minutes by design.
  • The slices are rebalanced on package boundaries from the measured per-package times: cloud.api and cloud.overseer run in the search slice (the cloud slice excludes them), and request moves from handler to search. The cloud root package alone is about 1180 s of suite time and sets the floor at roughly 11.5 min with two JVMs; the four slices should now land within about 3.5 min of each other. The same mechanism is the fallback if a slice drifts again.

In theory, this is how it will play out:
gh-parallel-tests-overview

This is an alternative to #5350 which is one serial test run

- tests-core.yml: solr/core split into four package slices (cloud, search,
  handler, other) run as a parallel matrix; cloud runs with tests.jvms=3
- tests-solrj.yml (was solrj-test.yml): also solrj-jetty, solrj-zookeeper,
  solrj-streaming and api
- tests-modules-search.yml and tests-modules-cloud.yml group the modules
- tests-extraction.yml (was gradle-extraction-check.yml)
- tests-misc.yml: test-framework, benchmark, ref-guide examples
- New tests.exclude option (comma-separated --tests patterns to exclude),
  needed for the exclusion-based "other" core slice
@github-actions

Copy link
Copy Markdown

Automated PR readiness judgements (LLM-generated; may be wrong; not updated on later pushes).

… job

An include that differs from every base combination on a key is added
as a new combination, so {cloud, jvms: 3} joined {cloud, jvms: 2}
instead of replacing it and the cloud slice ran twice. With jvms gone
from the base axis the include only decorates the existing cloud job,
and the step defaults the other slices to 2.
CoreContainerShutdownTest waits out the production close timeouts and
TestCpuAllowedLimit runs queries until the CPU limit trips: each holds
one test JVM for about two minutes on every PR run, by design. Both
keep running nightly.
The cloud slice set the critical path at 16.6 minutes against 12 for
search and other, and the cloud package alone is the largest unit in
solr/core. Its api and overseer sub-packages now run in the search
slice, which had the most room, and request moves there from handler.
Every slice runs with two test JVMs: the third JVM the cloud slice had
was measured slower, not faster, and the include that added it ran the
whole slice a second time.

@dsmiley dsmiley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this :-)

Comment thread .github/workflows/tests-core.yml Resolved
Comment thread .github/workflows/tests-core.yml Resolved
Comment thread .github/workflows/tests-core.yml Resolved
Comment thread .github/workflows/tests-extraction.yml Resolved
Comment thread .github/workflows/tests-modules-ops.yml Resolved

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

Comment on lines +42 to +47
# Top-level packages under org.apache.solr for each named slice.
# The "other" slice runs everything not listed here.
SLICE_CLOUD: cloud
SLICE_SEARCH: search uninverting spelling highlight analysis legacy index parser
SLICE_HANDLER: handler cli response request servlet jersey rest security api

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking aloud: If I could re-organize our core module code, it'd be around: cli (soon to be separated!), cloud, server, node, core. And maybe others, but those are the big pillars in my mind. Ah well. I'm tempted to propose a big bang break-up for Solr 11... at least for the code to be in the right paths with the right packages... even if the surrounding modules aren't ready to be formed yet.

Comment thread .github/workflows/tests-core.yml Resolved
Comment thread .github/workflows/tests-core.yml Resolved
Comment thread .github/workflows/tests-solrj.yml Resolved
The SolrJ job was the shortest of the test jobs and solrj-zookeeper is the
heaviest module in it, so move that module to the exclusion-based "other"
slice of the core workflow, which has the most headroom. The cloud slice is
the critical path, so it does not take on more work.

The "other" slice passes no --tests, only a -Ptests.exclude of the packages
owned by the three named slices, so a single gradle invocation can drive both
test tasks without the filter becoming ambiguous. None of the excluded
package prefixes match solrj-zookeeper's tests, which live in
org.apache.solr.client.solrj.impl and org.apache.solr.common.cloud.
The core tests build on the SolrJ modules, so a change confined to solrj,
solrj-jetty or solrj-streaming can break them without the core workflow
running at all. Add those to the existing "built on these" group, and list
solrj-zookeeper with solr/core since the workflow now tests it.
The handler slice was the second longest at 15.2 minutes while the
exclusion-based "other" slice ran 12.0, so drop cli from the handler list
and let it fall through to "other".
The modules it runs are the operational ones -- backup repositories, cross-dc,
jwt-auth, opentelemetry -- and "cloud" invited confusion with SolrCloud and
with the cloud slice of the core tests.
These modules are mostly unit tests that do not each start a cluster, so they
take a third JVM on the runner better than the core slices do.
Drop the percentage from the core workflow's header, since it only has to say
that solr/core dominates the suite, and put the extraction workflow's name
above its comment so every workflow file opens with its name.
@janhoy

janhoy commented Nov 17, 2026

Copy link
Copy Markdown
Contributor Author

@dsmiley Thanks for review, I did a lot of fixes, replied and closed most threds. A few open, all test workflows <18m. Could perhaps investigate if we can optimize some tests in the core/cloud one in order to get it down to 15min. Perhpas move some to @Nightly. But that could be followup. I'd say we merge, let it prove itself, and iterate with follow up commits?

Is this main branch code only, or do we need it on the other release branches for PRs targeting branch_10x directly? And what about 9x?

@dsmiley dsmiley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Jan! 🚢

env:
# Top-level packages under org.apache.solr for each named slice.
# The "other" slice runs everything not listed here.
SLICE_CLOUD: cloud

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we're going to balance by time, the slice names have lost their meaning. Let's use what use suggested of SLICE1, SLICE2, ...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the new name -- ops

@janhoy
janhoy merged commit ad81444 into apache:main Nov 18, 2026
12 checks passed
@janhoy
janhoy deleted the gh-parallel-tests branch November 18, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants