[CELEBORN-2432] Cache sbt launcher and dependencies in SBT CI workflow

[yew1eb]

What changes were proposed in this pull request?

Add an actions/cache step to every job in sbt.yml, caching:

• build/*.jar (the sbt launcher downloaded by build/sbt)
• ~/.sbt
• ~/.cache/coursier (dependency repository)

Keys are derived from hashFiles of the poms and build scripts, so downloads only happen when the build files change. This follows the pattern used by Spark's build_and_test.yml.

Why are the changes needed?

The sbt CI jobs currently configure no caching at all: every run of every matrix job (~40 jobs) re-downloads the sbt launcher and the entire dependency set, which exposes CI to network flakes. Recent examples on PR CI and on main:

• "Our attempt to download sbt locally to build/sbt-launch-1.9.4.jar failed" (job fails in ~15s)
• scala compile failing with MissingResourceException:
  org.scalactic.ScalacticBundle caused by java.io.IOException: Stream closed (corrupted jar download)

Does this PR resolve a correctness bug?

• Yes

Does this PR introduce any user-facing change?

• Ye

How was this patch tested?

Workflow-only change; the PR's own CI run exercises the new cache steps (first run populates the cache, subsequent runs hit it).

[Copilot]

Pull request overview

Adds dependency caching to the SBT GitHub Actions workflow to reduce repeated downloads across CI runs, improving build stability (fewer network flakes) and speeding up subsequent runs.

Changes:

• Adds actions/cache@v4 to each SBT CI job.
• Caches the downloaded SBT launcher (build/*.jar) plus SBT/Coursier home directories (~/.sbt, ~/.cache/coursier).

Suppressed comments (6)

.github/workflows/sbt.yml:179

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

        key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
        restore-keys: |
          sbt-

.github/workflows/sbt.yml:231

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

          key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
          restore-keys: |
            sbt-

.github/workflows/sbt.yml:273

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

          key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
          restore-keys: |
            sbt-

.github/workflows/sbt.yml:316

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

          key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
          restore-keys: |
            sbt-

.github/workflows/sbt.yml:354

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

          key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
          restore-keys: |
            sbt-

.github/workflows/sbt.yml:392

• The cache key is currently shared across different JDK / Scala / Spark/Flink matrix variants and only hashes a subset of the SBT build definition. Because GitHub Actions caches are immutable once saved, the first job that saves this key can “freeze” an incomplete cache, and later variants will keep re-downloading missing artifacts without ever being able to save them. Consider (1) including the SBT build definition files (project//*.scala, project//*.sbt, version.sbt, plugins.sbt) in hashFiles and (2) scoping the key by the relevant matrix dimensions (java/scala/spark/flink) while keeping restore-keys broad for reuse.

          key: sbt-${{ hashFiles('**/pom.xml', 'project/build.properties', 'build/sbt', 'build/sbt-launch-lib.bash') }}
          restore-keys: |
            sbt-

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[yew1eb]

Addressed: the cache key is now scoped by job and matrix dimensions (sbt-${{ github.job }}-${{ matrix.java }}-${{ matrix.scala }}-${{ matrix.spark }}${{ matrix.flink }}-...), and hashFiles covers the SBT build definition files (project/**/*.scala, project/**/*.sbt) instead of pom.xml.

[pan3793]

IIRC actions/setup-java provides some built-in cache for building systems, does that work?

BTW, you can customize build/sbt-config/repositories to set up mirrors.

[yew1eb]

IIRC actions/setup-java provides some built-in cache for building systems, does that work?

BTW, you can customize build/sbt-config/repositories to set up mirrors.

setup-java's cache: sbt doesn't cover build/sbt-launch-*.jar (the main flake source here), and its key isn't matrix-scoped — so we kept the manual cache. Re mirrors: already supported via DEFAULT_ARTIFACT_REPOSITORY for local dev; not a safe CI default since the GCS mirror is incompletely synced (sbt-launch jar 404s).