Skip to content

[BUILD] Allow auron-build.sh to build with an existing Maven and forward arbitrary Maven options #2460

Description

@ShreyeshArangath

Is your feature request related to a problem? Please describe.

Calling auron-build.sh from another build script is awkward, and on a machine without internet access it doesn't work at all. Three things get in the way.

It always builds through build/mvn, which downloads Maven from the Apache mirrors. That download fails in an offline or airgapped build, and there's no way to point the script at a Maven that's already installed. build/mvn does honour MVN_HOME and SKIP_MVN_VERSION_CHECK, but nothing at the top level lets a caller skip the wrapper.

Most Maven options still can't be passed through. #2227 fixed -D handling with MVN_D_ARGS, but -P, -B and long options like --no-transfer-progress are either swallowed by the break in the -* handler or rejected outright:

$ sh auron-build.sh --release --sparkver 3.5 --scalaver 2.12 --no-transfer-progress
[ERROR] Unknown option: --no-transfer-progress

The goal is hardcoded to install, so building a distribution also writes every module into ~/.m2. Nothing in the build needs that, since the assembly module is in the root reactor and its artifacts are read out of target/. It causes real trouble for anyone building several variants at one version: the intermediate modules carry no Spark version in their artifactId (auron-common_2.12, spark-extension_2.12, auron-core), so building Spark 3.1 and 3.5 at the same project.version writes two different jars over a single GAV. Whichever ran last is what every later build on that machine resolves.

Describe the solution you'd like

  • --mvn <PATH> to build with a given Maven instead of build/mvn. It should take either a path or a command on PATH, and say so clearly when it can't find either.
  • -- as an end-of-options marker, with everything after it passed to Maven untouched. This sits alongside the existing -D handling rather than replacing it.
  • --goal <GOAL> for the Maven goal, still defaulting to install so nothing changes for current callers, but letting a caller run package and leave the local repository alone.

That's enough for a wrapper script to drive the whole build:

auron-build.sh --release --mvn "$MVN" --goal package --sparkver 3.5 --scalaver 2.12 \
  -- -Pceleborn-0.6 -B --no-transfer-progress

Describe alternatives you've considered

Setting MVN_HOME and letting build/mvn pick it up. This works for the compile step, but build/mvn is still what gets invoked, so the version check and mirror logic stay in the path. A flag is clearer about intent and easier to pass down from a parent script.

Teaching MVN_D_ARGS to collect -P and long options too. Fewer flags, but then the collector has to know every Maven option worth forwarding, and that list keeps growing. -- is the usual answer and needs no upkeep as Maven changes.

Leaving --goal alone and always running install. That's the status quo, and it's what makes the multi-variant case above quietly wrong. Pointing -Dmaven.repo.local at a scratch directory works around it, but throws away the cache of every third-party dependency at the same time.

Changing the default to package. Tidier, but it would break any caller relying on the install side effect, so the default is better left as it is.

The -- part is useful on its own and could land separately as a follow-up to #2227 if you'd rather keep the changes small.

Additional context

The -- separator would also make auron-build.sh usable as a build step inside a larger script, which is the case that surfaced all three of these.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions