Pipeline for Maven

[chinyeungli]

• Issue: maven-heaven: Design and implement Scancode.io pipeline for a single Maven package #1763
• Create a pipeline for Maven package

[tdruez]

• Create a new maven pipe module in place of use resolve
• Opening and loading a large file to make edits multiple times in various steps is not great.
• To be discussed: Do we need a dedicated pipeline for just an extra step? Shouldn't the original scan_single_package detect that it's a Maven package and apply the necessary? Any reason to keep this new logic separated?

[chinyeungli]

@tdruez I’ve updated the code to include support for the "D2D" option.

The "deploy_to_devel" option is equivalent to the "map_deploy_to_develop" pipeline, which runs on Java, JavaScript, Kotlin, and Scala as these are the languages commonly found in Maven projects.

[chinyeungli]

@chinyeungli Can you merge in the new changes and fix the failing test for scan_maven_package_single_file (https://github.com/aboutcode-org/scancode.io/actions/runs/26379105691/job/77644819092?pr=1953#step:10:1020)

The failing test is related to #2172 (#1763 (comment))

Reference data:
https://github.com/aboutcode-org/scancode.io/blob/1763_pipeline_for_maven/scanpipe/tests/data/jvm/scancodeio_wisp-logging.json#L739

      "is_binary": true,
      "is_text": false,

However, in SCIO the file reports:

      "is_binary": false,
      "is_text": true,

So this discrepancy is not something to be fixed in this issue?

[chinyeungli]

I decided to change to another test file so that the test is passing.

[JonoYang]

There seems to be duplicate commits coming in when you updated your branch, I'm not sure how to resolve this.

[chinyeungli]

There seems to be duplicate commits coming in when you updated your branch, I'm not sure how to resolve this.

Are you referring to the DCO? Should I force it and click “Set DCO to pass”?

[tdruez]

I have doubts about this design, and I want to revisit a concern I raised earlier that wasn't addressed: #1953 (review)

"Do we need a dedicated pipeline for just an extra step? Shouldn't the original scan_single_package detect that it's a Maven package and apply the necessary? Any reason to keep this new logic separated?"

This new pipeline ScanMavenPackage implementation merges two fundamentally different processing paths: a basic scan and a full D2D analysis, into one class controlled by a d2d_option_enabled boolean.

The current design will be confusing to maintain and to users.

[chinyeungli]

I have doubts about this design, and I want to revisit a concern I raised earlier that wasn't addressed: #1953 (review)

"Do we need a dedicated pipeline for just an extra step? Shouldn't the original scan_single_package detect that it's a Maven package and apply the necessary? Any reason to keep this new logic separated?"

This new pipeline ScanMavenPackage implementation merges two fundamentally different processing paths: a basic scan and a full D2D analysis, into one class controlled by a d2d_option_enabled boolean.

The current design will be confusing to maintain and to users.

@tdruez The current pipeline design combines the basic scan and the D2D steps that already exist in other pipelines, with two additional steps:
fetch_and_scan_remote_pom() and update_package_license_from_resource_if_missing().

• fetch_and_scan_remote_pom()
  If a package doesn't have a pom.xml, try to find and fetch and scan the remote (upstream) pom.xml if exist.

• update_package_license_from_resource_if_missing()
  If a package doesn't have a declared/detected package license, but a license is found at the file level, update the package license using the license detected at the file level.

So, I guess the question is is it necessary to have a dedicated pipeline for these 2 extra steps.

@pombredanne any comment on it?

[pombredanne]

@chinyeungli I like the dedicated pipeline as it helps focus the analysis, we just need to make sure we have design that @tdruez agrees to. One question is the way to invoke this. IMHO the input should be ONE PURL, and based on that the POM, the src JAR, the binary JAR(s) should be fetched, and possibly also even the corresponding source git repo should be fetched and processed. That way I could run it at the scale of a whole Maven repo, spawning one SCIO project for each PURL. Another question is if we want to also include C/C++ and native binaries steps, but that's for later. Let's discuss the details in the next call, if that works for you and @tdruez

[pombredanne]

At last some feedback!

[pombredanne]

I am not sure you want to call all mapping steps inside a step? Should not these be their own steps?

[chinyeungli]

I've separated the logic into different steps

[pombredanne]

Should this be in ScanCode? Do we have that already somewhere in PurlDB too?

[chinyeungli]

removed as this is not needed.

[pombredanne]

Using this hardcoded URL means that we cannot run that on any other repo. We need to find a way to avoid that. If we use a PURL as a project input, then the Maven repo will be part of the PURL (most of the times as default), but then we do not have to further dive into that

[chinyeungli]

Updated code to use PURL as an input.

[pombredanne]

Suggested change

	def get_pom_url_list(input_source, packages):
	def get_pom_urls(input_source, packages):

[pombredanne]

May be use:

Suggested change

	pom_url_list = []
	pom_urls = []

[pombredanne]

See my comment above: no hardcoding of hosts should be needed

[pombredanne]

Do not we have code to do that in FetchCode, packageurl-python and/or scancode-toolkit? Also I have a hard time to understand all the parsing with indices. Do you have an example or tests of what you are trying to accomplish?

[pombredanne]

It is best to avoid a _list suffix IMHO

Suggested change

	def download_pom_files(pom_url_list):
	def download_pom_files(pom_url):

[pombredanne]

Could we use fetchcode to fetch instead?

[chinyeungli]

Yes, we could, but the issue is do we want to.
Following is the code if we switch to fetchcode which seems making things complicated, OR we can move the "fetch_http" to fetchcode and use it.

        import os
        import fetchcode
        downloaded_pom = fetchcode.fetch(pom_url)
        location = str(downloaded_pom.location)
        path = location + ".pom"
        # The fetch function from fetchcode save the file as /tmp/name
        # without an extension. We need to add the ".pom" extension so that
        # the package scan can work properly for this file.
        os.rename(location, path)

[chinyeungli]

I've redesigned the code. It now only takes 1 PURL as input.
I've also removed the manual option for d2d. If both the source and binary are available, d2d will be performed automatically, otherwise, only a basic scan is performed.

Example 1:
PURL: pkg:maven/aero/champ/cargojson@1.0

Without fetching the remote POM and updating the package data, no packages are detected because the fetched binary (https://repo1.maven.org/maven2/aero/champ/cargojson/1.0/cargojson-1.0.jar) doesn't contain a pom.xml.

However, with remote POM fetching and package data updates enabled, we successfully detect 1 package and update 17 dependencies in the summary:



(We can see the POM file was fetched remotely under "Datafile paths".)

Example 2:
PURL: pkg:maven/activemq/activemq-axis@1.2
https://repo1.maven.org/maven2/activemq/activemq-axis/1.2/

Since there is no source available for this package, only a basic scan is performed.