Features and code reorganization to support the KBase Lakehouse

[jeff-cohere]

This PR brings in some interoperability features needed to support a deployment of the DTS to the KBase Lakehouse environment:

• The concept of "connections" has been introduced to the Endpoint interface to allow endpoints with different providers to interoperate. A couple of supporting methods are added, along with one that reports the UUID for an endpoint itself.
• The "local" endpoint type has been modified to support transfers to Globus endpoints via HTTPS. This means the DTS can write a transfer manifest to its local storage and transfer this manifest directly to its destination without an additional dedicated Globus collection.
• The Globus endpoint can now interoperate with an S3 endpoint using the Globus Connect Server Manager API where it's available. DTS extracts a KBase user's S3 credentials from the KBase MMS during authentication (if it's available) and then adds the credentials, associating them with the authenticated user. This machinery needs to be tested in the Lakehouse environment. It probably also needs to be hardened a bit.

Because we're now using several different Globus APIs, I've also reorganized the Globus logic into its own source file. This makes the Globus endpoint logic more transparent.

Closes #242

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://kbase.github.io/dts/pr-preview/pr-243/
Built to branch gh-pages at 2026-09-30 23:51 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[jeff-cohere]

The minio/minio Docker image seems unavailable as of a few minutes ago. Routine botstorm or discontinued? I suppose we'll find out soon.

[Copilot]

🟡 Changes recommended

Cross-provider transfers currently contain blocking path, credential-handling, HTTP, concurrency, and secret-persistence defects.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds Lakehouse interoperability across local, Globus, and S3 endpoints.

Changes:

• Extends endpoint identity, path, and connection APIs.
• Adds Globus HTTPS and S3 credential integration via MMS.
• Introduces a KBase Lakehouse database stub and configuration updates.

File summaries

File	Description
transfers/transfers.go	Updates provider and database registration.
transfers/transfers_test.go	Migrates endpoint path fixtures.
transfers/store.go	Simplifies transfer creation messages.
transfers/mover.go	Adds cross-provider connection setup.
services/version.go	Bumps version to 0.15.0.
services/prototype.go	Clarifies KBase authentication fallback.
integration/irods/fixtures/test-config.yaml	Migrates local path configuration.
endpoints/s3/endpoint.go	Implements expanded endpoint interface.
endpoints/s3/endpoint_test.go	Updates S3 path assertions.
endpoints/local/endpoint.go	Adds split paths and Globus HTTPS uploads.
endpoints/local/endpoint_test.go	Updates local endpoint fixtures.
endpoints/globus/globus.go	Adds dedicated Globus API clients.
endpoints/globus/endpoint.go	Refactors Globus endpoint integration.
endpoints/endpoints.go	Expands the endpoint contract.
dtstest/dtstest.go	Updates endpoint test doubles.
deployment/dts.yaml	Revises deployment endpoint configuration.
databases/nmdc/database_test.go	Migrates Globus path fixtures.
databases/kbase_lakehouse/database.go	Adds the Lakehouse database stub.
auth/kbase_mms.go	Adds MMS credential retrieval.
auth/kbase_auth_server.go	Associates users with connection credentials.
auth/authenticator.go	Initializes credential maps.
auth/auth.go	Extends users and credentials for connections.

Review details

Suppressed comments (2)

endpoints/globus/globus.go:712

• As in get, filepath.Join corrupts an HTTPS URL (https://… becomes https:/…). This makes every Manager API POST, including user-credential registration, fail before reaching Globus.

	resourcePath := filepath.Join(c.Url, resource)

endpoints/globus/endpoint.go:271

• This upload path ignores DataPath, despite the endpoint contract defining it as the directory containing endpoint data. A configured destination such as globus-kbase (data_path: jeff_cohere) receives local uploads under the base path instead. Apply the destination data path exactly once and reconcile the custom-destination path construction, which currently embeds that path in DestinationPath.

	absPath := filepath.Join(ep.Paths.Base, resource)

• Files reviewed: 22/22 changed files
• Comments generated: 16
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jeff-cohere]

The DTS Globus/S3 integration isn't working yet. In order to transfer files from a Globus endpoint to an S3 endpoint, a user's S3 credentials must be registered with the storage gateway underlying the collection in which the files reside. During the registration process, the DTS presents the user's Globus ID, Globus username, and an S3 credentials policy containing credentials obtained from the KBase MMS (passed along via the store process, which records all information specific to a requested transfer when it's created. The error encountered is described in the code attempting the registration, and relevant output is dumped.

There's a test script on dev1 in my home directory: /home/ac.jeff/dtspy/jdp-xfer.py. It can be executed from that directory with

uv run jdp-xfer.py

and you can check the log messages in the DTS instance running in the dts-dev namespace in the BERDL Rancher 2 environment at Argonne.

Hopefully it won't take much more time/effort to take this to the finish line. I wish we had more time to get this working!