Skip to content

feat(storage): rewrite GCS provider to IStorageProvider parity#1509

Open
kkopanidis wants to merge 1 commit into
mainfrom
feat/storage-gcs-provider-rewrite
Open

feat(storage): rewrite GCS provider to IStorageProvider parity#1509
kkopanidis wants to merge 1 commit into
mainfrom
feat/storage-gcs-provider-rewrite

Conversation

@kkopanidis

Copy link
Copy Markdown
Contributor

Summary

The Google Cloud Storage provider was deprecated and missing critical IStorageProvider methods. This rewrites it to parity with AWS S3 and Azure.

Problems fixed:

  • deleteContainer and deleteFolder threw "not implemented"
  • getSignedUrl never returned a value
  • exists always returned true
  • Folder markers used name/keep.txt instead of the shared name.keep.txt convention
  • Public access relied on legacy ACLs (makePublic/makePrivate) incompatible with Uniform Bucket-Level Access

What changed:

  • Full IStorageProvider implementation with metrics parity (containers_total, folders_total increment/decrement)
  • Auth via service account key file, inline JSON (google.serviceAccountKeyJson), or Application Default Credentials
  • v4 signed URLs for read/upload and long-lived public URLs on private buckets
  • UBLA-safe bucket public access via IAM (roles/storage.objectViewer for allUsers)
  • New buckets created with Uniform Bucket-Level Access enabled

Test plan

  • Configure storage provider google with a GCS bucket (or fake-gcs-server emulator)
  • Auth: verify key file path, inline JSON, and ADC (empty credentials) each initialize successfully
  • Container lifecycle: createContainer (public + private), containerExists, setContainerPublicAccess, deleteContainer
  • Folder lifecycle: createFolder, folderExists (marker at {folder}.keep.txt), deleteFolder removes marker and prefixed objects
  • File ops: store, get, exists (returns false for missing files), delete
  • URLs: getSignedUrl returns a working v4 URL; getUploadUrl accepts PUT; getPublicUrl returns direct URL for public buckets and signed URL for private buckets
  • Metrics: confirm containers_total / folders_total increment on create and decrement on delete

Replace the deprecated Google Cloud Storage provider with a full
implementation aligned to AWS S3 and Azure, including UBLA-safe IAM
public access, v4 signed URLs, ADC/inline JSON auth, and fixes for
deleteContainer, deleteFolder, exists, getSignedUrl, and folder markers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant