Skip to content

Add garage 0.9.3 community pack (S3-compatible object store) #269

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3pings wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add garage 0.9.3 community pack (S3-compatible object store) #269

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 213 additions & 0 deletions packs/garage-0.9.3/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
# Garage

[Garage](https://garagehq.deuxfleurs.fr/) is a lightweight, S3-compatible distributed object store built by [Deuxfleurs](https://deuxfleurs.fr/). It is designed for self-hosted, geo-distributed deployments running on commodity hardware — it tolerates high-latency links between nodes and does not require a dedicated storage network, which makes it a good fit for edge clusters and small on-prem footprints where Ceph or MinIO would be too heavy.

This pack deploys Garage as a 3-replica StatefulSet using the upstream Helm chart (chart `0.9.3`, app `v2.3.0`). Nodes discover each other automatically through the built-in Kubernetes discovery mechanism, so no bootstrap peer list is needed.

> [!IMPORTANT]
> Garage does not self-assemble into a usable cluster. After the pods are running you **must** manually assign a layout to each node before the S3 API will accept any data. See [Post-Installation](#post-installation).


## Prerequisites

- A Kubernetes cluster with a default StorageClass, or an explicit `storageClass` set under `persistence` (see [Storage](#storage)).
- Permission to create cluster-scoped resources. The chart installs the `garagenodes.deuxfleurs.fr` CRD and a ClusterRole used for peer discovery. If you cannot grant cluster-scoped access, set `garage.kubernetesSkipCrd: true` and install the CRD out of band.
- At least 3 schedulable nodes if you keep the default `replicationFactor: "3"`. With fewer nodes, replicas will co-locate and you lose the durability the replication factor implies.


## Parameters

| **Parameter** | **Description** | **Type** | **Default Value** | **Required** |
|---|---|---|---|---|
| `deployment.kind` | `StatefulSet` (PVC-backed) or `DaemonSet` (hostPath-backed) | String | `StatefulSet` | No |
| `deployment.replicaCount` | Number of Garage nodes | Integer | `3` | No |
| `garage.replicationFactor` | Copies of each object across the cluster | String | `"3"` | No |
| `garage.consistencyMode` | `consistent` (read-after-write), `degraded`, or `dangerous` | String | `"consistent"` | No |
| `garage.dbEngine` | Metadata engine: `lmdb` or `sqlite` | String | `"lmdb"` | No |
| `garage.blockSize` | Data block size in bytes | String | `"1048576"` | No |
| `garage.compressionLevel` | zstd level for stored blocks | String | `"1"` | No |
| `garage.metadataAutoSnapshotInterval` | Interval for automatic metadata DB snapshots, e.g. `6h`. Empty disables | String | `""` | No |
| `garage.rpcSecret` | Shared secret for inter-node RPC. Generated and stored in a Secret if left empty | String | `""` | No |
| `garage.existingRpcSecret` | Name of an existing Secret holding the RPC secret under key `rpcSecret` | String | `""` | No |
| `garage.kubernetesSkipCrd` | Skip installing the `garagenodes` CRD | Boolean | `false` | No |
| `garage.s3.api.region` | S3 region name advertised to clients | String | `"garage"` | No |
| `garage.s3.api.rootDomain` | Suffix enabling virtual-hosted bucket addressing | String | `".s3.garage.tld"` | No |
| `garage.s3.web.rootDomain` | Suffix for static website hosting from buckets | String | `".web.garage.tld"` | No |
| `garage.additionalTopLevelConfig` | Raw TOML appended to `garage.toml` | String | `""` | No |
| `garage.garageTomlString` | Full `garage.toml` template. **Overrides all other `garage.*` values** | String | `""` | No |
| `persistence.enabled` | Persist metadata and data | Boolean | `true` | No |
| `persistence.meta.size` | Metadata volume size | String | `100Mi` | **Yes — raise it** |
| `persistence.data.size` | Data volume size | String | `100Mi` | **Yes — raise it** |
| `service.type` | `ClusterIP`, `NodePort`, or `LoadBalancer` | String | `ClusterIP` | No |
| `ingress.s3.api.enabled` | Expose the S3 API through an Ingress | Boolean | `false` | No |
| `ingress.s3.web.enabled` | Expose bucket static-website serving through an Ingress | Boolean | `false` | No |
| `monitoring.metrics.enabled` | Annotate a Service for Prometheus scraping | Boolean | `false` | No |
| `monitoring.metrics.serviceMonitor.enabled` | Create a `ServiceMonitor` (requires Prometheus Operator) | Boolean | `false` | No |
| `resources` | Pod resource requests and limits | Object | `{}` | No |

The full set of chart values is documented in [`charts/garage/README.md`](charts/garage/README.md).


## Storage

The chart ships with `100Mi` volumes for both metadata and data. **These defaults are placeholders and are not usable for real workloads** — size them before the first deploy:

```yaml
charts:
garage:
persistence:
meta:
storageClass: "fast-storage-class" # SSD-backed; LMDB is latency-sensitive
size: 5Gi
data:
storageClass: "bulk-storage-class" # Capacity matters more than latency here
size: 500Gi
```

Resizing after deployment is awkward: StatefulSet `volumeClaimTemplates` are immutable, so growing volumes requires orphaning the StatefulSet (`kubectl delete sts --cascade=orphan`), editing the PVCs, and letting the pack recreate the controller. Your StorageClass must have `allowVolumeExpansion: true`. Size generously up front.

Setting `deployment.kind: DaemonSet` switches Garage to hostPath storage at `persistence.meta.hostPath` and `persistence.data.hostPath`, placing one node per host. This suits edge clusters with direct-attached disks, but the paths must exist and be writable by UID/GID `1000`.


## Usage

To use this pack, create or edit an [add-on cluster profile](https://docs.spectrocloud.com/profiles/cluster-profiles/create-cluster-profiles/create-addon-profile/), search for the **garage** pack, and adjust the values. A minimal production-shaped override:

```yaml
charts:
garage:
deployment:
replicaCount: 3
garage:
replicationFactor: "3"
consistencyMode: "consistent"
metadataAutoSnapshotInterval: "6h"
s3:
api:
region: "us-east-1"
rootDomain: ".s3.example.com"
persistence:
meta:
storageClass: "fast-storage-class"
size: 5Gi
data:
storageClass: "bulk-storage-class"
size: 500Gi
resources:
requests:
cpu: 250m
memory: 1Gi
limits:
memory: 2Gi
```

The pack deploys into the `garage` namespace.

### Exposing the S3 API

By default the API is only reachable in-cluster at `garage.garage.svc.cluster.local:3900`. To reach it from outside, either set `service.type: LoadBalancer` or enable the Ingress. Virtual-hosted bucket addressing (`bucket.s3.example.com`) requires a wildcard host, and `garage.s3.api.rootDomain` must match:

```yaml
charts:
garage:
ingress:
s3:
api:
enabled: true
className: "nginx"
hosts:
- host: "s3.example.com"
paths:
- path: /
pathType: Prefix
- host: "*.s3.example.com"
paths:
- path: /
pathType: Prefix
tls:
- secretName: garage-s3-tls
hosts:
- "s3.example.com"
- "*.s3.example.com"
```

S3 clients that only support path-style addressing (`s3.example.com/bucket`) work without the wildcard host.


## Post-Installation

A freshly deployed Garage cluster has no layout, which means it has zero usable capacity and rejects all S3 operations. The `garage` CLI is bundled in the container image; run it via `kubectl exec` against any pod.

**1. Confirm all nodes see each other.** Each pod should be listed with a node ID:

```bash
kubectl exec -it -n garage garage-0 -- ./garage status
```

**2. Assign a layout role to each node.** Use the first several characters of each node ID (enough to be unique). `--zone` should reflect physical failure domains — with `replicationFactor: 3`, Garage spreads copies across distinct zones when it can. `--capacity` is the share of data the node accepts, and should track the size of its data volume:

```bash
kubectl exec -it -n garage garage-0 -- ./garage layout assign <node-id-1> -z dc1 -c 500G -t garage-0
kubectl exec -it -n garage garage-0 -- ./garage layout assign <node-id-2> -z dc2 -c 500G -t garage-1
kubectl exec -it -n garage garage-0 -- ./garage layout assign <node-id-3> -z dc3 -c 500G -t garage-2
```

**3. Review and apply.** The version number must be exactly one greater than the current layout version — `1` for the first apply:

```bash
kubectl exec -it -n garage garage-0 -- ./garage layout show
kubectl exec -it -n garage garage-0 -- ./garage layout apply --version 1
```

**4. Create a bucket and a key, then grant access:**

```bash
kubectl exec -it -n garage garage-0 -- ./garage bucket create my-bucket
kubectl exec -it -n garage garage-0 -- ./garage key create my-app-key
kubectl exec -it -n garage garage-0 -- ./garage bucket allow --read --write my-bucket --key my-app-key
```

`key create` prints the access key ID and secret access key **once** — capture them at that moment. Point any S3 client at the API endpoint using the region from `garage.s3.api.region`.

Adding or removing nodes later repeats steps 2–3 with an incremented `--version`. Garage rebalances data automatically after the new layout is applied.


## Monitoring

Garage exposes Prometheus metrics on the admin API port (`3903`). With the Prometheus Operator installed:

```yaml
charts:
garage:
monitoring:
metrics:
enabled: true
serviceMonitor:
enabled: true
interval: 30s
```

The admin API is deliberately excluded from the main Service because its responses are not consistent across nodes — query a specific pod when using it directly.


## Uninstalling

Removing the pack from the cluster profile deletes the workloads but leaves two things behind on purpose:

- **PersistentVolumeClaims** are retained by the StatefulSet controller. Delete them explicitly to reclaim storage — and only once you are certain the data is not needed.
- **The `garagenodes.deuxfleurs.fr` CRD** is not removed by Helm and must be deleted manually if you are permanently retiring Garage.


## Kubernetes Compatibility

Requires Kubernetes v1.19 or later (`networking.k8s.io/v1` Ingress).


## References

- [Garage documentation](https://garagehq.deuxfleurs.fr/documentation/)
- [Deploying on Kubernetes](https://garagehq.deuxfleurs.fr/documentation/cookbook/kubernetes/)
- [Creating a cluster layout](https://garagehq.deuxfleurs.fr/documentation/operations/layout/)
- [Configuration reference](https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/)
- [S3 compatibility matrix](https://garagehq.deuxfleurs.fr/documentation/reference-manual/s3-compatibility/)
- [Source code](https://git.deuxfleurs.fr/Deuxfleurs/garage)
Binary file added packs/garage-0.9.3/charts/garage-0.9.3.tgz
View file
Open in desktop
Binary file not shown.
23 changes: 23 additions & 0 deletions packs/garage-0.9.3/charts/garage/.helmignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*.orig
*~
# Various IDEs
.project
.idea/
*.tmproj
.vscode/
18 changes: 18 additions & 0 deletions packs/garage-0.9.3/charts/garage/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: v2
name: garage
description: S3-compatible object store for small self-hosted geo-distributed deployments
type: application
version: 0.9.3
appVersion: "v2.3.0"
home: https://garagehq.deuxfleurs.fr/
icon: https://garagehq.deuxfleurs.fr/images/garage-logo.svg

keywords:
- geo-distributed
- read-after-write-consistency
- s3-compatible

sources:
- https://git.deuxfleurs.fr/Deuxfleurs/garage.git

maintainers: []
100 changes: 100 additions & 0 deletions packs/garage-0.9.3/charts/garage/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# garage

![Version: 0.9.3](https://img.shields.io/badge/Version-0.9.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: v2.3.0](https://img.shields.io/badge/AppVersion-v2.3.0-informational?style=flat-square)

S3-compatible object store for small self-hosted geo-distributed deployments

**Homepage:** <https://garagehq.deuxfleurs.fr/>

## Source Code

* <https://git.deuxfleurs.fr/Deuxfleurs/garage.git>

## Values

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| affinity | object | `{}` | |
| commonLabels | object | `{}` | Extra labels for all resources |
| deployment.kind | string | `"StatefulSet"` | Switchable to DaemonSet |
| deployment.podManagementPolicy | string | `"OrderedReady"` | If using statefulset, allow Parallel or OrderedReady (default) |
| deployment.replicaCount | int | `3` | Number of StatefulSet replicas/garage nodes to start |
| environment | object | `{}` | |
| extraVolumeMounts | object | `{}` | |
| extraVolumes | object | `{}` | |
| fullnameOverride | string | `""` | |
| garage.blockSize | string | `"1048576"` | Defaults is 1MB An increase can result in better performance in certain scenarios https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#block_size |
| garage.bootstrapPeers | list | `[]` | This is not required if you use the integrated kubernetes discovery |
| garage.compressionLevel | string | `"1"` | zstd compression level of stored blocks https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#compression_level |
| garage.dbEngine | string | `"lmdb"` | Can be changed for better performance on certain systems https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#db_engine |
| garage.existingConfigMap | string | `""` | if not empty string, allow using an existing ConfigMap for the garage.toml, if set, ignores garage.toml |
| garage.garageTomlString | string | `""` | String Template for the garage configuration if set, ignores above values. Values can be templated, see https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/ |
| garage.kubernetesSkipCrd | bool | `false` | Set to true if you want to use k8s discovery but install the CRDs manually outside of the helm chart, for example if you operate at namespace level without cluster resources |
| garage.replicationFactor | string | `"3"` | Default to 3 replicas, see the replication_factor section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#replication_factor |
| garage.consistencyMode | string | `"consistent"` | Default to read-after-write consistency, see the consistency_mode section at https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#consistency_mode |
| garage.metadataAutoSnapshotInterval | string | `""` | If this value is set, Garage will automatically take a snapshot of the metadata DB file at a regular interval and save it in the metadata directory. https://garagehq.deuxfleurs.fr/documentation/reference-manual/configuration/#metadata_auto_snapshot_interval |
| garage.admin.apiBindAddr | string | `"[::]:3903"` | |
| garage.rpcBindAddr | string | `"[::]:3901"` | |
| garage.rpcSecret | string | `""` | If not given, a random secret will be generated and stored in a Secret object |
| garage.s3.api.bindAddr | string | `"[::]:3900"` | |
| garage.s3.api.region | string | `"garage"` | |
| garage.s3.api.rootDomain | string | `".s3.garage.tld"` | |
| garage.s3.web.index | string | `"index.html"` | |
| garage.s3.web.bindAddr | string | `"[::]:3902"` | |
| garage.s3.web.rootDomain | string | `".web.garage.tld"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"dxflrs/amd64_garage"` | default to amd64 docker image |
| image.tag | string | `""` | set the image tag, please prefer using the chart version and not this to avoid compatibility issues |
| imagePullSecrets | list | `[]` | set if you need credentials to pull your custom image |
| ingress.s3.api.annotations | object | `{}` | Rely _either_ on the className or the annotation below but not both! If you want to use the className, set className: "nginx" and replace "nginx" by an Ingress controller name, examples [here](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers). |
| ingress.s3.api.enabled | bool | `false` | |
| ingress.s3.api.hosts[0] | object | `{"host":"s3.garage.tld","paths":[{"path":"/","pathType":"Prefix"}]}` | garage S3 API endpoint, to be used with awscli for example |
| ingress.s3.api.hosts[1] | object | `{"host":"*.s3.garage.tld","paths":[{"path":"/","pathType":"Prefix"}]}` | garage S3 API endpoint, DNS style bucket access |
| ingress.s3.api.labels | object | `{}` | |
| ingress.s3.api.tls | list | `[]` | |
| ingress.s3.web.annotations | object | `{}` | Rely _either_ on the className or the annotation below but not both! If you want to use the className, set className: "nginx" and replace "nginx" by an Ingress controller name, examples [here](https://kubernetes.io/docs/concepts/services-networking/ingress-controllers). |
| ingress.s3.web.enabled | bool | `false` | |
| ingress.s3.web.hosts[0] | object | `{"host":"*.web.garage.tld","paths":[{"path":"/","pathType":"Prefix"}]}` | wildcard website access with bucket name prefix |
| ingress.s3.web.hosts[1] | object | `{"host":"mywebpage.example.com","paths":[{"path":"/","pathType":"Prefix"}]}` | specific bucket access with FQDN bucket |
| ingress.s3.web.labels | object | `{}` | |
| ingress.s3.web.tls | list | `[]` | |
| initImage.pullPolicy | string | `"IfNotPresent"` | |
| initImage.repository | string | `"busybox"` | |
| initImage.tag | string | `"stable"` | |
| livenessProbe | object | `{}` | Specifies a livenessProbe |
| monitoring.metrics.enabled | bool | `false` | If true, a service for monitoring is created with a prometheus.io/scrape annotation |
| monitoring.metrics.serviceMonitor.enabled | bool | `false` | If true, a ServiceMonitor CRD is created for a prometheus operator https://github.com/coreos/prometheus-operator |
| monitoring.metrics.serviceMonitor.interval | string | `"15s"` | |
| monitoring.metrics.serviceMonitor.labels | object | `{}` | |
| monitoring.metrics.serviceMonitor.path | string | `"/metrics"` | |
| monitoring.metrics.serviceMonitor.relabelings | list | `[]` | |
| monitoring.metrics.serviceMonitor.scheme | string | `"http"` | |
| monitoring.metrics.serviceMonitor.scrapeTimeout | string | `"10s"` | |
| monitoring.metrics.serviceMonitor.tlsConfig | object | `{}` | |
| monitoring.tracing.sink | string | `""` | specify a sink endpoint for OpenTelemetry Traces, eg. `http://localhost:4317` |
| nameOverride | string | `""` | |
| nodeSelector | object | `{}` | |
| persistence.data.hostPath | string | `"/var/lib/garage/data"` | |
| persistence.data.size | string | `"100Mi"` | |
| persistence.enabled | bool | `true` | |
| persistence.meta.hostPath | string | `"/var/lib/garage/meta"` | |
| persistence.meta.size | string | `"100Mi"` | |
| podAnnotations | object | `{}` | additional pod annotations |
| podSecurityContext.fsGroup | int | `1000` | |
| podSecurityContext.runAsGroup | int | `1000` | |
| podSecurityContext.runAsNonRoot | bool | `true` | |
| podSecurityContext.runAsUser | int | `1000` | |
| readinessProbe | object | `{}` | Specifies a readinessProbe |
| resources | object | `{}` | |
| securityContext.capabilities | object | `{"drop":["ALL"]}` | The default security context is heavily restricted, feel free to tune it to your requirements |
| securityContext.readOnlyRootFilesystem | bool | `true` | |
| service.s3.api.port | int | `3900` | |
| service.s3.web.port | int | `3902` | |
| service.type | string | `"ClusterIP"` | You can rely on any service to expose your cluster - ClusterIP (+ Ingress) - NodePort (+ Ingress) - LoadBalancer |
| serviceAccount.annotations | object | `{}` | Annotations to add to the service account |
| serviceAccount.create | bool | `true` | Specifies whether a service account should be created |
| serviceAccount.name | string | `""` | The name of the service account to use. If not set and create is true, a name is generated using the fullname template |
| tolerations | list | `[]` | |

----------------------------------------------
Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
Loading
Loading