diff --git a/docs/admin/configuration/storage/index.md b/docs/admin/configuration/storage/index.md index 5e06f6fa..86d426c3 100644 --- a/docs/admin/configuration/storage/index.md +++ b/docs/admin/configuration/storage/index.md @@ -14,3 +14,4 @@ Use these guides to configure the storage backend for your OpenCloud deployment. - [PosixFS](./posixfs.md) - Use a regular filesystem as the OpenCloud storage backend - [DecomposedS3](./decomposeds3.md) - Use S3-compatible object storage with MinIO +- [Cleaning Up Old Versions](./revisions-cleanup.md) - Delete old file versions automatically, according to a cleanup strategy diff --git a/docs/admin/configuration/storage/revisions-cleanup.md b/docs/admin/configuration/storage/revisions-cleanup.md new file mode 100644 index 00000000..20bbd2f1 --- /dev/null +++ b/docs/admin/configuration/storage/revisions-cleanup.md @@ -0,0 +1,382 @@ +--- +sidebar_position: 4 +id: storage-revisions-cleanup +title: 'Cleaning Up Old Versions' +description: How OpenCloud deletes old file versions according to a cleanup strategy +draft: false +--- + +# Cleaning Up Old Versions + +Every time a file is changed in OpenCloud, the previous content is kept as a version. Users can see these versions in the web interface and restore an earlier one, for example after saving a document by mistake. + +This is the case as long as versioning is enabled, which it is by default. Versioning can be turned off by setting `OC_DISABLE_VERSIONING` to `true`, see the [global environment variables](../../../dev/server/configuration/global-envvars.mdx). A changed file then simply replaces its previous content, and no new versions are created. + +Without a cleanup, OpenCloud keeps every version forever. On a server where files change often, the versions can take up much more space than the files themselves: a document that is saved twenty times a day has thousands of versions after a year, and every one of them is a full copy of the file at that time. + +The version cleanup deletes old versions according to a **cleanup strategy** that you choose, for example "keep the 10 newest versions", "delete versions older than 90 days", or "keep everything from the last week, and fewer versions the older they get". It works with all storage drivers: PosixFS, DecomposedFS and DecomposedS3. + +The cleanup is **disabled by default**. No file versions are deleted until you configure a strategy. + +:::note + +The configuration and the commands call versions _revisions_, for example `OC_REVISIONS_CLEANUP_STRATEGY`. Both words mean the same thing. + +::: + +## How the Cleanup Works + +### When the Cleanup Runs + +Once a strategy is configured, OpenCloud applies it to a file whenever the file gets a new version: + +- when a user uploads new content for an existing file, as soon as the upload is complete. If the upload is checked first, for example by the antivirus scanner, the cleanup runs once the check has passed. +- when a user restores an earlier version. The version that holds the content from just before the restore is kept, so that the restore can be undone, see [After a Restore](#after-a-restore) below. +- with PosixFS in [collaborative mode](./posixfs.md#posixfs-collaborative-mode), when a file is changed directly in the file system and OpenCloud picks up the change. This only creates a version if `STORAGE_USERS_POSIX_ENABLE_FS_REVISIONS` is set to `true`, which is not the default. + +The cleanup only looks at the file that got the new version. Files that are never changed again keep all their versions, even if the strategy would delete some of them. To clean those up too, or to apply a different strategy once, use the [command to clean up versions on demand](../../maintenance/maintenance-commands.md#clean-up-old-file-versions). + +The cleanup skips a file while an upload to it is still being checked. It cleans up that file with its next version. + +### After a Restore + +When a user restores an earlier version, the content the file had until then becomes a version. The cleanup keeps this version for as long as the file has the restored content, whatever the strategy says. That way, the user can always undo the restore. + +Usually, the strategy would keep this version anyway, as it is the newest one. But not always. An example with `auto`, which keeps only the oldest version of each time slot: + +1. At 10:00:00, a user saves a document. The previous content becomes a version. +2. At 10:00:05, the user restores an earlier version, by mistake. The content saved at 10:00:00 becomes a version. +3. Both versions were created within the same slot of 10 seconds. `auto` would keep the one from 10:00:00 and delete the one from the restore, and with it the content the document had until a few seconds ago. Instead, it is kept. +4. The user notices the mistake and restores that version again. + +The same goes for `0`, which deletes all versions, and for a duration: as long as the file has the restored content, `30d` doesn't delete the version from the restore, not even after 30 days. + +The protection ends as soon as the file is changed again: by an upload, by another restore, or with PosixFS, by a change in the file system. From then on, the version is treated like any other, and the next cleanup applies the strategy to it. + +So, for every restored file, at most one version is kept beyond what the strategy allows, until the file is changed again. + +### The Cleanup Runs in the Background + +With the default settings, the cleanup doesn't slow down uploads. When a file gets a new version, OpenCloud puts the file into a queue, and background workers clean up the files in the queue one after the other. Usually, the cleanup is done a moment after the upload. + +Two settings change this, and can make uploads slower: with `OC_REVISIONS_CLEANUP_ASYNC` set to `false`, an upload waits until the cleanup of the file is done, and with `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL` set to `block`, an upload waits for room in the queue when the queue is full. See [Background Workers](#background-workers) below. + +If a file gets several new versions while it waits in the queue, it is cleaned up only once. + +## Turning the Cleanup On + +The cleanup is configured with environment variables of the OpenCloud service. The most important one is `OC_REVISIONS_CLEANUP_STRATEGY`, which sets the strategy. All variables are listed in the [environment variables of the storage-users service](../../../dev/server/services/storage-users/env-vars.mdx). + +### With OpenCloud Compose + +If you run OpenCloud with [OpenCloud Compose](https://github.com/opencloud-eu/opencloud-compose), create a file of your own for the setting, for example `revisions-cleanup.yml` in your `opencloud-compose` directory. That way, your setting is kept when you update OpenCloud Compose. + +```bash +nano opencloud-compose/revisions-cleanup.yml +``` + +Set the strategy for the `opencloud` service. In this example, OpenCloud keeps all versions from the last 7 days, keeps fewer versions the older they get, and deletes versions older than 90 days (the strategies are explained below): + +```yaml +services: + opencloud: + environment: + OC_REVISIONS_CLEANUP_STRATEGY: '7d, auto, 90d' + STORAGE_USERS_FILEMETADATA_CACHE_STORE: 'nats-js-kv' +``` + +The second setting is only needed for the [command to clean up versions on demand](../../maintenance/maintenance-commands.md#clean-up-old-file-versions). It makes OpenCloud keep its cache of file metadata in NATS, where the command can update it, instead of in the memory of the OpenCloud process. Without it, the command refuses to delete versions, see [Share the file metadata cache](../../maintenance/maintenance-commands.md#share-the-file-metadata-cache). If you never run the command, you can leave it out. + +Then add the file to the `COMPOSE_FILE` variable in your `.env` file, at the end of the list. For example, if your `.env` file has + +```env +COMPOSE_FILE=docker-compose.yml:traefik/opencloud.yml +``` + +change it to + +```env +COMPOSE_FILE=docker-compose.yml:traefik/opencloud.yml:revisions-cleanup.yml +``` + +Restart OpenCloud to apply the change: + +```bash +docker compose down +docker compose up -d +``` + +### With Other Installations + +Set the environment variable for the OpenCloud process. How you do that depends on how you run OpenCloud. The spaces after the commas are optional: `7d,auto,90d` is the same strategy as `7d, auto, 90d`. Without spaces, the value needs no quotes, which avoids mistakes, so the following examples leave them out. + +In a systemd unit, for example in a drop-in file that you create with `systemctl edit` and the name of your OpenCloud unit: + +```ini +[Service] +Environment=OC_REVISIONS_CLEANUP_STRATEGY=7d,auto,90d +``` + +If you prefer the strategy with spaces, put the whole assignment in double quotes: `Environment="OC_REVISIONS_CLEANUP_STRATEGY=7d, auto, 90d"`. + +In an environment file, for example one that your systemd unit reads with `EnvironmentFile=`, or one that you pass to `docker run` with `--env-file`. If you want to run the [cleanup command](../../maintenance/maintenance-commands.md#clean-up-old-file-versions) by hand or with cron, prefer an environment file to `Environment=` lines: the command needs the same settings as OpenCloud, and it can load the same file, while `Environment=` lines only reach the OpenCloud service. + +```env +OC_REVISIONS_CLEANUP_STRATEGY=7d,auto,90d +``` + +When you start the container with `docker run` directly: + +```bash +docker run -e OC_REVISIONS_CLEANUP_STRATEGY=7d,auto,90d ... +``` + +With spaces, quote the value for the shell: `-e 'OC_REVISIONS_CLEANUP_STRATEGY=7d, auto, 90d'`. + +Then restart OpenCloud. After changing a systemd unit, run `systemctl daemon-reload` first. + +If you want to use the [command to clean up versions on demand](../../maintenance/maintenance-commands.md#clean-up-old-file-versions), also set `STORAGE_USERS_FILEMETADATA_CACHE_STORE` to `nats-js-kv` in the same way, see [Share the file metadata cache](../../maintenance/maintenance-commands.md#share-the-file-metadata-cache). + +### Checking the Configuration + +OpenCloud checks the strategy when it starts. If the strategy is invalid, for example because of a typo, the storage-users service doesn't start, and its log says what is wrong with the strategy. This is deliberate: a strategy that is silently ignored, or that deletes more than intended, would be worse. + +To see what a strategy would delete before you configure it, run the cleanup command with `--dry-run`, see [Trying Out a Strategy](#trying-out-a-strategy) below. + +## Choosing a Strategy + +### Numbers and Durations + +A strategy is made of up to three parts, separated by commas. Each part is one of these: + +- **A number**, for example `10`, means a number of versions, counted from the newest one. `10` means "the 10 newest versions". +- **A duration**, for example `30d`, means the versions from that period of time, counted back from now. The units are `w` (weeks), `d` (days), `h` (hours), `m` (minutes) and `s` (seconds). `30d` means "the versions of the last 30 days". +- **`auto`** thins out the versions: the older the versions are, the fewer of them are kept. It is explained in more detail below. + +The spaces after the commas are optional: `5, 30d` and `5,30d` are the same. + +### The Example File + +The examples below all use the same file, a report that was worked on over several months. Today is October 21 at 15:00, and the report has these 8 versions, besides its current content. Each version was created when the report was saved, and holds the content it had until then: + +| Version | Created on | +| ------- | ----------------------------------------- | +| 1 | October 21, 14:50 (10 minutes ago) | +| 2 | October 21, 14:20 | +| 3 | October 21, 11:05 | +| 4 | October 20, 16:30 (yesterday) | +| 5 | October 14, 10:00 (one week ago) | +| 6 | September 30, 09:00 (three weeks ago) | +| 7 | August 12, 13:00 (ten weeks ago) | +| 8 | April 2, 08:00 (more than six months ago) | + +The current content of a file is never deleted by the cleanup, whatever the strategy. Only versions are. + +The cleanup goes by the time a version was created. The version list in the web interface shows another date for each version, the time its content was saved, see [How Old Is a Version?](#how-old-is-a-version). + +### Keeping a Number of Versions: `10` + +A single number keeps that many versions and deletes all older ones. + +With `5`, OpenCloud keeps versions 1 to 5 and deletes versions 6, 7 and 8. With `10`, all 8 versions are kept, as the report has fewer than 10. + +`0` deletes **all versions of every file that gets a new version**. Unlike turning versioning off with `OC_DISABLE_VERSIONING`, this also deletes the versions that already exist. + +### Keeping Versions for a Period of Time: `30d` + +A single duration keeps the versions of that period and deletes the older ones. + +With `30d`, the period starts on September 21 at 15:00. Versions 1 to 6 are kept, versions 7 and 8 are deleted. With `14d`, the period starts on October 7, so versions 1 to 5 are kept and versions 6 to 8 are deleted. + +A version is as old as the time since it was created, see [How Old Is a Version?](#how-old-is-a-version). With `30d`, the content a file had before a change is kept for 30 days after the change, however old that content was. + +### Counting From Midnight: `1D` + +A duration in days or weeks counts back from the current time by default: `1d` means the last 24 hours. With an uppercase unit, `D` or `W`, it counts from midnight instead (in the time zone of the server): `1D` means "since yesterday at 00:00", `0D` means "since today at 00:00", and `1W` "since 00:00 seven days ago". + +For the report, `1d` keeps the versions since yesterday at 15:00, which are versions 1 to 4. `1D` keeps the versions since yesterday at 00:00, which are also versions 1 to 4. The difference shows for a version created yesterday at 09:00: `1d` deletes it, as it is older than 24 hours, while `1D` keeps it, because it is from yesterday. + +### Thinning Out Versions: `auto` + +`auto` keeps fewer versions the older they get. It divides time into slots, and keeps the oldest version of each slot: + +| Age of the version | `auto` keeps | +| ------------------ | ---------------------------- | +| less than a minute | one version every 10 seconds | +| less than an hour | one version per minute | +| less than a day | one version per hour | +| less than 30 days | one version per day | +| older | one version per week | + +`auto` never deletes the only version of a slot. It deletes versions that were created shortly after each other, of which it only needs one. + +For the report, `auto` keeps all 8 versions: each one is the only version in its slot. Version 2 and version 1 are only 30 minutes apart, but they are less than an hour old, so each of them has a slot of one minute. + +Here is an example where `auto` makes a difference. A user edits a presentation yesterday afternoon and saves it six times between 14:00 and 14:05. Each save makes the previous content a version, so there are six versions, created between 14:00 and 14:05. Today, they are more than a day old, so they share one slot, the one for yesterday. `auto` keeps the oldest of them, the one created at 14:00, which holds the presentation as it was before the user started editing it. It deletes the other five, which hold the intermediate states of those five minutes. The state from 14:05 is still there, as the current content or as a later version. If the user had saved the presentation six times within the last few minutes, `auto` would still keep all six for now, and thin them out later as they get older. + +Why the oldest of a slot, not the newest? The oldest version of a slot holds the content the file had when the slot began, which may have been there for a long time. Keeping the newest instead would delete that content as soon as a user saves the file twice within a few seconds. + +`auto` alone never deletes a version that is the only one of its week. Over the years, a file that changes every week keeps one version per week. + +### Combining a Number and a Duration: `5, 30d` + +With two parts, the first part sets what is always kept, and the second part sets what is deleted: + +- `5, 30d` deletes versions older than 30 days, but always keeps the 5 newest versions, however old they are. +- `7d, 10` keeps the 10 newest versions, and also every version from the last 7 days, even if there are more than 10 of them. + +Let's look at the report again. Today is October 21 at 15:00: + +| Version | Created on | +| ------- | ----------------------------------------- | +| 1 | October 21, 14:50 (10 minutes ago) | +| 2 | October 21, 14:20 | +| 3 | October 21, 11:05 | +| 4 | October 20, 16:30 (yesterday) | +| 5 | October 14, 10:00 (one week ago) | +| 6 | September 30, 09:00 (three weeks ago) | +| 7 | August 12, 13:00 (ten weeks ago) | +| 8 | April 2, 08:00 (more than six months ago) | + +For the report, `5, 30d` keeps versions 1 to 6: versions 1 to 5 are the 5 newest, and version 6 is from the last 30 days. Versions 7 and 8 are deleted. + +`5, 30d` is a good choice for files that are rarely changed. Take a contract that was edited a few times in spring and then put aside. When someone changes it again in October, the versions from spring are months old. `30d` alone would delete all of them, and keep only the version created by the October change. `5, 30d` keeps the 5 newest, so the contract can still be restored to how it was during the spring. + +One part must be a number and the other a duration. Two numbers or two durations, like `5, 10`, are rejected, as one of them would have no effect: `5, 10` is the same as `10`. The error message tells you which value to use instead. + +The versions of the first part are always kept, whatever the second part says. The number counts all versions, including the ones the first part keeps anyway. For the report, which has 4 versions from the last 7 days: + +- `7d, 5` keeps versions 1 to 5: versions 1 to 4 are from the last 7 days, and version 5 makes 5 versions in total. It doesn't keep 5 versions in addition to the 4 from the last 7 days. +- `7d, 2` keeps versions 1 to 4. They are from the last 7 days, so they are kept, although that's more than 2. + +### Combining `auto` With a Number or a Duration + +`auto` can be combined with a number or a duration before or after it: + +- **`14d, auto`** keeps all versions of the last 14 days, and thins out the older ones with `auto`. +- **`auto, 90d`** deletes versions older than 90 days, and thins out the rest with `auto`. +- **`auto, 50`** keeps at most 50 versions, and thins them out with `auto`. +- **`7d, auto, 90d`** keeps all versions of the last 7 days, deletes versions older than 90 days, and thins out the versions in between. + +For the report, `7d, auto, 90d` keeps versions 1 to 4, as they are from the last 7 days. Versions 5, 6 and 7 are between 7 and 90 days old, and `auto` keeps all three, as each is the only version of its slot. Version 8 is older than 90 days and is deleted. + +With a number after `auto`, as in `auto, 50`, the number is applied first, and `auto` thins out the rest: `auto, 50` keeps at most 50 versions, and often fewer. It doesn't mean "thin out first, then keep 50 of what is left". + +In `7d, auto, 90d` or `2, auto, 50`, the first value must be smaller than the last one, otherwise `auto` would never apply. + +### Overview + +| Strategy | Meaning | Examples | +| -------------- | --------------------------------------------------------------------------------------- | -------------------------------- | +| `disabled` | Keep all versions (the default) | `disabled` | +| `X` | Delete everything beyond `X` | `10`, `30d`, `0` | +| `auto` | Thin out versions the older they get | `auto` | +| `X, auto` | Keep everything within `X`, thin out the older versions | `14d, auto` | +| `auto, X` | Delete everything beyond `X`, thin out the rest | `auto, 90d`, `auto, 50` | +| `X1, X2` | Keep everything within `X1`, delete everything beyond `X2`; one number and one duration | `5, 30d`, `7d, 10` | +| `X1, auto, X2` | Keep everything within `X1`, delete everything beyond `X2`, thin out in between | `7d, auto, 90d`, `1D, auto, 30d` | + +### Suggestions + +Which strategy is right depends on how your users work and how much storage you have. Some starting points: + +- **Keep everything recent, and a history for a few months:** `7d, auto, 90d`. Users can go back to any version of the last week, and to about one version per day or week for three months. +- **Save storage, but never lose the last few versions of a file:** `5, 30d`. +- **A fixed upper limit per file:** `auto, 50` or simply `20`. +- **Keep a long history, but not every save:** `auto`. The number of versions grows slowly, by about one per week for files that keep changing. + +## How Old Is a Version? + +The age of a version is the time since it was created: when the file was changed, and its previous content became the version. OpenCloud records that time on the server when it creates the version. + +It is _not_ the date that the version list in the web interface shows for the version. That date is when the content of the version was saved, as the app that saved it says, so it can be any date, also one in the future. For example, a desktop client that uploads a file keeps the modification time the file has on the user's computer. + +For files that are changed often, the two dates are close together. For files that are left alone for a long time, they are far apart. An example: + +1. On August 14, a user saves a document. +2. Nobody touches the document for two months. +3. On October 14, the user changes it. The content from August 14 becomes a version. The version list shows it as August 14. +4. The cleanup runs right after the change. For the cleanup, the version is a few seconds old. + +With `30d`, this version is kept until November 13, 30 days after the change. So the content a file had before a change can always be restored for the duration of the strategy, however old that content was. + +The cleanup also orders the versions by the time they were created, for example to find the 10 newest. A version whose content is dated in the future therefore doesn't count as the newest one, and can't push the actual newest versions out. + +Versions created before OpenCloud recorded the time of their creation, that is, with an earlier version of OpenCloud, get an estimate instead: the time they were replaced, as far as OpenCloud can tell it from the dates of the newer versions and of the file. The estimate is never later than the current time. + +## Trying Out a Strategy + +Before you configure a strategy, or before you change it, you can see what it would delete. The cleanup command has a dry run, which deletes nothing and lists every version with what would happen to it and why: + +```bash +docker compose exec opencloud opencloud storage-users revisions cleanup --all --strategy "7d, auto, 90d" --dry-run --verbose +``` + +For the report from above, the output would look like this (shortened, the lines for versions 3 to 7 are left out): + +```plaintext +25b69780-5f39-43be-a7ac-a9b9e9fe4230 /Documents/report.odt (node 8a1c0e2f-6b4d-4f7a-9c3e-2d5b7f1a0c94): would delete 1 of 8 revisions (2.1 MiB) + protected 2026-10-21T14:50:00+02:00 2.3 MiB protected by "7d": newer than 2026-10-14T15:00:00+02:00 + protected 2026-10-21T14:20:00+02:00 2.3 MiB protected by "7d": newer than 2026-10-14T15:00:00+02:00 + ... + deleted 2026-04-02T08:00:00+02:00 2.1 MiB deleted by "90d": older than 2026-07-23T15:00:00+02:00 +``` + +A dry run works with any configuration of the file metadata cache, as it deletes nothing. See [Clean up old file versions](../../maintenance/maintenance-commands.md#clean-up-old-file-versions) for all options of the command. + +## Background Workers + +The settings of the background workers rarely need to be changed. The defaults work for most installations. + +| Variable | Default | Meaning | +| ----------------------------------------------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `OC_REVISIONS_CLEANUP_ASYNC` | `true` | Clean up in background workers. With `false`, the cleanup runs right away, and the operation that created the new version, for example the upload, waits until the cleanup is done. | +| `OC_REVISIONS_CLEANUP_ASYNC_WORKERS` | `0` (automatic) | The number of background workers. Automatically, half the number of CPUs, and at least 2. | +| `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_SIZE` | `0` (automatic) | The number of files that can wait for a worker. Automatically, 64 per worker. | +| `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL` | `drop` | What happens when the queue is full. With `drop`, the cleanup of that file is skipped, and happens with its next version instead. With `block`, the operation that created the new version, for example the upload, waits until there is room in the queue, but at most for `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL_TIMEOUT`. | +| `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL_TIMEOUT` | `5s` | How long `block` waits for room in the queue at most, for example `5s` or `1m`. After that, the cleanup of the file is skipped, as with `drop`. With `0`, `block` waits as long as it takes: no cleanup is ever skipped, but under heavy load, uploads can get slower and slower for as long as the load lasts. | + +The queue only fills up under heavy load, when files get new versions faster than the workers can clean them up. By default, OpenCloud then skips the cleanup of the files that don't fit into the queue, so that uploads never wait for the cleanup. + +A skipped cleanup isn't lost: the file is cleaned up with its next version, or by the [cleanup command](../../maintenance/maintenance-commands.md#clean-up-old-file-versions). Every skipped cleanup is logged as a warning, with the location of the file on disk: + +```text +revision cleanup: queue is full, skipping the cleanup of this node until its next revision +``` + +If you see this warning often, add workers with `OC_REVISIONS_CLEANUP_ASYNC_WORKERS`, or make the queue larger with `OC_REVISIONS_CLEANUP_ASYNC_QUEUE_SIZE`. + +If you prefer to wait for room in the queue rather than to skip cleanups, use `block`. Uploads can then get slower under heavy load, but at most by the timeout. In this example, an upload waits up to 30 seconds: + +```yaml +services: + opencloud: + environment: + OC_REVISIONS_CLEANUP_STRATEGY: '7d, auto, 90d' + OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL: 'block' + OC_REVISIONS_CLEANUP_ASYNC_QUEUE_FULL_TIMEOUT: '30s' +``` + +When the timeout is over, the cleanup of the file is skipped, as with `drop`, and logged as a warning as well: + +```text +revision cleanup: queue is still full after the timeout, skipping the cleanup of this node until its next revision +``` + +With a timeout of `0`, uploads wait as long as it takes, and no cleanup is ever skipped. Only use it if you can accept that uploads get slower and slower for as long as heavy load lasts. + +When OpenCloud stops, cleanups that are already running are finished, and the cleanups still waiting in the queue are abandoned. The queue isn't stored anywhere, so these cleanups are **not resumed** when OpenCloud starts again. Each of these files is cleaned up with its next version, or by the [cleanup command](../../maintenance/maintenance-commands.md#clean-up-old-file-versions). + +## Logs + +For every file it deleted versions of, the cleanup writes a line at log level `info`: `revision cleanup: purged old revisions`, with the number and the size of the deleted versions. Nothing is logged for files where nothing was deleted. + +At log level `debug`, every deleted version is logged as well, with the reason, for example `deleted by "30d": older than 2026-09-21T15:00:00Z`. See [Logging](../logging.md) for how to set the log level. + +## Things to Keep in Mind + +- **Versions count towards the quota of a space.** Deleting old versions frees storage and quota. +- **The cleanup can't be undone.** Deleted versions are gone. Try a new strategy with a dry run first. +- **Versions that are not deleted stay as they are.** The cleanup never changes a version, it only deletes versions. +- **A restore keeps one version beyond the strategy.** The version created by a restore is kept until the file is changed again, see [After a Restore](#after-a-restore). +- **Turning versioning off is different.** Turning versioning off with `OC_DISABLE_VERSIONING` only stops creating new versions, while the existing ones are kept. The cleanup deletes existing versions, gradually, as files get new versions, or all at once with the [cleanup command](../../maintenance/maintenance-commands.md#clean-up-old-file-versions). diff --git a/docs/admin/features.md b/docs/admin/features.md index 281c59f3..aaa9d41e 100644 --- a/docs/admin/features.md +++ b/docs/admin/features.md @@ -50,7 +50,7 @@ Mark frequently used files and folders for quick access. Favorites are stored se ### Versioning and Recovery -Every change is automatically saved as a version and can be restored at any time. +When versioning is enabled, which is the default, every change is automatically saved as a version and can be restored. Administrators can turn versioning off by setting `OC_DISABLE_VERSIONING` to `true`, see the [global environment variables](../dev/server/configuration/global-envvars.mdx). They also decide how long versions are kept: all of them, which is the default, or according to a cleanup strategy, see [Cleaning Up Old Versions](./configuration/storage/revisions-cleanup.md). OpenCloud supports storing any number of files in any format and size. diff --git a/docs/admin/maintenance/maintenance-commands.md b/docs/admin/maintenance/maintenance-commands.md index 0301ccd1..1689a345 100644 --- a/docs/admin/maintenance/maintenance-commands.md +++ b/docs/admin/maintenance/maintenance-commands.md @@ -135,6 +135,232 @@ When no arguments are specified, the default behavior is to start the storage sc The optional command-line option `--halt-on-error` (or `-E`) will make the command stop as soon as an error is encountered, whereas the default behaviour is to keep going by skipping paths that yield unrecoverable errors and proceeding to processing the subsequent ones. +### Clean up old file versions + +OpenCloud can delete old file versions automatically, according to a cleanup strategy, whenever a file gets a new version. That cleanup and its strategies are explained in [Cleaning Up Old Versions](../configuration/storage/revisions-cleanup.md). + +The automatic cleanup only looks at files that get a new version. Files that are never changed again keep all their versions. The `revisions cleanup` command cleans up the versions of whole spaces, of folders or of single files on demand. Use it: + +- once after turning the cleanup on, to clean up the versions that existed before, +- after changing the strategy, to apply the new one to all files, +- regularly, for example every night, so that files that are no longer changed are cleaned up too, +- to free storage, with a stricter strategy than the configured one. + +```bash +opencloud storage-users revisions cleanup [flags] +``` + +The command uses the configuration of the storage-users service, including the configured strategy, and can run while OpenCloud is running. Unlike the commands above, it doesn't need `--basepath`. With Docker Compose, run it inside the OpenCloud container: + +```bash +docker compose exec opencloud opencloud storage-users revisions cleanup --all --dry-run +``` + +`docker compose exec` runs the command with the environment variables of the container, so it gets the same settings as OpenCloud. Without containers, this isn't automatic. In particular, variables that you set with `Environment=` in a systemd unit only reach the OpenCloud service. A command that you run from a shell or from cron doesn't get them: it finds no strategy and stops with an error, and it doesn't see the settings of the file metadata cache either (see [Share the file metadata cache](#share-the-file-metadata-cache) below). + +On such a server, keep the settings in an environment file, for example `/etc/opencloud/opencloud.env`, and load it in the systemd unit with `EnvironmentFile=`: + +```ini +[Service] +EnvironmentFile=/etc/opencloud/opencloud.env +``` + +Then load the same file for the command, as the user OpenCloud runs as: + +```bash +sudo -u opencloud sh -c 'set -a; . /etc/opencloud/opencloud.env; exec opencloud storage-users revisions cleanup --all --dry-run' +``` + +`set -a` exports every variable the file sets, so that the command sees them. This works as long as the file only contains lines of the form `NAME=value`, with values that contain spaces in quotes, as in `OC_REVISIONS_CLEANUP_STRATEGY="7d, auto, 90d"`. + +:::caution + +Deleted versions can't be restored. Always run the command with `--dry-run` first, and check what it would delete. + +::: + +#### Try it out first + +`--dry-run` (`-n`) deletes nothing. It prints what would be deleted. Together with `--verbose` (`-v`), it lists every version, with what would happen to it and why: + +```bash +opencloud storage-users revisions cleanup --all --strategy "7d, auto, 90d" --dry-run --verbose +``` + +```plaintext +25b69780-5f39-43be-a7ac-a9b9e9fe4230 /Documents/report.odt (node 8a1c0e2f-6b4d-4f7a-9c3e-2d5b7f1a0c94): would delete 1 of 3 revisions (2.1 MiB) + protected 2026-10-21T14:50:00+02:00 2.3 MiB protected by "7d": newer than 2026-10-14T15:00:00+02:00 + kept 2026-09-30T09:00:00+02:00 2.2 MiB kept by "auto" (last 30 days): the oldest in its slot of 1 day starting 2026-09-30T00:00:00Z + deleted 2026-04-02T08:00:00+02:00 2.1 MiB deleted by "90d": older than 2026-07-23T15:00:00+02:00 +3 files in 1 space: would delete 1 revision (2.1 MiB) of 1 file, 0 skipped, 0 failed +``` + +Each version is either `deleted`, `kept` (by the strategy) or `protected` (always kept, for example because it is among the newest ones). The reason names the part of the strategy that decided, and the time it compared the version with. The time shown for each version is when it was created, which can differ from the date the version list in the web interface shows, see [How Old Is a Version?](../configuration/storage/revisions-cleanup.md#how-old-is-a-version). + +When the result looks right, run the same command without `--dry-run`. On a default installation, the command then stops with an error about the file metadata cache, see the next section. + +#### Share the file metadata cache + +OpenCloud keeps the metadata of files and their versions in a cache, the file metadata cache. By default, each process keeps a cache of its own, in its memory. The command runs in a process of its own, next to OpenCloud, so with the default cache, OpenCloud doesn't learn which versions the command deleted. + +OpenCloud lists the versions of a file from the storage, so a list of versions that is opened after the cleanup is right. But a list that was already open while the command ran, for example the Versions panel of a file in the web interface, still shows the deleted versions. Downloading or restoring one of them fails, as they are gone. Opening the list again shows the remaining versions. + +That's why the command refuses to delete versions when each process keeps a cache of its own, which is the case with the default, `memory`, and with `ocmem`. It stops right away, before it deletes anything, with this error: + +```plaintext +the file metadata cache of the storage-users service is kept in the memory of each process (store "memory"): the server doesn't learn about the revisions this command deletes, so a list of versions that is already open keeps showing them, and downloading or restoring one of them fails. Share the cache by setting STORAGE_USERS_FILEMETADATA_CACHE_STORE to 'nats-js-kv' (OC_CACHE_STORE would set the cache store of all services), or run the command anyway with --allow-local-cache +``` + +To share the cache, set `STORAGE_USERS_FILEMETADATA_CACHE_STORE` to `nats-js-kv`. OpenCloud then keeps the cache in NATS, and the command uses the same cache. With OpenCloud Compose, add the setting to the `opencloud` service, for example in the same file as the strategy, as described in [Turning the Cleanup On](../configuration/storage/revisions-cleanup.md#with-opencloud-compose): + +```yaml +services: + opencloud: + environment: + OC_REVISIONS_CLEANUP_STRATEGY: '7d, auto, 90d' + STORAGE_USERS_FILEMETADATA_CACHE_STORE: 'nats-js-kv' +``` + +Then restart OpenCloud: + +```bash +docker compose down +docker compose up -d +``` + +With other installations, set the variable for the OpenCloud process, in the same way as the strategy, and restart OpenCloud. The command must get the same setting, see above for a server that runs OpenCloud with systemd. + +`OC_CACHE_STORE` would also work, but it sets the cache of all services, not only the one of the storage. + +If you can accept that open lists of versions show deleted versions until they are opened again, for example because the command runs at night, when nobody is working, you can run it anyway with `--allow-local-cache`: + +```bash +opencloud storage-users revisions cleanup --all --allow-local-cache +``` + +A dry run needs neither: it deletes nothing, so there is nothing OpenCloud would have to learn about. It always keeps a cache of its own in its memory, whatever is configured, so that it doesn't fill the shared cache with the metadata of every file it looks at. + +#### Choose the files + +Exactly one of these options selects the files to clean up: + +- `--all` (`-a`): all files of all spaces. +- `--space `: all files of a space. The option can be given several times, for several spaces. +- `--space --path `: only a file, or a folder with everything in it, within one space. +- `--fs-path ` (`-p`): a file, or a folder with everything in it, by its path on disk. This only works with the PosixFS storage driver. + +Space IDs often contain a `$`, so put them in single quotes. The ID of a file or a folder, which looks like `$!`, isn't accepted as a space ID, so that a mix-up doesn't clean up the whole space. Give its space with `--space` and its path with `--path` instead: + +```bash +# all files of two spaces +opencloud storage-users revisions cleanup --space '' --space '' + +# only the Documents folder of a space +opencloud storage-users revisions cleanup --space '' --path /Documents + +# a folder of a project space, by its path on disk (PosixFS only) +opencloud storage-users revisions cleanup --fs-path /var/lib/opencloud/storage/users/projects//Documents +``` + +#### Choose the strategy + +Without `--strategy` (`-s`), the command uses the configured strategy, `OC_REVISIONS_CLEANUP_STRATEGY`. If none is configured, it stops with an error and doesn't delete anything. With `--strategy`, it uses the given one for this run only: + +```bash +# keep only the 10 newest versions of every file in a space +opencloud storage-users revisions cleanup --space '' --strategy 10 + +# delete all versions older than a year, but keep the 3 newest of every file +opencloud storage-users revisions cleanup --all --strategy "3, 52w" +``` + +The strategies are explained in [Choosing a Strategy](../configuration/storage/revisions-cleanup.md#choosing-a-strategy). + +#### Read the output + +By default, the command prints a line for every file whose versions it deleted, that failed, or that it skipped, followed by a summary: + +```plaintext +25b69780-5f39-43be-a7ac-a9b9e9fe4230 /Documents/report.odt: deleted 3 of 12 revisions (4.2 MiB) +25b69780-5f39-43be-a7ac-a9b9e9fe4230 /Photos/holiday.jpg: skipped: an upload to the file is still being processed +1520 files in 3 spaces: deleted 3 revisions (4.2 MiB) of 1 file, 1 skipped, 0 failed +``` + +Each line starts with the ID of the space and the path of the file within the space. A file is skipped while an upload to it is still being processed, for example by the antivirus scanner. A later run cleans it up. + +For scripts and monitoring, `--json` (`-j`) prints the same as JSON objects, one per line. The last line is the summary. Sizes are in bytes: + +```json +{"type":"file","space_id":"25b69780-5f39-43be-a7ac-a9b9e9fe4230","node_id":"8a1c0e2f-6b4d-4f7a-9c3e-2d5b7f1a0c94","path":"/Documents/report.odt","dry_run":false,"revisions":12,"deleted":3,"deleted_bytes":4404019,"failed":0} +{"type":"summary","dry_run":false,"spaces":3,"files":1520,"cleaned":1,"deleted":3,"deleted_bytes":4404019,"skipped":1,"failed":0} +``` + +#### Run it regularly + +To clean up files that are no longer changed, run the command regularly, for example every night with cron. On large installations, a single run can take a long time, so limit it with `--timeout` (`-t`), and add `--shuffle`: + +```bash +opencloud storage-users revisions cleanup --all --timeout 1h --shuffle +``` + +`--timeout` stops the cleanup gracefully: files that are being cleaned up are finished, and the rest is left for the next run. As every run starts from the beginning again, `--shuffle` makes it go through the spaces and folders in a random order. That way, each run starts in a different place, and over several nights, all files are reached. + +An example crontab entry for OpenCloud Compose, which runs the cleanup every night at 3:00 for at most one hour, and appends the output to a log file: + +```crontab +0 3 * * * cd /opt/opencloud-compose && docker compose exec -T opencloud opencloud storage-users revisions cleanup --all --timeout 1h --shuffle >> /var/log/opencloud-revisions-cleanup.log 2>&1 +``` + +Replace `/opt/opencloud-compose` with the directory of your OpenCloud Compose installation. `-T` is needed because cron doesn't provide a terminal. The entry needs the shared file metadata cache, see [Share the file metadata cache](#share-the-file-metadata-cache). Otherwise, every run ends with an error and exit code `1`, and deletes nothing. + +Without containers, load the environment file of OpenCloud in the crontab entry, as described at the beginning of this section, for example in the crontab of the user `opencloud`: + +```crontab +0 3 * * * sh -c 'set -a; . /etc/opencloud/opencloud.env; exec opencloud storage-users revisions cleanup --all --timeout 1h --shuffle' >> /var/log/opencloud/revisions-cleanup.log 2>&1 +``` + +The command ends with one of these exit codes, which you can use in scripts: + +- `0`: the cleanup is done. +- `1`: the cleanup, or the cleanup of at least one file, failed. The output says which and why. The command also ends with `1` when you interrupt it with Ctrl-C, or when it gets a `SIGTERM`. +- `2`: `--timeout` stopped the cleanup before all files were cleaned up, and nothing failed. + +#### While OpenCloud is running + +The command can run while OpenCloud is running. It works on the storage directly, in a process of its own, next to OpenCloud. Three things are good to know: + +- **Run it as the user OpenCloud runs as.** While the command cleans up a file, it locks it, and the first time a file is locked, a lock file is created. That lock file belongs to the user that runs the command, and only that user can open it. If the command runs as another user, for example as `root`, OpenCloud may no longer be able to lock these files, and changing, moving or deleting them fails. A dry run (`--dry-run`) changes nothing on disk and creates no lock files, so it does no harm, but the real cleanup must run as the right user. With Docker Compose, `docker compose exec` already runs the command as the user of the container. On a server where OpenCloud runs as the user `opencloud`, run the command with `sudo -u`, and load the environment file of OpenCloud, as described at the beginning of this section: + + ```bash + sudo -u opencloud sh -c 'set -a; . /etc/opencloud/opencloud.env; exec opencloud storage-users revisions cleanup --all --dry-run' + ``` + + When the storage belongs to another user than the one that runs the command, the command prints a warning before it starts, except in a dry run, which changes nothing on disk. For example: + + ```plaintext + 2026-11-17T10:00:00+01:00 WRN the storage /var/lib/opencloud/storage/users belongs to the user opencloud, but the command runs as root: the lock files it creates belong to root, and OpenCloud may not be able to lock these files anymore; run the command as opencloud, e.g. with 'sudo -u opencloud' + ``` + + If you see it, stop the command with Ctrl-C and run it again as the right user. + +- **The file metadata cache must be shared**, so that OpenCloud learns which versions the command deleted. See [Share the file metadata cache](#share-the-file-metadata-cache). +- **PosixFS needs NATS.** With the PosixFS driver, the command connects to NATS, like OpenCloud itself. When NATS runs within OpenCloud, which is the default where a single `opencloud server` process runs all services, the command only works while OpenCloud is running. When OpenCloud is stopped, the command gives up after 10 seconds, after a few log lines about the failed attempts to connect: + + ```plaintext + could not create the event stream for the posix driver: could not connect to NATS at 127.0.0.1:9233 within 10s: the command needs NATS, i.e. a running OpenCloud server or a separate NATS + ``` + + If NATS runs as a separate service in your installation, the command also works while OpenCloud is stopped, as long as NATS is running. + +#### Other options + +- `--async`, `--workers` and `--queue-size` override the settings of the background workers for this run, see [Background Workers](../configuration/storage/revisions-cleanup.md#background-workers). +- `--queue-full` and `--queue-full-timeout` say what happens when the queue of the background workers is full. Unlike the other settings, they don't default to the configured ones: the command finds files much faster than the workers can clean them up, so the queue is full most of the time, and skipping the files that don't fit would skip most of them. That's why the command waits for room in the queue by default (`--queue-full block`), as long as it takes (`--queue-full-timeout 0`). You rarely need to change them. With `--queue-full drop`, or with a timeout, for example `--queue-full-timeout 10s`, the skipped files are listed in the output, and are cleaned up with their next version or the next run. +- Ctrl-C stops the cleanup gracefully, like `--timeout`, but the command then ends with the exit code `1`, not `2`. Press Ctrl-C a second time to end the command right away, for example when it waits for a file that another process has locked, or for NATS. Use this only when the first Ctrl-C doesn't help: a version is deleted in two steps, first its entry, then its content. If the command ends between them, the content stays on disk without anything pointing to it, and takes up storage. +- A dry run changes nothing on disk. It doesn't keep the files locked while it decides, so its result is a snapshot: for a file that changes while the command runs, for example because a user uploads a new version of it, the real run may decide differently. + +Not to be confused with `opencloud revisions purge`, an older command that deletes all versions of a DecomposedFS storage, or of a single file or space, directly on disk and without a strategy. + ## Uploads The `opencloud storage-users uploads sessions` command lists and manages upload sessions. Flags can be combined to filter sessions by state. diff --git a/docs/admin/resources/faq.md b/docs/admin/resources/faq.md index 35ba2b57..7456eeb5 100644 --- a/docs/admin/resources/faq.md +++ b/docs/admin/resources/faq.md @@ -121,6 +121,8 @@ We provide apps for Windows, MacOS, Linux, iOS and Android. Please refer to the Yes. In case you made a mistake, you can always jump back to older versions of a file. We got your back! +By default, OpenCloud keeps all versions. Administrators can have old versions deleted automatically, according to a cleanup strategy, see [Cleaning Up Old Versions](../configuration/storage/revisions-cleanup.md). + ## Deployment and Installation