Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
fa4ec16
remove empty line
Hneuschmidt Jul 27, 2026
3da53a0
Update pixi.toml with features
Hneuschmidt Jul 27, 2026
0f25692
Make pixi.toml tables inline so they can work with external tools
Hneuschmidt Jul 27, 2026
70580db
Add pixi version to versions that must be checked
Hneuschmidt Jul 27, 2026
8073d5e
Add tomlq dependency for tools
Hneuschmidt Jul 27, 2026
4f4323a
Check if reference is empty
Hneuschmidt Jul 27, 2026
f1ea030
Install yq (tomlq) for check-version-matches update
Hneuschmidt Jul 27, 2026
2e7335c
Fix trailing whitespace across project
Hneuschmidt Jul 27, 2026
f9157b0
Comment out ruff from pre-commit for now
Hneuschmidt Jul 27, 2026
077e865
Add ruff to dev tools
Hneuschmidt Jul 27, 2026
851f4ae
Add debug-statements pre-commit hook
Hneuschmidt Jul 27, 2026
b46a23f
Remove bespoke environments per example
Hneuschmidt Jul 29, 2026
55ddf29
Remove unused templates in etc
Hneuschmidt Jul 29, 2026
09bafc6
Update github action names
Hneuschmidt Jul 30, 2026
76bf0db
Remove notebooks
Hneuschmidt Jul 30, 2026
4d659d1
Update integration tests and reorganize test parameters
Hneuschmidt Aug 11, 2026
ebcef82
Separate current tests and release tests. Use fixed workspace URL
Hneuschmidt Aug 11, 2026
715c7fe
Fix integration test catalog retreival for released FORCE version
Hneuschmidt Aug 11, 2026
4c187fa
Remove query_stac example
Hneuschmidt Aug 18, 2026
36352a6
Use topographic map for example
Hneuschmidt Aug 18, 2026
88eef02
Remove old stac generation implementation
Hneuschmidt Aug 18, 2026
834aa9b
Update examples readmes
Hneuschmidt Aug 18, 2026
a01c32c
Add main README.md
Hneuschmidt Aug 18, 2026
87fc2c2
Remove inspection section based on STAC URL from logs
Hneuschmidt Aug 18, 2026
e6b048b
remove commented out spatial extent
Hneuschmidt Aug 18, 2026
76bea8b
Use OpenTopoMap in example (replacing Mapnik)
Hneuschmidt Aug 18, 2026
e1b58ba
Update text
Hneuschmidt Aug 18, 2026
f64b615
Typo
Hneuschmidt Aug 18, 2026
07b6625
Update TSA example
Hneuschmidt Aug 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/automatically-publish-docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# if: github.event.pull_request.merged == true || github.event_name == 'workflow_dispatch'


name: Publish Docker image (automatic tag)
name: Publish Docker image
on:
workflow_dispatch:
pull_request:
Expand Down
42 changes: 0 additions & 42 deletions .github/workflows/publish-docker-image.yml

This file was deleted.

6 changes: 3 additions & 3 deletions .github/workflows/publish-documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,11 +69,11 @@ jobs:
## Automated documentation update
Source repository:
`${{ github.repository }}`

Release:
`${{ github.event.release.tag_name || 'manual' }}`

Commit:
`${{ github.sha }}`

This PR updates the documentation source for FORCE
2 changes: 1 addition & 1 deletion .github/workflows/python-tests.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Run Pytest
name: Python tests

on:
push:
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/test-and-release.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Test Build Release
name: Check release and release

on:
push:
Expand All @@ -12,7 +12,7 @@ on:
type: boolean
default: false
version:
description: Version name, must match the entry in pyproject.toml and docker-requirement.yaml
description: Version name, must match the entry in pixi.toml, pyproject.toml and docker-requirement.yaml
required: true
type: string

Expand Down Expand Up @@ -44,6 +44,10 @@ jobs:
- name: Info
run: |
echo ${{ env.RELEASE_VERSION }}
- name: install yq
run: |
sudo apt update
sudo apt istall yq
- name: Check versions
run: |
# Fails if versions in pyproject.toml, cwl/docker-requirement.yaml and the reference given
Expand Down
40 changes: 0 additions & 40 deletions .github/workflows/test-force-level2.yml

This file was deleted.

15 changes: 15 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.3.0
hooks:
- id: check-yaml
- id: trailing-whitespace
- id: check-added-large-files
- id: debug-statements


#- repo: https://github.com/astral-sh/ruff-pre-commit
# # Ruff version.
# rev: v0.16.0
# hooks:
# - id: ruff-format
105 changes: 103 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,103 @@
# apex-force-openeo
Extensions to FORCE toolbox for integration as processes in OpenEO
# FORCE as an Earth Observation Application Package, integrated into CDSE openEO

Run the [FORCE processing engine](https://force-eo.readthedocs.io/en/latest/) on the
[Copernicus Dataspace Ecosystem](https://dataspace.copernicus.eu/)'s (CDSE) openEO service, or deploy
FORCE as an Application package according to the
[OGC Best Practice for Earth Observation Application Package](http://www.opengis.net/doc/BP/eoap/1.0) (EOAP) on a
compatible service.

FORCE (Framework for Operational Radiometric Correction for Environmental Monitoring) is a processing framework
for Sentinel-2 and Landsat imagery. It supports the generation of Analysis Ready Data Cubes (level 2 processing) and
many higher level processing operations such as time series analysis.
[FORCE is developed](https://github.com/davidfrantz/force) by Prof. David Frantz
(*Geoinformatics - Spatial Data Science*, Trier University) and the FORCE Open Source community.
Please make sure to [acknowledge their work accordingly](https://force-eo.readthedocs.io/en/latest/policy/citation.html)
when processing with FORCE.

This repository wraps FORCE as an EOAP and adds tools necessary for integrating it into cloud services.
In particular, FORCE is made available as an openEO process in the CDSE openEO backend.

The cloud integration of FORCE has been performed in the context of the [European Space Agency](https://www.esa.int/)'s
(ESA) [Application Propagation Environments](https://apex.esa.int/) (APEx) initiative.

For more information, have a look at [the documentation](https://esa-apex.github.io/apex_toolbox_documentation/docs/force/)
on the [APEx toolbox documentation portal](https://esa-apex.github.io/apex_toolbox_documentation).

## Features

- FORCE level 2 processing through openEO on CDSE
- FORCE Time Series Analysis (TSA) through openEO on CDSE
- Automatic STAC generation for FORCE datacubes
- Discover parameters of cloudified FORCE modules using the openEO client
- Run level 2 processing once and re-use results without downloading multiple times
- Area of interest selection (level 2) with GEOJSON: parameter aoi of force_level2.
- Automatic DEM download based on selected area
- Integration with openEO workspaces to share results

STAC generation can be applied to any FORCE (level2 or TSA) data cubes, not just those produced by this EOAP.

## Getting Started

- Read the FORCE [user guide](https://esa-apex.github.io/apex_toolbox_documentation/docs/force/guide/intro.html)
in the offical APEx toolbox documentation portal

### Examples

For more detailed examples and more context, see
[the user guide](https://esa-apex.github.io/apex_toolbox_documentation/docs/force/guide/intro.html) and the
[example notebooks](examples).

You will need a (free) [CDSE account](https://dataspace.copernicus.eu/) to run this example, as well as the
[openEO Python client installed](https://open-eo.github.io/openeo-python-client/installation.html).

```Python
import openeo
from openeo.rest.stac_resource import StacResource
from openeo.internal.graph_building import PGNode


connection = openeo.connect("openeo.dataspace.copernicus.eu").authenticate_oidc()


# Select a STAC item to process (collections, catalogs and item collections are also supported!)
stac_item_url = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c/items/S2A_MSIL1C_20260419T100711_N0512_R022_T32TPQ_20260419T152521"

# Create the process graph
processing_name = "FORCE_level2"
force_l2_stac_resource = StacResource(
graph=PGNode(
process_id="force_level2",
arguments={
"stac_url": stac_item_url,
"name": processing_name,
"do_brdf": True,
# other FORCE level 2 parameters
},
),
connection=connection,
)

# Run processing
l2_job = force_l2_stac_resource.create_job(title=processing_name)
l2_job.start_and_wait()

# Download results (alternatively: Continue processing without download with Time Series Analysis. Check out the guide!)

l2_results = l2_job.get_results()
l2_results.download_files("force-level2-results")
```

## Structure

![FORCE onion](docs/force/img/onion-concrete.svg)

## Related

- [FORCE](https://force-eo.readthedocs.io/en/latest/) Framework for Operational Radiometric Correction for Environmental
monitoring, the processing engine wrapped by the EOAP implemented in this repository
- [Application Propagation Environments (APEx)](https://apex.esa.int/): ESA initiative to provide easy access to
earth observation application outcomes. FORCE was provided as a cloud-ready toolbox through the
[APEx *Toolbox Cloudification* activity](https://esa-apex.github.io/apex_documentation/propagation/toolboxcloud.html)
- [EOAP best practice](http://www.opengis.net/doc/BP/eoap/1.0) The *OGC Best Practice for Earth Observation Application Package*
a specification how to provide earth observation processors in a standardized format. This repository provides FORCE
in a accordance with this best practice.
2 changes: 1 addition & 1 deletion cwl/force-l2-workflow.cwl
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ steps:
if (inputs.cfg === null || inputs.cfg === undefined) {
return { cfg_json: null };
} else {
return { cfg_json: JSON.stringify(inputs.cfg) };
return { cfg_json: JSON.stringify(inputs.cfg) };
}
}
in:
Expand Down
5 changes: 2 additions & 3 deletions docs/force/feature_overview.qmd
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
---
title: Features and Limitations

---


The FORCE integration into to CDSE is in a proof-of-concept stage. It is possible to use the FORCE level 2 and Time Series Analysis (TSA) modules on a small scale. Most parameters supported by FORCE are available through the openEO interface.
The FORCE integration into to CDSE is in a proof-of-concept stage. It is possible to use the FORCE level 2 and Time Series Analysis (TSA) modules on a small scale. Most parameters supported by FORCE are available through the openEO interface.

Unsupported parameters are usually either managed by the implementation (parallelization options, input/output directores) or reuqire supplementary files which are not avaiable on the backend.
The `aoi` (openEO interface) / `FILE_AOI` (FORCE parameter file) parameter is good example of a parameter that is exposed in a different way in openEO to avoid the file interface. Instead of a shape file, a GEOJSON string is passed.
Expand Down Expand Up @@ -37,6 +36,6 @@ If you encounter limitations not mentioned here, please raise an issue in the [G
### Parallelizing large Time Series Analysis jobs

Server-side parallelization of the TSA module is not currently enabled. It is possible to parallelize TSA by creating one openEO job per tile by setting the `x_tile_range` and `y_tile_range` arguments to a single tile per job.
Furthermore, consider computing spatio-temporal metrics (STM) in separate jobs to reduce memory usage.
Furthermore, consider computing spatio-temporal metrics (STM) in separate jobs to reduce memory usage.

Finally, you may reduce the `chunk_size` which will reduce memory usage at the cost of a longer processing time.
11 changes: 5 additions & 6 deletions docs/force/guide/level2.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ order: 4

L1C_COLLECTION_URL = "https://stac.dataspace.copernicus.eu/v1/collections/sentinel-2-l1c"
STAC_ROOT_URL = "https://stac.dataspace.copernicus.eu/v1"
#w,s,e,n = 10.386, 44.437, 11.423, 44.973
w, s, e, n = 11.0, 44.5, 11.1, 44.6
spatial_extent = { "west": w, "south": s, "east": e, "north": n}
# ATTENTION: Inclusion of the second date in the search depends on the query method.
Expand Down Expand Up @@ -70,7 +69,7 @@ The integrated FORCE uses (lowercase) [`snake_case`](https://en.wikipedia.org/wi

The parameters are documented in the process description. We can inspect it using the Python client's [`describe_processes()`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.describe_process).

<!--
<!--

FIXME:
It is necessary to connect to the staging backend, until the custom processes are publicly
Expand Down Expand Up @@ -123,7 +122,7 @@ When running FORCE level2 on its own, we do not need to use a workspace. Simply

We can instruct openEO to store our results in a workspace using the [`export_workspace`](https://openeo.org/documentation/1.0/processes.html#export_workspace) openEO process.

When using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system.
When using `export_workspace`, we need to pass a `merge` string to identify your result. The merge will be a key prefix on object storage. The `merge` is something similar to a path in a file system.
It is very important to set a good `merge` parameter and remember the result, because results can only be accessed when the `merge` is known. Furthermore, all users of the workspace (in the case of the `apex-force-results-workspace`, anyone) may write data, so make sure to use a unique `merge`, to reduce the likelihood of accidentally modifying others' data and have others overwrite your data by accident.


Expand Down Expand Up @@ -367,7 +366,7 @@ While the cube will stay in place for multiple days, eventually the storage will
<!--

We use a succeeded job instead of the one created above to avoid having to run and
wait for an entire job every time the documentation is re-generated
wait for an entire job every time the documentation is re-generated
(happens only when the freeze cache is invalidated:
https://quarto.org/docs/projects/code-execution.html#freeze).

Expand Down Expand Up @@ -395,7 +394,7 @@ l2_job = connection.job("j-260702093443422db1ddead799544cff")
This step is optional
:::

The basic URL of the FORCE workspace `apex-force-results-workspace` is
The basic URL of the FORCE workspace `apex-force-results-workspace` is
<https://s3.waw4-1.cloudferro.com/apex-force-results-waw4-1-exotc5yuexi2c5tvwqhoivj62fz8v0uupy0me>. Together with the `merge` path defined for our processing, we can determine the path to the `catalog.json` for our FORCE level results:

```{.python}
Expand Down Expand Up @@ -424,7 +423,7 @@ This step is optional if you plan to continue processing with FORCE on CDSE
:::

Assets can be downloaded using the standard openEO mechanism to access result assets.
The STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files).
The STAC metadata specifies the directory structure of the resulting cube. This structure is respected by openEO's [`download_files`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_files).

To inspect a particular asset, we may use [`download_file`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.JobResults.download_file) to download singular files instead.

Expand Down
4 changes: 2 additions & 2 deletions docs/force/guide/parametrization.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ order: 6

The [FORCE openEO processes](processes.qmd) (`force_level2` and `force_tsa`) accept most of the parameters that can be passed using FORCE's native parametrization files ([level 2](https://force-eo.readthedocs.io/en/latest/components/lower-level/level2/param.html), [TSA](https://force-eo.readthedocs.io/en/latest/components/higher-level/tsa/param.html)). The exceptions to this rule are explained below.

There are 4 types of parameters that do not correspond exactly to the FORCE parameter files
There are 4 types of parameters that do not correspond exactly to the FORCE parameter files

1. Special parameters that require translation (e.g., the area of interest)
1. Additional parameters of the openEO integration (e.g., the processing name)
Expand Down Expand Up @@ -77,7 +77,7 @@ Input/Output directories (managed by the integration)

Parallel processing parameters (managed by the integration)

- `NPROC`
- `NPROC`
- `NTHREAD`
- `PARALLEL_READS`
- `DELAY`
Expand Down
4 changes: 2 additions & 2 deletions docs/force/guide/processes.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import openeo
connection = openeo.connect("https://openeo.dataspace.copernicus.eu")
connection.authenticate_oidc()
```
<!--
<!--

FIXME:
It is necessary to connect to the staging backend, until the custom processes are publicly
Expand Down Expand Up @@ -90,7 +90,7 @@ Please refer to the [Python client documentation](https://open-eo.github.io/open

Name | Qualified name (Python client) | Purpose
-----|----------------|--------
`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend
`create_job` | [`openeo.rest.connection.Connection.create_job`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.connection.Connection.create_job) | Submit a Process Graph to the backend
`start_and_wait` | [`openeo.rest.job.BatchJob.start_and_wait`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start_and_wait) | Start a submitted job and block until completion
`start` | [`openeo.rest.job.BatchJob.start`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.start) | Same as `start_and_wait` but do not block execution
`get_results` | [`openeo.rest.job.BatchJob.get_results`](https://open-eo.github.io/openeo-python-client/api.html#openeo.rest.job.BatchJob.get_results) | Get a `JobResults` object from a completed Job (contains references to output files)
Expand Down
Loading