Add attributes to the EOPF-like datatree

README.md

@@ -1,7 +1,7 @@
 # xarray-sentinel
 
 Easily access and explore the SAR data products of the
-[Copernicus Sentinel-1 satellite mission](https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-1)
+[Copernicus Sentinel-1 satellite mission](https://sentiwiki.copernicus.eu/web/sentinel-1)
 in Python.
 
 This Open Source project is sponsored by B-Open - <https://www.bopen.eu>.
@@ -543,12 +543,12 @@ Attributes: ...
 
 This is the list of the reference documents:
 
-- Sentinel-1 document library:
-  - [user guides](https://sentinels.copernicus.eu/web/sentinel/user-guides/sentinel-1-sar)
-  - [technical guides](https://sentinels.copernicus.eu/web/sentinel/technical-guides/sentinel-1-sar)
-- [Sentinel-1 Product Specification v3.9 07 May 2021 S1-RS-MDA-52-7441-3-9 documenting IPF 3.40](https://sentinel.esa.int/documents/247904/1877131/S1-RS-MDA-52-7441-3-9-2_Sentinel-1ProductSpecification.pdf)
-- [Sentinel-1 Product Specification v3.7 27 February 2020 S1-RS-MDA-52-7441 documenting IPF 3.30](https://sentinel.esa.int/documents/247904/1877131/Sentinel-1-Product-Specification)
-- [Radiometric Calibration of S-1 Level-1 Products Generated by the S-1 IPF v1.0 21/05/2015 ESA-EOPG-CSCOP-TN-0002](https://sentinel.esa.int/documents/247904/685163/S1-Radiometric-Calibration-V1.0.pdf)
+- Sentinel-1 on SentiWiki:
+  - [user guides](https://sentiwiki.copernicus.eu/web/sentinel-1)
+  - [documents](https://sentiwiki.copernicus.eu/web/s1-documents)
+- [Sentinel-1 Product Specification v3.9 07 May 2021 S1-RS-MDA-52-7441-3-9 documenting IPF 3.40](https://sentinels.copernicus.eu/documents/247904/1877131/S1-RS-MDA-52-7441-3-9-2_Sentinel-1ProductSpecification.pdf)
+- [Sentinel-1 Product Specification v3.7 27 February 2020 S1-RS-MDA-52-7441 documenting IPF 3.30](https://sentinels.copernicus.eu/documents/247904/1877131/Sentinel-1-Product-Specification)
+- [Radiometric Calibration of S-1 Level-1 Products Generated by the S-1 IPF v1.0 21/05/2015 ESA-EOPG-CSCOP-TN-0002](https://sentinels.copernicus.eu/documents/247904/685163/S1-Radiometric-Calibration-V1.0.pdf)
 
 ## Design decisions
 

docs/DATATREE.md

@@ -104,12 +104,12 @@ All fields come from the SAFE name, except `burst_id` from `swathTiming` and
 ### Attributes
 
 - root
-  - `stac_discovery`: STAC item from `manifest.safe`, one asset per node.
-  - `other_metadata`: `{"eopf_category": "eocontainer", "title": "S01SIWSLC"}`.
+  - `other_metadata`: `{"eopf_category": "eocontainer", "title": "S01SIWSLC"}`. The CPM
+    also has a root `stac_discovery`, we don't, see step 9.
 - node
   - `stac_discovery`: manifest and annotation properties, `sar:relative_burst`, burst
-    footprint as `geometry`, `detectionPerformed`, `thermalNoiseCorrectionPerformed`.
-  - `processing_history`: from the manifest `safe:processing`.
+    footprint as `geometry`.
+  - `processing_history`: from the manifest `safe:processing`, not set for now.
   - `other_metadata`: keyed by polarisation, each with `eopf_category: "eoproduct"`,
     `title`, `image_number`, `quality_information`, `general_annotation`,
     `image_annotation` and `burst_timing`. `burst_timing` holds the node's burst entry of
@@ -182,8 +182,14 @@ What we do in each case follows the decisions above.
    - `sar:relative_burst` needs the SAR extension v1.3.1, the CPM declares v1.3.0.
    - the eopf extension URL does not exist, we use
      `https://cs-si.github.io/eopf-stac-extension/v1.2.0/schema.json`.
-   - `detectionPerformed` and `thermalNoiseCorrectionPerformed` are strings, we use
-     bools, `links` points to a Zarr v2 `.zattrs.json`, we use none.
+   - `detectionPerformed` and `thermalNoiseCorrectionPerformed` are non-STAC top-level
+     keys, already in `other_metadata`, we drop them. `links` points to a Zarr v2
+     `.zattrs.json`, we use none.
+   - `eopf:instrument_mode` duplicates `sar:instrument_mode`, we drop it.
+1. `processing_history` is a single `"Unknown"` entry with no time, inputs or outputs:
+   `ToProcessingHistory` tests lxml attributes with `"name" in node`, which checks
+   the children. We walk the manifest as CPM 2.7 did, without its own conversion entry,
+   but don't set it yet.
 
 ## Open points
 
@@ -207,7 +213,8 @@ What we do in each case follows the decisions above.
 ## Plan
 
 One PR with tests per step. Step 1 comes first, steps 2 to 9 are independent and each
-adds converters to the step 1 registry, steps 10 and 11 come last.
+adds converters to the step 1 registry, steps 10 and 11 come last, step 12 can be
+done at any time.
 
 1. Done: skeleton and test harness.
    - `eopf_product.open_datatree` reads each XML once and opens each measurement once.
@@ -262,22 +269,42 @@ adds converters to the step 1 registry, steps 10 and 11 come last.
    xarray-sentinel finds one annotation per WV swath instead of one per vignette.
 1. Attributes, in four parts. `tests/data/eopf/S01SIWSLC_target_attrs.json` holds the
    key and type tree of the node attributes, the CPM 3.0 burst node with our deliberate
-   differences.
+   differences, with `other_metadata` of one polarisation and its annotation sections as
+   `dict` or `list` only, as their content depends on the IPF version.
    1. Done: node `stac_discovery`, `eopf_metadata.build_stac_discovery` following the
       CPM mapping keys `stac_discovery`, `stac_discovery_annotation` and
       `stac_discovery_annotation_manifest`, quirk 8. `geometry` is the node footprint
       interpolated from the GCPs, `sar:relative_burst` is set when the product has
       burst ids. The items validate against the declared STAC extensions.
-   1. root `stac_discovery` and `other_metadata`.
-   1. node `other_metadata` extending `eopf_metadata.build_other_metadata`:
+   1. Done: root `other_metadata`, `{eopf_category: "eocontainer", title}` with
+      `eopf_category` used by the CPM tools to recognise the container. No root
+      `stac_discovery`: the CPM reads it from the manifest and merges its properties
+      into every node, then adds assets from the variable short names, which collide
+      across nodes, and its own software versions.
+   1. Done: node `other_metadata`, `eopf_metadata.build_other_metadata` with
       `eopf_category`, `title`, `image_number`, `burst_timing` with the node's burst
       merged in, drop empty `swath_merging`, numeric lists instead of `$` strings,
-      `t_0` as `t0`. We keep the CPM `DEPRECATED_ATTRS`.
-   1. `processing_history`, by level from the manifest as in CPM 2.7, the CPM 3.0 one
-      is a single `"Unknown"` entry.
+      `t_0` as `t0`. `title` is the product type from the manifest, as the annotation
+      mode of SM is the beam. We keep the CPM `DEPRECATED_ATTRS`.
+   1. Done, not connected: node `processing_history`,
+      `eopf_metadata.build_processing_history`, quirk 9, entries keyed by output level
+      from the resource roles and sorted by time, AUX resources as `adfs` and not
+      walked. `open_datatree` does not set it for now.
 1. API and docs: `Sentinel1Backend.open_datatree` and `open_groups_as_dict`, README,
    this file.
 1. Validation: convert one or two full SAFE products with CPM 3.0 in a throwaway
    environment (the test SAFEs lack the `support/*.xsd` the CPM needs), compare
    structure exactly and values within tolerance, list the deviations due to the
    quirks. Optionally open our Zarr output with `xarray-eopf`.
+1. Parse once: decode each XML once and pass the result down.
+   - Manifest: `build_stac_discovery` and `build_processing_history` take the parsed
+     `ElementTree` root instead of the path, now `build_stac_discovery` parses the
+     manifest again for every node.
+   - Annotation, calibration and noise: `open_pol_dataset` and the `sentinel1`
+     openers of calibration, noise, replicas and `azimuth_fm_rate` still decode the
+     whole XML at every `parse_tag` or `parse_tag_as_list` call, the wrappers of the
+     `eopf_sentinel1` bases call `parse_xml` once per call. Switch them to the
+     `parse_xml` dict and `get_tag`.
+   - Then `open_datatree` keeps the parsed dicts instead of `BytesIO`, the manifest
+     keeps its `BytesIO` for `parse_manifest_sentinel1` until the manifest XSD
+     decode in [Open points](#open-points) is settled.

tests/data/eopf/S01SIWSLC_target_attrs.json

@@ -1,8 +1,36 @@
 {
+  "other_metadata": {
+    "burst_timing": {
+      "azimuth_anx_time": "float",
+      "azimuth_time": "str",
+      "burst_id": "int",
+      "first_valid_sample": "list",
+      "last_valid_sample": "list",
+      "lines_per_burst": "int",
+      "samples_per_burst": "int",
+      "sensing_time": "str"
+    },
+    "eopf_category": "str",
+    "general_annotation": {
+      "azimuth_fm_rate_list": "list",
+      "downlink_information_list": "list",
+      "noise_list": "list",
+      "product_information": "dict",
+      "raw_data_analysis_list": "list",
+      "replica_information_list": "list",
+      "terrain_height_list": "list"
+    },
+    "image_annotation": {
+      "image_information": "dict",
+      "processing_information": "dict"
+    },
+    "image_number": "str",
+    "quality_information": "dict",
+    "title": "str"
+  },
   "stac_discovery": {
     "assets": {},
     "bbox": "list",
-    "detectionPerformed": "bool",
     "geometry": {
       "coordinates": "list",
       "type": "str"
@@ -15,7 +43,6 @@
       "datetime": "str",
       "end_datetime": "str",
       "eopf:datatake_id": "str",
-      "eopf:instrument_mode": "str",
       "instruments": "list",
       "mission": "str",
       "platform": "str",
@@ -49,7 +76,6 @@
     },
     "stac_extensions": "list",
     "stac_version": "str",
-    "thermalNoiseCorrectionPerformed": "bool",
     "type": "str"
   }
 }

tests/test_15_eopf_metadata.py

@@ -37,16 +37,24 @@
 
 
 def test_to_snake_recursive() -> None:
-    metadata = {"qualityInformation": {"qualityDataList": [{"qualityData": 2}]}}
-    expected = {"quality_information": {"quality_data_list": [{"quality_data": 2}]}}
+    metadata = {
+        "qualityInformation": {"qualityDataList": [{"qualityData": 2}]},
+        "t0": 1,
+    }
+    expected = {
+        "quality_information": {"quality_data_list": [{"quality_data": 2}]},
+        "t0": 1,
+    }
 
     res = eopf_metadata.to_snake_recursive(metadata)
 
     assert res == expected
 
 
 def test_build_other_metadata() -> None:
-    res = eopf_metadata.build_other_metadata(esa_safe.parse_xml(SLC_S3_VH_annotation))
+    res = eopf_metadata.build_other_metadata(
+        esa_safe.parse_xml(SLC_S3_VH_annotation), "S01SSMSLC"
+    )
     expected_quality_information = {
         "product_quality_index": 0.0,
         "quality_data_list": [
@@ -109,15 +117,53 @@ def test_build_other_metadata() -> None:
     }
 
     assert set(res) == {
+        "eopf_category",
+        "title",
+        "image_number",
         "quality_information",
         "general_annotation",
         "image_annotation",
-        "swath_timing",
-        "swath_merging",
+        "burst_timing",
     }
 
     assert res["quality_information"] == expected_quality_information
 
+    general_annotation = res["general_annotation"]
+    replica = general_annotation["replica_information_list"][0]["reference_replica"]
+    fm_rate = general_annotation["azimuth_fm_rate_list"][0]
+
+    assert all(isinstance(c, float) for c in replica["amplitude_coefficients"])
+    assert all(isinstance(c, float) for c in fm_rate["azimuth_fm_rate_polynomial"])
+    assert "t0" in fm_rate
+    assert "burst_list" not in res["burst_timing"]
+
+    res = eopf_metadata.build_other_metadata(
+        esa_safe.parse_xml(SLC_IW1_VV_annotation), "S01SIWSLC", burst_index=1
+    )
+    burst_timing = res["burst_timing"]
+
+    assert all(isinstance(s, float) for s in burst_timing["first_valid_sample"])
+    assert "byte_offset" not in burst_timing
+    assert burst_timing["azimuth_time"] == "2021-04-01T05:26:26.966491"
+
+
+def test_build_processing_history() -> None:
+    res = eopf_metadata.build_processing_history(SLC_IW / "manifest.safe", SLC_IW.name)
+
+    assert set(res) == {
+        "Level-0 Product",
+        "Level-1 Intermediate SLC Product",
+        "Level-1 Product",
+    }
+    (product,) = res["Level-1 Product"]
+    assert product["outputs"] == [SLC_IW.name]
+    (intermediate,) = res["Level-1 Intermediate SLC Product"]
+    assert product["inputs"] == intermediate["outputs"]
+    assert len(intermediate["adfs"]) == 3
+    assert res["Level-0 Product"][-1]["facility"] == "PAC1"
+    times = [entry["time"] for entry in res["Level-0 Product"]]
+    assert times == sorted(times)
+
 
 @pytest.mark.parametrize(
     "safe,annotation",

tests/test_30_eopf_product.py

@@ -45,10 +45,13 @@ def datatree_structure(dt: xr.DataTree) -> dict[str, dict[str, list[Any]]]:
     }
 
 
-def attrs_structure(value: Any) -> Any:
-    if isinstance(value, dict):
-        return {key: attrs_structure(item) for key, item in value.items()}
-    return type(value).__name__
+def attrs_structure(value: Any, target: Any) -> Any:
+    if not (isinstance(value, dict) and isinstance(target, dict)):
+        return type(value).__name__
+    structure = {}
+    for key, item in value.items():
+        structure[key] = attrs_structure(item, target.get(key))
+    return structure
 
 
 @pytest.mark.parametrize(
@@ -149,8 +152,16 @@ def test_open_datatree_attrs() -> None:
     node = res.children["S01SIWSLC_20220414T102209_0027_A244_E677_051AA4_IW1_365915"]
     stac_discovery = node.attrs["stac_discovery"]
 
-    assert {
-        key: attrs_structure(node.attrs[key]) for key in EXPECTED_SLC_IW_ATTRS
-    } == EXPECTED_SLC_IW_ATTRS
+    assert res.attrs == {
+        "other_metadata": {"eopf_category": "eocontainer", "title": "S01SIWSLC"}
+    }
+    assert list(node.attrs["other_metadata"]) == ["HH"]
+
+    # other_metadata of one polarisation
+    node_attrs = dict(node.attrs) | {
+        "other_metadata": node.attrs["other_metadata"]["HH"]
+    }
+
+    assert attrs_structure(node_attrs, EXPECTED_SLC_IW_ATTRS) == EXPECTED_SLC_IW_ATTRS
     assert stac_discovery["id"] == node.name
     assert stac_discovery["properties"]["sar:relative_burst"] == 365915

xarray_sentinel/conventions.py

@@ -1,7 +1,7 @@
 """CF representation of metadata according to Sentinel-1 Product Specification.
 
 See: S1-RS-MDA-52-7441, DI-MPC-PB, MPC-0240, 3/7, 27/02/2020
-https://sentinel.esa.int/documents/247904/1877131/Sentinel-1-Product-Specification
+https://sentinels.copernicus.eu/documents/247904/1877131/Sentinel-1-Product-Specification
 """
 
 import xarray as xr