diff --git a/EUMETSAT_data_access.ipynb b/EUMETSAT_data_access.ipynb new file mode 100644 index 0000000..243dfba --- /dev/null +++ b/EUMETSAT_data_access.ipynb @@ -0,0 +1,764 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "88ee180f", + "metadata": {}, + "source": [ + "# EUMETSAT MetOp Data Store Access Notebook\n", + "\n", + "This notebook is a first working example for the MAAP/VEDA/FALCON support task: **use one MetOp product to test EUMETSAT authentication, token creation, product search, and download**.\n", + "\n", + "This notebook defaults to:\n", + "\n", + "`EO:EUM:DAT:METOP:AVHRRL1` — **AVHRR Level 1B - Metop - Global**\n", + "\n", + "\n", + "## What this notebook does\n", + "\n", + "1. Installs/imports `eumdac`\n", + "2. Reads your EUMETSAT consumer key and consumer secret securely\n", + "3. Creates a short-lived access token\n", + "4. Connects to the EUMETSAT Data Store\n", + "5. Selects a MetOp collection\n", + "6. Searches a small time window\n", + "7. Lists matching products\n", + "8. Downloads one product, or optionally one file inside the product\n", + "\n", + "## Notes\n", + "\n", + "- Do **not** hard-code your consumer key or secret in a committed notebook.\n", + "- Use environment variables or `getpass` prompts.\n", + "- The EUMETSAT Data Store path is not the same as a STAC workflow. This notebook focuses on the Data Store/EUMDAC path first. WEkEO/HDA STAC can be tested separately after the basic token workflow is confirmed.\n" + ] + }, + { + "cell_type": "markdown", + "id": "c13b59fc", + "metadata": {}, + "source": [ + "## 0. How to get the EUMETSAT API key\n", + "\n", + "Before running the notebook:\n", + "\n", + "1. Create/sign in to your [EUMETSAT account](https://user.eumetsat.int/cas/login).\n", + "2. Go to the EUMETSAT API key page: [https://api.eumetsat.int/api-key](https://api.eumetsat.int/api-key)\n", + "3. Copy the **consumer key** and **consumer secret**.\n", + "4. Come back to this notebook and paste them only when prompted." + ] + }, + { + "cell_type": "markdown", + "id": "3e2c4bca", + "metadata": {}, + "source": [ + "## 1. Install dependencies\n", + "\n", + "Run this once per clean environment. In MAAP Hub/Jupyter, restart the kernel after installation if imports fail.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "7698a175", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -q eumdac pandas tqdm\n" + ] + }, + { + "cell_type": "markdown", + "id": "1c392423", + "metadata": {}, + "source": [ + "## 2. Imports and output folder\n" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "7f68999b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Download folder: /home/jovyan/eumetsat_metop_downloads\n", + "eumdac version: 3.1.1\n" + ] + } + ], + "source": [ + "from pathlib import Path\n", + "from datetime import datetime, timezone\n", + "import os\n", + "import getpass\n", + "import shutil\n", + "import json\n", + "\n", + "import pandas as pd\n", + "from tqdm.auto import tqdm\n", + "\n", + "import eumdac\n", + "\n", + "DOWNLOAD_DIR = Path(\"eumetsat_metop_downloads\")\n", + "DOWNLOAD_DIR.mkdir(exist_ok=True)\n", + "\n", + "print(\"Download folder:\", DOWNLOAD_DIR.resolve())\n", + "print(\"eumdac version:\", getattr(eumdac, \"__version__\", \"version not available\"))\n" + ] + }, + { + "cell_type": "markdown", + "id": "68128e04", + "metadata": {}, + "source": [ + "## 3. Set the test collection\n", + "\n", + "Default product for the first token/search/download test:\n", + "\n", + "EO:EUM:DAT:METOP:AVHRRL1 — AVHRR Level 1B - Metop - Global\n", + "\n", + "This notebook uses the EUMETSAT AVHRR Level 1B - Metop - Global collection as a placeholder while the FALCON team confirms the exact product they need. AVHRR, or Advanced Very High Resolution Radiometer, is an imaging instrument flown on the Metop polar-orbiting satellites. This collection works well for an initial test because it is unambiguously a MetOp product, it is available through the EUMETSAT Data Store, and the collection ID can be swapped out later without touching any of the token, search, or download logic.\n", + "\n", + "Preferred reference:\n", + "[EUMETSAT product page — AVHRR Level 1B - Metop - Global](https://user.eumetsat.int/catalogue/EO%3AEUM%3ADAT%3AMETOP%3AAVHRRL1)\n", + "\n", + "Change only this variable later if FALCON asks for a different MetOp product." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "cd52bde8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collection ID: EO:EUM:DAT:METOP:AVHRRL1\n", + "Collection note: AVHRR Level 1B - Metop - Global\n" + ] + } + ], + "source": [ + "COLLECTION_ID = \"EO:EUM:DAT:METOP:AVHRRL1\"\n", + "COLLECTION_NAME_NOTE = \"AVHRR Level 1B - Metop - Global\"\n", + "\n", + "print(\"Collection ID:\", COLLECTION_ID)\n", + "print(\"Collection note:\", COLLECTION_NAME_NOTE)\n" + ] + }, + { + "cell_type": "markdown", + "id": "43f73cca", + "metadata": {}, + "source": [ + "## 4. Read credentials securely\n", + "\n", + "This cell checks for environment variables first. If they are not found, it asks you to paste the values.\n", + "\n", + "Environment variable names used here:\n", + "\n", + "- `EUMETSAT_CONSUMER_KEY`\n", + "- `EUMETSAT_CONSUMER_SECRET`\n" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "f18acc73", + "metadata": {}, + "outputs": [ + { + "name": "stdin", + "output_type": "stream", + "text": [ + "EUMETSAT consumer key: ········\n", + "EUMETSAT consumer secret: ········\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Credentials loaded. Secret values are not printed.\n" + ] + } + ], + "source": [ + "def read_secret(env_name: str, prompt: str) -> str:\n", + " value = os.environ.get(env_name)\n", + " if value:\n", + " print(f\"Using {env_name} from environment variables.\")\n", + " return value\n", + " return getpass.getpass(prompt)\n", + "\n", + "CONSUMER_KEY = read_secret(\"EUMETSAT_CONSUMER_KEY\", \"EUMETSAT consumer key: \")\n", + "CONSUMER_SECRET = read_secret(\"EUMETSAT_CONSUMER_SECRET\", \"EUMETSAT consumer secret: \")\n", + "\n", + "assert CONSUMER_KEY, \"Missing consumer key\"\n", + "assert CONSUMER_SECRET, \"Missing consumer secret\"\n", + "\n", + "print(\"Credentials loaded. Secret values are not printed.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "f943d716", + "metadata": {}, + "source": [ + "## 5. Create token and connect to the Data Store\n", + "\n", + "`eumdac.AccessToken` exchanges the consumer key/secret for a short-lived access token. Then `eumdac.DataStore` uses that token to interact with the EUMETSAT Data Store.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "aabfa56c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Access token created successfully.\n", + "Token expiration: 2026-06-20 16:50:01.131138\n" + ] + } + ], + "source": [ + "credentials = (CONSUMER_KEY, CONSUMER_SECRET)\n", + "\n", + "token = eumdac.AccessToken(credentials)\n", + "datastore = eumdac.DataStore(token)\n", + "\n", + "print(\"Access token created successfully.\")\n", + "\n", + "# Some eumdac versions expose expiry information; print it if available, but never print the token value.\n", + "for attr in [\"expiration\", \"expires\", \"expires_at\"]:\n", + " if hasattr(token, attr):\n", + " try:\n", + " print(f\"Token {attr}:\", getattr(token, attr))\n", + " except Exception:\n", + " pass\n" + ] + }, + { + "cell_type": "markdown", + "id": "5713d883", + "metadata": {}, + "source": [ + "## 6. Open the MetOp collection\n", + "\n", + "This should fail fast if the collection ID is wrong or if authentication is not working.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "7e9f2092", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Selected collection:\n", + "EO:EUM:DAT:METOP:AVHRRL1\n" + ] + } + ], + "source": [ + "collection = datastore.get_collection(COLLECTION_ID)\n", + "\n", + "print(\"Selected collection:\")\n", + "print(collection)\n" + ] + }, + { + "cell_type": "markdown", + "id": "9e0af463", + "metadata": {}, + "source": [ + "## 7. Search products in a small time window\n", + "\n", + "Start with a small time range so the search is fast. If you get zero products, change the dates.\n", + "\n", + "For a first access test, we only need a few matching products.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "21ecd46b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Searching:\n", + "Collection: EO:EUM:DAT:METOP:AVHRRL1\n", + "Start: 2024-01-01 00:00:00+00:00\n", + "End: 2024-01-01 06:00:00+00:00\n", + "Max results: 5\n", + "Found 5 product(s) in the first 5 returned results.\n", + "1. AVHR_xxx_1B_M03_20240101051303Z_20240101065503Z_N_O_20240101065154Z\n", + "2. AVHR_xxx_1B_M01_20240101041903Z_20240101060103Z_N_O_20240101050832Z\n", + "3. AVHR_xxx_1B_M03_20240101033103Z_20240101051303Z_N_O_20240101050949Z\n", + "4. AVHR_xxx_1B_M01_20240101023403Z_20240101041903Z_N_O_20240101032442Z\n", + "5. AVHR_xxx_1B_M03_20240101014603Z_20240101033103Z_N_O_20240101032627Z\n" + ] + } + ], + "source": [ + "# Change these dates if the collection returns no products for this window.\n", + "START = datetime(2024, 1, 1, 0, 0, tzinfo=timezone.utc)\n", + "END = datetime(2024, 1, 1, 6, 0, tzinfo=timezone.utc)\n", + "MAX_RESULTS = 5\n", + "\n", + "print(\"Searching:\")\n", + "print(\"Collection:\", COLLECTION_ID)\n", + "print(\"Start:\", START)\n", + "print(\"End:\", END)\n", + "print(\"Max results:\", MAX_RESULTS)\n", + "\n", + "products_iter = collection.search(dtstart=START, dtend=END)\n", + "\n", + "products = []\n", + "for product in products_iter:\n", + " products.append(product)\n", + " if len(products) >= MAX_RESULTS:\n", + " break\n", + "\n", + "print(f\"Found {len(products)} product(s) in the first {MAX_RESULTS} returned results.\")\n", + "\n", + "if not products:\n", + " print(\"No products found. Try widening START/END or searching without a time filter.\")\n", + "else:\n", + " for i, product in enumerate(products, start=1):\n", + " print(f\"{i}. {product}\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "80f4b046", + "metadata": {}, + "source": [ + "## 8. Optional fallback: search latest products without a time filter\n", + "\n", + "Run this only if the date-window search above returns no products.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "f4e9d2d3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Fallback search skipped. Set RUN_LATEST_FALLBACK = True if needed.\n" + ] + } + ], + "source": [ + "RUN_LATEST_FALLBACK = False\n", + "\n", + "if RUN_LATEST_FALLBACK:\n", + " latest_products = []\n", + " for product in collection.search():\n", + " latest_products.append(product)\n", + " if len(latest_products) >= MAX_RESULTS:\n", + " break\n", + "\n", + " print(f\"Found {len(latest_products)} latest product(s).\")\n", + " for i, product in enumerate(latest_products, start=1):\n", + " print(f\"{i}. {product}\")\n", + "\n", + " # Use fallback results for the rest of the notebook.\n", + " products = latest_products\n", + "else:\n", + " print(\"Fallback search skipped. Set RUN_LATEST_FALLBACK = True if needed.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "ef747470", + "metadata": {}, + "source": [ + "## 9. Put product names into a table\n", + "\n", + "EUMDAC product objects can vary by version/product type, so this helper records safe fields without assuming every metadata field exists.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "e4452601", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
product_stringsizesatelliteinstrumentsensing_startsensing_endcollection
0AVHR_xxx_1B_M03_20240101051303Z_20240101065503...403950Metop-CAVHRR2024-01-01 05:13:032024-01-01 06:55:03EO:EUM:DAT:METOP:AVHRRL1
1AVHR_xxx_1B_M01_20240101041903Z_20240101060103...455329Metop-BAVHRR2024-01-01 04:19:032024-01-01 06:01:03EO:EUM:DAT:METOP:AVHRRL1
2AVHR_xxx_1B_M03_20240101033103Z_20240101051303...402704Metop-CAVHRR2024-01-01 03:31:032024-01-01 05:13:03EO:EUM:DAT:METOP:AVHRRL1
3AVHR_xxx_1B_M01_20240101023403Z_20240101041903...456826Metop-BAVHRR2024-01-01 02:34:032024-01-01 04:19:03EO:EUM:DAT:METOP:AVHRRL1
4AVHR_xxx_1B_M03_20240101014603Z_20240101033103...404912Metop-CAVHRR2024-01-01 01:46:032024-01-01 03:31:03EO:EUM:DAT:METOP:AVHRRL1
\n", + "
" + ], + "text/plain": [ + " product_string size satellite \\\n", + "0 AVHR_xxx_1B_M03_20240101051303Z_20240101065503... 403950 Metop-C \n", + "1 AVHR_xxx_1B_M01_20240101041903Z_20240101060103... 455329 Metop-B \n", + "2 AVHR_xxx_1B_M03_20240101033103Z_20240101051303... 402704 Metop-C \n", + "3 AVHR_xxx_1B_M01_20240101023403Z_20240101041903... 456826 Metop-B \n", + "4 AVHR_xxx_1B_M03_20240101014603Z_20240101033103... 404912 Metop-C \n", + "\n", + " instrument sensing_start sensing_end \\\n", + "0 AVHRR 2024-01-01 05:13:03 2024-01-01 06:55:03 \n", + "1 AVHRR 2024-01-01 04:19:03 2024-01-01 06:01:03 \n", + "2 AVHRR 2024-01-01 03:31:03 2024-01-01 05:13:03 \n", + "3 AVHRR 2024-01-01 02:34:03 2024-01-01 04:19:03 \n", + "4 AVHRR 2024-01-01 01:46:03 2024-01-01 03:31:03 \n", + "\n", + " collection \n", + "0 EO:EUM:DAT:METOP:AVHRRL1 \n", + "1 EO:EUM:DAT:METOP:AVHRRL1 \n", + "2 EO:EUM:DAT:METOP:AVHRRL1 \n", + "3 EO:EUM:DAT:METOP:AVHRRL1 \n", + "4 EO:EUM:DAT:METOP:AVHRRL1 " + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def safe_getattr(obj, attr):\n", + " try:\n", + " value = getattr(obj, attr)\n", + " if callable(value):\n", + " return None\n", + " return value\n", + " except Exception:\n", + " return None\n", + "\n", + "records = []\n", + "for product in products:\n", + " record = {\"product_string\": str(product)}\n", + " for attr in [\n", + " \"id\", \"identifier\", \"title\", \"name\", \"size\", \"satellite\", \"instrument\",\n", + " \"sensing_start\", \"sensing_end\", \"publication\", \"collection\"\n", + " ]:\n", + " value = safe_getattr(product, attr)\n", + " if value is not None:\n", + " record[attr] = str(value)\n", + " records.append(record)\n", + "\n", + "products_df = pd.DataFrame(records)\n", + "products_df\n" + ] + }, + { + "cell_type": "markdown", + "id": "c06dc4fc", + "metadata": {}, + "source": [ + "## 10. Inspect one product\n", + "\n", + "This checks what files/entries are available inside the product. For some products this may be a ZIP-like product package.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "51a85f0d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Selected product:\n", + "AVHR_xxx_1B_M03_20240101051303Z_20240101065503Z_N_O_20240101065154Z\n", + "Number of entries detected: 3\n", + "- AVHR_xxx_1B_M03_20240101051303Z_20240101065503Z_N_O_20240101065154Z.nat\n", + "- EOPMetadata.xml\n", + "- manifest.xml\n" + ] + } + ], + "source": [ + "if not products:\n", + " raise RuntimeError(\"No products available. Run a successful search first.\")\n", + "\n", + "selected_product = products[0]\n", + "print(\"Selected product:\")\n", + "print(selected_product)\n", + "\n", + "entries = []\n", + "try:\n", + " entries = list(selected_product.entries)\n", + "except Exception as exc:\n", + " print(\"Could not list product entries:\", repr(exc))\n", + "\n", + "print(f\"Number of entries detected: {len(entries)}\")\n", + "for entry in entries[:20]:\n", + " print(\"-\", entry)\n", + "\n", + "if len(entries) > 20:\n", + " print(f\"... and {len(entries) - 20} more entries\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "d1695135", + "metadata": {}, + "source": [ + "## 11. Download the full selected product\n", + "\n", + "This can be large. The switch is set to `False` by default so the notebook does not accidentally download a huge file.\n", + "\n", + "Set `DOWNLOAD_FULL_PRODUCT = True` when you are ready.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "f073f301", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Full product download skipped. Set DOWNLOAD_FULL_PRODUCT = True to download.\n" + ] + } + ], + "source": [ + "DOWNLOAD_FULL_PRODUCT = False\n", + "\n", + "if DOWNLOAD_FULL_PRODUCT:\n", + " selected_product = products[0]\n", + " print(\"Downloading full product:\", selected_product)\n", + "\n", + " with selected_product.open() as source:\n", + " output_name = Path(source.name).name\n", + " output_path = DOWNLOAD_DIR / output_name\n", + " with open(output_path, \"wb\") as destination:\n", + " shutil.copyfileobj(source, destination)\n", + "\n", + " print(\"Downloaded to:\", output_path.resolve())\n", + " print(\"File size MB:\", round(output_path.stat().st_size / 1_000_000, 2))\n", + "else:\n", + " print(\"Full product download skipped. Set DOWNLOAD_FULL_PRODUCT = True to download.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "9255bdaa", + "metadata": {}, + "source": [ + "## 12. Optional: download only one entry/file inside the product\n", + "\n", + "Use this if the product has entries and you want a small metadata file first. This is useful for proving access without downloading the full science product.\n", + "\n", + "The cell tries to pick `manifest.xml` if it exists; otherwise it uses the first detected entry.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "70d90774", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Single-entry download skipped. Set DOWNLOAD_SINGLE_ENTRY = True to download one entry.\n" + ] + } + ], + "source": [ + "DOWNLOAD_SINGLE_ENTRY = False\n", + "\n", + "if DOWNLOAD_SINGLE_ENTRY:\n", + " if not entries:\n", + " raise RuntimeError(\"No entries were detected for this product. Use the full-product download instead.\")\n", + "\n", + " preferred = None\n", + " for entry in entries:\n", + " if \"manifest\" in str(entry).lower():\n", + " preferred = entry\n", + " break\n", + "\n", + " selected_entry = preferred or entries[0]\n", + " print(\"Downloading entry:\", selected_entry)\n", + "\n", + " with selected_product.open(entry=selected_entry) as source:\n", + " output_name = Path(source.name).name\n", + " output_path = DOWNLOAD_DIR / output_name\n", + " with open(output_path, \"wb\") as destination:\n", + " shutil.copyfileobj(source, destination)\n", + "\n", + " print(\"Downloaded to:\", output_path.resolve())\n", + " print(\"File size MB:\", round(output_path.stat().st_size / 1_000_000, 2))\n", + "else:\n", + " print(\"Single-entry download skipped. Set DOWNLOAD_SINGLE_ENTRY = True to download one entry.\")\n" + ] + }, + { + "cell_type": "markdown", + "id": "fb971070", + "metadata": {}, + "source": [ + "## 13. Reusable function for another MetOp collection\n", + "\n", + "When FALCON provides the exact product, change only the `collection_id`, `start`, and `end` values.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f55d6a2d", + "metadata": {}, + "outputs": [], + "source": [ + "def search_products(collection_id: str, start: datetime, end: datetime, max_results: int = 10):\n", + " # Return up to max_results EUMETSAT Data Store products for a collection and time range.\n", + " col = datastore.get_collection(collection_id)\n", + " results = []\n", + " for product in col.search(dtstart=start, dtend=end):\n", + " results.append(product)\n", + " if len(results) >= max_results:\n", + " break\n", + " return results\n", + "\n", + "# Example use:\n", + "# other_products = search_products(\n", + "# collection_id=\"EO:EUM:DAT:METOP:AVHRRL1\",\n", + "# start=datetime(2024, 1, 1, 0, 0, tzinfo=timezone.utc),\n", + "# end=datetime(2024, 1, 1, 6, 0, tzinfo=timezone.utc),\n", + "# max_results=3,\n", + "# )\n", + "# other_products\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.19" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}