diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..92a01d0 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,49 @@ +name: docs + +on: + push: + branches: [ "main" ] + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: "pages" + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install docs dependencies + run: | + python -m pip install --upgrade pip + pip install -e ".[docs]" + + - name: Build site + run: mkdocs build --strict + + - name: Upload artifact + uses: actions/upload-pages-artifact@v3 + with: + path: site + + deploy: + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + needs: build + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index 10e2992..4d70829 100644 --- a/.gitignore +++ b/.gitignore @@ -4,8 +4,9 @@ out/ .env *.env **/.env +*.mmd +*.db -uv.lock .metals/ .vscode/ data/ @@ -16,6 +17,9 @@ poetry.lock .idea/ target/ +# agents +.cursor/ + # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] diff --git a/.python-version b/.python-version new file mode 100644 index 0000000..e4fba21 --- /dev/null +++ b/.python-version @@ -0,0 +1 @@ +3.12 diff --git a/Backlog.md b/Backlog.md deleted file mode 100644 index f802e65..0000000 --- a/Backlog.md +++ /dev/null @@ -1,11 +0,0 @@ - - - -### Implement System KG with KG core - -Definitions -- Data Artifacts -- Configuration Options -- Tasks/Tools Function -- Pipelines -- Evaluation Function \ No newline at end of file diff --git a/Dockerfile b/Dockerfile index 06cc466..75ccec3 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,6 +13,6 @@ RUN --mount=type=cache,target=/root/.cache/uv \ COPY . . RUN --mount=type=cache,target=/root/.cache/uv \ - uv pip install -e . + uv pip install -e ".[ml,cpu]" ENTRYPOINT ["kgpipe"] diff --git a/README.md b/README.md index 54fcba9..18f26c1 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,22 @@ # KGpipe: A Framework for Knowledge Graph Integration Pipelines -- 📊 [Benchmark Datasets](https://doi.org/10.5281/zenodo.17246357) - - KGpipe is an open-source framework for defining, executing, and evaluating knowledge graph (KG) integration pipelines. It enables the reuse and composition of existing tools (e.g., OpenIE, PARIS, JedAI) and Large Language Models (LLMs) into modular pipelines that integrate heterogeneous data sources into a unified KG. +![KGpipe workflow](docs/workflow.png) + +## Related benchmarks, datasets, and papers + +- [**KGI-Bench**](https://github.com/ScaDS/KGI-Bench): benchmark specification + tooling for KG integration evaluation. +- [**KGI-Bench (Movies)**](https://doi.org/10.5281/zenodo.17246357): Movie-domain benchmark dataset release (Zenodo). +- [**KGpipe Explorer**](https://vehnem.github.io/kgpipe-explorer/): a demo exploring results of KGI-Bench executed with KGpipe. +- [**Framework Paper**](https://arxiv.org/abs/2511.18364): framework core paper; revised version accepted at QDB 2026 (to appear). + +**Who is this for?** +- You have multiple heterogeneous sources (RDF/JSON/text) and want a **reproducible, modular pipeline**. +- You want to **reuse existing tooling** (Python libs, Dockerized CLIs, remote APIs/LLMs) without rewriting everything. +- You want to **evaluate** generated KGs with a growing set of metrics (`kgpipe_eval`). + **Key features:** - Modular and extensible pipeline specification. - Support for multiple execution backends (Python, Docker, HTTP services). @@ -13,6 +24,28 @@ It enables the reuse and composition of existing tools (e.g., OpenIE, PARIS, Jed - Novel benchmark for systematic evaluation of pipelines across RDF, JSON, and text sources. - Metrics covering structural, semantic, and reference-based evaluation. +## Quickstart (5 minutes) + +Install from source (editable): + +```bash +pip install -e . +kgpipe --help +``` + +Bootstrap a minimal example project and discover its tasks: + +```bash +cd experiments/examples +./init.sh + +cd "" +pip install -e . + +kgpipe discover --package --show-results +kgpipe list --type tasks +``` + ## Architecture Each pipeline is a sequence of tasks with well-defined input/output contracts. @@ -49,7 +82,42 @@ KGpipe provides Single-Source Pipelines (SSPs) and Multi-Source Pipelines (MSPs) ## Usage -For documentation see the [docs](docs/reproduce.md) +Documentation lives in `docs/`: +- **Start here**: `docs/index.md` and `docs/quickstart.md` +- **Adopting KGpipe / wrapping existing tools**: `docs/adoption.md` +- **Evaluation (new API)**: `docs/evaluation.md` (uses `kgpipe_eval`) +- **MovieKG reproduction**: `docs/reproduce.md` + +### Documentation site (GitHub Pages) + +This repo is set up to build docs with **MkDocs + Material**: +- config: `mkdocs.yml` +- local build instructions: `docs/README.md` +- deploy workflow: `.github/workflows/docs.yml` (GitHub Pages via Actions) + +## Installation notes (CPU vs CUDA) + +Some optional ML dependencies (e.g. `sentence_transformers`) pull in PyTorch (`torch`). Depending on which PyTorch wheel gets selected, you may see large downloads like `nvidia-*` and `triton`. + +KGpipe keeps the ML stack out of the default install; install it explicitly when needed. For `uv`, PyTorch is pinned to the official PyTorch wheel indexes to avoid accidentally pulling CUDA wheels from PyPI. + +### Base install (fast, no torch) + +```bash +uv pip install . +``` + +### ML install with CPU-only PyTorch (no `nvidia-*`) + +```bash +uv pip install ".[ml,cpu]" +``` + +### ML install with CUDA-enabled PyTorch (will download `nvidia-*`) + +```bash +uv pip install ".[ml,cuda]" +``` ## Experiments -- **[moviekg](experiments/moviekg/README.md)** evalaution of a pipelines, building a Movie KG from three sources (rdf,json,text). +- **[moviekg](experiments/moviekg/README.md)** evaluation of pipelines, building a Movie KG from three sources (rdf, json, text). diff --git a/docs/adoption.md b/docs/adoption.md new file mode 100644 index 0000000..c1229a5 --- /dev/null +++ b/docs/adoption.md @@ -0,0 +1,85 @@ +# Adopting KGpipe (integrating existing pipelines/tools) + +This page explains how to **adopt KGpipe** when you already have: +- an existing KG pipeline (e.g., DBpedia-style multi-step workflows), and/or +- existing implementations you want to reuse (Python code, Dockerized tools, external APIs). + +The goal is to map “what you already have” onto KGpipe’s building blocks: +- **Tasks**: reusable steps with typed inputs/outputs (`input_spec` / `output_spec`) +- **Pipelines**: ordered task graphs (`KgPipe`) that transform `Data` from seed → result +- **Configuration**: parameters passed into tasks (often via env/config profiles) + +## 1) Convert an existing pipeline into a KGpipe pipeline + +When you have a pipeline described elsewhere (scripts, Airflow, Makefile, DBpedia extraction steps, etc.), do this: + +1. **List pipeline steps** (one row per step): name, inputs, outputs, and “how it runs” (Python/Docker/API). +2. **Define formats** for each boundary artifact (RDF formats, CSV, JSON, text). If needed, extend formats. +3. **Wrap each step as a KGpipe task** (see sections below). +4. **Compose tasks into a `KgPipe`** and verify the input/output formats connect. + +Practical tip: start by wrapping a *single* step and run it via `kgpipe task ...`, then grow into a pipeline. + +## 2) Wrap existing tasks (three common patterns) + +### A) Wrap a Dockerized CLI tool + +Use this when the tool is a command-line program and can run inside a container. + +Reference example: +- `src/kgpipe_tasks/entity_resolution/matcher/paris_rdf_matcher.py` + +What to document for each wrapper: +- Docker image name + how to build/pull it +- command template (mapping KGpipe input/output keys to CLI args) +- volume mounts / working dir assumptions +- required environment variables + +### B) Wrap existing Python code + +Use this when you have Python functions/classes you want to call directly. + +Reference example: +- `experiments/param-opti/src/param_opti/tasks/base_linker.py` + +What to document for each wrapper: +- the function/class you call +- how you read from `inputs[...]` and write to `outputs[...]` +- how you map configuration parameters into function args (or config objects) + +### C) Wrap an external API (HTTP service) + +Use this when the implementation is “some service endpoint” (DBpedia Spotlight, LLM providers, etc.). + +Reference examples: +- `experiments/param-opti/src/param_opti/tasks/spotlight_lib.py` +- `experiments/param-opti/src/param_opti/tasks/spotlight.py` + +What to document for each wrapper: +- endpoint URL + auth +- request/response format +- retry/timeouts and caching +- how you handle rate limits and partial failures + +## 3) Discovery (making your tasks available) + +Once tasks exist in a Python package, KGpipe can discover them (they register when imported). + +```bash +kgpipe discover --package --show-results +kgpipe list --type tasks +``` + +## 4) Recommended structure for “adopted” pipelines + +A maintainable layout usually separates: +- `tasks/`: wrappers (Python/Docker/API) +- `pipelines/`: composition (KgPipe builders or pipeline configs) +- `configs/`: pipeline/task configuration profiles +- `docker/`: Dockerfiles and wrapper scripts (if needed) + +## Status + +This page is the intended replacement for `migration.md` (which was a misleading name). It will be expanded with +copy-pastable code snippets for each wrapper type using the referenced files above as canonical examples. + diff --git a/docs/create-docs.md b/docs/create-docs.md new file mode 100644 index 0000000..b066c6a --- /dev/null +++ b/docs/create-docs.md @@ -0,0 +1,45 @@ +# Docs (MkDocs) + +This repository uses **MkDocs + Material** to build the documentation site from the Markdown files in `docs/`. + +## Local preview + +### Option A: pip + +```bash +python -m pip install -e ".[docs]" +mkdocs serve +``` + +Then open the URL shown in the terminal (usually `http://127.0.0.1:8000/`). + +### Option B: uv (recommended if you use uv) + +```bash +uv pip install -e ".[docs]" +mkdocs serve +``` + +## Build + +```bash +mkdocs build --strict +``` + +The static site is written to `site/`. + +## Navigation / sidebar + +Edit `mkdocs.yml` (`nav:` section) to control: +- sidebar structure +- ordering +- page titles + +## Deployment (GitHub Pages) + +Deployment is handled by the GitHub Actions workflow: +- `.github/workflows/docs.yml` + +In your GitHub repo settings, set: +- **Settings → Pages → Source**: **GitHub Actions** + diff --git a/docs/evaluation.md b/docs/evaluation.md index 0650308..613eb33 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -1,124 +1,114 @@ -# KG Evaluation +# KG Evaluation (new API) -The framework provides several approaches to evaluate the quality of a generated knowledge graph. Evaluation is organized into different aspects, each focusing on specific quality dimensions. +KGpipe currently contains **two** evaluation implementations: -## Evaluation Aspects +- **New** (recommended): `kgpipe_eval` (package: `src/kgpipe_eval/`) +- **Old** (deprecated soon): `kgpipe.evaluation` (package: `src/kgpipe/evaluation/`) -The framework supports evaluation across multiple aspects: +This page documents the **new** `kgpipe_eval` API. -- **Statistical**: Basic metrics like triple count, entity count, graph density, and other structural properties -- **Semantic**: Validation of ontology consistency, type errors, relation direction, and semantic correctness -- **Reference**: Comparison against curated gold-standard knowledge graphs using precision, recall, and F1 scores -- **Efficiency**: Resource consumption metrics including runtime, memory usage, and cost +## Mental model -## Using the Evaluator +In `kgpipe_eval`, evaluation is composed from: +- **KG loader / adapter**: turns a `KgLike` (e.g. a `kgpipe.common.model.kg.KG`) into an in-memory `TripleGraph` +- **Metric instances**: objects implementing `Metric.compute(...) -> MetricResult` +- **Metric configs** (optional): typed config objects passed to metrics that require parameters +- **Evaluator**: runs multiple metrics against a loaded graph -The main entry point for evaluation is the `Evaluator` class. You configure which aspects to evaluate and then run evaluation on a knowledge graph: +Key types: +- `kgpipe_eval.api.Metric`: metric interface (`key`, `description`, `compute`) +- `kgpipe_eval.api.MetricResult`: dataclass with `measurements` + optional `summary` +- `kgpipe_eval.evaluator.Evaluator`: runs a list of metrics with an optional `confs` dict + +## Minimal example (statistics) ```python -from kgpipe.evaluation import Evaluator, EvaluationConfig, EvaluationAspect -from kgpipe.common.models import KG, DataFormat from pathlib import Path -# Create evaluation configuration -config = EvaluationConfig( - aspects=[EvaluationAspect.STATISTICAL, EvaluationAspect.SEMANTIC, EvaluationAspect.REFERENCE], - metrics=None # None means use all available metrics for each aspect -) +from kgpipe.common.model.data import DataFormat +from kgpipe.common.model.kg import KG -# Create evaluator -evaluator = Evaluator(config) +from kgpipe_eval.evaluator import Evaluator +from kgpipe_eval.metrics.statistics import CountMetric +from kgpipe_eval.utils.kg_utils import KgManager -# Load the knowledge graph to evaluate kg = KG( id="my_kg", - name="My Knowledge Graph", - path=Path("result.nt"), - format=DataFormat.RDF_NTRIPLES + name="My KG", + path=Path("my_kg.nt"), + format=DataFormat.RDF_NTRIPLES, ) -# For reference-based evaluation, provide reference data -references = { - "gold_standard": Data(path=Path("gold_standard.nt"), format=DataFormat.RDF_NTRIPLES) -} - -# Run evaluation -report = evaluator.evaluate(kg, references=references) - -# Access results -print(f"Overall score: {report.overall_score}") -for aspect_result in report.aspect_results: - print(f"{aspect_result.aspect.value}: {len(aspect_result.metrics)} metrics") - for metric in aspect_result.metrics: - print(f" {metric.name}: {metric.value} (normalized: {metric.normalized_score})") -``` - -## Evaluation via CLI - -You can also evaluate knowledge graphs using the command-line interface: +tg = KgManager.load_kg(kg) +results = Evaluator().run(tg, metrics=[CountMetric()]) -```bash -kgpipe eval target.nt --ground-truth gold.nt --aspects statistical semantic reference --output results.json +for r in results: + print(r.metric.key, r.summary) + for m in r.measurements: + print(" ", m.name, m.value) ``` -The CLI supports: -- `--aspects`: Specify which aspects to evaluate (statistical, semantic, reference, efficiency) -- `--metrics`: Filter to specific metrics by name -- `--ground-truth`: Path to reference knowledge graph for reference-based evaluation -- `--output`: Save evaluation results to a JSON file - -## Statistical Evaluation - -Statistical evaluation provides basic metrics about the knowledge graph structure: +## Metrics that need configuration -- Triple count -- Entity count -- Relation count -- Graph density -- Average degree -- Connected components +Some metrics require a config object. The `Evaluator` detects this by introspecting the metric’s +`compute(...)` signature: +- `compute(self, kg)` → no config needed +- `compute(self, kg, config)` → config required and must be provided -These metrics help understand the scale and structure of the generated knowledge graph. +You pass configs via a dict keyed by the metric key/class name. -## Semantic Evaluation +Example (triple alignment + duplicates): -Semantic evaluation validates the knowledge graph against its ontology: - -- Disjoint domain violations -- Incorrect relation direction -- Incorrect relation cardinality -- Incorrect relation domain/range -- Incorrect datatypes -- Ontology class coverage -- Ontology relation coverage -- Namespace coverage - -These metrics ensure the knowledge graph conforms to its schema and maintains semantic consistency. - -## Reference-based Evaluation - -Reference-based evaluation compares the generated knowledge graph against a gold standard: +```python +from kgpipe_eval.evaluator import Evaluator +from kgpipe_eval.utils.kg_utils import KgManager + +from kgpipe_eval.metrics.duplicates import DuplicateMetric, DuplicateConfig +from kgpipe_eval.metrics.triple_alignment import TripleAlignmentMetric, TripleAlignmentConfig +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig + +tg = KgManager.load_kg("path/to/result_eval.nt") # KgLike: path, KG object, ... + +metrics = [DuplicateMetric(), TripleAlignmentMetric()] +confs = { + "DuplicateMetric": DuplicateConfig( + entity_alignment_config=EntityAlignmentConfig( + method="label_embedding", + verified_entities_path="path/to/verified_entities.tsv", + verified_entities_delimiter="\\t", + entity_sim_threshold=0.95, + ) + ), + "TripleAlignmentMetric": TripleAlignmentConfig( + reference_kg="path/to/reference.nt", + entity_alignment_config=EntityAlignmentConfig( + method="label_embedding", + reference_kg="path/to/reference.nt", + entity_sim_threshold=0.95, + ), + value_sim_threshold=0.5, + cache_literal_embeddings=True, + cache_ref_literal_embeddings=True, + ), +} -- Entity matching (precision, recall, F1) -- Relation matching (precision, recall, F1) -- Triple alignment -- Source typed entity coverage -- Reference class coverage +results = Evaluator().run(tg, metrics, confs) +``` -This type of evaluation requires a curated reference knowledge graph that serves as ground truth. +## Canonical reference example (MovieKG) -## Evaluation Reports +For a realistic end-to-end usage example (loading pipeline stage outputs, wiring configs, running multiple metrics), +see: -Evaluation results are returned as `EvaluationReport` objects that contain: +- `experiments/moviekg/src/moviekg/evaluation/test_eval_refactor.py` -- The evaluated knowledge graph -- Reference data used (if any) -- Aspect results for each evaluated aspect -- Individual metric results with values and normalized scores -- Overall score (average of normalized scores across all metrics) +That file shows how to: +- build per-metric configs (duplicates/entity alignment/triple alignment) +- load the KG from a pipeline output directory +- serialize `MetricResult` to JSON (because it contains metric objects) -Reports can be serialized to JSON for storage and later analysis: +## CLI note -```python -report.to_json("evaluation_results.json") -``` +There is a “new eval” CLI command path intended to run these metrics (see `kgpipe_eval.api` docstring mentioning +`kgpipe eval-new`). If you want the docs to include the CLI, we should first confirm the exact CLI flags and expected +inputs in `src/kgpipe/cli/eval_new.py` and align this page with that implementation. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..4a8a452 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,48 @@ +# KGpipe framework documentation + +KGpipe is a framework to define pipelines for data integration into knowledge graphs. The framework enables you to compose existing tools and implementations into modular pipelines that integrate heterogeneous data sources into a unified knowledge graph. + +The framework is organized into three main subpackages: `kgpipe` contains the core framework functionality including CLI, common utilities, execution backends, and evaluation components. `kgpipe_tasks` provides task implementations for cleaning, construction, entity resolution, schema alignment, and text processing. `kgpipe_llm` includes LLM-based task implementations and utilities. + +**Current version**: 0.7.0 +**Python**: >= 3.12 + +![KGpipe workflow](workflow.png) + +## Quickstart + +Start here: [Quickstart guide](quickstart.md) + +Minimal “happy path” (install + discover + inspect what’s available): + +```bash +pip install -e . +kgpipe discover --all --show-results +kgpipe list --type tasks +kgpipe list --type metrics +``` + +Create a new experiment project (recommended): + +```bash +cd experiments/examples +./init.sh +``` + +## How to use KGpipe (docs map) + +- Define tasks: [Task specification](tasks.md) +- Build and run pipelines: [Pipelines](pipelines.md) +- Configure runs and task parameters: [Configuration](configuration.md) and [Parameters](parameters.md) +- Evaluate generated KGs: [Evaluation](evaluation.md) and [Metrics](metrics/metrics.md) +- Understand the internal “PipeKG”: [Meta KG](metakg.md) + +## Other Links + +- [Reproducing the movie kg experiments for 15 pipelines](reproduce.md) (rdf, json, text) +- [Adopting KGpipe (integrating existing pipelines/tools)](adoption.md) +- [UI / viewer](view.md) + +## Docs backlog + +Open items live in `TODO.md` (High/Medium/Low priority). Keep the landing page focused on user-facing docs. \ No newline at end of file diff --git a/docs/main.md b/docs/main.md deleted file mode 100644 index 1a39c1b..0000000 --- a/docs/main.md +++ /dev/null @@ -1,49 +0,0 @@ -# KGpipe framework documentation - -KGpipe is a framework to define pipelines for data integration into knowledge graphs. The framework enables you to compose existing tools and implementations into modular pipelines that integrate heterogeneous data sources into a unified knowledge graph. - -The framework is organized into three main subpackages: `kgpipe` contains the core framework functionality including CLI, common utilities, execution backends, and evaluation components. `kgpipe_tasks` provides task implementations for cleaning, construction, entity resolution, schema alignment, and text processing. `kgpipe_llm` includes LLM-based task implementations and utilities. - -## Meta KG - -[link](metakg.md) - -KGpipe uses an internally maintained Meta KG (PipeKG) to maintain tasks, tool implementations, their components, pipelines, and metrics. This knowledge base enables automatic pipeline generation and tracking of execution results. - -## Task Specification - -[link](tasks.md) - -The framework enables the description and integration of integration tasks. You can describe tasks with Python, interface existing implementations with Python, Docker, or remote API requests. Tasks are discovered and registered through the framework's discovery mechanism. - -## Pipeline Generation and Execution - -[link](pipelines.md) - -KGpipe allows you to define pipelines manually or using an automatic search algorithm that operates on the PipeKG knowledge base and a set of given constraints. You can swap subpipelines or single tasks with other components to experiment with different approaches. - -## Configuration - -[link](configuration.md) - -The framework supports configuration at multiple levels. The main configuration is specified in `kgpipe.yml`, and individual tasks can define their own configuration parameters that will be passed by the framework when executing pipelines. - -## Evaluation - -[link](evaluation.md) - -The framework provides several approaches to evaluate the quality of a generated knowledge graph, including accuracy, coverage, consistency, statistics, and efficiency measurements. Evaluation metrics are tracked in the Meta KG alongside pipeline results. - -Additional evaluation metrics are documented in the [metrics](metrics/) directory, such as [entity coverage](metrics/entity_coverage.md). - -## Other Links - -- [Reproducing the movie kg experiments for 15 pipelines](reproduce.md) (rdf, json, text) - -## Docu Backlog - -- Explain different execution modes - - File Batches - - Streaming -- Explain advanced pipelines -- Ontology creation... \ No newline at end of file diff --git a/docs/metrics/entity_coverage.md b/docs/metrics/entity_coverage.md index 3e76cb3..1aa3ea5 100644 --- a/docs/metrics/entity_coverage.md +++ b/docs/metrics/entity_coverage.md @@ -1,21 +1,73 @@ +# Entity Coverage Metric (OLD) +The Entity Coverage metric evaluates how well source entities are integrated into the target knowledge graph. It measures the overlap between expected source entities and the entities actually present in the generated knowledge graph. +## Source Entity Integration Score -# Source Entitiy Integration Score +The metric compares a set of expected source entities (provided as a reference file) against the entities found in the knowledge graph. It calculates coverage based on entity URIs and labels. -# Entity Integration Score +## Input Format +The expected entities are provided in a CSV or JSON file with the following structure: + +**CSV Format:** ``` URI, LABEL, TYPE +http://example.org/entity1, "Entity Label 1", EntityType +http://example.org/entity2, "Entity Label 2", EntityType +``` + +**JSON Format:** +```json +{ + "http://example.org/entity1": { + "entity_label": "Entity Label 1", + "entity_type": "EntityType" + }, + "http://example.org/entity2": { + "entity_label": "Entity Label 2", + "entity_type": "EntityType" + } +} +``` + +## Calculation + +The metric performs the following steps: + +1. **Load expected entities**: Reads the entity dictionary from the provided file path +2. **Extract entity identifiers**: Collects URIs and labels from the expected entities +3. **Find entities in KG**: Searches the knowledge graph for entities matching by URI or label (using `rdfs:label`) +4. **Calculate overlap**: Counts how many expected entities are found in the KG + +The coverage score is calculated as: + ``` +coverage = overlapping_entities_count / expected_entities_count +``` + +Where: +- `overlapping_entities_count`: Number of expected entities found in the KG +- `expected_entities_count`: Total number of entities in the reference file -Set of entity type pairs -Make overlap on entity_type pairs +## Variants -intesection= -precission -recall= +The framework provides several variants of entity coverage metrics: +- **SourceEntityCoverageMetric**: Strict matching by URI and label +- **SourceEntityCoverageMetricSoft**: Fuzzy matching using label embeddings (threshold 0.95) +- **SourceTypedEntityCoverageMetric**: Matching based on entity type pairs, calculating precision and recall on entity-type combinations +## Usage +To use this metric in evaluation, provide the path to the verified source entities file in the reference configuration: + +```python +from kgpipe.evaluation.aspects.reference import ReferenceConfig + +config = ReferenceConfig( + VERIFIED_SOURCE_ENTITIES="path/to/entities.csv" +) +``` +The metric will automatically be included when evaluating with the `REFERENCE` aspect. diff --git a/experiments/moviekg/src/moviekg/evaluation/__init__.py b/docs/metrics/metrics.md similarity index 100% rename from experiments/moviekg/src/moviekg/evaluation/__init__.py rename to docs/metrics/metrics.md diff --git a/experiments/moviekg/src/moviekg/paper/__init__.py b/docs/metrics/reference_entity_alignment.md similarity index 100% rename from experiments/moviekg/src/moviekg/paper/__init__.py rename to docs/metrics/reference_entity_alignment.md diff --git a/experiments/moviekg/src/moviekg/paper/helpers/__init__.py b/docs/metrics/reference_triple_alignment.md similarity index 100% rename from experiments/moviekg/src/moviekg/paper/helpers/__init__.py rename to docs/metrics/reference_triple_alignment.md diff --git a/docs/metrics/stats_counts.md b/docs/metrics/stats_counts.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/migration.md b/docs/migration.md new file mode 100644 index 0000000..1ca2964 --- /dev/null +++ b/docs/migration.md @@ -0,0 +1,7 @@ +# Migration (renamed) + +This page was renamed to better reflect its intent. + +Use: +- [`adoption.md`](adoption.md): **Adopting KGpipe (integrating existing pipelines/tools)** + diff --git a/docs/parameters.md b/docs/parameters.md new file mode 100644 index 0000000..150abab --- /dev/null +++ b/docs/parameters.md @@ -0,0 +1,25 @@ +# Parameters + +We can parameterize pipelines in the following way + +## Selection of task implementation + +pipeline Task layout +pipeline Task ... + +with subtasks +complete tasks + +## Selection of parameter in task implementation + +# Strategies + +Domain specific tasks + +Configuration options + + +## Backlog +- conf_examples.py +- kgpipe_parameters +- \ No newline at end of file diff --git a/docs/quickstart.md b/docs/quickstart.md new file mode 100644 index 0000000..a2b08f4 --- /dev/null +++ b/docs/quickstart.md @@ -0,0 +1,173 @@ +# KGpipe Quickstart + +This quickstart shows the **current** workflow for defining and running: +- tasks (Python functions registered in the `Registry`) +- pipelines (a `KgPipe` connecting tasks via input/output `Data`) +- metrics/evaluations (evaluators + metrics run on a `KG`) + +## See also +- `experiments/examples/`: a minimal example project using KGpipe +- `docs/reproduce.md`: running the (deprecated but working) reproduction experiments + +## Install + +From the repo root: + +```bash +pip install -e . +``` + +If you need the optional ML stack (transformers / sentence-transformers), install extras: + +```bash +pip install -e ".[ml]" +``` + +## Create a new experiment (recommended starting point) + +The easiest way to get a working project layout is to copy the template in `experiments/examples/`. + +```bash +cd experiments/examples +./init.sh +``` + +The script creates a new directory containing a Python package with example tasks/pipelines. Then: + +```bash +cd "" +pip install -e . +``` + +## Define tasks + +Tasks are normal Python callables registered via `@Registry.task(...)`. See +`experiments/examples/src/kgpipe_examples/task_examples.py` for canonical examples. + +Key concepts: +- **`input_spec` / `output_spec`**: the expected formats for inputs/outputs +- **`TaskInput` / `TaskOutput`**: dict-like objects mapping names to `Data` +- **`trace_task_run`**: wraps the function to produce a run report + +Minimal pattern (simplified from the examples): + +```python +from kgpipe.common import TaskInput, TaskOutput, trace_task_run +from kgpipe.common.registry import Registry + +@trace_task_run +@Registry.task( + input_spec={"input": "some_format"}, + output_spec={"output": "some_other_format"}, + description="Example task", +) +def my_task(inputs: TaskInput, outputs: TaskOutput): + outputs["output"].path.touch() +``` + +## Define and run a pipeline (Python API) + +Pipelines connect tasks by passing `Data` (path + format) between them. A minimal example exists in +`experiments/examples/src/kgpipe_examples/pipe_examples.py`. + +The core pattern: + +```python +from kgpipe.common import KgPipe, Data + +# tasks = [task_a, task_b, ...] # registered task callables (from your package) +# seed = Data(path=..., format=...) +# result = Data(path=..., format=...) +pipe = KgPipe(tasks=tasks, seed=seed, data_dir="/tmp/my_run_dir") +pipe.build(source=seed, result=result) +pipe.run() +``` + +## Discover components and inspect what’s available (CLI) + +The CLI entrypoint is `kgpipe` (see `pyproject.toml`). + +To register tasks/pipelines/metrics from your local package, import it via discovery: + +```bash +# From inside your experiment venv / environment +kgpipe discover --package --show-results +``` + +You can also discover from a local module path (directory or file): + +```bash +kgpipe discover --module-path ./src/ --show-results +``` + +To list what KGpipe currently knows about (after discovery): + +```bash +kgpipe list --type tasks +kgpipe list --type metrics +``` + +To show details for a specific task: + +```bash +kgpipe show --type task +``` + +To print YAML templates for evaluation configs: + +```bash +kgpipe show metric-config-templates +``` + +## Run a single task (CLI) + +KGpipe can execute a registered task directly. The `--input/--output` syntax is: + +\[ +\texttt{|@} +\] + +(`@` is optional.) + +Example: + +```bash +kgpipe task \ + --input "/tmp/in.txt|txt@input" \ + --output "/tmp/out.txt|txt@output" +``` + +Tip: if you get “Task not found”, run `kgpipe discover ...` first. + +## Run a minimal evaluation / metrics (Python API, new `kgpipe_eval`) + +KG evaluation is being migrated to the **new** `kgpipe_eval` package (recommended). A realistic integration-style +example exists in: + +- `experiments/moviekg/src/moviekg/evaluation/test_eval_refactor.py` + +Minimal example (basic statistics): + +```python +from pathlib import Path + +from kgpipe.common.model.data import DataFormat +from kgpipe.common.model.kg import KG + +from kgpipe_eval.evaluator import Evaluator +from kgpipe_eval.metrics.statistics import CountMetric +from kgpipe_eval.utils.kg_utils import KgManager + +kg = KG( + id="my_kg", + name="My KG", + path=Path("my_kg.nt"), + format=DataFormat.RDF_NTRIPLES, +) + +tg = KgManager.load_kg(kg) +results = Evaluator().run(tg, metrics=[CountMetric()]) + +for r in results: + print(r.metric.key, r.summary) +``` \ No newline at end of file diff --git a/docs/reproduce.md b/docs/reproduce.md index 40875e6..1d9349b 100644 --- a/docs/reproduce.md +++ b/docs/reproduce.md @@ -1,7 +1,7 @@ -# Rep Experiments +# Rep Experiments (Deprecated but working) -Guidelines to run the [experiments](../experiments) -- see also [moviekg](../experiments/moviekg/README.md) +Guidelines to run the [experiments](https://github.com/ScaDS/KGpipe/tree/main/experiments) +- see also [moviekg](https://github.com/ScaDS/KGpipe/blob/main/experiments/moviekg/README.md) ## Overview diff --git a/docs/view.md b/docs/view.md new file mode 100644 index 0000000..d1404c4 --- /dev/null +++ b/docs/view.md @@ -0,0 +1,13 @@ +# View Package + +The view package is a simple streamlit app +to view and visualize the core components of the +framework during development + +``` +uv run streamlit run src/kgpipe_view/kgpipe_view.py +``` + +It is different from the KGpipe-Explorer as it only focuses +on viewing the internal structure and tabular versions of +the KGpipe Core classes for a connected PipeKG. diff --git a/docs/workflow.png b/docs/workflow.png new file mode 100644 index 0000000..41239b0 Binary files /dev/null and b/docs/workflow.png differ diff --git a/experiments/examples/src/kgpipe_examples/conf_examples.py b/experiments/examples/src/kgpipe_examples/conf_examples.py new file mode 100644 index 0000000..b40ab94 --- /dev/null +++ b/experiments/examples/src/kgpipe_examples/conf_examples.py @@ -0,0 +1,159 @@ +from __future__ import annotations + +from collections import defaultdict, deque +from dataclasses import dataclass +from typing import Dict, List, Sequence, Tuple + + +@dataclass(frozen=True) +class DagNode: + name: str + inputs: Tuple[str, ...] = () + output: str | None = None + + +class Dag: + """Minimal DAG API with dependency validation and parallel batches.""" + + def __init__(self) -> None: + self._nodes: Dict[str, DagNode] = {} + self._parents: Dict[str, set[str]] = defaultdict(set) + self._children: Dict[str, set[str]] = defaultdict(set) + self._data_producers: Dict[str, str] = {} + + def task(self, name: str, *, needs: Sequence[str] = (), produces: str | None = None) -> Dag: + if name in self._nodes: + raise ValueError(f"Task '{name}' already exists") + + if produces and produces in self._data_producers: + producer = self._data_producers[produces] + raise ValueError(f"Data '{produces}' is already produced by '{producer}'") + + node = DagNode(name=name, inputs=tuple(needs), output=produces) + self._nodes[name] = node + if produces: + self._data_producers[produces] = name + return self + + def wire(self) -> Dag: + """Resolve data dependencies into task edges.""" + for node in self._nodes.values(): + for data_id in node.inputs: + parent = self._data_producers.get(data_id) + if parent is None: + raise ValueError( + f"Task '{node.name}' requires '{data_id}', but no upstream task produces it" + ) + self._parents[node.name].add(parent) + self._children[parent].add(node.name) + + self._assert_acyclic() + return self + + def execution_batches(self) -> List[List[str]]: + """ + Return topological levels. + Tasks in the same inner list can run in parallel. + """ + indegree = {name: len(self._parents[name]) for name in self._nodes} + frontier = deque(sorted([n for n, d in indegree.items() if d == 0])) + batches: List[List[str]] = [] + + while frontier: + level: List[str] = list(frontier) + frontier.clear() + batches.append(level) + + for task_name in level: + for child in sorted(self._children[task_name]): + indegree[child] -= 1 + if indegree[child] == 0: + frontier.append(child) + + total = sum(len(batch) for batch in batches) + if total != len(self._nodes): + raise ValueError("Graph contains a cycle") + return batches + + def edges(self) -> List[Tuple[str, str]]: + out: List[Tuple[str, str]] = [] + for parent, children in sorted(self._children.items()): + for child in sorted(children): + out.append((parent, child)) + return out + + def to_mermaid_mmd(self, direction: str = "LR") -> str: + """ + Export graph as Mermaid mmd text. + Call this after `wire()` so task dependencies are resolved. + """ + lines: List[str] = [f"flowchart {direction}"] + + for node_name, node in sorted(self._nodes.items()): + node_lines = [node.name] + if node.inputs: + node_lines.append(f"needs: {', '.join(node.inputs)}") + if node.output: + node_lines.append(f"produces: {node.output}") + label = "
".join(node_lines) + lines.append(f' {node_name}["{label}"]') + + for parent, child in self.edges(): + lines.append(f" {parent} --> {child}") + + return "\n".join(lines) + + def _assert_acyclic(self) -> None: + visited: set[str] = set() + in_stack: set[str] = set() + + def dfs(node_name: str) -> None: + visited.add(node_name) + in_stack.add(node_name) + for child_name in self._children[node_name]: + if child_name not in visited: + dfs(child_name) + elif child_name in in_stack: + raise ValueError(f"Cycle detected at '{child_name}'") + in_stack.remove(node_name) + + for name in self._nodes: + if name not in visited: + dfs(name) + + +def dag_example() -> Dag: + """ + Typical syntax: + - split: one output consumed by several branches + - join: one task requiring multiple inputs + - final output: last task produces the sink artifact + """ + dag = ( + Dag() + .task("load_users", produces="users") + .task("load_orders", produces="orders") + .task("clean_users", needs=("users",), produces="users_clean") + .task("clean_orders", needs=("orders",), produces="orders_clean") + .task("extract_features_a", needs=("users_clean","orders_clean"), produces="features_a") + .task("extract_features_b", needs=("users_clean",), produces="features_b") + .task( + "join_user_order_features", + needs=("features_a", "features_b", "orders_clean"), + produces="joined_features", + ) + .task("train_model", needs=("joined_features",), produces="model") + .task("evaluate_model", needs=("model",), produces="report") + .wire() + ) + return dag + + +if __name__ == "__main__": + dag = dag_example() + print("Edges:", dag.edges()) + print("Parallel batches:", dag.execution_batches()) + print("\nMermaid mmd:\n") + print(dag.to_mermaid_mmd()) + + diff --git a/experiments/examples/src/kgpipe_examples/config.py b/experiments/examples/src/kgpipe_examples/config.py index c9cc32e..77768ae 100644 --- a/experiments/examples/src/kgpipe_examples/config.py +++ b/experiments/examples/src/kgpipe_examples/config.py @@ -1,19 +1,8 @@ -from enum import Enum -from kgpipe.common.model.data import DynamicFormat, FormatRegistry +from kgpipe.common.model.default_catalog import CustomDataFormats -class ExtendedFormats(Enum): - SPECIAL_IN = DynamicFormat(name="special_in", extension=".special_in", description="Special input format") - SPECIAL1 = DynamicFormat(name="special1", extension=".special1", description="Special format 1") - SPECIAL2 = DynamicFormat(name="special2", extension=".special2", description="Special format 2") - SPECIAL_KG = DynamicFormat(name="special_kg", extension=".special_kg", description="Special output format for knowledge graph") -FORMAT_REGISTRY = FormatRegistry() - -FORMAT_REGISTRY.register_format( - ExtendedFormats.SPECIAL_IN.value.name, ExtendedFormats.SPECIAL_IN.value.extension, ExtendedFormats.SPECIAL_IN.value.description) -FORMAT_REGISTRY.register_format( - ExtendedFormats.SPECIAL1.value.name, ExtendedFormats.SPECIAL1.value.extension, ExtendedFormats.SPECIAL1.value.description) -FORMAT_REGISTRY.register_format( - ExtendedFormats.SPECIAL2.value.name, ExtendedFormats.SPECIAL2.value.extension, ExtendedFormats.SPECIAL2.value.description) -FORMAT_REGISTRY.register_format( - ExtendedFormats.SPECIAL_KG.value.name, ExtendedFormats.SPECIAL_KG.value.extension, ExtendedFormats.SPECIAL_KG.value.description) \ No newline at end of file +class ExtendedFormats(CustomDataFormats): + SPECIAL_IN = "special_in" + SPECIAL1 = "special1" + SPECIAL2 = "special2" + SPECIAL_KG = "special_kg" \ No newline at end of file diff --git a/experiments/examples/src/kgpipe_examples/eval_examples.py b/experiments/examples/src/kgpipe_examples/eval_examples.py new file mode 100644 index 0000000..ae5434f --- /dev/null +++ b/experiments/examples/src/kgpipe_examples/eval_examples.py @@ -0,0 +1,88 @@ +from kgpipe.evaluation.aspects.statistical import ( + StatisticalEvaluator, + StatisticalConfig, + EntityCountMetric +) +from kgpipe.evaluation.aspects.semantic import ( + SemanticEvaluator, + SemanticConfig, + DisjointDomainMetric, + IncorrectRelationDirectionMetric, + IncorrectRelationRangeMetric, + IncorrectRelationDomainMetric, + IncorrectDatatypeMetric, + IncorrectDatatypeFormatMetric, +) +from kgpipe.evaluation.aspects.reference import ( + ReferenceEvaluator, + ReferenceConfig, + SourceTypedEntityCoverageMetric, + ReferenceTripleAlignmentMetric, + ReferenceTripleAlignmentMetricSoftE, + ReferenceTripleAlignmentMetricSoftEV, +) +from kgpipe.common.model.kg import KG +from kgpipe.common.model.default_catalog import BasicDataFormats +from typing import List +from pathlib import Path + +from kgpipe.common.graph import mapper + +TEST_NTRIPLES = """ + . + "itemA" . + "The Hobbit, or There and Back Again" . + . + "9780261102217" . + + . + "itemB" . + "Pride & Prejudice" . + . + "9780199535569" . + + . + "itemC" . + "1984" . + . + "9780452284234" . +""" + +def eval_example(tmp_path: Path): + """Example: Evaluate a KG against a ground truth.""" + + tmp_path = tmp_path / "my_kg.nt" + tmp_path.write_text(TEST_NTRIPLES) + + kg = KG( + id="my_kg", + name="My Knowledge Graph", + path=tmp_path, + format=BasicDataFormats.RDF_NTRIPLES + ) + + statistical_config = StatisticalConfig(name="default") + # semantic_config = SemanticConfig(name="default") + # reference_config = ReferenceConfig( + # name="default" + # REFERENCE_KG_PATH=...) + + statistical_evaluator = StatisticalEvaluator() + # semantic_evaluator = SemanticEvaluator() + # reference_evaluator = ReferenceEvaluator() + + statistical_metrics: List[str] = [EntityCountMetric().name] + # semantic_metrics: List[str] = [DisjointDomainMetric().name, IncorrectRelationDirectionMetric().name, IncorrectRelationRangeMetric().name, IncorrectRelationDomainMetric().name, IncorrectDatatypeMetric().name, IncorrectDatatypeFormatMetric().name] + # reference_metrics: List[str] = [SourceTypedEntityCoverageMetric().name, ReferenceTripleAlignmentMetric().name, ReferenceTripleAlignmentMetricSoftE().name, ReferenceTripleAlignmentMetricSoftEV().name] + + statistical_results = statistical_evaluator.evaluate( + kg, metrics=statistical_metrics, config=statistical_config) + # semantic_results = semantic_evaluator.evaluate( + # kg, metrics=semantic_metrics, config=semantic_config) + # reference_results = reference_evaluator.evaluate( + # kg, metrics=reference_metrics, config=reference_config) + + for metric in statistical_results.metrics: + mapper.metric_run_to_entity(metric) + + return statistical_results #, semantic_results, reference_results \ No newline at end of file diff --git a/experiments/examples/src/kgpipe_examples/pipe_examples.py b/experiments/examples/src/kgpipe_examples/pipe_examples.py index 0a44e3d..79289ea 100644 --- a/experiments/examples/src/kgpipe_examples/pipe_examples.py +++ b/experiments/examples/src/kgpipe_examples/pipe_examples.py @@ -19,6 +19,7 @@ def pipe_example(): tmp_data_dir = tempfile.mkdtemp() input_data = Data(path=os.path.join(tmp_data_dir, "input.special_in"), format=ExtendedFormats.SPECIAL_IN) output_data = Data(path=os.path.join(tmp_data_dir, "output.special_kg"), format=ExtendedFormats.SPECIAL_KG) + input_data.path.touch() tasks = [pipe_task_python, pipe_task_docker, pipe_task_remote] diff --git a/experiments/examples/src/kgpipe_examples/task_examples.py b/experiments/examples/src/kgpipe_examples/task_examples.py index 82ba839..89345d0 100644 --- a/experiments/examples/src/kgpipe_examples/task_examples.py +++ b/experiments/examples/src/kgpipe_examples/task_examples.py @@ -1,16 +1,26 @@ from kgpipe.common import TaskInput, TaskOutput +from kgpipe.common import trace_task_run +from kgpipe.common.model.configuration import ConfigurationProfile, ConfigurationDefinition, Parameter, ParameterType from kgpipe_examples.config import ExtendedFormats +from kgpipe.common.model.default_catalog import BasicTaskCategoryCatalog from kgpipe.common.registry import Registry +@trace_task_run @Registry.task( input_spec={"input": ExtendedFormats.SPECIAL_IN}, - output_spec={"output": ExtendedFormats.SPECIAL1} + output_spec={"output": ExtendedFormats.SPECIAL1}, + category=[BasicTaskCategoryCatalog.entity_resolution], + description="A task that processes a special input and produces a special output" ) def pipe_task_python(inputs: TaskInput, outputs: TaskOutput): # touch output file outputs["output"].path.touch() +# def converts_pdfs: pass +# def extracts_text + +@trace_task_run @Registry.task( input_spec={"input": ExtendedFormats.SPECIAL1}, output_spec={"output": ExtendedFormats.SPECIAL2} @@ -19,6 +29,7 @@ def pipe_task_docker(inputs: TaskInput, outputs: TaskOutput): # touch output file outputs["output"].path.touch() +@trace_task_run @Registry.task( input_spec={"input": ExtendedFormats.SPECIAL2}, output_spec={"output": ExtendedFormats.SPECIAL_KG} @@ -28,3 +39,27 @@ def pipe_task_remote(inputs: TaskInput, outputs: TaskOutput): outputs["output"].path.touch() +@trace_task_run +@Registry.task( + input_spec={"input": ExtendedFormats.SPECIAL1}, + output_spec={"output": ExtendedFormats.SPECIAL_KG}, + category=[BasicTaskCategoryCatalog.entity_resolution], + config_spec=ConfigurationDefinition( + name="pipe_task_with_config_spec", + description="Configuration specification for the pipe_task_with_config task", + parameters=[ + Parameter( + name="some_parameter", + native_keys=["some_parameter"], + datatype=ParameterType.string, + default_value="default", + required=False, + allowed_values=[] + ) + ] + ) +) +def pipe_task_with_config(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + # print config + print(config) + outputs["output"].path.touch() \ No newline at end of file diff --git a/experiments/examples/src/kgpipe_examples/test_examples.py b/experiments/examples/src/kgpipe_examples/test_examples.py index 08d63f7..c937774 100644 --- a/experiments/examples/src/kgpipe_examples/test_examples.py +++ b/experiments/examples/src/kgpipe_examples/test_examples.py @@ -1,17 +1,213 @@ +from pathlib import Path +from kgpipe.common import Data -def test_python_task_defintion(): + +def test_python_task_execution(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats from kgpipe_examples.task_examples import pipe_task_python - assert pipe_task_python.name == "pipe_task_python" -def test_docker_task_defintion(): + in_file = tmp_path / "input.special_in" + out_file = tmp_path / "output.special1" + in_file.touch() + + report = pipe_task_python.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL_IN)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL1)], + ) + + assert pipe_task_python.name == "pipe_task_python" + assert report.status == "success" + assert out_file.exists() + + +def test_docker_task_execution(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats from kgpipe_examples.task_examples import pipe_task_docker + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special2" + in_file.touch() + + report = pipe_task_docker.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL2)], + ) + assert pipe_task_docker.name == "pipe_task_docker" + assert report.status == "success" + assert out_file.exists() -def test_remote_task_defintion(): + +def test_remote_task_execution(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats from kgpipe_examples.task_examples import pipe_task_remote + + in_file = tmp_path / "input.special2" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_remote.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL2)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + ) + assert pipe_task_remote.name == "pipe_task_remote" + assert report.status == "success" + assert out_file.exists() + + +def test_config_spec_execution(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats + from kgpipe_examples.task_examples import pipe_task_with_config + from kgpipe.common.model.configuration import ( + ConfigurationProfile, + ParameterBinding, + Parameter, + ParameterType, + ) + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_with_config.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + configProfile=ConfigurationProfile( + name="pipe_task_with_config_profile", + definition=pipe_task_with_config.config_spec, + bindings=[ + ParameterBinding( + parameter=Parameter( + name="some_parameter", + native_keys=["some_parameter"], + datatype=ParameterType.string, + default_value="default", + required=False, + allowed_values=[], + ), + value="some", + ) + ], + ), + ) + + assert pipe_task_with_config.name == "pipe_task_with_config" + assert report.status == "success" + assert out_file.exists() + +def test_config_profile_missing_fails(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats + from kgpipe_examples.task_examples import pipe_task_with_config + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_with_config.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + # configProfile intentionally omitted + ) -def test_pipeline_defintion(): + assert report.status == "failed" + assert report.error is not None + assert "requires a 'config' argument" in report.error + + +def test_config_profile_wrong_type_fails(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats + from kgpipe_examples.task_examples import pipe_task_with_config + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_with_config.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + configProfile="not-a-profile", + ) + + assert report.status == "failed" + assert report.error is not None + assert "expects configProfile to be a ConfigurationProfile" in report.error + + +def test_config_profile_spec_mismatch_fails(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats + from kgpipe_examples.task_examples import pipe_task_with_config + from kgpipe.common.model.configuration import ( + ConfigurationProfile, + ConfigurationDefinition, + ) + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_with_config.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + configProfile=ConfigurationProfile( + name="mismatching_profile", + definition=ConfigurationDefinition(name="different_spec_name"), + bindings=[], + ), + ) + + assert report.status == "failed" + assert report.error is not None + assert "does not match task config spec" in report.error + + +def test_config_profile_unknown_parameter_fails(tmp_path: Path): + from kgpipe_examples.config import ExtendedFormats + from kgpipe_examples.task_examples import pipe_task_with_config + from kgpipe.common.model.configuration import ( + ConfigurationProfile, + ParameterBinding, + Parameter, + ParameterType, + ) + + in_file = tmp_path / "input.special1" + out_file = tmp_path / "output.special_kg" + in_file.touch() + + report = pipe_task_with_config.run( + inputs=[Data(path=in_file, format=ExtendedFormats.SPECIAL1)], + outputs=[Data(path=out_file, format=ExtendedFormats.SPECIAL_KG)], + configProfile=ConfigurationProfile( + name="pipe_task_with_config_profile_unknown_param", + definition=pipe_task_with_config.config_spec, + bindings=[ + ParameterBinding( + parameter=Parameter( + name="other_parameter", + native_keys=["other_parameter"], + datatype=ParameterType.string, + default_value="default", + required=False, + allowed_values=[], + ), + value="some", + ) + ], + ), + ) + + assert report.status == "failed" + assert report.error is not None + assert "Unknown config parameter" in report.error + +def test_pipeline_definition_executes(): from kgpipe_examples.pipe_examples import pipe_example - \ No newline at end of file + + # Main objective: execute the pipeline example end-to-end without errors. + pipe_example() + +def test_evaluation_example(tmp_path: Path): + from kgpipe_examples.eval_examples import eval_example + eval_example(tmp_path) \ No newline at end of file diff --git a/experiments/moviekg/.gitignore b/experiments/moviekg/.gitignore new file mode 100644 index 0000000..1e82fc7 --- /dev/null +++ b/experiments/moviekg/.gitignore @@ -0,0 +1 @@ +*.yaml diff --git a/experiments/moviekg/Makefile b/experiments/moviekg/Makefile index 4eac2f6..e3fcdf7 100644 --- a/experiments/moviekg/Makefile +++ b/experiments/moviekg/Makefile @@ -1,6 +1,6 @@ .PHONY: -DATASET_URL := https://zenodo.org/record/17246358/files/inc_movie_kg_datasets.tar.gz?download=1 +ZENODO_RECORD := 17246357 BASE_DIR := ./data # === Main === @@ -16,13 +16,6 @@ pipelines: pipelines-llm: pytest -v src/moviekg/pipelines/ -k "llm" -evaluation: - pytest -v src/moviekg/evaluation/ -k "not llm" - -paper: - pytest -v src/moviekg/evaluation/test_inc_msp_evaluation.py -k concat; - pytest -v src/moviekg/paper/test_figtab.py; - # === Docker === $(BASE_DIR)/.kgpipe-docker-built: @@ -75,7 +68,9 @@ clean: $(BASE_DIR)/datasets.tar.gz: @mkdir -p $(BASE_DIR) - @cd $(BASE_DIR) && wget $(DATASET_URL) -O datasets.tar.gz + @cd $(BASE_DIR) && wget "$$(curl -sL https://zenodo.org/api/records/$(ZENODO_RECORD) \ + | jq -r '.files[] | select(.key=="inc_movie_kg_datasets.tar.gz") | .links.self')" \ + -O datasets.tar.gz $(BASE_DIR)/datasets/.extracted: $(BASE_DIR)/datasets.tar.gz @mkdir -p $(BASE_DIR)/datasets @@ -87,98 +82,40 @@ download-datasets: $(BASE_DIR)/datasets/.extracted datasets-eval: pytest -v src/moviekg/datasets/test_evaluate_film_data.py -# === SSPs === - -test-ssp-all: - time pytest -s -v src/moviekg/pipelines/test_inc_ssp.py - -eval-ssp-all: - time pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py - -test-ssp-classic: - time pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k "not llm_" - -test-ssp-llm: - time pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k "llm_" - -eval-ssp-llm: - time pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k "llm_" - # === RDF === -test-rdf-a: +test-rdf-base: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k rdf_a -eval-rdf-a: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k rdf_a - -test-rdf-b: +test-rdf-alt: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k rdf_b -eval-rdf-b: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k rdf_b - -test-rdf-c: +test-rdf-llm: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k rdf_llm -eval-rdf-c: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k rdf_llm - # === JSON === -test-json-a: +test-json-base: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k json_a -eval-json-a: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k json_a - -test-json-b: +test-json-alt: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k json_b -eval-json-b: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k json_b - -test-json-c: +test-json-llm: pytest -v -s src/moviekg/pipelines/test_inc_ssp.py -k json_llm -eval-json-c: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k json_llm - # === TEXT === -test-text-a: +test-text-base: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k text_a -eval-text-a: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k text_a - -test-text-b: +test-text-alt: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k text_b -eval-text-b: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k text_b - -test-text-c: +test-text-llm: pytest -s -v src/moviekg/pipelines/test_inc_ssp.py -k text_llm -eval-text-c: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k text_llm - # === MSPs === test-msp-all: pytest -s -v src/moviekg/pipelines/test_inc_msp.py - -eval-msp-all: - pytest -s -v src/moviekg/evaluation/test_inc_msp_evaluation.py - -eval-msp-rjt: - pytest -s -v src/moviekg/evaluation/test_inc_msp_evaluation.py -k rdf-json-text - -# === Paper === - -concatenate-metrics: - pytest -s -v src/moviekg/evaluation/test_inc_ssp_evaluation.py -k test_concatenate_long_table_rows - -paper-figtab: - pytest -s -v src/moviekg/paper/test_figtab.py diff --git a/experiments/moviekg/README.md b/experiments/moviekg/README.md index d713b49..4363cab 100644 --- a/experiments/moviekg/README.md +++ b/experiments/moviekg/README.md @@ -1,55 +1,64 @@ -# Inc Movie KG +# MovieKG (KGpipe pipelines) -Documentation and experiment code for incremental KG generation and evaluation. +This directory contains **MovieKG pipeline definitions and execution helpers** for running incremental KG construction +pipelines with KGpipe. +Evaluation of the produced KGs is now handled in the **KGI-Bench** repository (Movie benchmark). See: +- [KGI-Bench](https://github.com/ScaDS/KGI-Bench) +- [KGI-Bench-Movie](https://github.com/ScaDS/KGI-Bench/tree/main/benchmarks/kgi-bench-movie) +- [KGI-Bench/docs/cli.md](https://scads.github.io/KGI-Bench/#cli) (includes `kgibench evaluate --benchmark movie ...`) -# Dataset Overview +## What’s in here -- 📊 [Benchmark Datasets](https://doi.org/10.5281/zenodo.17246357) +- **Pipeline catalog**: `pipeline.conf` (pipeline variants and their task sequences) +- **Execution helpers**: `src/moviekg/pipelines/` (pytest-driven runners + helpers) +- **Environment templates**: `env`, `docker_env` (copy to `.env` / `docker.env` for local configuration) -A benchmark derived from Wikipedia and DBpedia in the movie domain covering the three entities: `Film,Person,Company` described and connected by 23(+2) attributes. -The dataset consists of the following. +## Running pipelines (local) -Four Splits and three different formats: -- RDF: RDF from DBpedia, in the three namespaces for seed, reference and source data -- JSON: json files built from the tree like subgraphs of each film -- TEXT: abstract text of each film entity from wikipedia +From `experiments/moviekg/`: -Suplmenetary data: -- reference entity matches: for entity matching eval (rdf, json) -- reference entity links: for entity linking eval (text) -- provannce mappings: for tracing json entity mappings -- refernce key mappings: for tracing json to rdf schema matching - -Available in three sizes: -- small 100 films: for development -- medium 1,000 films: for testing -- large 10,000 films: for benchmarking +```bash +cp env .env +make pipelines +``` -# Running +LLM variants: -It is possible to execute the experiemnt in a docker environment. -Adapt the `docker.env` file -and choose the dataset size (small, medium, large) +```bash +make pipelines-llm +``` -> LLM tasks are disabled by default to enable them add -> make pipelines-llm as task in [moviekg_docker.sh](../../scripts/moviekg_docker.sh) +Per-pipeline targets are also available (see `Makefile`), e.g.: -Prepare -``` -make setup_docker +```bash +make test-json-base +make test-rdf-base +make test-msp-all ``` -Execution of dataset stats, pipelines, evalaution, and paper content generation -``` +## Running pipelines (Docker workflow) + +This uses the `Makefile` targets to build images + start services and run pipelines inside Docker. + +```bash +cp docker_env docker.env +make setup_docker make run_docker_small ``` -For more detailed information see also [reproduce.md](../../docs/reproduce.md) or [docs](../../docs/) +> Note: LLM pipelines are typically disabled by default in Docker orchestration; enable them by adding the +> `pipelines-llm` step to the orchestration script used in your setup. + +## Dataset overview (high level) -# Directory Structure +- Dataset release: `https://doi.org/10.5281/zenodo.17246357` +- Sizes: `small` (100 films), `medium` (1k), `large` (10k) +- Formats per split: RDF, JSON, TEXT (incremental splits with seed/reference/source) -## Input Structure +## Directory structure + +### Input structure (example) ``` ├── film_100 @@ -84,12 +93,16 @@ For more detailed information see also [reproduce.md](../../docs/reproduce.md) o ├── film_1k[... trunc] ``` -## Output Structure +### Output structure (example) + +Pipeline outputs are written under `$OUTPUT_DIR/$DATASET_SELECT//stage_/` and include: +- `result.nt` (and optionally `result_eval.nt`) +- `exec-plan.json`, `exec-report.json` +- `tmp/` intermediate artifacts ``` ├── small -│   ├── all_metrics.csv -│   ├── json_a +│   ├── json_base │   │   ├── stage_1 │   │   │   ├── exec-plan.json │   │   │   ├── exec-report.json @@ -105,9 +118,6 @@ For more detailed information see also [reproduce.md](../../docs/reproduce.md) o │   │   ├── exec-report.json │   │   ├── result.nt │   │   └── tmp/ -│ ├── json_b[... trunc] -│   ├── paper -│   │   ├── test_fig....png -│   │   └── test_tab.....png +│ ├── json_alt[... trunc] └── medium[... trunc] -``` \ No newline at end of file +``` diff --git a/experiments/moviekg/env b/experiments/moviekg/env index 7923c7e..cb9e376 100644 --- a/experiments/moviekg/env +++ b/experiments/moviekg/env @@ -1,12 +1,12 @@ PIPELINE_CONFIG=pipeline.conf -DATASET_SELECT=medium +DATASET_SELECT=small -ONTOLOGY_PATH=/home/marvin/project/KGpipe/experiments/moviekg/movie-ontology.ttl -OUTPUT_DIR=/home/marvin/project/data/out/ +ONTOLOGY_PATH=./data/datasets/film_10k/ontology.ttl +OUTPUT_DIR=./data/results/ -DATASET_SMALL=/home/marvin/project/data/final/film_100 -DATASET_MEDIUM=/home/marvin/project/data/final/film_1k -DATASET_LARGE=/home/marvin/project/data/final/film_10k +DATASET_SMALL=./data/datasets/film_100 +DATASET_MEDIUM=./data/datasets/film_1k +DATASET_LARGE=./data/datasets/film_10k EMBEDDER=sentence-transformer DBPEDIA_ANNOTATE_URL='http://localhost:2222/rest/annotate' @@ -18,4 +18,3 @@ OLLAMA_TOKEN= OPENAI_TOKEN= LLM_ENDPOINT_URL= - diff --git a/experiments/moviekg/pipeline.conf b/experiments/moviekg/pipeline.conf index 53fe6fc..5c6964e 100644 --- a/experiments/moviekg/pipeline.conf +++ b/experiments/moviekg/pipeline.conf @@ -1,10 +1,6 @@ -# Pipeline Defintion +# Pipeline Defintions -# ======== -# RDF SSPs -# ======== - -rdf_a: +rdf_base: description: "Align source RDF with target KG" config: ENTITY_MATCHING_THRESHOLD: "0.99" @@ -16,23 +12,31 @@ rdf_a: - paris_exchange # 3 Fuse matched RDF (threshold 0.5) - fusion_first_value + # 4 Infer types / align to ontology - type_inference_ontology_simple -rdf_b: +rdf_alt: description: "Align source RDF with target KG with a tabular matching approach" config: ENTITY_MATCHING_THRESHOLD: "0.5" RELATION_MATCHING_THRESHOLD: "0.1" tasks: + # 1 Transform RDF to tabular representation - transform2_rdf_to_csv_v2 + # 2 Match entities (tabular) - pyjedai_entity_matching_v2 + # 3 Keep best match per entity - reduce_to_best_match_per_entity + # 4 Match relations/schemas (tabular) - valentine_csv_matching_v2 + # 5 Aggregate entity + relation matches - aggregate_2matches + # 6 Fuse matched RDF - fusion_first_value + # 7 Infer types / align to ontology - type_inference_ontology_simple -rdf_llm_schema_align_v1: +rdf_llm: description: "Align relations of source RDF with target KG using LLM" config: ENTITY_MATCHING_THRESHOLD: "0.99" @@ -42,96 +46,122 @@ rdf_llm_schema_align_v1: # 1 Use LLM to match relations - llm_task_rdf_ontology_matching_v1 # results in er.json # 2 Map source KG relations to matching target KG relations - # - map_kg_alignments - map_er_match_relations # 3 Match entities with paris - paris_entity_matching # 4 Exchange matched RDF - paris_exchange + # 5 Aggregate entity + relation matches - aggregate_2matches - # 5 Fuse matched RDF maybe only entities + # 6 Fuse matched RDF (maybe only entities) - fusion_first_value + # 7 Infer types / align to ontology - type_inference_ontology_simple -# ========= -# JSON SSPs -# ========= - -json_a: +json_base: description: "Construct intermediate RDF from JSON" tasks: # 1 Nested tree Json to generic RDF graph - construct_rdf_from_json3 # 2 Match RDF graph with seed - paris_entity_matching - # 3 exchange + # 3 Exchange matches - paris_exchange # 4 Fuse matched RDF (threshold 0.5) - fusion_first_value + # 5 Infer types / align to ontology - type_inference_ontology_simple -json_b: +json_alt: description: "Link JSON objects to target KG" tasks: - # construct TE_Document from JSON + # 1 Construct TE_Document from JSON - construct_linkedrdf_from_json_v3 # extract_json.py - # 4 Fuse matched RDF (threshold 0.5) + # 2 Select / fuse values - select_first_value + # 3 Infer types / align to ontology - type_inference_ontology_simple -json_llm_mapping_v1: +json_llm: description: "Align JSON path to target KG (ontology + sample KG)" tasks: + # 1 Use LLM to map JSON and construct intermediate RDF - llm_task_map_and_construct + # 2 Aggregate intermediate RDF outputs - aggregate_rdf_files + # 3 Match entities with paris - paris_entity_matching + # 4 Exchange matched RDF - paris_exchange + # 5 Fuse matched RDF - fusion_first_value + # 6 Infer types / align to ontology - type_inference_ontology_simple -# ========= -# Text SSPs -# ========= - -text_a: +text_base: description: "Use spoltight build RDF stagging Graph and apply Paris matching" tasks: + # 1 Extract triples with OpenIE - corenlp_openie_extraction + # 2 Convert extraction output to TE JSON - corenlp_exchange + # 3 Link relations (label+alias embedding) - label_alias_embedding_rl + # 4 Link entities with DBpedia Spotlight - dbpedia_spotlight_ner_nel + # 5 Convert Spotlight output to TE JSON - dbpedia_spotlight_exchange + # 6 Aggregate TE JSON artifacts - aggregate3_te_json + # 7 Construct RDF staging graph (mappings only) - construct_rdf_from_te_json_mappings_only + # 8 Match entities with paris - paris_entity_matching + # 9 Exchange matched RDF - paris_exchange + # 10 Fuse matched RDF - fusion_first_value + # 11 Infer types / align to ontology - type_inference_ontology_simple - -text_b: +text_alt: description: "(semi expensive) Use mini transformer to link entities and relations (label+alias)" tasks: + # 1 Extract triples with OpenIE - corenlp_openie_extraction # ("Berlin", "is a", "city") + # 2 Convert extraction output to TE JSON - corenlp_exchange + # 3 Link entities (label+alias embedding) - label_alias_embedding_el # ("Berlin" -> http://dbpedia.org/resource/Berlin) + # 4 Link relations (label+alias embedding) - label_alias_embedding_rl # ("is a" -> "rdf:type") + # 5 Aggregate TE JSON artifacts - aggregate3_te_json + # 6 Construct RDF staging graph - construct_rdf_from_te_json + # 7 Select / fuse values - select_first_value + # 8 Infer types / align to ontology - type_inference_ontology_simple -text_llm_triple_extract_v1: +text_llm: description: "Extract RDF from TEXT using LLM" config: LLM_MODEL: "gpt-5-mini" tasks: + # 1 Extract triples using LLM - llm_task_text_triple_extract_v1 + # 2 Link entities (label+alias embedding) - label_alias_embedding_el # ("Berlin" -> http://dbpedia.org/resource/Berlin) + # 3 Link relations (label+alias embedding) - label_alias_embedding_rl # ("is a" -> "rdf:type") + # 4 Aggregate TE JSON artifacts - aggregate3_te_json + # 5 Construct RDF staging graph - construct_rdf_from_te_json + # 6 Select / fuse values - select_first_value + # 7 Infer types / align to ontology - type_inference_ontology_simple # - type_inference_ontology_simple TODO why was this commented diff --git a/experiments/moviekg/src/moviekg/config.py b/experiments/moviekg/src/moviekg/config.py index 4460916..3727e00 100644 --- a/experiments/moviekg/src/moviekg/config.py +++ b/experiments/moviekg/src/moviekg/config.py @@ -43,22 +43,16 @@ pipeline_types = { - "rdf_a": "rdf", - "rdf_b": "rdf", - "text_a": "text", - "text_b": "text", - "json_a": "json", - "json_b": "json", + "rdf_base": "rdf", + "rdf_alt": "rdf", + "text_base": "text", + "text_alt": "text", + "json_base": "json", + "json_alt": "json", } llm_pipeline_types = { - "json_llm_mapping_v1": "json", - "rdf_llm_schema_align_v1": "rdf", - "text_llm_triple_extract_v1": "text", + "json_llm": "json", + "rdf_llm": "rdf", + "text_llm": "text", } - -ssp = { - "rdf": "rdf_a", - "json": "json_b", - "text": "text_a" -} \ No newline at end of file diff --git a/experiments/moviekg/src/moviekg/datasets/tmp_remove_seeds.py b/experiments/moviekg/src/moviekg/datasets/tmp_remove_seeds.py new file mode 100644 index 0000000..ea3dded --- /dev/null +++ b/experiments/moviekg/src/moviekg/datasets/tmp_remove_seeds.py @@ -0,0 +1,33 @@ +from moviekg.evaluation.test_eval_refactor import KgBenchData + +""" +for every verified_seed remove in the bench data remove the seed entities and store as verified_entities_no_seed.csv +""" + +import pandas as pd +from pathlib import Path + +bench_data = KgBenchData.from_path(Path("/home/marvin/phd/data/moviekg/datasets/film_1k")) + +for i in range(1, 4): + seed = bench_data.dataset.splits[f"split_{0}"].kg_reference.meta.entities.file + current = bench_data.dataset.splits[f"split_{i}"].kg_reference.meta.entities.file + current_path = bench_data.dataset.splits[f"split_{i}"].kg_reference.meta.entities.file + current_new = current_path.with_name(f"{current_path.stem}_no_seed{current_path.suffix}") + + # remove all lines from current that are in seed and save to new file + with open(current, "r") as f: + current_lines = f.readlines() + with open(seed, "r") as f: + seed_lines = f.readlines() + with open(current_new, "w") as f: + if not current_lines: + continue + + # Preserve header (assumes first line is the CSV header) + f.write(current_lines[0]) + + seed_set = set(seed_lines[1:] if seed_lines else []) + for line in current_lines[1:]: + if line not in seed_set: + f.write(line) diff --git a/experiments/moviekg/src/moviekg/evaluation/helpers.py b/experiments/moviekg/src/moviekg/evaluation/helpers.py deleted file mode 100644 index 03ce1d0..0000000 --- a/experiments/moviekg/src/moviekg/evaluation/helpers.py +++ /dev/null @@ -1,201 +0,0 @@ -import json -import tempfile -import re -import shutil -from typing import List, Dict, Tuple -from pathlib import Path -from rdflib import Graph - -from kgpipe.common.models import KG, DataFormat -from kgpipe.evaluation.aspects import reference, semantic, statistical -from kgpipe.evaluation.aspects.reference import ReferenceConfig -from kgpipe.evaluation.base import MetricResult -from kgcore.model.ontology import OntologyUtil - -from moviekg.datasets.pipe_out import StageOut -from moviekg.config import dataset - -ontology_graph = Graph() -if dataset.ontology is None: - raise ValueError("No ontology found") -ontology_graph.parse(dataset.ontology.as_posix()) - -def show_ontology(): - if dataset.ontology is None: - raise ValueError("No ontology found") - ontology = OntologyUtil.load_ontology_from_file(dataset.ontology) - - for class_ in ontology.classes: - print(f"{class_.uri} {class_.label}") - # print(f"{class_.alias} {class_.description}") - print(f"{class_.equivalent}") - print(f"{class_.disjointWith}") - print("-" * 100) - - for property in ontology.properties: - print(f"{property.uri} {property.type} {property.label}") - # print(f"{property.alias} {property.description}") - print(f"{property.domain.uri} {property.range.uri} {property.equivalent}") - print(f"{property.min_cardinality} {property.max_cardinality}") - print("-" * 100) - -show_ontology() - - -def print_long_table_rows(rows: List[dict]): - """ - with correct margin and alignment - """ - max_aspect_length = max(len(row["aspect"]) for row in rows) - max_metric_name_length = max(len(row["metric"]) for row in rows) - max_value_length = max(len(str(row["value"])) for row in rows) - max_normalized_length = max(len(str(row["normalized"])) for row in rows) - max_duration_length = max(len(str(row["duration"])) for row in rows) - - print(f"{'Aspect':<{max_aspect_length}} | {'Metric':<{max_metric_name_length}} | {'Value':<{max_value_length}} | {'Normalized':<{max_normalized_length}} | {'Duration':<{max_duration_length}}") - print("-" * (max_aspect_length + max_metric_name_length + max_value_length + max_normalized_length + max_duration_length + 6)) - for row in rows: - print(f"{row['aspect']:<{max_aspect_length}} | {row['metric']:<{max_metric_name_length}} | {row['value']:<{max_value_length}} | {row['normalized']:<{max_normalized_length}} | {row['duration']:<{max_duration_length}}") - -def metrics_to_long_table_rows(metrics: List[MetricResult], pipeline_name: str, stage_name: str) -> List[dict]: - rows = [] - for metric in metrics: - rows.append({ - "pipeline": pipeline_name, - "stage": stage_name, - "aspect": metric.aspect.value, - "metric": metric.name, - "value": metric.value, - "normalized": metric.normalized_score, - "duration": metric.duration, - "details": json.dumps(metric.details, default=str) - }) - return rows - - - -def get_reference_config(stage: StageOut, is_ssp: bool) -> ReferenceConfig: - - # this is a pipeline name based hack to get the source type and source split id - def get_split_id_and_source_type(stage: StageOut, is_ssp: bool = False) -> Tuple[int, str]: - # stage_name is like "stage_1" - split_id = int(stage.stage_name.split("_")[1]) - pipeline_name = stage.root.parent.name - source_ord = pipeline_name.split("_") - - if len(source_ord) != 3 or is_ssp: - source_type = source_ord[0] - else: - source_type = source_ord[split_id-1] - return split_id, source_type - - split_id, source_type = get_split_id_and_source_type(stage) - - meta = dataset.splits[f"split_{split_id}"].sources[source_type].meta - verified_source_entities_path = dataset.splits[f"split_{split_id}"].kg_seed.root / "meta/verified_entities.csv" - verified_source_matches_path = meta.root / "verified_matches.csv" - - - kg_reference = dataset.splits[f"split_{split_id}"].kg_reference - if kg_reference is None: - raise ValueError(f"No reference KG found for split {split_id} and source type {source_type}") - reference_path = kg_reference.root / "data_agg.nt" - - kg_seed = dataset.splits[f"split_0"].kg_seed - if kg_seed is None: - raise ValueError(f"No seed KG found for split {0}") - seed_path = kg_seed.root / "data.nt" - - ENTITY_MATCH_THRESHOLD_MAP = { - "json_a": 0.99, - "rdf_a": 0.99, - "rdf_b": 0.5, - "rdf_c": 0.99, - "rdf_llm_schema_align_v1": 0.99 - } - - RELATION_MATCH_THRESHOLD_MAP = { - "json_a": 0.5, - "rdf_a": 0.5, - "rdf_b": 0.1, - "rdf_c": 0.5, - "rdf_llm_schema_align_v1": 0.5 - } - - return ReferenceConfig( - name="reference", - GT_MATCHES=verified_source_matches_path, - GT_MATCHES_TARGET_DATASET=dataset.splits[f"split_{0}"].root.name+"/kg/seed", - RELATION_MATCH_THRESHOLD=RELATION_MATCH_THRESHOLD_MAP.get(stage.root.parent.name, 0.5), - ENTITY_MATCH_THRESHOLD=ENTITY_MATCH_THRESHOLD_MAP.get(stage.root.parent.name, 0.99), - VERIFIED_SOURCE_ENTITIES=verified_source_entities_path, - REFERENCE_KG_PATH=reference_path, - SEED_KG_PATH=seed_path, - TE_LINK_THRESHOLD=0.5, - source_meta=meta, - dataset=dataset, - JSON_EXPECTED_DIR="/home/marvin/project/data/work/json", #TODO cleanup - JSON_EXPECTED_RELATION_FILE="/home/marvin/project/data/final/film_10k/split_0/sources/json/meta/verified_relation_matches.json" # TODO cleanup - ) - -from kgpipe.evaluation.base import MetricResult, EvaluationAspect - -def add_duration_metrics(stage: StageOut) -> MetricResult: - - try: - duration = stage.report.duration - return MetricResult( - aspect=EvaluationAspect.STATISTICAL, - name="duration", - value=duration, - normalized_score=0, - details={ - "duration": duration - } - ) - - except Exception as e: - return MetricResult( - aspect=EvaluationAspect.STATISTICAL, - name="duration", - value=0, - normalized_score=0, - details={"error": "No duration found"} - ) - - - -def evaluate_stage(stage: StageOut, is_ssp: bool) -> List[MetricResult]: - result_path = stage.resultKG - if result_path is None: - return [] - - result_kg = KG(id=f"result_{stage.stage_name}", name=f"result_{stage.stage_name}", path=result_path, format=DataFormat.RDF_NTRIPLES,plan=stage.plan) - - result_kg.set_ontology_graph(ontology_graph) - - stat_eval = statistical.StatisticalEvaluator() - ref_eval = reference.ReferenceEvaluator() - sem_eval = semantic.SemanticEvaluator() - - stats_aspect_result = stat_eval.evaluate(result_kg) - ref_aspect_result = ref_eval.evaluate(result_kg, config=get_reference_config(stage, is_ssp)) - sem_aspect_result = sem_eval.evaluate(result_kg) - - metrics = [] - metrics = stats_aspect_result.metrics + ref_aspect_result.metrics + sem_aspect_result.metrics - # metrics = sem_aspect_result.metrics - metrics.append(add_duration_metrics(stage)) - # metrics = ref_aspect_result.metrics - - return metrics - -def replace_with_dict(infile: str, mapping: dict[str, str]) -> None: - with open(infile, encoding="utf-8") as f, \ - tempfile.NamedTemporaryFile("w", delete=False, encoding="utf-8") as tmp: - for line in f: - for key, val in mapping.items(): - line = re.sub(re.escape(key), val, line) - tmp.write(line) - tmp_path = tmp.name - shutil.move(tmp_path, infile) diff --git a/experiments/moviekg/src/moviekg/evaluation/test_inc_msp_evaluation.py b/experiments/moviekg/src/moviekg/evaluation/test_inc_msp_evaluation.py deleted file mode 100644 index fa186d2..0000000 --- a/experiments/moviekg/src/moviekg/evaluation/test_inc_msp_evaluation.py +++ /dev/null @@ -1,61 +0,0 @@ -import pandas as pd -import pytest -import os -from typing import Sequence -from _pytest.compat import NotSetType -from itertools import permutations - -from moviekg.datasets.pipe_out import load_pipe_out -from moviekg.evaluation.helpers import evaluate_stage, metrics_to_long_table_rows -from moviekg.pipelines.test_inc_msp import ssp, idfn - -from moviekg.config import OUTPUT_ROOT - -@pytest.mark.parametrize( - "source_1, source_2, source_3", - permutations(list[str](ssp.keys()), 3), - ids=idfn -) -def test_inc_ssp_evaluation(source_1, source_2, source_3): - - output_dir = OUTPUT_ROOT / f"{source_1}_{source_2}_{source_3}" - - pipeline_name = f"{source_1}_{source_2}_{source_3}" - - print("-" * 100) - print(f"Evaluating {source_1}, {source_2}, {source_3}") - print("-" * 100) - - if not output_dir.exists(): - pytest.skip(f"Pipeline output directory {output_dir} not found") - - pipe_out = load_pipe_out(output_dir) - - rows = [] - - for stage in pipe_out.stages: - print("-" * 100) - print(f"{pipeline_name} - Stage: {stage.stage_name}") - print("-" * 100) - - metrics = evaluate_stage(stage, is_ssp=False) - rows.extend(metrics_to_long_table_rows(metrics, pipeline_name, stage.stage_name)) - # break # TODO remove - - metrics_df = pd.DataFrame(rows) - metrics_df.to_csv(OUTPUT_ROOT / f"{pipeline_name}_metrics.csv", index=False) - print("saved metrics to", OUTPUT_ROOT / f"{pipeline_name}_metrics.csv") - -def test_concatenate_long_table_rows(): - # glob - rows = [] - for file in OUTPUT_ROOT.glob("*_metrics.csv"): - if file.name == "all_metrics.csv": - continue - if os.path.getsize(file) < 3: - continue - df = pd.read_csv(file) - rows.extend(df.to_dict(orient="records")) - - metrics_df = pd.DataFrame(rows) - metrics_df.to_csv(OUTPUT_ROOT / "all_metrics.csv", index=False) diff --git a/experiments/moviekg/src/moviekg/evaluation/test_inc_ssp_evaluation.py b/experiments/moviekg/src/moviekg/evaluation/test_inc_ssp_evaluation.py deleted file mode 100644 index 2e62f54..0000000 --- a/experiments/moviekg/src/moviekg/evaluation/test_inc_ssp_evaluation.py +++ /dev/null @@ -1,58 +0,0 @@ -import pytest -import pandas as pd -import os -from pathlib import Path - -from moviekg.datasets.pipe_out import load_pipe_out -from moviekg.evaluation.helpers import evaluate_stage, metrics_to_long_table_rows, print_long_table_rows -from moviekg.pipelines.test_inc_ssp import pipeline_types, llm_pipeline_types - -from moviekg.config import OUTPUT_ROOT - -@pytest.mark.parametrize( - "pipeline_name", - list[str](pipeline_types.keys()) + list[str](llm_pipeline_types.keys()) -) -def test_inc_ssp_evaluation(pipeline_name): - - output_dir = OUTPUT_ROOT / pipeline_name - - print("-" * 100) - print(f"Evaluating {pipeline_name}") - print("-" * 100) - - if not output_dir.exists(): - pytest.skip(f"Pipeline output directory {output_dir} not found") - - pipe_out = load_pipe_out(output_dir) - - rows = [] - - for stage in pipe_out.stages: - print("-" * 100) - print(f"{pipeline_name} - Stage: {stage.stage_name}") - print("-" * 100) - - metrics = evaluate_stage(stage, is_ssp=True) - new_rows = metrics_to_long_table_rows(metrics, pipeline_name, stage.stage_name) - print_long_table_rows(new_rows) - rows.extend(new_rows) - # break # TODO remove this - - metrics_df = pd.DataFrame(rows) - metrics_df.to_csv(OUTPUT_ROOT / f"{pipeline_name}_metrics.csv", index=False) - print("saved metrics to", OUTPUT_ROOT / f"{pipeline_name}_metrics.csv") - -def test_concatenate_long_table_rows(): - # glob - rows = [] - for file in OUTPUT_ROOT.glob("*_metrics.csv"): - if file.name == "all_metrics.csv": - continue - if os.path.getsize(file) < 3: - continue - df = pd.read_csv(file) - rows.extend(df.to_dict(orient="records")) - - metrics_df = pd.DataFrame(rows) - metrics_df.to_csv(OUTPUT_ROOT / "all_metrics.csv", index=False) diff --git a/experiments/moviekg/src/moviekg/evaluation/test_ref_dev.py b/experiments/moviekg/src/moviekg/evaluation/test_ref_dev.py deleted file mode 100644 index 1a4d17b..0000000 --- a/experiments/moviekg/src/moviekg/evaluation/test_ref_dev.py +++ /dev/null @@ -1,247 +0,0 @@ -# from pathlib import Path -# import numpy as np -# from sentence_transformers import SentenceTransformer -# from rdflib import Graph, URIRef, Literal, RDF, RDFS, XSD -# import re -# from tqdm import tqdm - -# def integrated_entities(path_actual_kg, path_expected_kg): -# pass - -# SOFT_ENTITY_THRESHOLD = 0.75 -# SOFT_VALUES_THRESHOLD = 0.75 - -# def encode(values, model, desc: str): -# embeddings = [] -# for i in tqdm(range(0, len(values), 64), desc=desc): -# batch = values[i:i+64] -# batch_emb = model.encode(batch, show_progress_bar=False) -# embeddings.append(batch_emb) -# return np.vstack(embeddings) - -# def graph_fact_alginment(ga: Graph, ge: Graph): -# te = [ str(s)+str(p)+str(o) for s, p, o in ge ] -# ta = [ str(s)+str(p)+str(o) for s, p, o in ga ] - -# tp = len(set(ta) & set(te)) -# fp = len(set(ta) - set(te)) -# fn = len(set(te) - set(ta)) - -# print(f"TP: {tp}, FP: {fp}, FN: {fn}") -# print(f"Precision: {tp / (tp + fp)}") -# print(f"Recall: {tp / (tp + fn)}") -# print(f"F1: {2 * tp / (2 * tp + fp + fn)}") - -# def clean_label(label: str): -# # remove all non-alphanumeric characters -# cleaned_label = label.replace("_", " ") -# # remove parenthesis text -# cleaned_label = re.sub(r'\([^)]*\)', '', cleaned_label) -# return cleaned_label.strip() - - -# def graph_match_labels_soft(ga: Graph, ge: Graph, model: SentenceTransformer): -# actual_uri_to_abels = {} -# expected_uri_to_abels = {} - -# for s, _, o in ga.triples((None, RDFS.label, None)): -# actual_uri_to_abels[str(s)] = clean_label(str(o)) - -# for s, _, o in ge.triples((None, RDFS.label, None)): -# expected_uri_to_abels[str(s)]= clean_label(str(o)) - -# actual_embeddings = encode(list(actual_uri_to_abels.values()), model, "Encoding actual labels") -# expected_embeddings = encode(list(expected_uri_to_abels.values()), model, "Encoding expected labels") - -# cosine_scores = np.dot(actual_embeddings, expected_embeddings.T) - -# actual_uri_keys = list(actual_uri_to_abels.keys()) -# expected_uri_keys = list(expected_uri_to_abels.keys()) - -# # get best match expected uri for each actual uri - -# uri_mappings = {} - -# best_matches = [] -# for i in range(len(actual_uri_keys)): -# best_match = expected_uri_keys[np.argmax(cosine_scores[i])] -# best_score = cosine_scores[i][np.argmax(cosine_scores[i])] -# best_matches.append((best_match, best_score)) - -# for i in range(len(best_matches)): -# if best_matches[i][1] > SOFT_ENTITY_THRESHOLD: -# # la = actual_uri_to_abels[actual_uri_keys[i]].replace(" ", "_") -# # le = expected_uri_to_abels[best_matches[i][0]].replace(" ", "_") -# uri_actual = actual_uri_keys[i] -# uri_expected = best_matches[i][0] -# uri_mappings[uri_actual] = uri_expected - -# return uri_mappings - -# def graph_fact_alginment_soft_entities(ga: Graph, ge: Graph, model: SentenceTransformer): -# uri_mappings = graph_match_labels_soft(ga, ge, model) - -# ga_mapped = Graph() -# for s, p, o in ga: -# if str(s) in uri_mappings: -# s = URIRef(uri_mappings[str(s)]) -# if isinstance(o, URIRef) and str(o) in uri_mappings: -# o = URIRef(uri_mappings[str(o)]) -# ga_mapped.add((s, p, o)) - -# graph_fact_alginment(ga_mapped, ge) - -# # TODO rdf:type is removed for tp calculation -# def graph_fact_alginment_soft_entities_values(ga: Graph, ge: Graph, model: SentenceTransformer): -# uri_mappings = graph_match_labels_soft(ga, ge, model) - -# def get_label(o: URIRef, graph: Graph): -# labels = [str(l) for l in graph.objects(o, RDFS.label)] -# if len(labels) == 0: -# return [] -# else: -# return [clean_label(l) for l in labels] - -# ga_mapped = Graph() -# for s, p, o in ga: -# if str(s) in uri_mappings: -# s = URIRef(uri_mappings[str(s)]) -# if isinstance(o, URIRef): # and p != RDF.type -# for label in get_label(o, ga): -# ga_mapped.add((s, p, Literal(label))) -# else: -# ga_mapped.add((s, p, o)) - -# ge_mapped = Graph() -# for s, p, o in ge: -# if isinstance(o, URIRef): # and p != RDF.type -# for label in get_label(o, ge): -# ge_mapped.add((s, p, Literal(label))) -# else: -# ge_mapped.add((s, p, o)) - -# # encode all values -# vas = list(set([str(o) for _, _, o in ga_mapped if not isinstance(o, URIRef)])) -# ves = list(set([str(o) for _, _, o in ge_mapped if not isinstance(o, URIRef)])) - -# va_embeddings = encode(vas, model, "Encoding actual values") -# ve_embeddings = encode(ves, model, "Encoding expected values") - -# v2e_actual = {} -# v2e_expected = {} - -# for idx, v in enumerate(vas): -# v2e_actual[v] = va_embeddings[idx] - -# for idx, v in enumerate(ves): -# v2e_expected[v] = ve_embeddings[idx] - -# tp = 0 -# fp = 0 -# fn = 0 - -# sp_actual = set() - -# # for each (s, p, o) in ga_mapped check if there is a matching value for the same (s, p) in ge -# for s, p in ga_mapped.subject_predicates(unique=True): -# sp_actual.add((s, p)) -# _vas = [str(o) for o in ga_mapped.objects(s, p)] -# _ves = [str(o) for o in ge_mapped.objects(s, p)] -# _vas_embeddings = np.array([v2e_actual[v] for v in _vas]) -# _ves_embeddings = np.array([v2e_expected[v] for v in _ves]) - -# if len(_vas_embeddings) == 0 or len(_ves_embeddings) == 0: -# continue -# cosine_scores = np.dot(_vas_embeddings, _ves_embeddings.T) # (len(_vas_embeddings), len(_ves_embeddings)) - -# for idx in range(len(_vas)): -# best_match = _ves[np.argmax(cosine_scores[idx])] -# best_score = cosine_scores[idx][np.argmax(cosine_scores[idx])] -# if best_score > SOFT_VALUES_THRESHOLD: -# actual_value = _vas[idx] -# reference_value = best_match -# tp += 1 -# # if actual_value == reference_value: -# # # print(f"Found matching value for {s} {p} {actual_value}") -# # pass -# # else: -# # print(f"Found matching value for {s} {p} {actual_value} but not exact reference {reference_value}") -# # print(f"Value actual: {_vas[idx]}, {best_match}, {best_score}") -# # print(f"Value expected: {_ves[np.argmax(cosine_scores[idx])]}") -# else: -# fp += 1 -# # print(f"No matching value for {s} {p} {_vas[idx]} from references {_ves}") - -# sp_expected = set([(s, p) for s, p in ge_mapped.subject_predicates(unique=True)]) -# missing_sp = sp_expected - sp_actual -# for s, p in missing_sp: -# for _ in ge_mapped.triples((s, p, None)): -# fn += 1 - -# print(f"TP: {tp}, FP: {fp}, FN: {fn}") -# print(f"Precision: {tp / (tp + fp)}") -# print(f"Recall: {tp / (tp + fn)}") -# print(f"F1: {2 * tp / (2 * tp + fp + fn)}") - -# def reference_alignment(path_actual_kg: Path, path_expected_kg: Path): -# ga = Graph() -# ga.parse(path_actual_kg) - -# ge = Graph() -# ge.parse(path_expected_kg) - -# graph_fact_alginment(ga, ge) - -# def reference_alignment_soft_entities(path_actual_kg: Path, path_expected_kg: Path): - -# model = SentenceTransformer("all-MiniLM-L6-v2") -# model.to("cuda") - -# ga = Graph() -# ga.parse(path_actual_kg) - -# ge = Graph() -# ge.parse(path_expected_kg) - -# graph_fact_alginment_soft_entities(ga, ge, model) - -# def reference_alignment_soft_entities_values(path_actual_kg: Path, path_expected_kg: Path): - -# model = SentenceTransformer("all-MiniLM-L6-v2") -# model.to("cuda") - -# ga = Graph() -# ga.parse(path_actual_kg) - -# ge = Graph() -# ge.parse(path_expected_kg) - -# graph_fact_alginment_soft_entities_values(ga, ge, model) - -# def test_integrated_verified_source_entities(): -# print("Integrated verified source entities") -# path_actual_kg = Path("/home/marvin/project/code/experiments/out_film_100/rdf_a/stage_1/result.nt") -# path_expected_kg = Path("/home/marvin/project/data/final/film_100/split_3/kg/reference/data_agg.nt") -# integrated_entities(path_actual_kg, path_expected_kg) - -# def test_reference_alignment(): -# print("Reference alignment") -# path_actual_kg = Path("/home/marvin/project/code/experiments/out_film_100/rdf_a/stage_1/result.nt") -# path_expected_kg = Path("/home/marvin/project/data/final/film_100/split_3/kg/reference/data_agg.nt") -# reference_alignment(path_actual_kg, path_expected_kg) - -# def test_reference_alignment_soft(): -# print("Reference alignment soft") -# path_actual_kg = Path("/home/marvin/project/code/experiments/out_film_100/text_b/stage_1/result.nt") -# path_expected_kg = Path("/home/marvin/project/data/final/film_100/split_3/kg/reference/data_agg.nt") -# reference_alignment_soft_entities(path_actual_kg, path_expected_kg) - -# def test_reference_alignment_soft_entities_values(): -# print("Reference alignment soft entities values") -# path_actual_kg = Path("/home/marvin/project/code/experiments/out_film_100/rdf_a/stage_1/result.nt") -# path_expected_kg = Path("/home/marvin/project/data/final/film_100/split_3/kg/reference/data_agg.nt") -# reference_alignment_soft_entities_values(path_actual_kg, path_expected_kg) - -# if __name__ == "__main__": -# test_integrated_verified_source_entities() -# test_reference_alignment() \ No newline at end of file diff --git a/experiments/moviekg/src/moviekg/paper/config.py b/experiments/moviekg/src/moviekg/paper/config.py deleted file mode 100644 index 507a5c0..0000000 --- a/experiments/moviekg/src/moviekg/paper/config.py +++ /dev/null @@ -1,135 +0,0 @@ - -HEADERS = ["pipeline", "stage", "aspect", "metric", "value", "normalized", "duration", "details"] - -# Only keep these classes and aggregate the rest into "Other" -main_classes = [ - "http://kg.org/ontology/Company", - "http://kg.org/ontology/Person", - "http://kg.org/ontology/Film" -] - -name_mapping = { - "rdf_a": r"\sspRDFa", - "rdf_b": r"\sspRDFb", - "rdf_c": r"\sspRDFc", - "rdf_llm_schema_align_v1": r"\sspRDFc", - "json_a": r"\sspJSONa", - "json_b": r"\sspJSONb", - "json_baseA": r"\sspJSONbaseA", - "json_c": r"\sspJSONc", - "json_llm_mapping_v1": r"\sspJSONc", - "text_a": r"\sspTexta", - "text_b": r"\sspTextb", - "text_c": r"\sspTextc", - "text_llm_triple_extract_v1": r"\sspTextc", - "rdf_json_text": r"\mspRJT", - "rdf_text_json": r"\mspRTJ", - "json_rdf_text": r"\mspJRT", - "json_text_rdf": r"\mspJTR", - "text_rdf_json": r"\mspTRJ", - "text_json_rdf": r"\mspTJR", -} - -METRIC_NAME_MAP = { - "entity_count": "EC", - "relation_count": "RC", - "triple_count": "FC", - "class_count": "TC", - "duration": "Time", - "loose_entity_count": "LEC", - "shallow_entity_count": "SEC", - # Semantic/Reasoning metrics - "reasoning": "EO", - "disjoint_domain": "EO1", - "incorrect_relation_direction": "EO2", - "incorrect_relation_cardinality": "EO3", - "incorrect_relation_range": "EO4", - "incorrect_relation_domain": "EO5", - "incorrect_datatype": "EO6", - "incorrect_datatype_format": "EO7", - "ontology_class_coverage": "EO8", - "ontology_relation_coverage": "EO9", - "ontology_namespace_coverage": "E10", - # Reference metrics - "ReferenceTripleAlignmentMetric": "RTC", - "ReferenceTripleAlignmentMetricSoftE": "RTC-SoftE", - "ReferenceTripleAlignmentMetricSoftEV": "RTC-SoftEV", - "ReferenceClassCoverageMetric": "RCC", - # ER metrics - "ER_EntityMatchMetric": "ER-EM", - "ER_RelationMatchMetric": "ER-RM", - # TE metrics - "TE_ExpectedEntityLinkMetric": "TE-EEL", - "TE_ExpectedRelationLinkMetric": "TE-ERL", - # Source metrics - "SourceEntityCoverageMetric": "VSEC", - "SourceEntityCoverageMetricSoft": "VSEC-Soft", - "REI_precision": "REI-Precision", - -} - -# long: -# disjoint_domain -# incorrect_relation_domain -# incorrect_relation_range -# incorrect_relation_direction -# incorrect_datatype -# incorrect_datatype_format -# short:ODT OD OR ORD OLT OLF OAvg -SEM_METRIC_SHORT_NAMES = { - # "reasoning" : "EO0", - "disjoint_domain": "$O_{DT}$", - "incorrect_relation_direction": "$O_{RD}$", - "incorrect_relation_cardinality": "$O_{CA}$", - "incorrect_relation_range": "$O_{R}$", - "incorrect_relation_domain": "$O_{D}$", - "incorrect_datatype": "$O_{LT}$", - "incorrect_datatype_format": "$O_{LF}$", - # "ontology_class_coverage": "$O_{CC}$", - # "ontology_relation_coverage": "$O_{RC}$", - # "ontology_namespace_coverage": "$O_{NC}$", -} - -METRIC_NAME_INDEX_PRETTY = [ - ("duration", "Runtime Duration"), - ("triple_count", "Fact/Triple Count"), - ("entity_count", "Entity Count"), - ("relation_count", "Relation Count"), - ("class_count", "Entity Type Count"), - ("Person", "Persons"), - ("Film", "Films"), - ("Company", "Companies"), - # ("Other", "Other Type"), - ("loose_entity_count", "Empty Entities"), - ("shallow_entity_count", "Shallow Entities"), - # Semantic/Reasoning metrics - # ("reasoning", "Reasoning"), - ("disjoint_domain", "Disjoint Domain"), - ("incorrect_relation_direction", "Incorrect Relation Direction"), - ("incorrect_relation_cardinality", "Incorrect Relation Cardinality"), - ("incorrect_relation_range", "Incorrect Relation Range"), - ("incorrect_relation_domain", "Incorrect Relation Domain"), - ("incorrect_datatype", "Incorrect Datatype"), - ("incorrect_datatype_format", "Incorrect Datatype Format"), - # ("ontology_class_coverage", "Ontology Class Coverage"), - # ("ontology_relation_coverage", "Ontology Relation Coverage"), - # ("ontology_namespace_coverage", "Ontology Namespace Coverage"), - # Source metrics - ("SourceEntityCoverageMetric", "Source Entity Recall"), - ("SourceEntityCoverageMetricSoft", "Source Entity Recall (~ID)"), - ("REI_precision", "Source Entity Precision (~ID)"), - # Reference metrics - ("ReferenceTripleAlignmentMetric", "Reference Alignment (f1)"), - ("ReferenceTripleAlignmentMetricSoftE", "Reference Alignment (~ID) (f1)"), - ("ReferenceTripleAlignmentMetricSoftEV", "Reference Alignment (~ID~Value) (f1)"), - # ("ReferenceClassCoverageMetric", "Reference Class Coverage"), - # ER metrics - ("ER_EntityMatchMetric", "Entity Match (p)"), - ("ER_RelationMatchMetric", "Relation Match (p)"), - # TE metrics - ("TE_ExpectedEntityLinkMetric", "Expected Entity Link (p)"), - ("TE_ExpectedRelationLinkMetric", "Expected Relation Link (p)"), -] - -METRIC_NAME_MAP_PRETTY = {k: v for k, v in METRIC_NAME_INDEX_PRETTY} -SEM_METRIC_LONG_NAMES = {v: k for k, v in SEM_METRIC_SHORT_NAMES.items()} diff --git a/experiments/moviekg/src/moviekg/paper/helpers/agggregate.py b/experiments/moviekg/src/moviekg/paper/helpers/agggregate.py deleted file mode 100644 index adff60e..0000000 --- a/experiments/moviekg/src/moviekg/paper/helpers/agggregate.py +++ /dev/null @@ -1,224 +0,0 @@ -import pandas as pd -import json -import numpy as np -from moviekg.paper.config import SEM_METRIC_SHORT_NAMES -from moviekg.paper.helpers.helpers import load_metrics_from_file -from moviekg.config import OUTPUT_ROOT - -def agg_duration_over_stages_per_pipeline(metric_df): - # group by pipeline and stage and take mean of normalized - metric_df = metric_df[metric_df["metric"] == "duration"] - # print(metric_df) - metric_df = metric_df.groupby(["pipeline"])["value"].sum().reset_index() - # add stage column = stage 3 - metric_df["stage"] = "stage_3"# - metric_df["metric"] = "duration" - - # print(metric_df.to_string()) - return metric_df - -def norm_min(min, value): - return (min/value) - -def norm_max(max, value): - return 1 / (max/value) - -def get_average_f1_source_entity_f1(df: pd.DataFrame): - df = df[df["metric"] == "SourceEntityPrecisionMetric"] - - for row in df.itertuples(): - details = json.loads(row.details) - expected_entities_count = details["expected_entities_count"] - found_entities_count = details["found_entities_count"] - overlapping_entities_count = details["overlapping_entities_count"] - possible_duplicates_count = details["possible_duplicates_count"] - overlapping_entities_strict_count = details["overlapping_entities_strict_count"] - - # print(f"pipeline={row.pipeline}, stage={row.stage}, expected_entities_count={expected_entities_count}, found_entities_count={found_entities_count}, overlapping_entities_count={overlapping_entities_count}, possible_duplicates_count={possible_duplicates_count}, overlapping_entities_strict_count={overlapping_entities_strict_count}") - precision = overlapping_entities_strict_count / overlapping_entities_count - precision = precision if precision <= 1.0 else 1.0 - recall = overlapping_entities_count / expected_entities_count - recall = recall if recall <= 1.0 else 1.0 - f1 = 2 * (precision * recall) / (precision + recall) - df.loc[row.Index, "normalized"] = f1 - - df = df[["pipeline", "normalized"]] - - # save as csv - - # calculate the average of the metrics - df = df.groupby("pipeline").mean().reset_index() - # set as value for normalized and stage_3 - df["stage"] = "stage_3" - df["metric"] = "SourceEntityF1Metric" - df["value"] = df["normalized"] - df = df[["pipeline", "stage", "metric", "value"]] - - return df - -def aggregate_reference_metrics(df: pd.DataFrame): - metrics = [ - "ReferenceTripleAlignmentMetricSoftEV", - "SourceEntityPrecisionMetric", - ] - - source_entity_f1_df = get_average_f1_source_entity_f1(df) - df = pd.concat([df, source_entity_f1_df]) - - df = df[df["metric"].isin(metrics)] - # if metric is ReferenceTripleAlignmentMetricSoftEV get details["f1"] and set normalized to it - - df.loc[df["metric"] == "ReferenceTripleAlignmentMetricSoftEV", "normalized"] = df[df["metric"] == "ReferenceTripleAlignmentMetricSoftEV"]["details"].apply(lambda x: json.loads(x)["f1_score"]) - - df = df[["pipeline", "stage", "metric", "normalized"]] - - new_rows = [] - for pipeline in df["pipeline"].unique(): - new_rows.append({ - "pipeline": pipeline, - "stage": "stage_3", - "metric": "EntityMatchingMetric", - "normalized": 0.85 - }) - new_rows.append({ - "pipeline": pipeline, - "stage": "stage_3", - "metric": "OntologyMatchingMetric", - "normalized": 0.75 - }) - new_rows.append({ - "pipeline": pipeline, - "stage": "stage_3", - "metric": "EntityLinkingMetric", - "normalized": 0.44 - }) - - df = pd.concat([df, pd.DataFrame(new_rows)]) - - - # for each pipeline and stage = stage_3, calculate the average of the metrics - df = df[df["stage"] == "stage_3"] - - return df - -def aggregate_efficiency_metrics(df: pd.DataFrame): - metrics = ["duration", "memory_peak"] - df = df[df["metric"].isin(metrics)] - df = df[["pipeline", "stage", "metric", "value"]] - # for duration aggregate sum the values for each pipeline and stage - - duration_df = agg_duration_over_stages_per_pipeline(df) - # remove duration - df = df[df["metric"] != "duration"] - df = pd.concat([df, duration_df]) - - df["stage"] = "stage_3" - - def get_min_for_metric(metric): - return df[df["metric"] == metric]["value"].min() - - def get_max_for_metric(metric): - return df[df["metric"] == metric]["value"].max() - - for metric in df["metric"].unique(): - min_val = get_min_for_metric(metric) - max_val = get_max_for_metric(metric) - df.loc[df["metric"] == metric, "normalized"] = norm_min(min_val, df["value"]) - - return df - - -def aggregate_semantic_metrics(df: pd.DataFrame): - metrics = list(SEM_METRIC_SHORT_NAMES.keys()) - df = df[df["metric"].isin(metrics)] - df = df[["pipeline", "stage", "metric", "normalized"]] - # for each pipeline and stage = stage_3, calculate the average of the metrics - df = df[df["stage"] == "stage_3"] - - return df - -def aggregate_size_metrics(df: pd.DataFrame): - metrics = ["entity_count", "triple_count"] - df = df[df["metric"].isin(metrics)] - df = df[["pipeline", "stage", "metric", "value"]] - # for each pipeline and stage = stage_3, calculate the average of the metrics - df = df[df["stage"] == "stage_3"] - - # Pivot to compute density per pipeline - wide = df.pivot(index="pipeline", columns="metric", values="value") - - # Compute density = triple_count / entity_count (guard against zero/NaN) - denom = wide["entity_count"] - numer = wide["triple_count"] - density = np.where((denom > 0) & np.isfinite(denom), numer / denom, np.nan) - wide["density"] = density - - - # Return to long format: (pipeline, metric, value) - df = (wide.reset_index() - .melt(id_vars="pipeline", var_name="metric", value_name="value")) - - - def _normalize(group: pd.DataFrame): - vmax = group["value"].max() - vmin = group["value"].min() - - invert_normalization = False - if group.name == "density": - invert_normalization = True - - if invert_normalization: - group["normalized"] = norm_min(vmin, group["value"]) # largest→0, smallest→1 - else: - group["normalized"] = norm_max(vmax, group["value"]) #(group["value"] - vmin) / (vmax - vmin) # smallest→0, largest→1 - - return group - - df = df.groupby("metric", group_keys=False).apply(_normalize) - - return df - -def mean_scores(df, column_name): - df = df[["pipeline", "normalized"]] - # calculate the average of the metrics - df = df.groupby("pipeline").mean().reset_index() - # rename normalized to semantic - df = df.rename(columns={"normalized": column_name}) - return df - -def aggregate_ranking_df(): - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - - # # replace pipeline name with name_mapping - # metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name) - - # only pipelines where name contains 2 "_" chars - # metric_df = metric_df[metric_df["pipeline"].str.count("_") == 2] TODO - - norm_semantic_df = aggregate_semantic_metrics(metric_df) - norm_semantic_df = norm_semantic_df[["pipeline", "metric", "normalized"]] - agg_semantic_df = mean_scores(norm_semantic_df, "semantic") - - norm_reference_df = aggregate_reference_metrics(metric_df) - norm_reference_df = norm_reference_df[["pipeline", "metric", "normalized"]] - # print(norm_reference_df.to_string()) - agg_reference_df = mean_scores(norm_reference_df, "reference") - - norm_efficiency_df = aggregate_efficiency_metrics(metric_df) - # print(norm_efficiency_df) - norm_efficiency_df = norm_efficiency_df[["pipeline", "metric", "normalized"]] - agg_efficiency_df = mean_scores(norm_efficiency_df, "efficiency") - - norm_size_df = aggregate_size_metrics(metric_df) - norm_size_df = norm_size_df[["pipeline", "metric", "normalized"]] - agg_size_df = mean_scores(norm_size_df, "size") - - norm_df = pd.merge(norm_semantic_df, norm_reference_df, on=["pipeline", "metric", "normalized"], how="outer") - norm_df = pd.merge(norm_df, norm_efficiency_df, on=["pipeline", "metric", "normalized"], how="outer") - norm_df = pd.merge(norm_df, norm_size_df, on=["pipeline", "metric", "normalized"], how="outer") - - agg_df = pd.merge(agg_semantic_df, agg_reference_df, on=["pipeline"], how="left") - agg_df = pd.merge(agg_df, agg_efficiency_df, on=["pipeline"], how="left") - agg_df = pd.merge(agg_df, agg_size_df, on=["pipeline"], how="left") - - return norm_df, agg_df \ No newline at end of file diff --git a/experiments/moviekg/src/moviekg/paper/helpers/getter.py b/experiments/moviekg/src/moviekg/paper/helpers/getter.py deleted file mode 100644 index 7c5bec0..0000000 --- a/experiments/moviekg/src/moviekg/paper/helpers/getter.py +++ /dev/null @@ -1,525 +0,0 @@ - -import pandas as pd -from collections import defaultdict -import json -from typing import List, Callable - - -type pipeline_name = str -type stage_name = str -type metric_name = str -type metric_value = float -type pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]] -type pipeline_stage_metric_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]] - -""" -Helper file to map final metrics as kgpipe.evaluation... still in progress - -Each getter returns a nested dictionary of pipeline, stage, metric_name -{ - "pipeline": { - "stage": { - "metric_name": value - } - } -} -""" - -# Util - -def dict_for_metric_name(df: pd.DataFrame, metric_name: str, row_name: str = "value") -> pipeline_stage_dict: - df = df[df["metric"] == metric_name] - metric_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for index, row in df.iterrows(): - metric_dict[row["pipeline"]][row["stage"]] = row[row_name] - return metric_dict - -# Statistical metrics - -def sta_entity_count(df: pd.DataFrame): - # only pipeline, stage, value - return dict_for_metric_name(df, "entity_count") - -def sta_fact_count(df: pd.DataFrame): - return dict_for_metric_name(df, "triple_count") - -def sta_type_count(df: pd.DataFrame): - return dict_for_metric_name(df, "class_count") - -def sta_relation_count(df: pd.DataFrame): - return dict_for_metric_name(df, "relation_count") - -def sta_shallow_entity_count(df: pd.DataFrame): - return dict_for_metric_name(df, "shallow_entity_count") - -def sta_denisity(df: pd.DataFrame): - fact_count = sta_fact_count(df) - entity_count = sta_entity_count(df) - - metric_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for pipeline, stage_dict in fact_count.items(): - for stage, value in stage_dict.items(): - metric_dict[pipeline][stage] = value / entity_count[pipeline][stage] - - return metric_dict - -def sta_duration(df: pd.DataFrame): - return dict_for_metric_name(df, "duration") - -# def sta_memory_peak(df: pd.DataFrame): -# return dict_for_metric_name(df, "memory_peak") - -# Semantic metrics - -def sem_disjoint_domain(df: pd.DataFrame): - return dict_for_metric_name(df, "disjoint_domain", "normalized") - -def sem_incorrect_relation_direction(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_relation_direction", "normalized") - -def sem_incorrect_relation_cardinality(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_relation_cardinality", "normalized") - -def sem_incorrect_relation_range(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_relation_range", "normalized") - -def sem_incorrect_relation_domain(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_relation_domain", "normalized") - -def sem_incorrect_datatype(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_datatype", "normalized") - -def sem_incorrect_datatype_format(df: pd.DataFrame): - return dict_for_metric_name(df, "incorrect_datatype_format", "normalized") - -# Reference metrics -def ref_kg_f1(df: pd.DataFrame): - df = df[df["metric"] == "ReferenceTripleAlignmentMetricSoftEV"] - - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - # print(details) - f1 = details.get("f1_score", -1) - res[row.pipeline][row.stage] = f1 - return res - -def ref_kg_p(df: pd.DataFrame): - df = df[df["metric"] == "ReferenceTripleAlignmentMetricSoftEV"] - - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - # print(details) - p = details["precision"] - res[row.pipeline][row.stage] = p - return res - -def ref_kg_r(df: pd.DataFrame): - df = df[df["metric"] == "ReferenceTripleAlignmentMetricSoftE"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - # print(details) - r = details["recall"] - res[row.pipeline][row.stage] = r - return res - -def ref_source_entity_f1(df: pd.DataFrame) -> pipeline_stage_dict: - df = df[df["metric"] == "SourceEntityPrecisionMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - expected_entities_count = details["expected_entities_count"] - found_entities_count = details["found_entities_count"] - overlapping_entities_count = details["overlapping_entities_count"] - possible_duplicates_count = details["possible_duplicates_count"] - overlapping_entities_strict_count = details["overlapping_entities_strict_count"] - - precision = overlapping_entities_strict_count / overlapping_entities_count - precision = precision if precision <= 1.0 else 1.0 - recall = overlapping_entities_count / expected_entities_count - recall = recall if recall <= 1.0 else 1.0 - f1 = 2 * (precision * recall) / (precision + recall) - df.loc[row.Index, "normalized"] = f1 - res[row.pipeline][row.stage] = f1 - return res - -def ref_source_entity_p(df: pd.DataFrame): - df = df[df["metric"] == "SourceEntityPrecisionMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - precision = details["overlapping_entities_strict_count"] / details["overlapping_entities_count"] - res[row.pipeline][row.stage] = precision - return res - -def ref_source_entity_r(df: pd.DataFrame): - df = df[df["metric"] == "SourceEntityPrecisionMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - recall = details["overlapping_entities_count"] / details["expected_entities_count"] - res[row.pipeline][row.stage] = recall - return res - -def ref_entity_matching_f1(df: pd.DataFrame): - df = df[df["metric"] == "ER_EntityMatchMetric"] - - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_seed_match_cnt"] - fp = details["false_seed_match_cnt"] - fn = details["false_missing_seed_match_cnt"] - f1 = 2 * tp / (2 * tp + fp + fn) - res[row.pipeline][row.stage] = f1 - return res - -def ref_entity_matching_p(df: pd.DataFrame): - df = df[df["metric"] == "ER_EntityMatchMetric"] - - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_seed_match_cnt"] - fp = details["false_seed_match_cnt"] - fn = details["false_missing_seed_match_cnt"] - precision = tp / (tp + fp) - res[row.pipeline][row.stage] = precision - return res - -def ref_entity_matching_r(df: pd.DataFrame): - df = df[df["metric"] == "ER_EntityMatchMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_seed_match_cnt"] - fp = details["false_seed_match_cnt"] - fn = details["false_missing_seed_match_cnt"] - recall = tp / (tp + fn) - res[row.pipeline][row.stage] = recall - return res - -RM_DEFAULT_FN=24 # 23 + label - -def ref_relation_matching_f1(df: pd.DataFrame): - df = df[df["metric"] == "ER_RelationMatchMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_relation_match_cnt"] - fp = details["false_relation_match_cnt"] - fn = RM_DEFAULT_FN - (tp+fp) # details.get("false_missing_relation_match_cnt", 0) - f1 = 2 * tp / (2 * tp + fp + fn) - res[row.pipeline][row.stage] = f1 - return res - - -def ref_relation_matching_p(df: pd.DataFrame): - df = df[df["metric"] == "ER_RelationMatchMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_relation_match_cnt"] - fp = details["false_relation_match_cnt"] - fn = RM_DEFAULT_FN - (tp+fp) # details.get("false_missing_relation_match_cnt", 0) - print(f"tp, fp, fn for {row.pipeline} {row.stage}: {tp}, {fp}, {fn}") - precision = tp / (tp + fp) if (tp + fp) > 0 else 0 - res[row.pipeline][row.stage] = precision - return res - - -def ref_relation_matching_r(df: pd.DataFrame): - df = df[df["metric"] == "ER_RelationMatchMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_relation_match_cnt"] - fp = details["false_relation_match_cnt"] - fn = RM_DEFAULT_FN - (tp+fp) # details.get("false_missing_relation_match_cnt", 0) - recall = tp / (tp + fn) if (tp + fn) > 0 else 0 - res[row.pipeline][row.stage] = recall - return res - -def ref_entity_linking_r(df: pd.DataFrame): - df = df[df["metric"] == "TE_ExpectedEntityLinkMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - # print(details) - tp = details["true_link_cnt"] - fp = details["false_link_cnt"] - fn = details["false_missing_link_cnt"] - r = tp / (tp + fn) if (tp + fn) > 0 else 0 - res[row.pipeline][row.stage] = r - return res - -def ref_json_entity_matching_f1(df: pd.DataFrame): - df = df[df["metric"] == "JsonEntityMatchingMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - res[row.pipeline][row.stage] = details["f1_score"] - return res - -def ref_json_entity_matching_p(df: pd.DataFrame): - df = df[df["metric"] == "JsonEntityMatchingMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - res[row.pipeline][row.stage] = details["precision"] - return res - -def ref_json_entity_matching_r(df: pd.DataFrame): - df = df[df["metric"] == "JsonEntityMatchingMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - res[row.pipeline][row.stage] = details["recall"] - return res - -def ref_json_entity_linking_r(df: pd.DataFrame): - df = df[df["metric"] == "JsonEntityLinkingMetric"] - res: pipeline_stage_dict = defaultdict[pipeline_name, defaultdict[stage_name, metric_value]](lambda: defaultdict[stage_name, metric_value](lambda: None)) - for row in df.itertuples(): - details = json.loads(row.details) - if "error" in details: - res[row.pipeline][row.stage] = -1 - continue - res[row.pipeline][row.stage] = details["recall"] - return res - -TABLE_DISPLAY_NAMES = { - # Statistical metrics - sta_entity_count.__name__ : "EC", - sta_fact_count.__name__: "FC", - sta_type_count.__name__: "TC", - sta_relation_count.__name__: "RC", - sta_shallow_entity_count.__name__: "SEC", - sta_denisity.__name__: "D", - sta_duration.__name__: "T", - # sta_memory_peak.__name__: "M", - # Semantic metrics - sem_disjoint_domain.__name__: "ODT", - sem_incorrect_relation_direction.__name__: "ORD", - sem_incorrect_relation_cardinality.__name__: "OCA", - sem_incorrect_relation_range.__name__: "OR", - sem_incorrect_relation_domain.__name__: "OD", - sem_incorrect_datatype.__name__: "OLT", - sem_incorrect_datatype_format.__name__: "OLF", - # Reference metrics - ref_kg_f1.__name__: "RTC", - ref_kg_p.__name__: "RTC-SoftE", - ref_kg_r.__name__: "RTC-SoftE-R", - ref_source_entity_f1.__name__: "VSEC", - ref_source_entity_p.__name__: "VSEC-P", - ref_source_entity_r.__name__: "VSEC-R", - ref_entity_matching_f1.__name__: "ER-EM", - ref_entity_matching_p.__name__: "ER-EM-P", - ref_entity_matching_r.__name__: "ER-EM-R", - ref_relation_matching_f1.__name__: "ER-RM", - ref_relation_matching_p.__name__: "ER-RM-P", - ref_relation_matching_r.__name__: "ER-RM-R", - ref_entity_linking_r.__name__: "TE-EEL", - ref_json_entity_matching_f1.__name__: "JSON-EM", - ref_json_entity_matching_p.__name__: "JSON-EM-P", - ref_json_entity_matching_r.__name__: "JSON-EM-R", - ref_json_entity_linking_r.__name__: "JSON-EL", -} - -def dict_of_metrics(df: pd.DataFrame, metric_getters: List[Callable[[pd.DataFrame], dict]]) -> pipeline_stage_metric_dict: - """ - # call the getter functions for each metric name not the dict_for_metric_name - """ - - # Create a 3-level nested defaultdict: pipeline -> stage -> metric_name -> value - metric_dict = defaultdict(lambda: defaultdict(dict)) - - for metric_getter in metric_getters: - metric_name = metric_getter.__name__ # the metric name (e.g., "sta_entity_count") - metric_data = metric_getter(df) # returns pipeline->stage->value - - if metric_data is None: - continue - - for pipeline, stage_dict in metric_data.items(): - for stage, value in stage_dict.items(): - metric_dict[pipeline][stage][metric_name] = value - - return metric_dict - -def get_pipeline_stage_metric_dict(df: pd.DataFrame, metric_names: List[str]) -> pipeline_stage_metric_dict: - """ - # call the getter functions for each metric name not the dict_for_metric_name - """ - return dict_of_metrics(df, [globals()[f"{metric_name.lower()}"] for metric_name in TABLE_DISPLAY_NAMES.keys()]) - - -def normalize_min_best(values: List[float], value: float) -> float: - def norm_min(min, value): - return (min/value) - return norm_min(min(values), value) - - -def normalize_max_best(values: List[float], value: float) -> float: - # print(f"values: {values}, value: {value}") - def norm_max(max, value): - return 1 / (max/value) - return norm_max(max(values), value) - - -def normalize_metric(psmd: pipeline_stage_metric_dict, metric_name: str, stages: List[str], func: Callable[[list[float], float], float]) -> pipeline_stage_metric_dict: - values_for_metric = [] - - pipelines_to_normalize = [] - stages_to_normalize = [] - - - for pipeline, stage_dict in psmd.items(): - pipelines_to_normalize.append(pipeline) - for stage, metric_dict in stage_dict.items(): - if stage not in stages or metric_name not in metric_dict: - continue - stages_to_normalize.append(stage) - values_for_metric.append(metric_dict[metric_name]) - - for pipeline in pipelines_to_normalize: - for stage in stages_to_normalize: - if metric_name not in psmd[pipeline][stage]: - continue - value = psmd[pipeline][stage][metric_name] - if metric_name == sta_fact_count.__name__: - if pipeline in ["json_llm_mapping_v1", "text_llm_triple_extract_v1"]: - values_for_metric=[65000] - else: - values_for_metric=[340000] - print(f"setting max ec for norm {pipeline} {values_for_metric}") - psmd[pipeline][stage][metric_name+"_norm"] = func(values_for_metric, value) - - return psmd - -def update_task_selected_task_metric(psmd: pipeline_stage_metric_dict, metric_name: str) -> pipeline_stage_metric_dict: - - for pipleine, stage_dict in psmd.items(): - if pipleine in ["reference", "seed"]: - continue - for stage, metric_dict in stage_dict.items(): - entity_matching_f1 = metric_dict.get(ref_entity_matching_f1.__name__, -1) - relation_matching_f1 = metric_dict.get(ref_relation_matching_f1.__name__, -1) - entity_linking_r = metric_dict.get(ref_entity_linking_r.__name__, -1) - json_entity_matching_f1 = metric_dict.get(ref_json_entity_matching_f1.__name__, -1) - json_entity_linking_r = metric_dict.get(ref_json_entity_linking_r.__name__, -1) - - if json_entity_matching_f1 != -1: - metric_dict[metric_name] = json_entity_matching_f1 - metric_dict[metric_name+"_spec"] = "JSON ER" - elif entity_matching_f1 != -1: - metric_dict[metric_name] = (entity_matching_f1 + relation_matching_f1) / 2 - metric_dict[metric_name+"_spec"] = "RDF ER" - elif json_entity_linking_r != -1: - metric_dict[metric_name] = json_entity_linking_r - metric_dict[metric_name+"_spec"] = "JSON EL" - else: - metric_dict[metric_name] = entity_linking_r - metric_dict[metric_name+"_spec"] = "TE" - - return psmd - -def agg_avg(values: list[float]) -> float: - return sum(values) / len(values) - -def agg_sum(values: list[float]) -> float: - return sum(values) - -def agg_metric_over_stages(psmd: pipeline_stage_metric_dict, metric_name: str, suffix: str, agg_func: Callable[[list[float]], float]) -> pipeline_stage_metric_dict: - - values_for_metric_by_pipeline = defaultdict[pipeline_name, list[float]](lambda: []) - pipelines_to_agg = [] - stages_to_agg = [] - - for pipeline, stage_dict in psmd.items(): - if pipeline in ["reference", "seed"]: - continue - pipelines_to_agg.append(pipeline) - for stage, metric_dict in stage_dict.items(): - if metric_name not in metric_dict: - continue - stages_to_agg.append(stage) - values_for_metric_by_pipeline[pipeline].append(metric_dict[metric_name]) - - for pipeline in pipelines_to_agg: - try: - psmd[pipeline]["stage_3"][metric_name+suffix] = agg_func(values_for_metric_by_pipeline[pipeline]) - except Exception as e: - print(f"Error aggregating metric {metric_name} for pipeline {pipeline}: {e}") - print(values_for_metric_by_pipeline[pipeline]) - psmd[pipeline]["stage_3"][metric_name+suffix] = 0 - return psmd - -def apply_selected_updates(psmd: pipeline_stage_metric_dict) -> pipeline_stage_metric_dict: - normalize_metric(psmd, "sta_entity_count", ["stage_3"], normalize_max_best) - update_task_selected_task_metric(psmd, "ref_selected_task_metric") - agg_metric_over_stages(psmd, "ref_selected_task_metric", "_avg", agg_avg) - agg_metric_over_stages(psmd, "sta_duration", "_sum", agg_sum) - agg_metric_over_stages(psmd, "ref_source_entity_f1", "_avg", agg_avg) - agg_metric_over_stages(psmd, "ref_kg_f1", "_avg", agg_avg) - return psmd - -def test_getter(): - from pathlib import Path - from moviekg.paper.helpers.helpers import load_metrics_from_file - print(TABLE_DISPLAY_NAMES.keys()) - df = load_metrics_from_file(Path("/home/marvin/project/data/out/large") / "all_metrics.csv") - psmd = dict_of_metrics(df, [globals()[f"{metric_name.lower()}"] for metric_name in TABLE_DISPLAY_NAMES.keys()]) - - - apply_selected_updates(psmd) - - for pipeline, stage_dict in psmd.items(): - for stage, metric_dict in stage_dict.items(): - if pipeline in ["reference", "seed"]: - continue - # print(f"{pipeline} {stage} {metric_dict['ref_selected_task_metric']} {metric_dict['ref_selected_task_metric_spec']}") - if "stage_3" == stage: - # print(f"{pipeline} {stage} {metric_dict['ref_selected_task_metric_agg']}") - print(pipeline) - print(stage) - print(json.dumps(metric_dict, indent=4)) - print("--------------------------------") \ No newline at end of file diff --git a/experiments/moviekg/src/moviekg/paper/helpers/helpers.py b/experiments/moviekg/src/moviekg/paper/helpers/helpers.py deleted file mode 100644 index 161f55e..0000000 --- a/experiments/moviekg/src/moviekg/paper/helpers/helpers.py +++ /dev/null @@ -1,739 +0,0 @@ -from matplotlib.font_manager import font_scalings -import pandas as pd -import numpy as np -import matplotlib.pyplot as plt -import seaborn as sns -import json -from matplotlib.patches import Patch -from matplotlib.ticker import ScalarFormatter -from typing import Dict -import re -import pandas as pd - -import pandas as pd -from typing import List, Optional - -from moviekg.paper.config import HEADERS, main_classes -from moviekg.pipelines.test_inc_ssp import pipeline_types, llm_pipeline_types - - -def load_metrics_from_file(file_path): - # print("Loading metrics from file: ", file_path) - df = pd.read_csv(file_path, names=HEADERS, skiprows=1) - return df - -def plot_growth_v1(df, metrics): - """ - df: pandas DataFrame with columns: - pipeline, stage, aspect, metric, value, normalized, details - metrics: list[str] of metric names to plot - - Generates a subplot for each metric. - Each subplot has x-axis: stage, y-axis: value. - Each pipeline's value is a grouped bar at each stage. - Returns (fig, axes). - """ - required_cols = {"pipeline", "stage", "aspect", "metric", "value", "normalized", "details"} - missing = required_cols - set(df.columns) - if missing: - raise ValueError(f"DataFrame is missing required columns: {sorted(missing)}") - - if not isinstance(metrics, (list, tuple)) or len(metrics) == 0: - raise ValueError("`metrics` must be a non-empty list of metric names.") - - # Only keep rows for requested metrics - plot_df = df[df["metric"].isin(metrics)].copy() - if plot_df.empty: - raise ValueError("No rows found for the requested metrics.") - - # Create subplots - n_metrics = len(metrics) - fig, axes = plt.subplots(n_metrics, 1, figsize=(10, max(3.5, 2.8 * n_metrics)), squeeze=False) - axes = axes.ravel() - - # Overall (stable) pipeline order: alphabetical for consistency - all_pipelines = sorted(plot_df["pipeline"].dropna().unique().tolist()) - - for ax, metric in zip(axes, metrics): - mdf = plot_df[plot_df["metric"] == metric].copy() - if mdf.empty: - ax.set_visible(False) - continue - - # Preserve stage order as first-appearance order for this metric - stage_order = pd.Index(mdf["stage"].dropna().astype(str)).drop_duplicates().tolist() - if not stage_order: - ax.set_visible(False) - continue - - # Pivot to stage x pipeline = values - pivot = ( - mdf.assign(stage=pd.Categorical(mdf["stage"].astype(str), categories=stage_order, ordered=True)) - .pivot_table( - index="stage", - columns="pipeline", - values="value", - aggfunc="sum", - ) - .reindex(columns=all_pipelines) # ensure consistent pipeline order - .sort_index() - ) - - # If some pipelines/stages don't exist, fill with 0 (or use NaN if you prefer gaps) - vals = pivot.fillna(0.0).values - stages = pivot.index.astype(str).tolist() - pipelines = pivot.columns.astype(str).tolist() - - n_stages = len(stages) - n_pipes = max(1, len(pipelines)) - - x = np.arange(n_stages, dtype=float) - total_width = 0.8 - bar_w = total_width / n_pipes - - # Center the grouped bars around each stage tick - start = x - (total_width / 2) + (bar_w / 2) - - for i, pipe in enumerate(pipelines): - y = pivot[pipe].fillna(0.0).to_numpy() - ax.bar(start + i * bar_w, y, width=bar_w, label=pipe) - - ax.set_title(str(metric)) - ax.set_xlabel("stage") - ax.set_ylabel("value") - ax.set_xticks(x) - ax.set_xticklabels(stages, rotation=0, ha="center") - - # Only show legend if multiple pipelines - if n_pipes > 1: - ax.legend(title="pipeline", frameon=False, ncols=min(3, n_pipes)) - ax.grid(axis="y", linestyle=":", linewidth=0.7, alpha=0.6) - - fig.tight_layout() - return fig, axes - -# --- Hardcoded pipeline colors (light/dark for solos; mid-tone for combined) -PALETTE = { - # JSON solo - "json_a": "#9ecae1", "json_b": "#1f77b4", "json_c": "21f77b4", - # "json_baseA": "#9ecae1", - # RDF solo - "rdf_a": "#a1d99b", "rdf_b": "#2ca02c", "rdf_c": "#3ca02c", - # TEXT solo - "text_a": "#fdd0a2", "text_b": "#ff7f0e", "text_c": "#ff7f0e", - - # JSON mixed → violet - "json_rdf_text": "#756bb1", "json_text_rdf": "#756bb1", - # RDF mixed → teal - "rdf_json_text": "#1c9099", "rdf_text_json": "#1c9099", - # TEXT mixed → red-brown - "text_json_rdf": "#d95f0e", "text_rdf_json": "#d95f0e", -} - -HUE_ORDER = [ - "json_a","json_b","json_rdf_text","json_text_rdf", - "rdf_a","rdf_b","rdf_json_text","rdf_text_json", - "text_a","text_b","text_json_rdf","text_rdf_json" -] - -def plot_growth(df, metrics, kind="bar", references={}): - """ - df: pandas DataFrame with columns: - pipeline, stage, aspect, metric, value, normalized, details - metrics: list[str] of metric names to plot - kind: "bar" or "line" - - Generates a facet plot (subplot per metric). - Each subplot has x-axis: stage, y-axis: value, - with different pipelines distinguished by color. - """ - required_cols = {"pipeline", "stage", "aspect", "metric", "value", "normalized", "details"} - missing = required_cols - set(df.columns) - if missing: - raise ValueError(f"DataFrame is missing required columns: {sorted(missing)}") - - if not metrics: - raise ValueError("`metrics` must be a non-empty list of metric names.") - - # Filter to requested metrics - plot_df = df[df["metric"].isin(metrics)].copy() - if plot_df.empty: - raise ValueError("No rows found for the requested metrics.") - - # Consistent style - sns.set(style="whitegrid") - - stage_order = list(dict.fromkeys(plot_df["stage"])) - - # sns.set_context("notebook", font_scale=1.2) - - # Facet grid WITHOUT hue to avoid legend kwarg collisions - g = sns.FacetGrid( - plot_df, - col="metric", - col_wrap=len(metrics), - height=len(metrics)*1.6, - aspect=1.5, - sharey=False, - col_order=metrics, - legend_out=False, - ) - - if kind != "bar": - raise ValueError("`kind` must be 'bar' for per-bar labels.") - - # Draw grouped bars with hue specified inside map_dataframe - g.map_dataframe( - sns.barplot, - x="stage", - y="value", - hue="pipeline", - hue_order=HUE_ORDER, - palette=PALETTE, - order=stage_order, - dodge=True, - errorbar=None - ) - - - try: - g._legend.remove() - except Exception: - pass - - # build a single combined legend below everything - handles, labels = g.axes[0].get_legend_handles_labels() - g.fig.legend( - handles, labels, - loc="lower center", - ncol=min(6, len(labels)), # 6 items per row (→ 2 rows for 12 pipelines) - bbox_to_anchor=(0.5, -0.1), # adjust vertical offset - frameon=False - ) - - for ax_idx, ax in enumerate(g.axes.flat): - - # remove x axis label - ax.set_xlabel("") - - # numbers 1 to 3 - for stage_idx in range(1, 4): - value, nvalue, details = get_reference_value(df, metrics[ax_idx], "stage_"+str(stage_idx)) - # print(metrics[ax_idx], value) - xpos = stage_idx - if stage_idx == 0: - ax.axhline(value, ls="--", color="red") - else: - ax.axhline(value, ls="--", color="black") - - for ax in g.axes.flat: - ax.set_xlabel("") - # tidy up axes - ax.set_xticks(range(len(stage_order))) - ax.set_xticklabels(stage_order) - ax.yaxis.set_major_formatter(ScalarFormatter(useMathText=True)) - ax.ticklabel_format(style='sci', axis='y', scilimits=(0,0)) - ax.grid(True, axis="y", linestyle="--", alpha=0.3) - ax.margins(x=0.02) - - return g - -def _stage_sort_key(s): - """ - Convert 'stage_3' -> 3 for natural sorting; unknown formats go to +inf. - """ - m = re.search(r"(\d+)$", str(s)) - return int(m.group(1)) if m else float("inf") - -def _shorten_iri(iri): - """ - Turn 'http://kg.org/ontology/Person' -> 'Person' for cleaner legends. - """ - return str(iri).rstrip("/").split("/")[-1] - -def _flatten_to_df(nested): - """ - nested: dict like { - 'rdf_a': {'stage_1': {'iri': count, ...}, ...}, - 'reference': {...}, - ... - } - Returns a tidy DataFrame with columns: - Pipeline, Stage, Class, Actual, Expected - """ - - # Split out reference (Expected) from others (Actual) - if "reference" not in nested: - raise ValueError("Input must contain a 'reference' key with expected counts.") - ref = nested["reference"] - pipelines = {k: v for k, v in nested.items() if k != "reference"} - - # Collect all stages/classes across data to ensure aligned zeros - all_stages = sorted( - {s for d in nested.values() for s in d.keys()}, - key=_stage_sort_key - ) - - - - all_classes = sorted( - {c for d in nested.values() for s in d.values() for c in s.keys()} - ) - - - - # Build rows - rows = [] - for pipe, pdata in pipelines.items(): - for stage in all_stages: - for cls in all_classes: - actual = pdata.get(stage, {}).get(cls, 0) - expected = ref.get(stage, {}).get(cls, 0) - if cls not in main_classes: - cls = "Other" - rows.append({ - "Pipeline": pipe, - "Stage": stage, - "Class": cls, - "Actual": actual, - "Expected": expected, - "Class Short": _shorten_iri(cls), - }) - return pd.DataFrame(rows), [ _shorten_iri(c) for c in all_classes ], all_stages, list(pipelines.keys()) - -import pandas as pd -import matplotlib.pyplot as plt -from matplotlib.patches import Patch -import seaborn as sns - -def plot_actual_expected_stacked(df, - pipeline_order=None, - stage_order=None, - class_order=None, - col_wrap=3, - height=4, - suptitle="Actual vs Expected (stacked by Class) per Pipeline & Stage"): - # --- prep --- - df = df.copy() - # ensure numeric & fill NAs - for col in ["Actual", "Expected"]: - df[col] = pd.to_numeric(df[col], errors="coerce").fillna(0) - - # Use Class Short as plotting label (cleaner legend) - if "Class Short" not in df.columns: - df["Class Short"] = df["Class"] - - # Default orders (preserve first-seen order) - if pipeline_order is None: - pipeline_order = list(pd.unique(df["Pipeline"])) - if stage_order is None: - stage_order = list(pd.unique(df["Stage"])) - if class_order is None: - class_order = list(pd.unique(df["Class Short"])) - - # aggregate once - gdf = ( - df.groupby(["Pipeline", "Stage", "Class Short"], as_index=False) - .agg(Actual=("Actual","sum"), Expected=("Expected","sum")) - ) - - # full grid to align missing combos to 0 - full_index = pd.MultiIndex.from_product( - [pipeline_order, stage_order], names=["Pipeline","Stage"] - ) - - # pivots: (Pipeline, Stage) × Class - actual = (gdf.pivot_table(index=["Pipeline","Stage"], columns="Class Short", - values="Actual", aggfunc="sum") - .reindex(full_index) - .reindex(columns=class_order) - .fillna(0)) - expected = (gdf.pivot_table(index=["Pipeline","Stage"], columns="Class Short", - values="Expected", aggfunc="sum") - .reindex(full_index) - .reindex(columns=class_order) - .fillna(0)) - - # --- plot --- - sns.set(style="whitegrid") - n_pipes = len(pipeline_order) - ncols = min(col_wrap, n_pipes) - nrows = (n_pipes + ncols - 1) // ncols - fig, axes = plt.subplots(nrows, ncols, figsize=(ncols*height*1.6, nrows*height), squeeze=False, constrained_layout=True) - axes = axes.flatten() - - # palettes - blues = sns.color_palette("Blues", n_colors=max(3, len(class_order))) - oranges = sns.color_palette("Oranges", n_colors=max(3, len(class_order))) - color_map_actual = {cls: blues[i % len(blues)] for i, cls in enumerate(class_order)} - color_map_expected = {cls: oranges[i % len(oranges)] for i, cls in enumerate(class_order)} - - width = 0.4 - for ax, pipeline in zip(axes, pipeline_order): - act = actual.loc[pipeline] # index=Stage, cols=Class Short - exp = expected.loc[pipeline] # index=Stage, cols=Class Short - - x = range(len(stage_order)) - - # stacked bars - bottom_a = [0.0]*len(stage_order) - bottom_e = [0.0]*len(stage_order) - - for cls in class_order: - a_vals = act[cls].to_numpy() - e_vals = exp[cls].to_numpy() - - ax.bar([xi - 0.2 for xi in x], a_vals, width=width, bottom=bottom_a, color=color_map_actual[cls], edgecolor="none", label="Actual") - ax.bar([xi + 0.2 for xi in x], e_vals, width=width, bottom=bottom_e, color=color_map_expected[cls], edgecolor="none", label="Expected") - - # update bottoms - bottom_a = [b + v for b, v in zip(bottom_a, a_vals)] - bottom_e = [b + v for b, v in zip(bottom_e, e_vals)] - - # cosmetics - ax.set_title(pipeline) - ax.set_xticks(list(x)) - ax.set_xticklabels(stage_order) - ax.set_xlabel("Stage") - ax.set_ylabel("Count") - ax.grid(axis="y", linestyle=":", linewidth=0.7, alpha=0.6) - - # hide any unused axes - for j in range(len(pipeline_order), len(axes)): - fig.delaxes(axes[j]) - - # legend - handles = ( - [Patch(facecolor=color_map_actual[c], label=f"{c} • Actual") for c in class_order] + - [Patch(facecolor=color_map_expected[c], label=f"{c} • Expected") for c in class_order] - ) - - # legend (robust placement) - ncol_leg = min(4, len(handles)) - nrows_leg = int(np.ceil(len(handles) / ncol_leg)) - - leg = fig.legend( - handles=handles, - loc="lower center", - ncol=ncol_leg, - bbox_to_anchor=(0.5, 0.02), # inside the figure, just above bottom - frameon=False - ) - - # Title inside the top of the figure - fig.suptitle(suptitle, y=0.99, fontsize=14) - - # Give the legend guaranteed space at the bottom, proportional to its rows - # (works alongside constrained_layout) - plt.subplots_adjust(bottom=0.08 + 0.05 * max(0, nrows_leg - 1)) - - return fig - - -def plot_expected_actual_from_nested( - nested, - col_wrap=3, - height=4, - suptitle="Actual vs Expected (stacked by Class) per Pipeline & Stage" -): - """ - nested: dict structured like the user's example. - Creates one subplot per pipeline. For each Stage on that subplot, - draws two stacked bars (Actual & Expected), each stacked by Class. - """ - - df, class_labels, stage_order, pipeline_order = _flatten_to_df(nested) - - # We’ll use the *short* class labels for stacking order & legend - classes = class_labels - - # Prepare nice style - sns.set(style="whitegrid") - g = sns.FacetGrid( - df, - col="Pipeline", - col_wrap=col_wrap, - height=height, - sharey=True, - col_order=pipeline_order - ) - - df[['Actual','Expected']] = df[['Actual','Expected']].fillna(0) - - # Aggregate by Pipeline, Stage, Class, and Class Short - df = ( - df.groupby(['Pipeline', 'Stage', 'Class', 'Class Short'], as_index=False) - .agg({'Actual': 'sum', 'Expected': 'sum'}) - ) - - return plot_actual_expected_stacked(df, pipeline_order, stage_order, ["Other", "Person", "Company", "Film"], col_wrap, height, suptitle) - - -def plot_class_occurence(df): - """ - df: pandas dataframe with columns: pipeline, stage, aspect, metric, value, normalized, details - """ - - # filter df for metrics - df = df[df["metric"].isin(["class_occurrence"])] - # filter details contains unique_classes - df = df[df["details"].str.contains("unique_classes")] - # remove duration column - df = df.drop(columns=["duration"]) - # filter not seed pipeline - df = df[df["pipeline"] != "seed"] - - - class_counts_by_stage_by_pipeline = {} - # for each row - for index, row in df.iterrows(): - details = json.loads(row["details"]) - classes = details["classes"] - if row["pipeline"] not in class_counts_by_stage_by_pipeline: - class_counts_by_stage_by_pipeline[row["pipeline"]] = {} - # skip stage 0 - if row["stage"] == "stage_0": - continue - if row["stage"] not in class_counts_by_stage_by_pipeline[row["pipeline"]]: - class_counts_by_stage_by_pipeline[row["pipeline"]][row["stage"]] = {} - for class_name, count in classes.items(): - if class_name not in class_counts_by_stage_by_pipeline[row["pipeline"]][row["stage"]]: - class_counts_by_stage_by_pipeline[row["pipeline"]][row["stage"]][class_name] = 0 - class_counts_by_stage_by_pipeline[row["pipeline"]][row["stage"]][class_name] += count - - # remove stage_0 - class_counts_by_stage_by_pipeline = {k: v for k, v in class_counts_by_stage_by_pipeline.items() if k != "stage_0"} - - return plot_expected_actual_from_nested(class_counts_by_stage_by_pipeline, col_wrap=2, height=4, suptitle="Actual vs Reference by Stage • Stacked by Class") - - -def rank_pipeline_stage(group_df, metric_names, metric_weights): - weights = pd.Series(metric_weights, index=metric_names) - vals = ( - group_df.set_index("metric")["normalized"] - .reindex(metric_names) # align order - .astype(float) - ) - return float((vals * weights).sum()/len(vals)) - -def rank_metrics_apply(df, metric_names, metric_weights): - dff = df[df["metric"].isin(metric_names)] - return ( - dff.groupby(["pipeline", "stage"]) - .apply(lambda g: rank_pipeline_stage(g, metric_names, metric_weights)) - .rename("score") - .reset_index() - ) - - -def rank_metrics( - df: pd.DataFrame, - metric_names: List[str], - metric_weights: List[float], - *, - agg: str = "mean", - fill_missing: Optional[float] = 0.0, - score_col: str = "score", -) -> pd.DataFrame: - """ - Compute a weighted score per (pipeline, stage) using normalized metric values. - - Parameters - ---------- - df : DataFrame - Must include columns: pipeline, stage, metric, normalized - (other columns are ignored). - metric_names : list of str - Names of metrics to include, in the same order as their weights. - metric_weights : list of float - Weights aligned to metric_names. - agg : {"mean","sum","max","min"}, default "mean" - If there are duplicate rows per (pipeline, stage, metric), how to aggregate. - fill_missing : float or None, default 0.0 - Value to fill when a metric is missing for a (pipeline, stage). - Use None to leave as NaN (then the final score may be NaN). - score_col : str, default "score" - Name of the output score column. - - Returns - ------- - DataFrame with columns: pipeline, stage, - """ - if len(metric_names) != len(metric_weights): - raise ValueError("metric_names and metric_weights must have the same length") - - # Keep only what we need - dff = df.loc[df["metric"].isin(metric_names), ["pipeline", "stage", "metric", "normalized"]] - - # Aggregate duplicates per (pipeline, stage, metric) - agg_map = {"mean": "mean", "sum": "sum", "max": "max", "min": "min"} - if agg not in agg_map: - raise ValueError(f'agg must be one of {list(agg_map)}') - pivot = dff.pivot_table( - index=["pipeline", "stage"], - columns="metric", - values="normalized", - aggfunc=agg_map[agg], - ) - - # Enforce column order and align with weights - pivot = pivot.reindex(columns=metric_names) - if fill_missing is not None: - pivot = pivot.fillna(fill_missing) - - weights = pd.Series(metric_weights, index=metric_names) - scores = pivot.dot(weights).rename(score_col) - - return scores.reset_index() - -def get_reference_value(df, metric_name, stage): - df = df[df["metric"] == metric_name] - df = df[df["stage"] == stage] - df = df[df["pipeline"] == "reference"] - # print(df.to_string()) - value = df["value"].values[0] - nvalue = df["normalized"].values[0] - details = json.loads(df["details"].values[0]) - return value, nvalue, details - - -def get_reference_class_counts(df) -> Dict[str, Dict[str, int]]: - df = df[df["pipeline"] == "reference"] - reference_stage_class_count: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int)) - df = df[df["metric"] == "class_occurrence"] - for stage in df["stage"].unique(): - df_stage = df[df["stage"] == stage] - details = json.loads(df_stage["details"].values[0]) - class_counts = details["classes"] - for class_name, count in class_counts.items(): - reference_stage_class_count[stage][class_name.split("/")[-1]] += count - - return reference_stage_class_count - -# def subplot_source_entity_integration(df): -# pass - -from collections import defaultdict - -def plot_class_occurence_new(df, reference_stage_class_count, classes): - - df = df[df["metric"] == "class_occurrence"] - - - pipeline_stage_class_count = defaultdict(lambda: defaultdict(lambda: defaultdict(int))) - - rows = [] - - # for each pipeline and stage - for pipeline in df["pipeline"].unique(): - for stage in df["stage"].unique(): - df_pipeline_stage = df[df["pipeline"] == pipeline] - df_pipeline_stage = df_pipeline_stage[df_pipeline_stage["stage"] == stage] - details = json.loads(df_pipeline_stage["details"].values[0]) - class_counts = details["classes"] - for class_name, count in class_counts.items(): - if class_name not in classes: - class_name = "Other" - pipeline_stage_class_count[pipeline][stage][class_name] += count - - # convert dict of dict to rows - for pipeline, stage_class_count in pipeline_stage_class_count.items(): - for stage, class_count in stage_class_count.items(): - for class_name, count in class_count.items(): - rows.append({"pipeline": pipeline, "stage": stage, "class": class_name.split("/")[-1], "count": count}) - - # df: pipeline, stage, class, count - df = pd.DataFrame(rows) - df = df[df["class"] != "Other"] - - classes_short = [class_name.split("/")[-1] for class_name in classes] - - sns.set(style="whitegrid") - - stage_order = list(dict.fromkeys(df["stage"])) - g = sns.FacetGrid( - df, - col="class", - col_wrap=3, - height=4, - aspect=1.5, - sharey=False, - col_order=classes_short #+["Other"], # preserve requested order - ) - g.map_dataframe( - sns.barplot, - x="stage", - y="count", - hue="pipeline", - hue_order=HUE_ORDER, - palette=PALETTE, - order=stage_order, - dodge=True, - errorbar=None - ) - - - for ax_idx, ax in enumerate(g.axes.flat): - class_idx = ax_idx - class_name = classes_short[class_idx] - - # remove x axis label - ax.set_xlabel("") - - for stage, class_counts in reference_stage_class_count.items(): - xpos = int(stage.split("_")[1]) - if stage == "stage_0": - ax.axhline(class_counts[class_name], ls="--", color="red") - else: - ax.axhline(class_counts[class_name], ls="--", color="black") - - - # g.add_legend() - - if g.legend is not None: - g.legend.remove() - - # build a combined legend below everything - handles, labels = g.axes[0].get_legend_handles_labels() - g.fig.legend( - handles, labels, - loc="lower center", - ncol=min(6, len(labels)), # split across columns - bbox_to_anchor=(0.5, -0.02) # push below the grid - ) - - # make space at bottom so legend isn’t cut off - g.fig.subplots_adjust(bottom=0.2) - - plt.subplots_adjust(top=0.88) - - # g.savefig("class_occurence_new.png") - - return g - - -def plot_class_occ_4_bar_chart(df): - metrics = ["class_occurrence"] - stages = ["stage_1", "stage_2", "stage_3"] - all_reference_values = {} - for metric in metrics: - for stage in stages: - value, nvalue, details = get_reference_value(df, metric, stage) - all_reference_values[metric] = { - "value": value, - "nvalue": nvalue, - "details": details - } - - reference_stage_class_count = get_reference_class_counts(df) - - # remove seed and reference pipeline - df = df[df["pipeline"] != "seed"] - df = df[df["pipeline"] != "reference"] - - # subplot_source_entity_integration(df) - - classes = ["http://kg.org/ontology/Film", "http://kg.org/ontology/Person", "http://kg.org/ontology/Company"] - - - return plot_class_occurence_new(df, reference_stage_class_count, classes) diff --git a/experiments/moviekg/src/moviekg/paper/helpers/ranking.py b/experiments/moviekg/src/moviekg/paper/helpers/ranking.py deleted file mode 100644 index ba9102a..0000000 --- a/experiments/moviekg/src/moviekg/paper/helpers/ranking.py +++ /dev/null @@ -1,119 +0,0 @@ -import pandas as pd -from collections import defaultdict -from typing import Any, Mapping, List, Dict - -from moviekg.config import OUTPUT_ROOT -from moviekg.paper.helpers.getter import ( - pipeline_stage_metric_dict, pipeline_name, metric_name, metric_value, - TABLE_DISPLAY_NAMES, - normalize_metric, normalize_min_best, normalize_max_best, - sta_fact_count, sta_denisity, sta_duration, #memory_peak is not considered - ref_kg_p, ref_source_entity_f1, - sem_disjoint_domain, sem_incorrect_relation_direction, sem_incorrect_relation_range, sem_incorrect_relation_domain, sem_incorrect_datatype, sem_incorrect_datatype_format -) - -type pipeline_agg = Mapping[pipeline_name, float] - -def agg_metrics(psmd: pipeline_stage_metric_dict, metric_names: List[metric_name]) -> pipeline_agg: - values_by_pipeline: Dict[pipeline_name, List[metric_value]] = defaultdict[pipeline_name, List[metric_value]](lambda: []) - for pipeline, stage_dict in psmd.items(): - if pipeline in ["reference", "seed"]: - continue - for stage, metric_dict in stage_dict.items(): - if stage not in ["stage_3"]: # only stage 3 is considered - continue - for metric_name in metric_names: - if metric_name in metric_dict: - values_by_pipeline[pipeline].append(metric_dict[metric_name]) - else: - print(f"pipeline: {pipeline}") - print(f"stage: {stage}") - print(f"metric_names: {metric_names}") - print(f"metric_dict: {metric_dict}") - raise ValueError(f"Metric {metric_name} not found in metric_names") - - res: pipeline_agg = defaultdict[pipeline_name, float](lambda: 0.0) - - for pipeline, values in values_by_pipeline.items(): - filtered_values = [value for value in values if value >= 0] - res[pipeline] = sum(filtered_values) / len(filtered_values) - print(pipeline) - print(" |\t".join(metric_names)) - print(" |\t".join([ str(value) for value in values_by_pipeline[pipeline]])) - print("="+str(res[pipeline])) - print("--------------------------------") - - return res - -def _rank_and_save2csv(weights: dict, outfile_stem: str, psmd: pipeline_stage_metric_dict, round_digits: int = 3) -> None: - - # psmd = normalize_metric(psmd, sta_fact_count.__name__, ["stage_3"], normalize_max_best) - psmd = normalize_metric(psmd, sta_denisity.__name__, ["stage_3"], normalize_max_best) - psmd = normalize_metric(psmd, sta_fact_count.__name__, ["stage_3"], normalize_max_best) - sta_metric_names = [sta_denisity.__name__+"_norm", sta_fact_count.__name__+"_norm"] - sta_agg = agg_metrics(psmd, sta_metric_names) - - sem_metric_names = [ - sem_disjoint_domain.__name__, sem_incorrect_relation_direction.__name__, - sem_incorrect_relation_range.__name__, sem_incorrect_relation_domain.__name__, - sem_incorrect_datatype.__name__, sem_incorrect_datatype_format.__name__] - sem_agg = agg_metrics(psmd, sem_metric_names) - - ref_metric_names = [ref_kg_p.__name__, ref_source_entity_f1.__name__+"_avg", "ref_selected_task_metric_avg"] - ref_agg = agg_metrics(psmd, ref_metric_names) - - psmd = normalize_metric(psmd, sta_duration.__name__+"_sum", ["stage_3"], normalize_min_best) - eff_metric_names = [sta_duration.__name__+"_sum_norm"] - eff_agg = agg_metrics(psmd, eff_metric_names) - - import json - json.dump(psmd, open(OUTPUT_ROOT / f"paper/{outfile_stem}_psmd.json", "w"), indent=4) - - df_rows = [] - - for pipeline, value in sem_agg.items(): - df_rows.append( - { - "pipeline": pipeline, - "semantic": round(value, round_digits), - "reference": round(ref_agg[pipeline], round_digits), - "size": round(sta_agg[pipeline], round_digits), - "efficiency": round(eff_agg[pipeline], round_digits) - } - ) - - - df = pd.DataFrame(df_rows) - - cols = ["size", "semantic", "reference", "efficiency"] - # Ensure we only use known columns; fill missing weights with 0.0 - w = pd.Series(weights).reindex(cols, fill_value=0.0) - - # Compute combined score - df = df[["pipeline"] + cols].copy() - df["combined"] = (df[cols] * w).sum(axis=1).round(round_digits) - - print(df.to_string()) - - # Sort & save (keep default index=True to match original behavior) - out = df[["pipeline", "combined"]].sort_values(by="combined", ascending=False) - out.to_csv(OUTPUT_ROOT / f"paper/{outfile_stem}.csv", sep="\t") - -# TODO cleanup -# def _rank_and_save(weights: dict, outfile_stem: str, df: pd.DataFrame, round_digits: int = 3) -> None: -# """ -# Compute weighted 'combined' score and save a TSV sorted by 'combined'. -# Uses the same behavior as your original functions (round to 3, keep default index in CSV). -# """ -# cols = ["size", "semantic", "reference", "efficiency"] -# # Ensure we only use known columns; fill missing weights with 0.0 -# w = pd.Series(weights).reindex(cols, fill_value=0.0) - -# # Compute combined score -# df = df[["pipeline"] + cols].copy() -# df["combined"] = (df[cols] * w).sum(axis=1).round(round_digits) - -# # Sort & save (keep default index=True to match original behavior) -# out = df[["pipeline", "combined"]].sort_values(by="combined", ascending=False) -# out.to_csv(OUTPUT_ROOT / f"paper/{outfile_stem}.csv", sep="\t") - diff --git a/experiments/moviekg/src/moviekg/paper/test_figtab.py b/experiments/moviekg/src/moviekg/paper/test_figtab.py deleted file mode 100644 index 1600033..0000000 --- a/experiments/moviekg/src/moviekg/paper/test_figtab.py +++ /dev/null @@ -1,701 +0,0 @@ -import json -import pandas as pd -from pathlib import Path -from collections import defaultdict - -from moviekg.config import OUTPUT_DIR, DATASET_SELECT -from moviekg.paper.helpers.agggregate import agg_duration_over_stages_per_pipeline -from moviekg.paper.helpers.getter import get_pipeline_stage_metric_dict, TABLE_DISPLAY_NAMES, apply_selected_updates -from moviekg.paper.helpers.helpers import load_metrics_from_file, plot_growth, plot_class_occ_4_bar_chart -from moviekg.paper.helpers.ranking import _rank_and_save2csv -from moviekg.paper.config import ( - name_mapping, METRIC_NAME_MAP, SEM_METRIC_SHORT_NAMES, - METRIC_NAME_INDEX_PRETTY, METRIC_NAME_MAP_PRETTY, SEM_METRIC_LONG_NAMES -) - - -# === Preamble === -if not OUTPUT_DIR: - raise ValueError("OUTPUT_DIR is not set") -if not DATASET_SELECT: - raise ValueError("DATASET_SELECT is not set") - -OUTPUT_ROOT = Path(OUTPUT_DIR) / DATASET_SELECT -(OUTPUT_ROOT / "paper").mkdir(parents=True, exist_ok=True) - -PIPLEINE_NAME_MAP = { - "json_rdf_text": "JRT", - "json_text_rdf": "JTR", - "rdf_json_text": "RJT", - "rdf_text_json": "RTJ", - "text_json_rdf": "TJR", - "text_rdf_json": "TRJ", - "json_a": "J_A", - "json_b": "J_B", - "json_c": "J_C", - "json_llm_mapping_v1": "J_C", - "json_baseA": "J_baseA", - "rdf_a": "R_A", - "rdf_b": "R_B", - "rdf_c": "R_C", - "rdf_llm_schema_align_v1": "R_C", - "text_a": "T_A", - "text_b": "T_B", - "text_c": "T_C", - "text_llm_triple_extract_v1": "T_C", - } - -def map_pipeline_name_pretty(pipeline_name): - return PIPLEINE_NAME_MAP.get(pipeline_name, pipeline_name) - -# === Helper Functions === - -def map_pipeline_name(pipeline_name): - return name_mapping.get(pipeline_name, pipeline_name) - - -def map_metric_name(metric_name): - return METRIC_NAME_MAP.get(metric_name, metric_name) - - -def add_REI_precision(metric_df): - # REI_fscore = 2 * (precision * recall) / (precision + recall) - source_entity_coverage_metric_soft = metric_df[metric_df["metric"] == "SourceEntityCoverageMetricSoft"] - - additional_rows = [] - for index, row in source_entity_coverage_metric_soft.iterrows(): - details = json.loads(row["details"]) - #"{""expected_entities_count"": 2758, ""found_entities_count"": 3099, ""overlapping_entities_count"": 53}" - - expected_entities_count = details["expected_entities_count"] - #found_entities_count = details["found_entities_count"] - overlapping_entities_count = details["overlapping_entities_count"] - - tp = overlapping_entities_count if overlapping_entities_count <= expected_entities_count else expected_entities_count - fp = overlapping_entities_count - tp if overlapping_entities_count > tp else 0 - precision = tp / (tp + fp) - - additional_rows.append( - {"pipeline": row["pipeline"], - "stage": row["stage"], - "metric": "REI_precision", - "aspect": "reference", - "normalized": precision, - "value": precision, - "details": row["details"]}) - - additional_df = pd.DataFrame(additional_rows) - return pd.concat([metric_df, additional_df]) - -def extract_class_occurence_df(df): - - classes = ["http://kg.org/ontology/Film", "http://kg.org/ontology/Person", "http://kg.org/ontology/Company"] - - - pipeline_stage_class_count = defaultdict(lambda: defaultdict(lambda: defaultdict(int))) - - # for each pipeline and stage - for pipeline in df["pipeline"].unique(): - for stage in df["stage"].unique(): - df_pipeline_stage = df[df["pipeline"] == pipeline] - df_pipeline_stage = df_pipeline_stage[df_pipeline_stage["stage"] == stage] - try: - details = json.loads(df_pipeline_stage["details"].values[0]) - class_counts = details["classes"] - for class_name, count in class_counts.items(): - if class_name not in classes: - class_name = "Other" - pipeline_stage_class_count[pipeline][stage][class_name] += count - except: - print(f"Error loading details for {pipeline} {stage}") - # print(df_pipeline_stage["details"].values[0]) - - rows = [] - for pipeline, stage_class_count in pipeline_stage_class_count.items(): - for stage, class_count in stage_class_count.items(): - for class_name, count in class_count.items(): - rows.append({"pipeline": pipeline, "stage": stage, "metric": class_name.split("/")[-1], "score": count}) - - return pd.DataFrame(rows) - - - -def map_metric_name_pretty(metric_name): - return METRIC_NAME_MAP_PRETTY.get(metric_name, metric_name) # TODO: remove this - -def get_statistics_df(df): - # only pipeline, stage, metric, normalized - - class_occurence_df = df[df["metric"] == "class_occurrence"] - class_count_df = extract_class_occurence_df(class_occurence_df) - - # print(class_count_df) - - df = df[df["aspect"] == "statistical"] - metircs = ["entity_count", "relation_count", "triple_count", "class_count", "duration", "loose_entity_count", "shallow_entity_count"] - df = df[df["metric"].isin(metircs)] - - df = df[["pipeline", "stage", "metric", "value"]] - df["score"] = df["value"].round(2) - - # union df and class_count_df - df = pd.concat([df, class_count_df]) - df[["pipeline"]] = df[["pipeline"]].map(map_pipeline_name) - - # rename metric to short name - df["metric"] = df["metric"].map(map_metric_name_pretty) - - # make each metric a column - df = df.pivot(index=["pipeline", "stage"], columns="metric", values="score") - df = df.reset_index() - - - return df - -def get_semantic_df(df): - # only pipeline, stage, metric, normalized - df = df[df["aspect"] == "semantic"] - df = df[["pipeline", "stage", "metric", "normalized"]] - - metrics = list(SEM_METRIC_SHORT_NAMES.keys()) - df = df[df["metric"].isin(metrics)] - - df["score"] = df["normalized"].round(2) - - # rename metric to short name - df["metric"] = df["metric"].map(map_metric_name_pretty) - - # make each metric a column - df = df.pivot(index=["pipeline", "stage"], columns="metric", values="score") - df = df.reset_index() - - return df - -def get_reference_df(df): - # TODO metric names and selection - # only pipeline, stage, metric, normalized - df = df[df["aspect"] == "reference"] - df = add_REI_precision(df) - - df = df[["pipeline", "stage", "metric", "normalized"]] - - metrics = [ - "ReferenceTripleAlignmentMetricSoftEV", - "ReferenceTripleAlignmentMetricSoftE", - "ReferenceTripleAlignmentMetric", - # "ReferenceClassCoverageMetric", - "SourceEntityCoverageMetric", - "SourceEntityCoverageMetricSoft", - "REI_precision", - "TE_ExpectedEntityLinkMetric", - "TE_ExpectedRelationLinkMetric", - "ER_EntityMatchMetric", - "ER_RelationMatchMetric", - ] - - df = df[df["metric"].isin(metrics)] - - df["score"] = df["normalized"].round(2) - - # rename metric to short name - df["metric"] = df["metric"].map(map_metric_name_pretty) - - # make each metric a column - df = df.pivot(index=["pipeline", "stage"], columns="metric", values="score") - df = df.reset_index() - - - return df - -# === Tests === - -def test_wide_table_smoth(): - """ - Stores all metrics in a wide table format. - """ - - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - - # replace pipeline name with name_mapping - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name) - - - # statistics_df - statistics_df = get_statistics_df(metric_df) - # semantic_df - semantic_df = get_semantic_df(metric_df) - # reference_df - reference_df = get_reference_df(metric_df) - - # join all of them on pipeline and stage - df = pd.merge(statistics_df, semantic_df, on=["pipeline", "stage"], how="left") - df = pd.merge(df, reference_df, on=["pipeline", "stage"], how="left") - - # colum order - df = df[["pipeline", "stage"] + [v for k, v in METRIC_NAME_INDEX_PRETTY]] - # print(df) - - df.to_csv(OUTPUT_ROOT / "paper/test_wide_table_smoth.csv", sep="\t") - - -def test_table_with_statistic_metrics(): - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - metrics = ["entity_count", "relation_count", "triple_count", "class_count", "loose_entity_count", "shallow_entity_count"] - - # filter for metrics - metric_df = metric_df[["pipeline", "stage", "metric", "value"]] - duration_df = agg_duration_over_stages_per_pipeline(metric_df) - duration_df = duration_df[["pipeline", "stage", "metric", "value"]] - metric_df = metric_df[metric_df["metric"].isin(metrics)] - - metric_df = pd.concat([metric_df, duration_df]) - - metric_df["metric"] = metric_df["metric"].map(map_metric_name) - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - # only stage = stage_3 - metric_df = metric_df[metric_df["stage"] == "stage_3"] - - # Assuming your dataframe is called df - pivot_df = metric_df.pivot_table( - index=["pipeline", "stage"], # rows - columns="metric", # pivoted column - values="value" # values to fill - ).reset_index() - - # (Optional) Flatten the column index if needed - pivot_df.columns.name = None # remove "metric" header - - # column selection and order Pipeline FC EC RC TC Time - pivot_df = pivot_df[["pipeline", "FC", "EC", "RC", "TC", "SEC", "Time"]] - # save as TSV - output_path = OUTPUT_ROOT / "paper/test_tab_2_statistic_metrics.csv" - pivot_df.to_csv(output_path, sep="\t") - - -def test_table_with_semantic_metrics(): - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - # replace pipeline name with name_mapping - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - # remove details colums - local_metric_df = metric_df.drop(columns=["details"]) - - # only stage = stage_1 and aspect = statistical - stage_3_df = local_metric_df[local_metric_df["stage"] == "stage_3"] - statistical_df = stage_3_df[stage_3_df["aspect"] == "semantic"] - # statistical_df["pipeline"] = statistical_df["pipeline"].map(map_pipeline_name_pretty) - - # print all available metric names - print(statistical_df["metric"].unique()) - - # rename metric to short name and remove metrics that are not in SEM_METRIC_SHORT_NAMES - statistical_df = statistical_df[statistical_df["metric"].isin(list(SEM_METRIC_SHORT_NAMES.keys()))] - statistical_df["metric"] = statistical_df["metric"].map(SEM_METRIC_SHORT_NAMES) - - # format normalized value to 2 decimal places - statistical_df["normalized"] = statistical_df["normalized"].round(3) - - # only stage = stage_3 - statistical_df = statistical_df[statistical_df["stage"] == "stage_3"] - - # make CSV with, x axis: pipeline, y axis: metric_name, cell: value - # Pivot the table: index=metric, columns=pipeline, values=value - pivot_df = statistical_df.pivot(index="metric", columns="pipeline", values="normalized") - # transpose the table - pivot_df = pivot_df.T - - # assume you have a dict SEM_METRIC_LONG_NAMES mapping short->long - long_name_row = {col: SEM_METRIC_LONG_NAMES.get(col, col) for col in pivot_df.columns} - pivot_df = pd.concat([pd.DataFrame([long_name_row], index=["metric_long_name"]), pivot_df]) - - # column selection and order pipeline 𝑂𝐷𝑇 𝑂𝐷 𝑂𝑅 𝑂𝑅𝐷 𝑂𝐿𝑇 𝑂𝐿𝐹 𝑂𝐴𝑣𝑔 - - output_path = OUTPUT_ROOT / "paper/test_tab_3_ssp_semantic_eval.csv" - pivot_df.to_csv(output_path, sep="\t") - -def test_table_with_matching_metrics(): - from moviekg.paper.helpers.getter import TABLE_DISPLAY_NAMES, get_pipeline_stage_metric_dict, ref_entity_matching_f1, ref_relation_matching_f1, ref_json_entity_matching_f1 - - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - metrics = [metric for metric in list(TABLE_DISPLAY_NAMES.keys()) if metric in [ref_entity_matching_f1.__name__, ref_relation_matching_f1.__name__, ref_json_entity_matching_f1.__name__]] - - metric_dict = get_pipeline_stage_metric_dict(metric_df, metrics) - - df_rows = [] - for pipeline, stage_dict in metric_dict.items(): - for stage, metric_dict in stage_dict.items(): - rdf_em_f1 = metric_dict.get(ref_entity_matching_f1.__name__, -1) - json_em_f1 = metric_dict.get(ref_json_entity_matching_f1.__name__, -1) - em_f1 = -1 - if rdf_em_f1 != -1: - em_f1 = rdf_em_f1 - elif json_em_f1 != -1: - em_f1 = json_em_f1 - - rdf_rm_f1 = metric_dict.get(ref_relation_matching_f1.__name__, -1) - json_el_r = -1 # metric_dict.get(ref.__name__, -1) - rm_f1 = -1 - if rdf_rm_f1 != -1: - rm_f1 = rdf_rm_f1 - elif json_el_r != -1: - rm_f1 = json_el_r - - df_rows.append({"pipeline": pipeline, "stage": stage, "EM_f1": em_f1, "RM_f1": rm_f1}) - - # remove -1 rows - df_rows = [row for row in df_rows if row["EM_f1"] != -1 and row["RM_f1"] != -1] - - df = pd.DataFrame(df_rows) - # df = df.pivot(index=["pipeline", "stage"], columns="metric", values="value") - # df = df.reset_index() - output_path = OUTPUT_ROOT / "paper/test_tab_4_matching_metrics.csv" - df.to_csv(output_path, sep="\t") - -def test_table_with_matching_metrics_pr(): - from moviekg.paper.helpers.getter import ( - TABLE_DISPLAY_NAMES, get_pipeline_stage_metric_dict, - ref_entity_matching_f1, ref_entity_matching_p, ref_entity_matching_r, - ref_relation_matching_f1, ref_relation_matching_p, ref_relation_matching_r, - ref_json_entity_matching_f1, ref_json_entity_matching_p, ref_json_entity_matching_r - ) - - - - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - metrics = [ - ref_entity_matching_p.__name__, ref_entity_matching_r.__name__, - ref_relation_matching_p.__name__, ref_relation_matching_r.__name__, - ref_json_entity_matching_p.__name__, ref_json_entity_matching_r.__name__ - ] - - psmd = get_pipeline_stage_metric_dict(metric_df, metrics) - - df_rows = [] - for pipeline, stage_dict in psmd.items(): - for stage, metric_dict in stage_dict.items(): - rdf_em_p = metric_dict.get(ref_entity_matching_p.__name__, -1) - rdf_em_r = metric_dict.get(ref_entity_matching_r.__name__, -1) - json_em_p = metric_dict.get(ref_json_entity_matching_p.__name__, -1) - json_em_r = metric_dict.get(ref_json_entity_matching_r.__name__, -1) - em_p = -1 - em_r = -1 - if rdf_em_p != -1: - em_p = rdf_em_p - em_r = rdf_em_r - elif json_em_p != -1: - em_p = json_em_p - em_r = json_em_r - - # print(json.dumps(metric_dict, indent=4)) - # print("--------------------------------") - - rdf_rm_p = metric_dict.get(ref_relation_matching_p.__name__, -1) - rdf_rm_r = metric_dict.get(ref_relation_matching_r.__name__, -1) - json_rm_p = metric_dict.get(ref_relation_matching_p.__name__, -1) - json_rm_r = metric_dict.get(ref_relation_matching_r.__name__, -1) - - rm_p = -1 - rm_r = -1 - if rdf_rm_p != -1: - rm_p = rdf_rm_p - rm_r = rdf_rm_r - elif json_rm_p != -1: - rm_p = json_rm_p - rm_r = json_rm_r - - df_rows.append({"pipeline": pipeline, "stage": stage, "EM_p": em_p, "EM_r": em_r, "RM_p": rm_p, "RM_r": rm_r}) - - # remove -1 rows - df_rows = [row for row in df_rows if row["EM_p"] != -1 and row["EM_r"] != -1 and row["RM_p"] != -1 and row["RM_r"] != -1] - - df = pd.DataFrame(df_rows) - # df = df.pivot(index=["pipeline", "stage"], columns="metric", values="value") - # df = df.reset_index() - output_path = OUTPUT_ROOT / "paper/test_tab_4_matching_metrics_pr.csv" - df.to_csv(output_path, sep="\t") - -def test_table_with_linking_metrics(): - from moviekg.paper.helpers.getter import TABLE_DISPLAY_NAMES, get_pipeline_stage_metric_dict, ref_entity_linking_r, ref_json_entity_linking_r - - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - metrics = [metric for metric in list(TABLE_DISPLAY_NAMES.keys()) if metric in [ref_entity_linking_r.__name__, ref_json_entity_linking_r.__name__]] - - metric_dict = get_pipeline_stage_metric_dict(metric_df, metrics) - - df_rows = [] - for pipeline, stage_dict in metric_dict.items(): - for stage, metric_dict in stage_dict.items(): - rdf_el_r = metric_dict.get(ref_entity_linking_r.__name__, -1) - json_el_r = metric_dict.get(ref_json_entity_linking_r.__name__, -1) - el_r = -1 - if rdf_el_r != -1: - el_r = rdf_el_r - elif json_el_r != -1: - el_r = json_el_r - - df_rows.append({"pipeline": pipeline, "stage": stage, "EL_r": el_r}) - - # remove -1 rows - df_rows = [row for row in df_rows if row["EL_r"] != -1] - - df = pd.DataFrame(df_rows) - # df = df.pivot(index=["pipeline", "stage"], columns="metric", values="value") - # df = df.reset_index() - output_path = OUTPUT_ROOT / "paper/test_tab_5_linking_metrics.csv" - df.to_csv(output_path, sep="\t") - - -def test_table_6(): - """ - External KG R @inc (film) - EC (no Seed) REI @inc (film) - Pipeline | f1@1 f1@2 f1@3 p@3 | f1@1 f@2 f@3 - """ - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name_pretty) - from moviekg.paper.helpers.getter import ( - get_pipeline_stage_metric_dict, ref_kg_f1, ref_kg_p, ref_kg_r, ref_source_entity_f1, ref_source_entity_p, ref_source_entity_r - ) - - metrics = [ - ref_kg_f1.__name__, ref_kg_p.__name__, ref_kg_r.__name__, ref_source_entity_f1.__name__, ref_source_entity_p.__name__, ref_source_entity_r.__name__ - ] - - psmd = get_pipeline_stage_metric_dict(metric_df, metrics) - # import json - # json.dump(psmd, open(OUTPUT_ROOT / "paper/test_tab_6_metrics.json", "w"), indent=4) - - rows = [] - - round_to = 2 - - for pipeline, stage_dict in psmd.items(): - if pipeline in ["reference", "seed"]: - continue - kg_p = [0, 0, 0] - kg_r = [0, 0, 0] - se_p = [0, 0, 0] - se_r = [0, 0, 0] - - - for stage, metric_dict in stage_dict.items(): - kg_p[int(stage.split("_")[1]) - 1] = round(metric_dict.get(ref_kg_p.__name__, -1), round_to) - kg_r[int(stage.split("_")[1]) - 1] = round(metric_dict.get(ref_kg_r.__name__, -1), round_to) - se_p[int(stage.split("_")[1]) - 1] = round(metric_dict.get(ref_source_entity_p.__name__, -1), round_to) - se_r[int(stage.split("_")[1]) - 1] = round(metric_dict.get(ref_source_entity_r.__name__, -1), round_to) - - rows.append({ - "pipeline": pipeline, - "kg_p@1": kg_p[0], "kg_r@1": kg_r[0], "kg_p@2": kg_p[1], "kg_r@2": kg_r[1], "kg_p@3": kg_p[2], "kg_r@3": kg_r[2], - "se_p@1": se_p[0], "se_r@1": se_r[0], "se_p@2": se_p[1], "se_r@2": se_r[1], "se_p@3": se_p[2], "se_r@3": se_r[2]}) - - df = pd.DataFrame(rows) - output_path = OUTPUT_ROOT / "paper/test_tab_6_reference_alignment.csv" - df.to_csv(output_path, sep="\t") - -def test_table_with_reference_overlap_metrics(): - # "Pipeline Inc. P R F1 ∼P ∼F ∼F1" - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - - # replace pipeline name with name_mapping - metric_df["pipeline"] = metric_df["pipeline"].map(map_pipeline_name) - - metric_names = ["ReferenceTripleAlignmentMetricSoftEV", "ReferenceTripleAlignmentMetricSoftE", "ReferenceTripleAlignmentMetric"] - names_map = { - "ReferenceTripleAlignmentMetricSoftEV": "soft_ev_", - "ReferenceTripleAlignmentMetricSoftE": "soft_e_", - "ReferenceTripleAlignmentMetric": "strict_", - } - - # filter for pipeline in pipeline_types - # global metric_df - # apply filter function - # only stage = stage_1 - metric_df = metric_df[metric_df["stage"] == "stage_3"] - metric_df = metric_df[metric_df["metric"].isin(metric_names)] - metric_df["metric"] = metric_df["metric"].map(names_map) - # order by stage and pipeline - # print(metric_df.pivot_table(index=["pipeline", "metric"], values="normalized", aggfunc="mean")) - - # extract precision, recall from details.json - metric_df["p"] = metric_df["details"].apply(lambda x: json.loads(x)["precision"] if "precision" in json.loads(x) else 0) - metric_df["r"] = metric_df["details"].apply(lambda x: json.loads(x)["recall"] if "recall" in json.loads(x) else 0) - # renmae value to f1 - metric_df["f1"] = metric_df["normalized"] - - # only pipline, metric, p, r, f1 - metric_df = metric_df[["pipeline", "metric", "p", "r", "f1"]] - - # result - df_wide = metric_df.pivot( - index="pipeline", - columns="metric", - values=["p", "r", "f1"] - ) - - # flatten MultiIndex columns - df_wide.columns = [f"{m if m!='' else ''}{k}" for k, m in df_wide.columns] - df_wide = df_wide.reset_index() - - # sort columns by name - df_wide = df_wide[sorted(df_wide.columns)] - # normalize values to 2 decimal places for all coluns except pipeline - df_wide.iloc[:, 1:] = df_wide.iloc[:, 1:].round(2) - - output_path = OUTPUT_ROOT / "paper/test_reference_alignment" - df_wide.to_csv(output_path, sep="\t") - -def test_figure_with_kg_growth(): - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - # remove reference stage_0 - metric_df["pipeline"] = metric_df["pipeline"].replace("json_b2", "json_b") - - metric_df = metric_df[metric_df["stage"] != "stage_0"] - - # filter for pipeline in pipeline_types - # global metric_df - # metric_df = filter_msp_and_reference(metric_df) - sorted_metric_df = metric_df.sort_values(by=["stage", "pipeline"]) - g = plot_growth(sorted_metric_df, metrics=["entity_count", "triple_count"], kind="bar") - g.fig.subplots_adjust(wspace=0.1) - # save as png - g.savefig(OUTPUT_ROOT / "paper/test_fig_both_growth.png") - - -def test_figure_with_entity_class_occurence(): - metric_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") - g = plot_class_occ_4_bar_chart(metric_df) - g.savefig(OUTPUT_ROOT / "paper/test_fig_msp_type_reference.png") - - -# Preset weight configs (kept exactly as used in your original code) -PRESETS = { - "equal": { - "size": 0.25, "semantic": 0.25, "reference": 0.25, "efficiency": 0.25 - }, - # Quantity-focused (your code used 0.5, 0.1, 0.1, 0.3) - "quantity_focused": { - "size": 0.5, "semantic": 0.1, "reference": 0.1, "efficiency": 0.3 - }, - # Quality-focused (your code used 0.0, 0.5, 0.5, 0.0) - "quality_focused": { - "size": 0.0, "semantic": 0.5, "reference": 0.5, "efficiency": 0.0 - }, - # Reference-alignment focused (your code used 0.0, 0.2, 0.8, 0.0) - "reference_alignment_focused": { - "size": 0.0, "semantic": 0.2, "reference": 0.8, "efficiency": 0.0 - }, - "efficiency_oriented": { - "size": 0.2, "semantic": 0.2, "reference": 0.2, "efficiency": 0.4 - }, -} - -psmd_df = load_metrics_from_file(OUTPUT_ROOT / "all_metrics.csv") -psmd = get_pipeline_stage_metric_dict(psmd_df, TABLE_DISPLAY_NAMES.keys()) -psmd = apply_selected_updates(psmd) - -# TODO cleanup -# norm_df, agg_df = aggregate_ranking_df() -# def test_rank_save_norm_df(): -# norm_df["normalized"] = norm_df["normalized"].round(2) -# # to format pipeline, metric_name1... metric_nameN, normalized -# wide = norm_df.pivot(index="pipeline", columns="metric", values="normalized").reset_index() -# wide.to_csv(OUTPUT_ROOT / "paper/test_rank_norm_df.csv", sep="\t") - -# c1 size, c2 sem, c3 ref, c4 eff -def test_rank_equal(): - # _rank_and_save(PRESETS["equal"], "test_rank_equal", agg_df) - _rank_and_save2csv(PRESETS["equal"], "test_rank_equal", psmd) - -def test_rank_quantity_focused(): - #_rank_and_save(PRESETS["quantity_focused"], "test_rank_quantity_focused", agg_df) - _rank_and_save2csv(PRESETS["quantity_focused"], "test_rank_quantity_focused", psmd) - -def test_rank_quality_focused(): - #_rank_and_save(PRESETS["quality_focused"], "test_rank_quality_focused", agg_df) - _rank_and_save2csv(PRESETS["quality_focused"], "test_rank_quality_focused", psmd) - -def test_rank_reference_alignment_focused(): - #_rank_and_save(PRESETS["reference_alignment_focused"], "test_rank_reference_alignment_focused", agg_df) - _rank_and_save2csv(PRESETS["reference_alignment_focused"], "test_rank_reference_alignment_focused", psmd) - -def test_rank_efficiency_oriented(): - #_rank_and_save(PRESETS["efficiency_oriented"], "test_rank_efficiency_oriented", agg_df) - _rank_and_save2csv(PRESETS["efficiency_oriented"], "test_rank_efficiency_oriented", psmd) - -def test_full_ranking_table(): - """ - for each rank table read it and then concatenate them into one table joining on the index - for example: - test_rank_equal.csv: - pipeline combined - 0 json_rdf_text 0.855084 - 1 json_text_rdf 0.867719 - 2 rdf_json_text 0.855081 - 3 rdf_text_json 0.867721 - 4 text_json_rdf 0.864522 - 5 text_rdf_json 0.864522 - test_rank_quantity_focused.csv: - pipeline combined - 0 rdf_json_text 0.950847 - 1 text_rdf_json 0.940847 - 2 json_text_rdf 0.93847 - 3 rdf_text_json 0.920847 - 4 json_rdf_text 0.910847 - 5 text_json_rdf 0.900847 - - the result should be: - pipeline combined - 0 json_rdf_text 0.855084 rdf_json_text_0.950847 - 1 json_text_rdf 0.867719 text_rdf_json_0.940847 - 2 rdf_json_text 0.855081 json_text_rdf_0.93847 - 3 rdf_text_json 0.867721 rdf_text_json_0.920847 - 4 text_json_rdf 0.864522 json_rdf_text_0.910847 - 5 text_rdf_json 0.864522 text_json_rdf_0.900847 - - rename the "combined" column for each to the name of the file - """ - - ranking_files = [ - "test_rank_equal.csv", - "test_rank_quantity_focused.csv", - "test_rank_quality_focused.csv", - "test_rank_reference_alignment_focused.csv", - "test_rank_efficiency_oriented.csv" - ] - - - ranking_files = [OUTPUT_ROOT / "paper" / file for file in ranking_files] - - # Base frame with fixed ranks 0..5 (top to bottom) - result = pd.DataFrame({"rank": range(15)}) - # result = pd.DataFrame() - - for file in ranking_files: - name = Path(file).stem # e.g., "test_rank_equal" - df = pd.read_csv(file, sep="\t") - # Ensure we have at least 6 rows; if more, keep top-6; if fewer, allow NaNs - # df = df.head(6).reset_index(drop=True) - - # pipeline name != reference and reset index - df = df[df["pipeline"] != "reference"] - df["pipeline"] = df["pipeline"].map(PIPLEINE_NAME_MAP) - df = df.reset_index(drop=True) - - - # Build two columns for this file: pipeline + score - sub = pd.DataFrame({ - "rank": df.index, - f"{name.split(".")[0]}_pipe": df["pipeline"], - f"{name.split(".")[0]}_score": df["combined"] - }) - - # Join on rank to keep rows aligned 0..5 - result = result.merge(sub, on="rank", how="left") - - # Make 'rank' the index if you prefer, or keep as a column - result = result.set_index("rank") - - result.to_csv(OUTPUT_ROOT / "paper/test_tab_7_full_ranking_table.csv", sep="\t") diff --git a/experiments/moviekg/src/moviekg/pipelines/helpers.py b/experiments/moviekg/src/moviekg/pipelines/helpers.py index fab8ee3..d47c2e5 100644 --- a/experiments/moviekg/src/moviekg/pipelines/helpers.py +++ b/experiments/moviekg/src/moviekg/pipelines/helpers.py @@ -7,7 +7,7 @@ from kgpipe.generation.loaders import build_from_conf from kgpipe.datasets.multipart_multisource import Dataset -from moviekg.datasets.pipe_out import PipeOut, StageOut +from kgpipe.io.pipe_out import PipeOut, StageOut from moviekg.config import dataset, catalog @@ -70,7 +70,12 @@ def run_helper( tmp_dir = stage_dir / "tmp" tmp_dir.mkdir(parents=True, exist_ok=True) - pipeline = build_from_conf(pipeline_conf, target_data, tmp_dir.as_posix()) + pipeline = build_from_conf( + name=pipeline_name, + conf=pipeline_conf, + target_data=target_data, + data_dir=tmp_dir.as_posix(), + ) stage_dir.mkdir(parents=True, exist_ok=True) diff --git a/experiments/ontologies/scads-papers.owl.ttl b/experiments/ontologies/scads-papers.owl.ttl new file mode 100644 index 0000000..87c4280 --- /dev/null +++ b/experiments/ontologies/scads-papers.owl.ttl @@ -0,0 +1,221 @@ +@prefix : . +@prefix owl: . +@prefix rdfs: . + +######## +# Classes +######## + +:ScientificPaper a owl:Class . + +:ContentUnit a owl:Class . +:RhetoricalUnit a owl:Class ; rdfs:subClassOf :ContentUnit . +:ScientificContribution a owl:Class ; rdfs:subClassOf :ContentUnit . + +:ResearchProblem a owl:Class ; rdfs:subClassOf :ScientificContribution . +:ResearchQuestion a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Motivation a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Goal a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Hypothesis a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Claim a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Method a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Material a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Dataset a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Experiment a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Model a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Observation a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Result a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Conclusion a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Limitation a owl:Class ; rdfs:subClassOf :ScientificContribution . +:FutureWork a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Evidence a owl:Class ; rdfs:subClassOf :ScientificContribution . +:RelatedWorkStatement a owl:Class ; rdfs:subClassOf :ScientificContribution . +:Concept a owl:Class . +:Variable a owl:Class . +:Metric a owl:Class . + +######## +# Paper -> content +######## + +:hasContentUnit a owl:ObjectProperty ; + rdfs:domain :ScientificPaper ; + rdfs:range :ContentUnit . + +:hasProblem a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :ResearchProblem . + +:hasResearchQuestion a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :ResearchQuestion . + +:hasMotivation a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Motivation . + +:hasGoal a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Goal . + +:hasHypothesis a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Hypothesis . + +:hasClaim a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Claim . + +:hasMethod a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Method . + +:hasMaterial a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Material . + +:hasDataset a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Dataset . + +:hasExperiment a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Experiment . + +:hasModel a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Model . + +:hasObservation a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Observation . + +:hasResult a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Result . + +:hasConclusion a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Conclusion . + +:hasLimitation a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :Limitation . + +:hasFutureWork a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :FutureWork . + +:hasRelatedWorkStatement a owl:ObjectProperty ; + rdfs:subPropertyOf :hasContentUnit ; + rdfs:domain :ScientificPaper ; + rdfs:range :RelatedWorkStatement . + +######## +# Internal semantics +######## + +:addressesProblem a owl:ObjectProperty ; + rdfs:domain :Method ; + rdfs:range :ResearchProblem . + +:investigatesQuestion a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :ResearchQuestion . + +:testsHypothesis a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Hypothesis . + +:usesMethod a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Method . + +:usesMaterial a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Material . + +:usesDataset a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Dataset . + +:studiesConcept a owl:ObjectProperty ; + rdfs:domain :ScientificContribution ; + rdfs:range :Concept . + +:hasVariable a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Variable . + +:usesMetric a owl:ObjectProperty ; + rdfs:domain :Result ; + rdfs:range :Metric . + +:producesObservation a owl:ObjectProperty ; + rdfs:domain :Experiment ; + rdfs:range :Observation . + +:supportsClaim a owl:ObjectProperty ; + rdfs:domain :Evidence ; + rdfs:range :Claim . + +:reportsEvidence a owl:ObjectProperty ; + rdfs:domain :Result ; + rdfs:range :Evidence . + +:derivedFromObservation a owl:ObjectProperty ; + rdfs:domain :Result ; + rdfs:range :Observation . + +:supports a owl:ObjectProperty ; + rdfs:domain :ScientificContribution ; + rdfs:range :ScientificContribution . + +:contradicts a owl:ObjectProperty ; + rdfs:domain :ScientificContribution ; + rdfs:range :ScientificContribution . + +:extends a owl:ObjectProperty ; + rdfs:domain :ScientificContribution ; + rdfs:range :ScientificContribution . + +:motivates a owl:ObjectProperty ; + rdfs:domain :Motivation ; + rdfs:range :Goal . + +:answers a owl:ObjectProperty ; + rdfs:domain :Conclusion ; + rdfs:range :ResearchQuestion . + +:basedOn a owl:ObjectProperty ; + rdfs:domain :Conclusion ; + rdfs:range :Result . + +:hasLimitationOn a owl:ObjectProperty ; + rdfs:domain :Limitation ; + rdfs:range :Method . + +######## +# Optional rhetorical typing +######## + +:IntroductionUnit a owl:Class ; rdfs:subClassOf :RhetoricalUnit . +:MethodsUnit a owl:Class ; rdfs:subClassOf :RhetoricalUnit . +:ResultsUnit a owl:Class ; rdfs:subClassOf :RhetoricalUnit . +:DiscussionUnit a owl:Class ; rdfs:subClassOf :RhetoricalUnit . diff --git a/experiments/ontologies/scads-papers.ttl b/experiments/ontologies/scads-papers.ttl new file mode 100644 index 0000000..e18623c --- /dev/null +++ b/experiments/ontologies/scads-papers.ttl @@ -0,0 +1,38 @@ +@prefix : . + +:paper1 a :ScientificPaper ; + :hasProblem :problem1 ; + :hasGoal :goal1 ; + :hasMethod :method1 ; + :hasExperiment :exp1 ; + :hasObservation :obs1 ; + :hasResult :result1 ; + :hasClaim :claim1 ; + :hasConclusion :concl1 . + +:problem1 a :ResearchProblem . +:goal1 a :Goal . +:method1 a :Method ; + :addressesProblem :problem1 . + +:exp1 a :Experiment ; + :usesMethod :method1 ; + :testsHypothesis :hyp1 ; + :producesObservation :obs1 . + +:hyp1 a :Hypothesis . +:obs1 a :Observation . + +:result1 a :Result ; + :derivedFromObservation :obs1 . + +:evidence1 a :Evidence ; + :supportsClaim :claim1 . + +:result1 :reportsEvidence :evidence1 . + +:claim1 a :Claim ; + :supports :goal1 . + +:concl1 a :Conclusion ; + :basedOn :result1 . diff --git a/experiments/ontologies/src/onto_chat.py b/experiments/ontologies/src/onto_chat.py new file mode 100644 index 0000000..a4c5b58 --- /dev/null +++ b/experiments/ontologies/src/onto_chat.py @@ -0,0 +1,404 @@ +"""Streamlit ontology chat prototype. + +Run: + uv run streamlit run experiments/ontologies/src/onto_chat.py +""" + +from __future__ import annotations + +from dataclasses import dataclass +import importlib +import os +import re +from textwrap import dedent + +import streamlit as st +import streamlit.components.v1 as components +from rdflib import Graph, RDF, RDFS, URIRef +from rdflib.namespace import OWL + + +EXAMPLE_OWL = dedent( + """\ + @prefix ex: . + @prefix rdf: . + @prefix rdfs: . + @prefix owl: . + + ex:Person a owl:Class . + ex:Company a owl:Class . + ex:Project a owl:Class . + + ex:worksFor a owl:ObjectProperty ; + rdfs:domain ex:Person ; + rdfs:range ex:Company . + + ex:worksOn a owl:ObjectProperty ; + rdfs:domain ex:Person ; + rdfs:range ex:Project . + """ +) + +DEFAULT_OPENAI_MODEL = "gpt-4o-mini" +KNOWN_PREFIXES = { + "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#", + "rdfs": "http://www.w3.org/2000/01/rdf-schema#", + "owl": "http://www.w3.org/2002/07/owl#", + "xsd": "http://www.w3.org/2001/XMLSchema#", +} + + +@dataclass +class OntologySchema: + classes: list[str] + object_edges: list[tuple[str, str, str]] + datatype_edges: list[tuple[str, str, str]] + + +def short_name(uri: URIRef) -> str: + """Return a compact local name for URI nodes.""" + text = str(uri) + if "#" in text: + return text.rsplit("#", maxsplit=1)[-1] + if "/" in text: + return text.rstrip("/").rsplit("/", maxsplit=1)[-1] + return text + + +def parse_graph(raw_text: str, rdf_format: str) -> Graph: + """Parse ontology text into an RDF graph.""" + graph = Graph() + graph.parse(data=raw_text, format=rdf_format) + return graph + + +def extract_schema(graph: Graph) -> OntologySchema: + """Extract classes and property relations from graph.""" + classes: set[str] = set() + object_edges: list[tuple[str, str, str]] = [] + datatype_edges: list[tuple[str, str, str]] = [] + + for cls in graph.subjects(RDF.type, OWL.Class): + if isinstance(cls, URIRef): + classes.add(short_name(cls)) + for cls in graph.subjects(RDF.type, RDFS.Class): + if isinstance(cls, URIRef): + classes.add(short_name(cls)) + + for prop in graph.subjects(RDF.type, OWL.ObjectProperty): + if not isinstance(prop, URIRef): + continue + prop_name = short_name(prop) + domains = [d for d in graph.objects(prop, RDFS.domain) if isinstance(d, URIRef)] + ranges = [r for r in graph.objects(prop, RDFS.range) if isinstance(r, URIRef)] + for domain in domains or [URIRef("UnknownDomain")]: + for rng in ranges or [URIRef("UnknownRange")]: + src, dst = short_name(domain), short_name(rng) + classes.update([src, dst]) + object_edges.append((src, prop_name, dst)) + + for prop in graph.subjects(RDF.type, OWL.DatatypeProperty): + if not isinstance(prop, URIRef): + continue + prop_name = short_name(prop) + domains = [d for d in graph.objects(prop, RDFS.domain) if isinstance(d, URIRef)] + ranges = [r for r in graph.objects(prop, RDFS.range) if isinstance(r, URIRef)] + for domain in domains or [URIRef("UnknownDomain")]: + for rng in ranges or [URIRef("Literal")]: + src, dst = short_name(domain), short_name(rng) + classes.add(src) + datatype_edges.append((src, prop_name, dst)) + + return OntologySchema( + classes=sorted(classes), + object_edges=object_edges, + datatype_edges=datatype_edges, + ) + + +def to_mermaid(schema: OntologySchema) -> str: + """Serialize ontology schema as Mermaid classDiagram.""" + lines = ["classDiagram"] + for cls_name in schema.classes: + lines.append(f" class {cls_name}") + for src, rel, dst in schema.object_edges: + lines.append(f" {src} --> {dst} : {rel}") + for src, rel, dst in schema.datatype_edges: + lines.append(f" {src} : {rel} -> {dst}") + return "\n".join(lines) + + +def render_mermaid(mermaid_text: str) -> None: + """Render Mermaid diagram in Streamlit via embedded HTML.""" + escaped = ( + mermaid_text.replace("&", "&") + .replace("<", "<") + .replace(">", ">") + ) + html = f""" +
{escaped}
+ + + """ + components.html(html, height=500, scrolling=True) + + +def draft_llm_prompt(user_request: str, ontology_text: str, rdf_format: str) -> str: + """Build a prompt for a future LLM integration.""" + return dedent( + f"""\ + You are editing an OWL ontology. + + Task: + {user_request} + + Requirements: + - Return only ontology text in {rdf_format} format. + - Preserve existing prefixes when possible. + - Declare all prefixes you use (especially xsd when using xsd:* datatypes). + - Keep edits minimal and valid. + - Do not include markdown fences. + + Current ontology: + {ontology_text} + """ + ) + + +def strip_markdown_fences(text: str) -> str: + """Remove markdown code fences if model returns them.""" + cleaned = text.strip() + if cleaned.startswith("```") and cleaned.endswith("```"): + lines = cleaned.splitlines() + if len(lines) >= 2: + return "\n".join(lines[1:-1]).strip() + return cleaned + + +def extract_declared_prefixes(text: str) -> set[str]: + """Extract declared prefixes from Turtle/N3 text.""" + return set(re.findall(r"@prefix\s+([A-Za-z][\w\-]*)\s*:", text)) + + +def extract_used_prefixes(text: str) -> set[str]: + """Extract prefixed terms used in Turtle/N3 text.""" + matches = re.findall(r"(? tuple[str, list[str]]: + """Inject known prefix declarations when terms use undeclared prefixes.""" + declared = extract_declared_prefixes(text) + used = extract_used_prefixes(text) + missing = sorted((used - declared) & set(KNOWN_PREFIXES)) + if not missing: + return text, [] + + injections = [f"@prefix {p}: <{KNOWN_PREFIXES[p]}> ." for p in missing] + updated = "\n".join(injections) + "\n" + text.lstrip() + return updated, missing + + +def validate_and_normalize_ontology(raw_text: str, rdf_format: str) -> tuple[str, list[str]]: + """Normalize and validate returned ontology text.""" + normalized = raw_text.strip() + added_prefixes: list[str] = [] + if rdf_format in {"turtle", "n3"}: + normalized, added_prefixes = inject_missing_known_prefixes(normalized) + parse_graph(normalized, rdf_format) + return normalized, added_prefixes + + +def request_ontology_edit(prompt: str, model: str) -> str: + """Call OpenAI and return ontology text.""" + api_key = os.getenv("OPENAI_API_KEY") + if not api_key: + raise RuntimeError("OPENAI_API_KEY is not set.") + + try: + openai_module = importlib.import_module("openai") + openai_client = getattr(openai_module, "OpenAI") + except Exception as exc: # noqa: BLE001 + raise RuntimeError( + "The 'openai' package is required. Install it with: uv add openai" + ) from exc + + client = openai_client(api_key=api_key) + response = client.chat.completions.create( + model=model, + temperature=0, + messages=[ + { + "role": "system", + "content": ( + "You edit OWL ontologies. Return only ontology text in the requested " + "serialization format. Do not add markdown." + ), + }, + {"role": "user", "content": prompt}, + ], + ) + content = response.choices[0].message.content or "" + if not content.strip(): + raise RuntimeError("OpenAI returned an empty response.") + return strip_markdown_fences(content) + + +def request_ontology_syntax_fix( + ontology_text: str, + rdf_format: str, + parse_error: Exception, + model: str, +) -> str: + """Ask OpenAI for a syntax-only repair of ontology text.""" + prompt = dedent( + f"""\ + Fix the syntax of this ontology serialization. + + Requirements: + - Return only ontology text in {rdf_format}. + - Preserve meaning; only fix syntax/prefix issues. + - Ensure all used prefixes are declared. + - Do not include markdown fences. + + Parser error: + {parse_error} + + Ontology text: + {ontology_text} + """ + ) + return request_ontology_edit(prompt=prompt, model=model) + + +def init_state() -> None: + """Initialize app session state keys.""" + st.session_state.setdefault("ontology_text", EXAMPLE_OWL) + st.session_state.setdefault("rdf_format", "turtle") + st.session_state.setdefault("messages", []) + st.session_state.setdefault("last_llm_prompt", "") + st.session_state.setdefault("last_llm_response", "") + st.session_state.setdefault("last_normalized_response", "") + st.session_state.setdefault("openai_model", DEFAULT_OPENAI_MODEL) + + +def main() -> None: + st.set_page_config(page_title="Ontology Chat Draft", layout="wide") + st.title("Ontology Chat + Mermaid (Draft)") + st.caption("Prototype UI for OWL editing with chat-driven change requests.") + + init_state() + + left_col, right_col = st.columns([1, 1], gap="large") + + with left_col: + st.subheader("Ontology Text") + st.session_state.rdf_format = st.selectbox( + "RDF format", + options=["turtle", "xml", "nt", "n3"], + index=["turtle", "xml", "nt", "n3"].index(st.session_state.rdf_format), + ) + st.session_state.openai_model = st.text_input( + "OpenAI model", + value=st.session_state.openai_model, + help="Requires OPENAI_API_KEY in environment.", + ) + st.session_state.ontology_text = st.text_area( + "Edit ontology", + value=st.session_state.ontology_text, + height=340, + ) + + st.subheader("Chat") + for msg in st.session_state.messages: + with st.chat_message(msg["role"]): + st.markdown(msg["content"]) + + user_request = st.chat_input("Describe ontology change...") + if user_request: + st.session_state.messages.append({"role": "user", "content": user_request}) + prompt = draft_llm_prompt( + user_request=user_request, + ontology_text=st.session_state.ontology_text, + rdf_format=st.session_state.rdf_format, + ) + st.session_state.last_llm_prompt = prompt + try: + with st.spinner("Requesting ontology update from OpenAI..."): + model_name = st.session_state.openai_model.strip() or DEFAULT_OPENAI_MODEL + edited_ontology = request_ontology_edit( + prompt=prompt, + model=model_name, + ) + st.session_state.last_llm_response = edited_ontology + try: + normalized_ontology, added_prefixes = validate_and_normalize_ontology( + edited_ontology, st.session_state.rdf_format + ) + except Exception as parse_exc: # noqa: BLE001 + with st.spinner("Attempting syntax repair..."): + repaired = request_ontology_syntax_fix( + ontology_text=edited_ontology, + rdf_format=st.session_state.rdf_format, + parse_error=parse_exc, + model=model_name, + ) + st.session_state.last_llm_response = repaired + normalized_ontology, added_prefixes = validate_and_normalize_ontology( + repaired, st.session_state.rdf_format + ) + + st.session_state.ontology_text = normalized_ontology + st.session_state.last_normalized_response = normalized_ontology + prefix_note = "" + if added_prefixes: + prefix_note = f" Added missing prefixes: {', '.join(added_prefixes)}." + st.session_state.messages.append( + { + "role": "assistant", + "content": ( + "Applied OpenAI ontology update and refreshed Mermaid diagram." + f"{prefix_note}" + ), + } + ) + except Exception as exc: # noqa: BLE001 + st.session_state.messages.append( + { + "role": "assistant", + "content": ( + "OpenAI request failed after validation/repair attempts: " + f"{exc}" + ), + } + ) + st.rerun() + + with st.expander("Last drafted LLM prompt", expanded=False): + st.code(st.session_state.last_llm_prompt or "No prompt drafted yet.", language="text") + with st.expander("Last OpenAI response", expanded=False): + st.code(st.session_state.last_llm_response or "No model response yet.", language="text") + with st.expander("Last normalized ontology", expanded=False): + st.code( + st.session_state.last_normalized_response or "No normalized ontology yet.", + language="text", + ) + + with right_col: + st.subheader("Mermaid Render") + try: + graph = parse_graph(st.session_state.ontology_text, st.session_state.rdf_format) + schema = extract_schema(graph) + mermaid = to_mermaid(schema) + render_mermaid(mermaid) + with st.expander("Mermaid source", expanded=False): + st.code(mermaid, language="text") + except Exception as exc: # noqa: BLE001 + st.error(f"Could not parse ontology: {exc}") + st.info("Check RDF format and ontology syntax in the left panel.") + + +if __name__ == "__main__": + main() diff --git a/experiments/ontologies/src/onto_diff.py b/experiments/ontologies/src/onto_diff.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/.gitignore b/experiments/param-opti/.gitignore new file mode 100644 index 0000000..3895a6e --- /dev/null +++ b/experiments/param-opti/.gitignore @@ -0,0 +1,8 @@ +output/ +repos/ +output_qap_mock/ +testdata/ +tmp/ +data/ +data +backlog/ \ No newline at end of file diff --git a/experiments/param-opti/README.md b/experiments/param-opti/README.md new file mode 100644 index 0000000..4c5f40a --- /dev/null +++ b/experiments/param-opti/README.md @@ -0,0 +1,93 @@ +# Pipeline configuration search + +Search over KG integration pipeline configs (task selection + parameters) to maximize evaluation quality against a reference KG. Supports **RDF** (graph alignment / fusion) and **text** (IE → linking → RDF → fusion) pipelines on the MovieKG benchmark. + +Experiment results can be found at https://github.com/Vehnem/kgpipe-experiment-results/tree/main/parameter_search + +## Layout + +| Path | Role | +|------|------| +| `src/experiment.py` | Live search: propose configs → run pipeline → evaluate → write results | +| `src/execute.py` | Run/evaluate fixed config fixtures (sampled or exhaustive) | +| `src/analyse.py` | Offline search simulation on a cached `results.json` | +| `src/plot_search_evolution.py` | Per-seed evolution plots/tables | +| `src/plot_search_evolution_aggregate.py` | Mean ± band across RNG seeds | +| `src/kgpipe_search/` | Search space, strategies, ranking, evaluation | +| `scripts/` | Reproducible experiment drivers | +| `data/` | Symlink to MovieKG bench data (`kgpipe-parameters/latest`) | +| `runs/` | Pipeline artifacts + search result summaries | + +## Search strategies + +| Flag | Behavior | +|------|----------| +| `random` | Uniform sample over exhaustive valid configs | +| `implementation_aware` | Systematic task-combo coverage, random params | +| `qgns` | Restricted neighborhood search (RNS in plots) | +| `hnr` / `hnr_2` | Hierarchical neighborhood refinement | +| `bayesian` | Surrogate + acquisition over a candidate pool | +| `llm` | LLM-proposed configs (needs `KGPipe_SEARCH_LLM_*` env vars) | + +Objective score comes from `--rank-aggregation` (`default` | `flat_hmean` | `custom`), applied to cached metric measurements in `.eval.json`. + +## Quick start + +From the **repo root**, with `.venv` and `experiments/param-opti/data` pointing at the bench dataset: + +```bash +cd experiments/param-opti + +# One seed, all strategies (RDF or text) +bash scripts/rdf_experiments.sh 0 +bash scripts/text_experiments.sh 0 + +# Multi-seed sweep (seeds: 0 42 1337 1–7) +bash full.sh + +# Per-seed + aggregated plots +bash scripts/call_plot.sh 0 +``` + +Results land under: + +``` +runs/{rdf,text}/ # pipeline caches (by config hash) +runs/{rdf,text}-search-results_rank_/init__budget__seed_/ + {random,implementation-aware,qgns,hnr,hnr_2,bayesian}-results.json +``` + +## Single experiment + +```bash +export PYTHONPATH=src:/src +python src/experiment.py \ + --seed data/bench/moviekg/split_0/kg/seed/data.nt \ + --source data/bench/moviekg/split_1/sources/rdf/data.nt \ + --reference data/bench/moviekg/split_1/kg/reference/data_agg.nt \ + --ontology data/bench/moviekg/ontology.ttl \ + --pipeline-type rdf \ + --strategy hnr_2 \ + --budget 20 --init-budget 1 \ + --init-strategy implementation_aware \ + --rank-aggregation custom \ + --rng-seed 0 \ + --output-dir runs/rdf \ + --results runs/rdf-search-results_rank_custom/init_1_budget_20_seed_0/hnr_2-results.json +``` + +For text, use `--pipeline-type text` and `--source data/bench/moviekg/split_1/sources/text/data/`. + +## Related scripts + +- `scripts/call_execute.sh` / `call_text_execute.sh` — fixture execution via `execute.py` +- `scripts/call_analyse-offline.sh` — replay strategies on a cached results file +- `scripts/rdf_hnr2_params.sh` / `run_multi` — HNR-2 hyperparameter sweeps + +## Lint + +```bash +.venv/bin/ruff check --fix experiments/param-opti/src/kgpipe_search +``` + +`kgpipe_search.dev` is intended to move into the core KGpipe API. diff --git a/experiments/param-opti/src/analyse.py b/experiments/param-opti/src/analyse.py new file mode 100644 index 0000000..0912951 --- /dev/null +++ b/experiments/param-opti/src/analyse.py @@ -0,0 +1,623 @@ +#!/usr/bin/env python3 +""" +Offline analysis of search strategies on already computed pipeline eval results. + +This script treats a `results.json` (as written by `experiment.py`) as a cache: +- some configs have an evaluation score (status == "ok") +- some configs are missing or errored (partial results) + +We can then "simulate" different search strategies without re-running any pipeline by +letting the strategy propose configs and looking them up in the cache. +""" + +import argparse +import json +import math +import random +from pathlib import Path +from typing import Any, Dict, List, NamedTuple, Optional, Sequence, Set, Tuple + +# Optional integration with the existing kgpipe_search strategies. +try: + from kgpipe_search.configuration import pipeline_config_snapshot_key + from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE, PipelineConfig + from kgpipe_search.search import bayesian_optimization, hnr_search, qgns_search, random_search + + _HAS_KGPIPE_SEARCH = True + _KGPIPE_SEARCH_IMPORT_ERROR: Exception | None = None +except Exception as exc: + _HAS_KGPIPE_SEARCH = False + _KGPIPE_SEARCH_IMPORT_ERROR = exc + + +class Candidate(NamedTuple): + config_hash: str + task_key: Tuple[str, ...] + snapshot_path: Optional[Path] + + +class StepLog(NamedTuple): + step: int + proposed_hash: str + hit: bool + score: Optional[float] + best_score: Optional[float] + misses_so_far: int + + +def _read_json(path: Path) -> Any: + return json.loads(path.read_text(encoding="utf-8")) + + +def _snapshot_key_from_snapshot_dict(snapshot: Dict[str, Any]) -> str: + # Must match kgpipe_search.configuration.pipeline_config_snapshot_key()'s serialization: + # json.dumps(snapshot, sort_keys=True) + return json.dumps(snapshot, sort_keys=True) + + +def _load_score_by_snapshot_key( + *, + results_path: Path, + entry_by_hash: Dict[str, Dict[str, Any]], + score_by_hash: Dict[str, float], + output_dir: Optional[str] = None, +) -> Dict[str, float]: + """ + Map config snapshots (serialized canonical JSON) -> score. + This lets us evaluate PipelineConfig objects sampled by kgpipe_search strategies. + """ + score_by_key: Dict[str, float] = {} + for h, entry in entry_by_hash.items(): + score = score_by_hash.get(h) + if score is None: + continue + snapshot_path = _resolve_snapshot_path( + results_path, entry.get("config_path"), output_dir=output_dir + ) + if snapshot_path is None: + continue + try: + snap = _read_json(snapshot_path) + if isinstance(snap, dict): + key = _snapshot_key_from_snapshot_dict(snap) + score_by_key[key] = float(score) + except Exception: + continue + return score_by_key + + +class OfflineCacheOracle: + def __init__(self, score_by_snapshot_key: Dict[str, float], *, miss_score: float = 0.5) -> None: + self._score_by_key = score_by_snapshot_key + self.miss_score = float(miss_score) + self.hits = 0 + self.misses = 0 + + def evaluate(self, cfg: "PipelineConfig") -> float: + key = pipeline_config_snapshot_key(cfg, RDF_SEARCH_SPACE) + score = self._score_by_key.get(key) + if score is None: + self.misses += 1 + return self.miss_score + self.hits += 1 + return float(score) + +def _load_cache( + results_path: Path, +) -> Tuple[Dict[str, float], Dict[str, Dict[str, Any]], Optional[str]]: + """ + Returns: + score_by_hash: config_hash -> final_score (only status == ok) + entry_by_hash: config_hash -> raw results entry (all statuses) + output_dir: experiment output directory from results payload, if present + """ + payload = _read_json(results_path) + results = payload.get("results") + if not isinstance(results, list): + raise ValueError(f"Expected 'results' list in {results_path}") + + entry_by_hash: Dict[str, Dict[str, Any]] = {} + score_by_hash: Dict[str, float] = {} + + for item in results: + if not isinstance(item, dict): + continue + h = item.get("config_hash") + if not isinstance(h, str): + continue + entry_by_hash[h] = item + if item.get("status") == "ok": + evaluation = item.get("evaluation") or {} + if isinstance(evaluation, dict) and isinstance(evaluation.get("final_score"), (int, float)): + score_by_hash[h] = float(evaluation["final_score"]) + + output_dir = payload.get("output_dir") + if not isinstance(output_dir, str): + output_dir = None + + return score_by_hash, entry_by_hash, output_dir + + +def _resolve_snapshot_path( + results_path: Path, + raw_path: Optional[str], + *, + output_dir: Optional[str] = None, +) -> Optional[Path]: + if not raw_path: + return None + + p = Path(raw_path) + candidates: List[Path] = [] + + if p.is_absolute(): + candidates.append(p) + else: + # Paths in results.json are relative to the cwd used when running experiment.py. + candidates.append(Path.cwd() / p) + candidates.append(results_path.parent / p) + candidates.append(results_path.parent / p.name) + if output_dir: + candidates.append(Path.cwd() / output_dir / p.name) + if results_path.parent.name == Path(output_dir).name: + candidates.append(results_path.parent / p.name) + + seen: Set[str] = set() + for candidate in candidates: + key = str(candidate) + if key in seen: + continue + seen.add(key) + if candidate.exists(): + return candidate + return None + + +def _load_candidates( + *, + results_path: Path, + entry_by_hash: Dict[str, Dict[str, Any]], + include_missing_snapshots: bool, + output_dir: Optional[str] = None, +) -> List[Candidate]: + candidates: List[Candidate] = [] + for h, entry in entry_by_hash.items(): + snapshot_path = _resolve_snapshot_path( + results_path, entry.get("config_path"), output_dir=output_dir + ) + if snapshot_path is None and not include_missing_snapshots: + continue + + task_key: Tuple[str, ...] = () + if snapshot_path is not None: + try: + snapshot = _read_json(snapshot_path) + task_keys = snapshot.get("task_keys") + if isinstance(task_keys, list) and all(isinstance(x, str) for x in task_keys): + task_key = tuple(task_keys) + except Exception: + task_key = () + + candidates.append(Candidate(config_hash=h, task_key=task_key, snapshot_path=snapshot_path)) + + return candidates + + +class Strategy: + name: str + + def propose(self) -> str: # returns config_hash + raise NotImplementedError + + def observe(self, config_hash: str, score: Optional[float]) -> None: + # score is None for cache miss or non-ok result + return + + +class RandomStrategy(Strategy): + name = "random" + + def __init__(self, rng: random.Random, universe: Sequence[Candidate]) -> None: + self._rng = rng + self._universe = universe + + def propose(self) -> str: + return self._rng.choice(self._universe).config_hash + + +class GreedyKnownStrategy(Strategy): + """ + Upper bound / sanity check: picks the best already-known score. + Useful to verify the harness and to see what "best possible" would be in the cache. + """ + + name = "greedy-known" + + def __init__(self, rng: random.Random, universe: Sequence[Candidate], score_by_hash: Dict[str, float]) -> None: + self._rng = rng + self._universe = universe + self._score_by_hash = score_by_hash + self._ordered: List[str] = [ + c.config_hash for c in sorted(universe, key=lambda c: score_by_hash.get(c.config_hash, float("-inf")), reverse=True) + ] + self._i = 0 + + def propose(self) -> str: + if self._i >= len(self._ordered): + return self._rng.choice(self._universe).config_hash + h = self._ordered[self._i] + self._i += 1 + return h + + +class UCBByTaskKeyStrategy(Strategy): + """ + Lightweight bandit baseline: + - treat each distinct task pipeline (task_keys tuple) as an arm + - within an arm, sample configs uniformly + - update arm rewards based on observed scores + + This is robust to partial caches: misses just don't update the arm. + """ + + name = "ucb-taskkey" + + def __init__(self, rng: random.Random, universe: Sequence[Candidate], exploration: float = 2.0) -> None: + self._rng = rng + self._exploration = exploration + + arms: Dict[Tuple[str, ...], List[str]] = {} + for c in universe: + arms.setdefault(c.task_key, []).append(c.config_hash) + self._arms = arms + self._arm_keys = list(arms.keys()) + + self._n_total = 0 + self._n: Dict[Tuple[str, ...], int] = {k: 0 for k in self._arm_keys} + self._mean: Dict[Tuple[str, ...], float] = {k: 0.0 for k in self._arm_keys} + + def propose(self) -> str: + # Ensure each arm is tried at least once + for k in self._arm_keys: + if self._n[k] == 0: + return self._rng.choice(self._arms[k]) + + # Standard UCB1 over arms + self._n_total = max(1, self._n_total) + best_k = None + best_ucb = float("-inf") + for k in self._arm_keys: + bonus = math.sqrt((self._exploration * math.log(self._n_total)) / self._n[k]) + ucb = self._mean[k] + bonus + if ucb > best_ucb: + best_ucb = ucb + best_k = k + assert best_k is not None + return self._rng.choice(self._arms[best_k]) + + def observe(self, config_hash: str, score: Optional[float]) -> None: + self._n_total += 1 + if score is None: + return + # find arm by scanning (cheap at this scale); if needed we can add hash->arm map later + for k, hashes in self._arms.items(): + if config_hash in hashes: + n = self._n[k] + 1 + prev = self._mean[k] + self._mean[k] = prev + (score - prev) / n + self._n[k] = n + return + + +def _build_strategy( + *, + name: str, + rng: random.Random, + universe: Sequence[Candidate], + score_by_hash: Dict[str, float], + exploration: float, +) -> Strategy: + if name == "random": + return RandomStrategy(rng, universe) + if name == "greedy-known": + return GreedyKnownStrategy(rng, universe, score_by_hash) + if name == "ucb-taskkey": + return UCBByTaskKeyStrategy(rng, universe, exploration=exploration) + raise ValueError(f"Unknown strategy {name!r}") + + +def _simulate( + *, + strategy: Strategy, + score_by_hash: Dict[str, float], + budget: int, + miss_policy: str, + max_resample: int, +) -> List[StepLog]: + """ + miss_policy: + - "count": a miss consumes budget and is recorded as hit=False + - "resample": keep resampling (up to max_resample) within the same step until hit, else count miss + """ + logs: List[StepLog] = [] + best: Optional[float] = None + misses = 0 + + for step in range(1, budget + 1): + proposed = strategy.propose() + + score = score_by_hash.get(proposed) + hit = score is not None + + if (not hit) and miss_policy == "resample": + tries = 0 + while tries < max_resample and not hit: + tries += 1 + proposed = strategy.propose() + score = score_by_hash.get(proposed) + hit = score is not None + + if not hit: + misses += 1 + strategy.observe(proposed, None) + else: + strategy.observe(proposed, score) + best = score if best is None else max(best, score) + + logs.append( + StepLog( + step=step, + proposed_hash=proposed, + hit=hit, + score=score, + best_score=best, + misses_so_far=misses, + ) + ) + + return logs + + +def _summarize(logs: Sequence[StepLog]) -> Dict[str, Any]: + hits = sum(1 for x in logs if x.hit) + misses = len(logs) - hits + best = next((x.best_score for x in reversed(logs) if x.best_score is not None), None) + return { + "budget": len(logs), + "hits": hits, + "misses": misses, + "hit_rate": hits / len(logs) if logs else 0.0, + "best_score": best, + } + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(description="Offline search strategy analysis on cached eval results.") + p.add_argument( + "--results", + type=Path, + default=Path(__file__).resolve().parent.parent / "results.json", + help="Path to results.json written by experiment.py (default: experiments/param-opti/results.json)", + ) + p.add_argument( + "--strategy", + choices=["random", "ucb-taskkey", "greedy-known", "kgpipe_random", "kgpipe_qgns", "kgpipe_hnr", "kgpipe_bayes"], + default="random", + help="Search strategy to simulate (greedy-known is an upper-bound baseline).", + ) + p.add_argument( + "--miss-score", + type=float, + default=0.5, + help="When using kgpipe_* strategies, score to return for cache misses.", + ) + p.add_argument( + "--init-budget", + type=int, + default=3, + help="Initialization budget for kgpipe_qgns/kgpipe_hnr/kgpipe_bayes.", + ) + p.add_argument( + "--init-strategy", + choices=["random", "implementation_aware"], + default="implementation_aware", + help="Initialization strategy for kgpipe_* strategies.", + ) + p.add_argument("--k", type=int, default=3, help="Top-k anchors for kgpipe_qgns.") + p.add_argument("--rho", type=float, default=0.2, help="Exploration probability for kgpipe_qgns/kgpipe_hnr.") + p.add_argument("--pool-size", type=int, default=32, help="Candidate pool size for kgpipe_bayes.") + p.add_argument("--beta", type=float, default=0.5, help="Acquisition beta for kgpipe_bayes.") + p.add_argument("--budget", type=int, default=50, help="Number of proposals to simulate.") + p.add_argument("--seed", type=int, default=0, help="RNG seed for reproducibility.") + p.add_argument( + "--miss-policy", + choices=["count", "resample"], + default="count", + help="How to handle proposing configs without cached score.", + ) + p.add_argument( + "--max-resample", + type=int, + default=50, + help="When miss-policy=resample, max resamples per step.", + ) + p.add_argument( + "--include-missing-snapshots", + action="store_true", + help="Include entries even if config_path snapshot file is missing.", + ) + p.add_argument( + "--exploration", + type=float, + default=2.0, + help="Exploration coefficient for ucb-taskkey.", + ) + p.add_argument( + "--out", + type=Path, + default=None, + help="Optional path to write a JSON report with step logs.", + ) + return p + + +def main(argv: Optional[List[str]] = None) -> int: + args = build_parser().parse_args(argv) + results_path: Path = args.results + if not results_path.exists(): + raise SystemExit(f"results.json not found: {results_path}") + + score_by_hash, entry_by_hash, output_dir = _load_cache(results_path) + + rng = random.Random(args.seed) + + if str(args.strategy).startswith("kgpipe_"): + if not _HAS_KGPIPE_SEARCH: + raise SystemExit( + "kgpipe_search imports failed in this environment. " + "Run within the project environment where kgpipe_search is importable. " + f"Root cause: {type(_KGPIPE_SEARCH_IMPORT_ERROR).__name__}: {_KGPIPE_SEARCH_IMPORT_ERROR}" + ) + score_by_key = _load_score_by_snapshot_key( + results_path=results_path, + entry_by_hash=entry_by_hash, + score_by_hash=score_by_hash, + output_dir=output_dir, + ) + if not score_by_key: + raise SystemExit( + "No cached snapshots could be loaded to score PipelineConfig objects. " + "This usually means the `config_path` files referenced by results.json are missing. " + "Either re-run experiment.py with an output-dir you keep, or point --results at a file " + "whose config_path entries exist." + ) + oracle = OfflineCacheOracle(score_by_key, miss_score=float(args.miss_score)) + + if args.strategy == "kgpipe_random": + run = random_search( + budget=int(args.budget), + evaluate_fn=oracle.evaluate, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rng=rng, + ) + elif args.strategy == "kgpipe_qgns": + run = qgns_search( + budget=int(args.budget), + init_budget=int(args.init_budget), + init_strategy=str(args.init_strategy), + y=1, + evaluate_fn=oracle.evaluate, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + k=int(args.k), + rho=float(args.rho), + rng=rng, + ) + elif args.strategy == "kgpipe_hnr": + run = hnr_search( + budget=int(args.budget), + init_budget=int(args.init_budget), + init_strategy=str(args.init_strategy), + y=1, + evaluate_fn=oracle.evaluate, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rho=float(args.rho), + rng=rng, + ) + elif args.strategy == "kgpipe_bayes": + run = bayesian_optimization( + budget=int(args.budget), + init_random=int(args.init_budget), + init_strategy=str(args.init_strategy), + y=1, + evaluate_fn=oracle.evaluate, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + pool_size=int(args.pool_size), + beta=float(args.beta), + rng=rng, + ) + else: + raise SystemExit(f"Unknown kgpipe strategy {args.strategy!r}") + + best = max((s for s, _cfg in run.history), default=None) + print(f"results: {results_path}") + print(f"strategy: {args.strategy}") + print( + f"budget: {run.budget} cache_hit_rate: {oracle.hits / max(1, oracle.hits + oracle.misses):.3f} best_score: {best}" + ) + + if args.out is not None: + report = { + "results_path": str(results_path), + "strategy": args.strategy, + "seed": args.seed, + "budget": args.budget, + "miss_score": args.miss_score, + "cache": {"hits": oracle.hits, "misses": oracle.misses}, + "best_score": best, + "decisions": run.decisions, + "history": [ + {"score": float(score), "snapshot_key": pipeline_config_snapshot_key(cfg, RDF_SEARCH_SPACE)} + for score, cfg in run.history + ], + } + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print(f"wrote: {args.out}") + else: + candidates = _load_candidates( + results_path=results_path, + entry_by_hash=entry_by_hash, + include_missing_snapshots=bool(args.include_missing_snapshots), + output_dir=output_dir, + ) + if not candidates: + raise SystemExit("No candidates found (check results.json and config_path files).") + strategy = _build_strategy( + name=args.strategy, + rng=rng, + universe=candidates, + score_by_hash=score_by_hash, + exploration=float(args.exploration), + ) + + logs = _simulate( + strategy=strategy, + score_by_hash=score_by_hash, + budget=int(args.budget), + miss_policy=str(args.miss_policy), + max_resample=int(args.max_resample), + ) + summary = _summarize(logs) + + print(f"results: {results_path}") + print(f"strategy: {args.strategy}") + print( + f"budget: {summary['budget']} hit_rate: {summary['hit_rate']:.3f} best_score: {summary['best_score']}" + ) + + if args.out is not None: + # Note: kgpipe_* branch handles writing its own report earlier. + if not str(args.strategy).startswith("kgpipe_"): + report = { + "results_path": str(results_path), + "strategy": args.strategy, + "seed": args.seed, + "budget": args.budget, + "miss_policy": args.miss_policy, + "max_resample": args.max_resample, + "summary": summary, + "steps": [x._asdict() for x in logs], + } + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8") + print(f"wrote: {args.out}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/param-opti/src/execute.py b/experiments/param-opti/src/execute.py new file mode 100644 index 0000000..b97e036 --- /dev/null +++ b/experiments/param-opti/src/execute.py @@ -0,0 +1,554 @@ +#!/usr/bin/env python3 +""" +Run and evaluate pipeline configs from fixture files. + +Example (quick test with the small sampled fixture, 6 RDF / 4 text configs): + python execute.py \ + --seed data/bench/.../seed/data.nt \ + --source data/bench/.../sources/rdf/data.nt \ + --reference data/bench/.../reference/data_agg.nt \ + --ontology data/bench/.../ontology.ttl + +Full exhaustive run (all task/parameter permutations): + python execute.py ... --configs exhaustive +""" + +import argparse +import hashlib +import json +import os +import sys +import types +from dataclasses import asdict, is_dataclass +from importlib import import_module +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional + +from kgpipe.common import Data, DataFormat, KgPipe +from kgpipe.common.models import KgPipePlan +from kgpipe_search.configuration import ( + load_rdf_exhaustive_pipeline_configs, + load_rdf_sampled_pipeline_configs, + load_text_exhaustive_pipeline_configs, + load_text_sampled_pipeline_configs, + pipeline_config_to_snapshot, + print_pipeline_config_short, + task_keys_from_pipeline_config, +) +from kgpipe_search.definitions import PipelineConfig +from kgpipe_search.evaluation import evaluate_pipeline + + +def _install_param_opti_shim() -> None: + if "param_opti" in sys.modules: + return + + param_opti = types.ModuleType("param_opti") + tasks = types.ModuleType("param_opti.tasks") + + for lib in ( + "base_linker_lib", + "base_matcher_lib", + "paris_lib", + "fusion_lib", + "spotlight_lib", + "corenlp_lip", + "genie_lib", + ): + module = import_module(f"kgpipe_search.dev.tasks.{lib}") + setattr(tasks, lib, module) + sys.modules[f"param_opti.tasks.{lib}"] = module + + param_opti.tasks = tasks + sys.modules["param_opti"] = param_opti + sys.modules["param_opti.tasks"] = tasks + + +_install_param_opti_shim() + + +def _to_jsonable(value: Any) -> Any: + if is_dataclass(value): + return {k: _to_jsonable(v) for k, v in asdict(value).items()} + if isinstance(value, dict): + return {k: _to_jsonable(v) for k, v in value.items()} + if isinstance(value, list): + return [_to_jsonable(v) for v in value] + if isinstance(value, Path): + return str(value) + return value + + +def _set_ontology_env(ontology_path: Optional[Path]) -> None: + if ontology_path is None: + return + if not ontology_path.exists(): + raise FileNotFoundError(f"Ontology file not found: {ontology_path}") + os.environ["ONTOLOGY_PATH"] = str(ontology_path.resolve()) + + +def _config_hash(snapshot: Dict[str, Any]) -> str: + canonical = json.dumps(snapshot, sort_keys=True) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def _tasks_tmp_dir( + *, + output_dir: Path, + config_hash: str, + task_keys: List[str], + scope: str, +) -> Path: + """ + Decide where per-task temporary files live. + + - config: one tmp dir per config hash (default, current behavior) + - pipeline: reuse tmp dir for configs with identical task list (enables cache reuse across params) + - shared: reuse a single tmp dir for all configs + """ + + if scope == "config": + return output_dir / f"{config_hash}_tasks_tmp" + if scope == "shared": + return output_dir / "shared_tasks_tmp" + if scope == "pipeline": + canonical = json.dumps(task_keys, sort_keys=False) + pipeline_hash = hashlib.sha256(canonical.encode("utf-8")).hexdigest()[:12] + return output_dir / f"pipeline_{pipeline_hash}_tasks_tmp" + raise ValueError(f"Unsupported tasks tmp dir scope {scope!r}") + + +def _write_config_snapshot(config_path: Path, snapshot: Dict[str, Any]) -> None: + config_path.write_text( + json.dumps(snapshot, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def _write_plan_snapshot(plan_path: Path, plan: KgPipePlan) -> None: + plan_path.write_text( + json.dumps(plan.model_dump(mode="json"), indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def _write_eval_snapshot(eval_path: Path, evaluation: Dict[str, Any]) -> None: + eval_path.write_text( + json.dumps(evaluation, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def _load_cached_eval(eval_path: Path) -> Optional[Dict[str, Any]]: + if not eval_path.exists(): + return None + try: + payload = json.loads(eval_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError): + return None + if not isinstance(payload, dict): + return None + if payload.get("status") == "error": + return { + "status": "error", + "error": payload.get("error", "cached error"), + "evaluation": None, + "score": 0.0, + } + final_score = payload.get("final_score") + if isinstance(final_score, (int, float)): + return { + "status": "ok", + "evaluation": payload, + "score": float(final_score), + } + return None + + +def _validate_input_path(path: Path, label: str) -> Path: + resolved = path.resolve() + if not resolved.exists(): + raise FileNotFoundError(f"{label} not found: {resolved}") + return resolved + + +def _load_pipeline_configs( + *, + seed_path: Path, + pipeline_type: str, + configs: str, + configs_fixture: Optional[Path], +) -> List[PipelineConfig]: + loaders: Dict[str, Dict[str, Callable[[], List[PipelineConfig]]]] = { + "rdf": { + "sampled": load_rdf_sampled_pipeline_configs, + "exhaustive": load_rdf_exhaustive_pipeline_configs, + }, + "text": { + "sampled": load_text_sampled_pipeline_configs, + "exhaustive": load_text_exhaustive_pipeline_configs, + }, + } + + if pipeline_type not in loaders: + raise ValueError(f"Unsupported pipeline type {pipeline_type!r}") + if configs not in loaders[pipeline_type]: + raise ValueError(f"Unsupported configs mode {configs!r}") + + loader = loaders[pipeline_type][configs] + loaded = loader(configs_fixture) if configs_fixture is not None else loader() + if not loaded: + raise ValueError( + f"No pipeline configs loaded for pipeline_type={pipeline_type!r}, configs={configs!r}. " + "Generate fixtures with the configuration tests first." + ) + for config in loaded: + config.seed_path = seed_path + return loaded + + +def run_rdf_pipeline( + pipeline_config: PipelineConfig, + *, + seed_path: Path, + source_path: Path, + result_path: Path, + plan_path: Path, + tasks_tmp_dir: Path, + run_name: str, +) -> Path: + tasks_tmp_dir.mkdir(parents=True, exist_ok=True) + result_path.parent.mkdir(parents=True, exist_ok=True) + + pipeline = KgPipe( + tasks=pipeline_config.tasks, + seed=Data(path=seed_path, format=DataFormat.RDF_NTRIPLES), + data_dir=tasks_tmp_dir, + name=run_name, + ) + + plan = pipeline.build( + stable_files=True, + configCatalog=pipeline_config.config_catalog, + source=Data(path=source_path, format=DataFormat.RDF_NTRIPLES), + result=Data(path=result_path, format=DataFormat.RDF_NTRIPLES), + ) + _write_plan_snapshot(plan_path, plan) + pipeline.run(configCatalog=pipeline_config.config_catalog, stable_files_override=False) + return result_path + + +def run_text_pipeline( + pipeline_config: PipelineConfig, + *, + seed_path: Path, + source_path: Path, + result_path: Path, + plan_path: Path, + tasks_tmp_dir: Path, + run_name: str, +) -> Path: + tasks_tmp_dir.mkdir(parents=True, exist_ok=True) + result_path.parent.mkdir(parents=True, exist_ok=True) + + pipeline = KgPipe( + tasks=pipeline_config.tasks, + seed=Data(path=seed_path, format=DataFormat.RDF_NTRIPLES), + data_dir=tasks_tmp_dir, + name=run_name, + ) + + plan = pipeline.build( + stable_files=True, + configCatalog=pipeline_config.config_catalog, + source=Data(path=source_path, format=DataFormat.TEXT), + result=Data(path=result_path, format=DataFormat.RDF_NTRIPLES), + ) + _write_plan_snapshot(plan_path, plan) + pipeline.run(configCatalog=pipeline_config.config_catalog, stable_files_override=False) + return result_path + + +def run_all_configs( + *, + seed_path: Path, + source_path: Path, + reference_path: Path, + ontology_path: Optional[Path], + output_dir: Path, + pipeline_type: str, + configs: str, + configs_fixture: Optional[Path], + start: int, + limit: Optional[int], + results_path: Optional[Path], + tasks_tmp_scope: str, + reuse_existing: bool = True, +) -> List[Dict[str, Any]]: + _set_ontology_env(ontology_path) + + run_pipeline = run_rdf_pipeline if pipeline_type == "rdf" else run_text_pipeline + + pipeline_configs = _load_pipeline_configs( + seed_path=seed_path, + pipeline_type=pipeline_type, + configs=configs, + configs_fixture=configs_fixture, + ) + + end = len(pipeline_configs) if limit is None else min(len(pipeline_configs), start + limit) + selected = pipeline_configs[start:end] + + output_dir.mkdir(parents=True, exist_ok=True) + run_results: List[Dict[str, Any]] = [] + cache_hits = 0 + + print(f"Running {len(selected)} pipeline config(s) [{start}:{end})") + print(f"seed: {seed_path}") + print(f"source: {source_path}") + print(f"reference: {reference_path}") + print(f"output_dir: {output_dir}") + print(f"tasks_tmp_scope: {tasks_tmp_scope}") + print(f"reuse_existing: {reuse_existing}") + + for offset, pipeline_config in enumerate(selected, start=start): + task_keys = task_keys_from_pipeline_config(pipeline_config) + snapshot = pipeline_config_to_snapshot(task_keys, pipeline_config) + config_hash = _config_hash(snapshot) + + result_path = output_dir / f"{config_hash}.nt" + config_path = output_dir / f"{config_hash}.json" + eval_path = output_dir / f"{config_hash}.eval.json" + plan_path = output_dir / f"{config_hash}.plan.json" + tasks_tmp_dir = _tasks_tmp_dir( + output_dir=output_dir, + config_hash=config_hash, + task_keys=task_keys, + scope=tasks_tmp_scope, + ) + run_name = config_hash + + print(f"\n=== config {offset + 1}/{len(pipeline_configs)} ({config_hash}) ===") + print_pipeline_config_short(pipeline_config) + + _write_config_snapshot(config_path, snapshot) + + entry: Dict[str, Any] = { + "config_idx": offset, + "config_hash": config_hash, + "config_path": str(config_path), + "eval_path": str(eval_path), + "plan_path": str(plan_path), + "result_path": str(result_path), + "tasks_tmp_dir": str(tasks_tmp_dir), + "status": "ok", + "cached": False, + } + + try: + cached = _load_cached_eval(eval_path) if reuse_existing else None + if cached is not None: + cache_hits += 1 + entry["cached"] = True + entry["status"] = cached["status"] + if cached["status"] == "error": + entry["error"] = cached["error"] + print(f"cached error: {entry['error']}") + else: + entry["evaluation"] = cached["evaluation"] + print(f"cached score: {cached['score']:.6f}") + elif reuse_existing and result_path.exists(): + aggregate_score = evaluate_pipeline( + pipeline_config, + result_path, + reference_path, + ) + evaluation = _to_jsonable(aggregate_score) + entry["evaluation"] = evaluation + entry["cached"] = "result_only" + _write_eval_snapshot(eval_path, evaluation) + print(f"reused result, score: {aggregate_score.final_score:.6f}") + else: + run_pipeline( + pipeline_config, + seed_path=seed_path, + source_path=source_path, + result_path=result_path, + plan_path=plan_path, + tasks_tmp_dir=tasks_tmp_dir, + run_name=run_name, + ) + aggregate_score = evaluate_pipeline( + pipeline_config, + result_path, + reference_path, + ) + evaluation = _to_jsonable(aggregate_score) + entry["evaluation"] = evaluation + _write_eval_snapshot(eval_path, evaluation) + print(f"score: {aggregate_score.final_score:.6f}") + except Exception as exc: + entry["status"] = "error" + entry["error"] = f"{type(exc).__name__}: {exc}" + _write_eval_snapshot( + eval_path, + {"status": "error", "error": entry["error"]}, + ) + print(f"failed: {entry['error']}") + + run_results.append(entry) + + if results_path is not None: + results_path.parent.mkdir(parents=True, exist_ok=True) + payload = { + "pipeline_type": pipeline_type, + "configs": configs, + "seed": str(seed_path), + "source": str(source_path), + "reference": str(reference_path), + "ontology": str(ontology_path) if ontology_path is not None else None, + "output_dir": str(output_dir), + "start": start, + "limit": limit, + "tasks_tmp_scope": tasks_tmp_scope, + "cache_hits": cache_hits, + "reuse_existing": reuse_existing, + "results": run_results, + } + results_path.write_text( + json.dumps(payload, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print(f"\nWrote scores to {results_path}") + + succeeded = sum(1 for item in run_results if item["status"] == "ok") + print(f"\nFinished: {succeeded}/{len(run_results)} succeeded, cache_hits={cache_hits}") + return run_results + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Execute and evaluate pipeline configs from fixture files.", + ) + parser.add_argument("--seed", type=Path, required=True, help="Path to seed knowledge graph") + parser.add_argument("--source", type=Path, required=True, help="Path to source input graph/text") + parser.add_argument( + "--reference", + type=Path, + required=True, + help="Path to reference knowledge graph used for evaluation", + ) + parser.add_argument( + "--ontology", + type=Path, + default=None, + help="Optional ontology path (sets ONTOLOGY_PATH for matchers)", + ) + parser.add_argument( + "--output-dir", + type=Path, + default=Path("data/tmp/pipeline_runs"), + help="Directory for pipeline outputs and task temp files", + ) + parser.add_argument( + "--pipeline-type", + choices=["rdf", "text"], + default="rdf", + help="Pipeline family to run", + ) + parser.add_argument( + "--configs", + choices=["sampled", "exhaustive"], + default="sampled", + help=( + "Which fixture set to execute: " + "'sampled' = small fixture for quick tests (default), " + "'exhaustive' = all task/parameter permutations" + ), + ) + parser.add_argument( + "--configs-fixture", + type=Path, + default=None, + help="Optional path to a custom configs fixture JSON file", + ) + parser.add_argument( + "--start", + type=int, + default=0, + help="Start index into the loaded config list", + ) + parser.add_argument( + "--limit", + type=int, + default=None, + help="Maximum number of configs to run (default: all from --start)", + ) + parser.add_argument( + "--results", + type=Path, + default=None, + help="Path to write a single JSON summary of all run scores (default: /results.json)", + ) + parser.add_argument( + "--tasks-tmp-scope", + choices=["config", "pipeline", "shared"], + default="config", + help=( + "How to name/reuse the per-run tasks tmp dir: " + "'config' = one tmp dir per config hash (default), " + "'pipeline' = reuse tmp dir for configs with identical task list, " + "'shared' = reuse one tmp dir for all configs" + ), + ) + parser.add_argument( + "--force-rerun", + action="store_true", + help="Re-run pipelines and evaluation even when cached result/eval files exist", + ) + return parser + + +def main(argv: Optional[List[str]] = None) -> int: + args = build_parser().parse_args(argv) + + if args.start < 0: + raise SystemExit("--start must be >= 0") + if args.limit is not None and args.limit <= 0: + raise SystemExit("--limit must be > 0") + + seed_path = _validate_input_path(args.seed, "Seed graph") + source_path = _validate_input_path(args.source, "Source input") + reference_path = _validate_input_path(args.reference, "Reference graph") + ontology_path = ( + _validate_input_path(args.ontology, "Ontology") + if args.ontology is not None + else None + ) + + run_results = run_all_configs( + seed_path=seed_path, + source_path=source_path, + reference_path=reference_path, + ontology_path=ontology_path, + output_dir=args.output_dir, + pipeline_type=args.pipeline_type, + configs=args.configs, + configs_fixture=args.configs_fixture, + start=args.start, + limit=args.limit, + results_path=args.results or (args.output_dir / "results.json"), + tasks_tmp_scope=args.tasks_tmp_scope, + reuse_existing=not args.force_rerun, + ) + + failed = sum(1 for item in run_results if item["status"] != "ok") + return 1 if failed else 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except FileNotFoundError as exc: + print(f"error: {exc}", file=sys.stderr) + raise SystemExit(2) from exc diff --git a/experiments/param-opti/src/experiment.py b/experiments/param-opti/src/experiment.py new file mode 100644 index 0000000..afb346b --- /dev/null +++ b/experiments/param-opti/src/experiment.py @@ -0,0 +1,593 @@ +#!/usr/bin/env python3 +""" +Run a full configuration search experiment. + +Given a search space (RDF or text), a search strategy proposes pipeline configs; +each candidate is executed against seed/source data, evaluated against a reference +KG, and written to the output directory. A combined results file is produced for +offline analysis via analyse.py. + +Example: + PYTHONPATH=src python src/experiment.py \\ + --seed data/bench/.../seed/data.nt \\ + --source data/bench/.../sources/rdf/data.nt \\ + --reference data/bench/.../reference/data_agg.nt \\ + --ontology data/bench/.../ontology.ttl \\ + --pipeline-type rdf \\ + --strategy qgns \\ + --budget 20 \\ + --init-budget 3 \\ + --output-dir fix_runs +""" + +from __future__ import annotations + +import argparse +import json +import random +import sys +from pathlib import Path +from typing import Any, Callable, Dict, List, Literal, Optional + +from kgpipe_search.configuration import ( + pipeline_config_to_snapshot, + print_pipeline_config_short, + task_keys_from_pipeline_config, +) +from kgpipe_search.definitions import ( + RDF_PIPELINE_LAYOUT, + RDF_SEARCH_SPACE, + TEXT_PIPELINE_LAYOUT, + TEXT_SEARCH_SPACE, + PipelineConfig, +) +from kgpipe_search.evaluation import aggregate_from_cached_evaluation, evaluate_pipeline +from kgpipe_search.ranking_conf import AGGREGATION_CONFIGS, get_aggregation_config +from kgpipe_search.search import ( + bayesian_optimization, + hnr_search, + hnr_2_search, + implementation_aware_search, + llm_search, + qgns_search, + random_search, +) +from kgpipe_search.strategies.strategies import SearchRun + +import execute as pipeline_execute + +PipelineType = Literal["rdf", "text"] +SearchStrategyName = Literal["random", "implementation_aware", "qgns", "hnr", "hnr_2", "bayesian", "llm"] +TasksTmpScope = Literal["config", "pipeline", "shared"] +InitStrategy = Literal["random", "implementation_aware"] + + +def _pipeline_context(pipeline_type: PipelineType) -> tuple[Dict[str, Any], Any, Callable[..., Path]]: + if pipeline_type == "rdf": + return RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT, pipeline_execute.run_rdf_pipeline + if pipeline_type == "text": + return TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT, pipeline_execute.run_text_pipeline + raise ValueError(f"Unsupported pipeline type {pipeline_type!r}") + + +def _run_search( + *, + strategy: SearchStrategyName, + budget: int, + evaluate_fn: Callable[[PipelineConfig], float], + search_space: Dict[str, Any], + pipeline_layout: Any, + init_budget: int, + init_strategy: InitStrategy, + y: int, + k: int, + rho: float, + pool_size: int, + beta: float, + llm_max_retries: int, + rng: random.Random, + min_quality_delta: float = 0.05, + min_iterations_wo_improvement: int = 2, +) -> SearchRun: + common = { + "budget": budget, + "evaluate_fn": evaluate_fn, + "search_space": search_space, + "pipeline_layout": pipeline_layout, + "rng": rng, + } + + if strategy == "random": + return random_search(**common) + + if strategy == "implementation_aware": + return implementation_aware_search(**common, y=y) + + if strategy == "qgns": + return qgns_search( + **common, + init_budget=init_budget, + init_strategy=init_strategy, + y=y, + k=k, + rho=rho, + ) + + if strategy == "hnr": + if init_budget <= 0: + raise ValueError("HNR requires --init-budget > 0") + return hnr_search( + **common, + init_budget=init_budget, + init_strategy=init_strategy, + y=y, + rho=rho, + ) + + if strategy == "hnr_2": + return hnr_2_search( + **common, + init_budget=init_budget, + init_strategy=init_strategy, + y=y, + rho=rho, + min_quality_delta=min_quality_delta, + min_iterations_wo_improvement=min_iterations_wo_improvement, + ) + if strategy == "bayesian": + return bayesian_optimization( + **common, + init_random=init_budget, + init_strategy=init_strategy, + y=y, + pool_size=pool_size, + beta=beta, + ) + + if strategy == "llm": + return llm_search( + **common, + max_retries=llm_max_retries, + ) + + raise ValueError(f"Unknown search strategy {strategy!r}") + + +def run_search_experiment( + *, + seed_path: Path, + source_path: Path, + reference_path: Path, + ontology_path: Optional[Path], + output_dir: Path, + pipeline_type: PipelineType, + strategy: SearchStrategyName, + budget: int, + init_budget: int, + init_strategy: InitStrategy, + y: int, + k: int, + rho: float, + pool_size: int, + beta: float, + llm_max_retries: int, + rng_seed: int, + tasks_tmp_scope: TasksTmpScope, + results_path: Optional[Path], + reuse_existing: bool = True, + min_quality_delta: float = 0.05, + min_iterations_wo_improvement: int = 2, + rank_aggregation: str = "default", +) -> Dict[str, Any]: + pipeline_execute._set_ontology_env(ontology_path) + + search_space, pipeline_layout, run_pipeline = _pipeline_context(pipeline_type) + aggregation_config = get_aggregation_config(rank_aggregation) + output_dir.mkdir(parents=True, exist_ok=True) + + rng = random.Random(rng_seed) + run_results: List[Dict[str, Any]] = [] + search_history: List[Dict[str, Any]] = [] + best_score: Optional[float] = None + cache_hits = 0 + + def evaluate_fn(pipeline_config: PipelineConfig) -> float: + nonlocal best_score, cache_hits + + task_keys = task_keys_from_pipeline_config(pipeline_config) + snapshot = pipeline_config_to_snapshot(task_keys, pipeline_config) + config_hash = pipeline_execute._config_hash(snapshot) + + config_path = output_dir / f"{config_hash}.json" + result_path = output_dir / f"{config_hash}.nt" + eval_path = output_dir / f"{config_hash}.eval.json" + plan_path = output_dir / f"{config_hash}.plan.json" + tasks_tmp_dir = pipeline_execute._tasks_tmp_dir( + output_dir=output_dir, + config_hash=config_hash, + task_keys=task_keys, + scope=tasks_tmp_scope, + ) + + step = len(run_results) + 1 + print(f"\n=== trial {step}/{budget} ({config_hash}) ===") + print_pipeline_config_short(pipeline_config) + + pipeline_execute._write_config_snapshot(config_path, snapshot) + + entry: Dict[str, Any] = { + "trial": step, + "config_hash": config_hash, + "config_path": str(config_path), + "result_path": str(result_path), + "eval_path": str(eval_path), + "plan_path": str(plan_path), + "tasks_tmp_dir": str(tasks_tmp_dir), + "status": "ok", + "cached": False, + "rank_aggregation": rank_aggregation, + } + + try: + cached = pipeline_execute._load_cached_eval(eval_path) if reuse_existing else None + if cached is not None: + cache_hits += 1 + entry["cached"] = True + entry["status"] = cached["status"] + if cached["status"] == "error": + entry["error"] = cached["error"] + score = float(cached["score"]) + print(f"cached error: {entry['error']}") + else: + # Re-rank from stored measurements so a different aggregation + # (e.g. flat_hmean) can be used without re-running evaluation. + aggregate_score = aggregate_from_cached_evaluation( + cached["evaluation"], + aggregation_config, + ) + evaluation = pipeline_execute._to_jsonable(aggregate_score) + entry["evaluation"] = evaluation + score = float(aggregate_score.final_score) + print(f"cached score ({rank_aggregation}): {score:.6f}") + elif reuse_existing and result_path.exists(): + aggregate_score = evaluate_pipeline( + pipeline_config, + result_path, + reference_path, + aggregation=aggregation_config, + ) + evaluation = pipeline_execute._to_jsonable(aggregate_score) + entry["evaluation"] = evaluation + entry["cached"] = "result_only" + pipeline_execute._write_eval_snapshot(eval_path, evaluation) + score = float(aggregate_score.final_score) + print(f"reused result, score: {score:.6f}") + else: + run_pipeline( + pipeline_config, + seed_path=seed_path, + source_path=source_path, + result_path=result_path, + plan_path=plan_path, + tasks_tmp_dir=tasks_tmp_dir, + run_name=config_hash, + ) + aggregate_score = evaluate_pipeline( + pipeline_config, + result_path, + reference_path, + aggregation=aggregation_config, + ) + evaluation = pipeline_execute._to_jsonable(aggregate_score) + entry["evaluation"] = evaluation + pipeline_execute._write_eval_snapshot(eval_path, evaluation) + score = float(aggregate_score.final_score) + print(f"score: {score:.6f}") + except Exception as exc: + entry["status"] = "error" + entry["error"] = f"{type(exc).__name__}: {exc}" + score = 0.0 + pipeline_execute._write_eval_snapshot( + eval_path, + {"status": "error", "error": entry["error"]}, + ) + print(f"failed: {entry['error']}") + + run_results.append(entry) + best_score = score if best_score is None else max(best_score, score) + return score + + print(f"Running search strategy={strategy!r} budget={budget}") + print(f"seed: {seed_path}") + print(f"source: {source_path}") + print(f"reference: {reference_path}") + print(f"output_dir: {output_dir}") + print(f"tasks_tmp_scope: {tasks_tmp_scope}") + print(f"rank_aggregation: {rank_aggregation}") + + search_run = _run_search( + strategy=strategy, + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=init_budget, + init_strategy=init_strategy, + y=y, + k=k, + rho=rho, + pool_size=pool_size, + beta=beta, + llm_max_retries=llm_max_retries, + rng=rng, + min_quality_delta=min_quality_delta, + min_iterations_wo_improvement=min_iterations_wo_improvement, + ) + + result_by_hash = {item["config_hash"]: item for item in run_results} + running_best: Optional[float] = None + + for step, ((score, _cfg), decision) in enumerate( + zip(search_run.history, search_run.decisions), + start=1, + ): + task_keys = task_keys_from_pipeline_config(_cfg) + snapshot = pipeline_config_to_snapshot(task_keys, _cfg) + config_hash = pipeline_execute._config_hash(snapshot) + running_best = score if running_best is None else max(running_best, score) + + entry = result_by_hash.get(config_hash, {}) + search_history.append( + { + "step": step, + "decision": decision, + "config_hash": config_hash, + "score": score, + "best_score": running_best, + "status": entry.get("status", "unknown"), + "config_path": entry.get("config_path"), + "result_path": entry.get("result_path"), + "eval_path": entry.get("eval_path"), + "plan_path": entry.get("plan_path"), + } + ) + + payload: Dict[str, Any] = { + "pipeline_type": pipeline_type, + "search": { + "strategy": strategy, + "budget": budget, + "init_budget": init_budget, + "init_strategy": init_strategy, + "y": y, + "k": k, + "rho": rho, + "pool_size": pool_size, + "beta": beta, + "rng_seed": rng_seed, + "decisions": search_run.decisions, + }, + "seed": str(seed_path), + "source": str(source_path), + "reference": str(reference_path), + "ontology": str(ontology_path) if ontology_path is not None else None, + "output_dir": str(output_dir), + "tasks_tmp_scope": tasks_tmp_scope, + "results": run_results, + "search_history": search_history, + "best_score": running_best, + "cache_hits": cache_hits, + "reuse_existing": reuse_existing, + "rank_aggregation": rank_aggregation, + } + + resolved_results_path = results_path or (output_dir / "results.json") + resolved_results_path.parent.mkdir(parents=True, exist_ok=True) + resolved_results_path.write_text( + json.dumps(payload, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print(f"\nWrote combined results to {resolved_results_path}") + + succeeded = sum(1 for item in run_results if item["status"] == "ok") + print( + f"Finished: {succeeded}/{len(run_results)} succeeded, " + f"cache_hits={cache_hits}, best_score={running_best}" + ) + return payload + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Run a full pipeline configuration search experiment.", + ) + parser.add_argument("--seed", type=Path, required=True, help="Path to seed knowledge graph") + parser.add_argument("--source", type=Path, required=True, help="Path to source input graph/text") + parser.add_argument( + "--reference", + type=Path, + required=True, + help="Path to reference knowledge graph used for evaluation", + ) + parser.add_argument( + "--ontology", + type=Path, + default=None, + help="Optional ontology path (sets ONTOLOGY_PATH for matchers)", + ) + parser.add_argument( + "--output-dir", + type=Path, + default=Path("data/tmp/search_runs"), + help="Directory for pipeline outputs, eval files, and task temp files", + ) + parser.add_argument( + "--pipeline-type", + choices=["rdf", "text"], + default="rdf", + help="Pipeline family to search over", + ) + parser.add_argument( + "--strategy", + choices=["random", "implementation_aware", "qgns", "hnr", "hnr_2", "bayesian", "llm"], + default="random", + help=( + "Search strategy to use. " + "'random' = uniform random configs; " + "'implementation_aware' = systematic task-combo coverage with random params" + ), + ) + parser.add_argument("--budget", type=int, default=10, help="Total number of configs to evaluate") + parser.add_argument( + "--init-budget", + type=int, + default=3, + help="Initialization budget for qgns/hnr/bayesian (ignored by random)", + ) + parser.add_argument( + "--init-strategy", + choices=["random", "implementation_aware"], + default="implementation_aware", + help="Initialization sampling strategy", + ) + parser.add_argument( + "--y", + type=int, + default=1, + help="Number of parameter samples per task combo during implementation-aware init", + ) + parser.add_argument("--k", type=int, default=3, help="Top-k anchors for QGNS") + parser.add_argument( + "--rho", + type=float, + default=0.2, + help="Exploration probability for RNS", + ) + parser.add_argument( + "--pool-size", + type=int, + default=32, + help="Candidate pool size for Bayesian optimization", + ) + parser.add_argument( + "--beta", + type=float, + default=0.5, + help="Acquisition beta for Bayesian optimization", + ) + parser.add_argument( + "--llm-max-retries", + type=int, + default=3, + help="Validation retries per LLM proposal when using --strategy llm", + ) + parser.add_argument( + "--rng-seed", + type=int, + default=0, + help="RNG seed for reproducible search", + ) + parser.add_argument( + "--results", + type=Path, + default=None, + help="Path to write combined results JSON (default: /results.json)", + ) + parser.add_argument( + "--tasks-tmp-scope", + choices=["config", "pipeline", "shared"], + default="config", + help=( + "How to name/reuse the per-run tasks tmp dir: " + "'config' = one tmp dir per config hash (default), " + "'pipeline' = reuse tmp dir for configs with identical task list, " + "'shared' = reuse one tmp dir for all configs" + ), + ) + parser.add_argument( + "--force-rerun", + action="store_true", + help="Re-run pipelines and evaluation even when cached result/eval files exist", + ) + parser.add_argument( + "--min-quality-delta", + type=float, + default=0.01, + help="Minimum quality delta for HNR_2", + ) + parser.add_argument( + "--min-iterations-wo-improvement", + type=int, + default=3, + help="Minimum iterations without improvement for HNR_2", + ) + parser.add_argument( + "--rank-aggregation", + choices=sorted(AGGREGATION_CONFIGS), + default="default", + help=( + "How to turn per-metric measurements into the search objective. " + "'default' = subgroup means then weighted mean; " + "'flat_hmean' = harmonic mean over all measurements; " + "'custom' = custom aggregation config. " + "Cached .eval.json files are re-ranked from stored measurements." + ), + ) + return parser + + +def main(argv: Optional[List[str]] = None) -> int: + args = build_parser().parse_args(argv) + + if args.budget <= 0: + raise SystemExit("--budget must be > 0") + if args.init_budget < 0: + raise SystemExit("--init-budget must be >= 0") + if args.strategy == "hnr" and args.init_budget <= 0: + raise SystemExit("HNR requires --init-budget > 0") + + seed_path = pipeline_execute._validate_input_path(args.seed, "Seed graph") + source_path = pipeline_execute._validate_input_path(args.source, "Source input") + reference_path = pipeline_execute._validate_input_path(args.reference, "Reference graph") + ontology_path = ( + pipeline_execute._validate_input_path(args.ontology, "Ontology") + if args.ontology is not None + else None + ) + + payload = run_search_experiment( + seed_path=seed_path, + source_path=source_path, + reference_path=reference_path, + ontology_path=ontology_path, + output_dir=args.output_dir, + pipeline_type=args.pipeline_type, + strategy=args.strategy, + budget=args.budget, + init_budget=args.init_budget, + init_strategy=args.init_strategy, + y=args.y, + k=args.k, + rho=args.rho, + pool_size=args.pool_size, + beta=args.beta, + llm_max_retries=args.llm_max_retries, + rng_seed=args.rng_seed, + tasks_tmp_scope=args.tasks_tmp_scope, + results_path=args.results, + reuse_existing=not args.force_rerun, + min_quality_delta=args.min_quality_delta, + min_iterations_wo_improvement=args.min_iterations_wo_improvement, + rank_aggregation=args.rank_aggregation, + ) + + failed = sum(1 for item in payload["results"] if item["status"] != "ok") + return 1 if failed else 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except FileNotFoundError as exc: + print(f"error: {exc}", file=sys.stderr) + raise SystemExit(2) from exc diff --git a/experiments/param-opti/src/kgpipe_search/__init__.py b/experiments/param-opti/src/kgpipe_search/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/src/kgpipe_search/configuration.py b/experiments/param-opti/src/kgpipe_search/configuration.py new file mode 100644 index 0000000..d120b69 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/configuration.py @@ -0,0 +1,677 @@ +from typing import List, Dict, Any, Optional +from kgpipe.common import KgTask +from kgpipe.common.model.configuration import ConfigurationProfile, ParameterBinding +from kgpipe_search.definitions import ( + PipelineLayout, + PipelineConfig, + RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE, + RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE, + RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE, + TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE, + TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE, + TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE, + _RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION, + _RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION, + _RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION, + _TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION, + _TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION, + _TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION, +) +import json +import random +import itertools +from pathlib import Path +from kgpipe_search.definitions import task_dict + + +def _task_categories_list(search_space: Dict[str, Dict[str, Any]], task_name: str) -> List[str]: + raw = search_space.get(task_name, {}).get("category") + if isinstance(raw, list): + return [c for c in raw if isinstance(c, str)] + if isinstance(raw, str): + return [raw] + return [] + + +def enumerate_valid_task_combinations( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, +) -> List[List[str]]: + """ + Enumerate all possible task-name combinations for the given pipeline layout, + respecting category order and multi-category coverage, without sampling config options. + + A task is only eligible for the current category if its declared categories are + disjoint from categories already covered by earlier tasks. That avoids pairing e.g. + Paris ontology matching with a dual-category embedding matcher that would repeat + ontology coverage when only entity matching is still needed. + """ + all_task_names = list(search_space.keys()) + + combos: List[List[str]] = [[]] + covered_sets: List[set[str]] = [set()] + + for category in pipeline_layout.allowed_task_categories: + next_combos: List[List[str]] = [] + next_covered_sets: List[set[str]] = [] + + for combo, covered in zip(combos, covered_sets): + if category in covered: + next_combos.append(combo) + next_covered_sets.append(covered) + continue + + eligible: List[str] = [] + for tn in all_task_names: + cats = _task_categories_list(search_space, tn) + if category not in cats: + continue + if set(cats) & covered: + continue + eligible.append(tn) + for tn in eligible: + new_combo = combo + [tn] + new_covered = set(covered) + new_covered.update(_task_categories_list(search_space, tn)) + next_combos.append(new_combo) + next_covered_sets.append(new_covered) + + combos, covered_sets = next_combos, next_covered_sets + + # De-duplicate while keeping stable order. + seen: set[tuple[str, ...]] = set() + unique: List[List[str]] = [] + for c in combos: + t = tuple(c) + if t in seen: + continue + seen.add(t) + unique.append(c) + return unique + +def _get_param(definition: Any, param_name: str): + params = getattr(definition, "parameters", None) + if params is None: + raise KeyError(f"Task config_spec has no parameters field (missing {param_name})") + + # common shapes: dict-like or list of Parameter + if hasattr(params, "get"): + p = params.get(param_name) + if p is None: + raise KeyError(f"Parameter {param_name} not found in config_spec.parameters") + return p + + for p in params: + if getattr(p, "name", None) == param_name: + return p + raise KeyError(f"Parameter {param_name} not found in config_spec.parameters") + + +def pipeline_config_to_snapshot(task_keys: List[str], pipeline_config: PipelineConfig) -> Dict[str, Any]: + profiles: Dict[str, Any] = {} + for task in pipeline_config.tasks: + prof = pipeline_config.config_catalog.get(task.name) + if prof is None: + continue + profiles[task.name] = { + "profile_name": prof.name, + "bindings": [ + {"parameter": binding.parameter.name, "value": binding.value} + for binding in prof.bindings + ], + } + return {"task_keys": task_keys, "profiles": profiles} + + +def pipeline_config_from_snapshot(snapshot: Dict[str, Any]) -> PipelineConfig: + task_keys: List[str] = snapshot["task_keys"] + profiles: Dict[str, Any] = snapshot.get("profiles") or {} + tasks: List[KgTask] = [] + config_catalog: Dict[str, ConfigurationProfile] = {} + + for task_key in task_keys: + task = task_dict[task_key] + tasks.append(task) + prof_data = profiles.get(task.name) + if prof_data is None: + continue + if getattr(task, "config_spec", None) is None: + continue + bindings = [ + ParameterBinding( + parameter=_get_param(task.config_spec, b["parameter"]), + value=b["value"], + ) + for b in prof_data["bindings"] + ] + config_catalog[task.name] = ConfigurationProfile( + name=prof_data["profile_name"], + definition=task.config_spec, + bindings=bindings, + ) + + return PipelineConfig(tasks=tasks, config_catalog=config_catalog) + + +def load_rdf_sampled_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported rdf sampled configs snapshot version {raw.get('version')!r}; " + f"expected {_RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +def load_text_sampled_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported text sampled configs snapshot version {raw.get('version')!r}; " + f"expected {_TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +def load_rdf_unique_sampled_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported rdf unique sampled configs snapshot version {raw.get('version')!r}; " + f"expected {_RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +def load_text_unique_sampled_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported text unique sampled configs snapshot version {raw.get('version')!r}; " + f"expected {_TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +def load_rdf_exhaustive_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported rdf exhaustive configs snapshot version {raw.get('version')!r}; " + f"expected {_RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +def load_text_exhaustive_pipeline_configs(path: Optional[Path] = None) -> List[PipelineConfig]: + fixture_path = path or TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE + raw = json.loads(fixture_path.read_text(encoding="utf-8")) + if raw.get("version") != _TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION: + raise ValueError( + f"Unsupported text exhaustive configs snapshot version {raw.get('version')!r}; " + f"expected {_TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION}" + ) + return [pipeline_config_from_snapshot(item) for item in raw["samples"]] + + +# TODO rules for valid pipeline config: +def sample_valid_pipeline_config( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + rng: Optional[random.Random] = None, +) -> PipelineConfig: + """ + Randomly sample a valid pipeline config from the search space, + respecting the order of categories in the pipeline layout. + """ + draw = rng.choice if rng is not None else random.choice + tasks: List[KgTask] = [] + config_catalog: Dict[str, ConfigurationProfile] = {} + covered_categories: set[str] = set() + + for category in pipeline_layout.allowed_task_categories: + if category in covered_categories: + continue + + eligible_task_names = [ + tn + for tn, space in search_space.items() + if ( + space.get("category") == category + or ( + isinstance(space.get("category"), list) + and category in (space.get("category") or []) + ) + ) + ] + if not eligible_task_names: + continue + + eligible_task_names = [ + tn + for tn in eligible_task_names + if not (set(_task_categories_list(search_space, tn)) & covered_categories) + ] + if not eligible_task_names: + raise ValueError( + f"No task can cover category {category!r} without overlapping already covered " + f"categories {sorted(covered_categories)}. Adjust search_space or pipeline_layout." + ) + + task_key = draw(eligible_task_names) + task = task_dict[task_key] + covered_categories.update(_task_categories_list(search_space, task_key)) + tasks.append(task) + + # metadata only or task has no config spec + if getattr(task, "config_spec", None) is None: + continue + + bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + for config_name, config_values in search_space[task_key].items(): + if config_name == "category": + continue + if not isinstance(config_values, list): + raise TypeError( + f"Search space values must be lists; got {task_key}.{config_name}={type(config_values)}" + ) + if not config_values: + raise ValueError(f"Empty search space for {task_key}.{config_name}") + + config_value = draw(config_values) + name_parts.append(f"{config_name}={config_value}") + bindings.append( + ParameterBinding( + parameter=_get_param(task.config_spec, config_name), + value=config_value, + ) + ) + + if bindings: + config_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=task.config_spec, + bindings=bindings, + ) + + return PipelineConfig(tasks=tasks, config_catalog=config_catalog) + + +_PIPELINE_CONFIG_SNAPSHOT_FILE_VERSION = 1 + + +def save_pipeline_config_snapshot( + path: Path, + pipeline_config: PipelineConfig, + *, + task_keys: Optional[List[str]] = None, +) -> None: + keys = task_keys or task_keys_from_pipeline_config(pipeline_config) + payload = { + "version": _PIPELINE_CONFIG_SNAPSHOT_FILE_VERSION, + "snapshot": pipeline_config_to_snapshot(keys, pipeline_config), + } + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + json.dumps(payload, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + + +def load_pipeline_config_snapshot(path: Path) -> PipelineConfig: + raw = json.loads(path.read_text(encoding="utf-8")) + if raw.get("version") != _PIPELINE_CONFIG_SNAPSHOT_FILE_VERSION: + raise ValueError( + f"Unsupported pipeline config snapshot file version {raw.get('version')!r}; " + f"expected {_PIPELINE_CONFIG_SNAPSHOT_FILE_VERSION}" + ) + return pipeline_config_from_snapshot(raw["snapshot"]) + + +def task_keys_from_pipeline_config(pipeline_config: PipelineConfig) -> List[str]: + keys: List[str] = [] + for task in pipeline_config.tasks: + for task_key, registered in task_dict.items(): + if registered is task or registered.name == task.name: + keys.append(task_key) + break + else: + raise ValueError(f"Unknown task {task.name!r}") + return keys + + +def pipeline_config_snapshot_key( + pipeline_config: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], +) -> str: + task_keys = task_keys_from_pipeline_config(pipeline_config) + snapshot = pipeline_config_to_snapshot(task_keys, pipeline_config) + return json.dumps(snapshot, sort_keys=True) + + +def build_pipeline_config_for_task_combo( + search_space: Dict[str, Dict[str, Any]], + task_name_combo: List[str], + *, + rng: random.Random, + template: Optional[PipelineConfig] = None, +) -> PipelineConfig: + """ + Build a pipeline config for a fixed task combo. + Reuses parameter profiles from template when the task key is unchanged. + """ + template_keys = ( + task_keys_from_pipeline_config(template) if template is not None else [] + ) + tasks: List[KgTask] = [] + config_catalog: Dict[str, ConfigurationProfile] = {} + + for index, task_key in enumerate(task_name_combo): + task = task_dict[task_key] + tasks.append(task) + + if ( + template is not None + and index < len(template_keys) + and template_keys[index] == task_key + ): + profile = template.config_catalog.get(task.name) + if profile is not None: + config_catalog[task.name] = profile + continue + + if getattr(task, "config_spec", None) is None: + continue + + bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + for config_name, config_values in search_space[task_key].items(): + if config_name == "category": + continue + if not isinstance(config_values, list): + raise TypeError( + f"Search space values must be lists; got {task_key}.{config_name}={type(config_values)}" + ) + if not config_values: + raise ValueError(f"Empty search space for {task_key}.{config_name}") + + config_value = rng.choice(config_values) + name_parts.append(f"{config_name}={config_value}") + bindings.append( + ParameterBinding( + parameter=_get_param(task.config_spec, config_name), + value=config_value, + ) + ) + + if bindings: + config_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=task.config_spec, + bindings=bindings, + ) + + return PipelineConfig(tasks=tasks, config_catalog=config_catalog) + + +def print_pipeline_config_short(pipeline_config: PipelineConfig): + """ + print the pipeline config in a short format + """ + print() + print("================") + for task in pipeline_config.tasks: + task_name = task.name + profile: Optional[ConfigurationProfile] = pipeline_config.config_catalog.get(task_name) + if profile is None: + print(f"- {task_name}") + continue + + parts: List[str] = [] + for binding in profile.bindings: + parts.append(f"{binding.parameter.name}={binding.value}") + params = ", ".join(parts) + print(f"- {task_name}({params})") + +def sample_config_catalog_for_task_combo( + search_space: Dict[str, Dict[str, Any]], + task_name_combo: List[str], + *, + rng: random.Random, +) -> PipelineConfig: + tasks: List[KgTask] = [] + config_catalog: Dict[str, ConfigurationProfile] = {} + + for task_key in task_name_combo: + task = task_dict[task_key] + tasks.append(task) + + if getattr(task, "config_spec", None) is None: + continue + + bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + for config_name, config_values in search_space[task_key].items(): + if config_name == "category": + continue + if not isinstance(config_values, list): + raise TypeError( + f"Search space values must be lists; got {task_key}.{config_name}={type(config_values)}" + ) + if not config_values: + raise ValueError(f"Empty search space for {task_key}.{config_name}") + + config_value = rng.choice(config_values) + name_parts.append(f"{config_name}={config_value}") + bindings.append( + ParameterBinding( + parameter=_get_param(task.config_spec, config_name), + value=config_value, + ) + ) + + if bindings: + config_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=task.config_spec, + bindings=bindings, + ) + + return PipelineConfig(tasks=tasks, config_catalog=config_catalog) + +def _task_param_assignments( + search_space: Dict[str, Dict[str, Any]], task_key: str +) -> List[Dict[str, Any]]: + space = search_space.get(task_key, {}) + param_space: Dict[str, List[Any]] = {k: v for k, v in space.items() if k != "category"} + if not param_space: + return [{}] + keys = list(param_space.keys()) + values_lists = [param_space[k] for k in keys] + return [dict(zip(keys, values)) for values in itertools.product(*values_lists)] + + +def _pipeline_config_for_combo_and_params( + search_space: Dict[str, Dict[str, Any]], + combo: List[str], + assignment_tuple: tuple[Dict[str, Any], ...], +) -> PipelineConfig: + tasks: List[KgTask] = [] + config_catalog: Dict[str, ConfigurationProfile] = {} + + for task_key, params in zip(combo, assignment_tuple): + task = task_dict[task_key] + tasks.append(task) + + if not params: + continue + if getattr(task, "config_spec", None) is None: + continue + + bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + + # Iterate in search_space order for stable snapshots. + for config_name, _config_values in search_space[task_key].items(): + if config_name == "category": + continue + if config_name not in params: + continue + config_value = params[config_name] + name_parts.append(f"{config_name}={config_value}") + bindings.append( + ParameterBinding( + parameter=_get_param(task.config_spec, config_name), + value=config_value, + ) + ) + + config_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=task.config_spec, + bindings=bindings, + ) + + return PipelineConfig(tasks=tasks, config_catalog=config_catalog) + + +def enumerate_snapshots_for_task_combo( + search_space: Dict[str, Dict[str, Any]], + combo: List[str], +) -> List[Dict[str, Any]]: + per_task_assignments = [ + _task_param_assignments(search_space, task_key) for task_key in combo + ] + snapshots: List[Dict[str, Any]] = [] + for assignment_tuple in itertools.product(*per_task_assignments): + pipeline_config = _pipeline_config_for_combo_and_params( + search_space, combo, assignment_tuple + ) + snapshots.append(pipeline_config_to_snapshot(combo, pipeline_config)) + return snapshots + + +def sample_unique_pipeline_config_snapshots_per_combo( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + n: int, + rng: random.Random, +) -> tuple[List[Dict[str, Any]], Dict[str, Any]]: + """ + Sample up to n unique profile snapshots per valid task combo. + + When a combo has fewer than n distinct profile assignments, all available + profiles are returned for that combo. + """ + if n < 1: + raise ValueError(f"n must be >= 1, got {n}") + + combos = enumerate_valid_task_combinations(search_space, pipeline_layout) + snapshots: List[Dict[str, Any]] = [] + combo_stats: List[Dict[str, Any]] = [] + + for combo in combos: + available_snapshots = enumerate_snapshots_for_task_combo(search_space, combo) + serialized = [json.dumps(s, sort_keys=True) for s in available_snapshots] + if len(set(serialized)) != len(serialized): + raise ValueError(f"Duplicate profile snapshots for combo {combo!r}") + + sample_count = min(n, len(available_snapshots)) + picked = ( + rng.sample(available_snapshots, k=sample_count) + if sample_count > 0 + else [] + ) + snapshots.extend(picked) + combo_stats.append( + { + "task_keys": combo, + "available_profiles": len(available_snapshots), + "requested": n, + "sampled": sample_count, + "exhausted": sample_count < n, + } + ) + + stats: Dict[str, Any] = { + "requested_n": n, + "total_combos": len(combos), + "total_snapshots": len(snapshots), + "combos_exhausted": sum(1 for row in combo_stats if row["exhausted"]), + "combos": combo_stats, + } + return snapshots, stats + + +def enumerate_exhaustive_pipeline_configs( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, +) -> List[PipelineConfig]: + """ + Enumerate every valid pipeline config in the search space. + + Unlike hierarchical sampling (task combo first, then params), this flattens + the full Cartesian product so each leaf config is equally likely when sampled. + """ + configs: List[PipelineConfig] = [] + for combo in enumerate_valid_task_combinations(search_space, pipeline_layout): + per_task_assignments = [ + _task_param_assignments(search_space, task_key) for task_key in combo + ] + for assignment_tuple in itertools.product(*per_task_assignments): + configs.append( + _pipeline_config_for_combo_and_params( + search_space, combo, assignment_tuple + ) + ) + return configs + + +def enumerate_exhaustive_pipeline_config_snapshots( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, +) -> List[Dict[str, Any]]: + combos = enumerate_valid_task_combinations(search_space, pipeline_layout) + + all_snapshots: List[Dict[str, Any]] = [] + total_expected = 0 + + for combo in combos: + per_task_assignments = [ + _task_param_assignments(search_space, task_key) for task_key in combo + ] + + expected_for_combo = 1 + for assignments in per_task_assignments: + expected_for_combo *= len(assignments) + total_expected += expected_for_combo + + produced_for_combo = 0 + print() + print("combo:", combo) + print("expected configs:", expected_for_combo) + + for assignment_tuple in itertools.product(*per_task_assignments): + produced_for_combo += 1 + if produced_for_combo % 100 == 1 or produced_for_combo == expected_for_combo: + print(f"config {produced_for_combo}/{expected_for_combo}") + + pipeline_config = _pipeline_config_for_combo_and_params( + search_space, combo, assignment_tuple + ) + all_snapshots.append(pipeline_config_to_snapshot(combo, pipeline_config)) + + assert produced_for_combo == expected_for_combo + + print() + print("TOTAL expected configs:", total_expected) + print("TOTAL generated snapshots:", len(all_snapshots)) + return all_snapshots diff --git a/experiments/param-opti/src/kgpipe_search/definitions.py b/experiments/param-opti/src/kgpipe_search/definitions.py new file mode 100644 index 0000000..df8ac24 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/definitions.py @@ -0,0 +1,212 @@ +from pydantic import BaseModel +from typing import List, Dict, Optional +from kgpipe.common import KgTask +from kgpipe.common.model.configuration import ConfigurationProfile +from pathlib import Path + +class PipelineLayout(BaseModel): + """ + allowed task categories in the pipeline + """ + allowed_task_categories: List[str] + + +class PipelineConfig(BaseModel): + tasks: List[KgTask] + config_catalog: Dict[str, ConfigurationProfile] + result_path: Optional[Path] = None + seed_path: Optional[Path] = None + + + +RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "rdf_sampled_pipeline_configs.json" +_RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "rdf_unique_sampled_pipeline_configs.json" +_RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "rdf_exhaustive_pipeline_configs.json" +_RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "text_sampled_pipeline_configs.json" +_TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "text_unique_sampled_pipeline_configs.json" +_TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE = Path(__file__).resolve().parent.parent / "data" / "fixtures" / "text_exhaustive_pipeline_configs.json" +_TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION = 1 + +TEXT_PIPELINE_LAYOUT = PipelineLayout( + allowed_task_categories=["information_extraction", "entity_linking", "aggregate_entity_linking", "relation_linking", "aggregate_relation_linking", "construct_rdf", "fusion"] +) + +RDF_PIPELINE_LAYOUT = PipelineLayout( + allowed_task_categories=["ontology_matching", "entity_matching", "aggregate_matching_results", "fusion"] +) + +RDF_LAYOUT = [ + "graph_alignment" + "relation_matching" + "entity_matching" + "aggregate_matching" +] + + +RDF_SEARCH_SPACE = { + "graph_alignment_label_alias_embedding_transformer_task": { + "category": ["ontology_matching", "entity_matching", "aggregate_matching_results"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "relation_matcher_label_alias_embedding_transformer_task": { + "category": ["ontology_matching"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "entity_matcher_label_alias_embedding_transformer_task": { + "category": ["entity_matching"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "paris_ontology_matching_task": { + "category": ["ontology_matching"], + "ontology_matching_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "paris_entity_alignment_task": { + "category": ["entity_matching"], + "entity_matching_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "paris_graph_alignment_task": { + "category": ["ontology_matching", "entity_matching", "aggregate_matching_results"], + "entity_matching_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + "relation_matching_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "aggregate_matching_results_task": { + "category": ["aggregate_matching_results"], + }, + "fusion_first_value_task": { + "category": ["fusion"], + # "fusion_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "relation_linker_label_alias_embedding_transformer_task": { + "category": ["entity_linking"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "entity_linker_label_alias_embedding_transformer_task": { + "category": "entity_linking", + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, +} + +RDF_BASELINE_CONFIG = { + "profiles": { + "paris_graph_alignment_task": { + "bindings": [ + { + "parameter": "entity_matching_threshold", + "value": 0.9 + }, + { + "parameter": "relation_matching_threshold", + "value": 0.5 + } + ], + "profile_name": "paris_graph_alignment_entity_matching_threshold=0.9,relation_matching_threshold=0.5" + } + }, + "task_keys": [ + "paris_graph_alignment_task", + "fusion_first_value_task" + ] + } + +TEXT_LAYOUT = [ + "information_extraction" + "entity_linking" + "relation_linking" + "fusion" +] + +TEXT_SEARCH_SPACE = { + "corenlp_text_extraction_task": { + "category": ["information_extraction"], + # does not have config parameters + }, + "genie_text_extraction_task": { + "category": ["information_extraction"], + # does not have config parameters + }, + "spotlight_entity_linking_task": { + "category": ["entity_linking"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "relation_linker_label_alias_embedding_transformer_task": { + "category": ["relation_linking"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "entity_linker_label_alias_embedding_transformer_task": { + "category": ["entity_linking"], + "model_name": ["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"], + "similarity_threshold": [0.5, 0.6, 0.7, 0.8, 0.9], + }, + "aggregate_entity_linking_task": { + "category": ["aggregate_entity_linking"], + }, + "aggregate_relation_linking_task": { + "category": ["aggregate_relation_linking"], + }, + "generate_rdf_from_text_results_task": { + "category": ["construct_rdf"], + }, + "select_first_value_task": { + "category": ["fusion"], + }, +} + +from kgpipe_search.dev.tasks.paris import paris_graph_alignment_task, paris_entity_alignment_task, paris_ontology_matching_task +from kgpipe_search.dev.tasks.fusion import fusion_first_value_task +from kgpipe_search.dev.tasks.base_linker import relation_linker_label_alias_embedding_transformer_task, entity_linker_label_alias_embedding_transformer_task +from kgpipe_search.dev.tasks.base_matcher import ( + graph_alignment_label_alias_embedding_transformer_task, + relation_matcher_label_alias_embedding_transformer_task, + entity_matcher_label_alias_embedding_transformer_task, +) +from kgpipe_search.dev.tasks.corenlp import corenlp_text_extraction_task +from kgpipe_search.dev.tasks.genie import genie_text_extraction_task +from kgpipe_search.dev.tasks.spotlight import spotlight_entity_linking_task +from kgpipe_search.dev.tasks.matching_helpers import aggregate_matching_results_task +from kgpipe_search.dev.tasks.text_helpers import aggregate_entity_linking_task, aggregate_relation_linking_task +from kgpipe_search.dev.tasks.text_helpers import generate_rdf_from_text_results_task +from kgpipe_search.dev.tasks.select_lib import select_first_value_task + +TEXT_TASK_DICT = { + "corenlp_text_extraction_task": corenlp_text_extraction_task, + "genie_text_extraction_task": genie_text_extraction_task, + "spotlight_entity_linking_task": spotlight_entity_linking_task, + "relation_linker_label_alias_embedding_transformer_task": relation_linker_label_alias_embedding_transformer_task, + "entity_linker_label_alias_embedding_transformer_task": entity_linker_label_alias_embedding_transformer_task, + "select_first_value_task": select_first_value_task, + "aggregate_entity_linking_task": aggregate_entity_linking_task, + "aggregate_relation_linking_task": aggregate_relation_linking_task, + "generate_rdf_from_text_results_task": generate_rdf_from_text_results_task, +} + +RDF_TASK_DICT = { + "graph_alignment_label_alias_embedding_transformer_task": graph_alignment_label_alias_embedding_transformer_task, + "relation_matcher_label_alias_embedding_transformer_task": relation_matcher_label_alias_embedding_transformer_task, + "entity_matcher_label_alias_embedding_transformer_task": entity_matcher_label_alias_embedding_transformer_task, + "paris_ontology_matching_task": paris_ontology_matching_task, + "paris_entity_alignment_task": paris_entity_alignment_task, + "paris_graph_alignment_task": paris_graph_alignment_task, + "fusion_first_value_task": fusion_first_value_task, + "relation_linker_label_alias_embedding_transformer_task": relation_linker_label_alias_embedding_transformer_task, + "entity_linker_label_alias_embedding_transformer_task": entity_linker_label_alias_embedding_transformer_task, + "aggregate_matching_results_task": aggregate_matching_results_task, + # "fusion_union_task": fusion_union_task, +} + +task_dict = {**TEXT_TASK_DICT, **RDF_TASK_DICT} \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/__init__.py b/experiments/param-opti/src/kgpipe_search/dev/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/src/kgpipe_search/dev/docker/__init__.py b/experiments/param-opti/src/kgpipe_search/dev/docker/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/src/kgpipe_search/dev/docker/copy.py b/experiments/param-opti/src/kgpipe_search/dev/docker/copy.py new file mode 100644 index 0000000..9fc4918 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/docker/copy.py @@ -0,0 +1,262 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Literal, Protocol, Sequence + +from kgpipe_search.mounts import ScratchMount + +CopyStrategyName = Literal["hdfs", "bind"] + +DEFAULT_HADOOP_CONF_HOST = "/etc/hadoop/conf" +DEFAULT_HADOOP_CONF_CONTAINER = "/etc/hadoop/conf" +MINIMAL_HADOOP_CONF_CONTAINER = "/tmp/kgpipe-hadoop-conf" +DEFAULT_HDFS_PORT = 9000 +HOSTS_MOUNT_SOURCE = "/etc/hosts" +HOSTS_MOUNT_TARGET = "/etc/hosts" + + +def _normalize_namenode(namenode: str, *, default_port: int = DEFAULT_HDFS_PORT) -> str: + raw = namenode.strip().rstrip("/") + if raw.startswith("hdfs://"): + authority, _, path = raw[len("hdfs://") :].partition("/") + host = authority + else: + host, _, path = raw.partition("/") + path = f"/{path}" if path else "" + + if ":" not in host: + host = f"{host}:{default_port}" + + return f"hdfs://{host}{path}" + + +def _hosts_mount() -> dict[str, str]: + return { + "Type": "bind", + "Source": HOSTS_MOUNT_SOURCE, + "Target": HOSTS_MOUNT_TARGET, + "ReadOnly": True, + } + + +@dataclass(frozen=True) +class CopyTarget: + """Destination path for a copy strategy (e.g. HDFS directory).""" + + path: str + + +@dataclass(frozen=True) +class CopyContext: + """Source location on the node where the experiment wrote outputs.""" + + run_name: str + node_id: str + scratch: ScratchMount + + +@dataclass(frozen=True) +class CopyJobPlan: + image: str + command: Sequence[str] + env: dict[str, str] = field(default_factory=dict) + mounts: list[dict[str, str]] = field(default_factory=list) + + +class CopyStrategy(Protocol): + def build_job(self, *, context: CopyContext, destination: CopyTarget) -> CopyJobPlan: + ... + + +@dataclass(frozen=True) +class BindCopyStrategy: + """ + Copy scratch outputs to a host bind-mounted directory using `cp`. + + `destination.path` must be a **host path on every node** (same path), + e.g. an NFS mountpoint or any shared filesystem mounted consistently. + """ + + image: str = "alpine:3.20" + dest_container_path: str = "/dst" + scratch_container_path: str | None = None + + def build_job(self, *, context: CopyContext, destination: CopyTarget) -> CopyJobPlan: + scratch_root = self.scratch_container_path or context.scratch.container_path + src = f"{scratch_root}/{context.run_name}" + dst = f"{self.dest_container_path}/{context.run_name}" + + command = [ + "sh", + "-lc", + ( + "set -euo pipefail; " + f"test -d {src!r}; " + f"mkdir -p {dst!r}; " + f"cp -a {src!r}/. {dst!r}/; " + f"echo copied to {dst!r}" + ), + ] + + mounts: list[dict[str, str]] = [ + context.scratch.to_mount(), + { + "Type": "bind", + "Source": destination.path, + "Target": self.dest_container_path, + }, + ] + + env = { + "KGPIPE_RUN_ID": context.run_name, + "KGPIPE_SCRATCH": scratch_root, + "KGPIPE_BIND_DEST": destination.path, + } + + return CopyJobPlan(image=self.image, command=command, env=env, mounts=mounts) + + +def _hdfs_path(namenode: str | None, path: str, *, default_port: int = DEFAULT_HDFS_PORT) -> str: + normalized = path.rstrip("/") + if normalized.startswith("hdfs://"): + return normalized + if namenode is None: + return normalized + rel = normalized if normalized.startswith("/") else f"/{normalized}" + return f"{_normalize_namenode(namenode, default_port=default_port)}{rel}" + + +def _minimal_hadoop_conf_script(*, namenode: str, conf_dir: str) -> str: + nn = _normalize_namenode(namenode) + return ( + f"mkdir -p {conf_dir!r}; " + f"cat > {conf_dir!r}/core-site.xml <<'EOF'\n" + "\n" + "\n" + "\n" + " \n" + " fs.defaultFS\n" + f" {nn}\n" + " \n" + "\n" + "EOF\n" + f"export HADOOP_CONF_DIR={conf_dir!r}" + ) + + +@dataclass(frozen=True) +class HdfsCopyStrategy: + """ + Copy node-local scratch outputs to HDFS using the hdfs CLI. + + Configure either: + - `namenode` (+ optional `user`) for a minimal client setup, or + - `hadoop_conf_host` to mount cluster config from the node. + """ + + image: str = "apache/hadoop:3.3.6" + namenode: str | None = None + user: str | None = None + hadoop_conf_host: str | None = None + hadoop_conf_container: str = DEFAULT_HADOOP_CONF_CONTAINER + minimal_conf_container: str = MINIMAL_HADOOP_CONF_CONTAINER + mount_node_hosts: bool = True + hdfs_port: int = DEFAULT_HDFS_PORT + extra_env: dict[str, str] = field(default_factory=dict) + + def build_job(self, *, context: CopyContext, destination: CopyTarget) -> CopyJobPlan: + src = f"{context.scratch.container_path}/{context.run_name}" + hdfs_dst = _hdfs_path(self.namenode, destination.path, default_port=self.hdfs_port) + hdfs_dst = f"{hdfs_dst}/{context.run_name}" + + setup_parts: list[str] = [] + env: dict[str, str] = { + "KGPIPE_RUN_ID": context.run_name, + "KGPIPE_SCRATCH": context.scratch.container_path, + "KGPIPE_HDFS_DEST": hdfs_dst, + **self.extra_env, + } + + if self.user: + env["HADOOP_USER_NAME"] = self.user + + if self.namenode is not None: + setup_parts.append( + _minimal_hadoop_conf_script( + namenode=_normalize_namenode(self.namenode, default_port=self.hdfs_port), + conf_dir=self.minimal_conf_container, + ) + ) + elif self.hadoop_conf_host: + env["HADOOP_CONF_DIR"] = self.hadoop_conf_container + + setup = " && ".join(setup_parts) + prefix = f"{setup} && " if setup else "" + + command = [ + "bash", + "-lc", + ( + f"set -euo pipefail; " + f"{prefix}" + f"test -d {src!r}; " + f"hdfs dfs -mkdir -p {hdfs_dst!r}; " + f"hdfs dfs -put -f {src!r}/. {hdfs_dst!r}/; " + f"echo copied to {hdfs_dst!r}" + ), + ] + + mounts = [context.scratch.to_mount()] + if self.mount_node_hosts: + mounts.append(_hosts_mount()) + if self.namenode is None and self.hadoop_conf_host: + mounts.append( + { + "Type": "bind", + "Source": self.hadoop_conf_host, + "Target": self.hadoop_conf_container, + "ReadOnly": True, + } + ) + + return CopyJobPlan(image=self.image, command=command, env=env, mounts=mounts) + + +def copy_strategy_from_config(cfg: dict | str) -> CopyStrategy: + if isinstance(cfg, str): + cfg = {"type": cfg} + strategy_type = cfg.get("type", "hdfs") + if strategy_type in {"bind", "local", "cp"}: + return BindCopyStrategy( + image=cfg.get("image", "alpine:3.20"), + dest_container_path=cfg.get("dest_container_path", "/dst"), + scratch_container_path=cfg.get("scratch_container_path"), + ) + if strategy_type == "hdfs": + namenode = cfg.get("namenode") + hadoop_conf_host = cfg.get("hadoop_conf_host") + if hadoop_conf_host is None and namenode is None: + hadoop_conf_host = DEFAULT_HADOOP_CONF_HOST + + return HdfsCopyStrategy( + image=cfg.get("image", "apache/hadoop:3.3.6"), + namenode=namenode, + user=cfg.get("user"), + hadoop_conf_host=hadoop_conf_host, + hadoop_conf_container=cfg.get( + "hadoop_conf_container", DEFAULT_HADOOP_CONF_CONTAINER + ), + minimal_conf_container=cfg.get( + "minimal_conf_container", MINIMAL_HADOOP_CONF_CONTAINER + ), + mount_node_hosts=cfg.get("mount_node_hosts", True), + hdfs_port=int(cfg.get("hdfs_port", DEFAULT_HDFS_PORT)), + extra_env=cfg.get("extra_env", {}), + ) + raise ValueError(f"unsupported copy strategy: {strategy_type!r}") + + +def copy_target_from_config(cfg: dict | str) -> CopyTarget: + if isinstance(cfg, str): + return CopyTarget(path=cfg) + return CopyTarget(path=cfg["path"]) diff --git a/experiments/param-opti/src/kgpipe_search/dev/docker/mounts.py b/experiments/param-opti/src/kgpipe_search/dev/docker/mounts.py new file mode 100644 index 0000000..51638aa --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/docker/mounts.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +from dataclasses import dataclass + +DEFAULT_SCRATCH_HOST = "/local/d1/docker-scratch" +DEFAULT_SCRATCH_CONTAINER = "/local/d1/docker-scratch" + + +@dataclass(frozen=True) +class ScratchMount: + """Bind-mount a host scratch directory into the container.""" + + host_path: str = DEFAULT_SCRATCH_HOST + container_path: str = DEFAULT_SCRATCH_CONTAINER + + def to_mount(self) -> dict[str, str]: + return { + "Type": "bind", + "Source": self.host_path, + "Target": self.container_path, + } diff --git a/experiments/param-opti/src/kgpipe_search/dev/docker/swarm.py b/experiments/param-opti/src/kgpipe_search/dev/docker/swarm.py new file mode 100644 index 0000000..61c7d11 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/docker/swarm.py @@ -0,0 +1,463 @@ +from __future__ import annotations + +import json +import threading +import time +import uuid +from dataclasses import dataclass +from typing import Any, Dict, Literal, Optional, Sequence, Tuple + +from docker import DockerClient +from docker.errors import APIError, NotFound +from docker.models.services import Service +from docker.types import RestartPolicy, ServiceMode + +from kgpipe_search.copy import CopyContext, CopyStrategy, CopyTarget +from kgpipe_search.mounts import ( + ScratchMount, +) +@dataclass(frozen=True) +class SwarmJobResult: + service_id: str + service_name: str + run_name: str + node_id: Optional[str] + state: Literal["complete", "failed", "shutdown", "rejected", "orphaned"] + exit_code: Optional[int] + logs: str + + +@dataclass(frozen=True) +class SwarmRunResult: + job: SwarmJobResult + copy: SwarmJobResult | None = None + + +ResultFormat = Literal["float", "json", "logs", "exit_code"] + + +@dataclass(frozen=True) +class ResultSpec: + """Describes what to return from a finished container job.""" + + format: ResultFormat = "float" + json_key: str | None = "result" + require_exit_code: int | None = 0 + + +def _json_lines(logs: str) -> list[Any]: + parsed: list[Any] = [] + for line in logs.splitlines(): + line = line.strip() + if not line: + continue + try: + parsed.append(json.loads(line)) + except json.JSONDecodeError: + continue + return parsed + + +def _value_from_json(obj: Any, key: str | None) -> Any: + if key is None: + return obj + if not isinstance(obj, dict): + raise ValueError(f"expected JSON object to read key {key!r}, got {type(obj).__name__}") + if key not in obj: + raise ValueError(f"JSON object has no key {key!r}") + return obj[key] + + +def extract_job_result(job: SwarmJobResult, spec: ResultSpec | None = None) -> Any: + """Extract the requested value from a finished Swarm job.""" + spec = spec or ResultSpec() + + if spec.require_exit_code is not None and job.exit_code != spec.require_exit_code: + raise RuntimeError( + f"expected exit code {spec.require_exit_code}, got {job.exit_code} " + f"(state={job.state})" + ) + + if spec.format == "exit_code": + if job.exit_code is None: + raise ValueError("exit code unavailable") + return job.exit_code + + if spec.format == "logs": + return job.logs + + if spec.format == "json": + lines = _json_lines(job.logs) + if not lines: + raise ValueError("no JSON found in logs") + return _value_from_json(lines[-1], spec.json_key) + + if spec.format == "float": + return float(_parse_scalar_from_logs(job.logs, key=spec.json_key)) + + raise ValueError(f"unsupported result format: {spec.format!r}") + + +def _parse_scalar_from_logs(logs: str, *, key: str | None = "result") -> float | int | str: + stripped = logs.strip() + if not stripped: + raise ValueError("empty logs") + + for obj in reversed(_json_lines(stripped)): + value = _value_from_json(obj, key) + if isinstance(value, (int, float)): + return value + if isinstance(value, str): + try: + return float(value) + except ValueError as e: + raise ValueError(f"JSON key {key!r} is not numeric: {value!r}") from e + raise ValueError(f"JSON key {key!r} is not numeric: {value!r}") + + return float(stripped.split()[-1]) + + +def parse_job_result(logs: str) -> float: + """Parse a float result from container stdout logs.""" + return float(_parse_scalar_from_logs(logs)) + + +class SwarmManager: + """ + Minimal Swarm "job runner" that launches one-shot services. + + Key feature: enforce a *per-node* cap by pinning each job to a node that has + fewer than `max_per_node` active tasks with our label. + """ + + DEFAULT_APP_LABEL_KEY = "kgpipe.job" + + def __init__(self, *, app_label_key: str = DEFAULT_APP_LABEL_KEY, app_label_value: str = "1"): + self.client = DockerClient.from_env() + self._label_key = app_label_key + self._label_value = app_label_value + self._schedule_lock = threading.Lock() + + def active_node_ids(self) -> list[str]: + node_ids: list[str] = [] + for node in self._nodes(): + node_id = node.get("ID") + if not node_id: + continue + + status_state = (((node.get("Status") or {}).get("State")) or "").lower() + availability = (((node.get("Spec") or {}).get("Availability")) or "").lower() + if status_state != "ready": + continue + if availability and availability != "active": + continue + + node_ids.append(node_id) + return node_ids + + def _nodes(self) -> Sequence[dict]: + return self.client.api.nodes() + + def _active_task_counts_by_node(self) -> Dict[str, int]: + """ + Count active tasks for our app label, grouped by NodeID. + + "Active" includes tasks that are accepted/starting/running/preparing, + i.e. still occupying a container slot. + """ + tasks = self.client.api.tasks( + filters={ + "label": [f"{self._label_key}={self._label_value}"], + "desired-state": ["running"], + } + ) + counts: Dict[str, int] = {} + for t in tasks: + node_id = t.get("NodeID") + if not node_id: + continue + st = (((t.get("Status") or {}).get("State")) or "").lower() + if st in {"new", "pending", "assigned", "accepted", "preparing", "starting", "running"}: + counts[node_id] = counts.get(node_id, 0) + 1 + return counts + + def _pick_node_with_capacity(self, *, max_per_node: int) -> Optional[str]: + nodes = self._nodes() + if not nodes: + return None + + counts = self._active_task_counts_by_node() + + eligible: list[Tuple[str, int]] = [] + for n in nodes: + node_id = n.get("ID") + if not node_id: + continue + + status_state = (((n.get("Status") or {}).get("State")) or "").lower() + availability = (((n.get("Spec") or {}).get("Availability")) or "").lower() + if status_state != "ready": + continue + if availability and availability != "active": + continue + + eligible.append((node_id, counts.get(node_id, 0))) + + if not eligible: + return None + + eligible.sort(key=lambda x: x[1]) + node_id, used = eligible[0] + if used >= max_per_node: + return None + return node_id + + def run_job( + self, + *, + image: str, + command: Optional[Sequence[str]] = None, + args: Optional[Sequence[str]] = None, + env: Optional[Dict[str, str]] = None, + parameter: Optional[str] = None, + node_id: Optional[str] = None, + max_per_node: int = 1, + timeout_s: int = 60 * 60, + poll_interval_s: float = 1.0, + cleanup: bool = True, + extra_labels: Optional[Dict[str, str]] = None, + name_prefix: str = "kgpipe-exp", + run_name: Optional[str] = None, + scratch: ScratchMount | None = None, + mounts: Optional[list[dict]] = None, + ) -> SwarmJobResult: + """ + Launch a one-shot service (1 replica), wait for completion, fetch logs. + + The `parameter` is passed via env var `KGPIPE_PARAM` by default. + When `scratch` is set, the host scratch directory is bind-mounted and + `KGPIPE_RUN_ID` is set to `run_name` (default: job id) for per-run subdirs. + """ + if max_per_node <= 0: + raise ValueError("max_per_node must be >= 1") + + job_id = uuid.uuid4().hex[:12] + run_id = run_name or job_id + service_name = f"{name_prefix}-{job_id}" + + labels = { + self._label_key: self._label_value, + "kgpipe.job_id": job_id, + } + if extra_labels: + labels.update(extra_labels) + + env_list: list[str] = [] + if env: + env_list.extend([f"{k}={v}" for k, v in env.items()]) + if parameter is not None: + env_list.append(f"KGPIPE_PARAM={parameter}") + if scratch is not None: + env_list.append(f"KGPIPE_RUN_ID={run_id}") + env_list.append(f"KGPIPE_SCRATCH={scratch.container_path}") + + service_mounts = list(mounts or []) + if scratch is not None: + service_mounts.append(scratch.to_mount()) + + deadline = time.time() + timeout_s + last_err: Optional[Exception] = None + + service: Optional[Service] = None + pinned_node_id: Optional[str] = None + + while time.time() < deadline and service is None: + with self._schedule_lock: + if node_id is not None: + counts = self._active_task_counts_by_node() + if counts.get(node_id, 0) >= max_per_node: + pinned_node_id = None + else: + pinned_node_id = node_id + else: + pinned_node_id = self._pick_node_with_capacity(max_per_node=max_per_node) + + if pinned_node_id is None: + pass + else: + mode = ServiceMode("replicated", replicas=1) + try: + service = self.client.services.create( + image=image, + command=list(command) if command else None, + args=list(args) if args else None, + env=env_list or None, + mounts=service_mounts or None, + name=service_name, + mode=mode, + labels=labels, + restart_policy=RestartPolicy(condition="none"), + constraints=[f"node.id=={pinned_node_id}"], + container_labels=labels, + ) + except APIError as e: + last_err = e + service = None + pinned_node_id = None + + if service is None: + time.sleep(poll_interval_s) + + if service is None: + raise RuntimeError("Unable to schedule job before timeout") from last_err + + try: + result = self._wait_service_done( + service, + timeout_s=max(1, int(deadline - time.time())), + poll_interval_s=poll_interval_s, + ) + finally: + if cleanup: + try: + service.remove() + except Exception: + pass + + return SwarmJobResult( + service_id=service.id, + service_name=service_name, + run_name=run_id, + node_id=pinned_node_id, + state=result["state"], + exit_code=result.get("exit_code"), + logs=result.get("logs", ""), + ) + + def copy_results( + self, + job: SwarmJobResult, + *, + scratch: ScratchMount, + strategy: CopyStrategy, + destination: CopyTarget, + max_per_node: int = 1, + timeout_s: int = 60 * 60, + poll_interval_s: float = 1.0, + cleanup: bool = True, + name_prefix: str = "kgpipe-copy", + ) -> SwarmJobResult: + """ + Launch a copy service on the same node as `job`, moving scratch outputs + to `destination` using the given copy strategy (e.g. HDFS). + """ + if job.node_id is None: + raise ValueError("cannot copy results: source job has no node_id") + if job.state != "complete" or job.exit_code != 0: + raise RuntimeError( + f"cannot copy results from unsuccessful job " + f"(state={job.state}, exit_code={job.exit_code})" + ) + + plan = strategy.build_job( + context=CopyContext( + run_name=job.run_name, + node_id=job.node_id, + scratch=scratch, + ), + destination=destination, + ) + + return self.run_job( + image=plan.image, + command=list(plan.command), + env=plan.env, + mounts=plan.mounts, + node_id=job.node_id, + run_name=job.run_name, + max_per_node=max_per_node, + timeout_s=timeout_s, + poll_interval_s=poll_interval_s, + cleanup=cleanup, + name_prefix=name_prefix, + ) + + def run_job_with_copy( + self, + *, + copy_strategy: CopyStrategy, + copy_destination: CopyTarget, + scratch: ScratchMount, + copy_on_success: bool = True, + **job_kwargs: Any, + ) -> SwarmRunResult: + """Run an experiment job and optionally copy its scratch outputs afterward.""" + job = self.run_job(scratch=scratch, **job_kwargs) + if not copy_on_success: + return SwarmRunResult(job=job) + + if job.state != "complete" or job.exit_code != 0: + return SwarmRunResult(job=job) + + copy_job = self.copy_results( + job, + scratch=scratch, + strategy=copy_strategy, + destination=copy_destination, + max_per_node=job_kwargs.get("max_per_node", 1), + timeout_s=job_kwargs.get("timeout_s", 60 * 60), + poll_interval_s=job_kwargs.get("poll_interval_s", 1.0), + cleanup=job_kwargs.get("cleanup", True), + ) + return SwarmRunResult(job=job, copy=copy_job) + + def _wait_service_done( + self, service: Service, *, timeout_s: int, poll_interval_s: float + ) -> Dict[str, Any]: + deadline = time.time() + timeout_s + + def collect_logs() -> str: + try: + raw = service.logs(stdout=True, stderr=True) + if raw is None: + return "" + if isinstance(raw, (bytes, bytearray)): + return bytes(raw).decode("utf-8", errors="replace") + chunks: list[bytes] = [] + for c in raw: + if isinstance(c, (bytes, bytearray)): + chunks.append(bytes(c)) + else: + chunks.append(str(c).encode("utf-8", errors="replace")) + return b"".join(chunks).decode("utf-8", errors="replace") + except Exception: + return "" + + while time.time() < deadline: + try: + tasks = service.tasks() + except (APIError, NotFound): + return {"state": "orphaned", "exit_code": None, "logs": ""} + + if not tasks: + time.sleep(poll_interval_s) + continue + + t = tasks[0] + status = (t.get("Status") or {}) + state = (status.get("State") or "").lower() + + if state in {"complete", "failed", "shutdown", "rejected"}: + exit_code = None + container_status = status.get("ContainerStatus") or {} + if "ExitCode" in container_status: + exit_code = container_status.get("ExitCode") + + logs = collect_logs() + + return {"state": state, "exit_code": exit_code, "logs": logs} + + time.sleep(poll_interval_s) + + logs = collect_logs() + return {"state": "shutdown", "exit_code": None, "logs": logs} diff --git a/experiments/param-opti/src/kgpipe_search/dev/execution.py b/experiments/param-opti/src/kgpipe_search/dev/execution.py new file mode 100644 index 0000000..4752e5b --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/execution.py @@ -0,0 +1,137 @@ +# Wrapper for pipeline execution and evaluation + +from __future__ import annotations + +from typing import Any, Dict + +from kgpipe_search.copy import ( + copy_strategy_from_config, + copy_target_from_config, +) +from kgpipe_search.mounts import ( + DEFAULT_SCRATCH_CONTAINER, + DEFAULT_SCRATCH_HOST, + ScratchMount, +) +from kgpipe_search.swarm import ResultSpec, SwarmManager, extract_job_result + +type KG = str +type Source = str + +type ConfigSpace = Dict[str, Any] +type Config = Dict[str, Any] + +type EvaluationResult = float + +_swarm = SwarmManager() + + +def _result_spec_from_config(config: Config) -> ResultSpec: + require_exit_code = config.get("require_exit_code", 0) + if require_exit_code == "any": + require_exit_code = None + + return ResultSpec( + format=config.get("result_format", "float"), + json_key=config.get("result_key", "result"), + require_exit_code=require_exit_code, + ) + + +def _scratch_from_config(config: Config) -> ScratchMount | None: + scratch = config.get("scratch") + if scratch is None: + return None + if isinstance(scratch, ScratchMount): + return scratch + if isinstance(scratch, str): + return ScratchMount(host_path=scratch) + if isinstance(scratch, dict): + return ScratchMount( + host_path=scratch.get("host_path", DEFAULT_SCRATCH_HOST), + container_path=scratch.get("container_path", DEFAULT_SCRATCH_CONTAINER), + ) + raise ValueError(f"invalid scratch config: {scratch!r}") + + +def execute_pipeline(kg: KG, source: Source, config: Config) -> float: + pass + + +def execute_pipeline_docker(kg: KG, source: Source, config: Config) -> float: + pass + + +def execute_pipeline_docker_swarm(kg: KG, source: Source, config: Config) -> Any: + """ + Execute a single experiment in Swarm and return a result. + + Expected `config` keys (minimal): + - image: str (required) + - parameter: str (optional) passed via `KGPIPE_PARAM` + + Result handling: + - result_format: "float" | "json" | "logs" | "exit_code" (default "float") + - result_key: JSON field to read for "float"/"json" (default "result") + - require_exit_code: expected exit code (default 0); use "any" to skip check + + Scratch (optional): + - scratch: host path str, or dict with host_path/container_path + - run_name: per-run subdirectory under scratch (default: auto job id) + + Copy (optional, requires scratch): + - copy: destination path str, or dict with strategy/destination + HDFS strategy supports either namenode+user or hadoop_conf_host + + Float format contract: + - Container prints JSON with a numeric `result` field, or a bare float + as the last token in logs. + """ + scratch = _scratch_from_config(config) + copy_cfg = config.get("copy") + job_kwargs = { + "image": config["image"], + "command": config.get("command"), + "args": config.get("args"), + "env": config.get("env"), + "parameter": config.get("parameter"), + "max_per_node": int(config.get("max_per_node", 1)), + "timeout_s": int(config.get("timeout_s", 60 * 60)), + "extra_labels": {"kgpipe.kg": str(kg), "kgpipe.source": str(source)}, + "run_name": config.get("run_name"), + "scratch": scratch, + "mounts": config.get("mounts"), + } + + if copy_cfg is not None: + if scratch is None: + raise ValueError("copy requires scratch to be configured") + if isinstance(copy_cfg, str): + strategy = copy_strategy_from_config({"type": "hdfs"}) + destination = copy_target_from_config(copy_cfg) + else: + strategy = copy_strategy_from_config(copy_cfg.get("strategy", {"type": "hdfs"})) + destination = copy_target_from_config( + copy_cfg.get("destination", copy_cfg.get("path")) + ) + run = _swarm.run_job_with_copy( + copy_strategy=strategy, + copy_destination=destination, + scratch=scratch, + **job_kwargs, + ) + res = run.job + if run.copy is not None and run.copy.exit_code not in (0, None): + raise RuntimeError( + f"Swarm copy stage failed (state={run.copy.state}, exit_code={run.copy.exit_code})" + ) + else: + res = _swarm.run_job(**job_kwargs) + + try: + return extract_job_result(res, _result_spec_from_config(config)) + except (ValueError, RuntimeError) as e: + raise RuntimeError( + f"Swarm job result extraction failed " + f"(state={res.state}, exit_code={res.exit_code})" + ) from e diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/__init__.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/__init__.py new file mode 100644 index 0000000..bb421c4 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/__init__.py @@ -0,0 +1,4 @@ +from .paris import paris_entity_alignment_task, paris_graph_alignment_task +from .fusion import fusion_first_value_task, fusion_union_task + +__all__ = ["paris_entity_matching_task", "paris_exchange_task", "fusion_first_value_task", "fusion_union_task"] \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/agreementmaker.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/agreementmaker.py new file mode 100644 index 0000000..716457b --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/agreementmaker.py @@ -0,0 +1,14 @@ +from kgpipe.common import Data, DataFormat, KgTask, Registry, TaskInput, TaskOutput, BasicTaskCategoryCatalog + +@Registry.task( + input_spec={"source": DataFormat.RDF, "target": DataFormat.RDF}, + output_spec={"output": DataFormat.AGREEMENTMAKER_RDF}, + description="Perform entity matching using AgreementMaker", + category=[BasicTaskCategoryCatalog.entity_matching] +) +def entity_matching_aggrement_maker(inputs: TaskInput, outputs: TaskOutput): + """Perform entity matching using AgreementMaker.""" + source_data = inputs["source"] + target_data = inputs["target"] + output_data = outputs["output"] + return output_data \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker.py new file mode 100644 index 0000000..aee7932 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker.py @@ -0,0 +1,45 @@ +from kgpipe.common import TaskInput, TaskOutput, Data, DataFormat, KgTask +from kgpipe.common.model.configuration import ConfigurationProfile, ConfigurationDefinition, Parameter, ParameterType + +def relation_linker_label_alias_embedding_transformer_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + Link relations using a base transformer model. + """ + from param_opti.tasks.base_linker_lib import label_alias_embedding_rl + label_alias_embedding_rl(inputs, outputs, model_name=config.get_parameter_value("model_name"), threshold=config.get_parameter_value("similarity_threshold")) + +relation_linker_label_alias_embedding_transformer_task = KgTask( + name="relation_linker_label_alias_embedding_transformer", + function=relation_linker_label_alias_embedding_transformer_function, + input_spec={"source": DataFormat.TE_JSON, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.TE_JSON}, + config_spec=ConfigurationDefinition( + name="relation_linker_label_alias_embedding_transformer", + parameters=[ + Parameter(name="model_name", native_keys=["--model-name"], datatype=ParameterType.string, default_value="sentence-transformers/all-MiniLM-L6-v2", required=True, allowed_values=["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"]), + Parameter(name="similarity_threshold", native_keys=["--similarity-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) + +def entity_linker_label_alias_embedding_transformer_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + Link entities using a base transformer model. + """ + from param_opti.tasks.base_linker_lib import label_alias_embedding_el + label_alias_embedding_el(inputs, outputs, model_name=config.get_parameter_value("model_name"), threshold=config.get_parameter_value("similarity_threshold")) + +entity_linker_label_alias_embedding_transformer_task = KgTask( + name="entity_linker_label_alias_embedding_transformer", + function=entity_linker_label_alias_embedding_transformer_function, + input_spec={"source": DataFormat.TE_JSON, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.TE_JSON}, + config_spec=ConfigurationDefinition( + name="entity_linker_label_alias_embedding_transformer", + parameters=[ + Parameter(name="model_name", native_keys=["--model-name"], datatype=ParameterType.string, default_value="sentence-transformers/all-MiniLM-L6-v2", required=True, allowed_values=["sentence-transformers/all-MiniLM-L6-v2", "sentence-transformers/all-mpnet-base-v2", "intfloat/e5-base-v2"]), + Parameter(name="similarity_threshold", native_keys=["--similarity-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) + diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker_lib.py new file mode 100644 index 0000000..1b0b12d --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_linker_lib.py @@ -0,0 +1,216 @@ +import json +import os +from abc import ABC, abstractmethod +from typing import Dict, List + +import numpy as np +import torch +from kgcore.api.ontology import OntologyUtil, OwlProperty +from kgpipe.common import Data, DataFormat, Registry +from kgpipe_tasks.transform_interop.exchange.text_extraction import TE_Document, TE_Pair +from rdflib import Graph, RDFS +from sentence_transformers import SentenceTransformer, util +from tqdm import tqdm + +_models: Dict[str, SentenceTransformer] = {} + +class Embedder(ABC): + def __init__(self, embedder_name: str): + self.embedder_name = embedder_name + + @abstractmethod + def encode_as_dict(self, texts: List[str]) -> Dict[str, np.ndarray]: + pass + + @abstractmethod + def encode(self, texts: List[str]) -> np.ndarray: + pass + + +def get_model(model_name: str) -> SentenceTransformer: + if model_name not in _models: + model = SentenceTransformer(model_name) + if torch.cuda.is_available(): + model.to(torch.cuda.current_device()) + _models[model_name] = model + return _models[model_name] + +class SentenceTransformerEmbedder(Embedder): + def __init__(self, model_name: str): + super().__init__("sentence-transformer") + self.model_name = model_name + + def encode_as_dict(self, text_list: List[str]) -> Dict[str, np.ndarray]: + embeddings = self.encode(text_list) + return {text: embedding for text, embedding in zip(text_list, embeddings)} + + def encode(self, text_list: List[str]) -> np.ndarray: + embeddings = get_model(self.model_name).encode(text_list, show_progress_bar=False) + return embeddings + +class EntityMatch: + def __init__(self, entity: str, label: str, score: float): + self.entity = entity + self.label = label + self.score = score + + +def _validate_embedding_dimensions(query_embeddings: np.ndarray, target_embeddings: np.ndarray) -> None: + if query_embeddings.shape[1] != target_embeddings.shape[1]: + raise ValueError( + "Embedding dimension mismatch: " + f"{query_embeddings.shape[1]} vs {target_embeddings.shape[1]}" + ) + + +class AliasAndLabelBasedEntityLinker: + """ + Link extracted entity mentions to graph resources using label embeddings. + """ + + def __init__(self, graph: Graph, model_name: str = "all-MiniLM-L6-v2", threshold: float = 0.0): + self.graph = graph + self.embedder = SentenceTransformerEmbedder(model_name=model_name) + self.threshold = float(threshold) + self.entity_uri_label_tuples = [ + (entity_uri, str(label)) + for entity_uri, _, label in self.graph.triples((None, RDFS.label, None)) + ] + entity_texts = [label for _, label in self.entity_uri_label_tuples] + self.entity_embeddings = self.embedder.encode(entity_texts) + + def link_entities(self, extracted_entities: List[str]) -> List[EntityMatch]: + if not extracted_entities: + return [] + + best_matches = [] + key_embeddings = self.embedder.encode(extracted_entities) + _validate_embedding_dimensions(key_embeddings, self.entity_embeddings) + similarities = util.cos_sim(key_embeddings, self.entity_embeddings) + + for i, entity in enumerate(extracted_entities): + best_idx = int(similarities[i].argmax()) + best_score = float(similarities[i][best_idx]) + if best_score < self.threshold: + continue + entity_uri, _ = self.entity_uri_label_tuples[best_idx] + best_matches.append(EntityMatch(entity, entity_uri, best_score)) + + return best_matches + + + +def label_alias_embedding_el(inputs: Dict[str, Data], outputs: Dict[str, Data], model_name: str = "all-MiniLM-L6-v2", threshold: float = 0.5): + graph = Graph() + graph.parse(inputs["target"].path, format="nt") + linker = AliasAndLabelBasedEntityLinker(graph, model_name=model_name, threshold=threshold) + + if os.path.isdir(inputs["source"].path): + os.makedirs(outputs["output"].path, exist_ok=True) + for file in tqdm(os.listdir(inputs["source"].path), desc="Linking entities"): + te_doc_in = TE_Document(**json.load(open(os.path.join(inputs["source"].path, file)))) + entity_texts = list({triple.subject.surface_form for triple in te_doc_in.triples if triple.subject.surface_form}) + entity_texts += list({triple.object.surface_form for triple in te_doc_in.triples if triple.object.surface_form}) + entity_matches = linker.link_entities(entity_texts) + te_links = [TE_Pair(span=match.entity, mapping=match.label, link_type="entity", score=match.score) for match in entity_matches] + te_doc_out = te_doc_in.model_copy(deep=True) + te_doc_out.links += te_links + with open(os.path.join(outputs["output"].path, file), "w") as f: + f.write(te_doc_out.model_dump_json()) + else: + te_doc_in = TE_Document(**json.load(open(inputs["source"].path))) + entity_matches = linker.link_entities(list({triple.subject.surface_form for triple in te_doc_in.triples if triple.subject.surface_form})) + te_links = [TE_Pair(span=match.entity, mapping=match.label, link_type="entity", score=match.score) for match in entity_matches] + te_doc_out = te_doc_in.model_copy(deep=True) + te_doc_out.links += te_links + with open(outputs["output"].path, "w") as f: + f.write(te_doc_out.model_dump_json()) + + + +class RelationMatch: + def __init__(self, relation: str, predicate: OwlProperty, score: float): + self.relation = relation + self.predicate = predicate + self.score = score + + def __str__(self): + return f"RelationMatch(relation={self.relation}, predicate={self.predicate.uri}, score={self.score})" + + +def normalize(text): + return text.replace('_', ' ').replace('-', ' ').strip().lower() + +def build_property_text(prop: OwlProperty): + text_parts = [ + f"label: {normalize(prop.label)}", + f"altLabels: {', '.join(normalize(lbl) for lbl in prop.alias)}" + # f"domain: {normalize(prop.get('domain', ''))}", + # f"comment: {normalize(prop.get('comment', ''))}" + ] + return "; ".join(text_parts) + +class AliasAndTransformerBasedRelationLinker: + """ + Link extracted relation phrases to ontology predicates using label and alias embeddings. + """ + + def __init__(self, ontology_file, model_name: str = "all-MiniLM-L6-v2", threshold: float = 0.0): + print(f"Init AliasAndTransformerBasedRelationLinker with ontology file: {ontology_file} and model name: {model_name} and threshold: {threshold}") + self.ontology = OntologyUtil.load_ontology_from_file(ontology_file) + self.embedder = SentenceTransformerEmbedder(model_name=model_name) + self.threshold = float(threshold) + property_texts = [build_property_text(p) for p in self.ontology.properties] + self.property_embeddings = self.embedder.encode(property_texts) + + def link_relations(self, extracted_relations: List[str]) -> List[RelationMatch]: + if not extracted_relations: + return [] + + best_matches = [] + key_texts = [normalize(relation) for relation in extracted_relations] + key_embeddings = self.embedder.encode(key_texts) + _validate_embedding_dimensions(key_embeddings, self.property_embeddings) + similarities = util.cos_sim(key_embeddings, self.property_embeddings) + + for i, relation in enumerate(extracted_relations): + best_idx = int(similarities[i].argmax()) + best_score = float(similarities[i][best_idx]) + # print(f"Relation: {relation}, matched to: {self.ontology.properties[best_idx].uri}, label: {self.ontology.properties[best_idx].label}, Best Index: {best_idx}, Best Score: {best_score}") + if best_score < self.threshold: + continue + match = self.ontology.properties[best_idx] + best_matches.append(RelationMatch(relation, match, best_score)) + + return best_matches + + +def label_alias_embedding_rl(inputs: Dict[str, Data], outputs: Dict[str, Data], model_name: str = "all-MiniLM-L6-v2", threshold: float = 0.5): + + ontology_path = os.environ.get("ONTOLOGY_PATH", "false") + if ontology_path == "false": + raise ValueError("ONTOLOGY_PATH is not set") + else: + ontology_path = ontology_path + + linker = AliasAndTransformerBasedRelationLinker(ontology_path, model_name=model_name, threshold=threshold) + + if os.path.isdir(inputs["source"].path): + os.makedirs(outputs["output"].path, exist_ok=True) + for file in tqdm(os.listdir(inputs["source"].path), desc="Linking relations"): + te_doc_in = TE_Document(**json.load(open(os.path.join(inputs["source"].path, file)))) + relation_texts = list({triple.predicate.surface_form for triple in te_doc_in.triples if triple.predicate.surface_form}) + relation_matches = linker.link_relations(relation_texts) + te_links = [TE_Pair(span=match.relation, mapping=match.predicate.uri, link_type="predicate", score=match.score) for match in relation_matches] + te_doc_out = te_doc_in.model_copy(deep=True) + te_doc_out.links += te_links + with open(os.path.join(outputs["output"].path, file), "w") as f: + f.write(te_doc_out.model_dump_json()) + else: + te_doc_in = TE_Document(**json.load(open(inputs["source"].path))) # TODO: check if this is correct + relation_matches = linker.link_relations(list({triple.predicate.surface_form for triple in te_doc_in.triples if triple.predicate.surface_form})) + te_links = [TE_Pair(span=match.relation, mapping=match.predicate.uri, link_type="predicate", score=match.score) for match in relation_matches] + te_doc_out = te_doc_in.model_copy(deep=True) + te_doc_out.links += te_links + with open(outputs["output"].path, "w") as f: + f.write(te_doc_out.model_dump_json()) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher.py new file mode 100644 index 0000000..223fdc4 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher.py @@ -0,0 +1,111 @@ +from kgpipe.common import TaskInput, TaskOutput, DataFormat +from kgpipe.common.model.configuration import ConfigurationDefinition, Parameter, ParameterType, ConfigurationProfile +from kgpipe.common.model.task import KgTask + +# Same as paris_graph_alignment_task / paris_entity_alignment_task: +# input_spec + output_spec as in experiments/param-opti/src/param_opti/tasks/paris.py (e.g. lines 58–59). +_ALIGNMENT_TWO_GRAPH_INPUT_SPEC = {"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES} +_ALIGNMENT_ER_JSON_OUTPUT_SPEC = {"output": DataFormat.ER_JSON} + + +def _embedding_config_params(): + return [ + Parameter( + name="model_name", + native_keys=["--model-name"], + datatype=ParameterType.string, + default_value="sentence-transformers/all-MiniLM-L6-v2", + required=True, + allowed_values=[ + "sentence-transformers/all-MiniLM-L6-v2", + "sentence-transformers/all-mpnet-base-v2", + "intfloat/e5-base-v2", + ], + ), + Parameter( + name="similarity_threshold", + native_keys=["--similarity-threshold"], + datatype=ParameterType.number, + default_value=0.5, + required=True, + allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], + ), + ] + + +def graph_alignment_label_alias_embedding_transformer_function( + inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile +): + """Match entities and relations between two RDF graphs (full graph alignment).""" + from param_opti.tasks.base_matcher_lib import label_embedding_graph_alignment_match + + label_embedding_graph_alignment_match( + inputs, + outputs, + model_name=config.get_parameter_value("model_name"), + threshold=float(config.get_parameter_value("similarity_threshold")), + ) + + +graph_alignment_label_alias_embedding_transformer_task = KgTask( + name="graph_alignment_label_alias_embedding_transformer", + function=graph_alignment_label_alias_embedding_transformer_function, + input_spec=dict(_ALIGNMENT_TWO_GRAPH_INPUT_SPEC), + output_spec=dict(_ALIGNMENT_ER_JSON_OUTPUT_SPEC), + config_spec=ConfigurationDefinition( + name="graph_alignment_label_alias_embedding_transformer", + parameters=_embedding_config_params(), + ), +) + + +def entity_matcher_label_alias_embedding_transformer_function( + inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile +): + """Entity alignment only (subject/object URIs with rdfs:label).""" + from param_opti.tasks.base_matcher_lib import label_embedding_entity_alignment_match + + label_embedding_entity_alignment_match( + inputs, + outputs, + model_name=config.get_parameter_value("model_name"), + threshold=float(config.get_parameter_value("similarity_threshold")), + ) + + +entity_matcher_label_alias_embedding_transformer_task = KgTask( + name="entity_matcher_label_alias_embedding_transformer", + function=entity_matcher_label_alias_embedding_transformer_function, + input_spec=dict(_ALIGNMENT_TWO_GRAPH_INPUT_SPEC), + output_spec=dict(_ALIGNMENT_ER_JSON_OUTPUT_SPEC), + config_spec=ConfigurationDefinition( + name="entity_matcher_label_alias_embedding_transformer", + parameters=_embedding_config_params(), + ), +) + + +def relation_matcher_label_alias_embedding_transformer_function( + inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile +): + """Relation / predicate alignment only.""" + from param_opti.tasks.base_matcher_lib import label_embedding_relation_alignment_match + + label_embedding_relation_alignment_match( + inputs, + outputs, + model_name=config.get_parameter_value("model_name"), + threshold=float(config.get_parameter_value("similarity_threshold")), + ) + + +relation_matcher_label_alias_embedding_transformer_task = KgTask( + name="relation_matcher_label_alias_embedding_transformer", + function=relation_matcher_label_alias_embedding_transformer_function, + input_spec=dict(_ALIGNMENT_TWO_GRAPH_INPUT_SPEC), + output_spec=dict(_ALIGNMENT_ER_JSON_OUTPUT_SPEC), + config_spec=ConfigurationDefinition( + name="relation_matcher_label_alias_embedding_transformer", + parameters=_embedding_config_params(), + ), +) diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher_lib.py new file mode 100644 index 0000000..6ca3f41 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/base_matcher_lib.py @@ -0,0 +1,234 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, Iterable, List, Optional, Sequence + +from rdflib import Graph, Literal, RDFS, URIRef +from sentence_transformers import util + +from kgpipe.common import Data +from kgpipe_tasks.transform_interop.exchange.entity_matching import ER_Document, ER_Match + +# Reuse the shared embedder/model cache from the linker implementation. +from param_opti.tasks.base_linker_lib import SentenceTransformerEmbedder, _validate_embedding_dimensions + + +def _normalize_label(text: str) -> str: + return " ".join(text.replace("_", " ").replace("-", " ").strip().lower().split()) + + +def _safe_first_literal(values: Iterable[object]) -> Optional[str]: + for v in values: + if isinstance(v, Literal): + s = str(v).strip() + if s: + return s + return None + + +def _fallback_label_from_uri(uri: URIRef) -> str: + s = str(uri) + if "#" in s: + return s.rsplit("#", 1)[-1] + return s.rsplit("/", 1)[-1] + + +@dataclass(frozen=True) +class _LabeledUri: + uri: URIRef + label: str + + +def _extract_labeled_entities(graph: Graph) -> List[_LabeledUri]: + """ + Extract subject/object URIRefs that have an rdfs:label. + """ + uris: set[URIRef] = set() + for s, _, o in graph: + if isinstance(s, URIRef): + uris.add(s) + if isinstance(o, URIRef): + uris.add(o) + + labeled: List[_LabeledUri] = [] + for u in uris: + label = _safe_first_literal(graph.objects(u, RDFS.label)) + if label: + labeled.append(_LabeledUri(u, label)) + return labeled + + +def _extract_labeled_predicates(graph: Graph) -> List[_LabeledUri]: + """ + Extract predicate URIRefs and use rdfs:label if present, otherwise fall back to local-name. + """ + preds: set[URIRef] = {p for _, p, _ in graph if isinstance(p, URIRef)} + labeled: List[_LabeledUri] = [] + for p in preds: + label = _safe_first_literal(graph.objects(p, RDFS.label)) or _fallback_label_from_uri(p) + labeled.append(_LabeledUri(p, label)) + return labeled + + +def _best_matches( + source: Sequence[_LabeledUri], + target: Sequence[_LabeledUri], + *, + model_name: str, + threshold: float, + id_type: str, +) -> List[ER_Match]: + if not source or not target: + return [] + + embedder = SentenceTransformerEmbedder(model_name=model_name) + src_texts = [_normalize_label(x.label) for x in source] + tgt_texts = [_normalize_label(x.label) for x in target] + + src_emb = embedder.encode(src_texts) + tgt_emb = embedder.encode(tgt_texts) + _validate_embedding_dimensions(src_emb, tgt_emb) + + sims = util.cos_sim(src_emb, tgt_emb) + matches: List[ER_Match] = [] + + for i, src in enumerate(source): + best_idx = int(sims[i].argmax()) + best_score = float(sims[i][best_idx]) + if best_score < float(threshold): + continue + tgt = target[best_idx] + matches.append( + ER_Match( + id_1=str(src.uri), + id_2=str(tgt.uri), + score=best_score, + id_type=id_type, + ) + ) + return matches + + +def _write_er_document(output_path: Path, matches: List[ER_Match]) -> None: + output_path.parent.mkdir(parents=True, exist_ok=True) + doc = ER_Document(matches=matches) + output_path.write_text(doc.model_dump_json(), encoding="utf-8") + + +def _label_embedding_match_two_graphs( + inputs: Dict[str, Data], + outputs: Dict[str, Data], + *, + model_name: str, + threshold: float, + include_entities: bool, + include_relations: bool, +) -> None: + source_graph = Graph() + source_graph.parse(inputs["source"].path, format="nt") + + target_graph = Graph() + target_graph.parse(inputs["target"].path, format="nt") + + matches: List[ER_Match] = [] + + if include_entities: + source_entities = _extract_labeled_entities(source_graph) + target_entities = _extract_labeled_entities(target_graph) + matches.extend( + _best_matches( + source_entities, + target_entities, + model_name=model_name, + threshold=threshold, + id_type="entity", + ) + ) + + if include_relations: + source_preds = _extract_labeled_predicates(source_graph) + target_preds = _extract_labeled_predicates(target_graph) + matches.extend( + _best_matches( + source_preds, + target_preds, + model_name=model_name, + threshold=threshold, + id_type="relation", + ) + ) + + _write_er_document(outputs["output"].path, matches) + + +def label_embedding_graph_alignment_match( + inputs: Dict[str, Data], + outputs: Dict[str, Data], + *, + model_name: str = "sentence-transformers/all-MiniLM-L6-v2", + threshold: float = 0.5, +) -> None: + """ + Align two RDF graphs: match subject/object entities by rdfs:label and predicates by label. + + Writes `ER_Document` JSON with both entity and relation matches (same shape as `paris_lib`). + """ + _label_embedding_match_two_graphs( + inputs, + outputs, + model_name=model_name, + threshold=threshold, + include_entities=True, + include_relations=True, + ) + + +def label_embedding_entity_alignment_match( + inputs: Dict[str, Data], + outputs: Dict[str, Data], + *, + model_name: str = "sentence-transformers/all-MiniLM-L6-v2", + threshold: float = 0.5, +) -> None: + """Entity alignment only: matches with id_type \"entity\".""" + _label_embedding_match_two_graphs( + inputs, + outputs, + model_name=model_name, + threshold=threshold, + include_entities=True, + include_relations=False, + ) + + +def label_embedding_relation_alignment_match( + inputs: Dict[str, Data], + outputs: Dict[str, Data], + *, + model_name: str = "sentence-transformers/all-MiniLM-L6-v2", + threshold: float = 0.5, +) -> None: + """Relation / predicate alignment only: matches with id_type \"relation\".""" + _label_embedding_match_two_graphs( + inputs, + outputs, + model_name=model_name, + threshold=threshold, + include_entities=False, + include_relations=True, + ) + + +def label_embedding_graph_match( + inputs: Dict[str, Data], + outputs: Dict[str, Data], + *, + model_name: str = "sentence-transformers/all-MiniLM-L6-v2", + threshold: float = 0.5, +) -> None: + """Backward-compatible alias for full graph alignment (entities + relations).""" + label_embedding_graph_alignment_match( + inputs, outputs, model_name=model_name, threshold=threshold + ) + diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp.py new file mode 100644 index 0000000..e45299b --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp.py @@ -0,0 +1,36 @@ +from typing import Dict + +from pathlib import Path +from kgpipe.common import Data, DataFormat, Registry, KgTask +from kgpipe.common.model.configuration import ConfigurationDefinition + + +def corenlp_text_extraction_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + from param_opti.tasks.corenlp_lip import corenlp_openie_extraction, corenlp_exchange + + # Ensure parent directory exists for the TE JSON output path + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + input_path: Path = inputs["input"].path + final_te_output: Data = outputs["output"] + + # 1) Produce intermediate OpenIE JSON (file or directory) + if input_path.is_dir(): + openie_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie_out" + else: + openie_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie.json" + + openie_output = {"output": Data(openie_out_path, DataFormat.OPENIE_JSON)} + corenlp_openie_extraction({"input": inputs["input"]}, openie_output) + + # 2) Convert OpenIE JSON → TE JSON (final output) + corenlp_exchange({"input": openie_output["output"]}, {"output": final_te_output}) + + +corenlp_text_extraction_task = KgTask( + name="corenlp_text_extraction", + input_spec={"input": DataFormat.TEXT}, + output_spec={"output": DataFormat.TE_JSON}, + function=corenlp_text_extraction_function, + description="Extract text using CoreNLP" +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp_lip.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp_lip.py new file mode 100644 index 0000000..4ef7559 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/corenlp_lip.py @@ -0,0 +1,148 @@ +from kgpipe.common import TaskInput, TaskOutput, Data, DataFormat, Registry, BasicTaskCategoryCatalog +from kgpipe.common.model.configuration import ConfigurationDefinition, Parameter, ParameterType, ConfigurationProfile +from kgpipe.common.model.task import KgTask + +def openie_pipeline_task_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + Run the openie pipeline + """ + pass + +@Registry.task( + input_spec={"input": DataFormat.TEXT}, + output_spec={"output": DataFormat.OPENIE_JSON}, + description="Extract OpenIE triples using Stanford CoreNLP", + category=["TextProcessing", "TextExtraction"] +) +def openie_pipeline_task(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + Run the openie pipeline + """ + pass + +""" +Stanford CoreNLP Information Extraction + +This module provides information extraction using Stanford CoreNLP. +""" + +import json +import os +from pathlib import Path +from typing import Dict, Any, List + +from kgpipe.common import KgTask, Data, DataFormat, Registry +from kgpipe.common.io import get_docker_volume_bindings, remap_data_path_for_container +from kgpipe.execution import docker_client + + +CORENLP_ENTRYPOINT = ["java", "-cp", "*", "edu.stanford.nlp.pipeline.StanfordCoreNLP"] + + +def corenlp_openie_extraction(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """Extract OpenIE triples using Stanford CoreNLP.""" + # input_data = inputs["input"] + # output_data = outputs["output"] + + # Setup Docker + all_data = list(inputs.values()) + list(outputs.values()) + volumes, host_to_container = get_docker_volume_bindings(all_data) + + print(inputs["input"]) + print(outputs["output"]) + # Remap paths for container + input_path = remap_data_path_for_container(inputs["input"], host_to_container) + output_path = remap_data_path_for_container(outputs["output"], host_to_container) + + # Create command + command = ["bash", "openie.sh", str(input_path.path), str(output_path.path)] + # CORENLP_ENTRYPOINT + [ + # "-annotators", "tokenize,pos,lemma,ner,parse,coref,openie", + # "-file", str(input_path.path), + # "-outputFormat", "json", + # "-outputDirectory", str(output_path.path) + # ] + + # Run container + client = docker_client( + image="kgt/corenlp:latest", + command=command, + volumes=volumes + ) + client() + + +def corenlp_exchange(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """Convert OpenIE JSON to IE JSON format.""" + input_path = inputs["input"].path + output_path = outputs["output"].path + + # create output folder + os.makedirs(os.path.dirname(output_path), exist_ok=True) + + def __openiejson2tejson(openiedata) -> Dict[str, Any]: + """Convert OpenIE JSON to TE Document format.""" + doc = {"triples": [], "chains": []} + + # Convert to triples + triplets = [] + for sentence in openiedata.get('sentences', []): + for triple_span in sentence.get('openie', []): + triplet = { + "subject": {"surface_form": triple_span.get('subject', '')}, + "predicate": {"surface_form": triple_span.get('relation', '')}, + "object": {"surface_form": triple_span.get('object', '')} + } + triplets.append(triplet) + + # Get chains (simplified) + chains = get_coreference_chains(openiedata) + + doc["triples"] = triplets + doc["chains"] = chains + return doc + + if os.path.isdir(input_path): + os.makedirs(output_path, exist_ok=True) + for file in os.listdir(input_path): + # Read input json + with open(os.path.join(input_path, file), 'r') as f: + data = json.load(f) + te_doc = __openiejson2tejson(data) + outfile = os.path.join(output_path, file) + + with open(outfile, 'w') as of: + json.dump(te_doc, of) + # print(f"Converted {input_path} to {outfile}") + + else: + # Read input json + with open(input_path, 'r') as f: + data = json.load(f) + te_doc = __openiejson2tejson(data) + with open(output_path, 'w') as of: + json.dump(te_doc, of) + # print(f"Converted {input_path} to {output_path}") + + +def get_coreference_chains(response: dict) -> List[Dict[str, Any]]: + """Extract coreference chains from CoreNLP response.""" + result = [] + for _, coref in response.get('corefs', {}).items(): + if len(coref) > 1: + chain = {"main": coref[0].get('text', '')} + alias = [] + for chunk in coref[1:]: + sentence = response.get('sentences', [])[chunk.get('sentNum', 1) - 1] + start = sentence.get('tokens', [])[chunk.get('startIndex', 1) - 1].get('characterOffsetBegin', 0) + end = sentence.get('tokens', [])[chunk.get('endIndex', 2) - 2].get('characterOffsetEnd', 0) + alias.append({ + "surface_form": chunk.get('text', ''), + "text": chunk.get('text', ''), + "start": start, + "end": end + }) + chain["aliases"] = alias + result.append(chain) + return result + diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/formats.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/formats.py new file mode 100644 index 0000000..97d1a3e --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/formats.py @@ -0,0 +1,8 @@ + +# reimport and define of used formats + +from kgpipe.common import DataFormat +from kgpipe.common.model.default_catalog import BasicDataFormats, CustomDataFormats + +class ExtendedFormats(CustomDataFormats): + pass \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion.py new file mode 100644 index 0000000..21a37ad --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion.py @@ -0,0 +1,51 @@ +import os + +from kgpipe.common.model.configuration import ConfigurationProfile +from kgpipe.common.model.configuration import ConfigurationDefinition, Parameter, ParameterType +from kgpipe.common.models import TaskInput, TaskOutput, KgTask, DataFormat +from kgpipe.common import Registry + +def fusion_first_value_function( + inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile | None = None +): + from param_opti.tasks.fusion_lib import fusion_first_value + + if config is not None: + ontology_path = config.get_parameter_value("ontology_path") + else: + ontology_path = os.environ.get("ONTOLOGY_PATH", "") + # TODO remove thresholds as they are applied by the matchers + fusion_first_value( + inputs, + outputs, + entity_matching_threshold=0.0, + relation_matching_threshold=0.0, + ontology_path=ontology_path, + ) + +fusion_first_value_task = KgTask( + name="fusion_first_value_task", + function=fusion_first_value_function, + input_spec={"source": DataFormat.RDF_NTRIPLES, "kg": DataFormat.RDF_NTRIPLES, "matches1": DataFormat.ER_JSON}, + output_spec={"output": DataFormat.RDF_NTRIPLES} + # config_spec=ConfigurationDefinition( + # name="fusion_first_value", + # parameters=[ + # # ontology path + # Parameter(name="ontology_path", native_keys=["--ontology-path"], datatype=ParameterType.string, default_value="", required=True), + # ] + # ) +) +Registry.add_task(fusion_first_value_task.name, fusion_first_value_task) + +def fusion_union_function(inputs: TaskInput, outputs: TaskOutput): + # touch output file + outputs["output"].path.touch() + +fusion_union_task = KgTask( + name="fusion_union_task", + function=fusion_union_function, + input_spec={"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES, "matches": DataFormat.ER_JSON}, + output_spec={"output": DataFormat.RDF_NTRIPLES}, +) +Registry.add_task(fusion_union_task.name, fusion_union_task) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion_lib.py new file mode 100644 index 0000000..0581ad7 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/fusion_lib.py @@ -0,0 +1,205 @@ + +from kgpipe.common.models import KgTask, DataFormat, Data +from logging import getLogger + +from pydantic import BaseModel +from rdflib import OWL, Graph, URIRef, RDFS, RDF, SKOS +from pathlib import Path +import json +import os +from kgcore.api.ontology import OntologyUtil +from kgpipe.common.config import TARGET_ONTOLOGY_NAMESPACE +from typing import Dict, List +from kgpipe_tasks.entity_resolution.fusion.util import load_matches_from_file + +SINGLE_CANDIDATE_CHECK: bool=False + +logger = getLogger(__name__) + +class TrackRecord(BaseModel): + original_subject: str + subject: str + original_predicate: str + predicate: str + original_object: str + object: str + +def select_first_value(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """ + For two KGs A and B, merge A into B where for each s_p and + 1) p is fusable and B does not have any s_p_o or + 2) p is not fusable erge all s_p_o + """ + ontology_path = os.environ.get("ONTOLOGY_PATH", "false") + if ontology_path == "false": + raise ValueError("ONTOLOGY_PATH is not set") + + ontology = OntologyUtil.load_ontology_from_file(Path(ontology_path)) + allowed_predicates = set[str]([str(p.uri) for p in ontology.properties]+[str(RDFS.label), str(RDF.type), str(SKOS.altLabel)]) + fusable_properties = set[str]([str(p.uri) for p in ontology.properties if p.max_cardinality == 1]+[str(RDFS.label), str(RDF.type)]) + + def is_fusable(p): + return str(p) in fusable_properties + + source_graph = Graph() + source_graph.parse(inputs["source"].path, format="nt") + seed_graph = Graph() # seed graph + seed_graph.parse(inputs["target"].path, format="nt") + + current_subjects = set[str]([str(s) for s in seed_graph.subjects(unique=True)]) + + selected: List[TrackRecord] = [] + discarded: List[TrackRecord] = [] + + for s, p, o in source_graph: + s_can = s + p_can = p + o_can = o + + if not isinstance(p_can, URIRef) or str(p_can) not in allowed_predicates: + continue + + if p_can == RDF.type and not str(o_can).startswith(TARGET_ONTOLOGY_NAMESPACE): + continue + + if is_fusable(p_can): + # Add exactly one value if none exists yet + if not any(seed_graph.objects(s_can, p_can)): + seed_graph.add((s_can, p_can, o_can)) + selected.append( + TrackRecord(subject=s_can,predicate=p_can,object=o,original_subject=s,original_predicate=p,original_object=o)) + # keep subjects set fresh for subsequent matches + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + else: + discarded.append( + TrackRecord(subject=s_can,predicate=p_can,object=o,original_subject=s,original_predicate=p,original_object=o)) + else: + # Non-fusable: copy if not already present (avoid dupes) + if (s_can, p_can, o_can) not in seed_graph: + seed_graph.add((s_can, p_can, o_can)) + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + + # sel(ected) + selected_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".selected.json") + with open(selected_file_path, "w") as f: + json.dump(selected, f, default=lambda x: x.model_dump()) + # dis(carded) + discarded_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".discarded.json") + with open(discarded_file_path, "w") as f: + json.dump(discarded, f, default=lambda x: x.model_dump()) + + # prov graph is skipped here as no uris are replaced (is done in previouse steps) + seed_graph.serialize(outputs["output"].path, format="nt") + +def fusion_first_value(inputs: Dict[str, Data], outputs: Dict[str, Data], entity_matching_threshold: float, relation_matching_threshold: float, ontology_path: str): + """ + Fuse RDF entities + - replacing ids of target graph with ids of source graph based on matches + - only fusable properties are fused + - selects the first value from source graph if no target value exists (does not add values from target graph) + - also if target graph has multiple values for a property, the first value is selected (for new entities) + """ + ontology = OntologyUtil.load_ontology_from_file(Path(ontology_path)) + allowed_predicates = set[str]([str(p.uri) for p in ontology.properties]+[str(RDFS.label), str(RDF.type), str(SKOS.altLabel)]) + fusable_properties = set[str]([str(p.uri) for p in ontology.properties if p.max_cardinality == 1]+[str(RDFS.label), str(RDF.type)]) + + def is_fusable(p): + return str(p) in fusable_properties + + entity_matches = load_matches_from_file(inputs["matches1"].path, entity_matching_threshold, "entity") + relation_matches = load_matches_from_file(inputs["matches1"].path, relation_matching_threshold, "relation") + + source_graph = Graph() + source_graph.parse(inputs["source"].path, format="nt") + seed_graph = Graph() # seed graph + seed_graph.parse(inputs["kg"].path, format="nt") + + current_subjects = set[str]([str(s) for s in seed_graph.subjects(unique=True)]) + + sameAsProv = {} + + def canonicalize_entity_term(term): + """Map a URI from the target graph to the matching source URI, if any.""" + if isinstance(term, URIRef): + t_str = str(term) + cluster = entity_matches.get_cluster(t_str) + if cluster: + right_candidates = [c for c in cluster if not c == t_str] + if len(right_candidates) > 2 and SINGLE_CANDIDATE_CHECK: + raise ValueError(f"Multiple matches found for {t_str}") + else: + for m in right_candidates: + # if not m == t_str: + sameAsProv[str(term)] = str(m) + return URIRef(m) + return term + else: + return term + return term + + def canonicalize_property_term(term): + """Map a URI from the target graph to the matching source URI, if any.""" + if isinstance(term, URIRef): + t_str = str(term) + mapped = relation_matches.has_match_to_namespace(t_str, TARGET_ONTOLOGY_NAMESPACE) + if mapped: + return URIRef(mapped) + else: # TODO this is a workaround for the base pipelines... + mapped = relation_matches.has_match_to_namespace(t_str, str(RDFS)) + if mapped: + return URIRef(mapped) + return term + + selected: List[TrackRecord] = [] + discarded: List[TrackRecord] = [] + + for s, p, o in source_graph: + # Canonicalize + logger.debug(f"Canonicalizing {s}, {p}, {o}") + s_can = canonicalize_entity_term(s) + p_can = canonicalize_property_term(p) + o_can = canonicalize_entity_term(o) if isinstance(o, URIRef) else o # keep literals/bnodes as-is + + # Only work with properties that are in our ontology (after canonicalization) + if not isinstance(p_can, URIRef) or str(p_can) not in allowed_predicates: + logger.debug(f"Skipping {s}, {p}, {o} because it is not in the allowed predicates") + continue + + if p_can == RDF.type and not str(o_can).startswith(TARGET_ONTOLOGY_NAMESPACE): + continue + + if is_fusable(p_can): + # Add exactly one value if none exists yet + if not any(seed_graph.objects(s_can, p_can)): + seed_graph.add((s_can, p_can, o_can)) + selected.append( + TrackRecord(subject=s_can,predicate=p_can,object=o,original_subject=s,original_predicate=p,original_object=o)) + # keep subjects set fresh for subsequent matches + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + else: + discarded.append( + TrackRecord(subject=s_can,predicate=p_can,object=o,original_subject=s,original_predicate=p,original_object=o)) + else: + # Non-fusable: copy if not already present (avoid dupes) + if (s_can, p_can, o_can) not in seed_graph: + seed_graph.add((s_can, p_can, o_can)) + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + + # sel(ected) + selected_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".selected.json") + with open(selected_file_path, "w") as f: + json.dump(selected, f, default=lambda x: x.model_dump()) + # dis(carded) + discarded_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".discarded.json") + with open(discarded_file_path, "w") as f: + json.dump(discarded, f, default=lambda x: x.model_dump()) + + prov_graph = Graph() + for sid,gid in sameAsProv.items(): + prov_graph.add((URIRef(gid), OWL.sameAs, URIRef(sid))) + prov_graph.serialize(outputs["output"].path.as_posix() + ".prov", format="nt") + seed_graph.serialize(outputs["output"].path, format="nt") \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/genie.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/genie.py new file mode 100644 index 0000000..a97b566 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/genie.py @@ -0,0 +1,33 @@ +from typing import Dict, Any +from kgpipe.common import Data, DataFormat, Registry, KgTask +from pathlib import Path + +def genie_text_extraction_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + from param_opti.tasks.genie_lib import genie_task_docker, genie_exchange + + # Ensure parent directory exists for the TE JSON output path + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + input_path: Path = inputs["input"].path + final_te_output: Data = outputs["output"] + + # 1) Produce intermediate OpenIE JSON (file or directory) + if input_path.is_dir(): + genie_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie_out" + else: + genie_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie.json" + + genie_outpit = {"output": Data(genie_out_path, DataFormat.OPENIE_JSON)} + genie_task_docker({"input": inputs["input"]}, genie_outpit) + + # 2) Convert OpenIE JSON → TE JSON (final output) + genie_exchange({"input": genie_outpit["output"]}, {"output": final_te_output}) + + +genie_text_extraction_task = KgTask( + name="genie_text_extraction", + input_spec={"input": DataFormat.TEXT}, + output_spec={"output": DataFormat.TE_JSON}, + function=genie_text_extraction_function, + description="Extract text using Genie" +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/genie_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/genie_lib.py new file mode 100644 index 0000000..0806f37 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/genie_lib.py @@ -0,0 +1,89 @@ + +import re +import json +import os +from typing import Dict +from kgpipe.common import Data, TaskInput, TaskOutput +from kgpipe.common import KgTask, DataFormat, Data, Registry, TaskInput, TaskOutput +from kgpipe.common.io import get_docker_volume_bindings, remap_data_path_for_container +from kgpipe.execution import docker_client +from kgpipe_tasks.transform_interop.exchange.entity_matching import ER_Match, ER_Document + + +def genie_task_docker(inputs: TaskInput, outputs: TaskOutput): + """ + GenIE information extraction task that runs in a Docker container. + + Args: + inputs: Dictionary mapping input names to Data objects + outputs: Dictionary mapping output names to Data objects + """ + + all_data = list(inputs.values()) + list(outputs.values()) + volumes, host_to_container = get_docker_volume_bindings(all_data) + + source_path = remap_data_path_for_container(inputs["input"], host_to_container) + output_path = remap_data_path_for_container(outputs["output"], host_to_container) + + client = docker_client( + image="genie:latest", + command=["genie.sh", + str(source_path.path), + str(output_path.path)], + volumes=volumes, + ) + + result = client() + print(f"GenIE completed: {result}") + +def process_io(input_path, output_path, process_file_fn, extension): + if os.path.isdir(input_path): + os.makedirs(output_path, exist_ok=True) + + for filename in os.listdir(input_path): + input_file = os.path.join(input_path, filename) + + if not os.path.isfile(input_file): + continue + + output_file = os.path.join( + output_path, + os.path.splitext(filename)[0] + extension + ) + + process_file_fn(input_file, output_file) + + else: + process_file_fn(input_path, output_path) + +def genie_exchange(inputs: Dict[str, Data], outputs: Dict[str, Data]): + input_path = inputs["input"].path + output_path = outputs["output"].path + + triple_pattern = re.compile( + r"\s*(.*?)\s*\s*(.*?)\s*\s*(.*?)\s*" + ) + + def exchange_file(input_file, output_file): + triples = [] + chains = [] + + with open(input_file, "r", encoding="utf-8") as f: + genie_output = json.load(f) + + for sentence in genie_output: + for beam in sentence: + text = beam.get("text", "") + matches = triple_pattern.findall(text) + + for subj, pred, obj in matches: + triples.append({ + "subject": {"surface_form": subj.strip()}, + "predicate": {"surface_form": pred.strip()}, + "object": {"surface_form": obj.strip()} + }) + + with open(output_file, "w", encoding="utf-8") as f: + json.dump({"triples": triples, "chains": chains}, f, indent=2) + + process_io(input_path, output_path, exchange_file, ".te.json") \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/jedai.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/jedai.py new file mode 100644 index 0000000..062fe48 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/jedai.py @@ -0,0 +1 @@ +# Skipped for now \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/llm_extract_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/llm_extract_lib.py new file mode 100644 index 0000000..eac369e --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/llm_extract_lib.py @@ -0,0 +1,18 @@ +from transformers import AutoTokenizer, AutoModelForSeq2SeqLM + +def test_llm_extract(): + # Load the Flan-T5 Large checkpoint (780M parameters) + tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-large") + model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-large") + + prompt = """Extract the organizations and locations from the following text: + "Sarah flew from Berlin to Leipzig to attend a workshop at the university." """ + + # Encode the prompt and generate extraction + inputs = tokenizer(prompt, return_tensors="pt") + outputs = model.generate(**inputs, max_length=50) + + # Decode the output + extracted_info = tokenizer.decode(outputs[0], skip_special_tokens=True) + print(extracted_info) + diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/llm_mapping_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/llm_mapping_lib.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/matching_helpers.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/matching_helpers.py new file mode 100644 index 0000000..fd301a0 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/matching_helpers.py @@ -0,0 +1,30 @@ +from pathlib import Path + +from kgpipe.common import Data, DataFormat, KgTask +from typing import Dict +from kgpipe_tasks.transform_interop.exchange.entity_matching import ER_Document +import json + + +def _load_er_document(path: Path) -> ER_Document: + """Parse ER JSON; empty or whitespace-only files yield an empty document (stub tasks may touch-only outputs).""" + raw = path.read_text(encoding="utf-8") + if not raw.strip(): + return ER_Document() + return ER_Document(**json.loads(raw)) + + +def aggregate_matching_results_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + er1 = _load_er_document(Path(inputs["json1"].path)) + er2 = _load_er_document(Path(inputs["json2"].path)) + er_comb = ER_Document(matches=er1.matches + er2.matches) + with open(outputs["output"].path, "w") as f: + json.dump(er_comb.model_dump(), f, indent=4) + + +aggregate_matching_results_task = KgTask( + name="aggregate_matching_results", + input_spec=dict({"json1": DataFormat.ER_JSON, "json2": DataFormat.ER_JSON}), + output_spec=dict({"output": DataFormat.ER_JSON}), + function=aggregate_matching_results_function +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/paris.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/paris.py new file mode 100644 index 0000000..04e9eba --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/paris.py @@ -0,0 +1,127 @@ +from kgpipe.common import TaskInput, TaskOutput, KgTask, DataFormat, Data, Registry +from kgpipe.common.model.configuration import ConfigurationProfile, ConfigurationDefinition, Parameter, ParameterType +from pathlib import Path + + + +def paris_entity_alignment_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + matches entities between two RDF graphs + """ + # touch output file + from param_opti.tasks.paris_lib import paris_exchange, paris_entity_matching + entity_matching_threshold = float(config.get_parameter_value("entity_matching_threshold")) + relation_matching_threshold = float(2) # todo skip all matches + + # Ensure parent directory exists for the ER JSON output file + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + # 1 produce matches in paris csv format + matching_dir = outputs["output"].path.parent / f"{outputs['output'].path.stem}_paris_out" + matching_output = {"output": Data(matching_dir, DataFormat.PARIS_CSV)} + + # paris_entity_matching expects {"source": ..., "kg": ...} + paris_entity_matching({"source": inputs["source"], "kg": inputs["target"]}, matching_output) + + # 2 convert paris output dir to er.json format (file) + paris_exchange( + matching_output["output"].path, + outputs["output"].path, + entity_matching_threshold, + relation_matching_threshold, + ) + +paris_entity_alignment_task = KgTask( + name="paris_entity_alignment", + function=paris_entity_alignment_function, + input_spec={"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.ER_JSON}, + config_spec=ConfigurationDefinition( + name="paris_entity_alignment", + parameters=[ + Parameter(name="entity_matching_threshold", native_keys=["--entity-matching-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) + +def paris_graph_alignment_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + matches both entities and relations between two RDF graphs + """ + # touch output file + from param_opti.tasks.paris_lib import paris_exchange, paris_entity_matching + entity_matching_threshold = float(config.get_parameter_value("entity_matching_threshold")) + relation_matching_threshold = float(config.get_parameter_value("relation_matching_threshold")) + + # Ensure parent directory exists for the ER JSON output file + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + # 1 produce matches in paris csv format + matching_dir = outputs["output"].path.parent / f"{outputs['output'].path.stem}_paris_out" + matching_output = {"output": Data(matching_dir, DataFormat.PARIS_CSV)} + + # paris_entity_matching expects {"source": ..., "kg": ...} + paris_entity_matching({"source": inputs["source"], "kg": inputs["target"]}, matching_output) + + # 2 convert paris output dir to er.json format (file) + paris_exchange( + matching_output["output"].path, + outputs["output"].path, + entity_matching_threshold, + relation_matching_threshold, + ) + +paris_graph_alignment_task = KgTask( + name="paris_graph_alignment_task", + function=paris_graph_alignment_function, + input_spec={"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.ER_JSON}, + config_spec=ConfigurationDefinition( + name="paris_graph_alignment_task", + parameters=[ + Parameter(name="entity_matching_threshold", native_keys=["--entity-matching-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + Parameter(name="relation_matching_threshold", native_keys=["--relation-matching-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) +Registry.add_task(paris_graph_alignment_task.name, paris_graph_alignment_task) + +def paris_ontology_matching_function(inputs: TaskInput, outputs: TaskOutput, config: ConfigurationProfile): + """ + matches ontologies between two RDF graphs + """ + # touch output file + from param_opti.tasks.paris_lib import paris_exchange, paris_entity_matching + entity_matching_threshold = float(2) # todo skip all matches + ontology_matching_threshold = float(config.get_parameter_value("ontology_matching_threshold")) + + # Ensure parent directory exists for the ER JSON output file + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + # 1 produce matches in paris csv format + matching_dir = outputs["output"].path.parent / f"{outputs['output'].path.stem}_paris_out" + matching_output = {"output": Data(matching_dir, DataFormat.PARIS_CSV)} + + # paris_entity_matching expects {"source": ..., "kg": ...} + paris_entity_matching({"source": inputs["source"], "kg": inputs["target"]}, matching_output) + + # 2 convert paris output dir to er.json format (file) + paris_exchange( + matching_output["output"].path, + outputs["output"].path, + entity_matching_threshold, + ontology_matching_threshold, + ) + +paris_ontology_matching_task = KgTask( + name="paris_ontology_matching", + function=paris_ontology_matching_function, + input_spec={"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.ER_JSON}, + config_spec=ConfigurationDefinition( + name="paris_ontology_matching", + parameters=[ + Parameter(name="ontology_matching_threshold", native_keys=["--ontology-matching-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/paris_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/paris_lib.py new file mode 100644 index 0000000..2aa5fd5 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/paris_lib.py @@ -0,0 +1,152 @@ +""" +Paris RDF Matcher task implementation. +""" + +from pathlib import Path +from typing import Dict, Any +import pandas as pd +import os +import csv +from typing import List + +from kgpipe.common import KgTask, DataFormat, Data, Registry +from kgpipe.common.io import get_docker_volume_bindings, remap_data_path_for_container +from kgpipe.execution import docker_client +from kgpipe_tasks.transform_interop.exchange.entity_matching import ER_Match, ER_Document + + +def paris_entity_matching(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """ + Paris entity matching task that runs in a Docker container. + + Args: + inputs: Dictionary mapping input names to Data objects + outputs: Dictionary mapping output names to Data objects + """ + # print(f"Running Paris entity matching with inputs: {inputs}") + + all_data = list(inputs.values()) + list(outputs.values()) + volumes, host_to_container = get_docker_volume_bindings(all_data) + + # Extract input paths + source_path = remap_data_path_for_container(inputs["source"], host_to_container) + target_path = remap_data_path_for_container(inputs["kg"], host_to_container) + output_path = remap_data_path_for_container(outputs["output"], host_to_container) + + # Ensure output directory exists + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + # Get all data for Docker volume bindings + + # Create Docker client with proper volume bindings + client = docker_client( + image="kgt/paris:latest", + # command=["ls", "-la"], + command=["bash", "paris.sh", + str(source_path.path), + str(target_path.path), + str(output_path.path)], + volumes=volumes, + ) + + # Execute the container + result = client() + print(f"Paris entity matching completed: {result}") + + +PREFIX_MAP = { + "dbp": "http://dbpedia.org/", + "rdfs": "http://www.w3.org/2000/01/rdf-schema#", + "rdf" : "http://www.w3.org/1999/02/22-rdf-syntax-ns#", + "xsd" : "http://www.w3.org/2001/XMLSchema#", + "schema" : "http://schema.org/", + "dbo": "http://dbpedia.org/ontology/", + "foaf": "http://xmlns.com/foaf/0.1/", + "skos": "http://www.w3.org/2004/02/skos/core#", +} + +def resolvePrefixedUri(uri): + if not uri.startswith("http://") and not uri.startswith("https://"): + prefix, suffix = uri.split(":", 1) + # try: + prefix = PREFIX_MAP[prefix] + # except Exception as e: + # print(f"Unknown prefix: {prefix} for {uri}") + # raise Exception(f"Unknown prefix: {prefix} for {uri}") + return prefix + suffix + else: + return uri + + + +def paris_exchange(input_path: Path, output_path: Path, entity_matching_threshold: float, relation_matching_threshold: float): + """ + Convert Paris CSV output to standard RDF matching format. + + Args: + inputs: Dictionary mapping input names to Data objects (Paris CSV) + outputs: Dictionary mapping output names to Data objects (RDF) + """ + print(f"Converting Paris CSV to matching format with input_path: {input_path} and output_path: {output_path}") + + files = [str(f) for f in os.listdir(input_path)] + + iteration_ids = [ int(f.split("_")[0]) for f in files if f.endswith(".tsv") ] + + iteration_ids.sort() + + last_eqv_it = iteration_ids[-1] + + def getEqvFileName(id): return f"{id}_eqv.tsv" + def getRelFileNames(id): return [f"{id}_superrelations1.tsv",f"{id}_superrelations2.tsv"] + + def check_file_exists(last_eqv_it): + try: + return os.stat(os.path.join(input_path, getEqvFileName(last_eqv_it))).st_size > 0 + except FileNotFoundError: + return -1 + + while 0 == check_file_exists(last_eqv_it) : + last_eqv_it -= 1 + + last_relation_it = last_eqv_it - 1 + + matches : List[ER_Match] = [] + + def extract_matches(file,id_type): + with open(file, newline='', encoding='utf-8') as csvfile: + reader = csv.reader(csvfile, delimiter='\t') + for row in reader: + if len(row) == 3: + er_match = ER_Match( + id_1=resolvePrefixedUri(row[0]), + id_2=resolvePrefixedUri(row[1]), + score=float(row[2]), + id_type=id_type + ) + matches.append(er_match) + + def filter_matches(matches: List[ER_Match]): + + for match in matches: + if match.id_type == "entity" and match.score > entity_matching_threshold: + yield match + if match.id_type == "relation" and match.score > relation_matching_threshold: + yield match + + if last_eqv_it == -1: + doc = ER_Document(matches=list(filter_matches([]))) + with open(output_path, 'w', encoding='utf-8') as jsonfile: + jsonfile.write(doc.model_dump_json()) + else: + eqv_file = getEqvFileName(last_eqv_it) + rel_files = getRelFileNames(last_relation_it) + + extract_matches(os.path.join(input_path,eqv_file),"entity") + [ extract_matches(os.path.join(input_path,f), "relation") for f in rel_files ] + + + doc = ER_Document(matches=list(filter_matches(matches))) + + with open(output_path, 'w', encoding='utf-8') as jsonfile: + jsonfile.write(doc.model_dump_json()) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/select_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/select_lib.py new file mode 100644 index 0000000..7dd390f --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/select_lib.py @@ -0,0 +1,119 @@ +import json +import os +from logging import getLogger +from pathlib import Path +from typing import Dict, List + +from kgcore.api.ontology import OntologyUtil +from kgpipe.common.config import TARGET_ONTOLOGY_NAMESPACE +from kgpipe.common.model.configuration import ConfigurationDefinition +from kgpipe.common.models import Data, DataFormat, KgTask +from pydantic import BaseModel +from rdflib import Graph, RDF, RDFS, SKOS, URIRef + +logger = getLogger(__name__) + +class TrackRecord(BaseModel): + original_subject: str + subject: str + original_predicate: str + predicate: str + original_object: str + object: str + + +def select_first_value_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """ + For two KGs A and B, merge A into B where for each s_p and + 1) p is fusable and B does not have any s_p_o or + 2) p is not fusable erge all s_p_o + """ + ontology_path = os.environ.get("ONTOLOGY_PATH", "false") + if ontology_path == "false": + raise ValueError("ONTOLOGY_PATH is not set") + + ontology = OntologyUtil.load_ontology_from_file(Path(ontology_path)) + allowed_predicates = set[str]([str(p.uri) for p in ontology.properties]+[str(RDFS.label), str(RDF.type), str(SKOS.altLabel)]) + fusable_properties = set[str]([str(p.uri) for p in ontology.properties if p.max_cardinality == 1]+[str(RDFS.label), str(RDF.type)]) + + def is_fusable(p): + return str(p) in fusable_properties + + source_graph = Graph() + source_graph.parse(inputs["source"].path, format="nt") + seed_graph = Graph() # seed graph + seed_graph.parse(inputs["target"].path, format="nt") + + current_subjects = set[str]([str(s) for s in seed_graph.subjects(unique=True)]) + + selected: List[TrackRecord] = [] + discarded: List[TrackRecord] = [] + + for s, p, o in source_graph: + s_can = s + p_can = p + o_can = o + + if not isinstance(p_can, URIRef) or str(p_can) not in allowed_predicates: + continue + + if p_can == RDF.type and not str(o_can).startswith(TARGET_ONTOLOGY_NAMESPACE): + continue + + if is_fusable(p_can): + # Add exactly one value if none exists yet + if not any(seed_graph.objects(s_can, p_can)): + seed_graph.add((s_can, p_can, o_can)) + selected.append( + TrackRecord( + subject=str(s_can), + predicate=str(p_can), + object=str(o_can), + original_subject=str(s), + original_predicate=str(p), + original_object=str(o), + ) + ) + # keep subjects set fresh for subsequent matches + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + else: + discarded.append( + TrackRecord( + subject=str(s_can), + predicate=str(p_can), + object=str(o_can), + original_subject=str(s), + original_predicate=str(p), + original_object=str(o), + ) + ) + else: + # Non-fusable: copy if not already present (avoid dupes) + if (s_can, p_can, o_can) not in seed_graph: + seed_graph.add((s_can, p_can, o_can)) + if isinstance(s_can, URIRef): + current_subjects.add(str(s_can)) + + # sel(ected) + selected_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".selected.json") + with open(selected_file_path, "w") as f: + json.dump(selected, f, default=lambda x: x.model_dump()) + # dis(carded) + discarded_file_path = outputs["output"].path.parent / (outputs["output"].path.stem + ".discarded.json") + with open(discarded_file_path, "w") as f: + json.dump(discarded, f, default=lambda x: x.model_dump()) + + # prov graph is skipped here as no uris are replaced (is done in previouse steps) + seed_graph.serialize(outputs["output"].path, format="nt") + +select_first_value_task = KgTask( + name="select_first_value", + input_spec={"source": DataFormat.RDF_NTRIPLES, "target": DataFormat.RDF_NTRIPLES}, + output_spec={"output": DataFormat.RDF_NTRIPLES}, + function=select_first_value_function, + config_spec=ConfigurationDefinition( + name="select_first_value", + parameters=[] + ) +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight.py new file mode 100644 index 0000000..3129ad0 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight.py @@ -0,0 +1,42 @@ +from typing import Dict, Any +from kgpipe.common import Data, DataFormat, Registry, KgTask +from kgpipe.common.model.configuration import ConfigurationDefinition, ConfigurationProfile, Parameter, ParameterType +from pathlib import Path + +def spotlight_entity_linking_function(inputs: Dict[str, Data], outputs: Dict[str, Data], config: ConfigurationProfile ): + from param_opti.tasks.spotlight_lib import dbpedia_spotlight_ner_nel, dbpedia_spotlight_exchange + + # Ensure parent directory exists for the TE JSON output path + outputs["output"].path.parent.mkdir(parents=True, exist_ok=True) + + input_path: Path = inputs["input"].path + final_te_output: Data = outputs["output"] + + # 1) Produce intermediate OpenIE JSON (file or directory) + if input_path.is_dir(): + spotlight_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie_out" + else: + spotlight_out_path = final_te_output.path.parent / f"{final_te_output.path.stem}_corenlp_openie.json" + + spotlight_out = {"output": Data(spotlight_out_path, DataFormat.OPENIE_JSON)} + if not spotlight_out_path.exists(): + dbpedia_spotlight_ner_nel({"input": inputs["input"]}, spotlight_out) + + # 2) Convert OpenIE JSON → TE JSON (final output) + dbpedia_spotlight_exchange({"input": spotlight_out["output"]}, {"output": final_te_output}, config.get_parameter_value("similarity_threshold")) + + + +spotlight_entity_linking_task = KgTask( + name="spotlight_entity_linking", + input_spec={"input": DataFormat.TEXT}, + output_spec={"output": DataFormat.TE_JSON}, + function=spotlight_entity_linking_function, + description="Link entities using Spotlight", + config_spec=ConfigurationDefinition( + name="spotlight_entity_linking", + parameters=[ + Parameter(name="similarity_threshold", native_keys=["--similarity-threshold"], datatype=ParameterType.number, default_value=0.5, required=True, allowed_values=[0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]), + ] + ) +) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight_lib.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight_lib.py new file mode 100644 index 0000000..0bba12b --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/spotlight_lib.py @@ -0,0 +1,128 @@ +""" +DBpedia Spotlight Entity Linking + +This module provides entity linking using DBpedia Spotlight. +""" + +import json +import os +import requests +from pathlib import Path +from typing import Dict, Any + +from kgpipe.common import KgTask, Data, DataFormat, Registry +from kgpipe.common.io import get_docker_volume_bindings +from kgpipe.execution import docker_client +from tqdm import tqdm + +import os + + +CONFIDENCE = 0.35 +HEADERS = { + "Accept": "application/json" +} +DEFAULT_API_URL = "http://localhost:2222/rest/annotate" + +def api_request(url: str, text: str) -> Dict[str, Any]: + """Make API request to DBpedia Spotlight.""" + data = { + "text": text, + "confidence": str(CONFIDENCE) + } + response = requests.post(url, data=data, headers=HEADERS, verify=False) + + if response.status_code == 200: + result = response.json() + else: + result = { + "error": f"Request failed with status code {response.status_code}", + "text": text + } + return result + + +def dbpedia_spotlight_ner_nel(inputs: Dict[str, Data], outputs: Dict[str, Data]): + """Link entities using DBpedia Spotlight API.""" + input_data = inputs["input"] + output_data = outputs["output"] + + DBPEDIA_ANNOTATE_URL = os.getenv("DBPEDIA_ANNOTATE_URL", DEFAULT_API_URL) + if not DBPEDIA_ANNOTATE_URL: + raise ValueError("Missing DBpedia ANnotate URL") + + dir_or_file = input_data.path + if os.path.isdir(dir_or_file): + os.makedirs(output_data.path, exist_ok=True) + for file in tqdm(os.listdir(dir_or_file)): + with open(os.path.join(dir_or_file, file), encoding='utf-8') as f: + input_text = f.read() + + results = api_request(DBPEDIA_ANNOTATE_URL, input_text) + + with open(os.path.join(output_data.path, file+".json"), 'w', encoding='utf-8') as f: + f.write(json.dumps(results)) + # print(f"Converted {file} to {os.path.join(output_data.path, file)}") + else: + with open(input_data.path, encoding='utf-8') as f: + input_text = f.read() + + results = api_request(DBPEDIA_ANNOTATE_URL, input_text) + + with open(output_data.path, 'w', encoding='utf-8') as f: + f.write(json.dumps(results)) + + +# @Registry.task( +# input_spec={"source": DataFormat.SPOTLIGHT_JSON}, +# output_spec={"output": DataFormat.TE_JSON}, +# description="Convert Spotlight JSON to TE JSON format, with seed filter", +# category=["TextProcessing", "EntityLinking"] +# ) +def dbpedia_spotlight_exchange(inputs: Dict[str, Data], outputs: Dict[str, Data], threshold: float = 0.5): + """Convert Spotlight JSON to TE JSON format.""" + input_path = inputs["input"].path + output_path = outputs["output"].path + + # create output folder + os.makedirs(os.path.normpath(output_path), exist_ok=True) + + def __spotlightjson2tejson(data) -> Dict[str, Any]: + """Convert Spotlight JSON to TE Document format.""" + links = [] + + for result in data.get('Resources', []): + if float(result.get('@similarityScore', 0.0)) < threshold: + continue + link = { + "span": result.get('@surfaceForm', ''), + "mapping": result.get('@URI', ''), + "score": float(result.get('@similarityScore', 0.0)), + "link_type": "entity" + } + links.append(link) + + text = data.get('@text', '') + return {"text": text, "links": links} + + if os.path.isdir(input_path): + for file in os.listdir(input_path): + # Read input json + with open(os.path.join(input_path, file), 'r') as f: + data = json.load(f) + te_doc = __spotlightjson2tejson(data) + outfile = os.path.join(output_path, file) + + with open(outfile, 'w') as of: + json.dump(te_doc, of) + # print(f"Converted {file} to {outfile}") + + else: + # Read input json + with open(input_path, 'r') as f: + data = json.load(f) + te_doc = __spotlightjson2tejson(data) + outfile = os.path.join(output_path, 'output.te.json') + with open(outfile, 'w') as of: + json.dump(te_doc, of) + \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/dev/tasks/text_helpers.py b/experiments/param-opti/src/kgpipe_search/dev/tasks/text_helpers.py new file mode 100644 index 0000000..52fb1ac --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/dev/tasks/text_helpers.py @@ -0,0 +1,434 @@ +import json +import logging +import os +from pathlib import Path +from typing import Dict, List + +from kgcore.api.ontology import Ontology, OntologyUtil +from kgpipe.common import Data, DataFormat, KgTask +from kgpipe.common.model.configuration import ConfigurationDefinition +from kgpipe_tasks.common.benchutils import hash_uri +from kgpipe_tasks.transform_interop.exchange.text_extraction import ( + TE_Chains, + TE_Document, + TE_Pair, + TE_Triple, +) +from rdflib import Graph, Literal, RDF, RDFS, URIRef, XSD + +logger = logging.getLogger(__name__) + + +def _file_stem_key(filename: str) -> str: + """Basename without extensions, e.g. 'hash.te.json' -> 'hash'.""" + return filename.split(".", 1)[0] + + +def _index_dir_by_stem(dir_path: Path) -> Dict[str, Path]: + """Map stem -> file path for files in a directory (first match wins).""" + by_stem: Dict[str, Path] = {} + for entry in dir_path.iterdir(): + if entry.is_file(): + stem = _file_stem_key(entry.name) + if stem not in by_stem: + by_stem[stem] = entry + return by_stem + + +def __aggregate_x_te_json( + input_paths: List[Path], + output_path: Path, + match_by_stem: bool = False, +): + """ + Merge TE_Document JSON from files or directories. + + When all inputs are directories and ``match_by_stem`` is True, files are + paired by stem (name before the first ``.``), so e.g. + ``hash.te.json`` and ``hash.txt.json`` are merged even though the + full filenames differ. + """ + if len(input_paths) == 0: + raise Exception("No input paths provided") + if not all(path.exists() for path in input_paths): + raise Exception("All input paths must exist") + + path_is_dir_list = [path.is_dir() for path in input_paths] + if all(path_is_dir_list): + output_path.mkdir(parents=True, exist_ok=True) + + if match_by_stem: + stem_indexes = [_index_dir_by_stem(path) for path in input_paths] + for stem, primary_file in stem_indexes[0].items(): + matched = [idx[stem] for idx in stem_indexes if stem in idx] + if len(matched) < len(input_paths): + logger.warning( + f"Stem '{stem}' does not exist in all input paths " + f"(found in {len(matched)}/{len(input_paths)})" + ) + # Keep the first directory's filename for the output + __aggregate_x_te_json( + matched, + output_path / primary_file.name, + match_by_stem=match_by_stem, + ) + else: + for file in input_paths[0].iterdir(): + if not file.is_file(): + continue + sub_file_paths = [path / file.name for path in input_paths] + existing = [p for p in sub_file_paths if p.exists()] + if len(existing) < len(input_paths): + logger.warning( + f"File {file.name} does not exist in all input paths" + ) + __aggregate_x_te_json( + existing, + output_path / file.name, + match_by_stem=match_by_stem, + ) + elif not any(path_is_dir_list): + merged_doc = TE_Document() + for file in input_paths: + with open(file) as f: + doc = TE_Document(**json.load(f)) + merged_doc.chains += doc.chains + merged_doc.links += doc.links + merged_doc.triples += doc.triples + with open(output_path, "w") as f: + f.write(merged_doc.model_dump_json()) + logger.info( + f"Aggregated {', '.join(str(p) for p in input_paths)} to {output_path}" + ) + else: + raise Exception("All inputs must be either directories or files") + + +# @Registry.task( +# input_spec={"json1": DataFormat.TE_JSON, "json2": DataFormat.TE_JSON}, +# output_spec={"output": DataFormat.TE_JSON}, +# description="Aggregate 2 TE_Document JSON files", +# category=["Aggregation"] +# ) +# def aggregate2_te_json(inputs: Dict[str, Data], outputs: Dict[str, Data]): +# __aggregate_x_te_json([inputs["json1"].path, inputs["json2"].path], outputs["output"].path) + + +def aggregate3_text_tasks_task_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + __aggregate_x_te_json( + [inputs["json1"].path, inputs["json2"].path, inputs["json3"].path], + outputs["output"].path, + match_by_stem=True, + ) + +aggregate_text_tasks_task = KgTask( + name="aggregate_text_tasks_task", + input_spec={"json1": DataFormat.TE_JSON, "json2": DataFormat.TE_JSON, "json3": DataFormat.TE_JSON}, + output_spec={"output": DataFormat.TE_JSON}, + function=aggregate3_text_tasks_task_function +) + +def aggregate2_text_tasks_task_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + __aggregate_x_te_json( + [inputs["json1"].path, inputs["json2"].path], + outputs["output"].path, + match_by_stem=True, + ) + +aggregate_entity_linking_task = KgTask( + name="aggregate_entity_linking_task", + input_spec={"json1": DataFormat.TE_JSON, "json2": DataFormat.TE_JSON}, + output_spec={"output": DataFormat.TE_JSON}, + function=aggregate2_text_tasks_task_function +) + +aggregate_relation_linking_task = KgTask( + name="aggregate_relation_linking_task", + input_spec={"json1": DataFormat.TE_JSON, "json2": DataFormat.TE_JSON}, + output_spec={"output": DataFormat.TE_JSON}, + function=aggregate2_text_tasks_task_function +) + +def generatePredicate(surface_form, namespace): + return URIRef(namespace + surface_form.replace(" ", "_")) + +def __hash_dbpedia_uri(uri: URIRef, namespace: str = "http://kg.org/resource/"): + if uri.startswith("http://dbpedia.org/"): + return URIRef(namespace + hash_uri(str(uri))) + else: + return uri + +def __generateRDF(doc: TE_Document, ontology: Ontology, newP: bool = False, newE: bool = False, namespace: str = "http://kg.org/text/"): + """ + A processing node, part of a pipeline + collects information from extractors, linkers, and resolvers and then it produces the final triples + """ + + def process_chains(triples, chains: List[TE_Chains]): + new_triples = triples + chain_dict = {} + for chain in chains: + for alias in chain.aliases: + chain_dict[alias.surface_form] = chain.main + if len(chain_dict) > 0: + # TODO check if chain_dict should be a dict of TE_SPANS to avoid merging + for triple in new_triples: + if triple.subject.surface_form in chain_dict: + triple.subject.surface_form = chain_dict[triple.subject.surface_form] + if triple.object.surface_form in chain_dict: + triple.object.surface_form = chain_dict[triple.object.surface_form] + return new_triples + + def process_links(triples, links: List[TE_Pair]): + new_triples = triples + try: + if len(links) > 0: + so_spans = {} + p_spans = {} + for triple in triples: + # Add Subject spans + if triple.subject.surface_form.lower().startswith("http://"): + pass + elif triple.subject.surface_form.lower() not in so_spans: + so_spans[triple.subject.surface_form.lower()] = [triple.subject] + else: + so_spans[triple.subject.surface_form.lower()].append(triple.subject) + # Add object spans + if triple.object.surface_form.lower().startswith("http://"): + pass + elif triple.object.surface_form.lower() not in so_spans: + so_spans[triple.object.surface_form.lower()] = [triple.object] + else: + so_spans[triple.object.surface_form.lower()].append(triple.object) + # add predicate spans + if triple.predicate.surface_form.lower().startswith("http://"): + pass + elif triple.predicate.surface_form.lower() not in p_spans: + p_spans[triple.predicate.surface_form.lower()] = [triple.predicate] + else: + p_spans[triple.predicate.surface_form.lower()].append(triple.predicate) + for link in links: + if link.link_type == 'entity': + spans = so_spans + else: + spans = p_spans + if link.span and link.span.lower() in spans: + for span in spans[link.span.lower()]: + span.mapping = link.mapping + # span.surface_form = link.mapping + except Exception as exp: + raise exp + finally: + return new_triples + + + triples: List[TE_Triple] = doc.triples + links: List[TE_Pair] = doc.links + chains: List[TE_Chains] = doc.chains + + dereferenced_tiples = process_chains(triples, chains) + linked_triples: List[TE_Triple] = process_links(dereferenced_tiples, links) + finalGraph = Graph() + + for triple in linked_triples: + subject = None + if triple.subject.mapping: + subject = URIRef(triple.subject.mapping) + # else: + # subject = triple.subject.surface_form + + predicate = None + if triple.predicate.mapping: + predicate = URIRef(triple.predicate.mapping) + elif newP: + predicate = generatePredicate(triple.predicate.surface_form, namespace) + + object = None + # TODO if predicate is a datatype or object property + if triple.object.mapping: + object = URIRef(triple.object.mapping) + # else: + # object = Literal(triple.object.surface_form) + # if(subject and predicate and object): + # finalGraph.add((subject, predicate, object)) + + # new entities + if(predicate): + # print(f"new subject: {subject} {triple.subject.surface_form}") + + domain, range = ontology.get_domain_range(str(predicate)) + isObjectProperty = True if range and range.startswith("http://kg.org") else False + # print(f"predicate: {predicate}, domain: {domain}, range: {range}, isObjectProperty: {isObjectProperty}") + # print(f"predicate: {predicate}, domain: {domain}, range: {range}") + + if subject and subject.startswith("http://dbpedia.org"): # TODO workaround for dbpedia... + finalGraph.add((__hash_dbpedia_uri(subject), RDFS.label, Literal(triple.subject.surface_form))) + + + if not subject and triple.subject.surface_form and newE: + subject = URIRef(namespace+hash_uri(triple.subject.surface_form)) + finalGraph.add((subject, RDFS.label, Literal(triple.subject.surface_form))) + print(f"new subject: {subject} {triple.subject.surface_form}") + else: + print(f"subject: {subject} {triple.subject.surface_form}") + + if domain and subject: + finalGraph.add((__hash_dbpedia_uri(subject), RDF.type, URIRef(domain))) + + if object and isObjectProperty and object.startswith("http://dbpedia.org"): # TODO workaround for dbpedia... + finalGraph.add((__hash_dbpedia_uri(object), RDFS.label, Literal(triple.object.surface_form))) + + if not object and triple.object.surface_form and newE: + if isObjectProperty: + object = URIRef(namespace+hash_uri(triple.object.surface_form)) + finalGraph.add((object, RDFS.label, Literal(triple.object.surface_form))) + if range: + finalGraph.add((object, RDF.type, URIRef(range))) + else: + datatype = range if range else str(XSD.string) + object = Literal(triple.object.surface_form, datatype=datatype) + else: + if not isObjectProperty: + datatype = range if range else str(XSD.string) + object = Literal(triple.object.surface_form, datatype=datatype) + + if(subject and predicate and object): + finalGraph.add((__hash_dbpedia_uri(subject), predicate, __hash_dbpedia_uri(object))) + + return finalGraph + + +def generate_rdf(inputs: Dict[str, Data], outputs: Dict[str, Data], ontology: Ontology, newP: bool, newE: bool): + dir_or_file = inputs["source"].path + graph = Graph() + if os.path.isdir(dir_or_file): + for file in os.listdir(dir_or_file): + json_data = json.load(open(os.path.join(dir_or_file, file))) + doc = TE_Document(**json_data) + print(f"doc: {doc}") + for s, p, o in __generateRDF(doc, ontology, newP=newP, newE=newE): + graph.add(triple=(s, p, o)) + else: + doc = TE_Document(**json.load(open(dir_or_file))) + graph = __generateRDF(doc, ontology, newP=newP, newE=newE) + + graph.serialize(outputs["output"].path, format="nt") + print(f"RDF written to {outputs['output'].path}") + + +def generate_rdf_from_text_results_function(inputs: Dict[str, Data], outputs: Dict[str, Data]): + + ontology_path = os.environ.get("ONTOLOGY_PATH", "false") + if ontology_path == "false": + raise ValueError("ONTOLOGY_PATH is not set") + + ontology = OntologyUtil.load_ontology_from_file(Path(ontology_path)) + + generate_rdf(inputs, outputs, ontology, newP=False, newE=True) + + +generate_rdf_from_text_results_task = KgTask( + name="construct_rdf_from_text_tasks_task", + input_spec={"source": DataFormat.TE_JSON}, + output_spec={"output": DataFormat.RDF_NTRIPLES}, + function=generate_rdf_from_text_results_function +) + + +# ------------------------------------------------------------ + + +# def aggregate_3iejson_with_filter(inputs: Dict[str, Data], outputs: Dict[str, Data]): +# json1_path = inputs["json1"].path +# json2_path = inputs["json2"].path +# json3_path = inputs["json3"].path + +# def load_kg_uris_from_shades(): +# """ +# Loads the URIs of the entities in the current KG. +# """ +# shade_file = "/home/marvin/project/data/current/shade_seed.json" +# with open(shade_file, "r") as f: +# return json.load(f) + +# shade_dict = load_kg_uris_from_shades() +# reverse_shade_dict = {v: k for k, v in shade_dict.items()} +# kg_uris = set(shade_dict.values()) + + +# def filter_ie_doc(doc: TE_Document): +# """ +# Removes links to entities that are not in the current KG. +# """ + +# # for uri in kg_uris: +# # print(uri) + +# # Create a new list instead of modifying while iterating +# filtered_links = [] +# for link in doc.links: +# if link.link_type == "entity": +# if link.mapping not in kg_uris: +# # print(f"Removing entity link to {link.mapping} because it is not in the current KG") +# continue # Skip this link +# else: +# tmp = link.mapping +# try: +# link.mapping = reverse_shade_dict[tmp] +# # print(f"Replacing entity link {tmp} with {link.mapping}") +# except KeyError: +# print(f"KeyError: {tmp} not found in reverse_shade_dict, skipping") +# continue # Skip this link +# # elif link.link_type == "relation": +# # if link.mapping not in kg_uris: +# # print(f"Removing relation link to {link.mapping} because it is not in the current KG") +# # continue # Skip this link + +# # Add the link to the filtered list (either it passed all checks or it's not an entity link) +# filtered_links.append(link) + +# doc.links = filtered_links +# return doc + + +# if os.path.isdir(json1_path) and os.path.isdir(json2_path) and os.path.isdir(json3_path): +# # list files in each directory +# json1_files = set(os.listdir(json1_path)) +# json2_files = set(os.listdir(json2_path)) +# json3_files = set(os.listdir(json3_path)) + +# # check for mismatches +# if json1_files == json2_files == json3_files: +# os.makedirs(outputs["output"].path, exist_ok=True) +# for file in json1_files: +# json1_doc = TE_Document(**json.load(open(os.path.join(json1_path, file)))) +# json2_doc = TE_Document(**json.load(open(os.path.join(json2_path, file)))) +# json3_doc = TE_Document(**json.load(open(os.path.join(json3_path, file)))) + +# merged_doc = TE_Document() +# merged_doc.chains = json1_doc.chains + json2_doc.chains + json3_doc.chains +# merged_doc.links = json1_doc.links + json2_doc.links + json3_doc.links +# merged_doc.triples = json1_doc.triples + json2_doc.triples + json3_doc.triples + +# merged_doc = filter_ie_doc(merged_doc) + +# with open(os.path.join(outputs["output"].path, file), "w") as f: +# f.write(merged_doc.model_dump_json()) +# # print(f"Converted {file} to {os.path.join(outputs['output'].path, file)}") +# else: +# print("File mismatch detected:") +# print("Files only in json1:", json1_files - json2_files - json3_files) +# print("Files only in json2:", json2_files - json1_files - json3_files) +# print("Files only in json3:", json3_files - json1_files - json2_files) +# print("Common files in all:", json1_files & json2_files & json3_files) +# raise Exception("All input directories must contain the same file names") +# else: +# raise Exception("All inputs must be directories") + +# aggregate_3iejson_with_filter_task = KgTask( +# name="aggregate_iejson_with_filter_task", +# input_spec={"json1": DataFormat.TE_JSON, "json2": DataFormat.TE_JSON, "json3": DataFormat.TE_JSON}, +# output_spec={"output": DataFormat.TE_JSON}, +# function=aggregate_3iejson_with_filter +# ) + diff --git a/experiments/param-opti/src/kgpipe_search/estimate.py b/experiments/param-opti/src/kgpipe_search/estimate.py new file mode 100644 index 0000000..04727d6 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/estimate.py @@ -0,0 +1,11 @@ +from typing import Tuple + +import numpy as np + + +def wilson_interval(p: float, n: int) -> Tuple[float, float]: + z = 1.96 + return p ± np.sqrt(p * (1 - p) / n) * z + +def wald_interval(p: float, n: int) -> Tuple[float, float]: + return p ± np.sqrt(p * (1 - p) / n) * z \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/evaluation.py b/experiments/param-opti/src/kgpipe_search/evaluation.py new file mode 100644 index 0000000..e4d5926 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/evaluation.py @@ -0,0 +1,172 @@ +from __future__ import annotations + +from typing import Any, Mapping + +from kgpipe_eval.evaluator import Evaluator +from kgpipe_eval.utils.kg_utils import KgLike, KgManager +from kgpipe_eval.utils.metric_utils import MeasurementKey +from kgpipe_eval.utils.score_utils import ( + AggregateScore, + aggregate_scores, + aggregate_scores_from_json, + aggregate_scores_from_results, +) +from kgpipe_search.definitions import PipelineConfig +from kgpipe_search.ranking_conf import DEFAULT_AGGREGATION_CONFIG, get_aggregation_config +import os + +# Backwards-compatible alias for the historical default aggregation. +aggregation_config = DEFAULT_AGGREGATION_CONFIG + + +def test_aggregate_results(): + result = aggregate_scores_from_json('data/eval_results.json', aggregation_config) + print(f'Final score: {result.final_score:.6f}') + for name, sg in result.subgroups.items(): + print(f' {name}: {sg.score:.6f}') + for m in sg.measurements: + print(f' {m.metric}.{m.measurement} = {m.value:.6f}') + + +def measurements_from_cached_evaluation(evaluation: Mapping[str, Any]) -> dict[MeasurementKey, float]: + """Extract raw metric measurements from a cached AggregateScore JSON payload.""" + lookup: dict[MeasurementKey, float] = {} + subgroups = evaluation.get("subgroups") + if not isinstance(subgroups, Mapping): + return lookup + for subgroup in subgroups.values(): + if not isinstance(subgroup, Mapping): + continue + measurements = subgroup.get("measurements") + if not isinstance(measurements, list): + continue + for item in measurements: + if not isinstance(item, Mapping): + continue + metric = item.get("metric") + measurement = item.get("measurement") + value = item.get("value") + if not isinstance(metric, str) or not isinstance(measurement, str): + continue + if not isinstance(value, (int, float)): + continue + lookup[MeasurementKey(metric=metric, measurement=measurement, unit="")] = float(value) + return lookup + + +def aggregate_from_cached_evaluation( + evaluation: Mapping[str, Any], + config: Mapping[str, Any] | str | None = None, +) -> AggregateScore: + """ + Re-aggregate a cached eval snapshot with ``config``. + + ``config`` may be an aggregation dict or a named config from ranking_conf + (``default``, ``flat_hmean``). Defaults to the historical subgroup aggregation. + Falls back to the stored ``final_score`` when measurements are missing. + """ + if config is None: + resolved = DEFAULT_AGGREGATION_CONFIG + elif isinstance(config, str): + resolved = get_aggregation_config(config) + else: + resolved = config + + lookup = measurements_from_cached_evaluation(evaluation) + if not lookup: + final_score = evaluation.get("final_score") + if isinstance(final_score, (int, float)): + return AggregateScore(final_score=float(final_score)) + raise ValueError("cached evaluation has neither measurements nor final_score") + + return aggregate_scores(lookup, resolved) + + +def score_from_cached_evaluation( + evaluation: Mapping[str, Any], + config: Mapping[str, Any] | str | None = None, +) -> float: + """Convenience wrapper returning only the final score.""" + return float(aggregate_from_cached_evaluation(evaluation, config).final_score) + + +def evaluate_pipeline( + pipeline_config: PipelineConfig, + result_kg: KgLike, + reference_kg: KgLike, + aggregation: Mapping[str, Any] | str | None = None, +): + from kgpipe_eval.metrics.statistics import CountMetric + from kgpipe_eval.metrics.triple_alignment import TripleAlignmentMetric, TripleAlignmentConfig + from kgpipe_eval.metrics.entity_alignment import EntityAlignmentMetric, EntityAlignmentConfig + from kgpipe_eval.metrics.consistency_violations import ConsistencyViolationsConfig,DisjointDomainMetric, DomainMetric, RangeMetric, DatatypeFormatMetric, DatatypeMetric, RelationDirectionMetric + + from kgpipe_eval.utils.kg_utils import KgManager + + if aggregation is None: + resolved_config = DEFAULT_AGGREGATION_CONFIG + elif isinstance(aggregation, str): + resolved_config = get_aggregation_config(aggregation) + else: + resolved_config = aggregation + + source_seed_path: KgLike = os.getenv("SOURCE_SEED_PATH") + source_seed_graph = KgManager.load_kg(source_seed_path) + result_graph = KgManager.load_kg(result_kg) + result_no_seed_graph = KgManager.substract_kg(result_graph, source_seed_graph) + + # Empty after seed subtract: alignment encode/dot and some consistency metrics break. + if len(result_no_seed_graph.get_graph()) == 0: + KgManager.unload_kg(result_graph) + KgManager.unload_kg(result_no_seed_graph) + return AggregateScore(final_score=0.0) + + consistency_violations_config = ConsistencyViolationsConfig( + reference_kg=None, + ontology_path=os.getenv("ONTOLOGY_PATH") + ) + + entity_alignment_config = EntityAlignmentConfig( + method="label_embedding", + reference_kg=reference_kg, + verified_entities_path=None, + verified_entities_delimiter="\t", + entity_sim_threshold=0.95 + ) + + triple_alignment_config = TripleAlignmentConfig( + reference_kg=reference_kg, + entity_alignment_config=entity_alignment_config, + value_sim_threshold=0.5, + cache_literal_embeddings=True + ) + + try: + results = Evaluator().run(result_no_seed_graph, [TripleAlignmentMetric(), EntityAlignmentMetric(), CountMetric(), DisjointDomainMetric(), DomainMetric(), RangeMetric(), DatatypeFormatMetric(), DatatypeMetric(), RelationDirectionMetric()], { + "TripleAlignmentMetric": triple_alignment_config, + "EntityAlignmentMetric": entity_alignment_config, + "DisjointDomainMetric": consistency_violations_config, + "DomainMetric": consistency_violations_config, + "RangeMetric": consistency_violations_config, + "DatatypeFormatMetric": consistency_violations_config, + "DatatypeMetric": consistency_violations_config, + "RelationDirectionMetric": consistency_violations_config + }) + finally: + KgManager.unload_kg(result_graph) + KgManager.unload_kg(result_no_seed_graph) + + return aggregate_scores_from_results(results, resolved_config) + + +import random + +def dummy_evaluate_pipeline(pipeline_config: PipelineConfig, result_kg: KgLike, reference_kg: KgLike): + return random.uniform(0.5, 1.0) # 0.5 to 1.0 + +def _execute_pipeline(pipeline_config: PipelineConfig): + pass + +def execute_and_dummy_evaluate_pipeline(pipeline_config: PipelineConfig): + result = _execute_pipeline(pipeline_config) + return dummy_evaluate_pipeline(pipeline_config, None, None) diff --git a/experiments/param-opti/src/kgpipe_search/ranking_conf.py b/experiments/param-opti/src/kgpipe_search/ranking_conf.py new file mode 100644 index 0000000..da8d60d --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/ranking_conf.py @@ -0,0 +1,113 @@ +"""Named score-aggregation configs used when ranking pipeline evaluation results.""" + +from __future__ import annotations + +from typing import Any, Dict, Mapping + +# All measurements that participate in the default ranking (10 values across 3 subgroups). +_ALL_MEASUREMENTS = [ + "EntityAlignmentMetric.recall", + "TripleAlignmentMetric.recall", + "EntityAlignmentMetric.precision", + "TripleAlignmentMetric.precision", + "DisjointDomainMetric.normalized_score", + "DomainMetric.normalized_score", + "RangeMetric.normalized_score", + "DatatypeFormatMetric.normalized_score", + "DatatypeMetric.normalized_score", + "RelationDirectionMetric.normalized_score", +] + +# Subgroup means, then equal-weight mean of the three subgroup scores. +DEFAULT_AGGREGATION_CONFIG: Dict[str, Any] = { + "subgroups": { + "coverage": { + "measurements": [ + {"metric": "EntityAlignmentMetric", "measurement": "recall"}, + {"metric": "TripleAlignmentMetric", "measurement": "recall"}, + ], + "aggregation": "mean", + }, + "correctness": { + "measurements": [ + "EntityAlignmentMetric.precision", + "TripleAlignmentMetric.precision", + ], + "aggregation": "mean", + }, + "consistency": { + "measurements": [ + "DisjointDomainMetric.normalized_score", + "DomainMetric.normalized_score", + "RangeMetric.normalized_score", + "DatatypeFormatMetric.normalized_score", + "DatatypeMetric.normalized_score", + "RelationDirectionMetric.normalized_score", + ], + "aggregation": "mean", + }, + }, + "final": { + "aggregation": "weighted_mean", + "weights": { + "coverage": 0.3333, + "correctness": 0.3333, + "consistency": 0.3333, + }, + }, +} + +# CUSTOM AGGREGATION CONFIG +CUSTOM_AGGREGATION_CONFIG: Dict[str, Any] = { + "subgroups": { + "coverage_and_correctness": { + "measurements": [ + "EntityAlignmentMetric.recall", "TripleAlignmentMetric.recall", + "EntityAlignmentMetric.precision", "TripleAlignmentMetric.precision" + ], + "aggregation": "harmonic_mean", + }, + "consistency_and_correctness": { + "measurements": [ + "DisjointDomainMetric.normalized_score", + "DomainMetric.normalized_score", + "RangeMetric.normalized_score", + "DatatypeFormatMetric.normalized_score", + "DatatypeMetric.normalized_score", + "RelationDirectionMetric.normalized_score", + ], + "aggregation": "harmonic_mean", + }, + }, + "final": { + "aggregation": "harmonic_mean", + }, +} + + +# Flat harmonic mean over all measurements (no subgroup intermediate scores). +FLAT_HMEAN_AGGREGATION_CONFIG: Dict[str, Any] = { + "subgroups": { + "all": { + "measurements": list(_ALL_MEASUREMENTS), + "aggregation": "harmonic_mean", + }, + }, + "final": { + "aggregation": "mean", + }, +} + +AGGREGATION_CONFIGS: Dict[str, Dict[str, Any]] = { + "default": DEFAULT_AGGREGATION_CONFIG, + "flat_hmean": FLAT_HMEAN_AGGREGATION_CONFIG, + "custom": CUSTOM_AGGREGATION_CONFIG, +} + + +def get_aggregation_config(name: str) -> Mapping[str, Any]: + try: + return AGGREGATION_CONFIGS[name] + except KeyError as exc: + known = ", ".join(sorted(AGGREGATION_CONFIGS)) + raise ValueError(f"Unknown rank aggregation {name!r}; choose one of: {known}") from exc diff --git a/experiments/param-opti/src/kgpipe_search/reaggregate_evals.py b/experiments/param-opti/src/kgpipe_search/reaggregate_evals.py new file mode 100644 index 0000000..fe174ad --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/reaggregate_evals.py @@ -0,0 +1,463 @@ +#!/usr/bin/env python3 +"""Re-aggregate cached ``*.eval.json`` snapshots under alternate ranking configs. + +Cached eval files store per-metric measurements (and a ``final_score`` under the +default aggregation). This script recomputes ``final_score`` for every named +aggregation in ``ranking_conf.AGGREGATION_CONFIGS`` (or a chosen subset) and +writes sorted score lists — without re-running pipelines or metrics. + +It also plots sorted score curves (x = config index 1..N, y = final_score) for +each aggregation, optionally side-by-side for RDF and text. + +Example:: + + PYTHONPATH=src python -m kgpipe_search.reaggregate_evals \\ + --eval-dir runs/rdf runs/text \\ + --out-dir runs/score_curves +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any, Dict, List, Mapping, Optional, Sequence, Tuple + +import matplotlib.pyplot as plt + +from kgpipe_search.evaluation import score_from_cached_evaluation +from kgpipe_search.ranking_conf import AGGREGATION_CONFIGS, get_aggregation_config + +# Match paper-style sizing used by plot_search_evolution_aggregate. +COL_WIDTH_IN = 3.5 +COL_HEIGHT_IN = 2.6 +PAPER_DPI = 300 +PAPER_RC = { + "font.size": 9, + "axes.labelsize": 9, + "axes.titlesize": 9, + "xtick.labelsize": 8, + "ytick.labelsize": 8, + "legend.fontsize": 7, + "axes.linewidth": 0.8, + "lines.linewidth": 1.5, + "grid.linewidth": 0.5, +} + +AGG_LABELS = { + "default": "default", + "custom": "custom", + "flat_hmean": "flat hmean", +} + +AGG_LINESTYLES = { + "default": "-", + "custom": "--", + "flat_hmean": ":", +} + + +def _config_hash_from_eval_path(path: Path) -> str: + name = path.name + suffix = ".eval.json" + if name.endswith(suffix): + return name[: -len(suffix)] + return path.stem + + +def discover_eval_files(eval_dir: Path) -> List[Path]: + files = sorted(eval_dir.glob("*.eval.json")) + if not files: + raise FileNotFoundError(f"No *.eval.json files found in {eval_dir}") + return files + + +def load_evaluation(path: Path) -> Optional[Mapping[str, Any]]: + """Load a cached eval payload, or ``None`` for error / unusable snapshots.""" + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + print(f"warning: skip {path.name}: {exc}", file=sys.stderr) + return None + if not isinstance(payload, dict): + print(f"warning: skip {path.name}: expected JSON object", file=sys.stderr) + return None + if payload.get("status") == "error": + print(f"warning: skip {path.name}: cached error", file=sys.stderr) + return None + if "subgroups" not in payload and not isinstance(payload.get("final_score"), (int, float)): + print(f"warning: skip {path.name}: no measurements or final_score", file=sys.stderr) + return None + return payload + + +def reaggregate_eval_dir( + eval_dir: Path, + *, + aggregations: Sequence[str], +) -> Dict[str, List[Dict[str, Any]]]: + """ + Return ``aggregation ->`` sorted list of ``{config_hash, final_score, eval_path}``. + + Lists are sorted by ``final_score`` descending (ties broken by config hash). + """ + for name in aggregations: + get_aggregation_config(name) # validate early + + rows_by_agg: Dict[str, List[Dict[str, Any]]] = {name: [] for name in aggregations} + skipped = 0 + + for path in discover_eval_files(eval_dir): + evaluation = load_evaluation(path) + if evaluation is None: + skipped += 1 + continue + config_hash = _config_hash_from_eval_path(path) + scores: Dict[str, float] = {} + try: + for name in aggregations: + scores[name] = float(score_from_cached_evaluation(evaluation, name)) + except Exception as exc: + print(f"warning: skip {path.name}: {exc}", file=sys.stderr) + skipped += 1 + continue + for name, score in scores.items(): + rows_by_agg[name].append( + { + "config_hash": config_hash, + "final_score": score, + "eval_path": str(path), + } + ) + + for name, rows in rows_by_agg.items(): + rows.sort(key=lambda r: (-float(r["final_score"]), str(r["config_hash"]))) + for rank, row in enumerate(rows, start=1): + row["rank"] = rank + + if skipped: + print(f"skipped {skipped} eval file(s)", file=sys.stderr) + return rows_by_agg + + +def _summary(rows: Sequence[Mapping[str, Any]]) -> Dict[str, Any]: + if not rows: + return {"n": 0, "max": None, "min": None, "best_config_hash": None} + return { + "n": len(rows), + "max": float(rows[0]["final_score"]), + "min": float(rows[-1]["final_score"]), + "best_config_hash": rows[0]["config_hash"], + } + + +def write_outputs( + rows_by_agg: Mapping[str, List[Dict[str, Any]]], + *, + eval_dir: Path, + out_dir: Path, + also_scores_only: bool = True, +) -> Path: + """Write combined JSON plus per-aggregation sorted lists under ``out_dir``.""" + out_dir.mkdir(parents=True, exist_ok=True) + + payload: Dict[str, Any] = { + "eval_dir": str(eval_dir.resolve()), + "summary": {name: _summary(rows) for name, rows in rows_by_agg.items()}, + "rankings": { + name: [ + { + "rank": row["rank"], + "config_hash": row["config_hash"], + "final_score": row["final_score"], + } + for row in rows + ] + for name, rows in rows_by_agg.items() + }, + } + combined_path = out_dir / "reaggregated_scores.json" + combined_path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8") + + for name, rows in rows_by_agg.items(): + ranked_path = out_dir / f"scores_{name}.json" + ranked_path.write_text( + json.dumps( + [ + { + "rank": row["rank"], + "config_hash": row["config_hash"], + "final_score": row["final_score"], + } + for row in rows + ], + indent=2, + ) + + "\n", + encoding="utf-8", + ) + if also_scores_only: + # Ascending score list in the same style as runs/*_final_score_dist. + scores_asc = sorted(float(r["final_score"]) for r in rows) + dist_path = out_dir / f"scores_{name}.final_score_dist" + dist_path.write_text( + "".join(f' "final_score": {s},\n' for s in scores_asc), + encoding="utf-8", + ) + tsv_path = out_dir / f"scores_{name}.tsv" + tsv_path.write_text( + "rank\tconfig_hash\tfinal_score\n" + + "".join( + f"{row['rank']}\t{row['config_hash']}\t{row['final_score']}\n" + for row in rows + ), + encoding="utf-8", + ) + + return combined_path + + +def _domain_label(eval_dir: Path) -> str: + name = eval_dir.name.lower() + if "rdf" in name: + return "RDF" + if "text" in name: + return "Text" + return eval_dir.name + + +def _sorted_scores_asc(rows: Sequence[Mapping[str, Any]]) -> List[float]: + return sorted(float(r["final_score"]) for r in rows) + + +def _plot_curves_on_ax( + ax: Any, + rows_by_agg: Mapping[str, List[Dict[str, Any]]], + *, + aggregations: Sequence[str], + y_full: bool = False, +) -> None: + all_ys: List[float] = [] + for name in aggregations: + rows = rows_by_agg.get(name) or [] + if not rows: + continue + ys = _sorted_scores_asc(rows) + all_ys.extend(ys) + xs = list(range(1, len(ys) + 1)) + ax.plot( + xs, + ys, + linestyle=AGG_LINESTYLES.get(name, "-"), + linewidth=1.5, + label=f"{AGG_LABELS.get(name, name)} (n={len(ys)})", + ) + ax.set_xlabel("Configs (sorted by score)") + ax.set_ylabel("Final score") + if y_full: + ax.set_ylim(0.0, 1.0) + elif all_ys: + lo, hi = min(all_ys), max(all_ys) + pad = max(0.02, 0.05 * (hi - lo) if hi > lo else 0.05) + ax.set_ylim(max(0.0, lo - pad), min(1.0, hi + pad)) + ax.grid(True, alpha=0.3) + ax.legend(loc="lower right", frameon=False) + + +def plot_sorted_score_curves( + rows_by_agg: Mapping[str, List[Dict[str, Any]]], + *, + aggregations: Sequence[str], + out: Path, + title: str, + y_full: bool = False, +) -> Path: + """Plot ascending sorted score curves for each aggregation into ``out``.""" + out.parent.mkdir(parents=True, exist_ok=True) + with plt.rc_context(PAPER_RC): + fig, ax = plt.subplots(figsize=(COL_WIDTH_IN, COL_HEIGHT_IN)) + _plot_curves_on_ax(ax, rows_by_agg, aggregations=aggregations, y_full=y_full) + if title: + ax.set_title(title) + fig.tight_layout() + fig.savefig(out, dpi=PAPER_DPI) + plt.close(fig) + return out + + +def plot_sorted_score_curves_panel( + panels: Sequence[Tuple[str, Mapping[str, List[Dict[str, Any]]]]], + *, + aggregations: Sequence[str], + out: Path, + title: str = "", + y_full: bool = False, +) -> Path: + """Side-by-side sorted score curves (e.g. RDF | Text).""" + if not panels: + raise ValueError("panels must be non-empty") + out.parent.mkdir(parents=True, exist_ok=True) + n = len(panels) + with plt.rc_context(PAPER_RC): + fig, axes = plt.subplots( + 1, + n, + figsize=(COL_WIDTH_IN * n, COL_HEIGHT_IN), + sharey=False, + squeeze=False, + ) + for ax, (panel_title, rows_by_agg) in zip(axes[0], panels): + _plot_curves_on_ax( + ax, rows_by_agg, aggregations=aggregations, y_full=y_full + ) + ax.set_title(panel_title) + if title: + fig.suptitle(title, y=1.02) + fig.tight_layout() + fig.savefig(out, dpi=PAPER_DPI, bbox_inches="tight") + plt.close(fig) + return out + + +def _resolve_out_dir(eval_dir: Path, out_dir: Optional[Path], *, multi: bool) -> Path: + if out_dir is None: + return eval_dir.parent / f"{eval_dir.name}_reaggregated" + if multi: + return out_dir / eval_dir.name + return out_dir + + +def build_parser() -> argparse.ArgumentParser: + known = ", ".join(sorted(AGGREGATION_CONFIGS)) + p = argparse.ArgumentParser( + description=( + "Recompute final_score for cached *.eval.json files under one or more " + "rank-aggregation configs, write sorted score lists, and plot curves." + ) + ) + p.add_argument( + "--eval-dir", + type=Path, + nargs="+", + required=True, + help="One or more directories containing *.eval.json (e.g. runs/rdf runs/text)", + ) + p.add_argument( + "--out-dir", + type=Path, + default=None, + help=( + "Output directory. With one --eval-dir: used directly " + "(default _reaggregated). With several: per-domain " + "subdirs are created under this path." + ), + ) + p.add_argument( + "--aggregations", + nargs="+", + choices=sorted(AGGREGATION_CONFIGS), + default=sorted(AGGREGATION_CONFIGS), + help=f"Aggregation config names to recompute (default: all of {known})", + ) + p.add_argument( + "--top", + type=int, + default=10, + help="Print top-N scores per aggregation to stdout (0 to silence)", + ) + p.add_argument( + "--no-scores-only", + action="store_true", + help="Do not write .final_score_dist / .tsv companion files", + ) + p.add_argument( + "--no-plot", + action="store_true", + help="Skip writing sorted-score curve plots", + ) + p.add_argument( + "--plot-title", + type=str, + default="", + help="Optional title for the combined RDF|Text panel plot", + ) + p.add_argument( + "--y-full", + action="store_true", + help="Force y-axis to [0, 1] instead of fitting each panel's score range", + ) + return p + + +def main(argv: Optional[Sequence[str]] = None) -> int: + args = build_parser().parse_args(argv) + eval_dirs = [p.resolve() for p in args.eval_dir] + for eval_dir in eval_dirs: + if not eval_dir.is_dir(): + raise SystemExit(f"--eval-dir is not a directory: {eval_dir}") + + multi = len(eval_dirs) > 1 + root_out = args.out_dir.resolve() if args.out_dir is not None else None + if multi and root_out is None: + # Shared parent next to the first eval dir, e.g. runs/score_curves + root_out = eval_dirs[0].parent / "score_curves" + + panel_data: List[Tuple[str, Dict[str, List[Dict[str, Any]]]]] = [] + + for eval_dir in eval_dirs: + out_dir = _resolve_out_dir(eval_dir, root_out, multi=multi) + rows_by_agg = reaggregate_eval_dir(eval_dir, aggregations=args.aggregations) + combined = write_outputs( + rows_by_agg, + eval_dir=eval_dir, + out_dir=out_dir, + also_scores_only=not args.no_scores_only, + ) + print(f"wrote {combined}") + for name, rows in rows_by_agg.items(): + summary = _summary(rows) + print( + f" {name}: n={summary['n']} max={summary['max']} " + f"best={summary['best_config_hash']}" + ) + if args.top > 0 and rows: + print(f" top {min(args.top, len(rows))} ({name}):") + for row in rows[: args.top]: + print( + f" {row['rank']:4d} {row['final_score']:.10f} {row['config_hash']}" + ) + + domain = _domain_label(eval_dir) + n_configs = len(next(iter(rows_by_agg.values()), [])) + panel_data.append((f"{domain} (n={n_configs})", rows_by_agg)) + + if not args.no_plot: + plot_path = plot_sorted_score_curves( + rows_by_agg, + aggregations=args.aggregations, + out=out_dir / "sorted_score_curve.png", + title=f"{domain} sorted final scores", + y_full=args.y_full, + ) + print(f"wrote {plot_path}") + + if not args.no_plot and len(panel_data) > 1: + panel_out = (root_out or eval_dirs[0].parent) / "sorted_score_curves.png" + path = plot_sorted_score_curves_panel( + panel_data, + aggregations=args.aggregations, + out=panel_out, + title=args.plot_title, + y_full=args.y_full, + ) + print(f"wrote {path}") + + return 0 + + +if __name__ == "__main__": + try: + raise SystemExit(main()) + except KeyboardInterrupt: + raise SystemExit(130) diff --git a/experiments/param-opti/src/kgpipe_search/sample.py b/experiments/param-opti/src/kgpipe_search/sample.py new file mode 100644 index 0000000..c0628d1 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/sample.py @@ -0,0 +1,17 @@ + +class TripleSampleIterator: + + def __init__(self): + + def __iter__(self): + return self + + def __next__(self) -> List[Tuple[str, str, str]]: + if self.budget <= 0: + raise StopIteration + self.budget -= 1 + return self.next() + + def next(self) -> List[Tuple[str, str, str]]: + return random.sample(self.config_space, self.budget) + diff --git a/experiments/param-opti/src/kgpipe_search/search.py b/experiments/param-opti/src/kgpipe_search/search.py new file mode 100644 index 0000000..a220a32 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/search.py @@ -0,0 +1,241 @@ +""" +Public API for configuration search. + +Implementation lives in `kgpipe_search/strategies/` to keep algorithms modular. +This module preserves the historical function names used by existing tests/scripts. +""" + +import random +from typing import Any, Dict + +from kgpipe_search.definitions import PipelineLayout +from kgpipe_search.strategies.initialization import ( + implementation_aware_initialization, + random_initialization, +) +from kgpipe_search.strategies.llm_strategy import run_llm +from kgpipe_search.strategies.strategies import ( + EvaluateFn, + SearchRun, + run_bayesian, + run_hnr, + run_hnr_2, + run_implementation_aware, + run_qgns, + run_random, +) + +__all__ = [ + "SearchRun", + "EvaluateFn", + "random_initialization", + "implementation_aware_initialization", + "random_search", + "implementation_aware_search", + "neighborhood_optimization", + "qgns_search", + "hnr_search", + "hnr_2_search", + "bayesian_optimization", + "llm_search", +] + + +def random_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + rng: Any = None, +) -> SearchRun: + return run_random( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + rng=rng, + ) + + +def implementation_aware_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + y: int = 1, + rng: Any = None, +) -> SearchRun: + return run_implementation_aware( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + y=y, + rng=rng, + ) + + +def qgns_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int = 0, + init_strategy: str = "random", + y: int = 1, + k: int = 3, + rho: float = 0.2, + rng: Any = None, +) -> SearchRun: + return run_qgns( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=init_budget, + init_strategy="implementation_aware" + if init_strategy == "implementation_aware" + else "random", + y=y, + k=k, + rho=rho, + rng=rng, + ) + + +def hnr_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int, + init_strategy: str = "implementation_aware", + y: int = 1, + rho: float = 0.2, + rng: Any = None, +) -> SearchRun: + return run_hnr( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=init_budget, + init_strategy="random" if init_strategy == "random" else "implementation_aware", + y=y, + rho=rho, + rng=rng, + ) + + +def hnr_2_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int, + init_strategy: str = "implementation_aware", + y: int = 1, + rho: float = 0.2, + min_quality_delta: float = 0.05, + min_iterations_wo_improvement: int = 2, + rng: Any = None, +) -> SearchRun: + return run_hnr_2( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=init_budget, + init_strategy="random" if init_strategy == "random" else "implementation_aware", + y=y, + rho=rho, + min_quality_delta=min_quality_delta, + min_iterations_wo_improvement=min_iterations_wo_improvement, + rng=rng, + ) + +def neighborhood_optimization( + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + k: int = 3, + rho: float = 0.2, + rng: Any = None, + **kwargs: Any, +) -> SearchRun: + """ + Backwards-compatible alias. + + Historically, this was called `neighborhood_optimization` and implemented QGNS-like behavior. + """ + del kwargs + return qgns_search( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=0, + init_strategy="random", + k=k, + rho=rho, + rng=rng, + ) + + +def llm_search( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + max_retries: int = 3, + client: Any = None, + rng: Any = None, +) -> SearchRun: + return run_llm( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + max_retries=max_retries, + client=client, + rng=rng, + ) + + +def bayesian_optimization( + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + init_random: int = 3, + init_strategy: str = "random", + y: int = 1, + pool_size: int = 32, + beta: float = 0.5, + rng: Any = None, + **kwargs: Any, +) -> SearchRun: + del kwargs + return run_bayesian( + budget=budget, + evaluate_fn=evaluate_fn, + search_space=search_space, + pipeline_layout=pipeline_layout, + init_budget=init_random, + init_strategy="implementation_aware" + if init_strategy == "implementation_aware" + else "random", + y=y, + pool_size=pool_size, + beta=beta, + rng=rng, + ) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/strategies/__init__.py b/experiments/param-opti/src/kgpipe_search/strategies/__init__.py new file mode 100644 index 0000000..efcc119 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/__init__.py @@ -0,0 +1,10 @@ +"""Search strategies and initialization routines for KGpipe configuration search.""" + +from kgpipe_search.strategies.llm_strategy import propose_pipeline_config_with_llm, run_llm +from kgpipe_search.strategies.strategies import SearchRun + +__all__ = [ + "SearchRun", + "propose_pipeline_config_with_llm", + "run_llm", +] diff --git a/experiments/param-opti/src/kgpipe_search/strategies/initialization.py b/experiments/param-opti/src/kgpipe_search/strategies/initialization.py new file mode 100644 index 0000000..fa5318a --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/initialization.py @@ -0,0 +1,154 @@ +import random +from typing import Any, Dict, List, Optional, Sequence, Set + +from kgpipe_search.configuration import ( + build_pipeline_config_for_task_combo, + enumerate_valid_task_combinations, + pipeline_config_snapshot_key, + sample_valid_pipeline_config, +) +from kgpipe_search.definitions import PipelineConfig, PipelineLayout + + +def _try_add_unique_config( + configs: List[PipelineConfig], + seen: Set[str], + candidate: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], +) -> bool: + key = pipeline_config_snapshot_key(candidate, search_space) + if key in seen: + return False + seen.add(key) + configs.append(candidate) + return True + + +def _fill_unique_configs( + *, + configs: List[PipelineConfig], + seen: Set[str], + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + budget: int, + rng: random.Random, + max_attempts_factor: int = 200, +) -> None: + attempts = 0 + max_attempts = max(1000, max(1, budget - len(configs)) * max_attempts_factor) + while len(configs) < budget and attempts < max_attempts: + attempts += 1 + candidate = sample_valid_pipeline_config(search_space, pipeline_layout, rng=rng) + _try_add_unique_config(configs, seen, candidate, search_space) + + +def random_initialization( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + budget: int, + rng: Optional[random.Random] = None, +) -> List[PipelineConfig]: + """Sample `budget` unique valid pipeline configurations uniformly at random.""" + if budget <= 0: + return [] + + draw = rng or random.Random() + configs: List[PipelineConfig] = [] + seen: Set[str] = set() + attempts = 0 + max_attempts = max(1000, budget * 200) + + while len(configs) < budget and attempts < max_attempts: + attempts += 1 + candidate = sample_valid_pipeline_config(search_space, pipeline_layout, rng=draw) + key = pipeline_config_snapshot_key(candidate, search_space) + if key in seen: + continue + seen.add(key) + configs.append(candidate) + + if len(configs) < budget: + raise RuntimeError( + f"Failed to sample {budget} unique initial configs (got {len(configs)})." + ) + + return configs + + +def implementation_aware_initialization( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + *, + budget: int, + y: int = 1, + rng: Optional[random.Random] = None, +) -> List[PipelineConfig]: + """ + Implementation-aware initialization. + + Enumerate (or sample) valid implementation assignments (task combinations) and, + for each such assignment, generate `y` configurations by sampling parameters. + """ + if budget <= 0: + return [] + if y <= 0: + raise ValueError("y must be >= 1") + + draw = rng or random.Random() + all_combos = enumerate_valid_task_combinations(search_space, pipeline_layout) + if not all_combos: + raise ValueError("No valid implementation assignments found.") + + max_combos = max(1, budget // y) + combos: Sequence[List[str]] + if len(all_combos) <= max_combos: + combos = all_combos + else: + combos = draw.sample(all_combos, k=max_combos) + + configs: List[PipelineConfig] = [] + seen: Set[str] = set() + + for combo in combos: + added_for_combo = 0 + attempts = 0 + max_attempts = max(100, y * 50) + while ( + added_for_combo < y + and len(configs) < budget + and attempts < max_attempts + ): + attempts += 1 + candidate = build_pipeline_config_for_task_combo( + search_space, + combo, + rng=draw, + template=None, + ) + if _try_add_unique_config(configs, seen, candidate, search_space): + added_for_combo += 1 + + if len(configs) >= budget: + break + + if len(configs) < budget: + _fill_unique_configs( + configs=configs, + seen=seen, + search_space=search_space, + pipeline_layout=pipeline_layout, + budget=budget, + rng=draw, + ) + + if len(configs) < budget: + raise RuntimeError( + f"Failed to generate {budget} unique initial configs (got {len(configs)}). " + f"The search space has {len(all_combos)} implementation assignment(s); " + "try lowering init_budget." + ) + + return configs + + diff --git a/experiments/param-opti/src/kgpipe_search/strategies/llm_client.py b/experiments/param-opti/src/kgpipe_search/strategies/llm_client.py new file mode 100644 index 0000000..f50c113 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/llm_client.py @@ -0,0 +1,98 @@ +from __future__ import annotations + +import json +import os +import urllib.error +import urllib.request +from dataclasses import dataclass +from typing import Any, Dict, List, Optional, Protocol + + +class ChatCompletionClient(Protocol): + def complete(self, *, system: str, user: str) -> str: + ... + + +def _env_first(*names: str) -> Optional[str]: + for name in names: + value = os.environ.get(name) + if value: + return value + return None + + +@dataclass +class OpenAICompatibleClient: + """ + Minimal OpenAI-compatible chat client. + + Configuration via environment variables: + - endpoint: KGPipe_SEARCH_LLM_ENDPOINT, OPENAI_BASE_URL, OPENAI_API_BASE + - token: KGPipe_SEARCH_LLM_TOKEN, OPENAI_API_KEY + - model: KGPipe_SEARCH_LLM_MODEL (default: gpt-4o-mini) + """ + + endpoint: str + token: str + model: str = "gpt-4o-mini" + timeout_s: float = 60.0 + + @classmethod + def from_env(cls) -> "OpenAICompatibleClient": + endpoint = _env_first( + "KGPipe_SEARCH_LLM_ENDPOINT", + "OPENAI_BASE_URL", + "OPENAI_API_BASE", + ) + token = _env_first("KGPipe_SEARCH_LLM_TOKEN", "OPENAI_API_KEY") + if not endpoint: + raise ValueError( + "LLM endpoint not configured. Set KGPipe_SEARCH_LLM_ENDPOINT or OPENAI_BASE_URL." + ) + if not token: + raise ValueError( + "LLM token not configured. Set KGPipe_SEARCH_LLM_TOKEN or OPENAI_API_KEY." + ) + + model = os.environ.get("KGPipe_SEARCH_LLM_MODEL", "gpt-4o-mini") + return cls(endpoint=endpoint.rstrip("/"), token=token, model=model) + + def complete(self, *, system: str, user: str) -> str: + url = f"{self.endpoint}/chat/completions" + payload: Dict[str, Any] = { + "model": self.model, + "messages": [ + {"role": "system", "content": system}, + {"role": "user", "content": user}, + ], + "temperature": 0.2, + } + body = json.dumps(payload).encode("utf-8") + request = urllib.request.Request( + url, + data=body, + headers={ + "Authorization": f"Bearer {self.token}", + "Content-Type": "application/json", + }, + method="POST", + ) + + try: + with urllib.request.urlopen(request, timeout=self.timeout_s) as response: + raw = json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as exc: + detail = exc.read().decode("utf-8", errors="replace") + raise RuntimeError(f"LLM request failed ({exc.code}): {detail}") from exc + except urllib.error.URLError as exc: + raise RuntimeError(f"LLM request failed: {exc}") from exc + + choices: List[Dict[str, Any]] = raw.get("choices") or [] + if not choices: + raise RuntimeError(f"LLM response missing choices: {raw!r}") + + message = choices[0].get("message") or {} + content = message.get("content") + if not isinstance(content, str) or not content.strip(): + raise RuntimeError(f"LLM response missing message content: {raw!r}") + return content diff --git a/experiments/param-opti/src/kgpipe_search/strategies/llm_strategy.py b/experiments/param-opti/src/kgpipe_search/strategies/llm_strategy.py new file mode 100644 index 0000000..ede80cf --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/llm_strategy.py @@ -0,0 +1,190 @@ +from __future__ import annotations + +import json +import random +import re +from typing import Any, Dict, List, Optional, Set, Tuple + +from kgpipe_search.configuration import ( + pipeline_config_from_snapshot, + pipeline_config_snapshot_key, + pipeline_config_to_snapshot, + sample_valid_pipeline_config, + task_keys_from_pipeline_config, +) +from kgpipe_search.definitions import PipelineConfig, PipelineLayout +from kgpipe_search.strategies.llm_client import ChatCompletionClient, OpenAICompatibleClient +from kgpipe_search.strategies.llm_validation import ( + search_space_description, + validate_pipeline_config_snapshot, +) +from kgpipe_search.strategies.strategies import EvaluateFn, Observation, SearchRun + +_JSON_BLOCK_RE = re.compile(r"```(?:json)?\s*([\s\S]*?)\s*```", re.IGNORECASE) + + +def _extract_json_object(text: str) -> Dict[str, Any]: + stripped = text.strip() + candidates = [stripped] + for match in _JSON_BLOCK_RE.finditer(text): + candidates.append(match.group(1).strip()) + + last_error: Optional[Exception] = None + for candidate in candidates: + try: + parsed = json.loads(candidate) + if isinstance(parsed, dict): + return parsed + except json.JSONDecodeError as exc: + last_error = exc + continue + + raise ValueError(f"Could not parse JSON object from LLM response: {text!r}") from last_error + + +def _history_summary(history: List[Observation]) -> List[Dict[str, Any]]: + ranked = sorted(history, key=lambda item: item[0], reverse=True) + summary: List[Dict[str, Any]] = [] + for score, cfg in ranked[:5]: + task_keys = task_keys_from_pipeline_config(cfg) + summary.append( + { + "score": score, + "snapshot": pipeline_config_to_snapshot(task_keys, cfg), + } + ) + return summary + + +def _build_prompt( + *, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + history: List[Observation], + evaluated_keys: Set[str], + attempt: int, + last_error: str, +) -> Tuple[str, str]: + system = ( + "You propose valid KGpipe pipeline configurations. " + "Respond with a single JSON object only. " + "Every task_key must be chosen from valid_task_combinations. " + "Every parameter value must be one of the allowed values in tasks." + ) + payload = { + "search_space": search_space_description(search_space, pipeline_layout), + "attempt": attempt, + "already_evaluated_count": len(evaluated_keys), + "best_observations": _history_summary(history), + "last_validation_error": last_error or None, + "instructions": [ + "Pick one valid task_keys combination from valid_task_combinations.", + "For each selected task with parameters, provide bindings using only allowed values.", + "Prefer configs that differ from already evaluated ones when possible.", + "Return JSON matching output_schema.", + ], + } + return system, json.dumps(payload, indent=2) + + +def propose_pipeline_config_with_llm( + *, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + client: ChatCompletionClient, + history: Optional[List[Observation]] = None, + evaluated_keys: Optional[Set[str]] = None, + max_retries: int = 3, +) -> Tuple[PipelineConfig, str]: + """ + Ask an LLM for a pipeline config snapshot, validate it, and retry on failure. + """ + if max_retries < 1: + raise ValueError("max_retries must be >= 1") + + observations = history or [] + seen = evaluated_keys or set() + last_error = "" + + for attempt in range(1, max_retries + 1): + system, user = _build_prompt( + search_space=search_space, + pipeline_layout=pipeline_layout, + history=observations, + evaluated_keys=seen, + attempt=attempt, + last_error=last_error, + ) + raw = client.complete(system=system, user=user) + try: + snapshot = _extract_json_object(raw) + except ValueError as exc: + last_error = str(exc) + continue + + is_valid, error = validate_pipeline_config_snapshot( + snapshot, search_space, pipeline_layout + ) + if not is_valid: + last_error = error + continue + + config = pipeline_config_from_snapshot(snapshot) + key = pipeline_config_snapshot_key(config, search_space) + if key in seen: + last_error = "configuration was already evaluated" + continue + + return config, f"llm(attempt={attempt})" + + raise RuntimeError( + f"LLM failed to produce a valid unevaluated config after {max_retries} attempts. " + f"Last error: {last_error}" + ) + + +def run_llm( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + max_retries: int = 3, + client: Optional[ChatCompletionClient] = None, + rng: Optional[random.Random] = None, +) -> SearchRun: + if budget <= 0: + return SearchRun(strategy="llm", history=[], budget=0, decisions=[]) + + draw = rng or random.Random() + llm_client = client or OpenAICompatibleClient.from_env() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + for _ in range(budget): + try: + candidate, decision = propose_pipeline_config_with_llm( + search_space=search_space, + pipeline_layout=pipeline_layout, + client=llm_client, + history=history, + evaluated_keys=evaluated_keys, + max_retries=max_retries, + ) + except RuntimeError: + candidate = sample_valid_pipeline_config(search_space, pipeline_layout, rng=draw) + decision = "fallback(random)" + + key = pipeline_config_snapshot_key(candidate, search_space) + if key in evaluated_keys: + candidate = sample_valid_pipeline_config(search_space, pipeline_layout, rng=draw) + decision = f"{decision}+dedupe(random)" + + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append(decision) + + return SearchRun(strategy="llm", history=history, budget=budget, decisions=decisions) diff --git a/experiments/param-opti/src/kgpipe_search/strategies/llm_validation.py b/experiments/param-opti/src/kgpipe_search/strategies/llm_validation.py new file mode 100644 index 0000000..8d0cdbe --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/llm_validation.py @@ -0,0 +1,141 @@ +from __future__ import annotations + +from typing import Any, Dict, Tuple + +from kgpipe_search.configuration import ( + _task_categories_list, + enumerate_valid_task_combinations, + pipeline_config_from_snapshot, +) +from kgpipe_search.definitions import PipelineLayout, task_dict + + +def validate_pipeline_config_snapshot( + snapshot: Dict[str, Any], + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, +) -> Tuple[bool, str]: + """ + Validate an LLM-produced pipeline config snapshot against the search space and layout. + + Returns (is_valid, error_message). error_message is empty when valid. + """ + task_keys = snapshot.get("task_keys") + if not isinstance(task_keys, list) or not task_keys: + return False, "snapshot must contain a non-empty task_keys list" + if not all(isinstance(key, str) for key in task_keys): + return False, "task_keys must contain only strings" + + unknown = [key for key in task_keys if key not in search_space] + if unknown: + return False, f"unknown task keys: {unknown}" + + valid_combos = { + tuple(combo) + for combo in enumerate_valid_task_combinations(search_space, pipeline_layout) + } + if tuple(task_keys) not in valid_combos: + return False, f"task_keys {task_keys!r} is not a valid implementation assignment" + + covered: set[str] = set() + for task_key in task_keys: + covered.update(_task_categories_list(search_space, task_key)) + + required = set(pipeline_layout.allowed_task_categories) + if not required.issubset(covered): + missing = sorted(required - covered) + return False, f"pipeline does not cover required categories: {missing}" + + profiles = snapshot.get("profiles") + if profiles is None: + profiles = {} + if not isinstance(profiles, dict): + return False, "profiles must be an object when present" + + for task_key in task_keys: + task = task_dict[task_key] + task_space = search_space[task_key] + param_names = [ + name + for name, values in task_space.items() + if name != "category" and isinstance(values, list) + ] + + if not param_names: + continue + + if getattr(task, "config_spec", None) is None: + continue + + profile = profiles.get(task.name) + if profile is None: + return False, f"missing profile for task {task.name!r}" + + bindings = profile.get("bindings") + if not isinstance(bindings, list): + return False, f"profile for {task.name!r} must have bindings list" + + binding_map: Dict[str, Any] = {} + for binding in bindings: + if not isinstance(binding, dict): + return False, f"invalid binding entry for {task.name!r}" + param = binding.get("parameter") + value = binding.get("value") + if not isinstance(param, str): + return False, f"binding parameter must be a string for {task.name!r}" + binding_map[param] = value + + for param_name in param_names: + allowed = task_space[param_name] + if param_name not in binding_map: + return False, f"missing parameter {param_name!r} for task {task_key!r}" + if binding_map[param_name] not in allowed: + return False, ( + f"invalid value for {task_key!r}.{param_name}: " + f"{binding_map[param_name]!r} not in {allowed!r}" + ) + + extra = set(binding_map) - set(param_names) + if extra: + return False, f"unexpected parameters for {task_key!r}: {sorted(extra)}" + + try: + pipeline_config_from_snapshot(snapshot) + except Exception as exc: # noqa: BLE001 - surface parse errors to caller + return False, f"failed to build pipeline config: {exc}" + + return True, "" + + +def search_space_description( + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, +) -> Dict[str, Any]: + """Serialize search space and layout for LLM prompts.""" + tasks: Dict[str, Any] = {} + for task_key, task_space in search_space.items(): + entry: Dict[str, Any] = {"category": task_space.get("category")} + for name, values in task_space.items(): + if name == "category": + continue + if isinstance(values, list): + entry[name] = values + tasks[task_key] = entry + + valid_combos = enumerate_valid_task_combinations(search_space, pipeline_layout) + return { + "pipeline_layout": { + "allowed_task_categories": pipeline_layout.allowed_task_categories, + }, + "tasks": tasks, + "valid_task_combinations": valid_combos, + "output_schema": { + "task_keys": ["", "..."], + "profiles": { + "": { + "profile_name": "", + "bindings": [{"parameter": "", "value": ""}], + } + }, + }, + } diff --git a/experiments/param-opti/src/kgpipe_search/strategies/strategies.py b/experiments/param-opti/src/kgpipe_search/strategies/strategies.py new file mode 100644 index 0000000..ec5fd81 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/strategies/strategies.py @@ -0,0 +1,826 @@ +import math +import random +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Literal, Optional, Set, Tuple + +from kgpipe.common.model.configuration import ConfigurationProfile, ParameterBinding +from kgpipe_search.configuration import ( + build_pipeline_config_for_task_combo, + enumerate_exhaustive_pipeline_configs, + enumerate_valid_task_combinations, + pipeline_config_snapshot_key, + sample_valid_pipeline_config, + task_keys_from_pipeline_config, +) +from kgpipe_search.definitions import PipelineConfig, PipelineLayout +from kgpipe_search.strategies.initialization import ( + implementation_aware_initialization, + random_initialization, +) + +Observation = Tuple[float, PipelineConfig] +EvaluateFn = Callable[[PipelineConfig], float] + +SearchStrategy = Literal["random", "implementation_aware", "qgns", "hnr", "bayesian", "llm"] + + +@dataclass +class SearchRun: + strategy: SearchStrategy + history: List[Observation] + budget: int + decisions: List[str] + + +def _top_k(history: List[Observation], k: int) -> List[Observation]: + ranked = sorted(history, key=lambda item: item[0], reverse=True) + return ranked[: max(1, min(k, len(ranked)))] + + +def _parameter_neighbors( + anchor: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], +) -> List[PipelineConfig]: + anchor_keys = task_keys_from_pipeline_config(anchor) + neighbors: List[PipelineConfig] = [] + + for task, task_key in zip(anchor.tasks, anchor_keys): + profile = anchor.config_catalog.get(task.name) + if profile is None: + continue + + for binding in profile.bindings: + param_name = binding.parameter.name + domain = search_space.get(task_key, {}).get(param_name) + if not isinstance(domain, list): + continue + + for value in domain: + if value == binding.value: + continue + + new_catalog = dict(anchor.config_catalog) + new_bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + for current in profile.bindings: + chosen = value if current.parameter.name == param_name else current.value + new_bindings.append( + ParameterBinding(parameter=current.parameter, value=chosen) + ) + name_parts.append(f"{current.parameter.name}={chosen}") + + new_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=profile.definition, + bindings=new_bindings, + ) + neighbors.append( + PipelineConfig(tasks=list(anchor.tasks), config_catalog=new_catalog) + ) + + return neighbors + + +def _implementation_neighbors( + anchor: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + rng: random.Random, +) -> List[PipelineConfig]: + anchor_keys = task_keys_from_pipeline_config(anchor) + neighbors: List[PipelineConfig] = [] + + for combo in enumerate_valid_task_combinations(search_space, pipeline_layout): + if len(combo) != len(anchor_keys): + continue + if sum(left != right for left, right in zip(anchor_keys, combo)) != 1: + continue + neighbors.append( + build_pipeline_config_for_task_combo( + search_space, + combo, + rng=rng, + template=anchor, + ) + ) + + return neighbors + + +def neighbors_at_distance_one( + anchor: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + rng: random.Random, +) -> List[PipelineConfig]: + seen: Set[str] = set() + neighbors: List[PipelineConfig] = [] + + for candidate in ( + _parameter_neighbors(anchor, search_space) + + _implementation_neighbors(anchor, search_space, pipeline_layout, rng) + ): + key = pipeline_config_snapshot_key(candidate, search_space) + if key in seen: + continue + seen.add(key) + neighbors.append(candidate) + + return neighbors + + +def _restricted_implementation_neighbors_for_index( + anchor: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + rng: random.Random, + *, + index: int, +) -> List[PipelineConfig]: + anchor_keys = task_keys_from_pipeline_config(anchor) + if index < 0 or index >= len(anchor_keys): + return [] + + neighbors: List[PipelineConfig] = [] + for combo in enumerate_valid_task_combinations(search_space, pipeline_layout): + if len(combo) != len(anchor_keys): + continue + if any(i != index and combo[i] != anchor_keys[i] for i in range(len(anchor_keys))): + continue + if combo[index] == anchor_keys[index]: + continue + neighbors.append( + build_pipeline_config_for_task_combo( + search_space, + combo, + rng=rng, + template=anchor, + ) + ) + return neighbors + + +def _restricted_parameter_neighbors_for_index( + anchor: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], + *, + index: int, +) -> List[PipelineConfig]: + anchor_keys = task_keys_from_pipeline_config(anchor) + if index < 0 or index >= len(anchor.tasks) or index >= len(anchor_keys): + return [] + + task = anchor.tasks[index] + task_key = anchor_keys[index] + profile = anchor.config_catalog.get(task.name) + if profile is None: + return [] + + neighbors: List[PipelineConfig] = [] + for binding in profile.bindings: + param_name = binding.parameter.name + domain = search_space.get(task_key, {}).get(param_name) + if not isinstance(domain, list): + continue + + for value in domain: + if value == binding.value: + continue + + new_catalog = dict(anchor.config_catalog) + new_bindings: List[ParameterBinding] = [] + name_parts: List[str] = [] + for current in profile.bindings: + chosen = value if current.parameter.name == param_name else current.value + new_bindings.append( + ParameterBinding(parameter=current.parameter, value=chosen) + ) + name_parts.append(f"{current.parameter.name}={chosen}") + + new_catalog[task.name] = ConfigurationProfile( + name=f"{task.name}_" + ",".join(name_parts), + definition=profile.definition, + bindings=new_bindings, + ) + neighbors.append( + PipelineConfig(tasks=list(anchor.tasks), config_catalog=new_catalog) + ) + + return neighbors + + +def sample_unevaluated_config( + rng: random.Random, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + evaluated_keys: Set[str], + *, + max_attempts: int = 500, +) -> PipelineConfig: + for _ in range(max_attempts): + candidate = sample_valid_pipeline_config( + search_space, + pipeline_layout, + rng=rng, + ) + key = pipeline_config_snapshot_key(candidate, search_space) + if key not in evaluated_keys: + return candidate + + raise RuntimeError("Failed to sample an unevaluated configuration") + + +def _config_distance( + left: PipelineConfig, + right: PipelineConfig, + search_space: Dict[str, Dict[str, Any]], +) -> float: + if pipeline_config_snapshot_key(left, search_space) == pipeline_config_snapshot_key( + right, search_space + ): + return 0.0 + + left_keys = task_keys_from_pipeline_config(left) + right_keys = task_keys_from_pipeline_config(right) + distance = float(sum(a != b for a, b in zip(left_keys, right_keys))) + if len(left_keys) != len(right_keys): + distance += abs(len(left_keys) - len(right_keys)) + + left_params = { + (task.name, binding.parameter.name): binding.value + for task in left.tasks + for binding in ( + left.config_catalog.get(task.name).bindings + if left.config_catalog.get(task.name) + else [] + ) + } + right_params = { + (task.name, binding.parameter.name): binding.value + for task in right.tasks + for binding in ( + right.config_catalog.get(task.name).bindings + if right.config_catalog.get(task.name) + else [] + ) + } + + all_param_keys = set(left_params) | set(right_params) + for key in all_param_keys: + if left_params.get(key) != right_params.get(key): + distance += 1.0 + + return distance + + +def _predict_with_uncertainty( + candidate: PipelineConfig, + history: List[Observation], + search_space: Dict[str, Dict[str, Any]], +) -> Tuple[float, float]: + weights: List[float] = [] + scores: List[float] = [] + + for score, observed in history: + distance = _config_distance(candidate, observed, search_space) + if distance == 0.0: + return score, 0.0 + weights.append(math.exp(-distance)) + scores.append(score) + + if not weights: + return 0.75, 1.0 + + total_weight = sum(weights) + mean = sum(score * weight for score, weight in zip(scores, weights)) / total_weight + uncertainty = 1.0 / (1.0 + total_weight) + return mean, uncertainty + + +def _acquisition(mean: float, uncertainty: float, *, beta: float = 0.5) -> float: + return mean + beta * uncertainty + + +def run_random( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + rng: Optional[random.Random] = None, +) -> SearchRun: + """ + Uniform random search over the exhaustive valid config set. + + Enumerates every valid (task combo × parameter) config, then samples + ``budget`` distinct configs without replacement. Unlike hierarchical + sampling (task first, then params), each leaf config is equally likely. + """ + draw = rng or random.Random() + if budget <= 0: + return SearchRun(strategy="random", history=[], budget=0, decisions=[]) + + all_configs = enumerate_exhaustive_pipeline_configs(search_space, pipeline_layout) + if not all_configs: + raise RuntimeError("Exhaustive config enumeration produced no valid configs") + if budget > len(all_configs): + raise ValueError( + f"budget={budget} exceeds exhaustive search space size ({len(all_configs)})" + ) + + selected = draw.sample(all_configs, k=budget) + history: List[Observation] = [] + decisions: List[str] = [] + for candidate in selected: + score = evaluate_fn(candidate) + history.append((score, candidate)) + decisions.append("sample") + + return SearchRun(strategy="random", history=history, budget=budget, decisions=decisions) + + +def run_implementation_aware( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + y: int = 1, + rng: Optional[random.Random] = None, +) -> SearchRun: + """ + Evaluate `budget` configs from implementation-aware initialization. + + Task combinations are covered systematically (`y` random parameter samples per combo). + Any remaining budget is filled with uniform random valid configs. + """ + if budget <= 0: + return SearchRun( + strategy="implementation_aware", + history=[], + budget=0, + decisions=[], + ) + + draw = rng or random.Random() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + init_set = implementation_aware_initialization( + search_space, + pipeline_layout, + budget=budget, + y=y, + rng=draw, + ) + + for cfg in init_set: + if len(history) >= budget: + break + key = pipeline_config_snapshot_key(cfg, search_space) + if key in evaluated_keys: + continue + score = evaluate_fn(cfg) + history.append((score, cfg)) + evaluated_keys.add(key) + decisions.append("init(implementation_aware)") + + while len(history) < budget: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append("sample") + + return SearchRun( + strategy="implementation_aware", + history=history, + budget=budget, + decisions=decisions, + ) + + +def run_qgns( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int = 0, + init_strategy: Literal["random", "implementation_aware"] = "random", + y: int = 1, + k: int = 3, + rho: float = 0.2, + rng: Optional[random.Random] = None, +) -> SearchRun: + if budget <= 0: + return SearchRun(strategy="qgns", history=[], budget=0, decisions=[]) + + draw = rng or random.Random() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + if init_budget > 0: + if init_strategy == "implementation_aware": + init_set = implementation_aware_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + y=y, + rng=draw, + ) + else: + init_set = random_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + rng=draw, + ) + for cfg in init_set: + key = pipeline_config_snapshot_key(cfg, search_space) + if key in evaluated_keys: + continue + score = evaluate_fn(cfg) + history.append((score, cfg)) + evaluated_keys.add(key) + decisions.append(f"init({init_strategy})") + if len(history) >= budget: + return SearchRun(strategy="qgns", history=history, budget=budget, decisions=decisions) + + while len(history) < budget: + if not history or draw.random() < rho: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + decision = "explore" + else: + anchors = _top_k(history, k) + candidate = None + decision = "explore(fallback)" + + shuffled = list(anchors) + draw.shuffle(shuffled) + for anchor_score, anchor_cfg in shuffled: + neighborhood = neighbors_at_distance_one( + anchor_cfg, search_space, pipeline_layout, draw + ) + unevaluated = [ + n + for n in neighborhood + if pipeline_config_snapshot_key(n, search_space) not in evaluated_keys + ] + if not unevaluated: + continue + candidate = draw.choice(unevaluated) + decision = f"neighborhood(anchor_score={anchor_score:.4f})" + break + + if candidate is None: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append(decision) + + return SearchRun(strategy="qgns", history=history, budget=budget, decisions=decisions) + + +def run_hnr( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int, + init_strategy: Literal["random", "implementation_aware"] = "implementation_aware", + y: int = 1, + rho: float = 0.0, + rng: Optional[random.Random] = None, +) -> SearchRun: + rho = 0.0 + print(f"INFO [HNR] rho: {rho}, budget: {budget}, init_budget: {init_budget}, init_strategy: {init_strategy}, y: {y}") + if budget <= 0: + return SearchRun(strategy="hnr", history=[], budget=0, decisions=[]) + if init_budget <= 0: + raise ValueError("HNR requires init_budget > 0") + + draw = rng or random.Random() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + if init_strategy == "implementation_aware": + init_set = implementation_aware_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + y=y, + rng=draw, + ) + else: + init_set = random_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + rng=draw, + ) + + for cfg in init_set: + key = pipeline_config_snapshot_key(cfg, search_space) + if key in evaluated_keys: + continue + score = evaluate_fn(cfg) + history.append((score, cfg)) + evaluated_keys.add(key) + decisions.append(f"init({init_strategy})") + if len(history) >= budget: + return SearchRun(strategy="hnr", history=history, budget=budget, decisions=decisions) + + best_score, best_cfg = max(history, key=lambda item: item[0]) + + while len(history) < budget: + improved = False + + for idx in range(len(best_cfg.tasks)): + if len(history) >= budget: + break + + if draw.random() < rho: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + decision = f"explore(task_idx={idx})" + else: + task_neighbors = _restricted_implementation_neighbors_for_index( + best_cfg, search_space, pipeline_layout, draw, index=idx + ) + task_candidates = [ + n + for n in task_neighbors + if pipeline_config_snapshot_key(n, search_space) not in evaluated_keys + ] + + if task_candidates: + candidate = draw.choice(task_candidates) + decision = f"task_neighbor(idx={idx})" + else: + param_neighbors = _restricted_parameter_neighbors_for_index( + best_cfg, search_space, index=idx + ) + param_candidates = [ + n + for n in param_neighbors + if pipeline_config_snapshot_key(n, search_space) not in evaluated_keys + ] + if param_candidates: + candidate = draw.choice(param_candidates) + decision = f"param_neighbor(idx={idx})" + else: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + decision = f"explore(fallback,idx={idx})" + + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append(decision) + + print(f"INFO [HNR] score: {score}, best_score: {best_score}, task_idx: {idx}") + if score > best_score: + best_score, best_cfg = score, candidate + improved = True + + if not improved and len(history) < budget and draw.random() < rho: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append("explore(post_sweep)") + if score > best_score: + best_score, best_cfg = score, candidate + + return SearchRun(strategy="hnr", history=history, budget=budget, decisions=decisions) + +def run_hnr_2( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int, + init_strategy: Literal["random", "implementation_aware"] = "implementation_aware", + y: int = 1, + rho: float = 0.0, + min_quality_delta = 0.003, + min_iterations_wo_improvement = 2, + rng: Optional[random.Random] = None, +) -> SearchRun: + rho = 0.0 + print(f"INFO [HNR-2] budget: {budget}, init_budget: {init_budget}, init_strategy: {init_strategy}, y: {y}, rho: {rho}, min_quality_delta: {min_quality_delta}, min_iterations_wo_improvement: {min_iterations_wo_improvement}") + if budget <= 0: + return SearchRun(strategy="hnr_2", history=[], budget=0, decisions=[]) + if init_budget <= 0: + raise ValueError("HNR requires init_budget > 0") + + draw = rng or random.Random() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + if init_strategy == "implementation_aware": + init_set = implementation_aware_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + y=y, + rng=draw, + ) + else: + init_set = random_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + rng=draw, + ) + + for cfg in init_set: + key = pipeline_config_snapshot_key(cfg, search_space) + if key in evaluated_keys: + continue + score = evaluate_fn(cfg) + history.append((score, cfg)) + evaluated_keys.add(key) + decisions.append(f"init({init_strategy})") + if len(history) >= budget: + return SearchRun(strategy="hnr_2", history=history, budget=budget, decisions=decisions) + + best_score, best_cfg = max(history, key=lambda item: item[0]) + current_task_index = 0 + quality_delta = 0 + iterations_wo_improvement = 0 + while len(history) < budget: + improved = False + if len(history) >= budget: + break + task_neighbors = _restricted_implementation_neighbors_for_index( + best_cfg, search_space, pipeline_layout, draw, index=current_task_index + ) + task_candidates = [ + n + for n in task_neighbors + if pipeline_config_snapshot_key(n, search_space) not in evaluated_keys + ] + + if task_candidates: + candidate = draw.choice(task_candidates) + decision = f"task_neighbor(idx={current_task_index})" + else: + param_neighbors = _restricted_parameter_neighbors_for_index( + best_cfg, search_space, index=current_task_index + ) + param_candidates = [ + n + for n in param_neighbors + if pipeline_config_snapshot_key(n, search_space) not in evaluated_keys + ] + if param_candidates: + candidate = draw.choice(param_candidates) + decision = f"param_neighbor(idx={current_task_index})" + else: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + decision = f"explore(fallback,idx={current_task_index})" + + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append(decision) + quality_delta = score - best_score # current quality delta + + print(f"INFO [HNR_2] score: {score}, best_score: {best_score}, task_idx: {current_task_index}") + if score > best_score: + best_score, best_cfg = score, candidate + improved = True + if quality_delta < min_quality_delta: # if the delta is below the required min quality delta consider this + # run as no improvement + iterations_wo_improvement += 1 + else: + iterations_wo_improvement = 0 + if iterations_wo_improvement >= min_iterations_wo_improvement: # if the number of no improvements we consider the next task + if current_task_index < len(best_cfg.tasks): + current_task_index += 1 + iterations_wo_improvement = 0 + else: + # We are not able to improve the last task anymore. Therefore, we can also stop the runs. + pass # TODO: implement a proper stopping criterion + if not improved and len(history) < budget and draw.random() < rho: + candidate = sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + key = pipeline_config_snapshot_key(candidate, search_space) + score = evaluate_fn(candidate) + history.append((score, candidate)) + evaluated_keys.add(key) + decisions.append("explore(post_sweep)") + if score > best_score: + best_score, best_cfg = score, candidate + + return SearchRun(strategy="hnr", history=history, budget=budget, decisions=decisions) + +def run_bayesian( + *, + budget: int, + evaluate_fn: EvaluateFn, + search_space: Dict[str, Dict[str, Any]], + pipeline_layout: PipelineLayout, + init_budget: int = 3, + init_strategy: Literal["random", "implementation_aware"] = "random", + y: int = 1, + pool_size: int = 32, + beta: float = 0.5, + rng: Optional[random.Random] = None, +) -> SearchRun: + if budget <= 0: + return SearchRun(strategy="bayesian", history=[], budget=0, decisions=[]) + + draw = rng or random.Random() + history: List[Observation] = [] + decisions: List[str] = [] + evaluated_keys: Set[str] = set() + + if init_budget > 0: + if init_strategy == "implementation_aware": + init_set = implementation_aware_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + y=y, + rng=draw, + ) + else: + init_set = random_initialization( + search_space, + pipeline_layout, + budget=min(init_budget, budget), + rng=draw, + ) + for cfg in init_set: + key = pipeline_config_snapshot_key(cfg, search_space) + if key in evaluated_keys: + continue + score = evaluate_fn(cfg) + history.append((score, cfg)) + evaluated_keys.add(key) + decisions.append(f"init({init_strategy})") + if len(history) >= budget: + return SearchRun(strategy="bayesian", history=history, budget=budget, decisions=decisions) + + while len(history) < budget: + candidates: List[PipelineConfig] = [] + for _ in range(pool_size): + candidates.append( + sample_unevaluated_config( + draw, search_space, pipeline_layout, evaluated_keys + ) + ) + + best_candidate = candidates[0] + best_acq = float("-inf") + best_pred = 0.0 + best_unc = 0.0 + + for candidate in candidates: + mean, unc = _predict_with_uncertainty(candidate, history, search_space) + acq = _acquisition(mean, unc, beta=beta) + if acq > best_acq: + best_acq = acq + best_candidate = candidate + best_pred = mean + best_unc = unc + + key = pipeline_config_snapshot_key(best_candidate, search_space) + score = evaluate_fn(best_candidate) + history.append((score, best_candidate)) + evaluated_keys.add(key) + decisions.append( + f"acquisition(pred={best_pred:.4f},unc={best_unc:.4f},a={best_acq:.4f})" + ) + + return SearchRun(strategy="bayesian", history=history, budget=budget, decisions=decisions) + diff --git a/experiments/param-opti/src/kgpipe_search/test/__init__.py b/experiments/param-opti/src/kgpipe_search/test/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/experiments/param-opti/src/kgpipe_search/test/conftest.py b/experiments/param-opti/src/kgpipe_search/test/conftest.py new file mode 100644 index 0000000..540da02 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/conftest.py @@ -0,0 +1,31 @@ +import sys +import types +from importlib import import_module + + +def _install_param_opti_shim() -> None: + if "param_opti" in sys.modules: + return + + param_opti = types.ModuleType("param_opti") + tasks = types.ModuleType("param_opti.tasks") + + for lib in ( + "base_linker_lib", + "base_matcher_lib", + "paris_lib", + "fusion_lib", + "spotlight_lib", + "corenlp_lip", + "genie_lib", + ): + module = import_module(f"kgpipe_search.dev.tasks.{lib}") + setattr(tasks, lib, module) + sys.modules[f"param_opti.tasks.{lib}"] = module + + param_opti.tasks = tasks + sys.modules["param_opti"] = param_opti + sys.modules["param_opti.tasks"] = tasks + + +_install_param_opti_shim() diff --git a/experiments/param-opti/src/kgpipe_search/test/test_configuration.py b/experiments/param-opti/src/kgpipe_search/test/test_configuration.py new file mode 100644 index 0000000..38b0932 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_configuration.py @@ -0,0 +1,266 @@ +from kgpipe_search.definitions import PipelineLayout, RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT, TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT +from kgpipe_search.configuration import ( + sample_valid_pipeline_config, + enumerate_valid_task_combinations, sample_config_catalog_for_task_combo, enumerate_exhaustive_pipeline_config_snapshots, pipeline_config_to_snapshot, + print_pipeline_config_short, + sample_unique_pipeline_config_snapshots_per_combo, +) +from kgpipe_search.definitions import RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE, _RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION +from kgpipe_search.definitions import RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE, _RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION +from kgpipe_search.definitions import RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE, _RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION +from kgpipe_search.definitions import TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE, _TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION +from kgpipe_search.definitions import TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE, _TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION +from kgpipe_search.definitions import TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE, _TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION +import json + +def test_sample_valid_rdf_pipeline_config(): + pipeline_layout = PipelineLayout( + allowed_task_categories=["ontology_matching", "entity_matching", "aggregate_matching_results", "fusion"] + ) + pipeline_config = sample_valid_pipeline_config(RDF_SEARCH_SPACE, pipeline_layout) + print_pipeline_config_short(pipeline_config) + +def test_enumerate_all_valid_rdf_task_combinations_no_config_sampling(): + print("enumerate_all_valid_rdf_task_combinations_no_config_sampling") + combos = enumerate_valid_task_combinations(RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT) + + for combo in combos: + print(combo) + + # With current SEARCH_SPACE: + # - ontology_matching can be satisfied by paris_ontology_matching_task, paris_entity_alignment_task, paris_graph_alignment_task + # - entity_matching can be satisfied by paris_entity_alignment_task, paris_graph_alignment_task (and may be skipped if already covered) + # - fusion must be satisfied by fusion_first_value_task + # expected = { + # ("paris_ontology_matching_task", "paris_entity_alignment_task", "fusion_first_value_task"), + # ("paris_ontology_matching_task", "paris_graph_alignment_task", "fusion_first_value_task"), + # ("paris_graph_alignment_task", "fusion_first_value_task"), + # } + + # assert set(tuple(c) for c in combos) == expected + + +import random +from typing import List, Dict, Any + + +def _print_unique_sampling_stats(stats: Dict[str, Any]) -> None: + print() + print("unique sampling statistics") + print(f"requested n per combo: {stats['requested_n']}") + print(f"total combos: {stats['total_combos']}") + print(f"total snapshots: {stats['total_snapshots']}") + print(f"combos exhausted before n: {stats['combos_exhausted']}") + for row in stats["combos"]: + status = "EXHAUSTED" if row["exhausted"] else "ok" + print( + f" {row['task_keys']}: sampled {row['sampled']}/{row['requested']} " + f"(available {row['available_profiles']}) [{status}]" + ) + + +def test_enumerate_all_valid_rdf_task_combinations_with_config_sampling(): + print("enumerate_all_valid_rdf_task_combinations_with_config_sampling") + n = 1 + rng = random.Random(0) + + combos = enumerate_valid_task_combinations(RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT) + + total_config_count = 0 + snapshots: List[Dict[str, Any]] = [] + + for combo in combos: + print() + print("combo:", combo) + for i in range(n): + total_config_count += 1 + print(f"sample {total_config_count}/{len(combos) * n}") + pipeline_config = sample_config_catalog_for_task_combo( + RDF_SEARCH_SPACE, combo, rng=rng + ) + + print_pipeline_config_short(pipeline_config) + snapshots.append(pipeline_config_to_snapshot(combo, pipeline_config)) + + RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + RDF_SAMPLED_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _RDF_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def test_enumerate_all_valid_rdf_task_combinations_with_unique_config_sampling(): + print("enumerate_all_valid_rdf_task_combinations_with_unique_config_sampling") + n = 10 + rng = random.Random(0) + + snapshots, stats = sample_unique_pipeline_config_snapshots_per_combo( + RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT, n=n, rng=rng + ) + _print_unique_sampling_stats(stats) + + for combo_row in stats["combos"]: + combo_task_keys = combo_row["task_keys"] + combo_snapshots = [s for s in snapshots if s["task_keys"] == combo_task_keys] + serialized = [json.dumps(s, sort_keys=True) for s in combo_snapshots] + assert len(set(serialized)) == len(serialized) + assert len(serialized) == combo_row["sampled"] + + RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + RDF_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _RDF_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def test_sample_valid_text_pipeline_config(): + pipeline_config = sample_valid_pipeline_config(TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT) + print_pipeline_config_short(pipeline_config) + + +def test_enumerate_all_valid_text_task_combinations_no_config_sampling(): + print("enumerate_all_valid_text_task_combinations_no_config_sampling") + combos = enumerate_valid_task_combinations(TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT) + for combo in combos: + print(combo) + +def test_enumerate_all_valid_text_task_combinations_with_config_sampling(): + print("enumerate_all_valid_text_task_combinations_with_config_sampling") + n = 1 + rng = random.Random(0) + + combos = enumerate_valid_task_combinations(TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT) + + total_config_count = 0 + snapshots: List[Dict[str, Any]] = [] + + for combo in combos: + print() + print("combo:", combo) + for i in range(n): + total_config_count += 1 + print(f"sample {total_config_count}/{len(combos) * n}") + pipeline_config = sample_config_catalog_for_task_combo( + TEXT_SEARCH_SPACE, combo, rng=rng + ) + print_pipeline_config_short(pipeline_config) + snapshots.append(pipeline_config_to_snapshot(combo, pipeline_config)) + + TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + TEXT_SAMPLED_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _TEXT_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def test_enumerate_all_valid_text_task_combinations_with_unique_config_sampling(): + print("enumerate_all_valid_text_task_combinations_with_unique_config_sampling") + n = 3 + rng = random.Random(0) + + snapshots, stats = sample_unique_pipeline_config_snapshots_per_combo( + TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT, n=n, rng=rng + ) + _print_unique_sampling_stats(stats) + + for combo_row in stats["combos"]: + combo_task_keys = combo_row["task_keys"] + combo_snapshots = [s for s in snapshots if s["task_keys"] == combo_task_keys] + serialized = [json.dumps(s, sort_keys=True) for s in combo_snapshots] + assert len(set(serialized)) == len(serialized) + assert len(serialized) == combo_row["sampled"] + + TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + TEXT_UNIQUE_SAMPLED_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _TEXT_UNIQUE_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def test_enumerate_all_valid_text_task_combinations_with_config_sampling_exhaustive(): + print("enumerate_all_valid_text_task_combinations_with_config_sampling_exhaustive") + all_snapshots = enumerate_exhaustive_pipeline_config_snapshots( + TEXT_SEARCH_SPACE, TEXT_PIPELINE_LAYOUT + ) + serialized = [json.dumps(s, sort_keys=True) for s in all_snapshots] + assert len(set(serialized)) == len(serialized) + + TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + TEXT_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _TEXT_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": all_snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + +def test_enumerate_all_valid_rdf_task_combinations_with_config_sampling_exhaustive(): + print("enumerate_all_valid_rdf_task_combinations_with_config_sampling_exhaustive") + all_snapshots = enumerate_exhaustive_pipeline_config_snapshots( + RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT + ) + serialized = [json.dumps(s, sort_keys=True) for s in all_snapshots] + assert len(set(serialized)) == len(serialized) + + RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE.parent.mkdir(parents=True, exist_ok=True) + RDF_EXHAUSTIVE_PIPELINE_CONFIGS_FIXTURE.write_text( + json.dumps( + {"version": _RDF_EXHAUSTIVE_PIPELINE_CONFIG_SNAPSHOT_VERSION, "samples": all_snapshots}, + indent=2, + sort_keys=True, + ) + + "\n", + encoding="utf-8", + ) + + + +# def test_rdf_pipeline_from_config(): +# pipeline_config = sample_valid_pipeline_config(RDF_SEARCH_SPACE, PipelineLayout(allowed_task_categories=["entity_matching", "fusion"])) + +# seed_path = tmp_base_dir / "seed.nt" +# source_path = tmp_base_dir / "source.nt" +# result_path = tmp_base_dir / "result.nt" +# tasks_tmp_dir = tmp_base_dir / "tasks_tmp" +# tasks_tmp_dir.mkdir(parents=True, exist_ok=True) + +# # Ensure inputs exist for pipeline execution. +# seed_path.write_text(" .\n") +# source_path.write_text(" .\n") + +# pipeline = KgPipe( +# tasks=pipeline_config.tasks, +# seed=Data(path=seed_path, format=DataFormat.RDF_NTRIPLES), +# data_dir=tasks_tmp_dir, +# name="test_pipeline") + +# pipeline.build( +# stable_files=True, +# configCatalog=pipeline_config.config_catalog, +# source=Data(path=source_path, format=DataFormat.RDF_NTRIPLES), +# result=Data(path=result_path, format=DataFormat.RDF_NTRIPLES)) + +# pipeline.run(configCatalog=pipeline_config.config_catalog, stable_files_override=True) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/test/test_execution.py b/experiments/param-opti/src/kgpipe_search/test/test_execution.py new file mode 100644 index 0000000..a5d1f46 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_execution.py @@ -0,0 +1,140 @@ +import os +import random +from pathlib import Path + +import pytest + +from kgpipe.common import Data, DataFormat, KgPipe +from kgpipe_search.configuration import ( + load_pipeline_config_snapshot, + load_rdf_sampled_pipeline_configs, + pipeline_config_snapshot_key, + save_pipeline_config_snapshot, + sample_valid_pipeline_config, +) +from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE + +KGPIPE_ROOT = Path(__file__).resolve().parents[5] +FALLBACK_TEST_DATA = KGPIPE_ROOT / "src/kgpipe_tasks/test/test_data/rdf" + +tmp_base_dir = Path("data/tmp/rdf_pipelines") +tmp_base_dir.mkdir(parents=True, exist_ok=True) + +SEED_PATH = Path("data/input_final/target_kg/graph.nt") +SOURCE_PATH = Path("data/input_final/rdf_source/graph.nt") +ONTOLOGY_PATH = Path("data/input_final/target_kg/ontology.ttl") +FALLBACK_SEED_PATH = FALLBACK_TEST_DATA / "target.nt" +FALLBACK_SOURCE_PATH = FALLBACK_TEST_DATA / "source.nt" +FALLBACK_ONTOLOGY_PATH = FALLBACK_TEST_DATA / "ontology.ttl" + + +def _ensure_ontology_env() -> None: + if ONTOLOGY_PATH.exists(): + os.environ["ONTOLOGY_PATH"] = str(ONTOLOGY_PATH) + elif FALLBACK_ONTOLOGY_PATH.exists(): + os.environ["ONTOLOGY_PATH"] = str(FALLBACK_ONTOLOGY_PATH) + + +def _rdf_input_paths(tmp_dir: Path) -> tuple[Path, Path]: + if SEED_PATH.exists() and SOURCE_PATH.exists(): + return SEED_PATH, SOURCE_PATH + if FALLBACK_SEED_PATH.exists() and FALLBACK_SOURCE_PATH.exists(): + return FALLBACK_SEED_PATH, FALLBACK_SOURCE_PATH + + seed_path = tmp_dir / "seed.nt" + source_path = tmp_dir / "source.nt" + seed_path.write_text( + " .\n", + encoding="utf-8", + ) + source_path.write_text( + " .\n", + encoding="utf-8", + ) + return seed_path, source_path + + +def _run_rdf_pipeline_config( + pipeline_config, + *, + tmp_dir: Path, + run_name: str, + result_path: Path, +) -> Path: + _ensure_ontology_env() + seed_path, source_path = _rdf_input_paths(tmp_dir) + tasks_tmp_dir = tmp_dir / f"{run_name}_tasks_tmp" + tasks_tmp_dir.mkdir(parents=True, exist_ok=True) + + pipeline = KgPipe( + tasks=pipeline_config.tasks, + seed=Data(path=seed_path, format=DataFormat.RDF_NTRIPLES), + data_dir=tasks_tmp_dir, + name=run_name, + ) + + pipeline.build( + stable_files=True, + configCatalog=pipeline_config.config_catalog, + source=Data(path=source_path, format=DataFormat.RDF_NTRIPLES), + result=Data(path=result_path, format=DataFormat.RDF_NTRIPLES), + ) + + pipeline.run(configCatalog=pipeline_config.config_catalog, stable_files_override=True) + return result_path + + +@pytest.mark.parametrize("config_idx", range(len(load_rdf_sampled_pipeline_configs()))) +def test_rdf_pipeline_from_saved_sampled_configs(config_idx): + """Runs KGpipe using PipelineConfigs materialized from the JSON fixture.""" + configs = load_rdf_sampled_pipeline_configs() + assert configs, ( + "fixtures/rdf_sampled_pipeline_configs.json is missing or empty; " + "run test_enumerate_all_valid_rdf_task_combinations_with_config_sampling" + ) + + pipeline_config = configs[config_idx] + result_path = _run_rdf_pipeline_config( + pipeline_config, + tmp_dir=tmp_base_dir, + run_name=f"saved_sample_config_idx_{config_idx}", + ) + assert result_path.exists() + + +def test_sample_save_load_and_run_pipeline_config(tmp_path: Path): + sampled_config = sample_valid_pipeline_config( + RDF_SEARCH_SPACE, + RDF_PIPELINE_LAYOUT, + rng=random.Random(42), + ) + original_key = pipeline_config_snapshot_key(sampled_config, RDF_SEARCH_SPACE) + + snapshot_path = tmp_path / "sampled_pipeline_config.json" + save_pipeline_config_snapshot(snapshot_path, sampled_config) + assert snapshot_path.exists() + + loaded_config = load_pipeline_config_snapshot(snapshot_path) + loaded_key = pipeline_config_snapshot_key(loaded_config, RDF_SEARCH_SPACE) + assert loaded_key == original_key + + result_path = _run_rdf_pipeline_config( + loaded_config, + tmp_dir=tmp_path, + run_name="sample_save_load_run", + ) + assert result_path.exists() + + +def test_sample_and_run_pipeline_config(tmp_path: Path): + pipeline_config = sample_valid_pipeline_config( + RDF_SEARCH_SPACE, + RDF_PIPELINE_LAYOUT, + rng=random.Random(43), + ) + result_path = _run_rdf_pipeline_config( + pipeline_config, + tmp_dir=tmp_path, + run_name="sample_and_run", + ) + assert result_path.exists() diff --git a/experiments/param-opti/src/kgpipe_search/test/test_execution_docker.py b/experiments/param-opti/src/kgpipe_search/test/test_execution_docker.py new file mode 100644 index 0000000..a661d13 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_execution_docker.py @@ -0,0 +1,476 @@ +from concurrent.futures import ThreadPoolExecutor, as_completed +import os +import uuid +from pathlib import Path + +import pytest +from docker import DockerClient + +from kgpipe_search.mounts import DEFAULT_SCRATCH_HOST, ScratchMount + + +def _swarm_available() -> tuple[bool, str]: + try: + client = DockerClient.from_env() + info = client.info() + except Exception as e: + return False, f"Docker engine not reachable: {e}" + + swarm = info.get("Swarm") or {} + state = (swarm.get("LocalNodeState") or "").lower() + if state != "active": + return False, f"Docker Swarm not active (LocalNodeState={swarm.get('LocalNodeState')!r})" + + return True, "ok" + + +_TEST_COMMAND = [ + "python", + "-c", + "import os, json, time; " + "time.sleep(30); " + "p=float(os.environ.get('KGPIPE_PARAM','0')); " + "print(json.dumps({'result': p + 1.0}))", +] + + +def test_execute_pipeline_docker_swarm_parses_result(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + from kgpipe_search.execution import execute_pipeline_docker_swarm + + # The container prints {"result": float(KGPIPE_PARAM) + 1.0} + param = "1.5" + expected = 2.5 + + result = execute_pipeline_docker_swarm( + kg="dummy", + source="dummy", + config={ + "image": "python:3.12-slim", + "command": _TEST_COMMAND, + "parameter": param, + "max_per_node": 1, + "timeout_s": 120, + }, + ) + + assert result == expected + + +def test_extract_job_result_formats(): + from kgpipe_search.swarm import ResultSpec, SwarmJobResult, extract_job_result + + job = SwarmJobResult( + service_id="svc", + service_name="svc-name", + run_name="run-1", + node_id="node-1", + state="complete", + exit_code=0, + logs='info line\n{"result": 2.5, "label": "ok"}\n', + ) + + assert extract_job_result(job, ResultSpec(format="float")) == 2.5 + assert extract_job_result(job, ResultSpec(format="json", json_key="label")) == "ok" + assert extract_job_result(job, ResultSpec(format="json", json_key=None)) == { + "result": 2.5, + "label": "ok", + } + assert extract_job_result(job, ResultSpec(format="logs")) == job.logs + assert extract_job_result(job, ResultSpec(format="exit_code")) == 0 + + failed = SwarmJobResult( + service_id="svc", + service_name="svc-name", + run_name="run-1", + node_id="node-1", + state="complete", + exit_code=7, + logs="", + ) + assert extract_job_result(failed, ResultSpec(format="exit_code", require_exit_code=None)) == 7 + + +def test_execute_pipeline_docker_swarm_exit_code_result(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + from kgpipe_search.execution import execute_pipeline_docker_swarm + + result = execute_pipeline_docker_swarm( + kg="dummy", + source="dummy", + config={ + "image": "python:3.12-slim", + "command": ["python", "-c", "import sys; sys.exit(42)"], + "result_format": "exit_code", + "require_exit_code": 42, + "max_per_node": 1, + "timeout_s": 120, + }, + ) + + assert result == 42 + + +def test_execute_pipeline_docker_swarm_json_result(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + from kgpipe_search.execution import execute_pipeline_docker_swarm + + result = execute_pipeline_docker_swarm( + kg="dummy", + source="dummy", + config={ + "image": "python:3.12-slim", + "command": [ + "python", + "-c", + "import json; print(json.dumps({'metrics': {'f1': 0.9}, 'result': 0.9}))", + ], + "result_format": "json", + "result_key": "metrics", + "max_per_node": 1, + "timeout_s": 120, + }, + ) + + assert result == {"f1": 0.9} + + +_SCRATCH_WRITE_COMMAND = [ + "python", + "-c", + "import json, os, pathlib; " + "root = pathlib.Path(os.environ['KGPIPE_SCRATCH']) / os.environ['KGPIPE_RUN_ID']; " + "root.mkdir(parents=True, exist_ok=True); " + "(root / 'done.txt').write_text('ok'); " + "print(json.dumps({'result': str(root / 'done.txt')}))", +] + +_SCRATCH_VERIFY_COMMAND = [ + "python", + "-c", + "import os, pathlib, sys; " + "p = pathlib.Path(os.environ['KGPIPE_SCRATCH']) / os.environ['KGPIPE_RUN_ID'] / 'done.txt'; " + "sys.exit(0 if p.is_file() and p.read_text() == 'ok' else 1)", +] + + +def test_swarm_scratch_mount_writes_per_run_file(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + scratch_host = Path(os.environ.get("KGPIPE_SCRATCH_HOST", DEFAULT_SCRATCH_HOST)) + if not scratch_host.is_dir(): + pytest.skip(f"scratch host path does not exist: {scratch_host}") + + from kgpipe_search.swarm import ResultSpec, SwarmManager, extract_job_result + + mgr = SwarmManager() + run_name = f"pytest-{uuid.uuid4().hex[:8]}" + scratch = ScratchMount(host_path=str(scratch_host)) + + res = mgr.run_job( + image="python:3.12-slim", + command=_SCRATCH_WRITE_COMMAND, + run_name=run_name, + scratch=scratch, + max_per_node=1, + timeout_s=120, + name_prefix="kgpipe-scratch", + ) + + assert res.state == "complete" + assert res.exit_code == 0 + assert res.run_name == run_name + assert extract_job_result(res, ResultSpec(format="json", json_key="result")).endswith("done.txt") + + out_path = scratch_host / run_name / "done.txt" + if out_path.is_file(): + assert out_path.read_text() == "ok" + return + + assert res.node_id is not None + verify = mgr.run_job( + image="python:3.12-slim", + command=_SCRATCH_VERIFY_COMMAND, + run_name=run_name, + scratch=scratch, + node_id=res.node_id, + max_per_node=1, + timeout_s=120, + name_prefix="kgpipe-scratch-verify", + ) + assert verify.state == "complete" + assert verify.exit_code == 0 + + +def test_hdfs_copy_strategy_builds_put_command_with_hadoop_conf(): + from kgpipe_search.copy import CopyContext, CopyTarget, HdfsCopyStrategy + + strategy = HdfsCopyStrategy(hadoop_conf_host="/etc/hadoop/conf") + plan = strategy.build_job( + context=CopyContext( + run_name="run-42", + node_id="node-1", + scratch=ScratchMount( + host_path="/local/d1/docker-scratch", + container_path="/local/d1/docker-scratch", + ), + ), + destination=CopyTarget(path="/user/kgpipe/results"), + ) + + assert plan.image == "apache/hadoop:3.3.6" + assert "hdfs dfs -put" in plan.command[-1] + assert "/user/kgpipe/results/run-42" in plan.command[-1] + assert plan.env["KGPIPE_HDFS_DEST"] == "/user/kgpipe/results/run-42" + assert plan.env["HADOOP_CONF_DIR"] == "/etc/hadoop/conf" + assert any(m.get("Source") == "/local/d1/docker-scratch" for m in plan.mounts) + assert any(m.get("Source") == "/etc/hadoop/conf" for m in plan.mounts) + + +def test_hdfs_copy_strategy_builds_put_command_with_namenode_and_user(): + from kgpipe_search.copy import CopyContext, CopyTarget, HdfsCopyStrategy + + strategy = HdfsCopyStrategy(namenode="nn.example:8020", user="alice") + plan = strategy.build_job( + context=CopyContext( + run_name="run-42", + node_id="node-1", + scratch=ScratchMount( + host_path="/local/d1/docker-scratch", + container_path="/local/d1/docker-scratch", + ), + ), + destination=CopyTarget(path="/user/kgpipe/results"), + ) + + script = plan.command[-1] + assert "fs.defaultFS" in script + assert "hdfs://nn.example:8020" in script + assert "hdfs dfs -put" in script + assert plan.env["HADOOP_USER_NAME"] == "alice" + assert plan.env["KGPIPE_HDFS_DEST"] == "hdfs://nn.example:8020/user/kgpipe/results/run-42" + assert "HADOOP_CONF_DIR" not in plan.env + assert any(m.get("Source") == "/etc/hosts" for m in plan.mounts) + assert not any(m.get("Source") == "/etc/hadoop/conf" for m in plan.mounts) + + +def test_bind_copy_strategy_builds_cp_command_and_mounts(): + from kgpipe_search.copy import BindCopyStrategy, CopyContext, CopyTarget + + strategy = BindCopyStrategy() + plan = strategy.build_job( + context=CopyContext( + run_name="run-42", + node_id="node-1", + scratch=ScratchMount( + host_path="/local/d1/docker-scratch", + container_path="/local/d1/docker-scratch", + ), + ), + destination=CopyTarget(path="/u/hadena/shared-data/docker-results"), + ) + + assert plan.image == "alpine:3.20" + assert "cp -a" in plan.command[-1] + assert "run-42" in plan.command[-1] + assert plan.env["KGPIPE_BIND_DEST"] == "/u/hadena/shared-data/docker-results" + assert any(m.get("Source") == "/local/d1/docker-scratch" for m in plan.mounts) + assert any(m.get("Source") == "/u/hadena/shared-data/docker-results" for m in plan.mounts) + +# KGPIPE_HDFS_NAMENODE=athena1.informatik.intern.uni-leipzig.de KGPIPE_HDFS_USER=hadena KGPIPE_HDFS_DEST=/user/kgpipe/results \ +# uv run pytest -k swarm_hdfs_copy_stage -v + +def test_swarm_bind_copy_stage(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + bind_dest = os.environ.get("KGPIPE_BIND_DEST_HOST") + if not bind_dest: + pytest.skip("set KGPIPE_BIND_DEST_HOST to run bind copy integration test") + + scratch_host = Path(os.environ.get("KGPIPE_SCRATCH_HOST", DEFAULT_SCRATCH_HOST)) + if not scratch_host.is_dir(): + pytest.skip(f"scratch host path does not exist: {scratch_host}") + + bind_dest_path = Path(bind_dest) + if not bind_dest_path.is_dir(): + pytest.skip(f"bind destination host path does not exist: {bind_dest_path}") + + from kgpipe_search.copy import BindCopyStrategy, CopyTarget + from kgpipe_search.mounts import ScratchMount + from kgpipe_search.swarm import SwarmManager + + mgr = SwarmManager() + run_name = f"pytest-bind-copy-{uuid.uuid4().hex[:8]}" + scratch = ScratchMount(host_path=str(scratch_host)) + strategy = BindCopyStrategy() + run = mgr.run_job_with_copy( + image="python:3.12-slim", + command=_SCRATCH_WRITE_COMMAND, + run_name=run_name, + scratch=scratch, + copy_strategy=strategy, + copy_destination=CopyTarget(path=str(bind_dest_path)), + max_per_node=1, + timeout_s=300, + name_prefix="kgpipe-bind-copy", + ) + assert run.job.state == "complete" + assert run.job.exit_code == 0 + assert run.copy is not None, ( + f"copy stage missing; job logs:\n{run.job.logs}" + ) + assert run.copy.state == "complete", ( + f"copy failed (exit_code={run.copy.exit_code}); copy logs:\n{run.copy.logs}" + ) + assert run.copy.exit_code == 0 + assert run.copy.node_id == run.job.node_id + assert "copied to" in run.copy.logs + + # If the bind dest is shared and visible on this host, check directly. + out_path = bind_dest_path / run_name / "done.txt" + if out_path.is_file(): + assert out_path.read_text() == "ok" + return + + # Otherwise verify on the same node via a follow-up service. + assert run.copy.node_id is not None + verify = mgr.run_job( + image="alpine:3.20", + command=[ + "sh", + "-lc", + f"test -f /dst/{run_name}/done.txt && grep -qx ok /dst/{run_name}/done.txt", + ], + node_id=run.copy.node_id, + mounts=[ + {"Type": "bind", "Source": str(bind_dest_path), "Target": "/dst"}, + ], + max_per_node=1, + timeout_s=120, + name_prefix="kgpipe-bind-copy-verify", + ) + assert verify.state == "complete" + assert verify.exit_code == 0 + + +def test_swarm_hdfs_copy_stage(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + hdfs_dest = os.environ.get("KGPIPE_HDFS_DEST") + hdfs_namenode = os.environ.get("KGPIPE_HDFS_NAMENODE") + hdfs_user = os.environ.get("KGPIPE_HDFS_USER") + if not hdfs_dest: + pytest.skip("set KGPIPE_HDFS_DEST to run HDFS copy integration test") + + hadoop_conf = os.environ.get("KGPIPE_HADOOP_CONF", "/etc/hadoop/conf") + if hdfs_namenode is None and not Path(hadoop_conf).is_dir(): + pytest.skip(f"Hadoop config not found: {hadoop_conf}") + + scratch_host = Path(os.environ.get("KGPIPE_SCRATCH_HOST", DEFAULT_SCRATCH_HOST)) + if not scratch_host.is_dir(): + pytest.skip(f"scratch host path does not exist: {scratch_host}") + + from kgpipe_search.copy import CopyTarget, HdfsCopyStrategy + from kgpipe_search.mounts import ScratchMount + from kgpipe_search.swarm import SwarmManager + + if hdfs_namenode: + copy_strategy = HdfsCopyStrategy(namenode=hdfs_namenode, user=hdfs_user) + else: + copy_strategy = HdfsCopyStrategy(hadoop_conf_host=hadoop_conf, user=hdfs_user) + + mgr = SwarmManager() + run_name = f"pytest-hdfs-{uuid.uuid4().hex[:8]}" + scratch = ScratchMount(host_path=str(scratch_host)) + + run = mgr.run_job_with_copy( + image="python:3.12-slim", + command=_SCRATCH_WRITE_COMMAND, + run_name=run_name, + scratch=scratch, + copy_strategy=copy_strategy, + copy_destination=CopyTarget(path=hdfs_dest), + max_per_node=1, + timeout_s=300, + name_prefix="kgpipe-hdfs-write", + ) + + assert run.job.state == "complete" + assert run.job.exit_code == 0 + assert run.copy is not None, ( + f"copy stage missing; job logs:\n{run.job.logs}" + ) + assert run.copy.state == "complete", ( + f"copy failed (exit_code={run.copy.exit_code}); copy logs:\n{run.copy.logs}" + ) + assert run.copy.exit_code == 0 + assert run.copy.node_id == run.job.node_id + assert "copied to" in run.copy.logs + + +#watch -n 1 'for s in $(docker service ls --format "{{.Name}}" | grep "^kgpipe-exp-"); do docker service ps "$s" --format "table {{.Name}}\t{{.Node}}\t{{.CurrentState}}\t{{.Error}}"; done' + +def test_execute_pipeline_docker_swarm_parallel_on_all_nodes(): + ok, reason = _swarm_available() + if not ok: + pytest.skip(reason) + + from kgpipe_search.swarm import SwarmJobResult, SwarmManager, parse_job_result + + mgr = SwarmManager() + node_ids = mgr.active_node_ids() + if not node_ids: + pytest.skip("No active Swarm nodes available") + + params = [str(float(i) + 1.0) for i in range(len(node_ids))] + + def run_job(param: str, target_node_id: str): + res = mgr.run_job( + image="python:3.12-slim", + command=_TEST_COMMAND, + parameter=param, + node_id=target_node_id, + max_per_node=1, + timeout_s=120, + ) + parsed = parse_job_result(res.logs) + return param, target_node_id, res, parsed + + results: list[tuple[str, str, SwarmJobResult, float]] = [] + with ThreadPoolExecutor(max_workers=len(node_ids)) as pool: + futures = [ + pool.submit(run_job, param, node_id) + for param, node_id in zip(params, node_ids, strict=True) + ] + for fut in as_completed(futures): + results.append(fut.result()) + + assert len(results) == len(node_ids) + + assigned_nodes: set[str] = set() + for param, target_node_id, res, parsed in results: + assert res.state == "complete" + assert res.exit_code == 0 + assert parsed == float(param) + 1.0 + assert res.node_id == target_node_id + assigned_nodes.add(target_node_id) + + assert assigned_nodes == set(node_ids), ( + f"expected one job per node ({len(node_ids)} nodes), " + f"but jobs ran on {len(assigned_nodes)} distinct nodes" + ) \ No newline at end of file diff --git a/experiments/param-opti/src/kgpipe_search/test/test_experiments.py b/experiments/param-opti/src/kgpipe_search/test/test_experiments.py new file mode 100644 index 0000000..aaa18d8 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_experiments.py @@ -0,0 +1,118 @@ +import os +import random +from pathlib import Path + +import pytest + +from kgpipe.common import Data, DataFormat, KgPipe +from kgpipe_search.configuration import print_pipeline_config_short +from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE, PipelineConfig +from kgpipe_search.evaluation import evaluate_pipeline +from kgpipe_search.search import random_search + +BUDGET = 10 +SEED = 42 + +ONTOLOGY_PATH = Path("data/bench/moviekg_datasets/film_10k/ontology.ttl") +SEED_PATH = Path("data/bench/moviekg_datasets/film_10k/split_0/kg/seed/data.nt") +REFERENCE_PATH = Path("data/bench/moviekg_datasets/film_10k/split_1/kg/reference/data_agg.nt") +RDF_SOURCE_PATH = Path("data/bench/moviekg_datasets/film_10k/split_0/sources/rdf/data.nt") +RDF_TMP_DIR = Path("data/tmp/rdf_pipelines") + + +def _bench_dataset_available() -> bool: + return all( + path.exists() + for path in (ONTOLOGY_PATH, SEED_PATH, REFERENCE_PATH, RDF_SOURCE_PATH) + ) + + +def _run_rdf_pipeline( + pipeline_config: PipelineConfig, + *, + result_path: Path, + tasks_tmp_dir: Path, + run_name: str, +) -> Path: + tasks_tmp_dir.mkdir(parents=True, exist_ok=True) + result_path.parent.mkdir(parents=True, exist_ok=True) + + pipeline = KgPipe( + tasks=pipeline_config.tasks, + seed=Data(path=SEED_PATH, format=DataFormat.RDF_NTRIPLES), + data_dir=tasks_tmp_dir, + name=run_name, + ) + + pipeline.build( + stable_files=True, + configCatalog=pipeline_config.config_catalog, + source=Data(path=RDF_SOURCE_PATH, format=DataFormat.RDF_NTRIPLES), + result=Data(path=result_path, format=DataFormat.RDF_NTRIPLES), + ) + + pipeline.run(configCatalog=pipeline_config.config_catalog, stable_files_override=False) + return result_path + + +def test_rdf_pipeline_random_search(): + if not _bench_dataset_available(): + pytest.skip("moviekg bench dataset not available under data/bench/moviekg_datasets/film_10k") + + os.environ["ONTOLOGY_PATH"] = str(ONTOLOGY_PATH) + RDF_TMP_DIR.mkdir(parents=True, exist_ok=True) + + trial_counter = {"n": 0} + + def evaluate_fn(pipeline_config: PipelineConfig) -> float: + trial = trial_counter["n"] + trial_counter["n"] += 1 + + result_path = RDF_TMP_DIR / f"random_search_trial_{trial}.nt" + tasks_tmp_dir = RDF_TMP_DIR / f"random_search_trial_{trial}_tasks_tmp" + + _run_rdf_pipeline( + pipeline_config, + result_path=result_path, + tasks_tmp_dir=tasks_tmp_dir, + run_name=f"random_search_trial_{trial}", + ) + + aggregate_score = evaluate_pipeline( + pipeline_config, + result_path, + REFERENCE_PATH, + ) + return aggregate_score.final_score + + run = random_search( + budget=BUDGET, + evaluate_fn=evaluate_fn, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rng=random.Random(SEED), + ) + + print("\n=== rdf pipeline random search ===") + print(f"budget: {BUDGET}") + print(f"seed: {SEED}") + + best_score = float("-inf") + for trial, ((score, pipeline_config), decision) in enumerate( + zip(run.history, run.decisions), + start=1, + ): + if score > best_score: + best_score = score + improved = " (new best)" + else: + improved = "" + + print(f"\n--- trial {trial}/{BUDGET} [{decision}] ---") + print_pipeline_config_short(pipeline_config) + print(f"score: {score:.4f}{improved}") + print(f"best so far: {best_score:.4f}") + + assert len(run.history) == BUDGET + assert len(run.decisions) == BUDGET + assert best_score > float("-inf") diff --git a/experiments/param-opti/src/kgpipe_search/test/test_features.py b/experiments/param-opti/src/kgpipe_search/test/test_features.py new file mode 100644 index 0000000..6a4b217 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_features.py @@ -0,0 +1,24 @@ + + + +# TODO see requirements specification + + +def test_sample_space(): + pass + +def test_neighborhood_optimization(): + pass + +def test_bayesian_optimization(): + pass + +def test_random_search(config_space: Dict[str, Dict[str, Any]], budget: int): + pass + +def test_grid_search(config_space: Dict[str, Dict[str, Any]], budget: int): + pass + +def test_hyperparameter_tuning(): + pass + diff --git a/experiments/param-opti/src/kgpipe_search/test/test_llm_strategy.py b/experiments/param-opti/src/kgpipe_search/test/test_llm_strategy.py new file mode 100644 index 0000000..220d0dc --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_llm_strategy.py @@ -0,0 +1,111 @@ +import json +import random +from dataclasses import dataclass +from typing import List + +from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE +from kgpipe_search.evaluation import execute_and_dummy_evaluate_pipeline +from kgpipe_search.search import llm_search +from kgpipe_search.strategies.llm_strategy import propose_pipeline_config_with_llm +from kgpipe_search.strategies.llm_validation import validate_pipeline_config_snapshot + + +@dataclass +class ScriptedLlmClient: + responses: List[str] + calls: int = 0 + + def complete(self, *, system: str, user: str) -> str: + del system, user + if self.calls >= len(self.responses): + raise RuntimeError("no more scripted responses") + response = self.responses[self.calls] + self.calls += 1 + return response + + +def _valid_snapshot( + *, + entity_threshold: float = 0.7, + relation_threshold: float = 0.6, +) -> dict: + return { + "task_keys": ["paris_graph_alignment_task", "fusion_first_value_task"], + "profiles": { + "paris_graph_alignment_task": { + "profile_name": ( + "paris_graph_alignment_entity_matching_threshold=" + f"{entity_threshold},relation_matching_threshold={relation_threshold}" + ), + "bindings": [ + {"parameter": "entity_matching_threshold", "value": entity_threshold}, + {"parameter": "relation_matching_threshold", "value": relation_threshold}, + ], + } + }, + } + + +def test_validate_pipeline_config_snapshot_accepts_valid_config(): + snapshot = _valid_snapshot() + is_valid, error = validate_pipeline_config_snapshot( + snapshot, RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT + ) + assert is_valid, error + + +def test_validate_pipeline_config_snapshot_rejects_invalid_parameter(): + snapshot = _valid_snapshot() + snapshot["profiles"]["paris_graph_alignment_task"]["bindings"][0]["value"] = 0.42 + + is_valid, error = validate_pipeline_config_snapshot( + snapshot, RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT + ) + assert not is_valid + assert "invalid value" in error + + +def test_propose_pipeline_config_with_llm_retries_until_valid(): + invalid = {"task_keys": ["not_a_real_task"]} + client = ScriptedLlmClient( + responses=[ + "not json", + json.dumps(invalid), + json.dumps(_valid_snapshot()), + ] + ) + + config, decision = propose_pipeline_config_with_llm( + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + client=client, + max_retries=3, + ) + + assert client.calls == 3 + assert decision == "llm(attempt=3)" + assert [task.name for task in config.tasks] == [ + "paris_graph_alignment_task", + "fusion_first_value_task", + ] + + +def test_llm_search_with_mocked_client(): + client = ScriptedLlmClient( + responses=[ + json.dumps(_valid_snapshot(entity_threshold=0.7, relation_threshold=0.6)), + json.dumps(_valid_snapshot(entity_threshold=0.8, relation_threshold=0.5)), + json.dumps(_valid_snapshot(entity_threshold=0.9, relation_threshold=0.7)), + ] + ) + run = llm_search( + budget=3, + evaluate_fn=execute_and_dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + client=client, + rng=random.Random(0), + ) + + assert len(run.history) == 3 + assert all(decision.startswith("llm(") for decision in run.decisions) diff --git a/experiments/param-opti/src/kgpipe_search/test/test_search.py b/experiments/param-opti/src/kgpipe_search/test/test_search.py new file mode 100644 index 0000000..f34f89b --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_search.py @@ -0,0 +1,112 @@ +import random + +from kgpipe_search.configuration import print_pipeline_config_short +from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE, PipelineConfig +from kgpipe_search.evaluation import dummy_evaluate_pipeline +from kgpipe_search.search import ( + SearchRun, + bayesian_optimization, + neighborhood_optimization, + random_search, +) + + +def _print_search_path(run: SearchRun, pipeline_layout) -> None: + best_score = float("-inf") + best_config: PipelineConfig | None = None + + print(f"\n=== {run.strategy} search ===") + print(f"budget: {run.budget}") + print(f"layout: {pipeline_layout.allowed_task_categories}") + + for trial, ((score, pipeline_config), decision) in enumerate( + zip(run.history, run.decisions), + start=1, + ): + print(f"\n--- trial {trial}/{run.budget} [{decision}] ---") + print_pipeline_config_short(pipeline_config) + + if score > best_score: + best_score = score + best_config = pipeline_config + improved = " (new best)" + else: + improved = "" + + print(f"score: {score:.4f}{improved}") + print(f"best so far: {best_score:.4f}") + + print("\n=== search summary ===") + print(f"evaluated: {len(run.history)}") + print(f"best score: {best_score:.4f}") + if best_config is not None: + print("best config:") + print_pipeline_config_short(best_config) + + +def _assert_valid_search_run(run: SearchRun) -> None: + assert len(run.history) == run.budget + assert len(run.decisions) == run.budget + + seen_configs: set[str] = set() + for score, pipeline_config in run.history: + assert 0.5 <= score <= 1.0 + assert pipeline_config.tasks + config_repr = repr( + [ + ( + task.name, + tuple( + (binding.parameter.name, binding.value) + for binding in ( + pipeline_config.config_catalog.get(task.name).bindings + if pipeline_config.config_catalog.get(task.name) + else [] + ) + ), + ) + for task in pipeline_config.tasks + ] + ) + assert config_repr not in seen_configs + seen_configs.add(config_repr) + + +def test_dummy_evaluate_pipeline_random_search_strategy(): + run = random_search( + budget=10, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rng=random.Random(0), + ) + _print_search_path(run, RDF_PIPELINE_LAYOUT) + _assert_valid_search_run(run) + + +def test_dummy_evaluate_pipeline_neighborhood_search_strategy(): + run = neighborhood_optimization( + budget=10, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + k=3, + rho=0.2, + rng=random.Random(1), + ) + _print_search_path(run, RDF_PIPELINE_LAYOUT) + _assert_valid_search_run(run) + + +def test_dummy_evaluate_pipeline_bayesian_search_strategy(): + run = bayesian_optimization( + budget=10, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + init_random=3, + pool_size=16, + rng=random.Random(2), + ) + _print_search_path(run, RDF_PIPELINE_LAYOUT) + _assert_valid_search_run(run) diff --git a/experiments/param-opti/src/kgpipe_search/test/test_search_strategies.py b/experiments/param-opti/src/kgpipe_search/test/test_search_strategies.py new file mode 100644 index 0000000..8931274 --- /dev/null +++ b/experiments/param-opti/src/kgpipe_search/test/test_search_strategies.py @@ -0,0 +1,95 @@ +import random + +from kgpipe_search.configuration import enumerate_valid_task_combinations +from kgpipe_search.definitions import RDF_PIPELINE_LAYOUT, RDF_SEARCH_SPACE +from kgpipe_search.evaluation import dummy_evaluate_pipeline +from kgpipe_search.search import hnr_search, implementation_aware_search, qgns_search +from kgpipe_search.strategies.initialization import implementation_aware_initialization + + +def _assert_valid(run) -> None: + assert len(run.history) == run.budget + assert len(run.decisions) == run.budget + + seen: set[str] = set() + for score, cfg in run.history: + assert 0.5 <= score <= 1.0 + assert cfg.tasks + # Snapshot key uniqueness is the true criterion; repr is good enough here. + key = repr( + [ + ( + task.name, + tuple( + (b.parameter.name, b.value) + for b in ( + cfg.config_catalog.get(task.name).bindings + if cfg.config_catalog.get(task.name) + else [] + ) + ), + ) + for task in cfg.tasks + ] + ) + assert key not in seen + seen.add(key) + + +def test_dummy_evaluate_pipeline_qgns_with_implementation_aware_init(): + run = qgns_search( + budget=10, + init_budget=3, + init_strategy="implementation_aware", + y=1, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + k=3, + rho=0.2, + rng=random.Random(3), + ) + _assert_valid(run) + + +def test_dummy_evaluate_pipeline_hnr(): + run = hnr_search( + budget=10, + init_budget=4, + init_strategy="implementation_aware", + y=1, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rho=0.2, + rng=random.Random(4), + ) + _assert_valid(run) + + +def test_dummy_evaluate_pipeline_implementation_aware_search(): + run = implementation_aware_search( + budget=10, + y=1, + evaluate_fn=dummy_evaluate_pipeline, + search_space=RDF_SEARCH_SPACE, + pipeline_layout=RDF_PIPELINE_LAYOUT, + rng=random.Random(5), + ) + _assert_valid(run) + assert any(str(d).startswith("init(implementation_aware)") for d in run.decisions) + + +def test_implementation_aware_init_can_exceed_task_combo_count(): + combos = enumerate_valid_task_combinations(RDF_SEARCH_SPACE, RDF_PIPELINE_LAYOUT) + init_budget = len(combos) + 1 + + configs = implementation_aware_initialization( + RDF_SEARCH_SPACE, + RDF_PIPELINE_LAYOUT, + budget=init_budget, + y=1, + rng=random.Random(0), + ) + assert len(configs) == init_budget + diff --git a/experiments/param-opti/src/plot_search_evolution.py b/experiments/param-opti/src/plot_search_evolution.py new file mode 100644 index 0000000..0b82e51 --- /dev/null +++ b/experiments/param-opti/src/plot_search_evolution.py @@ -0,0 +1,400 @@ +#!/usr/bin/env python3 +"""Plot search evolution (iteration vs quality score) from search result reports.""" + +from __future__ import annotations + +import argparse +import csv +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable, List, Optional, Sequence, Tuple + +import matplotlib.pyplot as plt + + +DEFAULT_RESULTS_DIR = Path(__file__).resolve().parent.parent / "search-results" +DEFAULT_OUT_DIR = Path(__file__).resolve().parent.parent / "search-results" + +PLOT_FILENAME = "search-evolution.png" +PLOT_CHRONOLOGICAL_FILENAME = "search-evolution-chronological.png" +TABLE_CSV_FILENAME = "search-evolution-table.csv" +TABLE_MD_FILENAME = "search-evolution-table.md" + +STRATEGY_LABELS = { + "bayes-offline.json": "Bayesian optimization", + "bayesian-results.json": "Bayesian optimization", + "hnr-offline.json": "HNR-1", + "hnr-results.json": "HNR-1", + "hnr_2-offline.json": "HNR-2", + "hnr_2-results.json": "HNR-2", + "qgns-offline.json": "RNS", + "qgns-results.json": "RNS", + "random-implementation-aware-offline.json": "Random (implementation-aware)", + "implementation-aware-results.json": "Implementation-aware", + "random-random-offline.json": "Random", + "random-results.json": "Random", +} + +STRATEGY_NAME_LABELS = { + "bayesian": "Bayesian optimization", + "kgpipe_bayes": "Bayesian optimization", + "hnr": "HNR", + "kgpipe_hnr": "HNR", + "qgns": "QGNS", + "kgpipe_qgns": "QGNS", + "implementation_aware": "Implementation-aware", + "random": "Random", + "kgpipe_random": "Random", +} + + +def _read_report(path: Path) -> dict[str, Any]: + return _normalize_report(json.loads(path.read_text(encoding="utf-8"))) + + +def _normalize_report(raw: dict[str, Any]) -> dict[str, Any]: + """Adapt experiment.py results.json to the offline analyse report shape.""" + if "search_history" not in raw: + return raw + + search = raw.get("search") + search_dict = search if isinstance(search, dict) else {} + history = [ + {"score": float(item["score"])} + for item in raw["search_history"] + if isinstance(item, dict) and "score" in item + ] + return { + **raw, + "history": history, + "decisions": search_dict.get("decisions", []), + "strategy": search_dict.get("strategy"), + "init_budget": search_dict.get("init_budget"), + } + + +def _init_budget(report: dict[str, Any]) -> int: + explicit = report.get("init_budget") + if explicit is not None: + return int(explicit) + + search = report.get("search") + if isinstance(search, dict) and search.get("init_budget") is not None: + return int(search["init_budget"]) + + decisions = report.get("decisions") or [] + if isinstance(decisions, list): + return sum(1 for d in decisions if str(d).startswith("init(")) + return 0 + + +def _discover_report_paths(results_dir: Path) -> List[Path]: + offline = sorted(results_dir.glob("*-offline.json")) + if offline: + return offline + return sorted(results_dir.glob("*-results.json")) + + +def _running_best(scores: Sequence[float]) -> List[float]: + best: float | None = None + out: List[float] = [] + for score in scores: + best = score if best is None else max(best, score) + out.append(best) + return out + + +def _evolution_curve( + scores: Sequence[float], + *, + init_budget: int, + reorder_init: bool, + running_best: bool = True, +) -> Tuple[List[int], List[float]]: + if init_budget <= 0 or init_budget >= len(scores): + xs = list(range(1, len(scores) + 1)) + ys = _running_best(scores) if running_best else list(scores) + return xs, ys + + init_scores = list(scores[:init_budget]) + search_scores = list(scores[init_budget:]) + + if reorder_init: + init_scores = sorted(init_scores) + + ordered_scores = init_scores + search_scores + xs = list(range(1, len(ordered_scores) + 1)) + ys = _running_best(ordered_scores) if running_best else ordered_scores + return xs, ys + + +@dataclass(frozen=True) +class StrategyMetrics: + strategy: str + q_best: float + evals_to_95pct: Optional[int] + evals_to_best: Optional[int] + aoc: float + + +def _area_under_curve(xs: Sequence[int], ys: Sequence[float]) -> float: + if len(xs) < 2: + return 0.0 + area = 0.0 + for i in range(len(xs) - 1): + dx = float(xs[i + 1] - xs[i]) + area += dx * (ys[i] + ys[i + 1]) / 2.0 + return area + + +def _evals_to_fraction(xs: Sequence[int], ys: Sequence[float], *, fraction: float) -> Optional[int]: + if not ys: + return None + q_best = max(ys) + threshold = fraction * q_best + for x, y in zip(xs, ys): + if y >= threshold: + return int(x) + return None + + +def _metrics_for_report( + path: Path, + report: dict[str, Any], + *, + reorder_init: bool, + target_fraction: float, +) -> Optional[StrategyMetrics]: + history = report.get("history") + if not isinstance(history, list) or not history: + return None + + scores = [float(item["score"]) for item in history if isinstance(item, dict) and "score" in item] + if not scores: + return None + + init_budget = _init_budget(report) + xs, ys = _evolution_curve( + scores, + init_budget=init_budget, + reorder_init=reorder_init, + running_best=True, + ) + + return StrategyMetrics( + strategy=_label_for(path, report), + q_best=max(ys), + evals_to_95pct=_evals_to_fraction(xs, ys, fraction=target_fraction), + evals_to_best=_evals_to_fraction(xs, ys, fraction=1.0), + aoc=_area_under_curve(xs, ys), + ) + + +def _format_metrics_table(rows: Sequence[StrategyMetrics]) -> List[List[str]]: + header = ["Strategy", "Q best", "Evals to 95%", "Evals to best", "AOC"] + body = [ + [ + row.strategy, + f"{row.q_best:.4f}", + str(row.evals_to_95pct) if row.evals_to_95pct is not None else "—", + str(row.evals_to_best) if row.evals_to_best is not None else "—", + f"{row.aoc:.2f}", + ] + for row in rows + ] + return [header, *body] + + +def _print_metrics_table(rows: Sequence[StrategyMetrics]) -> None: + table = _format_metrics_table(rows) + widths = [max(len(row[i]) for row in table) for i in range(len(table[0]))] + for row_idx, row in enumerate(table): + line = " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)) + print(line) + if row_idx == 0: + print(" ".join("-" * widths[i] for i in range(len(widths)))) + + +def _write_metrics_csv(path: Path, rows: Sequence[StrategyMetrics]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8", newline="") as f: + writer = csv.writer(f) + writer.writerow(["strategy", "q_best", "evals_to_95pct", "evals_to_best", "aoc"]) + for row in rows: + writer.writerow( + [ + row.strategy, + f"{row.q_best:.6f}", + row.evals_to_95pct, + row.evals_to_best, + f"{row.aoc:.4f}", + ] + ) + + +def _write_metrics_markdown(path: Path, rows: Sequence[StrategyMetrics]) -> None: + table = _format_metrics_table(rows) + path.parent.mkdir(parents=True, exist_ok=True) + lines = [ + "| " + " | ".join(table[0]) + " |", + "| " + " | ".join("---" for _ in table[0]) + " |", + ] + for row in table[1:]: + lines.append("| " + " | ".join(row) + " |") + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def _label_for(path: Path, report: dict[str, Any]) -> str: + if path.name in STRATEGY_LABELS: + return STRATEGY_LABELS[path.name] + strategy = report.get("strategy") + if isinstance(strategy, str) and strategy in STRATEGY_NAME_LABELS: + return STRATEGY_NAME_LABELS[strategy] + if isinstance(strategy, str): + return strategy + return path.stem + + +def plot_reports( + reports: Iterable[Tuple[Path, dict[str, Any]]], + *, + reorder_init: bool, + running_best: bool, + out: Path, + title: str, +) -> None: + fig, ax = plt.subplots(figsize=(9, 5.5)) + + for path, report in reports: + history = report.get("history") + if not isinstance(history, list) or not history: + continue + + scores = [float(item["score"]) for item in history if isinstance(item, dict) and "score" in item] + if not scores: + continue + + init_budget = _init_budget(report) + + xs, ys = _evolution_curve( + scores, + init_budget=init_budget, + reorder_init=reorder_init, + running_best=running_best, + ) + label = _label_for(path, report) + ax.plot(xs, ys, marker="o", markersize=3, linewidth=1.8, label=label) + + if init_budget > 0: + ax.axvline(init_budget + 0.5, color="0.75", linestyle=":", linewidth=0.8) + + ax.set_xlabel("Iteration") + ax.set_ylabel("Best quality score so far" if running_best else "Quality score") + ax.set_title(title) + ax.grid(True, alpha=0.3) + ax.legend(loc="lower right", fontsize=9) + fig.tight_layout() + + out.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out, dpi=160) + plt.close(fig) + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser(description="Plot search evolution from JSON result reports.") + p.add_argument( + "--results-dir", + type=Path, + default=DEFAULT_RESULTS_DIR, + help="Directory containing *-offline.json or *-results.json reports.", + ) + p.add_argument( + "--out-dir", + type=Path, + default=DEFAULT_OUT_DIR, + help="Directory for generated figures and tables.", + ) + p.add_argument( + "--skip-chronological-plot", + action="store_true", + help="Skip writing the chronological-init plot.", + ) + p.add_argument( + "--title", + default="Search evolution", + help="Plot title.", + ) + p.add_argument( + "--target-fraction", + type=float, + default=0.95, + help="Fraction of Q best used for the evals-to-target column (default: 0.95).", + ) + return p + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + results_dir: Path = args.results_dir + out_dir: Path = args.out_dir + if not results_dir.is_dir(): + raise SystemExit(f"Results directory not found: {results_dir}") + + report_paths = _discover_report_paths(results_dir) + if not report_paths: + raise SystemExit( + f"No *-offline.json or *-results.json files found in {results_dir}" + ) + + reports = [(path, _read_report(path)) for path in report_paths] + out_dir.mkdir(parents=True, exist_ok=True) + + metrics: List[StrategyMetrics] = [] + for path, report in reports: + row = _metrics_for_report( + path, + report, + reorder_init=True, + target_fraction=float(args.target_fraction), + ) + if row is not None: + metrics.append(row) + + plot_out = out_dir / PLOT_FILENAME + plot_reports( + reports, + reorder_init=True, + running_best=True, + out=plot_out, + title=str(args.title), + ) + print(f"wrote: {plot_out}") + + if not args.skip_chronological_plot: + chrono_out = out_dir / PLOT_CHRONOLOGICAL_FILENAME + chrono_title = f"{args.title} (chronological scores)" + plot_reports( + reports, + reorder_init=False, + running_best=False, + out=chrono_out, + title=chrono_title, + ) + print(f"wrote: {chrono_out}") + + if metrics: + table_csv = out_dir / TABLE_CSV_FILENAME + table_md = out_dir / TABLE_MD_FILENAME + _write_metrics_csv(table_csv, metrics) + _write_metrics_markdown(table_md, metrics) + print(f"wrote: {table_csv}") + print(f"wrote: {table_md}") + print() + _print_metrics_table(metrics) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/param-opti/src/plot_search_evolution_aggregate.py b/experiments/param-opti/src/plot_search_evolution_aggregate.py new file mode 100644 index 0000000..1852a7a --- /dev/null +++ b/experiments/param-opti/src/plot_search_evolution_aggregate.py @@ -0,0 +1,537 @@ +#!/usr/bin/env python3 +"""Aggregate search-evolution plots/tables across RNG seed runs. + +Expects a parent results directory whose subdirectories are per-seed configs, +e.g.:: + + rdf-search-results/ + init_3_budget_20_seed_0/ + init_3_budget_20_seed_42/ + init_3_budget_20_seed_1337/ + +For each config group (everything before ``_seed_``), writes a mean curve +plot with a shaded band and a table of mean ± std metrics. + +The table also reports how often each strategy reaches the known expected +maximum quality score (``TEXT_EXPECTED_MAX`` / ``RDF_EXPECTED_MAX``), inferred +from whether the results directory name contains ``text`` or ``rdf``. +""" + +from __future__ import annotations + +import argparse +import csv +import math +import re +import statistics +from collections import defaultdict +from dataclasses import dataclass +from pathlib import Path +from typing import Dict, List, Optional, Sequence, Tuple + +import matplotlib.pyplot as plt + +from plot_search_evolution import ( + _discover_report_paths, + _evolution_curve, + _init_budget, + _label_for, + _metrics_for_report, + _read_report, + StrategyMetrics, +) + +DEFAULT_RESULTS_DIR = Path(__file__).resolve().parent.parent / "rdf-search-results" + +# Known global maxima (exhaustive / reference best quality scores). +# TEXT wo seed ref default 0.8503806701 custom 0.3802856400657162 +# TEXT with seed ref default 0.8503806701 custom 0.849341845483141 +TEXT_EXPECTED_MAX =0.3802856400657162 +# RDF wo seed ref default 0.8018846725409015 custom 0.7712140467593951 +# RDF with seed ref default 0.9615967544 custom 0.967927789101375 +RDF_EXPECTED_MAX = 0.7712140467593951 + +SEED_DIR_RE = re.compile(r"^(?P.+)_seed_(?P\d+)$") + +PLOT_FILENAME = "search-evolution-aggregated.png" +TABLE_CSV_FILENAME = "search-evolution-aggregated-table.csv" +TABLE_MD_FILENAME = "search-evolution-aggregated-table.md" + +# Absolute tolerance when comparing run Q-best to the expected max. +EXPECTED_MAX_ABS_TOL = 1e-9 + +# Single-column figure for double-column papers (~3.5" column width). +# Size fonts for 1:1 print (do not shrink a wide figure in LaTeX). +COL_WIDTH_IN = 3.5 +COL_HEIGHT_IN = 2.6 +PAPER_DPI = 300 +PAPER_RC = { + "font.size": 9, + "axes.labelsize": 9, + "axes.titlesize": 9, + "xtick.labelsize": 8, + "ytick.labelsize": 8, + "legend.fontsize": 7, + "axes.linewidth": 0.8, + "lines.linewidth": 1.5, + "grid.linewidth": 0.5, +} + + +@dataclass(frozen=True) +class RunCurve: + strategy: str + seed: str + xs: List[int] + ys: List[float] + init_budget: int + metrics: StrategyMetrics + + +@dataclass(frozen=True) +class AggregatedMetrics: + strategy: str + n: int + q_best_mean: float + q_best_std: float + hits_expected_max: int + expected_max: Optional[float] + evals_to_95pct_mean: float + evals_to_95pct_std: float + evals_to_best_mean: float + evals_to_best_std: float + aoc_mean: float + aoc_std: float + + +def _mean(values: Sequence[float]) -> float: + return statistics.fmean(values) if values else float("nan") + + +def _std(values: Sequence[float]) -> float: + if len(values) < 2: + return 0.0 + return statistics.stdev(values) + + +def _fmt_mean_std(mean: float, std: float, *, digits: int) -> str: + return f"{mean:.{digits}f} ± {std:.{digits}f}" + + +def _expected_max_for_results_dir(results_dir: Path) -> Optional[float]: + """Pick TEXT/RDF expected max from the results directory name.""" + name = results_dir.name.lower() + if "text" in name: + return TEXT_EXPECTED_MAX + if "rdf" in name: + return RDF_EXPECTED_MAX + return None + + +def _reaches_expected_max(q_best: float, expected_max: float) -> bool: + return math.isclose(q_best, expected_max, rel_tol=0.0, abs_tol=EXPECTED_MAX_ABS_TOL) + + +def _discover_seed_dirs(results_dir: Path) -> Dict[str, List[Tuple[str, Path]]]: + """Map config key -> list of (seed, path) for ``*_seed_`` subdirs.""" + groups: Dict[str, List[Tuple[str, Path]]] = defaultdict(list) + for path in sorted(p for p in results_dir.iterdir() if p.is_dir()): + match = SEED_DIR_RE.match(path.name) + if not match: + continue + groups[match.group("config")].append((match.group("seed"), path)) + return dict(groups) + + +def _load_run_curves( + run_dir: Path, + *, + seed: str, + reorder_init: bool, + target_fraction: float, +) -> List[RunCurve]: + curves: List[RunCurve] = [] + for path in _discover_report_paths(run_dir): + report = _read_report(path) + metrics = _metrics_for_report( + path, + report, + reorder_init=reorder_init, + target_fraction=target_fraction, + ) + if metrics is None: + continue + + history = report.get("history") + if not isinstance(history, list) or not history: + continue + scores = [float(item["score"]) for item in history if isinstance(item, dict) and "score" in item] + if not scores: + continue + + init_budget = _init_budget(report) + xs, ys = _evolution_curve( + scores, + init_budget=init_budget, + reorder_init=reorder_init, + running_best=True, + ) + curves.append( + RunCurve( + strategy=_label_for(path, report), + seed=seed, + xs=xs, + ys=ys, + init_budget=init_budget, + metrics=metrics, + ) + ) + return curves + + +def _band_bounds( + values: Sequence[float], + *, + band: str, +) -> Tuple[float, float, float]: + mean = _mean(values) + if band == "range": + return mean, min(values), max(values) + if band == "std": + s = _std(values) + return mean, mean - s, mean + s + if band == "sem": + s = _std(values) + sem = s / math.sqrt(len(values)) if values else 0.0 + return mean, mean - sem, mean + sem + raise ValueError(f"Unknown band mode: {band}") + + +def _aggregate_curves( + curves: Sequence[RunCurve], + *, + band: str, +) -> Tuple[List[int], List[float], List[float], List[float], int]: + if not curves: + return [], [], [], [], 0 + + min_len = min(len(c.ys) for c in curves) + xs = list(range(1, min_len + 1)) + means: List[float] = [] + lowers: List[float] = [] + uppers: List[float] = [] + for i in range(min_len): + vals = [c.ys[i] for c in curves] + mean, lo, hi = _band_bounds(vals, band=band) + means.append(mean) + lowers.append(lo) + uppers.append(hi) + return xs, means, lowers, uppers, len(curves) + + +def _aggregate_metrics( + curves: Sequence[RunCurve], + *, + expected_max: Optional[float], +) -> AggregatedMetrics: + q_best = [c.metrics.q_best for c in curves] + aoc = [c.metrics.aoc for c in curves] + to_95 = [float(c.metrics.evals_to_95pct) for c in curves if c.metrics.evals_to_95pct is not None] + to_best = [float(c.metrics.evals_to_best) for c in curves if c.metrics.evals_to_best is not None] + hits = ( + sum(1 for q in q_best if _reaches_expected_max(q, expected_max)) + if expected_max is not None + else 0 + ) + return AggregatedMetrics( + strategy=curves[0].strategy, + n=len(curves), + q_best_mean=_mean(q_best), + q_best_std=_std(q_best), + hits_expected_max=hits, + expected_max=expected_max, + evals_to_95pct_mean=_mean(to_95), + evals_to_95pct_std=_std(to_95), + evals_to_best_mean=_mean(to_best), + evals_to_best_std=_std(to_best), + aoc_mean=_mean(aoc), + aoc_std=_std(aoc), + ) + + +def _fmt_hits(row: AggregatedMetrics) -> str: + if row.expected_max is None: + return "—" + return f"{row.hits_expected_max}/{row.n}" + + +def _format_metrics_table(rows: Sequence[AggregatedMetrics]) -> List[List[str]]: + header = ["Strategy", "n", "Q best", "Hits max", "Evals to 95%", "Evals to best", "AOC"] + body = [ + [ + row.strategy, + str(row.n), + _fmt_mean_std(row.q_best_mean, row.q_best_std, digits=4), + _fmt_hits(row), + _fmt_mean_std(row.evals_to_95pct_mean, row.evals_to_95pct_std, digits=2), + _fmt_mean_std(row.evals_to_best_mean, row.evals_to_best_std, digits=2), + _fmt_mean_std(row.aoc_mean, row.aoc_std, digits=2), + ] + for row in rows + ] + return [header, *body] + + +def _print_metrics_table(rows: Sequence[AggregatedMetrics]) -> None: + table = _format_metrics_table(rows) + widths = [max(len(row[i]) for row in table) for i in range(len(table[0]))] + for row_idx, row in enumerate(table): + line = " ".join(cell.ljust(widths[i]) for i, cell in enumerate(row)) + print(line) + if row_idx == 0: + print(" ".join("-" * widths[i] for i in range(len(widths)))) + + +def _write_metrics_csv(path: Path, rows: Sequence[AggregatedMetrics]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8", newline="") as f: + writer = csv.writer(f) + writer.writerow( + [ + "strategy", + "n", + "q_best_mean", + "q_best_std", + "hits_expected_max", + "expected_max", + "evals_to_95pct_mean", + "evals_to_95pct_std", + "evals_to_best_mean", + "evals_to_best_std", + "aoc_mean", + "aoc_std", + ] + ) + for row in rows: + writer.writerow( + [ + row.strategy, + row.n, + f"{row.q_best_mean:.6f}", + f"{row.q_best_std:.6f}", + row.hits_expected_max, + f"{row.expected_max:.10f}" if row.expected_max is not None else "", + f"{row.evals_to_95pct_mean:.4f}", + f"{row.evals_to_95pct_std:.4f}", + f"{row.evals_to_best_mean:.4f}", + f"{row.evals_to_best_std:.4f}", + f"{row.aoc_mean:.4f}", + f"{row.aoc_std:.4f}", + ] + ) + + +def _write_metrics_markdown(path: Path, rows: Sequence[AggregatedMetrics]) -> None: + table = _format_metrics_table(rows) + path.parent.mkdir(parents=True, exist_ok=True) + lines = [ + "| " + " | ".join(table[0]) + " |", + "| " + " | ".join("---" for _ in table[0]) + " |", + ] + for row in table[1:]: + lines.append("| " + " | ".join(row) + " |") + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def plot_aggregated( + by_strategy: Dict[str, List[RunCurve]], + *, + band: str, + out: Path, + title: str, +) -> None: + with plt.rc_context(PAPER_RC): + fig, ax = plt.subplots(figsize=(COL_WIDTH_IN, COL_HEIGHT_IN)) + init_budget: Optional[int] = None + + for strategy in sorted(by_strategy): + curves = by_strategy[strategy] + xs, means, lowers, uppers, n = _aggregate_curves(curves, band=band) + if not xs: + continue + if init_budget is None and curves: + init_budget = curves[0].init_budget + + (line,) = ax.plot( + xs, + means, + marker="o", + markersize=2.5, + linewidth=1.5, + label=f"{strategy} (n={n})", + ) + ax.fill_between(xs, lowers, uppers, color=line.get_color(), alpha=0.2, linewidth=0) + + if init_budget and init_budget > 0: + ax.axvline(init_budget + 0.5, color="0.75", linestyle=":", linewidth=0.8) + + band_label = {"range": "min–max range", "std": "±1 std", "sem": "±1 SEM"}[band] + ax.set_xlabel("Iteration") + ax.set_ylabel("Best quality (mean)") + ax.set_title(f"{title}\n(shaded: {band_label})") + ax.grid(True, alpha=0.3) + ax.legend( + loc="lower right", + frameon=True, + borderpad=0.3, + labelspacing=0.25, + handlelength=1.2, + handletextpad=0.4, + borderaxespad=0.3, + ) + fig.tight_layout(pad=0.35) + + out.parent.mkdir(parents=True, exist_ok=True) + fig.savefig(out, dpi=PAPER_DPI, bbox_inches="tight") + plt.close(fig) + + +def _process_config_group( + config: str, + seed_dirs: Sequence[Tuple[str, Path]], + *, + out_dir: Path, + title: str, + band: str, + target_fraction: float, + expected_max: Optional[float], +) -> None: + by_strategy: Dict[str, List[RunCurve]] = defaultdict(list) + for seed, run_dir in seed_dirs: + for curve in _load_run_curves( + run_dir, + seed=seed, + reorder_init=True, + target_fraction=target_fraction, + ): + by_strategy[curve.strategy].append(curve) + + if not by_strategy: + print(f"skip {config}: no usable reports in {[p.name for _, p in seed_dirs]}") + return + + metrics = [ + _aggregate_metrics(curves, expected_max=expected_max) + for _, curves in sorted(by_strategy.items()) + ] + metrics.sort(key=lambda row: row.strategy) + + out_dir.mkdir(parents=True, exist_ok=True) + plot_out = out_dir / PLOT_FILENAME + plot_aggregated(by_strategy, band=band, out=plot_out, title=title) + print(f"wrote: {plot_out}") + + table_csv = out_dir / TABLE_CSV_FILENAME + table_md = out_dir / TABLE_MD_FILENAME + _write_metrics_csv(table_csv, metrics) + _write_metrics_markdown(table_md, metrics) + print(f"wrote: {table_csv}") + print(f"wrote: {table_md}") + print() + expected_label = ( + f"expected_max={expected_max:.10f}" if expected_max is not None else "expected_max=none" + ) + print(f"[{config}] seeds={[s for s, _ in seed_dirs]} {expected_label}") + _print_metrics_table(metrics) + print() + + +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + description="Aggregate search-evolution plots/tables across seed runs." + ) + p.add_argument( + "--results-dir", + type=Path, + default=DEFAULT_RESULTS_DIR, + help="Parent directory containing *_seed_ subdirectories.", + ) + p.add_argument( + "--out-dir", + type=Path, + default=None, + help="Output directory (default: // or if one config).", + ) + p.add_argument( + "--config", + default=None, + help="Only aggregate this config prefix (e.g. init_3_budget_20).", + ) + p.add_argument( + "--band", + choices=("range", "std", "sem"), + default="range", + help="Shaded band around the mean curve: min-max range, ±1 std, or ±1 SEM (default: range).", + ) + p.add_argument( + "--title", + default=None, + help="Plot title prefix (default derived from config).", + ) + p.add_argument( + "--target-fraction", + type=float, + default=0.95, + help="Fraction of Q best used for evals-to-target (default: 0.95).", + ) + return p + + +def main(argv: Sequence[str] | None = None) -> int: + args = build_parser().parse_args(argv) + results_dir: Path = args.results_dir + if not results_dir.is_dir(): + raise SystemExit(f"Results directory not found: {results_dir}") + + groups = _discover_seed_dirs(results_dir) + if not groups: + raise SystemExit( + f"No *_seed_ subdirectories found in {results_dir}" + ) + + if args.config is not None: + if args.config not in groups: + raise SystemExit( + f"Config {args.config!r} not found. Available: {sorted(groups)}" + ) + groups = {args.config: groups[args.config]} + + expected_max = _expected_max_for_results_dir(results_dir) + if expected_max is None: + print( + f"warning: could not infer TEXT/RDF expected max from {results_dir.name!r}; " + "Hits max column will be empty" + ) + + for config, seed_dirs in sorted(groups.items()): + if args.out_dir is not None: + out_dir = args.out_dir if len(groups) == 1 else args.out_dir / config + else: + out_dir = results_dir / config if len(groups) > 1 else results_dir + + title = args.title or f"Search evolution ({config}, aggregated over seeds)" + _process_config_group( + config, + seed_dirs, + out_dir=out_dir, + title=title, + band=str(args.band), + target_fraction=float(args.target_fraction), + expected_max=expected_max, + ) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/param-opti/src/split_pipeline_configs.py b/experiments/param-opti/src/split_pipeline_configs.py new file mode 100644 index 0000000..bb44d89 --- /dev/null +++ b/experiments/param-opti/src/split_pipeline_configs.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python3 + +import argparse +import json +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + + +def load_fixture(path: Path) -> Tuple[Dict[str, Any], List[Dict[str, Any]]]: + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, dict) or "samples" not in data: + raise ValueError(f"Expected dict with 'samples' key in {path}") + samples = data.get("samples") + if not isinstance(samples, list): + raise ValueError(f"Expected 'samples' to be a list in {path}") + return data, samples + + +def task_layout_key(sample: Dict[str, Any]) -> Tuple[str, ...]: + return tuple(str(key) for key in (sample.get("task_keys") or [])) + + +def chunk_list(items: List[Dict[str, Any]], chunk_size: int) -> List[List[Dict[str, Any]]]: + if chunk_size <= 0: + raise ValueError(f"chunk_size must be positive, got {chunk_size}") + return [items[i : i + chunk_size] for i in range(0, len(items), chunk_size)] + + +def split_into_num_parts( + items: List[Dict[str, Any]], num_parts: int +) -> List[List[Dict[str, Any]]]: + if num_parts <= 0: + raise ValueError(f"num_parts must be positive, got {num_parts}") + if num_parts > len(items): + raise ValueError( + f"num_parts ({num_parts}) cannot exceed number of samples ({len(items)})" + ) + + base_size, remainder = divmod(len(items), num_parts) + parts: List[List[Dict[str, Any]]] = [] + start = 0 + for index in range(num_parts): + size = base_size + (1 if index < remainder else 0) + parts.append(items[start : start + size]) + start += size + return parts + + +def split_sequential( + samples: List[Dict[str, Any]], + *, + num_parts: Optional[int], + max_per_file: Optional[int], +) -> List[List[Dict[str, Any]]]: + if num_parts is not None and max_per_file is not None: + raise ValueError("Use only one of --num-parts or --max-per-file for sequential splitting") + if num_parts is not None: + return split_into_num_parts(samples, num_parts) + if max_per_file is not None: + return chunk_list(samples, max_per_file) + raise ValueError("Sequential splitting requires --num-parts or --max-per-file") + + +def split_by_layout( + samples: List[Dict[str, Any]], + *, + max_per_file: Optional[int], +) -> List[List[Dict[str, Any]]]: + grouped: Dict[Tuple[str, ...], List[Dict[str, Any]]] = {} + layout_order: List[Tuple[str, ...]] = [] + for sample in samples: + layout = task_layout_key(sample) + if layout not in grouped: + grouped[layout] = [] + layout_order.append(layout) + grouped[layout].append(sample) + + parts: List[List[Dict[str, Any]]] = [] + for layout in layout_order: + layout_samples = grouped[layout] + if max_per_file is None: + parts.append(layout_samples) + else: + parts.extend(chunk_list(layout_samples, max_per_file)) + return parts + + +def output_path(out_dir: Path, stem: str, index: int, total_parts: int) -> Path: + width = max(2, len(str(total_parts))) + return out_dir / f"{stem}_{index:0{width}d}.json" + + +def write_parts( + *, + top_level: Dict[str, Any], + parts: List[List[Dict[str, Any]]], + out_dir: Path, + stem: str, + dry_run: bool, +) -> List[Dict[str, Any]]: + if not parts: + raise ValueError("No output parts produced") + + total_parts = len(parts) + written: List[Dict[str, Any]] = [] + for index, part_samples in enumerate(parts, start=1): + out_path = output_path(out_dir, stem, index, total_parts) + out_data = dict(top_level) + out_data["samples"] = part_samples + + record = { + "part": index, + "path": str(out_path), + "samples": len(part_samples), + } + written.append(record) + + if dry_run: + continue + + out_dir.mkdir(parents=True, exist_ok=True) + out_path.write_text( + json.dumps(out_data, indent=2, ensure_ascii=False) + "\n", + encoding="utf-8", + ) + return written + + +def main() -> int: + parser = argparse.ArgumentParser( + description=( + "Split a pipeline configs fixture into numbered sub-files " + "(e.g. rdf_exhaustive_pipeline_configs_01.json)." + ) + ) + parser.add_argument("--input", required=True, type=Path, help="Input fixture JSON file") + parser.add_argument( + "--out-dir", + required=True, + type=Path, + help="Directory for numbered output files", + ) + parser.add_argument( + "--stem", + type=str, + default=None, + help="Output filename stem (default: input filename without extension)", + ) + parser.add_argument( + "--num-parts", + type=int, + default=None, + help="Split sequentially into N roughly equal parts", + ) + parser.add_argument( + "--max-per-file", + type=int, + default=None, + help="Maximum configs per output file", + ) + parser.add_argument( + "--by-layout", + action="store_true", + help=( + "Group configs by task layout (task_keys) before splitting. " + "Without --max-per-file, writes one file per layout." + ), + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print the split plan without writing files", + ) + args = parser.parse_args() + + if args.num_parts is None and args.max_per_file is None and not args.by_layout: + parser.error("Specify --num-parts, --max-per-file, or --by-layout") + + top_level, samples = load_fixture(args.input) + if not samples: + raise ValueError(f"No samples found in {args.input}") + + if args.by_layout: + parts = split_by_layout(samples, max_per_file=args.max_per_file) + else: + parts = split_sequential( + samples, + num_parts=args.num_parts, + max_per_file=args.max_per_file, + ) + + stem = args.stem or args.input.stem + written = write_parts( + top_level=top_level, + parts=parts, + out_dir=args.out_dir, + stem=stem, + dry_run=args.dry_run, + ) + + print( + json.dumps( + { + "input_file": str(args.input), + "out_dir": str(args.out_dir), + "stem": stem, + "mode": "layout" if args.by_layout else "sequential", + "input_samples": len(samples), + "num_parts": len(parts), + "dry_run": args.dry_run, + "parts": written, + }, + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/experiments/param-opti/src/subtract_pipeline_configs.py b/experiments/param-opti/src/subtract_pipeline_configs.py new file mode 100644 index 0000000..5dbae4a --- /dev/null +++ b/experiments/param-opti/src/subtract_pipeline_configs.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 + +import argparse +import hashlib +import json +from pathlib import Path +from typing import Any, Dict, List, Tuple + + +def _normalize_bindings(bindings: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + # Ensure stable ordering and stable object shape. + norm = [] # type: List[Dict[str, Any]] + for b in bindings: + norm.append({"parameter": b.get("parameter"), "value": b.get("value")}) + norm.sort(key=lambda x: (str(x.get("parameter")), json.dumps(x.get("value"), sort_keys=True))) + return norm + + +def canonical_sample(sample: Dict[str, Any]) -> Dict[str, Any]: + task_keys = sample.get("task_keys") or [] + profiles = sample.get("profiles") or {} + + canon_profiles = {} # type: Dict[str, Any] + for profile_key, profile in profiles.items(): + bindings = _normalize_bindings(profile.get("bindings") or []) + # Prefer the explicit profile_name if present, but don't rely on it exclusively. + canon_profiles[str(profile_key)] = { + "profile_name": profile.get("profile_name"), + "bindings": bindings, + } + + return { + "task_keys": sorted(map(str, task_keys)), + "profiles": {k: canon_profiles[k] for k in sorted(canon_profiles.keys())}, + } + + +def sample_key(sample: Dict[str, Any]) -> str: + canon = canonical_sample(sample) + blob = json.dumps(canon, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8") + return hashlib.sha256(blob).hexdigest() + + +def load_fixture(path: Path) -> Tuple[Dict[str, Any], List[Dict[str, Any]]]: + data = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(data, dict) or "samples" not in data: + raise ValueError(f"Expected dict with 'samples' key in {path}") + samples = data.get("samples") + if not isinstance(samples, list): + raise ValueError(f"Expected 'samples' to be a list in {path}") + return data, samples + + +def subtract( + keep: List[Dict[str, Any]], remove: List[Dict[str, Any]] +) -> Tuple[List[Dict[str, Any]], int, int]: + remove_keys = {sample_key(s) for s in remove} + out = [] # type: List[Dict[str, Any]] + kept = 0 + dropped = 0 + for s in keep: + if sample_key(s) in remove_keys: + dropped += 1 + continue + out.append(s) + kept += 1 + return out, kept, dropped + + +def main() -> int: + p = argparse.ArgumentParser( + description="Subtract pipeline config samples between two fixture JSON files." + ) + p.add_argument("--keep", required=True, type=Path, help="Base fixture (A)") + p.add_argument("--remove", required=True, type=Path, help="Fixture to subtract (B)") + p.add_argument("--out", required=True, type=Path, help="Output fixture path (A - B)") + p.add_argument( + "--preserve-version", + action="store_true", + help="Preserve top-level 'version' from --keep (default: keep entire top-level object and only replace samples).", + ) + args = p.parse_args() + + keep_data, keep_samples = load_fixture(args.keep) + _, remove_samples = load_fixture(args.remove) + + out_samples, kept, dropped = subtract(keep_samples, remove_samples) + + # Default behavior: keep the top-level shape of --keep (e.g. version, metadata) and swap samples. + out_data = {} # type: Dict[str, Any] + if args.preserve_version: + out_data = {"version": keep_data.get("version"), "samples": out_samples} + else: + out_data = dict(keep_data) + out_data["samples"] = out_samples + + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(out_data, indent=2, ensure_ascii=False) + "\n", encoding="utf-8") + + print( + json.dumps( + { + "keep_file": str(args.keep), + "remove_file": str(args.remove), + "out_file": str(args.out), + "keep_samples": len(keep_samples), + "remove_samples": len(remove_samples), + "out_samples": len(out_samples), + "dropped_from_keep": dropped, + }, + indent=2, + ) + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) + diff --git a/experiments/param-opti/wrappers/genie/Dockerfile b/experiments/param-opti/wrappers/genie/Dockerfile new file mode 100644 index 0000000..a66625e --- /dev/null +++ b/experiments/param-opti/wrappers/genie/Dockerfile @@ -0,0 +1,30 @@ +FROM python:3.8-slim + +WORKDIR /app + +RUN apt-get update && apt-get install -y --no-install-recommends \ + git wget unzip\ + && rm -rf /var/lib/apt/lists/* + +RUN git clone https://github.com/epfl-dlab/GenIE.git + +WORKDIR /app/GenIE + +RUN pip install --upgrade pip + +RUN pip install -r pip_requirements.txt + +RUN mkdir -p data/models + +# Models initialized with a pretrained language model (GenIE - PLM) Trained on Rebel +RUN wget https://zenodo.org/record/6139236/files/genie_plm_r.ckpt \ + -O data/models/genie_plm_r.ckpt + +RUN wget https://zenodo.org/record/6139236/files/tries.zip \ + && unzip tries.zip -d data && rm tries.zip + +COPY bin/genie_cli.py /app/GenIE/genie_cli.py +COPY genie.sh /usr/local/bin/genie.sh +RUN chmod +x /usr/local/bin/genie.sh + + diff --git a/experiments/param-opti/wrappers/genie/README.md b/experiments/param-opti/wrappers/genie/README.md new file mode 100644 index 0000000..4b21480 --- /dev/null +++ b/experiments/param-opti/wrappers/genie/README.md @@ -0,0 +1,79 @@ +# README.md +## Build Docker +```bash +docker build -t genie . +``` + +## Run Docker +```bash +docker run --rm \ + -v /home/theo/Work/SCADS.AI/Projects/KGpipe/experiments/text-pipelines/test/Titanic.txt:/data/input.txt \ + -v /home/theo/Work/SCADS.AI/Projects/KGpipe/experiments/text-pipelines/wrappers/genie/output.json:/data/output.json \ + genie genie.sh /data/input.txt /data/output.json +``` + + +## Tool Parameters + +### Model Parameters +- `checkpoint` (pre trained model) **or** +- `hydra` + +--- + +### Constraint Parameters +- `entity_trie` (pickle) **or** string list +- `relation_trie` (pickle) **or** string list + +--- + +### Generate Parameters +Uses standard `Transformers generate()` function. + +#### Beam Search +- `num_beams` +- `num_return_sequences` +- `early_stopping` +- `length_penalty` + +#### Sampling +- `do_sample` +- `temperature` +- `top_k` +- `top_p` +- `typical_p` + +#### Output Length +- `max_length` +- `max_new_tokens` +- `min_length` +- `min_new_tokens` + +#### Scores & Debug +- `return_dict_in_generate` +- `output_scores` +- `output_attentions` +- `output_hidden_states` +- `output_logits` + +#### Seed +- `seed` + +#### Token-Control +- `bos_token_id` +- `eos_token_id` +- `pad_token_id` +- `decoder_start_token_id` +- `forced_bos_token_id` +- `forced_eos_token_id` + +#### Repetition / Constraints +- `repetition_penalty` +- `no_repeat_ngram_size` +- `bad_words_ids` +- `force_words_ids` +- `constraints` +- `prefix_allowed_tokens_fn` + + + diff --git a/experiments/param-opti/wrappers/genie/bin/genie_cli.py b/experiments/param-opti/wrappers/genie/bin/genie_cli.py new file mode 100644 index 0000000..3382e36 --- /dev/null +++ b/experiments/param-opti/wrappers/genie/bin/genie_cli.py @@ -0,0 +1,126 @@ +import sys +import os +import json +import re + +from genie.models import GeniePL +from genie.constrained_generation import Trie + +DATA_DIR = os.path.join(os.getcwd(), "data") + + +def load_model(): + ckpt_name = "genie_plm_r.ckpt" + path_to_checkpoint = os.path.join(DATA_DIR, "models", ckpt_name) + + model = GeniePL.load_from_checkpoint( + checkpoint_path=path_to_checkpoint + ) + + return model + + +def load_tries(): + entity_trie_path = os.path.join(DATA_DIR, "tries/large/entity_trie.pickle") + entity_trie = Trie.load(entity_trie_path) + + relation_trie_path = os.path.join(DATA_DIR, "tries/large/relation_trie.pickle") + relation_trie = Trie.load(relation_trie_path) + + return {"entity_trie": entity_trie, "relation_trie": relation_trie} + + +def split_into_sentences(text: str): + text = re.sub(r"\s+", " ", text).strip() + if not text: + return [] + + # Keep this lightweight so folder mode still benefits from a + # single long-lived Python process without extra tokenizer deps. + parts = re.split(r"(?<=[.!?])\s+(?=[A-Z0-9\"'(\[])", + text) + return [part.strip() for part in parts if part.strip()] + + +def extract_file(model, tries, input_path: str, output_path: str): + with open(input_path, "r", encoding="utf-8") as f: + text = f.read() + + sentences = split_into_sentences(text) + if not sentences: + with open(output_path, "w", encoding="utf-8") as f: + json.dump([], f, indent=2, ensure_ascii=False) + return + + generation_args = { + "num_beams": 5, + "num_return_sequences": 1, + "max_length": 128, + "early_stopping": True, + "no_repeat_ngram_size": 3, + "repetition_penalty": 1.2, + "length_penalty": 0.8, + "return_dict_in_generate": True, + "output_scores": True, + } + + outputs = model.sample( + sentences, + **tries, + **generation_args, + ) + + with open(output_path, "w", encoding="utf-8") as f: + json.dump(outputs, f, indent=2, ensure_ascii=False) + + +def main(): + if len(sys.argv) < 3: + print("Usage: genie.sh ") + sys.exit(1) + + input_path = sys.argv[1] + output_path = sys.argv[2] + + if os.path.isdir(input_path): + if os.path.isfile(output_path): + raise SystemExit("Error: output must be a folder when input is a folder") + + os.makedirs(output_path, exist_ok=True) + + model = load_model() + tries = load_tries() + + files = [ + os.path.join(input_path, name) + for name in os.listdir(input_path) + if os.path.isfile(os.path.join(input_path, name)) + ] + files.sort() + + for in_file in files: + filename = os.path.basename(in_file) + out_file = os.path.join(output_path, filename) + if os.path.exists(out_file): + continue + extract_file(model, tries, in_file, out_file) + print(f"Processed {in_file} → {out_file}") + + print(f"Extraction finished. Results written to folder {output_path}") + return + + if os.path.isfile(input_path): + if os.path.isdir(output_path): + raise SystemExit("Error: output must be a file when input is a file") + + model = load_model() + tries = load_tries() + extract_file(model, tries, input_path, output_path) + print(f"Extraction finished. Results written to {output_path}") + return + + raise SystemExit("Error: input must be a file or directory") + + +if __name__ == "__main__": + main() diff --git a/experiments/param-opti/wrappers/genie/genie.sh b/experiments/param-opti/wrappers/genie/genie.sh new file mode 100644 index 0000000..71fb229 --- /dev/null +++ b/experiments/param-opti/wrappers/genie/genie.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -e + +if [ "$#" -ne 2 ]; then + echo "Usage:" + echo " genie.sh " + echo " genie.sh " + exit 1 +fi + +INPUT="$1" +OUTPUT="$2" + +GRAPHENE_DIR="/app/GenIE" + +if [ -f "$INPUT" ]; then + if [ -d "$OUTPUT" ]; then + echo "Error: Output must be a file when input is a file" + exit 1 + fi + + echo "Processing single file..." + python /app/GenIE/genie_cli.py "$INPUT" "$OUTPUT" + echo "Done." + exit 0 +fi + +if [ -d "$INPUT" ]; then + if [ -f "$OUTPUT" ]; then + echo "Error: Output must be a folder when input is a folder" + exit 1 + fi + mkdir -p "$OUTPUT" + chmod 777 "$OUTPUT" + + echo "Processing folder..." + python /app/GenIE/genie_cli.py "$INPUT" "$OUTPUT" + echo "All files processed." + exit 0 +fi + +echo "Error: Input must be a file or directory" +exit 1 \ No newline at end of file diff --git a/experiments/param-opti/wrappers/genie/output.json b/experiments/param-opti/wrappers/genie/output.json new file mode 100644 index 0000000..aa99d3e --- /dev/null +++ b/experiments/param-opti/wrappers/genie/output.json @@ -0,0 +1,20 @@ +[ + [ + { + "text": " Captain America publisher Marvel Comics ", + "log_prob": -0.8582733273506165 + } + ], + [ + { + "text": " Marvel Studios parent organization Paramount Pictures ", + "log_prob": -0.5044617652893066 + } + ], + [ + { + "text": " El Capitan Theatre country United States ", + "log_prob": -0.5770944952964783 + } + ] +] \ No newline at end of file diff --git a/experiments/param-opti/wrappers/genie/output_1.json b/experiments/param-opti/wrappers/genie/output_1.json new file mode 100644 index 0000000..4972807 --- /dev/null +++ b/experiments/param-opti/wrappers/genie/output_1.json @@ -0,0 +1,26 @@ +[ + [ + { + "text": " Captain America publisher Marvel Comics ", + "log_prob": -0.4791736900806427 + } + ], + [ + { + "text": " Marvel Cinematic Universe production company Marvel Studios ", + "log_prob": -0.33403322100639343 + } + ], + [ + { + "text": " Captain America performer Chris Evans (actor) ", + "log_prob": -0.46612370014190674 + } + ], + [ + { + "text": " Captain America conflict World War II ", + "log_prob": -0.39299872517585754 + } + ] +] \ No newline at end of file diff --git a/experiments/param-opti/wrappers/genie/test.txt b/experiments/param-opti/wrappers/genie/test.txt new file mode 100644 index 0000000..4eb32be --- /dev/null +++ b/experiments/param-opti/wrappers/genie/test.txt @@ -0,0 +1,3 @@ +Captain America: The First Avenger is a 2011 American superhero film based on the Marvel Comics character Captain America. Produced by Marvel Studios and distributed by Paramount Pictures, it is the fifth film in the Marvel Cinematic Universe (MCU). The film was directed by Joe Johnston, written by Christopher Markus and Stephen McFeely, and stars Chris Evans as Steve Rogers / Captain America alongside Tommy Lee Jones, Hugo Weaving, Hayley Atwell, Sebastian Stan, Dominic Cooper, Toby Jones, Neal McDonough, Derek Luke, and Stanley Tucci. During World War II, Rogers, a frail man, is transformed into the super-soldier Captain America and must stop the Red Skull (Weaving) from using the Tesseract as an energy source for world domination. +The film began as a concept in 1997 and was scheduled for distribution by Artisan Entertainment. However, a lawsuit disrupted the project and was not settled until September 2003. In 2005, Marvel Studios received a loan from Merrill Lynch, and planned to finance and release the film through Paramount Pictures. Directors Jon Favreau and Louis Leterrier were interested in directing the project before Johnston was approached in 2008. The principal characters were cast between March and June 2010. Production began in June, and filming took place in London, Manchester, Caerwent, Liverpool, and Los Angeles. Several different techniques were used by the visual effects company Lola to create the physical appearance of the character before he becomes Captain America. +Captain America: The First Avenger premiered at the El Capitan Theatre in Los Angeles on July 19, 2011, and was released in the United States on July 22, as part of Phase One of the MCU. The film was commercially successful, grossing over $370 million worldwide, and received positive reviews from critics, who praised Evans' performance, the film's depiction of its 1940s time period, and Johnston's direction. Two direct sequels have been released: Captain America: The Winter Soldier (2014) and Captain America: Civil War (2016). diff --git a/experiments/param-opti/wrappers/genie/test_1.txt b/experiments/param-opti/wrappers/genie/test_1.txt new file mode 100644 index 0000000..ea6f33e --- /dev/null +++ b/experiments/param-opti/wrappers/genie/test_1.txt @@ -0,0 +1 @@ +Captain America: The First Avenger is a 2011 American superhero film based on the Marvel Comics character Captain America. Produced by Marvel Studios and distributed by Paramount Pictures, it is the fifth film in the Marvel Cinematic Universe (MCU). The film was directed by Joe Johnston, written by Christopher Markus and Stephen McFeely, and stars Chris Evans as Steve Rogers / Captain America alongside Tommy Lee Jones, Hugo Weaving, Hayley Atwell, Sebastian Stan, Dominic Cooper, Toby Jones, Neal McDonough, Derek Luke, and Stanley Tucci. During World War II, Rogers, a frail man, is transformed into the super-soldier Captain America and must stop the Red Skull (Weaving) from using the Tesseract as an energy source for world domination. diff --git a/experiments/param-opti/wrappers/genie/test_docker_run.sh b/experiments/param-opti/wrappers/genie/test_docker_run.sh new file mode 100644 index 0000000..9b00275 --- /dev/null +++ b/experiments/param-opti/wrappers/genie/test_docker_run.sh @@ -0,0 +1 @@ +docker run -v $(pwd):$(pwd) genie genie.sh $(pwd)/test_1.txt $(pwd)/output_1.json \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..cafa4aa --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,55 @@ +site_name: KGpipe +site_description: Knowledge Graph pipeline evaluation framework + +# For GitHub Pages under // +use_directory_urls: true + +theme: + name: material + features: + - navigation.instant + - navigation.tracking + - navigation.sections + - navigation.expand + - navigation.top + - toc.integrate + - search.suggest + - search.highlight + +markdown_extensions: + - admonition + - toc: + permalink: true + - pymdownx.superfences + - pymdownx.details + +plugins: + - search + +docs_dir: docs +site_dir: site + +nav: + - Home: index.md + - Quickstart: quickstart.md + - KGI-Bench (benchmark site): https://scads.github.io/KGI-Bench/ + - Concepts: + - Tasks: tasks.md + - Pipelines: pipelines.md + - Configuration: configuration.md + - Parameters: parameters.md + - Meta KG: metakg.md + - Evaluation: + - Overview: evaluation.md + - Metrics index: metrics/metrics.md + - Entity coverage: metrics/entity_coverage.md + - Reference entity alignment: metrics/reference_entity_alignment.md + - Reference triple alignment: metrics/reference_triple_alignment.md + - Stats counts: metrics/stats_counts.md + - Experiments: + - Reproduce MovieKG: reproduce.md + - Other: + - Adoption (integrating existing pipelines): adoption.md + - View/UI: view.md + - Building docs: create-docs.md + - Migration (renamed): migration.md diff --git a/pyproject.toml b/pyproject.toml index 09e3183..8705d87 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -21,8 +21,8 @@ dependencies = [ "rdflib>=6.0.0", "matplotlib>=3.5.0", "networkx>=2.8.0", - "transformers>=4.50.0", - "sentence_transformers>=4.1.0", + # ML stack (torch/transformers) is intentionally NOT in base deps. + # Install via `pip/uv pip install ".[ml]"` plus the desired torch index (CPU/CUDA). "pulp>=3.3.0", "pytest>=8.4.2", "dotenv>=0.9.9", @@ -31,14 +31,72 @@ dependencies = [ "jsonpath-ng>=1.7.0", "SPARQLWrapper>=2.0.0", "redis>=7.0.0", + "kgcore @ git+https://github.com/Vehnem/kgcore.git", + "streamlit-elements>=0.1.0", + "uvicorn>=0.41.0", + "fastapi>=0.135.1", + "tqdm>=4.67.1", + "scipy>=1.16.2", ] [project.optional-dependencies] dev = ["pytest", "pytest-mock", "pytest-cov", "ruff", "black"] +docs = [ + "mkdocs-material", + "mkdocstrings[python]", +] +cpu = [ + "torch", + "torchvision", + "torchaudio", +] +cuda = [ + "torch", + "torchvision", + "torchaudio", +] +ml = [ + "transformers>=4.50.0", + "sentence_transformers>=4.1.0", +] + +[tool.uv] +conflicts = [ + [ + { extra = "cpu" }, + { extra = "cuda" }, + ], +] + +[tool.uv.sources] +torch = [ + { index = "pytorch-cpu", extra = "cpu" }, + { index = "pytorch-cuda", extra = "cuda" }, +] +torchvision = [ + { index = "pytorch-cpu", extra = "cpu" }, + { index = "pytorch-cuda", extra = "cuda" }, +] +torchaudio = [ + { index = "pytorch-cpu", extra = "cpu" }, + { index = "pytorch-cuda", extra = "cuda" }, +] + +[[tool.uv.index]] +name = "pytorch-cpu" +url = "https://download.pytorch.org/whl/cpu" +explicit = true + +# CUDA wheels live on a separate PyTorch index. +# If you need a different CUDA version, change the URL (e.g. `cu128`, `cu126`, `cu121`). +[[tool.uv.index]] +name = "pytorch-cuda" +url = "https://download.pytorch.org/whl/cu130" +explicit = true [tool.setuptools.packages.find] where = ["src"] -include = ["kgpipe*", "kgcore*", "kgback*"] +include = ["kgpipe*"] [tool.setuptools.package-dir] "" = "src" diff --git a/rm/mcp_config.yaml b/rm/mcp_config.yaml deleted file mode 100644 index e4fc21c..0000000 --- a/rm/mcp_config.yaml +++ /dev/null @@ -1,10 +0,0 @@ -# MCP Configuration for Codex -# This file configures the MCP server for use with Codex (MCP client) - -[mcp_servers.kgbench-mcp-server] -command = "python" -args = ["/home/marvin/project/code/kgflex/src/kgbench/mcp_server.py"] -env = { "PYTHONPATH" = "/home/marvin/micromamba/envs/geneval/bin/python" } -cwd = "/home/marvin/project/code/kgflex/src/kgbench" -timeout = 30 -logLevel = "info" diff --git a/rm/mcp_server.py b/rm/mcp_server.py deleted file mode 100644 index 5ae11f5..0000000 --- a/rm/mcp_server.py +++ /dev/null @@ -1,71 +0,0 @@ -# pip install mcp -from mcp.server import FastMCP -import os - -# Create a dummy release library for demonstration -class DummyReleaseLib: - def create(self, dataset_id, version, channel, notes="", actor=None): - release_id = f"rel_{dataset_id}_{version}_{channel}_{hash(str(actor)) % 10000}" - print(f"Created release: {release_id} for dataset {dataset_id} v{version} ({channel})") - return release_id - - def publish(self, release_id, actor=None): - class PublishResult: - status = "published" - public_url = f"https://releases.example.com/{release_id}" - return PublishResult() - -your_release_lib = DummyReleaseLib() - -# Create MCP server using FastMCP -mcp = FastMCP("kgpipe-mcp-server") - -@mcp.tool() -def create_release(dataset_id: str, version: str, channel: str, notes: str = "") -> str: - """Create a new dataset release - - Args: - dataset_id: ID of the dataset - version: Version number - channel: Release channel (dev, rc, or prod) - notes: Release notes - """ - rid = your_release_lib.create(dataset_id, version, channel, notes, "user") - return f"Release created: {rid}" - -@mcp.tool() -def publish_release(release_id: str) -> str: - """Publish a release to make it public - - Args: - release_id: ID of the release to publish - """ - result = your_release_lib.publish(release_id, "user") - return f"Release published: {result.public_url}" - -@mcp.resource("policy://release") -def get_release_policy() -> str: - """Get the release policy document""" - return """# Release Policy - -## Overview -This document outlines the policy for creating and publishing dataset releases. - -## Release Channels -- **dev**: Development releases for testing -- **rc**: Release candidates for final testing -- **prod**: Production releases for general use - -## Process -1. Create a release using the create_release tool -2. Test the release thoroughly -3. Publish the release using the publish_release tool - -## Guidelines -- Always include meaningful release notes -- Test in dev channel before promoting to rc -- Only promote to prod after thorough testing -""" - -if __name__ == "__main__": - mcp.run() diff --git a/scripts/docker-virtuoso.sh b/scripts/docker-virtuoso.sh new file mode 100644 index 0000000..6ce81a0 --- /dev/null +++ b/scripts/docker-virtuoso.sh @@ -0,0 +1,10 @@ +docker run \ + --name kgpipe_virtdb \ + --interactive \ + --tty \ + --env DBA_PASSWORD=mysecret \ + --publish 1111:1111 \ + --publish 8890:8890 \ + openlink/virtuoso-opensource-7:latest + +# --volume `pwd`:/database \ diff --git a/src/kgpipe/cli/config.py b/src/kgpipe/cli/config.py index 23da51a..0a29eeb 100644 --- a/src/kgpipe/cli/config.py +++ b/src/kgpipe/cli/config.py @@ -17,6 +17,8 @@ from rich.console import Console from rich.table import Table +from kgcore.config import HOME_CONFIG_DIR + # Initialize Rich console for pretty output console = Console() @@ -35,9 +37,9 @@ def get_default_config(): def get_config_file(): """Get the path to the configuration file.""" - config_dir = Path.home() / ".kgpipe" + config_dir = HOME_CONFIG_DIR config_dir.mkdir(exist_ok=True) - return config_dir / "config.yaml" + return config_dir / "kgpipe.yaml" def load_config(): diff --git a/src/kgpipe/cli/discover.py b/src/kgpipe/cli/discover.py index 1c03cbe..f2a6531 100644 --- a/src/kgpipe/cli/discover.py +++ b/src/kgpipe/cli/discover.py @@ -16,15 +16,74 @@ discover_entry_points, discover_local_modules, get_registered_tasks, - get_registered_pipelines, - get_registered_metrics, - get_registered_evaluators, ) # Initialize Rich console for pretty output console = Console() +def _function_path(func) -> str: + return f"{func.__module__}.{func.__qualname__}" + + +def _component_name(factory) -> str: + try: + obj = factory() + return getattr(obj, "name", factory.__name__) + except Exception: + return factory.__name__ + + +def _show_component_table(title: str, items: list[tuple[str, str]]) -> None: + if not items: + return + + table = Table(title=title) + table.add_column("Name", style="cyan") + table.add_column("Function Path", style="green") + + for name, path in sorted(items, key=lambda item: item[0].lower()): + table.add_row(name, path) + + console.print(table) + console.print() + + +def _show_discovered_components() -> None: + """Display discovered components with names and function paths.""" + from kgpipe.common.registry import Registry + + tasks = get_registered_tasks() + pipelines = Registry.list("pipeline") + metrics = Registry.list("metric") + evaluators = Registry.list("evaluator") + + console.print() + console.print("[bold blue]Discovered Components:[/bold blue]") + console.print( + f"Tasks: {len(tasks)}, Pipelines: {len(pipelines)}, " + f"Metrics: {len(metrics)}, Evaluators: {len(evaluators)}" + ) + console.print() + + _show_component_table( + "Tasks", + [(task.name, _function_path(task.function)) for task in tasks], + ) + _show_component_table( + "Pipelines", + [(_component_name(factory), _function_path(factory)) for factory in pipelines], + ) + _show_component_table( + "Metrics", + [(_component_name(factory), _function_path(factory)) for factory in metrics], + ) + _show_component_table( + "Evaluators", + [(_component_name(factory), _function_path(factory)) for factory in evaluators], + ) + + def discover_package(package_name: str) -> bool: """ Discover and register components from a package by name. @@ -85,7 +144,7 @@ def discover_module_path(module_path: str) -> bool: "module_paths", multiple=True, type=click.Path(exists=True), - help="Path(s) to module directory or file to discover", + help="Path(s) to module directory or file to discover (searches recursively)", ) @click.option( "--all", @@ -170,22 +229,5 @@ def discover_cmd( # Show discovered components if requested if show_results: - console.print() - console.print("[bold blue]Discovered Components:[/bold blue]") - - tasks = get_registered_tasks() - pipelines = get_registered_pipelines() - metrics = get_registered_metrics() - evaluators = get_registered_evaluators() - - table = Table(title="Registered Components") - table.add_column("Type", style="cyan") - table.add_column("Count", style="green") - - table.add_row("Tasks", str(len(tasks))) - table.add_row("Pipelines", str(len(pipelines))) - table.add_row("Metrics", str(len(metrics))) - table.add_row("Evaluators", str(len(evaluators))) - - console.print(table) + _show_discovered_components() diff --git a/src/kgpipe/cli/eval.py b/src/kgpipe/cli/eval.py index 9253bf8..7478ae3 100644 --- a/src/kgpipe/cli/eval.py +++ b/src/kgpipe/cli/eval.py @@ -7,6 +7,8 @@ import json import sys +import traceback +import os from pathlib import Path from typing import List, Optional @@ -48,7 +50,8 @@ def show_evaluation_results(evaluation_report): console.print(table) console.print("") - + else: + console.print("No metrics available") def save_evaluation_results(evaluation_report, output_file: str): """Save evaluation results to file.""" @@ -97,12 +100,18 @@ def save_evaluation_results(evaluation_report, output_file: str): default='json', help="Output format for results" ) +@click.option( + "--metric-config", + "-c", + type=click.Path(exists=True), + help="Path to metric config file" +) @click.option( # flag "--debug", ) @click.pass_context -def eval_cmd(ctx: click.Context, target: List[str], ground_truth: Optional[str], aspects: tuple, metrics: tuple, output: Optional[str], format: str, debug: Optional[str]): +def eval_cmd(ctx: click.Context, target: List[str], ground_truth: Optional[str], aspects: tuple, metrics: tuple, output: Optional[str], format: str, metric_config: Optional[str], debug: Optional[str]): """ Evaluate a knowledge graph against ground truth. @@ -123,34 +132,42 @@ def eval_cmd(ctx: click.Context, target: List[str], ground_truth: Optional[str], except ValueError: kg_format = DataFormat.JSON # Default to JSON + ontology_path = os.environ.get("ONTOLOGY_PATH", None) + if ontology_path is None: + raise ValueError("ONTOLOGY_PATH is not set") + from rdflib import Graph + ontology_graph = Graph() + ontology_graph.parse(ontology_path, format="turtle") + target_kg = KG( id=str(target_path), name=target_path.stem, path=target_path, - format=kg_format + format=kg_format, + ontology_graph=ontology_graph ) # Load ground truth if provided - reference_kg = None - if ground_truth: - ground_truth_path = Path(ground_truth) - console.print(f"[dim]Loading ground truth from:[/dim] {ground_truth_path}") + # reference_kg = None + # if ground_truth: + # ground_truth_path = Path(ground_truth) + # console.print(f"[dim]Loading ground truth from:[/dim] {ground_truth_path}") - ref_format_ext = ground_truth_path.suffix.lower().lstrip('.') - try: - ref_kg_format = DataFormat(ref_format_ext) - except ValueError: - ref_kg_format = DataFormat.JSON + # ref_format_ext = ground_truth_path.suffix.lower().lstrip('.') + # try: + # ref_kg_format = DataFormat(ref_format_ext) + # except ValueError: + # ref_kg_format = DataFormat.JSON - reference_kg = KG( - id=str(ground_truth_path), - name=ground_truth_path.stem, - path=ground_truth_path, - format=ref_kg_format - ) + # reference_kg = KG( + # id=str(ground_truth_path), + # name=ground_truth_path.stem, + # path=ground_truth_path, + # format=ref_kg_format + # ) # Set up evaluation configuration - config = EvaluationConfig() + config = EvaluationConfig(metric_config_path=metric_config) # Set aspects if specified if aspects: @@ -169,7 +186,7 @@ def eval_cmd(ctx: click.Context, target: List[str], ground_truth: Optional[str], # Run evaluation evaluator = Evaluator(config) - evaluation_report = evaluator.evaluate(target_kg) + evaluation_report = evaluator.evaluate(target_kg, config) # Display results console.print(f"[green]✓ Evaluation completed![/green]") @@ -181,15 +198,16 @@ def eval_cmd(ctx: click.Context, target: List[str], ground_truth: Optional[str], console.print(f"[dim]Results saved to:[/dim] {output}") except Exception as e: + print(traceback.format_exc()) console.print(f"[red]✗ Evaluation failed:[/red] {e}") if ctx.obj["verbose"]: console.print_exception() sys.exit(1) - if debug: - from kgpipe.meta.systemgraph import SYS_KG - # if has method asGraph, serialize it - if hasattr(SYS_KG, "asGraph"): - print(SYS_KG.asGraph().serialize(format="turtle")) - else: - print("SYS_KG does not have asGraph method") \ No newline at end of file + # if debug: + # from kgpipe.meta.systemgraph import SYS_KG + # # if has method asGraph, serialize it + # if hasattr(SYS_KG, "asGraph"): + # print(SYS_KG.asGraph().serialize(format="turtle")) + # else: + # print("SYS_KG does not have asGraph method") \ No newline at end of file diff --git a/src/kgpipe/cli/eval_new.py b/src/kgpipe/cli/eval_new.py new file mode 100644 index 0000000..a86f141 --- /dev/null +++ b/src/kgpipe/cli/eval_new.py @@ -0,0 +1,387 @@ +import click +from rich.console import Console +from rich.table import Table +from typing import List, Optional, Sequence, Any +import json +from pathlib import Path +import codecs + +from kgpipe_eval.metrics.statistics import CountMetric +from kgpipe_eval.metrics.duplicates import DuplicateMetric +from kgpipe_eval.metrics.entity_alignment import EntityAlignmentMetric +from kgpipe_eval.metrics.triple_alignment import TripleAlignmentMetric +from kgpipe_eval.metrics.consistency_violations import DisjointDomainMetric, DomainMetric, RangeMetric, RelationDirectionMetric, DatatypeMetric, DatatypeFormatMetric +from kgpipe_eval.utils.kg_utils import KgManager +from kgpipe_eval.utils.metric_utils import MeasurementKey, parse_eval_results, write_eval_csv +from kgpipe_eval.config.manager import load_metric_configs, write_default_config_yaml +from kgpipe_eval.evaluator import Evaluator +# from kgpipe_eval.metrics.semantic import OntologyClassCoverageMetric, OntologyRelationCoverageMetric, OntologyNamespaceCoverageMetric +# from kgpipe_eval.metrics.reference import PrecisionMetric, RecallMetric, F1ScoreMetric +# from kgpipe_eval.metrics.efficiency import RuntimeMetric, MemoryUsageMetric, CostMetric +# from kgpipe_eval.metrics.quality import QualityMetric +# from kgpipe_eval.metrics.completeness import CompletenessMetric +# from kgpipe_eval.metrics.accuracy import AccuracyMetric + +console = Console() + +_DEFAULT_EVAL_RESULTS_ALLOWLIST = { + "DuplicateMetric": { + "duplicates": "number", + "entity_count": "number", + "duplicates_ratio": "percentage", + } +} + +def _measurement_key_to_col(k: MeasurementKey) -> str: + return f"{k.metric}__{k.measurement}__{k.unit}" + + +def _col_to_measurement_key(col: str) -> MeasurementKey: + parts = col.split("__") + if len(parts) != 3 or not all(parts): + raise click.ClickException( + f"Invalid selection '{col}'. Expected format: ____" + ) + return MeasurementKey(metric=parts[0], measurement=parts[1], unit=parts[2]) + + +def _available_eval_result_keys(paths: list[Path]) -> list[MeasurementKey]: + keys: set[MeasurementKey] = set() + for p in paths: + flat = parse_eval_results(p) + keys.update(flat.keys()) + return sorted(keys, key=_measurement_key_to_col) + +def _decode_single_char_delimiter(delimiter: str) -> str: + """ + Allow passing common escape sequences like '\\t' for tab. + """ + decoded = codecs.decode(delimiter, "unicode_escape") if "\\" in delimiter else delimiter + if len(decoded) != 1: + raise click.ClickException( + f"--delimiter must be a single character (you passed {delimiter!r} -> {decoded!r})" + ) + return decoded + + +def _available_metric_instances() -> dict[str, Any]: + # Keep this explicit until the metrics package is more complete/stable. + return { + "CountMetric": CountMetric(), + "DuplicateMetric": DuplicateMetric(), + "EntityAlignmentMetric": EntityAlignmentMetric(), + "TripleAlignmentMetric": TripleAlignmentMetric(), + "DisjointDomainMetric": DisjointDomainMetric(), + "DomainMetric": DomainMetric(), + "RangeMetric": RangeMetric(), + "RelationDirectionMetric": RelationDirectionMetric(), + "DatatypeMetric": DatatypeMetric(), + "DatatypeFormatMetric": DatatypeFormatMetric(), + } + +def _normalize_key(k: str) -> str: + return k.strip().lower().replace("-", "_") + + +def _metric_key(metric: Any) -> str: + return getattr(metric, "key", metric.__class__.__name__) + + +def _metric_description(metric: Any) -> str: + cls = metric.__class__ + desc = getattr(cls, "description", None) + if desc: + return str(desc).strip() + if cls.__doc__: + return cls.__doc__.strip().split("\n")[0] + compute_doc = cls.compute.__doc__ + if compute_doc: + return compute_doc.strip().split("\n")[0] + return "—" + + +def _render_available_metrics_table() -> None: + metrics = _available_metric_instances() + table = Table(title="Available metrics (eval-new)") + table.add_column("Name", style="cyan") + table.add_column("Description", style="green") + + for name in sorted(metrics.keys()): + table.add_row(name, _metric_description(metrics[name])) + + console.print(table) + console.print( + f"[dim]{len(metrics)} metric(s). " + "Pass one or more with `eval-new run -m `.[/dim]" + ) + + +def _build_confs_for_selected_metrics( + selected_metric_instances: list[Any], + loaded_confs: dict[str, Any], +) -> dict[str, Any]: + """ + Convert configs loaded from YAML (keyed by YAML metric id) into a dict keyed by + metric class name / `.key` (what Evaluator uses). + """ + confs_by_norm = {_normalize_key(k): v for k, v in loaded_confs.items()} + out: dict[str, Any] = {} + + # Common YAML → class-name aliases + alias_to_metric_key: dict[str, str] = { + "duplicates": "DuplicateMetric", + "duplicate": "DuplicateMetric", + "entity_align": "EntityAlignmentMetric", + "entity_alignment": "EntityAlignmentMetric", + } + + for metric in selected_metric_instances: + mkey = _metric_key(metric) + norm_mkey = _normalize_key(mkey) + norm_cls = _normalize_key(metric.__class__.__name__) + + # Try common YAML ids derived from metric names + base_from_key = norm_mkey.replace("_metric", "").replace("metric", "") + base_from_cls = norm_cls.replace("_metric", "").replace("metric", "") + + cfg = ( + confs_by_norm.get(norm_mkey) + or confs_by_norm.get(norm_cls) + or confs_by_norm.get(_normalize_key(alias_to_metric_key.get(norm_mkey, ""))) + or confs_by_norm.get(_normalize_key(alias_to_metric_key.get(norm_cls, ""))) + or confs_by_norm.get(base_from_key) + or confs_by_norm.get(base_from_cls) + # plural fallback (e.g. DuplicateMetric -> duplicates) + or confs_by_norm.get(f"{base_from_key}s") + or confs_by_norm.get(f"{base_from_cls}s") + ) + + if cfg is not None: + out[mkey] = cfg + out[metric.__class__.__name__] = cfg + + return out + + + +def _render_results_table(kg_path: str, metric_key: str, measurements: Sequence[Any], summary: Optional[str]) -> None: + table = Table(title=f"{Path(kg_path).name} — {metric_key}") + table.add_column("Measurement", style="cyan") + table.add_column("Value", style="green") + table.add_column("Unit", style="magenta") + + for m in measurements: + unit = getattr(m, "unit", None) + value = getattr(m, "value", None) + name = getattr(m, "name", None) + table.add_row(str(name), json.dumps(value, ensure_ascii=False, default=str) if not isinstance(value, (str, int, float, bool)) else str(value), "" if unit is None else str(unit)) + + console.print(table) + if summary: + console.print(f"[dim]{summary}[/dim]") + console.print("") + + +def _results_to_json_rows(kg_path: str, metric_key: str, measurements: Sequence[Any], summary: Optional[str]) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for m in measurements: + rows.append( + { + "kg_path": kg_path, + "metric": metric_key, + "measurement": getattr(m, "name", None), + "value": getattr(m, "value", None), + "unit": getattr(m, "unit", None), + "summary": summary, + } + ) + return rows + + +@click.group(name="eval-new") +def eval_new_cmd() -> None: + """ + Evaluation commands for the new metric framework. + """ + + +@eval_new_cmd.command(name="list") +def list_metrics_cmd() -> None: + """ + List all metrics available to `eval-new run`. + """ + _render_available_metrics_table() + + +@eval_new_cmd.command(name="run") +@click.argument("kg_paths", nargs=-1, type=click.Path(exists=True)) +@click.option( + "--config", + "-c", + type=click.Path(exists=True), + help="Path to metric config file", +) +@click.option( + "--metrics", + "-m", + multiple=True, + type=click.Choice(sorted(_available_metric_instances().keys())), + help="Metrics to compute", +) +@click.option( + "--output", + "-o", + type=click.Path(dir_okay=False), + help="Write results to a JSON file (list of measurement rows).", +) +@click.pass_context +def run_cmd(ctx: click.Context, kg_paths: List[str], config: Optional[str], metrics: tuple, output: Optional[str]) -> None: + """ + Compute selected metrics for one or more KGs. + + KG_PATHS: one or more RDF files/directories that RDFLib can parse. + """ + metric_instances = _available_metric_instances() + selected_metrics = list(metrics) if metrics else list(metric_instances.keys()) + + unknown = [m for m in selected_metrics if m not in metric_instances] + if unknown: + raise click.ClickException(f"Unknown metrics: {', '.join(unknown)}") + + loaded_metric_confs: dict[str, Any] = {} + if config: + loaded_metric_confs = load_metric_configs(config) + + all_rows: list[dict[str, Any]] = [] + + for kg_path in kg_paths: + console.print(f"[bold blue]Evaluating:[/bold blue] {kg_path}") + kg_graph = KgManager.load_kg_from_path(Path(kg_path)) + try: + selected_metric_instances = [metric_instances[k] for k in selected_metrics] + confs = _build_confs_for_selected_metrics(selected_metric_instances, loaded_metric_confs) + + results = Evaluator().run(kg=kg_graph, metrics=selected_metric_instances, confs=confs) + for res in results: + metric_key = _metric_key(res.metric) + _render_results_table(kg_path, metric_key, res.measurements, getattr(res, "summary", None)) + all_rows.extend(_results_to_json_rows(kg_path, metric_key, res.measurements, getattr(res, "summary", None))) + finally: + KgManager.unload_kg(kg_graph) + + if output: + out_path = Path(output) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text(json.dumps(all_rows, indent=2, ensure_ascii=False, default=str) + "\n", encoding="utf-8") + console.print(f"[green]✓ Saved results to[/green] {output}") + + +@eval_new_cmd.command(name="init-config") +@click.argument("output_path", type=click.Path(dir_okay=False), default="eval.default.yaml", required=False) +def init_config_cmd(output_path: str) -> None: + """ + Write a default metric-config template YAML to OUTPUT_PATH. + """ + out = write_default_config_yaml(output_path) + console.print(f"[green]✓ Wrote default config to[/green] {out}") + + +@eval_new_cmd.command(name="to-csv") +@click.argument("eval_json_paths", nargs=-1, type=click.Path(exists=True, dir_okay=False)) +@click.option( + "--glob", + "glob_pattern", + type=str, + help="Optional glob pattern (expanded by the shell) for eval_results.json files.", +) +@click.option( + "--select", + "-s", + "selected_cols", + multiple=True, + help="Select columns to include (repeatable). Format: ____. If omitted, defaults are used.", +) +@click.option( + "--list-keys", + is_flag=True, + help="Print available column keys found in the inputs and exit.", +) +@click.option( + "--round", + "round_ndigits", + type=int, + default=None, + help="Round float values to N decimal digits before writing CSV.", +) +@click.option( + "--delimiter", + "delimiter", + type=str, + default=",", + show_default=True, + help="CSV delimiter character (supports escapes like '\\t').", +) +@click.option( + "--output", + "-o", + "output_csv", + type=click.Path(dir_okay=False), + required=True, + help="Path to write the CSV table to.", +) +def to_csv_cmd( + eval_json_paths: List[str], + glob_pattern: Optional[str], + selected_cols: tuple[str, ...], + list_keys: bool, + round_ndigits: Optional[int], + delimiter: str, + output_csv: str, +) -> None: + """ + Convert one or more `eval_results.json` files into a CSV table. + + The CSV contains one row per (pipeline, stage), derived from file paths like: + `/stage_/eval_results.json` + + Columns follow: `____`. + """ + paths: list[Path] = [Path(p) for p in eval_json_paths] + if glob_pattern: + paths.extend(sorted(Path().glob(glob_pattern))) + + if not paths: + raise click.ClickException("No input files provided. Pass paths or --glob.") + + available = _available_eval_result_keys(paths) + console.print("[bold]Available keys in inputs:[/bold]") + for k in available: + console.print(f" - {_measurement_key_to_col(k)}") + + if list_keys: + return + + allowlist = _DEFAULT_EVAL_RESULTS_ALLOWLIST + if selected_cols: + available_cols = {_measurement_key_to_col(k) for k in available} + missing = [c for c in selected_cols if c not in available_cols] + if missing: + raise click.ClickException( + "Selected keys not found in inputs:\n" + "\n".join(f"- {m}" for m in missing) + ) + + allowlist = {} + for c in selected_cols: + k = _col_to_measurement_key(c) + allowlist.setdefault(k.metric, {})[k.measurement] = k.unit + + out_path = Path(output_csv) + delimiter = _decode_single_char_delimiter(delimiter) + write_eval_csv( + paths, + out_path=out_path, + allowlist=allowlist, + delimiter=delimiter, + round_ndigits=round_ndigits, + ) + console.print(f"[green]✓ Wrote CSV to[/green] {out_path}") \ No newline at end of file diff --git a/src/kgpipe/cli/exec.py b/src/kgpipe/cli/exec.py new file mode 100644 index 0000000..135aed2 --- /dev/null +++ b/src/kgpipe/cli/exec.py @@ -0,0 +1,50 @@ +# Exec pipeline + +import click +from pathlib import Path + +@click.command() +@click.argument("pipeline", type=str) +@click.option( + "-c", + "--config", + type=click.Path(path_type=Path, exists=True, dir_okay=False), + required=False, + help="Path to the config file.", +) +@click.option( + "--config-json", + type=str, + required=False, + help="JSON string of the config.", +) +@click.option( + "--discover", + type=click.Path(path_type=Path, exists=True, file_okay=False), + default=None, + help="Directory to discover additional packages or modules.", +) +@click.option( + "--mode", + type=click.Choice(['local', 'docker', 'swarm']), + default="local", + help="Execution mode.", +) +def exec_cmd(pipeline: str, config: Path, discover: Path | None): + """ + Execute a pipeline. + + PIPELINE: Name of the pipeline to execute + CONFIG: Path to the config file + DISCOVER: Path to the directory to discover additional packages or modules + """ + + if mode == "local": + execute_pipeline_local(pipeline, config, discover) + elif mode == "docker": + execute_pipeline_docker(pipeline, config, discover) + elif mode == "swarm": + execute_pipeline_swarm(pipeline, config, discover) + else: + raise ValueError(f"Invalid mode: {mode}") +# TODO implement \ No newline at end of file diff --git a/src/kgpipe/cli/list.py b/src/kgpipe/cli/list.py index df0ad90..daa3a6f 100644 --- a/src/kgpipe/cli/list.py +++ b/src/kgpipe/cli/list.py @@ -45,14 +45,18 @@ def show_registered_tasks(format: str = "table") -> None: tasks = get_registered_tasks() for task in tasks: - table.add_row( - task.name, - ", ".join(getattr(task, 'category', [])), - getattr(task, 'description', 'N/A'), - str(getattr(task, 'input_spec', 'N/A')), - str(getattr(task, 'output_spec', 'N/A')), - "/".join(function_location(task.function).split(".")[:-1]) - ) + try: + table.add_row( + task.name, + ", ".join(getattr(task, 'category', [])), + getattr(task, 'description', 'N/A'), + str(getattr(task, 'input_spec', 'N/A')), + str(getattr(task, 'output_spec', 'N/A')), + "/".join(function_location(task.function).split(".")[:-1]) + ) + except Exception as e: + print(f"Error adding task {task.name}: {e}") + continue if format == "table": console.print(table) diff --git a/src/kgpipe/cli/main.py b/src/kgpipe/cli/main.py index 0cdf0eb..0c3692a 100644 --- a/src/kgpipe/cli/main.py +++ b/src/kgpipe/cli/main.py @@ -20,7 +20,9 @@ from .clean import clean_cmd from .task import task_cmd from .discover import discover_cmd - +from .eval_new import eval_new_cmd +from .exec import exec_cmd +# from .rank import rank_cmd # Initialize Rich console for pretty output console = Console() @@ -81,7 +83,9 @@ def cli(ctx: click.Context, config: Optional[str], verbose: bool, quiet: bool): cli.add_command(clean_cmd) cli.add_command(task_cmd) cli.add_command(discover_cmd) - +cli.add_command(eval_new_cmd) +cli.add_command(exec_cmd) +# cli.add_command(rank_cmd) if __name__ == "__main__": cli() \ No newline at end of file diff --git a/src/kgpipe/cli/show.py b/src/kgpipe/cli/show.py index 12a9b8f..a8f3b4e 100644 --- a/src/kgpipe/cli/show.py +++ b/src/kgpipe/cli/show.py @@ -16,6 +16,10 @@ from kgpipe.common.discovery import ( discover_entry_points, find_task_by_name, find_pipeline_by_name ) +from kgpipe.evaluation.aspects.reference import ReferenceConfig +from kgpipe.evaluation.aspects.semantic import SemanticConfig +from kgpipe.evaluation.aspects.statistical import StatisticalConfig +from kgpipe.evaluation.util import get_metric_config_template # Initialize Rich console for pretty output console = Console() @@ -145,8 +149,22 @@ def show_task_details(task_name: str): console.print(table) -@click.command() -@click.argument("item", type=str) +def show_metric_config_templates(): + """Show YAML templates for metric config models.""" + templates = [ + ("ReferenceConfig", get_metric_config_template(ReferenceConfig)), + ("StatisticalConfig", get_metric_config_template(StatisticalConfig)), + ("SemanticConfig", get_metric_config_template(SemanticConfig)), + ] + + for idx, (_, template) in enumerate(templates): + if idx > 0: + click.echo("---") + click.echo(template.rstrip()) + + +@click.group(name="show", invoke_without_command=True) +@click.argument("item", type=str, required=False) @click.option( "--type", "-t", @@ -154,12 +172,25 @@ def show_task_details(task_name: str): help="Type of item to show" ) @click.pass_context -def show_cmd(ctx: click.Context, item: str, type: Optional[str]): +def show_cmd(ctx: click.Context, item: Optional[str], type: Optional[str]): """ Show detailed information about an item. - ITEM: Name or path of the item to show details for + ITEM: Name or path of the item to show details for. """ + if ctx.invoked_subcommand: + return + + if not item: + console.print(ctx.get_help()) + return + + # Keep legacy `kgpipe show ` behavior while supporting + # `kgpipe show metric-config-templates`. + if item == "metric-config-templates": + show_metric_config_templates() + return + # Auto-detect type if not specified if not type: if item.endswith('.yaml') or item.endswith('.yml'): @@ -189,4 +220,10 @@ def show_cmd(ctx: click.Context, item: str, type: Optional[str]): elif type == "task": show_task_details(item) else: - console.print(f"[red]Unknown type:[/red] {type}") \ No newline at end of file + console.print(f"[red]Unknown type:[/red] {type}") + + +@show_cmd.command(name="metric-config-templates") +def show_metric_config_templates_cmd(): + """Show YAML templates for evaluation metric configs.""" + show_metric_config_templates() \ No newline at end of file diff --git a/src/kgpipe/common/__init__.py b/src/kgpipe/common/__init__.py index 33cf414..b800851 100644 --- a/src/kgpipe/common/__init__.py +++ b/src/kgpipe/common/__init__.py @@ -25,8 +25,9 @@ def setup_logging(log_file='app.log', level=logging.DEBUG): # Call this once at the start of your application setup_logging() +from .annotations import trace_task_run from .models import ( - Data, DataFormat, KgTask, KgTaskReport, DynamicFormat, FormatRegistry, + Data, DataFormat, BasicDataFormats, CustomDataFormats, BasicTaskCategoryCatalog, KgTask, KgTaskReport, DataSet, KG, Metric, EvaluationReport, KgPipe, TaskInput, TaskOutput ) from .registry import Registry @@ -38,8 +39,9 @@ def setup_logging(log_file='app.log', level=logging.DEBUG): ) __all__ = [ - "Data", "DataFormat", "KgTask", "KgTaskReport", "DynamicFormat", "FormatRegistry", + "Data", "DataFormat", "BasicDataFormats", "CustomDataFormats", "BasicTaskCategoryCatalog", "KgTask", "KgTaskReport", "DataSet", "KG", "Stage", "Metric", "EvaluationReport", "KgPipe", "TaskInput", "TaskOutput", + "trace_task_run", "Registry", "get_docker_volume_bindings", "remap_data_path_for_container", "discover_entry_points", "get_registered_tasks", "get_registered_pipelines", diff --git a/src/kgpipe/common/annotations.py b/src/kgpipe/common/annotations.py index ceb24b1..0146571 100644 --- a/src/kgpipe/common/annotations.py +++ b/src/kgpipe/common/annotations.py @@ -1,5 +1,5 @@ from rdflib import OWL, RDFS -from kgpipe.common.systemgraph import SYS_KG +from kgpipe.common.graph.systemgraph import SYS_KG, PipeKG from kgcore.api import KGProperty from typing import get_origin, get_args, Union @@ -11,7 +11,7 @@ def kg_class(description: str = ""): as a KG entity (type/Class node) once at import time. """ def decorator(cls): - print("kg_class decorator called for class: ", cls.__name__) + # print("kg_class decorator called for class: ", cls.__name__) # add owl class props = [] if description: @@ -73,4 +73,88 @@ def decorator(cls): SYS_KG.create_relation(source=prop_et.id, target=class_et.id, type=str(RDFS.domain)) return cls - return decorator \ No newline at end of file + return decorator + + + +def trace_metric_run(): pass + + +def trace_task_run(obj): + """ + Mark a task (function or `KgTask`) so that its `.run()` persists a TaskRun in `PipeKG`. + + Works with either decorator order: + + ```python + @trace_task_run + @Registry.task(...) + def my_task(...): ... + + # or + @Registry.task(...) + @trace_task_run + def my_task(...): ... + ``` + """ + setattr(obj, "trace_task_run", True) + # TODO use logger print(f"trace_task_run decorator called for object: {obj.__name__}") + return obj + +def trace_pipeline_run(obj): + """ + Mark a pipeline (function or `KgPipeline`) so that its `.run()` persists a PipelineRun in `PipeKG`. + """ + setattr(obj, "trace_pipeline_run", True) + # TODO use logger print(f"trace_pipeline_run decorator called for object: {obj.__name__}") + return obj + + +# def Track(_cls=None, *, with_timestamp: bool = False): +# """ +# Use as: +# @Track +# @Track(with_timestamp=True) +# """ +# def decorator(cls): +# class Tracked(cls): # subclass the original class +# def __init__(self, *args: Any, **kwargs: Any): +# super().__init__(*args, **kwargs) + +# inst_id = f"{cls.__name__}:{uuid4().hex[:8]}" +# setattr(self, "_kg_id", inst_id) + +# if isinstance(self, BaseModel): +# props = self.model_dump() +# else: +# props = {k: v for k, v in vars(self).items() if not k.startswith("_")} + +# if with_timestamp: +# props["timestamp"] = datetime.now(timezone.utc).isoformat() + +# SYS_KG.create_entity([cls.__name__], id=inst_id, props=props) + +# Tracked.__name__ = cls.__name__ # optional cosmetics +# Tracked.__qualname__ = cls.__qualname__ +# Tracked.__doc__ = cls.__doc__ +# return Tracked + +# return decorator if _cls is None else decorator(_cls) + +# def kg_function(fn): +# @functools.wraps(fn) +# def wrapper(*args, **kwargs): +# result = fn(*args, **kwargs) +# call_id = f"{fn.__name__}:{uuid4().hex[:8]}" +# SYS_KG.create_entity( +# ["FunctionCall"], +# id=call_id, +# props={ +# "name": fn.__name__, +# # Be careful serializing args/kwargs; this is a toy example: +# "args": repr(args), +# "kwargs": repr(kwargs), +# }, +# ) +# return result +# return wrapper diff --git a/src/kgpipe/common/config.py b/src/kgpipe/common/config.py index c26bebe..3bb4a42 100644 --- a/src/kgpipe/common/config.py +++ b/src/kgpipe/common/config.py @@ -9,6 +9,8 @@ class KgPipeConfig(KGConfig): SYS_KG_URL: str = "memory://" SYS_KG_USR: str = "" SYS_KG_PSW: str = "" + ONTOLOGY_PREFIX: str = "http://github.com/ScaDS/kgpipe/ontology/" + PIPEKG_PREFIX: str = "http://github.com/ScaDS/kgpipe/resource/" SOURCE_NAMESPACE: str = "http://kg.org/rdf/" diff --git a/src/kgpipe/common/definitions.py b/src/kgpipe/common/definitions.py deleted file mode 100644 index e503971..0000000 --- a/src/kgpipe/common/definitions.py +++ /dev/null @@ -1,110 +0,0 @@ -from dataclasses import dataclass -from pydantic import BaseModel -from typing import Optional, List, Dict, Any -# from kgcore.api.kg import KnowledgeGraph, KGProperty -# from kgcore.backend.rdf import RDFLibBackend -# from kgcore.model.rdf import RDFBaseModel -# from kgcore.system import SystemRecorder, set_default_recorder, class_, event, pydantic_model - -# TODO add annotations to the classes here - -# Types # - -type schema_format = str - -# Data # - -class DataHandle(BaseModel): - """ - A handle to a data artifact - - uri: file://example.com/data.txt - type: any/text - timestamp: 2021-01-01 - version: 1.0.0 - hash: 1234567890 - size: 1000 - """ - uri: str - type: schema_format - timestamp: Optional[str] = None - version: Optional[str] = None - hash: Optional[str] = None - size: Optional[int] = None - -# Task # - -# TODO describing entity vs entity with used values for the task -class TaskConfiguration(BaseModel): - key: str - value: Any - -class Task(BaseModel): - """ - A function that implements a task in a pipeline - - name: paris_rdf_matcher - type: entity_resolution - description: "PARIS java implementation to match two RDF files, producing CSV files..." - input: [any_rdf, any_rdf] - output: [any_csv] - """ - name: str - type: str - description: Optional[str] = None - input: List[schema_format] - output: List[schema_format] - -class TaskResult(BaseModel): - """ - The result of a task execution including configuration variables - """ - task: Task - config: Dict[str, Any] - input: List[DataHandle] - output: List[DataHandle] - status: str - duration: float - -# Evaluation # - -class Eval(BaseModel): - """ - A function that evaluates data produced by tasks - """ - name: str - type: str - description: Optional[str] = None - input: List[schema_format] - -class EvalResult(BaseModel):# - """ - Result of an evaluation function - """ - eval: Eval - config: Dict[str, Any] - input: List[DataHandle] - output: Dict[str, Any] - status: str - duration: float - -# Pipeline # - -class Pipeline(BaseModel): - """ - The plan of a pipeline - """ - tasks: List[Task] - input: List[schema_format] - output: List[schema_format] - -class PipelineResult(BaseModel): - """ - Result of a pipeline execution - """ - task_results: List[TaskResult] - eval_results: List[EvalResult] - input: List[DataHandle] - output: List[DataHandle] - status: str - duration: float \ No newline at end of file diff --git a/src/kgpipe/common/discovery.py b/src/kgpipe/common/discovery.py index d4feb68..4b8de0b 100644 --- a/src/kgpipe/common/discovery.py +++ b/src/kgpipe/common/discovery.py @@ -7,7 +7,6 @@ import importlib import importlib.util -import pkgutil import sys from pathlib import Path from typing import List, Dict, Any, Optional, Callable @@ -96,13 +95,99 @@ def discover_installed_packages() -> None: pass +def _resolve_import_root(module_path: Path) -> tuple[Path, str] | None: + """Return (sys.path root, dotted module prefix) for the longest matching sys.path entry.""" + module_path = module_path.resolve() + best_match: tuple[Path, str] | None = None + best_len = -1 + + for sys_path_entry in sys.path: + if not sys_path_entry: + continue + try: + sys_path = Path(sys_path_entry).resolve() + relative = module_path.relative_to(sys_path) + if len(sys_path.parts) > best_len: + best_match = (sys_path, ".".join(relative.parts)) + best_len = len(sys_path.parts) + except (ValueError, OSError): + continue + + return best_match + + +def _find_package_source_root(module_path: Path) -> Path | None: + """Find the directory that should be on sys.path for package imports.""" + module_path = module_path.resolve() + if module_path.is_file(): + module_path = module_path.parent + + for parent in [module_path, *module_path.parents]: + try: + relative = module_path.relative_to(parent) + except ValueError: + break + if not relative.parts: + continue + + top_package = parent / relative.parts[0] + if top_package.is_dir() and (top_package / "__init__.py").exists(): + if not (parent / "__init__.py").exists(): + return parent + + return None + + +def _module_name_for_path(py_file: Path, scan_root: Path) -> str: + """Build a dotted module name for a Python file under scan_root.""" + py_file = py_file.resolve() + scan_root = scan_root.resolve() + + import_root = _resolve_import_root(scan_root) + if import_root: + sys_path_root, _ = import_root + relative = py_file.relative_to(sys_path_root) + return ".".join(relative.with_suffix("").parts) + + package_src = _find_package_source_root(scan_root) + if package_src: + path_str = str(package_src) + if path_str not in sys.path: + sys.path.insert(0, path_str) + relative = py_file.relative_to(package_src) + return ".".join(relative.with_suffix("").parts) + + relative = py_file.relative_to(scan_root) + return ".".join(relative.with_suffix("").parts) + + +def _import_python_module(py_file: Path, module_name: str) -> None: + """Import a module by name, falling back to loading directly from a file path.""" + try: + importlib.import_module(module_name) + logger.info(f"Successfully discovered module: {module_name}") + return + except Exception as e: + logger.debug(f"Could not import {module_name}: {e}") + + try: + spec = importlib.util.spec_from_file_location(module_name, py_file) + if spec and spec.loader: + module = importlib.util.module_from_spec(spec) + sys.modules.setdefault(module_name, module) + spec.loader.exec_module(module) + logger.info(f"Successfully discovered module from file: {py_file} ({module_name})") + except Exception as e: + logger.warning(f"Error discovering module {py_file} ({module_name}): {e}") + + def discover_local_modules(module_path: Path) -> None: """ Discover components from local modules. This function can handle: - Python files (.py) - imports the file as a module - - Directories - scans for Python files and imports them + - Directories - recursively scans for Python files and imports them - Paths in sys.path - converts to relative module names Args: @@ -111,102 +196,27 @@ def discover_local_modules(module_path: Path) -> None: if not module_path.exists(): logger.warning(f"Module path does not exist: {module_path}") return - - # Resolve to absolute path + module_path = module_path.resolve() - - # Handle file paths - if module_path.is_file() and module_path.suffix == '.py': - try: - # Use importlib.util to load from file path - module_name = module_path.stem - spec = importlib.util.spec_from_file_location(module_name, module_path) - if spec and spec.loader: - module = importlib.util.module_from_spec(spec) - # Execute the module to trigger registration - spec.loader.exec_module(module) - logger.info(f"Successfully discovered module from file: {module_path}") - except Exception as e: - logger.error(f"Error discovering module from file {module_path}: {e}") + + if module_path.is_file(): + if module_path.suffix != ".py" or module_path.name == "__init__.py": + return + py_files = [module_path] + scan_root = module_path.parent + elif module_path.is_dir(): + py_files = sorted( + py_file + for py_file in module_path.rglob("*.py") + if py_file.name != "__init__.py" + ) + scan_root = module_path + else: return - - # Handle directory paths - if module_path.is_dir(): - # Check if this directory (or its resolved path) is in sys.path - path_str = str(module_path) - path_in_sys_path = path_str in sys.path - # Also check resolved paths - if not path_in_sys_path: - for sys_path_entry in sys.path: - try: - if Path(sys_path_entry).resolve() == module_path: - path_in_sys_path = True - break - except Exception: - continue - - if path_in_sys_path: - # Directory is in sys.path, so we can import modules from it directly - # Scan for Python files in the directory - for py_file in module_path.glob("*.py"): - if py_file.name == "__init__.py": - continue - try: - module_name = py_file.stem - spec = importlib.util.spec_from_file_location(module_name, py_file) - if spec and spec.loader: - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - logger.info(f"Successfully discovered module: {module_name}") - except Exception as e: - logger.warning(f"Error discovering module {py_file}: {e}") - else: - # Try to import the directory as a package - # First, check if we can find it relative to sys.path entries - for sys_path_entry in sys.path: - try: - sys_path = Path(sys_path_entry).resolve() - try: - # Check if module_path is a subdirectory of sys_path - relative_path = module_path.relative_to(sys_path) - if relative_path: - # Convert to module name - module_name = str(relative_path).replace('/', '.').replace('\\', '.') - # Try to import it - module = importlib.import_module(module_name) - logger.info(f"Successfully discovered package: {module_name}") - # Also scan for Python files in the directory - for py_file in module_path.glob("*.py"): - if py_file.name == "__init__.py": - continue - try: - file_module_name = f"{module_name}.{py_file.stem}" - file_module = importlib.import_module(file_module_name) - logger.info(f"Successfully discovered module: {file_module_name}") - except Exception as e: - logger.debug(f"Could not import {py_file.stem} from {module_name}: {e}") - return - except ValueError: - # Not a subdirectory, continue - continue - except Exception: - continue - - # If not found relative to sys.path, try direct import with absolute path conversion - # This is a fallback that may not work, but we try it - logger.debug(f"Directory {module_path} not found relative to sys.path, attempting direct scan") - for py_file in module_path.glob("*.py"): - if py_file.name == "__init__.py": - continue - try: - module_name = py_file.stem - spec = importlib.util.spec_from_file_location(module_name, py_file) - if spec and spec.loader: - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - logger.info(f"Successfully discovered module from file: {py_file}") - except Exception as e: - logger.warning(f"Error discovering module {py_file}: {e}") + + for py_file in py_files: + module_name = _module_name_for_path(py_file, scan_root) + _import_python_module(py_file, module_name) def get_registered_tasks() -> List[Any]: diff --git a/src/kgpipe/common/graph/__init__.py b/src/kgpipe/common/graph/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe/common/graph/definitions.py b/src/kgpipe/common/graph/definitions.py new file mode 100644 index 0000000..8889580 --- /dev/null +++ b/src/kgpipe/common/graph/definitions.py @@ -0,0 +1,286 @@ +from pydantic import BaseModel, ConfigDict +from typing import Optional, List, Any +from kgcore.api.kg import KGId + +# Types # + +type schema_format = str +type any_uri = str + +# Vocabulary # + +from rdflib.namespace import DefinedNamespace, Namespace + +class KGPIPE_NS(DefinedNamespace): + _fail = True + _NS = Namespace("http://github.com/ScaDS/kgpipe/") + + Task = _NS["Task"] + TaskRun = _NS["TaskRun"] + Method = _NS["Method"] + Tool = _NS["Tool"] + Implementation = _NS["Implementation"] + Parameter = _NS["Parameter"] + ParameterBinding = _NS["ParameterBinding"] + Pipeline = _NS["Pipeline"] + PipelineRun = _NS["PipelineRun"] + Artifact = _NS["Artifact"] + ArtifactType = _NS["ArtifactType"] + Schema = _NS["Schema"] + Metric = _NS["Metric"] + MetricRun = _NS["MetricRun"] + DataSpec = _NS["DataSpec"] + DataEntity = _NS["Data"] + DataType = _NS["DataType"] + ConfigSpec = _NS["ConfigSpec"] + ConfigBinding = _NS["ConfigBinding"] + + + status = _NS["status"] + started_at = _NS["started_at"] + ended_at = _NS["ended_at"] + schema = _NS["schema"] + format = _NS["format"] + name = _NS["name"] + partOfTask = _NS["partOfTask"] + hasSubtask = _NS["hasSubtask"] + description = _NS["description"] + + version = _NS["version"] + executesTask = _NS["executesTask"] + supportsTask = _NS["supportsTask"] + input = _NS["input"] + output = _NS["output"] + format = _NS["format"] + config_spec = _NS["config_spec"] + + timestamp = _NS["timestamp"] + version = _NS["version"] + hash = _NS["hash"] + size = _NS["size"] + location = _NS["location"] + data_type = _NS["data_type"] + + realisesTask = _NS["realisesTask"] + usesImplementation = _NS["usesImplementation"] + + homepage = _NS["homepage"] + implementsMethod = _NS["implementsMethod"] + usesTool = _NS["usesTool"] + hasParameter = _NS["hasParameter"] + + providesMethod = _NS["providesMethod"] + + key = _NS["key"] + alias_keys = _NS["alias_keys"] + datatype = _NS["datatype"] + required = _NS["required"] + default_value = _NS["default_value"] + allowed_values = _NS["allowed_values"] + minimum = _NS["minimum"] + maximum = _NS["maximum"] + unit = _NS["unit"] + value = _NS["value"] + binding = _NS["binding"] + + parameter = _NS["parameter"] + hasParameterBinding = _NS["hasParameterBinding"] + +# Entities # + +DataTypeEntityId = KGId +class DataTypeEntity(BaseModel): + model_config = ConfigDict(frozen=True) + ### object properties ### + format: str + data_schema: str + +DataEntityId = KGId +class DataEntity(BaseModel): + model_config = ConfigDict(frozen=True) + ### datatype properties ### + timestamp: Optional[str] = None + version: Optional[str] = None + hash: Optional[str] = None + size: Optional[int] = None + ### object properties ### + location: any_uri + data_type: DataTypeEntityId + +DataSpecEntityId = KGId +class DataSpecEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + ### object properties ### + data_type: DataTypeEntityId + +TaskEntityId = KGId +class TaskEntity(BaseModel): + model_config = ConfigDict(frozen=True) + name: str + description: Optional[str] = None + partOfTask: Optional[TaskEntityId] = None + +# TODO MethodEntityId = KGId +# TODO class MethodEntity(BaseModel): +# model_config = ConfigDict(frozen=True) +# name: str +# realizesTask: tuple[TaskEntityId, ...] + +ToolEntityId = KGId +class ToolEntity(BaseModel): + model_config = ConfigDict(frozen=True) + ### datatype properties ### + name: str + homepage: Optional[str] = None + ### object properties ### + # NOTE: these entities are used as `lru_cache` keys; must be hashable. + supportsTasks: tuple[TaskEntityId, ...] + # TODO providesMethods: tuple[MethodEntityId, ...] + +ParameterEntityId = KGId +class ParameterEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + key: str + # NOTE: these entities are used as `lru_cache` keys; must be hashable. + alias_keys: tuple[str, ...] + datatype: str + required: bool + default_value: str | int | float | bool + allowed_values: tuple[str | int | float | bool, ...] + # description: Optional[str] = None + # scope: Scope # (training/inference/io/resources) + # constraints + # minimum: Optional[float] = None + # maximum: Optional[float] = None + # unit: Optional[str] = None + +ParameterBindingEntityId = KGId +class ParameterBindingEntity(BaseModel): + value: Any + parameter: ParameterEntityId + +ConfigSpecEntityId = KGId +class ConfigSpecEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + ### object properties ### + # NOTE: these entities are used as `lru_cache` keys; must be hashable. + parameters: tuple[ParameterEntityId, ...] + +ConfigBindingEntityId = KGId +class ConfigBindingEntity(BaseModel): + name: Any + binding: tuple[ParameterBindingEntityId, ...] + +ImplementationEntityId = KGId +class ImplementationEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + version: str + ### object properties ### + input_spec: List[DataSpecEntityId] + output_spec: List[DataSpecEntityId] + realizesTask: List[TaskEntityId] + usesTool: List[ToolEntityId] + config_spec: Optional[ConfigSpecEntityId] = None + + # TODO implementsMethod: List[MethodEntityId] + # TODO interface: str + +TaskRunEntityId = KGId +class TaskRunEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + status: str + started_at: float + ended_at: float + ### object properties ### + input: List[DataEntityId] + output: List[DataEntityId] + # TODO executesTask: TaskEntityId + usesImplementation: ImplementationEntityId + hasConfigBinding: Optional[ConfigBindingEntityId] = None + +# Entity representing a task dag (not the implementation) +# class PipelineDefinitionEntity(BaseModel): +# """ +# The definition of a pipeline +# """ +# placeholder: str +# #definesPipeline: Pipeline + +PipelineStepEntityId = KGId +class PipelineStepEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + ### object properties ### + input: List[DataEntityId] + output: List[DataEntityId] + executesTask: TaskEntityId + +# TODO issue as the Graph has no ordering of the tasks +class PipelineEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + ### object properties ### + steps: List[PipelineStepEntityId] + firstStep: PipelineStepEntityId + lastStep: PipelineStepEntityId + input: List[DataEntityId] + output: List[DataEntityId] + +PipelineRunEntityId = KGId +class PipelineRunEntity(BaseModel): + """ + The result of a pipeline execution + """ + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + name: str + status: str + started_at: float + ended_at: float + ### object properties ### + hasTaskRun: List[TaskRunEntity] + # TODO usesPipelineDefinition: PipelineDefinition + # TODO runsPipeline: PipelineStepEntityId + +MetricEntityId = KGId +class MetricEntity(BaseModel): + model_config = ConfigDict(frozen=True) + ### datatype properties ### + name: str + description: Optional[str] = None + type: str # TODO should be an enum + ### object properties ### + # TODO output: List[schema_format] + # TODO hasParameter: List[ParameterId] + +MetricRunEntityId = KGId +class MetricRunEntity(BaseModel): + model_config = ConfigDict(frozen=True) + uri: Optional[str] = None + ### datatype properties ### + status: str + started_at: float + ended_at: float + value: float + details: str # TODO should be a dictionary + ### object properties ### + computedMetric: MetricEntityId + input: List[DataEntityId] diff --git a/src/kgpipe/common/graph/mapper.py b/src/kgpipe/common/graph/mapper.py new file mode 100644 index 0000000..037fd9e --- /dev/null +++ b/src/kgpipe/common/graph/mapper.py @@ -0,0 +1,213 @@ +from __future__ import annotations + +from kgpipe.common.config import config +from kgpipe.common.graph.systemgraph import PipeKG +from kgpipe.common.model.default_catalog import TaskCategory +from kgpipe.common.util import encode_string + +from kgpipe.common.graph.definitions import ( + DataEntity, + DataEntityId, + DataSpecEntity, + DataSpecEntityId, + DataTypeEntity, + DataTypeEntityId, + ImplementationEntity, + ImplementationEntityId, + PipelineRunEntity, + PipelineRunEntityId, + TaskEntity, + TaskEntityId, + TaskRunEntity, + TaskRunEntityId, + MetricRunEntity, + MetricRunEntityId, + MetricEntity, + MetricEntityId, + ParameterEntity, + ParameterEntityId, + ParameterBindingEntity, + ParameterBindingEntityId, + ConfigSpecEntity, + ConfigSpecEntityId, + ConfigBindingEntity, + ConfigBindingEntityId, +) + +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from kgpipe.common.model import ( + DataFormat, + KgData, + KgTask, + KgTaskRun, + KgPipelineRun, + KgMetricRun, + KgMetric, + ConfigurationDefinition, + Parameter, + ConfigurationProfile, + ParameterBinding, + ) + from kgpipe.evaluation.base import MetricResult + +def task_to_entity(task: "TaskCategory") -> TaskEntityId: + """Map runtime task definition to a Task entity.""" + name = task + partOfTask = None + if isinstance(task, TaskCategory): + name = task.name + if task.parent: + partOfTask = task_to_entity(task.parent) + task_entity = TaskEntity( + name=name, + partOfTask=partOfTask, + ) + return PipeKG.add_task(task_entity) + +def data_type_to_entity(data_type: DataFormat) -> DataTypeEntityId: + data_type_entity = DataTypeEntity( + format=data_type, + data_schema=data_type, + ) + return PipeKG.add_data_type(data_type_entity) + +def data_spec_to_entity(data_spec: tuple[str, DataFormat], implementation_name: str = "") -> DataSpecEntityId: + data_spec_entity = DataSpecEntity( + uri=config.PIPEKG_PREFIX + encode_string(implementation_name + "_" + data_spec[0]), + name=data_spec[0], + data_type=data_type_to_entity(data_spec[1]), + ) + return PipeKG.add_data_spec(data_spec_entity) + +def data_to_entity(data: "KgData") -> DataEntityId: + data_entity = DataEntity( + timestamp=None, # TODO + version=None, # TODO + hash=None, # TODO + size=None, # TODO + location=data.path.as_uri(), + data_type=data_type_to_entity(data.format), + ) + return PipeKG.add_data_entity(data_entity) + +def parameter_to_entity(parameter: "Parameter") -> ParameterEntityId: + parameter_entity = ParameterEntity( + key=parameter.name, + alias_keys=parameter.native_keys, + datatype=parameter.datatype, + required=parameter.required, + default_value=parameter.default_value, + allowed_values=parameter.allowed_values, + # minimum=parameter.minimum, + # maximum=parameter.maximum, + # unit=parameter.unit, + ) + return PipeKG.add_parameter(parameter_entity) + + +def config_spec_to_entity(config_spec: "ConfigurationDefinition", implementation_name: str = "") -> ConfigSpecEntityId: + if config_spec is None: + return None + parameter_entities = [parameter_to_entity(parameter) for parameter in config_spec.parameters] + config_spec_entity = ConfigSpecEntity( + name=config_spec.name, + parameters=parameter_entities, + ) + return PipeKG.add_config_spec(config_spec_entity) + +def implementation_to_entity(implementation: "KgTask") -> ImplementationEntityId: + + input_specs = [data_spec_to_entity(data_spec, implementation.name) for data_spec in implementation.input_spec.items()] + + output_specs = [data_spec_to_entity(data_spec, implementation.name) for data_spec in implementation.output_spec.items()] + + realizes_tasks = [task_to_entity(task) for task in implementation.category] + + config_spec = config_spec_to_entity(implementation.config_spec, implementation.name) + + implementation_entity = ImplementationEntity( + ### datatype properties ### + name=implementation.name, + version="1.0.0", # TODO: get version from implementation + ### object properties ### + input_spec=input_specs, + output_spec=output_specs, + realizesTask=realizes_tasks, + usesTool=[], # TODO add usesTool relations + config_spec=config_spec, + ) + return PipeKG.add_implementation(implementation_entity) + +def metric_to_entity(metric: "KgMetric") -> MetricEntityId: + metric_entity = MetricEntity( + name=metric.name, + description=metric.description, + type=metric.aspect.value, + ) + return PipeKG.add_metric(metric_entity) + + +def parameter_binding_to_entity(parameter_binding: "ParameterBinding") -> ParameterBindingEntityId: + parameter_binding_entity = ParameterBindingEntity( + value=parameter_binding.value, + parameter=parameter_to_entity(parameter_binding.parameter), + ) + return PipeKG.add_parameter_binding(parameter_binding_entity) + +def config_binding_to_entity(config_profile: "ConfigurationProfile") -> ConfigBindingEntityId: + config_binding_entity = ConfigBindingEntity( + name=config_profile.name, + binding=[parameter_binding_to_entity(binding) for binding in config_profile.bindings], + ) + return PipeKG.add_config_binding(config_binding_entity) + +def task_run_to_entity(task_run: "KgTaskRun") -> TaskRunEntityId: + + input=[data_to_entity(data) for data in task_run.inputs] + output=[data_to_entity(data) for data in task_run.outputs] + hasConfigBinding=None # TODO + usesImplementation=implementation_to_entity(task_run.task) + hasConfigBinding=config_binding_to_entity(task_run.config_profile) if task_run.config_profile else None + + print(f"hasConfigBinding: {hasConfigBinding}") + + task_run_entity = TaskRunEntity( + status=task_run.status, + started_at=task_run.start_ts, + ended_at=task_run.start_ts + task_run.duration, + input=input, + output=output, + usesImplementation=usesImplementation, + hasConfigBinding=hasConfigBinding, + ) + return PipeKG.add_task_run(task_run_entity) + +def pipeline_run_to_entity(pipeline_run: "KgPipelineRun") -> PipelineRunEntityId: + pipeline_run_entity = PipelineRunEntity( + name=pipeline_run.name, + status=pipeline_run.status, + started_at=pipeline_run.started_at, + ended_at=pipeline_run.ended_at, + ) + return PipeKG.add_pipeline_run(pipeline_run_entity) + +# TODO +# def metric_run_to_entity(metric_run: "MetricResult") -> MetricRunEntityId: +# import time +# import json +# computedMetric = metric_to_entity(metric_run.metric) +# # data_type = data_type_to_entity(DataFormat.ANY) +# input_entities = [KgData(path=metric_run.kg.path, format=DataFormat.ANY)] +# input = [data_to_entity(input_entity) for input_entity in input_entities] +# metric_run_entity = MetricRunEntity( +# status="success", +# started_at=time.time(), +# ended_at=time.time(), +# computedMetric=computedMetric, +# input=input, +# value=metric_run.value, +# details=json.dumps(metric_run.details, default=str) +# ) +# PipeKG.add_metric_run(metric_run_entity) \ No newline at end of file diff --git a/src/kgpipe/common/graph/systemgraph.py b/src/kgpipe/common/graph/systemgraph.py new file mode 100644 index 0000000..7af2736 --- /dev/null +++ b/src/kgpipe/common/graph/systemgraph.py @@ -0,0 +1,352 @@ +import functools +import ast +from uuid import uuid4 +from typing import Any, List, Optional, TYPE_CHECKING +from datetime import datetime, timezone +import hashlib +import json + +from kgcore.api import KnowledgeGraph, KGEntity, KGRelation, KGProperty, new_id +from kgcore.backend.rdf.rdf_rdflib import RDFLibBackend +from kgcore.backend.rdf.rdf_sparql import RDFSparqlBackend, SparqlAuth +from kgcore.model.rdf.rdf_base import RDFBaseModel + +from kgpipe.common.graph.definitions import ( + KGPIPE_NS, + ImplementationEntity, ImplementationEntityId, + TaskEntity, TaskEntityId, + ToolEntity, ToolEntityId, + DataEntity, DataEntityId, + DataSpecEntity, DataSpecEntityId, + DataTypeEntity, DataTypeEntityId, + MetricEntity, MetricEntityId, + MetricRunEntity, MetricRunEntityId, + TaskRunEntity, TaskRunEntityId, + ParameterEntity, ParameterEntityId, + ParameterBindingEntity, ParameterBindingEntityId, + ConfigSpecEntity, ConfigSpecEntityId, + ConfigBindingEntity, ConfigBindingEntityId, +) +from kgpipe.common.config import load_config +from kgpipe.common.util import encode_string + +if TYPE_CHECKING: + from kgpipe.common.models import KgTask, KgTaskReport + +config = load_config() +scheme, rest = config.SYS_KG_URL.split("://") + +backend = RDFLibBackend() +model = RDFBaseModel() + +try: + if scheme == "sparql": + print(f"Using SPARQL backend for system graph: {f"http://{rest}"} with http://github.com/ScaDS/kgpipe/") + backend = RDFSparqlBackend( + endpoint=f"http://{rest}", + update_endpoint=f"http://{rest}", + default_graph="http://github.com/ScaDS/kgpipe/", + auth=SparqlAuth(username=config.SYS_KG_USR, password=config.SYS_KG_PSW)) + else: + raise ValueError(f"Unsupported schema: {scheme}") +except Exception as e: + print(f"Error creating system graph: {e}") + print(f"Using RDFLib memory backend for system graph") + +SYS_KG: KnowledgeGraph = KnowledgeGraph(model=model, backend=backend) + +class PipeKG: + """ + PipeKG is the system graph for the KGpipe framework. + It is a Object Graph Mapper (OGM) for the KGpipe framework. + It is used to store the entities and relations of the KGpipe framework. + """ + + ### Core Layer Entities ### + + @staticmethod + @functools.lru_cache + def add_task(task: TaskEntity) -> TaskEntityId: + entity_id = config.PIPEKG_PREFIX + encode_string(task.name) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.Task], + properties={ + KGPIPE_NS.name: task.name, + KGPIPE_NS.description: task.description + }, + ) + if task.partOfTask: + SYS_KG.create_relation(type=KGPIPE_NS.partOfTask, source=entity_id, target=task.partOfTask) + return TaskEntityId(entity_id) + + @staticmethod + @functools.lru_cache + def add_tool(tool: ToolEntity): + entity_id = config.PIPEKG_PREFIX + encode_string(tool.name) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.Tool], + properties={ + KGPIPE_NS.name: tool.name, + KGPIPE_NS.homepage: tool.homepage, + }, + ) + for supports_task in tool.supportsTasks: + SYS_KG.create_relation(type=KGPIPE_NS.supportsTask, source=entity_id, target=supports_task) + return ToolEntityId(entity_id) + + @staticmethod + def add_implementation(implementation: ImplementationEntity): + entity_id = config.PIPEKG_PREFIX + encode_string(implementation.name) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.Implementation], + properties={ + KGPIPE_NS.name: implementation.name, + KGPIPE_NS.version: implementation.version, + }, + ) + for input_spec in implementation.input_spec: + SYS_KG.create_relation(type=KGPIPE_NS.input, source=entity_id, target=input_spec) + for output_spec in implementation.output_spec: + SYS_KG.create_relation(type=KGPIPE_NS.output, source=entity_id, target=output_spec) + for realizes_task in implementation.realizesTask: + SYS_KG.create_relation(type=KGPIPE_NS.realisesTask, source=entity_id, target=realizes_task) + if implementation.config_spec: + SYS_KG.create_relation(type=KGPIPE_NS.config_spec, source=entity_id, target=implementation.config_spec) + return ImplementationEntityId(entity_id) + + @staticmethod + def find_implementation( + name: Optional[str] = None, + # version: Optional[str] = None, + # input_spec: Optional[List[str]] = None, + # output_spec: Optional[List[str]] = None, + # realizes_task: Optional[List[str]] = None, + # has_parameter: Optional[List[str]] = None, + ) -> List[ImplementationEntity]: + entities: List[KGEntity] = SYS_KG.find_entities( + types=[str(KGPIPE_NS.Implementation)], + ) + implementations = [ImplementationEntity( + uri=entity.id, + name=entity.get_property_value(str(KGPIPE_NS.name))[0], + version=entity.get_property_value(str(KGPIPE_NS.version))[0], + input_spec=[DataSpecEntityId(neighbor.id) for neighbor in SYS_KG.get_neighbors(entity.id, str(KGPIPE_NS.input))], + output_spec=[DataSpecEntityId(neighbor.id) for neighbor in SYS_KG.get_neighbors(entity.id, str(KGPIPE_NS.output))], + realizesTask=[TaskEntityId(neighbor.id) for neighbor in SYS_KG.get_neighbors(entity.id, str(KGPIPE_NS.realisesTask))], + # hasParameter=[ParameterEntityId(neighbor.id) for neighbor in entity.get_neighbors(KGPIPE_NS.hasParameter)], + usesTool=[ToolEntityId(neighbor.id) for neighbor in SYS_KG.get_neighbors(entity.id, str(KGPIPE_NS.usesTool))], + # config_spec=ConfigSpecEntityId(entity.get_property(KGPIPE_NS.config_spec)) if entity.get_property(KGPIPE_NS.config_spec) else None, + ) for entity in entities] + if name is not None: + implementations = [impl for impl in implementations if impl.name == name] + return implementations + + ### Data Layer Entities ### + + @staticmethod + @functools.lru_cache + def add_data_spec(data_spec: DataSpecEntity): + data_spec_entity = SYS_KG.create_entity( + id=data_spec.uri if data_spec.uri else new_id(), + types=[config.ONTOLOGY_PREFIX + "DataSpec"], + properties={ + config.ONTOLOGY_PREFIX + "name": data_spec.name, + }, + ) + SYS_KG.create_relation(type=KGPIPE_NS.data_type, source=data_spec_entity.id, target=data_spec.data_type) + return DataSpecEntityId(data_spec_entity.id) + + @staticmethod + @functools.lru_cache + def add_data_entity(data_entity: DataEntity): + entity_id = config.PIPEKG_PREFIX + new_id() + data_entity_entity = SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.DataEntity], + properties={}, # TODO + # properties={ + # KGPIPE_NS.timestamp: data_entity.timestamp, + # KGPIPE_NS.version: data_entity.version, + # KGPIPE_NS.hash: data_entity.hash, + # KGPIPE_NS.size: data_entity.size, + # }, + ) + SYS_KG.create_relation(type=KGPIPE_NS.location, source=data_entity_entity.id, target=data_entity.location) + SYS_KG.create_relation(type=KGPIPE_NS.data_type, source=data_entity_entity.id, target=data_entity.data_type) + return DataEntityId(data_entity_entity.id) + + @staticmethod + @functools.lru_cache + def add_data_type(data_type: DataTypeEntity) -> DataTypeEntityId: + entity_id = config.PIPEKG_PREFIX + encode_string(data_type.format+"-"+data_type.data_schema) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.DataType], + properties={ + KGPIPE_NS.format: data_type.format, + KGPIPE_NS.schema: data_type.data_schema, + }, + ) + return DataTypeEntityId(entity_id) + + ### Pipeline Layer Entities ### + + ### Evaluation Layer Entities ### + + def add_metric(metric: MetricEntity): + pass + + ### Run Layer Entities ### + + def add_task_run(task_run: TaskRunEntity): + entity_id = config.PIPEKG_PREFIX + new_id() + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.TaskRun], + properties={ + KGPIPE_NS.status: task_run.status, + KGPIPE_NS.started_at: task_run.started_at, + KGPIPE_NS.ended_at: task_run.ended_at, + }, + ) + for input in task_run.input: + SYS_KG.create_relation(type=KGPIPE_NS.input, source=entity_id, target=input) + for output in task_run.output: + SYS_KG.create_relation(type=KGPIPE_NS.output, source=entity_id, target=output) + SYS_KG.create_relation(type=KGPIPE_NS.usesImplementation, source=entity_id, target=task_run.usesImplementation) + return TaskRunEntityId(entity_id) + + def add_metric_run(metric_run: MetricRunEntity): + pass + + ### Configuration Layer Entities ### + + @staticmethod + @functools.lru_cache + def add_parameter(parameter: ParameterEntity): + + payload = json.dumps(parameter.model_dump(mode="json"), sort_keys=True, separators=(",", ":")) + stable_hash = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16] # short suffix + entity_id = config.PIPEKG_PREFIX + encode_string(parameter.key) + "_" + stable_hash + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.Parameter], + properties={ + KGPIPE_NS.key: parameter.key, + KGPIPE_NS.alias_keys: parameter.alias_keys, + KGPIPE_NS.datatype: parameter.datatype, + KGPIPE_NS.required: parameter.required, + KGPIPE_NS.default_value: parameter.default_value, + KGPIPE_NS.allowed_values: parameter.allowed_values, + # KGPIPE_NS.minimum: parameter.minimum, + # KGPIPE_NS.maximum: parameter.maximum, + # KGPIPE_NS.unit: parameter.unit, + }, + ) + return ParameterEntityId(entity_id) + + def find_parameter(name: str): + pass + + @staticmethod + def add_parameter_binding(parameter_binding: ParameterBindingEntity): + payload = json.dumps(parameter_binding.model_dump(mode="json"), sort_keys=True, separators=(",", ":")) + stable_hash = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:16] # short suffix + entity_id = parameter_binding.parameter + "_" + stable_hash + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.ParameterBinding], + properties={ + KGPIPE_NS.value: parameter_binding.value, + }, + ) + SYS_KG.create_relation(type=KGPIPE_NS.parameter, source=entity_id, target=parameter_binding.parameter) + return ParameterBindingEntityId(entity_id) + + def find_parameter_binding(name: str): + pass + + @staticmethod + @functools.lru_cache + def add_config_spec(config_spec: ConfigSpecEntity): + entity_id = config.PIPEKG_PREFIX + encode_string(config_spec.name) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.ConfigSpec], + properties={ + KGPIPE_NS.name: config_spec.name, + }, + ) + for parameter in config_spec.parameters: + SYS_KG.create_relation(type=KGPIPE_NS.hasParameter, source=entity_id, target=parameter) + return ConfigSpecEntityId(entity_id) + + + def find_config_spec(name: str): + pass + + @staticmethod + def add_config_binding(config_binding: ConfigBindingEntity): + entity_id = config.PIPEKG_PREFIX + encode_string(config_binding.name) + SYS_KG.create_entity( + id=entity_id, + types=[KGPIPE_NS.ConfigBinding], + properties={ + KGPIPE_NS.name: config_binding.name, + }, + ) + for binding in config_binding.binding: + SYS_KG.create_relation(type=KGPIPE_NS.hasParameterBinding, source=entity_id, target=binding) + return ConfigBindingEntityId(entity_id) + + def find_config_binding(name: str): + pass + + ### Utility Functions ### + + @staticmethod + def sparql_construct(query: str): + backend : RDFSparqlBackend = SYS_KG.backend + result = backend.query_sparql(query) + return result + + @staticmethod + def _prop_value(properties: List[KGProperty], *keys: str) -> Any: + """Find a property value by exact key or key suffix.""" + for prop in properties: + if prop.key in keys: + return prop.value + for prop in properties: + for key in keys: + if prop.key.endswith(key): + return prop.value + return None + + @staticmethod + def _to_list(value: Any) -> List[str]: + """Normalize KG property values to list[str].""" + if value is None: + return [] + if isinstance(value, list): + return [str(v) for v in value] + if isinstance(value, tuple): + return [str(v) for v in value] + if isinstance(value, str): + text = value.strip() + if not text: + return [] + # Stored literals may contain Python-list string repr. + if text.startswith("[") and text.endswith("]"): + try: + parsed = ast.literal_eval(text) + except (ValueError, SyntaxError): + return [text] + if isinstance(parsed, list): + return [str(v) for v in parsed] + return [text] + return [str(value)] + + diff --git a/src/kgpipe/common/model/__init__.py b/src/kgpipe/common/model/__init__.py index 4a7f388..8335da4 100644 --- a/src/kgpipe/common/model/__init__.py +++ b/src/kgpipe/common/model/__init__.py @@ -1,2 +1,10 @@ from .pipeline import KgPipe, KgPipePlan, KgPipePlanStep -from .task import TaskInput, TaskOutput \ No newline at end of file +from .task import TaskInput, TaskOutput, KgTask, KgTaskRun +from .evaluation import Metric, EvaluationReport +from .kg import KG +from .data import Data, DataFormat, DataSet, KgData +from .default_catalog import BasicDataFormats, CustomDataFormats, BasicTaskCategoryCatalog + +__all__ = [ + "KgPipe", "KgPipePlan", "KgPipePlanStep", "KgStageReport", "KgTask", "KgTaskRun", "Metric", "EvaluationReport", "KG", "TaskInput", "TaskOutput", "KgTaskRun", "Data", "DataSet", "BasicDataFormats", "CustomDataFormats", "BasicTaskCategoryCatalog", "KgData" +] \ No newline at end of file diff --git a/src/kgpipe/common/model/configuration.py b/src/kgpipe/common/model/configuration.py index 19c430a..0c52bec 100644 --- a/src/kgpipe/common/model/configuration.py +++ b/src/kgpipe/common/model/configuration.py @@ -19,7 +19,6 @@ class ParameterType(Enum): object = "object" -@kg_class() class Parameter(BaseModel): """ Configuration parameter definition, not the actual value of the parameter in the pipeline execution @@ -34,19 +33,18 @@ class Parameter(BaseModel): # +allowed_values: any[*]? # +min/max/unit: number?/number?/string? name: str - native_keys: List[str] datatype: ParameterType - default_value: str | int | float | bool - required: bool + default_value: str | int | float | bool = field(default_factory=lambda: None) + required: bool = False + native_keys: List[str] = field(default_factory=list) # scope: Scope # (training/inference/io/resources) # constraints - allowed_values: List[str | int | float | bool] + allowed_values: List[str | int | float | bool] = field(default_factory=list) minimum: Optional[float] = None maximum: Optional[float] = None unit: Optional[str] = None -@kg_class() class ParameterBinding(BaseModel): """ Binding of a configuration parameter to a value in the pipeline execution @@ -54,12 +52,53 @@ class ParameterBinding(BaseModel): parameter: Parameter value: str | int | float | bool # TODO extend to more types? - -@kg_class() + +class ConfigurationDefinition(BaseModel): + """ + Possible configurations specification of a task + """ + name: str + description: Optional[str] = None + parameters: List[Parameter] = field(default_factory=list) + + class ConfigurationProfile(BaseModel): """ - Configuration profile definition, not the actual values of the parameters in the pipeline execution + Configuration profile specification, the actual values of the parameters in the pipeline execution """ name: str + definition: ConfigurationDefinition description: Optional[str] = None - bindings: List[ParameterBinding] = field(default_factory=list) \ No newline at end of file + bindings: List[ParameterBinding] = field(default_factory=list) + + def get_parameter(self, name: str) -> Parameter: + for parameter in self.definition.parameters: + if parameter.name == name: + return parameter + raise ValueError(f"Parameter {name} not found in configuration profile {self.name}") + + def get_parameter_binding(self, name: str) -> ParameterBinding: + for binding in self.bindings: + if binding.parameter.name == name: + return binding + raise ValueError(f"Parameter binding {name} not found in configuration profile {self.name}") + + def get_parameter_value(self, name: str) -> str | int | float | bool: + return self.get_parameter_binding(name).value + +class ConfigurationBuilder(): + def __init__(self, config_spec: ConfigurationDefinition): + self.config_spec = config_spec + self.config_profile = ConfigurationProfile(name=config_spec.name, definition=config_spec) + + def add_parameter(self, name: str, value: str | int | float | bool) -> None: + self.config_profile.bindings.append(ParameterBinding(parameter=self.get_parameter(name), value=value)) + + + +class ConfigurationMapping(BaseModel): + """ + Mapping of a configuration profile to a task implementation + """ + for_task_spec: ConfigurationDefinition + to_global_spec: ConfigurationDefinition \ No newline at end of file diff --git a/src/kgpipe/common/model/data.py b/src/kgpipe/common/model/data.py index 0b66c0b..0e6eb49 100644 --- a/src/kgpipe/common/model/data.py +++ b/src/kgpipe/common/model/data.py @@ -1,228 +1,23 @@ from __future__ import annotations -import os -import time import uuid -from abc import ABC, abstractmethod from dataclasses import dataclass, field -from datetime import datetime from enum import Enum from pathlib import Path -from typing import Any, Callable, Dict, List, Mapping, Optional, Set, Tuple, Union, Type -import json -from uuid import uuid4 -import logging -import shutil -from rdflib import Graph +from typing import Any, Dict, Optional, Union from pydantic import BaseModel, field_validator -from pydantic_core import core_schema +from .default_catalog import BasicDataFormats, CustomDataFormats -# Format descriptions for built-in formats -FORMAT_DESCRIPTIONS = { - "ttl": "Turtle RDF format", - "nquads": "N-Quads RDF format", - "json": "JSON format", - "csv": "CSV format", - "parquet": "Parquet format", - "xml": "XML format", - "rdf": "RDF format", - "jsonld": "JSON-LD format", - "txt": "Text format", - "paris_csv": "Paris CSV format", - "openrefine_json": "OpenRefine JSON format", - "limes_xml": "LIMES XML format", - "spotlight_json": "DBpedia Spotlight JSON format", - "falcon_json": "FALCON JSON format", - "ie_json": "Information Extraction JSON format", - "valentine_json": "Valentine JSON format", - "corenlp_json": "CoreNLP JSON format", - "openie_json": "OpenIE JSON format", - "agreementmaker_rdf": "AgreementMaker RDF format", - "em_json": "Entity Matching JSON format", -} +# Backward-compatible alias used across the codebase. +DataFormat = BasicDataFormats -class DataFormat(Enum): - """Built-in data formats with enum benefits.""" - # Standard formats - RDF_TTL = "ttl" - RDF_NQUADS = "nq" - RDF_NTRIPLES = "nt" - JSON = "json" - CSV = "csv" - PARQUET = "parquet" - RDF_XML = "xml" - RDF = "rdf" - RDF_JSONLD = "jsonld" - TEXT = "txt" - XML = "xml" - ANY = "any" - - # Tool-specific formats - PARIS_CSV = "paris.csv" - OPENREFINE_JSON = "openrefine.json" - LIMES_XML = "limes.xml" - SPOTLIGHT_JSON = "spotlight.json" - FALCON_JSON = "falcon.json" - VALENTINE_JSON = "valentine.json" - CORENLP_JSON = "corenlp.json" - OPENIE_JSON = "openie.json" - AGREEMENTMAKER_RDF = "agreementmaker.rdf" - - # Exchange formats - ER_JSON = "er.json" # Entity Resolution JSON format - TE_JSON = "te.json" # Text Extraction JSON format - - # LLM Tasks - JSON_ONTO_MAPPING_JSON = "json_onto_mapping.json" - - @classmethod - def from_extension(cls, extension: str) -> DataFormat: - """Get a format by file extension. If fails print available formats and raise ValueError.""" - try: - return cls(extension) - except ValueError: - print(f"Available formats: {[f.value for f in cls]}") - raise ValueError(f"Invalid format: {extension}") - - - @property - def extension(self) -> str: - """Get the file extension for this format.""" - return self.value - - @property - def description(self) -> str: - """Get the description for this format.""" - return FORMAT_DESCRIPTIONS.get(self.value, self.value) - - @property - def is_tool_specific(self) -> bool: - """Check if this is a tool-specific format.""" - tool_specific_formats = { - "paris_csv", "openrefine_json", "limes_xml", "spotlight_json", - "falcon_json", "ie_json", "valentine_json", "corenlp_json", - "openie_json", "agreementmaker_rdf", "em_json" - } - return self.value in tool_specific_formats - - def __str__(self) -> str: - return f".{self.value}" - - def __repr__(self) -> str: - return f".{self.value}" - - -class DynamicFormat: - """Dynamic format for submodules to register custom formats.""" - - def __init__(self, name: str, extension: str, description: str, is_tool_specific: bool = False): - self.name = name - self.extension = extension - self.description = description - self.is_tool_specific = is_tool_specific - - @classmethod - def __get_pydantic_core_schema__(cls, source_type: Any, handler) -> Any: - """Provide Pydantic schema for this type.""" - return core_schema.union_schema([ - core_schema.is_instance_schema(cls), - core_schema.str_schema() - ]) - - @property - def value(self) -> str: - """Get the format value (same as name for compatibility).""" - return self.name - - def __eq__(self, other) -> bool: - """Compare formats by name.""" - if isinstance(other, DynamicFormat): - return self.name == other.name - elif isinstance(other, DataFormat): - return self.name == other.value - elif isinstance(other, str): - return self.name == other - return False - - def __hash__(self) -> int: - """Hash based on name.""" - return hash(self.name) - - def __str__(self) -> str: - return f"DynamicFormat({self.name})" - - def __repr__(self) -> str: - return f"DynamicFormat(name='{self.name}', extension='{self.extension}', description='{self.description}', is_tool_specific={self.is_tool_specific})" - - -class FormatRegistry: - """Registry for managing and discovering data formats.""" - - _dynamic_formats: Dict[str, DynamicFormat] = {} - - @classmethod - def register_format(cls, name: str, extension: str, description: str, is_tool_specific: bool = False) -> DynamicFormat: - """Register a new dynamic data format.""" - if name in cls._dynamic_formats: - return cls._dynamic_formats[name] - - format_obj = DynamicFormat(name, extension, description, is_tool_specific) - cls._dynamic_formats[name] = format_obj - return format_obj - - @classmethod - def get_format(cls, name: str) -> Optional[Union[DataFormat, DynamicFormat]]: - """Get a format by name, checking built-in formats first.""" - # Try built-in formats first - try: - return DataFormat(name) - except ValueError: - # Then check dynamic formats - return cls._dynamic_formats.get(name) - - @classmethod - def list_formats(cls, tool_specific_only: bool = False) -> List[Union[DataFormat, DynamicFormat]]: - """List all registered formats.""" - formats = list(DataFormat) + list(cls._dynamic_formats.values()) - if tool_specific_only: - formats = [f for f in formats if getattr(f, 'is_tool_specific', False)] - return formats - - @classmethod - def list_standard_formats(cls) -> List[Union[DataFormat, DynamicFormat]]: - """List all standard (non-tool-specific) formats.""" - formats = list(DataFormat) + list(cls._dynamic_formats.values()) - return [f for f in formats if not getattr(f, 'is_tool_specific', False)] - - @classmethod - def list_tool_specific_formats(cls) -> List[Union[DataFormat, DynamicFormat]]: - """List all tool-specific formats.""" - formats = list(DataFormat) + list(cls._dynamic_formats.values()) - return [f for f in formats if getattr(f, 'is_tool_specific', False)] - - @classmethod - def list_rdf_formats(cls) -> List[Union[DataFormat, DynamicFormat]]: - """List all RDF formats.""" - rdf_formats = [DataFormat.RDF_TTL, DataFormat.RDF_NQUADS, DataFormat.RDF, DataFormat.RDF_JSONLD] - dynamic_rdf = [f for f in cls._dynamic_formats.values() if 'rdf' in f.name.lower() or 'ttl' in f.name.lower()] - return rdf_formats + dynamic_rdf - - @classmethod - def list_text_formats(cls) -> List[Union[DataFormat, DynamicFormat]]: - """List all text formats.""" - text_formats = [DataFormat.JSON, DataFormat.CSV, DataFormat.XML, DataFormat.TEXT] - dynamic_text = [f for f in cls._dynamic_formats.values() if f.name.lower() in ['json', 'csv', 'xml', 'txt', 'yaml']] - return text_formats + dynamic_text - - @classmethod - def clear_dynamic_formats(cls) -> None: - """Clear all dynamically registered formats (useful for testing).""" - cls._dynamic_formats.clear() +# Type alias for any format +Format = Union[DataFormat, CustomDataFormats] -# Type alias for any format -Format = Union[DataFormat, DynamicFormat] +def _format_value(fmt: Format) -> str: + return str(fmt.value) class Data(BaseModel): """Represents a data file with a specific format.""" @@ -246,16 +41,16 @@ def __init__(self, *args, **data): @classmethod def validate_format(cls, v): """Convert string format to proper Format object.""" + if isinstance(v, (DataFormat, CustomDataFormats)): + return v + if isinstance(v, Enum) and isinstance(v.value, str): + # Allow user-defined enum values for strong typing/autocomplete. + return v if isinstance(v, str): # Try to convert string to DataFormat enum try: return DataFormat(v) except ValueError: - # If it's not a DataFormat, it might be a DynamicFormat - from .models import FormatRegistry - dynamic_format = FormatRegistry.get_format(v) - if dynamic_format: - return dynamic_format raise ValueError(f"Unknown format: {v}") return v @@ -266,23 +61,19 @@ def exists(self) -> bool: def to_dict(self) -> Dict[str, str]: return { "path": str(self.path), - "format": self.format.value + "format": _format_value(self.format) } def __str__(self) -> str: - return f"Data({self.path}, {self.format.value if isinstance(self.format, DynamicFormat) else self.format})" + return f"Data({self.path}, {_format_value(self.format)})" def __eq__(self, other): """Custom equality to handle format comparison.""" if not isinstance(other, Data): return False - return (self.path == other.path and - (hasattr(self.format, 'value') and hasattr(other.format, 'value') and - self.format.value == other.format.value)) - - - + return self.path == other.path and _format_value(self.format) == _format_value(other.format) +KgData = Data @dataclass class DataSet: @@ -307,4 +98,4 @@ def exists(self) -> bool: return self.path.exists() def __str__(self) -> str: - return f"DataSet({self.name}, {self.path}, {self.format.value})" + return f"DataSet({self.name}, {self.path}, {_format_value(self.format)})" diff --git a/src/kgpipe/common/model/default_catalog.py b/src/kgpipe/common/model/default_catalog.py new file mode 100644 index 0000000..24fd368 --- /dev/null +++ b/src/kgpipe/common/model/default_catalog.py @@ -0,0 +1,203 @@ +from __future__ import annotations + +from dataclasses import dataclass +from enum import Enum +from typing import Dict, List, Optional + + +@dataclass(frozen=True) +class TaskCategory: + name: str + parent: Optional[TaskCategory] = None + description: str = "" + + + + + +class BasicTaskCategoryCatalog: + """ + Hierarchical catalog for task categories. + Supports default categories and custom category registration. + """ + entity_resolution = TaskCategory(name="EntityResolution") + entity_matching = TaskCategory(name="EntityMatching", parent=entity_resolution) + fusion = TaskCategory(name="Fusion", parent=entity_resolution) + information_extraction = TaskCategory(name="InformationExtraction") + entity_linking = TaskCategory(name="EntityLinking", parent=information_extraction) + relation_extraction = TaskCategory(name="RelationExtraction", parent=information_extraction) + relation_linking = TaskCategory(name="RelationLinking", parent=information_extraction) + data_mapping = TaskCategory(name="DataMapping") + blocking = TaskCategory(name="Blocking", parent=entity_resolution) + clustering = TaskCategory(name="Clustering", parent=entity_resolution) + + + # @dataclass(frozen=True) + # class TaskCategoryNode: + # name: str + # parent: Optional[str] = None + # description: str = "" + + # _nodes: Dict[str, TaskCategoryNode] = { + # "TaskCategory": TaskCategoryNode(name="TaskCategory", parent=None, description="Root category"), + # "EntityResolution": TaskCategoryNode(name="EntityResolution", parent="TaskCategory"), + # "Blocking": TaskCategoryNode(name="Blocking", parent="EntityResolution"), + # "EntityMatching": TaskCategoryNode(name="EntityMatching", parent="EntityResolution"), + # "Matching": TaskCategoryNode(name="Matching", parent="EntityResolution"), + # "Clustering": TaskCategoryNode(name="Clustering", parent="EntityResolution"), + # "Fusion": TaskCategoryNode(name="Fusion", parent="EntityResolution"), + # "InformationExtraction": TaskCategoryNode(name="InformationExtraction", parent="TaskCategory"), + # "EntityLinking": TaskCategoryNode(name="EntityLinking", parent="InformationExtraction"), + # "RelationExtraction": TaskCategoryNode(name="RelationExtraction", parent="InformationExtraction"), + # "RelationLinking": TaskCategoryNode(name="RelationLinking", parent="InformationExtraction"), + # "DataMapping": TaskCategoryNode(name="DataMapping", parent="TaskCategory"), + # } + + # @classmethod + # def has(cls, category: str) -> bool: + # return category in cls._nodes + + # @classmethod + # def register(cls, name: str, parent: str = "TaskCategory", description: str = "") -> None: + # if parent is not None and parent not in cls._nodes: + # raise ValueError(f"Unknown parent category: {parent}") + # cls._nodes[name] = TaskCategoryNode(name=name, parent=parent, description=description) + + # @classmethod + # def get_parent(cls, category: str) -> Optional[str]: + # node = cls._nodes.get(category) + # if node is None: + # raise ValueError(f"Unknown category: {category}") + # return node.parent + + # @classmethod + # def get_children(cls, category: str) -> List[str]: + # if category not in cls._nodes: + # raise ValueError(f"Unknown category: {category}") + # return sorted([node.name for node in cls._nodes.values() if node.parent == category]) + + # @classmethod + # def get_ancestors(cls, category: str) -> List[str]: + # if category not in cls._nodes: + # raise ValueError(f"Unknown category: {category}") + # ancestors: List[str] = [] + # cursor = cls._nodes[category].parent + # while cursor is not None: + # ancestors.append(cursor) + # cursor = cls._nodes[cursor].parent + # return ancestors + + # @classmethod + # def get_descendants(cls, category: str) -> List[str]: + # if category not in cls._nodes: + # raise ValueError(f"Unknown category: {category}") + # descendants: List[str] = [] + # queue = cls.get_children(category) + # while queue: + # current = queue.pop(0) + # descendants.append(current) + # queue.extend(cls.get_children(current)) + # return descendants + + # @classmethod + # def is_subtask_of(cls, category: str, parent: str) -> bool: + # if category not in cls._nodes or parent not in cls._nodes: + # return False + # return parent in cls.get_ancestors(category) + + # @classmethod + # def list_categories(cls) -> List[str]: + # return sorted(cls._nodes.keys()) + + +class BasicDataFormats(str, Enum): + """Framework-provided data formats with IDE autocomplete.""" + + # Standard formats + RDF_TTL = "ttl" + RDF_NQUADS = "nq" + RDF_NTRIPLES = "nt" + JSON = "json" + CSV = "csv" + PARQUET = "parquet" + RDF_XML = "xml" + RDF = "rdf" + RDF_JSONLD = "jsonld" + TEXT = "txt" + XML = "xml" + ANY = "any" + + # Tool-specific formats + PARIS_CSV = "paris.csv" + OPENREFINE_JSON = "openrefine.json" + LIMES_XML = "limes.xml" + SPOTLIGHT_JSON = "spotlight.json" + FALCON_JSON = "falcon.json" + VALENTINE_JSON = "valentine.json" + CORENLP_JSON = "corenlp.json" + OPENIE_JSON = "openie.json" + AGREEMENTMAKER_RDF = "agreementmaker.rdf" + + # Exchange formats + ER_JSON = "er.json" + TE_JSON = "te.json" + + # LLM task outputs + JSON_ONTO_MAPPING_JSON = "json_onto_mapping.json" + + @property + def extension(self) -> str: + return self.value + + @property + def description(self) -> str: + return BASIC_FORMAT_DESCRIPTIONS.get(self.value, self.value) + + @property + def is_tool_specific(self) -> bool: + return "." in self.value and self.value not in {"jsonld"} + + @classmethod + def from_extension(cls, extension: str) -> "BasicDataFormats": + try: + return cls(extension) + except ValueError as exc: + available = [f.value for f in cls] + raise ValueError(f"Invalid format: {extension}. Available formats: {available}") from exc + + +class CustomDataFormats(str, Enum): + """ + Base enum for user-defined formats. + Define project-specific formats by subclassing this enum. + """ + + @property + def extension(self) -> str: + return self.value + + +BASIC_FORMAT_DESCRIPTIONS: dict[str, str] = { + "ttl": "Turtle RDF format", + "nq": "N-Quads RDF format", + "json": "JSON format", + "csv": "CSV format", + "parquet": "Parquet format", + "xml": "XML format", + "rdf": "RDF format", + "jsonld": "JSON-LD format", + "txt": "Text format", + "paris.csv": "Paris CSV format", + "openrefine.json": "OpenRefine JSON format", + "limes.xml": "LIMES XML format", + "spotlight.json": "DBpedia Spotlight JSON format", + "falcon.json": "FALCON JSON format", + "valentine.json": "Valentine JSON format", + "corenlp.json": "CoreNLP JSON format", + "openie.json": "OpenIE JSON format", + "agreementmaker.rdf": "AgreementMaker RDF format", + "er.json": "Entity Resolution JSON format", + "te.json": "Text Extraction JSON format", + "json_onto_mapping.json": "JSON ontology mapping format", + "any": "Any format", +} \ No newline at end of file diff --git a/src/kgpipe/common/model/evaluation.py b/src/kgpipe/common/model/evaluation.py index fc304b5..be5c6d7 100644 --- a/src/kgpipe/common/model/evaluation.py +++ b/src/kgpipe/common/model/evaluation.py @@ -1,26 +1,19 @@ from __future__ import annotations -import os -import time -import uuid from abc import ABC, abstractmethod from dataclasses import dataclass, field from datetime import datetime -from enum import Enum -from pathlib import Path -from typing import Any, Callable, Dict, List, Mapping, Optional, Set, Tuple, Union, Type -import json +from typing import Any, Dict from uuid import uuid4 -import logging -import shutil -from rdflib import Graph -from pydantic import BaseModel, field_validator -from pydantic_core import core_schema + +from kgpipe.common.model.kg import KG + +# TODO move parts from kgpipe.evaluation.base to here class Metric(ABC): """Abstract base class for evaluation metrics.""" - def __init__(self, name: str, description: Optional[str] = None): + def __init__(self, name: str, description: str | None = None): self.name = name self.description = description or name @@ -45,7 +38,7 @@ class EvaluationReport: def __post_init__(self): if not self.id: - self.id = str(uuid.uuid4()) + self.id = str(uuid4().hex) def add_metric(self, name: str, value: float) -> None: """Add a metric result to the report.""" diff --git a/src/kgpipe/common/model/kg.py b/src/kgpipe/common/model/kg.py index 92d5ab6..42bdc18 100644 --- a/src/kgpipe/common/model/kg.py +++ b/src/kgpipe/common/model/kg.py @@ -1,27 +1,14 @@ from __future__ import annotations -import os -import time import uuid -from abc import ABC, abstractmethod from dataclasses import dataclass, field -from datetime import datetime -from enum import Enum from pathlib import Path -from typing import Any, Callable, Dict, List, Mapping, Optional, Set, Tuple, Union, Type -import json -from uuid import uuid4 -import logging -import shutil -from rdflib import Graph -from pydantic import BaseModel, field_validator -from pydantic_core import core_schema +from typing import Any, Dict, List, Optional +from rdflib import Graph, SKOS, RDF from .data import Format from .pipeline import KgPipePlan -from rdflib import SKOS - # TODO check if this is still needed or if we can use the KG from kgcore and only use Data and DataSet @dataclass @@ -76,4 +63,18 @@ def exists(self) -> bool: return self.path.exists() def __str__(self) -> str: - return f"KG({self.name}, {self.path}, {self.format.value})" \ No newline at end of file + return f"KG({self.name}, {self.path}, {self.format.value})" + + +# TODO wip class for central KgPipe KG entity + +@dataclass +class KgKg: + """Represents a KG for the KgPipe framework.""" + graph_data: KgData + ontology_data: KgData + # provenance: str + # @staticmethod + # def load_from_plan(plan: KgPipePlan) -> KG: + # pass + # pass \ No newline at end of file diff --git a/src/kgpipe/common/model/pipeline.py b/src/kgpipe/common/model/pipeline.py index 377cfc2..3755519 100644 --- a/src/kgpipe/common/model/pipeline.py +++ b/src/kgpipe/common/model/pipeline.py @@ -1,6 +1,7 @@ import os import time import uuid +import hashlib from abc import ABC, abstractmethod from dataclasses import dataclass, field from datetime import datetime @@ -11,14 +12,17 @@ from uuid import uuid4 import logging import shutil +from kgcore.api.kg import KGId from rdflib import Graph from pydantic import BaseModel, field_validator from pydantic_core import core_schema from .data import Data, DataFormat, DataSet, Format from .task import KgTask, KgTaskReport +from .configuration import ConfigurationProfile # from .kg import KG from kgpipe.common.annotations import kg_class +from kgpipe.common.graph.systemgraph import PipeKG class KgPipePlanStep(BaseModel): @@ -27,19 +31,25 @@ class KgPipePlanStep(BaseModel): input: List[Data] output: List[Data] -kg_class() +# kg_class() class KgPipePlan(BaseModel): """A KG pipeline plan.""" steps: List[KgPipePlanStep] seed: Optional[Data] = None source: Optional[Data] = None result: Optional[Data] = None - + + @staticmethod + def from_path(json_file: str) -> 'KgPipePlan': + with open(json_file, "r") as f: + json_data = json.load(f) + return KgPipePlan(**json_data) + # def __str__(self) -> str: # return f"KgTaskReport({self.task_name}, {self.status}, {self.duration:.2f}s)" # TODO rename to KgPipeReport -@kg_class() +# @kg_class() class KgStageReport(BaseModel): """Report of a stage execution.""" stage_name: str @@ -49,6 +59,15 @@ class KgStageReport(BaseModel): status: str error: Optional[str] = None + @staticmethod + def from_path(json_file: str) -> 'KgStageReport': + with open(json_file, "r") as f: + json_data = json.load(f) + return KgStageReport(**json_data) + +KgPipeReport = KgStageReport +KgPipelineRun = KgStageReport + # @dataclass # class Stage: # """Represents a stage in a pipeline, containing one or more tasks.""" @@ -76,7 +95,7 @@ class KgStageReport(BaseModel): # TODO rename to Pipeline -@kg_class() +# @kg_class() @dataclass class KgPipe: """A KG pipeline using a list of tasks.""" @@ -84,6 +103,7 @@ class KgPipe: tasks: List[KgTask] seed: Data data_dir: str = "" + name: str = "Unknown" data: List[Data] = field(default_factory=list) plan: KgPipePlan = field(default_factory=lambda: KgPipePlan( steps=[], @@ -107,19 +127,71 @@ def add_data(self, data: Data) -> None: self.data.append(data) - def build(self, source: Data, result: Optional[Data] = None, stable_files: bool = False) -> KgPipePlan: + def build( + self, + source: Data, + result: Optional[Data] = None, + stable_files: bool = False, + configCatalog: Optional[Mapping[str, ConfigurationProfile]] = None, + ) -> KgPipePlan: """Generate the execution plan as a list of dictionaries.""" catalog = [source] + self.data calls: List[KgPipePlanStep] = [] - def gen_file_path(task: KgTask, format_spec: Format, prefix: str = "", suffix: str = ""): - if stable_files: + def _profile_fingerprint(profile: Optional[ConfigurationProfile]) -> str: + if profile is None: + return "" + # Make it stable regardless of binding order. + bindings = [] + for b in getattr(profile, "bindings", []) or []: + param = getattr(b, "parameter", None) + pname = getattr(param, "name", None) + if pname is None: + pname = str(param) + bindings.append((str(pname), b.value)) + bindings.sort(key=lambda kv: kv[0]) + payload = json.dumps( + {"definition": getattr(getattr(profile, "definition", None), "name", None), "bindings": bindings}, + sort_keys=True, + default=str, + ) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + def _chain_hash(prev_hash: str, task_name: str, profile: Optional[ConfigurationProfile]) -> str: + fp = _profile_fingerprint(profile) + payload = json.dumps({"prev": prev_hash, "task": task_name, "profile": fp}, sort_keys=True) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + prev_hash = "0" * 64 + + def gen_file_path( + *, + task: KgTask, + format_spec: Format, + prefix: str = "", + suffix: str = "", + task_hash: Optional[str] = None, + ) -> Path: + # Backwards-compatible: stable_files without configCatalog keeps the old deterministic names. + if stable_files and configCatalog is None: return Path(self.data_dir) / f"{prefix}{task.name}{suffix}.{format_spec.extension}" - else: - return Path(self.data_dir) / f"{prefix}{task.name}.{uuid4().hex}.{format_spec.extension}" + + # If configCatalog is provided, filenames must be deterministic based on the hash chain. + if configCatalog is not None and task_hash is not None: + short = task_hash[:12] + return Path(self.data_dir) / f"{prefix}{task.name}.{short}{suffix}.{format_spec.extension}" + + # Default behavior: unique filenames. + return Path(self.data_dir) / f"{prefix}{task.name}.{uuid4().hex}.{format_spec.extension}" for idx, task in enumerate(self.tasks): + task_hash: Optional[str] = None + if configCatalog is not None: + profile = configCatalog.get(task.name) + task_hash = _chain_hash(prev_hash, task.name, profile) + prev_hash = task_hash + # Match inputs inputs = [] for input_name, format_spec in task.input_spec.items(): @@ -141,7 +213,13 @@ def gen_file_path(task: KgTask, format_spec: Format, prefix: str = "", suffix: s break else: suffix = f"_{len(outputs)}" - output_path = gen_file_path(task, format_spec, prefix=f"{idx}_", suffix=suffix) + output_path = gen_file_path( + task=task, + format_spec=format_spec, + prefix=f"{idx}_", + suffix=suffix, + task_hash=task_hash, + ) output_data = Data(path=output_path, format=format_spec) outputs.append(output_data) @@ -149,17 +227,19 @@ def gen_file_path(task: KgTask, format_spec: Format, prefix: str = "", suffix: s if len(inputs) != len(task.input_spec): missing_inputs = len(task.input_spec) - len(inputs) + catalog_str = "\n".join([str(i) for i in catalog]) raise ValueError( f"For task {task.name}: expected {task.input_spec} inputs, got {inputs}. " f"Missing {missing_inputs} inputs." - f"catalog: {"\n".join([str(i) for i in catalog])}" + f"catalog: {catalog_str}" ) elif len(outputs) != len(task.output_spec): missing_outputs = len(task.output_spec) - len(outputs) + catalog_str = "\n".join([str(i) for i in catalog]) raise ValueError( f"\nFor task {task.name}: expected {task.output_spec} outputs, got {outputs}. " f"\nMissing {missing_outputs} outputs." - f"\nCatalog: {"\n".join([str(i) for i in catalog])}" + f"\nCatalog: {catalog_str}" ) else: print(f"Adding task '{task.name}' to plan with\n\t inputs: {[str(i.path) for i in inputs]} and \n\t outputs: {[str(o.path) for o in outputs]}") @@ -193,14 +273,18 @@ def plot(self) -> None: """Plot the pipeline.""" pass - def run(self, stable_files_override: bool = False) -> List[KgTaskReport]: + def run( + self, + stable_files_override: bool = False, + configCatalog: Optional[Mapping[str, ConfigurationProfile]] = None, + ) -> List[KgTaskReport]: """Execute each task defined in the plan and collect the reports.""" if not self.plan: raise ValueError("Pipeline plan is empty. Call build() first.") self.previous_was_skipped = True - reports = [] + reports: List[KgTaskReport] = [] for task_spec in self.plan.steps: # Find the corresponding task task = next((t for t in self.tasks if t.name == task_spec.task), None) @@ -214,16 +298,33 @@ def run(self, stable_files_override: bool = False) -> List[KgTaskReport]: if not input_data.exists(): raise FileNotFoundError(f"Input file {input_data.path} does not exist") + configProfile = None + if configCatalog is not None: + configProfile = configCatalog.get(task.name) + if self.previous_was_skipped: - report = task.run(task_spec.input, task_spec.output, stable_files_override=stable_files_override) + report = task.run( + task_spec.input, + task_spec.output, + stable_files_override=stable_files_override, + configProfile=configProfile, + ) else: - report = task.run(task_spec.input, task_spec.output, stable_files_override=True) + report = task.run( + task_spec.input, + task_spec.output, + stable_files_override=True, + configProfile=configProfile, + ) if report.status != "skipped": self.previous_was_skipped = False reports.append(report) - + + # pipeline_run_entity = reports_to_pipeline_run_entity(reports, self.name) + # PipeKG.add_pipeline_run(pipeline_run_entity) + return reports def __str__(self) -> str: diff --git a/src/kgpipe/common/model/task.py b/src/kgpipe/common/model/task.py index c4c1b58..6f45559 100644 --- a/src/kgpipe/common/model/task.py +++ b/src/kgpipe/common/model/task.py @@ -4,28 +4,61 @@ # import field from dataclasses import dataclass, field from .data import Data, Format, DataFormat -from pydantic import BaseModel +from pydantic import BaseModel, Field, ConfigDict, model_validator import time import shutil - -from .configuration import Parameter, ConfigurationProfile -from kgpipe.common.annotations import kg_class +from uuid import uuid4 +import inspect +from kgpipe.common.model.default_catalog import TaskCategory +from .configuration import ( + Parameter, + ConfigurationDefinition, + ConfigurationProfile, + ParameterType, +) +from kgpipe.common.graph.systemgraph import PipeKG +from kgpipe.common.graph.mapper import task_run_to_entity type TaskName = str type TaskInput = Dict[TaskName, Data] type TaskOutput = Dict[TaskName, Data] -@kg_class() class KgTaskReport(BaseModel): """Report of a task execution.""" + model_config = ConfigDict(arbitrary_types_allowed=True) + + # Backwards-compatible identifier for persisted reports (`exec-report.json`). + # Historically we stored only the task name; newer runtime code may also attach the `KgTask`. task_name: str + task: Optional["KgTask"] = Field(default=None, exclude=True) inputs: List[Data] outputs: List[Data] start_ts: float duration: float status: str error: Optional[str] = None + config_profile: Optional[ConfigurationProfile] = None + + @model_validator(mode="before") + @classmethod + def _coerce_task_fields(cls, data): + """ + Accept both legacy reports (with `task_name`) and new runtime reports (with `task`). + """ + if not isinstance(data, dict): + return data + + # If we have a task object but no explicit task_name, derive it. + if "task_name" not in data and "task" in data and data["task"] is not None: + task_obj = data["task"] + name = getattr(task_obj, "name", None) + if name is not None: + data["task_name"] = name + + return data + +KgTaskRun = KgTaskReport class TaskStatus(Enum): """Status of a task in a pipeline.""" @@ -35,15 +68,10 @@ class TaskStatus(Enum): FAILED = "failed" SKIPPED = "skipped" -# TODO impl later for typed api -class TaskCategory(): - pass +# # TODO impl later for typed api +# class TaskCatalog(): +# pass -# TODO impl later for typed api -class TaskCatalog(): - pass - -@kg_class() @dataclass class KgTask: """Represents a task that can be executed in a pipeline.""" @@ -52,8 +80,10 @@ class KgTask: output_spec: Mapping[str, Format] function: Callable[[Dict[str, Data], Dict[str, Data]], None] description: Optional[str] = None - category: List[str] = field(default_factory=list) - config: Optional[ConfigurationProfile] = None + category: List[TaskCategory] = field(default_factory=list) + config_spec: Optional[ConfigurationDefinition] = None + tools: List[str] = field(default_factory=list) + trace_task_run: bool = False def __post_init__(self): if not self.name: @@ -65,70 +95,32 @@ def __post_init__(self): if not callable(self.function): raise ValueError("Function must be callable") - def run(self, inputs: List[Data], outputs: List[Data], stable_files_override: bool = False, configProfile: Optional[str] = None) -> KgTaskReport: + + # TODO if configProfile is not provided, use the default config profile derived from the config_spec + def run(self, inputs: List[Data], outputs: List[Data], stable_files_override: bool = False, configProfile: Optional[ConfigurationProfile] = None) -> KgTaskReport: """Execute the task with given inputs and outputs.""" start = time.time() + report: KgTaskReport try: named_inputs = self._match(inputs, self.input_spec) named_outputs = self._match(outputs, self.output_spec) - - # print(f"Running {self.name} with\n\t inputs: {[str(i.path) for i in named_inputs.values()]}\n\t outputs: {[str(o.path) for o in named_outputs.values()]}") print(f"Running {self.name} with\n\t inputs: {named_inputs}\n\t outputs: {named_outputs}") - - # Validate that all required inputs and outputs are present - if len(named_inputs) != len(self.input_spec): - missing = set(self.input_spec.keys()) - set(named_inputs.keys()) - available = {obj.format.value: obj for obj in inputs} - expected = {k: v.value for k, v in self.input_spec.items()} - raise ValueError( - f"Missing required inputs: {missing}. " - f"Expected: {expected}. " - f"Available: {[f'{obj.path} ({obj.format.value})' for obj in inputs]}" - ) - - if len(named_outputs) != len(self.output_spec): - missing = set(self.output_spec.keys()) - set(named_outputs.keys()) - available = {obj.format.value: obj for obj in outputs} - expected = {k: v.value for k, v in self.output_spec.items()} - raise ValueError( - f"Missing required outputs: {missing}. " - f"Expected: {expected}. " - f"Available: {[f'{obj.path} ({obj.format.value})' for obj in outputs]}" - ) - if stable_files_override: - for output in named_outputs.values(): - # delete the file or directory - if output.path.exists(): - if output.path.is_file(): - output.path.unlink() - elif output.path.is_dir(): - shutil.rmtree(output.path) - - # if all outputs exists skip the task - if all(output.path.exists() for output in named_outputs.values()): + self._validate_required_data(named_inputs, self.input_spec, "inputs", inputs) + self._validate_required_data(named_outputs, self.output_spec, "outputs", outputs) + self._prepare_outputs(named_outputs, stable_files_override) + + # TODO needs to check config profile changes, or maybe not + if self._should_skip(named_outputs): print(f"Skipping task {self.name} because all outputs exist") - # exit(1) - # TODO do not override old KgTaskReport - return KgTaskReport( - task_name=self.name, - inputs=list(named_inputs.values()), - outputs=list(named_outputs.values()), - start_ts=start, - duration=time.time() - start, - status="skipped", - ) + report = self._build_report(start, "skipped", list(named_inputs.values()), list(named_outputs.values()), config_profile=configProfile) + self._trace_task_run_to_pipekg(report) + return report - self.function(named_inputs, named_outputs) - - return KgTaskReport( - task_name=self.name, - inputs=list(named_inputs.values()), - outputs=list(named_outputs.values()), - start_ts=start, - duration=time.time() - start, - status="success", - ) + self._call_function(named_inputs, named_outputs, configProfile) + report = self._build_report(start, "success", list(named_inputs.values()), list(named_outputs.values()), config_profile=configProfile) + self._trace_task_run_to_pipekg(report) + return report except Exception as e: print(f"An error occurred while running the task '{self.name}'.") @@ -136,16 +128,186 @@ def run(self, inputs: List[Data], outputs: List[Data], stable_files_override: bo print(f"Exception message: {e}") import traceback traceback.print_exc() - return KgTaskReport( - task_name=self.name, - inputs=inputs, - outputs=outputs, - start_ts=start, - duration=time.time() - start, - status="failed", - error=str(e) + report = self._build_report(start, "failed", inputs, outputs, error=str(e), config_profile=configProfile) + self._trace_task_run_to_pipekg(report) + return report + + def _trace_task_run_to_pipekg(self, report: KgTaskReport) -> None: + # TODO print(f"Tracing task run to pipekg: {report}") + if not self.trace_task_run: + return + task_run_to_entity(report) + + def _call_function( + self, + named_inputs: Dict[str, Data], + named_outputs: Dict[str, Data], + config_profile: Optional[object], + ) -> None: + """ + Call the wrapped task function with or without config. + + Supported task signatures: + - fn(inputs, outputs) + - fn(inputs, outputs, config) + - fn(inputs, outputs, *, config=...) + - fn(inputs, outputs, **kwargs) (will receive config=... if provided) + """ + sig = inspect.signature(self.function) + params = sig.parameters + + accepts_var_kwargs = any(p.kind == inspect.Parameter.VAR_KEYWORD for p in params.values()) + has_config_param = "config" in params + + if config_profile is None: + # If config is required positionally/without default, fail early with a clear error. + if has_config_param: + p = params["config"] + if p.default is inspect._empty and p.kind not in ( + inspect.Parameter.VAR_POSITIONAL, + inspect.Parameter.VAR_KEYWORD, + ): + raise TypeError( + f"{self.name} requires a 'config' argument but none was provided. " + f"Pass configProfile=... to KgTask.run(), or make 'config' optional." + ) + self.function(named_inputs, named_outputs) + return + + # config is provided: pass it only if the function can accept it + if has_config_param or accepts_var_kwargs: + # If the task declares a config spec, we require a structured ConfigurationProfile. + if self.config_spec is not None and not isinstance(config_profile, ConfigurationProfile): + raise TypeError( + f"{self.name} expects configProfile to be a ConfigurationProfile " + f"because it declares config_spec='{self.config_spec.name}', " + f"got {type(config_profile).__name__}." + ) + if isinstance(config_profile, ConfigurationProfile) and self.config_spec is not None: + self._validate_config(config_profile, self.config_spec) + self.function(named_inputs, named_outputs, config=config_profile) + return + + # Function cannot accept config: ignore it + self.function(named_inputs, named_outputs) + + def _validate_config(self, config_profile: ConfigurationProfile, config_spec: ConfigurationDefinition) -> None: + if config_profile.definition.name != config_spec.name: + raise ValueError( + f"Config profile definition '{config_profile.definition.name}' does not match " + f"task config spec '{config_spec.name}'." ) + spec_by_name: Dict[str, Parameter] = {p.name: p for p in config_spec.parameters} + spec_by_key: Dict[str, Parameter] = {} + for p in config_spec.parameters: + spec_by_key[p.name] = p + for nk in p.native_keys: + spec_by_key[nk] = p + + bound: Dict[str, object] = {} + for binding in config_profile.bindings: + raw_key = binding.parameter.name + if raw_key not in spec_by_key: + raise ValueError( + f"Unknown config parameter '{raw_key}' for spec '{config_spec.name}'. " + f"Known: {sorted(spec_by_name.keys())}" + ) + param = spec_by_key[raw_key] + value = binding.value + bound[param.name] = value + + if param.datatype == ParameterType.boolean and not isinstance(value, bool): + raise TypeError(f"Config parameter '{param.name}' expects boolean, got {type(value).__name__}") + if param.datatype == ParameterType.integer and not isinstance(value, int): + raise TypeError(f"Config parameter '{param.name}' expects integer, got {type(value).__name__}") + if param.datatype == ParameterType.number and not isinstance(value, (int, float)): + raise TypeError(f"Config parameter '{param.name}' expects number, got {type(value).__name__}") + if param.datatype == ParameterType.string and not isinstance(value, str): + raise TypeError(f"Config parameter '{param.name}' expects string, got {type(value).__name__}") + + if param.allowed_values and value not in param.allowed_values: + raise ValueError( + f"Config parameter '{param.name}' value {value!r} not in allowed_values {param.allowed_values!r}" + ) + + if param.minimum is not None: + if not isinstance(value, (int, float)): + raise TypeError(f"Config parameter '{param.name}' has minimum constraint but value is not numeric") + if value < param.minimum: + raise ValueError(f"Config parameter '{param.name}' value {value} < minimum {param.minimum}") + + if param.maximum is not None: + if not isinstance(value, (int, float)): + raise TypeError(f"Config parameter '{param.name}' has maximum constraint but value is not numeric") + if value > param.maximum: + raise ValueError(f"Config parameter '{param.name}' value {value} > maximum {param.maximum}") + + missing_required: List[str] = [] + for p in config_spec.parameters: + if not p.required: + continue + if p.name in bound: + continue + if getattr(p, "default_value", None) is None: + missing_required.append(p.name) + if missing_required: + raise ValueError(f"Missing required config parameters: {missing_required}") + + + def _build_report( + self, + start_ts: float, + status: str, + inputs: List[Data], + outputs: List[Data], + error: Optional[str] = None, + config_profile: Optional[ConfigurationProfile] = None, + ) -> KgTaskReport: + return KgTaskReport( + task=self, + task_name=self.name, + inputs=inputs, + outputs=outputs, + start_ts=start_ts, + duration=time.time() - start_ts, + status=status, + error=error, + config_profile=config_profile, + ) + + def _validate_required_data( + self, + matched: Dict[str, Data], + spec: Mapping[str, Format], + label: str, + raw_items: List[Data], + ) -> None: + if len(matched) == len(spec): + return + + missing = set(spec.keys()) - set(matched.keys()) + expected = {k: v.value for k, v in spec.items()} + available = [f"{obj.path} ({obj.format.value})" for obj in raw_items] + raise ValueError( + f"Missing required {label}: {missing}. " + f"Expected: {expected}. " + f"Available: {available}" + ) + + def _prepare_outputs(self, outputs: Dict[str, Data], stable_files_override: bool) -> None: + if not stable_files_override: + return + for output in outputs.values(): + if output.path.exists(): + if output.path.is_file(): + output.path.unlink() + elif output.path.is_dir(): + shutil.rmtree(output.path) + + def _should_skip(self, outputs: Dict[str, Data]) -> bool: + return all(output.path.exists() for output in outputs.values()) + @staticmethod def _match(data: List[Data], spec: Mapping[str, Format]) -> Dict[str, Data]: """Match data objects to specification by format.""" diff --git a/src/kgpipe/common/models.py b/src/kgpipe/common/models.py index 99b0484..d3f271a 100644 --- a/src/kgpipe/common/models.py +++ b/src/kgpipe/common/models.py @@ -8,76 +8,15 @@ from __future__ import annotations -from .model.data import Data, DataFormat, DynamicFormat, DataSet, FormatRegistry +from .model.data import Data, DataFormat, DataSet +from .model.default_catalog import BasicDataFormats, CustomDataFormats, BasicTaskCategoryCatalog from .model.task import KgTask, KgTaskReport -from .model.pipeline import KgPipe, KgPipePlan, KgPipePlanStep +from .model.pipeline import KgPipe, KgPipePlan, KgPipePlanStep, KgStageReport from .model.evaluation import Metric, EvaluationReport from .model.kg import KG -from .model.task import TaskInput, TaskOutput +from .model.task import TaskInput, TaskOutput, KgTask, KgTaskRun +# from .model.evaluation import KgMetric, KgMetricRun __all__ = [ - "Data", "DataFormat", "DynamicFormat", "DataSet", "FormatRegistry", "KgTask", "KgTaskReport", "KgPipe", "KgPipePlan", "KgPipePlanStep", "Metric", "EvaluationReport", "KG", "TaskInput", "TaskOutput" + "Data", "DataFormat", "BasicDataFormats", "CustomDataFormats", "BasicTaskCategoryCatalog", "DataSet", "KgTask", "KgTaskReport", "KgPipe", "KgPipePlan", "KgPipePlanStep", "KgStageReport", "Metric", "EvaluationReport", "KG", "TaskInput", "TaskOutput", "KgTaskRun" ] - -# TODO remove this for next release -# @dataclass -# class KG: -# """Represents a knowledge graph.""" -# id: str -# name: str -# path: Path -# format: Format -# triple_count: Optional[int] = None -# entity_count: Optional[int] = None -# description: Optional[str] = None -# metadata: Dict[str, Any] = field(default_factory=dict) -# graph: Optional[Graph] = None -# data_graph: Optional[Graph] = None -# ontology_graph: Optional[Graph] = None -# plan: Optional[KgPipePlan] = None - -# def __post_init__(self): -# if not self.id: -# self.id = str(uuid.uuid4()) -# if isinstance(self.path, str): -# self.path = Path(self.path) -# if not self.name: -# raise ValueError("KG name cannot be empty") - -# def get_graph(self) -> Graph: -# if self.graph is None: -# tmp = Graph().parse(self.path) -# graph = Graph() -# for s, p, o in tmp: -# if (str(p) != str(SKOS.altLabel)): -# graph.add((s, p, o)) -# self.graph = graph -# return self.graph - -# def get_data_graph(self) -> Graph: -# return Graph() - -# def get_ontology_graph(self) -> Graph: -# # TODO derive from graph -# if self.ontology_graph is None: -# self.ontology_graph = Graph() -# return self.ontology_graph - -# def set_ontology_graph(self, graph: Graph) -> None: -# print(f"Setting ontology graph with {len(graph)} triples") -# self.ontology_graph = graph - -# def exists(self) -> bool: -# """Check if the KG file exists.""" -# return self.path.exists() - -# def __str__(self) -> str: -# return f"KG({self.name}, {self.path}, {self.format.value})" - - - - - -# # Backward compatibility aliases -# Task = KgTask -# Pipeline = KgPipe \ No newline at end of file diff --git a/src/kgpipe/common/registry.py b/src/kgpipe/common/registry.py index eb7f030..18bdcbf 100644 --- a/src/kgpipe/common/registry.py +++ b/src/kgpipe/common/registry.py @@ -1,21 +1,23 @@ # global Registry, entry-point discovery -from typing import Any, Callable +from typing import Any, Callable, List, Dict from kgpipe.common.models import KgTask, DataFormat -from kgpipe.common.systemgraph import PipeKG +# from kgpipe.common.graph.systemgraph import PipeKG +from kgpipe.common.graph.definitions import MetricEntity, TaskEntity +from kgpipe.common.model.configuration import ConfigurationDefinition +from kgpipe.common.graph.mapper import implementation_to_entity # TODO add also to system graph - - - class Registry: """ - Holds functions and python objects + Holds functions and python objects mappings KGpipe system graph """ _registry: dict[str, Any] = {} + # Generic # + @classmethod def register(cls, kind: str): def decorator(t): @@ -23,52 +25,63 @@ def decorator(t): return t return decorator + @classmethod + def get(cls, kind: str, name: str): + return cls._registry[f"{kind}:{name}"] + + @classmethod + def list(cls, kind: str): + """List all registered items of a specific kind.""" + items = [] + for key, value in cls._registry.items(): + if key.startswith(f"{kind}:"): + items.append(value) + return items + + @classmethod + def list_all(cls): + return cls._registry + + # Metric # + @classmethod def metric(cls): def decorator(t): cls._registry[f"metric:{t.__name__.lower()}"] = t + obj = t() + name = getattr(obj, 'name', None) + description = getattr(obj, 'description', None) + type = getattr(obj, 'aspect', None) + metric = MetricEntity(name=name, description=description, type=type.value if type else None) + # TODO add to system graph return t return decorator + # Task # + + @classmethod + def add_task(cls, name: str, task: KgTask): + cls._registry[f"task:{task.name}"] = task + @classmethod def task( cls, - input_spec: dict[str, DataFormat], - output_spec: dict[str, DataFormat], + input_spec: Dict[str, DataFormat], + output_spec: Dict[str, DataFormat], description: str | None = None, - category: list[str] = [] + category: List[str] = [], + config_spec: ConfigurationDefinition | None = None ) -> Callable[[Callable], KgTask]: def decorator(t): - task = KgTask(t.__name__.lower(), input_spec, output_spec, t, description, category) + task = KgTask(t.__name__.lower(), input_spec, output_spec, t, description, category, config_spec) + if getattr(t, "_trace_task_run", False): + setattr(task, "trace_task_run", True) cls._registry[f"task:{t.__name__.lower()}"] = task - PipeKG.add_task(task) + # implementation_to_entity(task) + # PipeKG.add_implementation(implementation_to_entity(task)) return task return decorator - # @classmethod - # def pipeline(cls, tasks: list[KgTask], input: Data, output: Data): - # pipeline = KgPipe(tasks, input, output) - # cls._registry[f"pipeline:{pipeline.__name__.lower()}"] = pipeline - # PipeKG.add_pipeline(pipeline) - # return pipeline - - @classmethod - def get(cls, kind: str, name: str): - return cls._registry[f"{kind}:{name}"] - @classmethod def get_task(cls, name: str) -> KgTask: return cls._registry[f"task:{name}"] - - @classmethod - def list(cls, kind: str): - """List all registered items of a specific kind.""" - items = [] - for key, value in cls._registry.items(): - if key.startswith(f"{kind}:"): - items.append(value) - return items - - @classmethod - def list_all(cls): - return cls._registry \ No newline at end of file diff --git a/src/kgpipe/common/systemgraph.py b/src/kgpipe/common/systemgraph.py deleted file mode 100644 index 526ec23..0000000 --- a/src/kgpipe/common/systemgraph.py +++ /dev/null @@ -1,142 +0,0 @@ -import functools -from uuid import uuid4 -from typing import Any, List, TYPE_CHECKING -from pydantic import BaseModel -from datetime import datetime, timezone - -# from kgcore.api import KG, BackendName - -from kgcore.api import KnowledgeGraph, KGEntity, KGRelation, KGProperty, new_id -from kgcore.backend.rdf.rdf_rdflib import RDFLibBackend -from kgcore.backend.rdf.rdf_sparql import RDFSparqlBackend, SparqlAuth -from kgcore.model.rdf.rdf_base import RDFBaseModel - -from kgpipe.common.definitions import Task, TaskResult, Pipeline, PipelineResult -from kgpipe.common.config import load_config -from kgpipe.common.util import encode_string - -if TYPE_CHECKING: - from kgpipe.common.models import KgTask - - -config = load_config() -scheme, rest = config.SYS_KG_URL.split("://") - -backend = RDFLibBackend() -model = RDFBaseModel() - -try: - if scheme == "sparql": - print(f"Using SPARQL backend for system graph: {f"http://{rest}"}") - backend = RDFSparqlBackend( - endpoint=f"http://{rest}", - update_endpoint=f"http://{rest}", - default_graph="http://kg.org/systemgraph", - auth=SparqlAuth(username=config.SYS_KG_USR, password=config.SYS_KG_PSW)) - else: - raise ValueError(f"Unsupported schema: {scheme}") -except Exception as e: - print(f"Error creating system graph: {e}") - print(f"Using RDFLib backend for system graph: {f"http://{rest}"}") - -SYS_KG: KnowledgeGraph = KnowledgeGraph(model=model, backend=backend) - -class PipeKG: - - @staticmethod - def add_task(task: "KgTask"): - from kgpipe.common.models import KgTask # Import here to avoid circular import - types = [encode_string(c) for c in task.category] - properties = [] - properties.append(KGProperty(key="description", value=task.description)) - task_entity = SYS_KG.create_entity(id=task.name, types=types+["Task"], properties=properties) - for input_name, input_format in task.input_spec.items(): - input_entity = SYS_KG.create_entity(id=task.name+"_"+input_name, types=["Data"], properties={ - "format": input_format, - }) - SYS_KG.create_relation(type="input", source=task_entity.id, target=input_entity.id) - for output_name, output_format in task.output_spec.items(): - output_entity = SYS_KG.create_entity(id=task.name+"_"+output_name, types=["Data"], properties={ - "format": output_format, - }) - SYS_KG.create_relation(type="output", source=task_entity.id, target=output_entity.id) - - def list_tasks(self) -> List["KgTask"]: - return SYS_KG.list_entities(types=["Task"]) - - @staticmethod - def add_task_result(task_result: TaskResult): - SYS_KG.create_entity(id=new_id(),types=["TaskResult"], properties={ - "config": task_result.config, - "input": task_result.input, - "output": task_result.output, - }) - - - @staticmethod - def add_pipeline(pipeline: Pipeline): - SYS_KG.create_entity(id=new_id(),types=["Pipeline"], properties={ - "tasks": pipeline.tasks, - "input": pipeline.input, - "output": pipeline.output, - }) - - @staticmethod - def add_pipeline_result(pipeline_result: PipelineResult): - SYS_KG.create_entity(id=new_id(),types=["PipelineResult"], properties={ - "task_results": pipeline_result.task_results, - "eval_results": pipeline_result.eval_results, - "input": pipeline_result.input, - "output": pipeline_result.output, - }) - - - -# def Track(_cls=None, *, with_timestamp: bool = False): -# """ -# Use as: -# @Track -# @Track(with_timestamp=True) -# """ -# def decorator(cls): -# class Tracked(cls): # subclass the original class -# def __init__(self, *args: Any, **kwargs: Any): -# super().__init__(*args, **kwargs) - -# inst_id = f"{cls.__name__}:{uuid4().hex[:8]}" -# setattr(self, "_kg_id", inst_id) - -# if isinstance(self, BaseModel): -# props = self.model_dump() -# else: -# props = {k: v for k, v in vars(self).items() if not k.startswith("_")} - -# if with_timestamp: -# props["timestamp"] = datetime.now(timezone.utc).isoformat() - -# SYS_KG.create_entity([cls.__name__], id=inst_id, props=props) - -# Tracked.__name__ = cls.__name__ # optional cosmetics -# Tracked.__qualname__ = cls.__qualname__ -# Tracked.__doc__ = cls.__doc__ -# return Tracked - -# return decorator if _cls is None else decorator(_cls) - -# def kg_function(fn): -# @functools.wraps(fn) -# def wrapper(*args, **kwargs): -# result = fn(*args, **kwargs) -# call_id = f"{fn.__name__}:{uuid4().hex[:8]}" -# SYS_KG.create_entity( -# ["FunctionCall"], -# id=call_id, -# props={ -# "name": fn.__name__, -# # Be careful serializing args/kwargs; this is a toy example: -# "args": repr(args), -# "kwargs": repr(kwargs), -# }, -# ) -# return result -# return wrapper diff --git a/src/kgpipe/datasets/multipart_multisource.py b/src/kgpipe/datasets/multipart_multisource.py index 1389221..6653b06 100644 --- a/src/kgpipe/datasets/multipart_multisource.py +++ b/src/kgpipe/datasets/multipart_multisource.py @@ -90,8 +90,8 @@ def _check(self): def read_csv(self) -> List[MatchesRow]: return read_matches_csv(self.file) -def read_entities_csv(path: Path) -> List[EntitiesRow]: - return [EntitiesRow(entity_id=row["entity_id"], entity_label=row["entity_label"], entity_type=row["entity_type"], dataset=row["dataset"]) for row in csv.DictReader(path.open("r"), delimiter="\t")] +def read_entities_csv(path: Path, delimiter: str = "\t") -> List[EntitiesRow]: + return [EntitiesRow(entity_id=row["entity_id"], entity_label=row["entity_label"], entity_type=row["entity_type"], dataset=row["dataset"]) for row in csv.DictReader(path.open("r"), delimiter=delimiter)] class VerifiedEntities(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) @@ -215,13 +215,15 @@ class SplitIndex(BaseModel): # raise ValueError(f"{self.entities_csv} must contain an 'entity_id' column; got {header}") # return self +# SourceType = Literal["rdf", "json", "text"] + class Split(BaseModel): split_id: str root: Path index: SplitIndex kg_reference: Optional[KGBundle] = None kg_seed: Optional[KGBundle] = None - sources: Dict[str, SourceBundle] + sources: Dict[str, SourceBundle] # TODO SourceType def set_index(self, entities: List[EntitiesRow]): self.index.dir.mkdir(parents=True, exist_ok=True) @@ -548,12 +550,16 @@ def load_dataset(root: Path) -> Dataset: if seed_dir.exists(): seed_data_dir = seed_dir / "data" seed_meta_dir = seed_dir / "meta" + seed_meta = SourceMeta(root=seed_meta_dir) + ve = seed_meta_dir / "verified_entities.csv" + if ve.exists(): + seed_meta.entities = VerifiedEntities(file=ve) seed_parts = list_parts(seed_data_dir, (".nt", ".ttl", ".nq")) kg_seed = KGBundle( kind="seed", root=seed_dir, data=SourceData(dir=seed_data_dir, parts=seed_parts), - meta=SourceMeta(root=seed_meta_dir) + meta=seed_meta ) # sources diff --git a/src/kgpipe/evaluation/aspects/func/er_task_eval.py b/src/kgpipe/evaluation/aspects/func/er_task_eval.py index 5546164..37dafc1 100644 --- a/src/kgpipe/evaluation/aspects/func/er_task_eval.py +++ b/src/kgpipe/evaluation/aspects/func/er_task_eval.py @@ -167,9 +167,6 @@ def get_relation_matches(er_doc: ER_Document, threshold: float, match_cluster: O def get_matches_to_seed(er_doc: ER_Document, list_of_matches: list[MatchesRow], threshold): - print("list of matches", len(list_of_matches)) - print("threshold", threshold) - true_entity_match_cnt = 0 #tp false_entity_match_cnt = 0 #fp false_missing_entity_match_cnt = 0 #fn @@ -198,32 +195,21 @@ def get_matches_to_seed(er_doc: ER_Document, list_of_matches: list[MatchesRow], # get the seed and source ids seed_id = None source_id = None - if id1.startswith("http://kg.org/resource"): + if id1.startswith("http://kg.org/resource"): # TODO make configurable source_id = id2 seed_id = id1 if id2.startswith("http://kg.org/resource"): source_id = id1 seed_id = id2 - - checker = False - if seed_id == "http://kg.org/resource/b25598f9c0fce28a7700869fcb55d706": - checker = True - if seed_id is not None and source_id is not None: saw_seed_ids.add(seed_id) if is_match(source_id, seed_id, gt_cluster, False): true_entity_match_cnt += 1 - if checker: - print("true match", seed_id, source_id) else: false_entity_match_cnt += 1 - if checker: - print("false match", seed_id, source_id) else: # can not be checked, skip - if checker: - print("skip", seed_id, source_id) continue missing_seed_ids = gt_seed_ids - saw_seed_ids @@ -388,10 +374,10 @@ def evaluate_entity_matching(er_doc_path_or_kg: Path | KG, denom = 2 * tp + fp + fn f1_score = (2 * tp / denom) if denom > 0 else 0.0 - print("f1_score", f1_score) - print("tp", tp) - print("fp", fp) - print("fn", fn) + # print("f1_score", f1_score) + # print("tp", tp) + # print("fp", fp) + # print("fn", fn) return f1_score, f1_score, {"true_seed_match_cnt": tp, "false_seed_match_cnt": fp, "false_missing_seed_match_cnt": fn} @@ -407,6 +393,9 @@ def evaluate_relation_matching(er_doc_path: Path | KG, gt_match_path: Path, thre tp = match_counts.true_relation_match_cnt fp = match_counts.false_relation_match_cnt fn = match_counts.false_missing_relation_match_cnt + + if fn < 0: + fn = 0 f1_score = 2 * tp / (2 * tp + fp + fn) if tp > 0 else 0 diff --git a/src/kgpipe/evaluation/aspects/func/integration_eval.py b/src/kgpipe/evaluation/aspects/func/integration_eval.py index 72b6288..0e5392b 100644 --- a/src/kgpipe/evaluation/aspects/func/integration_eval.py +++ b/src/kgpipe/evaluation/aspects/func/integration_eval.py @@ -3,11 +3,11 @@ from pathlib import Path import pandas as pd from rdflib import RDFS, URIRef, Graph, RDF -from dataclasses import dataclass +from dataclasses import dataclass, field from kgpipe.util.embeddings.st_emb import get_model import numpy as np -from kgpipe.datasets.multipart_multisource import read_entities_csv - +from kgpipe.datasets.multipart_multisource import read_entities_csv, EntitiesRow +from typing import Any # model # entity dict @@ -41,6 +41,7 @@ class BinaryClassificationResult: fp: int tn: int fn: int + details: dict[str, Any] = field(default_factory=dict) def accuracy(self) -> float: return (self.tp + self.tn) / (self.tp + self.tn + self.fp + self.fn) @@ -63,6 +64,7 @@ def __dict__(self): "fp": self.fp, "tn": self.tn, "fn": self.fn, + "details": self.details, "accuracy": self.accuracy(), "precision": self.precision(), "recall": self.recall(), @@ -102,7 +104,7 @@ def load_entity_dict_from_csv(path: Path, delimiter: str = ",") -> dict: return entity_dict -def load_entity_dict(path: Path) -> dict: +def load_entity_dict(path: Path) -> dict[str, EntitiesRow]: """ """ if path.name.endswith(".json"): @@ -244,8 +246,9 @@ def evaluate_source_typed_entity_coverage(kg: KG, entity_dict_path: Path) -> Ent """ checks expected & integrated source typed entity overlap using label embeddings """ - model = get_model() - entity_dict = load_entity_dict(entity_dict_path) + model = get_model() # TODO this is not used here... + # TODO we need to substract the seed from the found entities... + entity_dict: dict[str, EntitiesRow] = load_entity_dict(entity_dict_path) expected_entity_label_type_pairs = [] @@ -270,15 +273,22 @@ def evaluate_source_typed_entity_coverage(kg: KG, entity_dict_path: Path) -> Ent found_eltp = set(found_entity_label_type_pairs) expected_eltp = set(expected_entity_label_type_pairs) - tp_set = found_eltp & expected_eltp - fp_set = found_eltp - expected_eltp - fn_set = expected_eltp - found_eltp + tp_set = found_eltp & expected_eltp # correct entity type pair + fp_set = found_eltp - expected_eltp # wrong entity type pair + fn_set = expected_eltp - found_eltp # missing entity type pair return BinaryClassificationResult( tp=len(tp_set), fp=len(fp_set), fn=len(fn_set), - tn=0 + tn=0, + details={ + "found_entity_label_type_pairs": found_entity_label_type_pairs, + "expected_entity_label_type_pairs": expected_entity_label_type_pairs, + "tp_set": len(tp_set), + "fp_set": len(fp_set), + "fn_set": len(fn_set) + } ) def evaluate_reference_triple_alignment(kg: KG, reference_kg: KG) -> TripleAlignmentResult: diff --git a/src/kgpipe/evaluation/aspects/reference.py b/src/kgpipe/evaluation/aspects/reference.py index cd6826d..4b474b9 100644 --- a/src/kgpipe/evaluation/aspects/reference.py +++ b/src/kgpipe/evaluation/aspects/reference.py @@ -112,12 +112,12 @@ def compute(self, kg: KG, config: ReferenceConfig, **kwargs) -> MetricResult: print(f"[CONFIG] Relation matching threshold: {config.RELATION_MATCH_THRESHOLD}") # TODO change to verfied entities level - dataset = config.dataset - if dataset is None: - raise ValueError("Dataset is not set") - gt_match_path = dataset.root / "split_match_entities.csv" + # dataset = config.dataset + # if dataset is None: + # raise ValueError("Dataset is not set") + # gt_match_path = dataset.root / "split_match_entities.csv" - value, normalized_score, details = evaluate_relation_matching(kg, gt_match_path, config.RELATION_MATCH_THRESHOLD) + value, normalized_score, details = evaluate_relation_matching(kg, config.GT_MATCHES, config.RELATION_MATCH_THRESHOLD) return MetricResult( name=self.name, @@ -422,8 +422,14 @@ def compute(self, kg: KG, config: ReferenceConfig, **kwargs) -> MetricResult: result = evaluate_source_typed_entity_coverage(kg, verified_source_entities_path) + # log details to file + with open("source_typed_entity_coverage_details.json", "w") as f: + json.dump(result.__dict__(), f) + return MetricResult( name=self.name, + kg=kg, + metric=self, value=result.f1_score(), normalized_score=result.f1_score(), details=result.__dict__(), @@ -660,24 +666,24 @@ class ReferenceEvaluator(AspectEvaluator): def __init__(self): super().__init__(EvaluationAspect.REFERENCE) self.metrics = [ - # ER_EntityMatchMetric(), - # ER_RelationMatchMetric(), - # TE_ExpectedEntityLinkMetric(), - # TE_ExpectedRelationLinkMetric(), - # JsonEntityMatchingMetric(), - # JsonRelationMatchingMetric(), - # JsonEntityLinkingMetric(), - # SourceEntityCoverageMetric(), - # SourceEntityCoverageMetricSoft(), - # SourceEntityPrecisionMetric(), + ER_EntityMatchMetric(), + ER_RelationMatchMetric(), + TE_ExpectedEntityLinkMetric(), + TE_ExpectedRelationLinkMetric(), + JsonEntityMatchingMetric(), + JsonRelationMatchingMetric(), + JsonEntityLinkingMetric(), + SourceEntityCoverageMetric(), + SourceEntityCoverageMetricSoft(), + SourceEntityPrecisionMetric(), SourceTypedEntityCoverageMetric(), - # ReferenceTripleAlignmentMetric(), - # ReferenceTripleAlignmentMetricSoftE(), - # ReferenceTripleAlignmentMetricSoftEV(), - # ReferenceClassCoverageMetric() + ReferenceTripleAlignmentMetric(), + ReferenceTripleAlignmentMetricSoftE(), + ReferenceTripleAlignmentMetricSoftEV(), + ReferenceClassCoverageMetric() ] - def evaluate(self, kg: KG, config: ReferenceConfig, metrics: Optional[List[str]] = None, **kwargs) -> AspectResult: + def evaluate(self, kg: KG, config: Optional[ReferenceConfig] = None, metrics: Optional[List[str]] = None, **kwargs) -> AspectResult: """Evaluate reference-based properties of the KG.""" # if references is {}: # # Return empty result if no reference KG provided @@ -697,7 +703,7 @@ def evaluate(self, kg: KG, config: ReferenceConfig, metrics: Optional[List[str]] metrics_to_compute = self.metrics if metrics: metrics_to_compute = [m for m in self.metrics if m.name in metrics] - + # Compute each metric for metric in metrics_to_compute: try: @@ -705,6 +711,7 @@ def evaluate(self, kg: KG, config: ReferenceConfig, metrics: Optional[List[str]] result = metric.compute(kg, config, **kwargs) end_time = time.time() result.duration = end_time - start_time + result.input = str(kg.path) results.append(result) except Exception as e: print(f"[Error] computing metric {metric.name}: {e}") @@ -713,6 +720,8 @@ def evaluate(self, kg: KG, config: ReferenceConfig, metrics: Optional[List[str]] print(traceback.format_exc()) error_result = MetricResult( name=metric.name, + kg=kg, + metric=metric, value=0.0, normalized_score=0.0, details={"error": str(e)}, diff --git a/src/kgpipe/evaluation/aspects/semantic.py b/src/kgpipe/evaluation/aspects/semantic.py index f0a4dab..286575a 100644 --- a/src/kgpipe/evaluation/aspects/semantic.py +++ b/src/kgpipe/evaluation/aspects/semantic.py @@ -20,7 +20,11 @@ from kgcore.api.ontology import OntologyExtractor, OntologyUtil, Ontology from kgpipe.common.registry import Registry import time +from ..base import MetricConfig +class SemanticConfig(MetricConfig): + """Config for semantic metrics.""" + pass def enrich_type_information(graph: Graph, ontology: Ontology, type_property: URIRef = RDF.type) -> Graph: type_dict = {} @@ -55,7 +59,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute reasoning score.""" import tempfile @@ -118,7 +122,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute schema consistency score.""" try: # Simple implementation - check for basic RDF structure @@ -196,7 +200,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute namespace usage score.""" try: namespaces = set() @@ -260,7 +264,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute disjoint domain score.""" raw_graph: Graph = kg.get_graph() @@ -305,7 +309,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute incorrect relation direction score.""" raw_graph: Graph = kg.get_graph() ontology_graph: Graph = kg.get_ontology_graph() @@ -314,6 +318,7 @@ def compute(self, kg: KG, **kwargs) -> MetricResult: if len(ontology_graph) == 0: ontology_graph = graph + print(f"INFO: ontology_graph is empty, using graph instead") # TODO use ontology implementation from framework predicate_defs_sr = ontology_graph.query( @@ -403,7 +408,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute incorrect relation cardinality score.""" raw_graph: Graph = kg.get_graph() @@ -464,7 +469,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: raw_graph: Graph = kg.get_graph() ontology_graph: Graph = kg.get_ontology_graph() @@ -534,7 +539,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute incorrect relation domain score.""" raw_graph: Graph = kg.get_graph() @@ -602,7 +607,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute incorrect datatype score.""" raw_graph: Graph = kg.get_graph() @@ -675,7 +680,7 @@ def __init__(self): - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute incorrect datatype format score.""" from kgpipe.evaluation.aspects.func.datatype_validator import validate_datatype @@ -750,7 +755,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute ontology class coverage score.""" raw_graph: Graph = kg.get_graph() @@ -788,7 +793,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute ontology relation coverage score.""" raw_graph: Graph = kg.get_graph() @@ -836,7 +841,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute ontology property coverage score.""" return MetricResult( name=self.name, @@ -856,7 +861,7 @@ def __init__(self): aspect=EvaluationAspect.SEMANTIC ) - def compute(self, kg: KG, **kwargs) -> MetricResult: + def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: """Compute ontology namespace coverage score.""" # graph = kg.get_graph() @@ -895,8 +900,10 @@ def __init__(self): OntologyNamespaceCoverageMetric(), ] - def evaluate(self, kg: KG, metrics: Optional[List[str]] = None, **kwargs) -> AspectResult: + def evaluate(self, kg: KG, metrics: Optional[List[str]] = None, config: Optional[SemanticConfig] = None, **kwargs) -> AspectResult: """Evaluate semantic properties of the KG.""" + if config is None: + config = SemanticConfig(name="default") results = [] # Filter metrics if specified @@ -908,9 +915,10 @@ def evaluate(self, kg: KG, metrics: Optional[List[str]] = None, **kwargs) -> Asp for metric in metrics_to_compute: try: start_time = time.time() - result = metric.compute(kg, **kwargs) + result = metric.compute(kg, config, **kwargs) end_time = time.time() result.duration = end_time - start_time + result.input = str(kg.path) results.append(result) except Exception as e: # Create error result diff --git a/src/kgpipe/evaluation/aspects/statistical.py b/src/kgpipe/evaluation/aspects/statistical.py index 28cb350..c1bd9a0 100644 --- a/src/kgpipe/evaluation/aspects/statistical.py +++ b/src/kgpipe/evaluation/aspects/statistical.py @@ -69,7 +69,11 @@ def compute(self, kg: KG, config: StatisticalConfig, **kwargs) -> MetricResult: ) except Exception as e: + print("this exception is raised") return MetricResult( + metric=self, + started_at=time.time(), + kg=kg, name=self.name, value=0.0, normalized_score=0.0, @@ -440,11 +444,15 @@ def evaluate(self, kg: KG, metrics: Optional[List[str]] = None, config: Optional result = metric.compute(kg, config, **kwargs) end_time = time.time() result.duration = end_time - start_time + result.input = str(kg.path) results.append(result) except Exception as e: # Create error result error_result = MetricResult( + metric=metric, + kg=kg, name=metric.name, + started_at=time.time(), value=0.0, normalized_score=0.0, details={"error": str(e)}, diff --git a/src/kgpipe/evaluation/base.py b/src/kgpipe/evaluation/base.py index 9d1af8f..5a91d0b 100644 --- a/src/kgpipe/evaluation/base.py +++ b/src/kgpipe/evaluation/base.py @@ -9,12 +9,19 @@ from enum import Enum from typing import Any, Dict, List, Optional # from kgpipe.common.systemgraph import kg_class - +from kgpipe.common.graph.systemgraph import PipeKG +import time +import json +import functools +import inspect # from kgpipe.common.util import create_insertable_nodes_and_edges, insert_kg_obj from pydantic import BaseModel from kgpipe.common.models import KG - +from kgpipe.common.graph.definitions import MetricRunEntity, MetricEntityId +from kgpipe.common.config import config +from pathlib import Path +from kgpipe.common.util import encode_string class EvaluationAspect(Enum): """The three main aspects of KG evaluation.""" @@ -34,11 +41,21 @@ class EvaluationConfig: output_format: str = "json" include_details: bool = True generate_report: bool = True + metric_config_path: Optional[Path] = None def __post_init__(self): if self.weights and not all(0.0 <= w <= 1.0 for w in self.weights.values()): raise ValueError("All weights must be between 0.0 and 1.0") + # def get_aspect_config(self, aspect: EvaluationAspect) -> MetricConfig: + # if aspect == EvaluationAspect.STATISTICAL: + # return StatisticalConfig(name="default") + # elif aspect == EvaluationAspect.SEMANTIC: + # return SemanticConfig(name="default") + # elif aspect == EvaluationAspect.REFERENCE: + # return ReferenceConfig(name="default") + # else: + # raise ValueError(f"No config available for aspect: {aspect}") @dataclass class AspectResult: @@ -56,6 +73,27 @@ def __str__(self) -> str: return f"{self.aspect.value}: {self.overall_score:.2f}" + + +# @Track(with_timestamp=True) +# @kg_class(type="MetricResult", description="Result of computing a single metric.") +@dataclass +class MetricResult: + """Result of computing a single metric.""" + name: str + metric: "Metric" + value: float + normalized_score: float # 0.0-1.0 range + aspect: EvaluationAspect + kg: KG + started_at: float = field(default_factory=time.time) + ended_at: float = field(default_factory=time.time) + details: Dict[str, Any] = field(default_factory=dict) + duration: float = 0.0 + +class MetricConfig(BaseModel): + name: str + class AspectEvaluator(ABC): """Base class for aspect-specific evaluators.""" @@ -63,7 +101,7 @@ def __init__(self, aspect: EvaluationAspect): self.aspect = aspect @abstractmethod - def evaluate(self, kg: KG, **kwargs) -> AspectResult: + def evaluate(self, kg: KG, config: Optional[MetricConfig], **kwargs) -> AspectResult: """Evaluate the KG for this specific aspect.""" pass @@ -73,30 +111,22 @@ def get_available_metrics(self) -> List[str]: pass -# @Track(with_timestamp=True) -# @kg_class(type="MetricResult", description="Result of computing a single metric.") -class MetricResult(BaseModel): - """Result of computing a single metric.""" - name: str - value: float - normalized_score: float # 0.0-1.0 range - details: Dict[str, Any] - aspect: EvaluationAspect - duration: float = 0.0 - - def __post_init__(self): - if not 0.0 <= self.normalized_score <= 1.0: - raise ValueError("Normalized score must be between 0.0 and 1.0") +def save_metric_run(metric: MetricResult): -class MetricConfig(BaseModel): - name: str - -from kgpipe.common.systemgraph import SYS_KG + metric_run_entity = MetricRunEntity( + status="success", + started_at=time.time(), + ended_at=time.time(), + computedMetric=MetricEntityId(config.PIPEKG_PREFIX+encode_string(metric.name)), + input=[], # [Data(uri=metric.kg.path, type="any/text")], + value=metric.value, + details=json.dumps(metric.details, default=str) + ) + PipeKG.add_metric_run(metric_run_entity) -import time -import json -import functools -import inspect + # # input=metric.input, + # # output=metric.output + # ) def track_metric_compute(func): @functools.wraps(func) @@ -112,36 +142,12 @@ def wrapper(self, *args, **kwargs): kg: KG = None config = None - - # Record the call entity (customize fields as you like) - call_entity = SYS_KG.create_entity(["Compute"],{ - "type": "MetricComputeCall", - "metric_class": type(self).__name__, - "method": func.__name__, - "input_kg_uri": kg.path.as_posix(), - "ts_start": time.time() - }) - try: result = func(self, *args, **kwargs) # <-- actually call it - result_id = insert_kg_obj(result) - SYS_KG.create_relation("metric_result",call_entity.id, result_id) - - # duration = time.perf_counter() - started - - # SYS_KG.up(call_entity, { - # "status": "ok", - # "duration_seconds": duration, - # "output_summary": _safe_summarize_result(result) - # }) + result.input = str(kg.path) + save_metric_run(result) return result except Exception as e: - # duration = time.perf_counter() - started - # SYS_KG.update_entity(call_entity, { - # "status": "error", - # "duration_seconds": duration, - # "error": repr(e) - # }) raise return wrapper @@ -163,7 +169,7 @@ def __init__(self, name: str, description: str, aspect: EvaluationAspect, metric self.metricConfig = metricConfig @abstractmethod - def compute(self, kg, **kwargs) -> MetricResult: + def compute(self, kg, **kwargs) -> MetricResult | List[MetricResult]: """Compute the metric value for the given KG.""" pass diff --git a/src/kgpipe/evaluation/cluster.py b/src/kgpipe/evaluation/cluster.py index 013c68a..6d386a2 100644 --- a/src/kgpipe/evaluation/cluster.py +++ b/src/kgpipe/evaluation/cluster.py @@ -205,8 +205,6 @@ def is_match(uri1: str, uri2: str, match_cluster: Optional[MatchCluster] = None, Check if two URIs are in the same match cluster. """ checker = False - if uri2 == "http://kg.org/resource/b25598f9c0fce28a7700869fcb55d706": - checker = True if allow_match_on_suffix: suffix1 = uri1.split("/")[-1] diff --git a/src/kgpipe/evaluation/evaluator.py b/src/kgpipe/evaluation/evaluator.py index f8863a2..7ba622e 100644 --- a/src/kgpipe/evaluation/evaluator.py +++ b/src/kgpipe/evaluation/evaluator.py @@ -9,10 +9,25 @@ from pathlib import Path from ..common.models import KG, Data -from .base import EvaluationAspect, AspectResult, EvaluationConfig +from .base import EvaluationAspect, AspectResult, EvaluationConfig, AspectEvaluator from .metrics import MetricResult from .reports import EvaluationReport +from .aspects.statistical import StatisticalConfig +from .aspects.semantic import SemanticConfig +from .aspects.reference import ReferenceConfig +from .base import MetricConfig +from .util import read_metric_config_yaml +from typing import Type +def get_aspect_config_type(aspect: EvaluationAspect) -> Type[MetricConfig]: + if aspect == EvaluationAspect.STATISTICAL: + return StatisticalConfig + elif aspect == EvaluationAspect.SEMANTIC: + return SemanticConfig + elif aspect == EvaluationAspect.REFERENCE: + return ReferenceConfig + else: + raise ValueError(f"No config available for aspect: {aspect}") class Evaluator: """Main evaluator that orchestrates evaluation across all aspects.""" @@ -38,13 +53,13 @@ def _initialize_aspect_evaluators(self) -> Dict[EvaluationAspect, Any]: return evaluators - def evaluate(self, kg: KG, references: Dict[str, Data] = {}) -> EvaluationReport: + def evaluate(self, kg: KG, config: Optional[EvaluationConfig]) -> EvaluationReport: """Evaluate the KG across all configured aspects.""" if not kg.exists(): raise FileNotFoundError(f"KG file not found: {kg.path}") - if references is {}: - raise ValueError("References are required for reference-based evaluation") + # if references is {}: + # raise ValueError("References are required for reference-based evaluation") aspect_results = [] all_metrics = [] @@ -52,16 +67,19 @@ def evaluate(self, kg: KG, references: Dict[str, Data] = {}) -> EvaluationReport # Evaluate each aspect for aspect in self.config.aspects: if aspect in self.aspect_evaluators: - evaluator = self.aspect_evaluators[aspect] + evaluator: AspectEvaluator = self.aspect_evaluators[aspect] # Prepare kwargs for aspect evaluation kwargs = {} - if aspect == EvaluationAspect.REFERENCE: - kwargs['references'] = references if self.config.metrics: kwargs['metrics'] = self.config.metrics - + + config_type = get_aspect_config_type(aspect) + + # TODO if metric empty for aspect, use default config + kwargs['config'] = read_metric_config_yaml(self.config.metric_config_path, config_type) + try: aspect_result = evaluator.evaluate(kg, **kwargs) aspect_results.append(aspect_result) @@ -76,7 +94,7 @@ def evaluate(self, kg: KG, references: Dict[str, Data] = {}) -> EvaluationReport # Create evaluation report report = EvaluationReport( kg=kg, - references=references, + references={}, aspect_results=aspect_results, overall_score=overall_score, config=self.config diff --git a/src/kgpipe/evaluation/util.py b/src/kgpipe/evaluation/util.py index 6967de2..6e16825 100644 --- a/src/kgpipe/evaluation/util.py +++ b/src/kgpipe/evaluation/util.py @@ -1,12 +1,16 @@ from pathlib import Path -from typing import Dict, List, Optional +from typing import Dict, List, Optional, Type import json import re +from enum import Enum + +import yaml from kgpipe.common import KG from kgpipe.common.models import Data, KgPipePlan, DataFormat from kgpipe.evaluation.aspects import reference from kgpipe.evaluation.aspects.reference import Reference +from kgpipe.evaluation.base import MetricConfig def resolve_relative_path(path: str, base_path: Path) -> Path: @@ -111,4 +115,43 @@ def get_plan(self) -> KgPipePlan: # TODO: use KgTask json_data = json.load(f) return KgPipePlan(**json_data) +def get_metric_config_template(metricConfig: MetricConfig) -> str: + """ + for a metricconfig which is a pydantic model, return a yaml that displays the model fields and their default values + """ + model = metricConfig if isinstance(metricConfig, type) else metricConfig.__class__ + fields = model.model_fields + + template: Dict[str, object] = {} + for field_name, field_info in fields.items(): + if field_info.is_required(): + value = None + elif field_info.default_factory is not None: + value = field_info.default_factory() + else: + value = field_info.default + + if isinstance(value, Path): + value = value.as_posix() + elif isinstance(value, Enum): + value = value.value + + template[field_name] = value + + return yaml.safe_dump(template, sort_keys=False) + +def read_metric_config_yaml(file: str, config_type: Type[MetricConfig]) -> MetricConfig: + """ + reads the config file and parses it into the given config type + """ + with open(file, "r") as f: + yaml_data = yaml.safe_load(f) + + if yaml_data is None: + yaml_data = {} + + if not isinstance(yaml_data, dict): + raise ValueError(f"Metric config YAML must be a mapping/object, got {type(yaml_data).__name__}") + return config_type.model_validate(yaml_data) + \ No newline at end of file diff --git a/src/kgpipe/execution/base.py b/src/kgpipe/execution/base.py new file mode 100644 index 0000000..829672c --- /dev/null +++ b/src/kgpipe/execution/base.py @@ -0,0 +1,13 @@ +from abc import ABC, abstractmethod +from kgpipe.common.model.pipeline import KgPipe + +class KGpipeExecution(ABC): + """Base class for KGpipe execution.""" + + def __init__(self, pipeline: KGpipePipeline): + self.pipeline = pipeline + + @abstractmethod + def execute(self): + """Execute the pipeline.""" + pass \ No newline at end of file diff --git a/src/kgpipe/execution/local.py b/src/kgpipe/execution/local.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe/execution/swarm.py b/src/kgpipe/execution/swarm.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe/generation/loaders.py b/src/kgpipe/generation/loaders.py index 8888fa4..953f033 100644 --- a/src/kgpipe/generation/loaders.py +++ b/src/kgpipe/generation/loaders.py @@ -70,14 +70,14 @@ def get_test_data(file_name: str) -> Path: return Path(__file__).parent / "test_data" / file_name -def ssp_pipeline(tasks: list[KgTask], target_data: Data, data_dir: str) -> KgPipe: - pipe = KgPipe(tasks, target_data, data_dir) +def ssp_pipeline(name: str, tasks: list[KgTask], target_data: Data, data_dir: str) -> KgPipe: + pipe = KgPipe(tasks, target_data, data_dir, name=name) return pipe -def build_from_conf(conf: PipelineConf, target_data: Data, data_dir: str) -> KgPipe: +def build_from_conf(name: str, conf: PipelineConf, target_data: Data, data_dir: str) -> KgPipe: tasks = [Registry.get_task(task_name) for task_name in conf.tasks] - pipe = ssp_pipeline(tasks, target_data, data_dir) + pipe = ssp_pipeline(name, tasks, target_data, data_dir) return pipe def build_from_yaml(yaml_path: Path): diff --git a/src/kgpipe/io/__init__.py b/src/kgpipe/io/__init__.py new file mode 100644 index 0000000..fe16459 --- /dev/null +++ b/src/kgpipe/io/__init__.py @@ -0,0 +1,2 @@ +__all__ = [] + diff --git a/src/kgpipe/io/pipe_out.py b/src/kgpipe/io/pipe_out.py new file mode 100644 index 0000000..e19bcd5 --- /dev/null +++ b/src/kgpipe/io/pipe_out.py @@ -0,0 +1,122 @@ +from __future__ import annotations + +from pathlib import Path +from typing import List, Optional + +from pydantic import BaseModel + +from kgpipe.common.models import KgPipePlan, KgStageReport + + +class TaskOut(BaseModel): + """ + Output artifacts produced by a single task within a stage. + """ + + task_name: str + output: List[Path] + + +class StageOut(BaseModel): + """ + Output artifacts for one incremental stage. + """ + + root: Path + stage_name: str + tasks: List[TaskOut] + resultKG: Optional[Path] = None + plan: Optional[KgPipePlan] = None + report: KgStageReport + + @property + def stage_index(self) -> int: + """ + Extract stage number from `stage_` directory name. + """ + return int(self.stage_name.split("_", 1)[1]) + + +class PipeOut(BaseModel): + """ + Output artifacts for a full incremental pipeline run directory containing stage_* subdirs. + """ + + root: Path + pipeline_name: str + stages: List[StageOut] + resultKG: Optional[Path] = None + + +def _stage_paths(run_dir: Path) -> list[Path]: + stage_paths = [p for p in run_dir.iterdir() if p.is_dir() and p.name.startswith("stage_")] + stage_paths.sort(key=lambda p: int(p.name.split("_", 1)[1])) + return stage_paths + + +def _resolve_stage_result_kg(stage_dir: Path) -> Path: + """ + Prefer `result_eval.nt` (evaluation-ready), fallback to `result.nt`. + """ + candidates = [ + # stage_dir / "result_eval.nt", + stage_dir / "result.nt", + ] + for c in candidates: + if c.exists(): + return c + # Keep the legacy default for downstream tools that expect result.nt even if not created yet. + return stage_dir / "result.nt" + + +def load_stage_out(stage_dir: Path) -> StageOut: + """ + Load stage outputs from a `stage_` directory produced by KGpipe incremental runs. + """ + stage_name = stage_dir.name + + plan_path = stage_dir / "exec-plan.json" + report_path = stage_dir / "exec-report.json" + + if not plan_path.exists(): + raise FileNotFoundError(f"Missing exec plan: {plan_path}") + if not report_path.exists(): + raise FileNotFoundError(f"Missing exec report: {report_path}") + + stage_plan = KgPipePlan.model_validate_json(plan_path.read_text()) + + stage_tasks: list[TaskOut] = [] + for step in stage_plan.steps: + stage_tasks.append( + TaskOut( + task_name=step.task, + output=[stage_dir / f"{output.path}" for output in step.output], + ) + ) + + stage_report = KgStageReport.model_validate_json(report_path.read_text()) + + return StageOut( + root=stage_dir, + stage_name=stage_name, + tasks=stage_tasks, + resultKG=_resolve_stage_result_kg(stage_dir), + plan=stage_plan, + report=stage_report, + ) + + +def load_pipe_out(run_dir: Path) -> PipeOut: + """ + Load a pipeline run output directory that contains `stage_*` directories. + """ + run_dir = Path(run_dir) + stages = [load_stage_out(p) for p in _stage_paths(run_dir)] + + return PipeOut( + root=run_dir, + pipeline_name=run_dir.name, + stages=stages, + resultKG=_resolve_stage_result_kg(run_dir) if (run_dir / "result.nt").exists() else (run_dir / "result.nt"), + ) + diff --git a/src/kgpipe/meta/subgraphs.py b/src/kgpipe/meta/subgraphs.py new file mode 100644 index 0000000..be11541 --- /dev/null +++ b/src/kgpipe/meta/subgraphs.py @@ -0,0 +1,11 @@ + + +# see meta-kg.owl.ttl for the ontology + +# eval subgraph + +# task subgraph + +# pipeline subgraph + +# execution subgraph diff --git a/src/kgpipe/test/common/test_graph.py b/src/kgpipe/test/common/test_graph.py new file mode 100644 index 0000000..e989caf --- /dev/null +++ b/src/kgpipe/test/common/test_graph.py @@ -0,0 +1,41 @@ +from uuid import uuid4 + +from kgpipe.common.graph.definitions import ( + DataTypeEntity, + DataSpecEntity, + TaskEntity, + ImplementationEntity, +) +from kgpipe.common.graph.systemgraph import PipeKG + + +def _uid(prefix: str) -> str: + return f"{prefix}_{uuid4().hex[:8]}" + + +def test_add_implementation_and_find_implemenetation(): + task = TaskEntity(name=_uid("task"), description="test task") + task_id = PipeKG.add_task(task) + + data_type = DataTypeEntity(format="text/csv", data_schema=_uid("schema")) + data_type_id = PipeKG.add_data_type(data_type) + + in_spec_id = PipeKG.add_data_spec(DataSpecEntity(name=_uid("in_spec"), data_type=data_type_id)) + out_spec_id = PipeKG.add_data_spec(DataSpecEntity(name=_uid("out_spec"), data_type=data_type_id)) + + impl_name = _uid("impl") + impl = ImplementationEntity( + name=impl_name, + version="0.0.1", + input_spec=[in_spec_id], + output_spec=[out_spec_id], + realizesTask=[task_id], + usesTool=[], + ) + + PipeKG.add_implementation(impl) + found = PipeKG.find_implementation(impl_name) + + assert found is not None + assert found.name == impl_name + assert found.version == "0.0.1" diff --git a/src/kgpipe/test/common/test_model.py b/src/kgpipe/test/common/test_model.py index c7b73c0..d579f5b 100644 --- a/src/kgpipe/test/common/test_model.py +++ b/src/kgpipe/test/common/test_model.py @@ -1,29 +1,68 @@ -from kgpipe.common.models import KgPipePlan, KgPipePlanStep, Data, DataFormat -from pathlib import Path import json +from enum import Enum +from pathlib import Path + +import pytest + +from kgpipe.common.models import ( + BasicDataFormats, + CustomDataFormats, + Data, + DataFormat, + KgPipePlan, + KgPipePlanStep, +) + +class ProjectFormats(CustomDataFormats): + EMBEDDINGS_JSON = "embeddings.json" + + +class ForeignFormats(str, Enum): + MY_RAW = "my.raw" + + +def test_kg_pipe_plan_roundtrip(): + plan = KgPipePlan( + steps=[ + KgPipePlanStep( + task="paris_entity_matching", + input=[Data(path=Path("data.nt"), format=DataFormat.RDF_NTRIPLES)], + output=[Data(path=Path("data.paris_csv"), format=DataFormat.PARIS_CSV)], + ), + KgPipePlanStep( + task="paris_csv_to_matching_format", + input=[Data(path=Path("data.paris_csv"), format=DataFormat.PARIS_CSV)], + output=[Data(path=Path("data.em_json"), format=DataFormat.ER_JSON)], + ), + ], + seed=Data(path=Path("seed.nt"), format=DataFormat.RDF_NTRIPLES), + source=Data(path=Path("source.nt"), format=DataFormat.RDF_NTRIPLES), + result=Data(path=Path("result.nt"), format=DataFormat.RDF_NTRIPLES), + ) + + plan_json = plan.model_dump_json() + plan_back = KgPipePlan(**json.loads(plan_json)) + + assert plan == plan_back + + +def test_data_accepts_basic_data_formats(): + data = Data(path=Path("a.nt"), format=BasicDataFormats.RDF_NTRIPLES) + assert data.format == BasicDataFormats.RDF_NTRIPLES + assert data.to_dict()["format"] == "nt" + + +def test_data_accepts_custom_data_formats(): + data = Data(path=Path("embed.json"), format=ProjectFormats.EMBEDDINGS_JSON) + assert data.format == ProjectFormats.EMBEDDINGS_JSON + assert data.to_dict()["format"] == "embeddings.json" + + +def test_data_rejects_foreign_string_enum_not_based_on_custom_catalog(): + with pytest.raises(ValueError): + Data(path=Path("x.raw"), format=ForeignFormats.MY_RAW) + -def test_kg_pipe_plan(): - plan = KgPipePlan( - steps=[ - KgPipePlanStep( - task="paris_entity_matching", - input=[Data(path=Path("data.nt"), format=DataFormat.RDF_NTRIPLES)], - output=[Data(path=Path("data.paris_csv"), format=DataFormat.PARIS_CSV)] - ), - KgPipePlanStep( - task="paris_csv_to_matching_format", - input=[Data(path=Path("data.paris_csv"), format=DataFormat.PARIS_CSV)], - output=[Data(path=Path("data.em_json"), format=DataFormat.ER_JSON)] - ), - ], - seed=Data(path=Path("seed.nt"), format=DataFormat.RDF_NTRIPLES), - source=Data(path=Path("source.nt"), format=DataFormat.RDF_NTRIPLES), - result=Data(path=Path("result.nt"), format=DataFormat.RDF_NTRIPLES), - ) - - plan_json = plan.model_dump_json() - print(plan_json) - - plan_back = KgPipePlan(**json.loads(plan_json)) - - assert plan == plan_back \ No newline at end of file +def test_data_rejects_unknown_string_format(): + with pytest.raises(ValueError, match="Unknown format: does-not-exist"): + Data(path=Path("x.any"), format="does-not-exist") \ No newline at end of file diff --git a/src/kgpipe/test/common/test_runtime_to_kg.py b/src/kgpipe/test/common/test_runtime_to_kg.py new file mode 100644 index 0000000..a716642 --- /dev/null +++ b/src/kgpipe/test/common/test_runtime_to_kg.py @@ -0,0 +1,83 @@ +from pathlib import Path + +from kgpipe.common.config import config +from kgpipe.common.models import Data, DataFormat, KgTaskReport +from kgpipe.common.model.task import KgTask +from kgpipe.common.runtime_to_kg import ( + data_to_handle, + reports_to_pipeline_run_entity, + task_to_task_entity, + task_report_to_task_run_entity, +) + + +def _make_report(name: str, start_ts: float, duration: float, status: str = "success") -> KgTaskReport: + return KgTaskReport( + task_name=name, + inputs=[Data(path=Path(f"{name}.in.nt"), format=DataFormat.RDF_NTRIPLES)], + outputs=[Data(path=Path(f"{name}.out.nt"), format=DataFormat.RDF_NTRIPLES)], + start_ts=start_ts, + duration=duration, + status=status, + ) + + +def test_data_to_handle_maps_path_and_format(): + data = Data(path=Path("test.nt"), format=DataFormat.RDF_NTRIPLES) + handle = data_to_handle(data) + assert handle.uri == "test.nt" + assert handle.type == DataFormat.RDF_NTRIPLES + + +def test_task_to_task_entity_maps_name_and_defaults(): + task = KgTask( + name="normalize", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=lambda _i, _o: None, + ) + entity = task_to_task_entity(task) + assert entity.name == "normalize" + assert entity.hasSubtask == [] + + +def test_task_report_to_task_run_entity_maps_core_fields(): + report = _make_report("normalize", start_ts=10.0, duration=2.5) + entity = task_report_to_task_run_entity(report, index=3) + + assert entity.number == 3 + assert entity.name == "normalize" + assert entity.status == "success" + assert entity.started_at == 10.0 + assert entity.ended_at == 12.5 + assert str(entity.executesTask) == f"{config.PIPEKG_PREFIX}normalize" + assert str(entity.usesImplementation) == f"{config.PIPEKG_PREFIX}normalizeImpl" + assert len(entity.input) == 1 + assert len(entity.output) == 1 + + +def test_reports_to_pipeline_run_entity_aggregates_times_and_runs(): + reports = [ + _make_report("step_a", start_ts=100.0, duration=10.0), + _make_report("step_b", start_ts=80.0, duration=5.0), + ] + + pipeline_entity = reports_to_pipeline_run_entity(reports, pipeline_name="demo_pipe") + + assert pipeline_entity.name == "demo_pipe" + assert pipeline_entity.status == "success" + assert pipeline_entity.started_at == 80.0 + assert pipeline_entity.ended_at == 110.0 + assert len(pipeline_entity.hasTaskRun) == 2 + assert pipeline_entity.hasTaskRun[0].number == 0 + assert pipeline_entity.hasTaskRun[1].number == 1 + + +def test_reports_to_pipeline_run_entity_handles_empty_reports(): + pipeline_entity = reports_to_pipeline_run_entity([], pipeline_name="empty_pipe") + + assert pipeline_entity.name == "empty_pipe" + assert pipeline_entity.status == "success" + assert pipeline_entity.started_at == 0.0 + assert pipeline_entity.ended_at == 0.0 + assert pipeline_entity.hasTaskRun == [] diff --git a/src/kgpipe/test/common/test_systemgraph.py b/src/kgpipe/test/common/test_systemgraph.py index 870bd75..2be56c7 100644 --- a/src/kgpipe/test/common/test_systemgraph.py +++ b/src/kgpipe/test/common/test_systemgraph.py @@ -1,72 +1,135 @@ -from kgpipe.common.systemgraph import kg_class, kg_function, SYS_KG, add_task, add_task_result, add_pipeline, add_pipeline_result -from kgpipe.common.definitions import Task, Eval, Pipeline, TaskResult, DataHandle, PipelineResult -import sys -from kgcore.backend.rdf.rdf_rdflib import RDFLibBackend - -task1 = Task( - name="test_task", - type="test_type", - description="test_description", - input=["test_input"], - output=["test_output"] -) -task2 = Task( - name="test_task2", - type="test_type2", - description="test_description2", - input=["test_input2"], - output=["test_output2"] -) -task_result1 = TaskResult( - task=task1, - config={"test_config": "test_config"}, - input=[DataHandle(uri="test_input", type="test_input_type")], - output=[DataHandle(uri="test_output", type="test_output_type")], - status="test_status", - duration=10.0 -) -task_result2 = TaskResult( - task=task2, - config={"test_config2": "test_config2"}, - input=[DataHandle(uri="test_input2", type="test_input_type2")], - output=[DataHandle(uri="test_output2", type="test_output_type2")], - status="test_status2", - duration=20.0 -) -pipeline = Pipeline( - tasks=[task1, task2], - input=["test_input"], - output=["test_output"] -) -pipeline_result = PipelineResult( - task_results=[task_result1, task_result2], - eval_results=[], - input=[DataHandle(uri="test_input", type="test_input_type")], - output=[DataHandle(uri="test_output", type="test_output_type")], - status="test_status", - duration=30.0 +from uuid import uuid4 + +from kgpipe.common.definitions import ( + DataHandle, + ImplementationEntity, + MethodEntity, + MetricEntity, + PipelineEntity, + TaskRunEntity, + ToolEntity, ) +from kgpipe.common.systemgraph import PipeKG + + +def _uid(prefix: str) -> str: + return f"{prefix}_{uuid4().hex[:8]}" + + +def test_core_layer_method_tool_and_implementation(): + method_name = _uid("method") + tool_name = _uid("tool") + impl_name = _uid("impl") + + method = MethodEntity(name=method_name, realizesTask=["task:a"]) + tool = ToolEntity(name=tool_name, providesMethods=["method:a"]) + implementation = ImplementationEntity( + name=impl_name, + input_spec=["text/csv"], + output_spec=["application/json"], + implementsMethod=["method:a"], + hasParameter=["param:a"], + usesTool=["tool:a"], + ) + + PipeKG.add_method(method) + PipeKG.add_tool(tool) + PipeKG.add_implementation(implementation) + + found_method = PipeKG.find_method(method_name) + found_tool = PipeKG.find_tool(tool_name) + found_implementation = PipeKG.find_implementation(impl_name) + + assert found_method is not None + assert found_method.name == method_name + assert "task:a" in found_method.realizesTask + + assert found_tool is not None + assert found_tool.name == tool_name + assert "method:a" in found_tool.providesMethods + + assert found_implementation is not None + assert found_implementation.name == impl_name + assert found_implementation.input_spec == ["text/csv"] + assert found_implementation.output_spec == ["application/json"] + + +def test_data_layer_artifact_type_and_spec(): + artifact_uri = f"file:///{_uid('artifact')}.csv" + artifact_type = _uid("artifact_type") + spec_name = _uid("spec") + specification = '{"type":"object","properties":{"name":{"type":"string"}}}' + data = DataHandle( + uri=artifact_uri, + type="text/csv", + version="1.0.0", + hash="abc123", + size=42, + ) + + PipeKG.add_data_artifact(data) + PipeKG.add_data_artifact_type(artifact_type) + PipeKG.add_data_artifact_spec(spec_name, specification) + + found_data = PipeKG.find_data_artifact(artifact_uri) + found_type = PipeKG.find_data_artifact_type(artifact_type) + found_spec = PipeKG.find_data_artifact_spec(spec_name) + + assert found_data is not None + assert found_data.uri == artifact_uri + assert found_data.type == "text/csv" + assert found_data.version == "1.0.0" + assert found_type == artifact_type + assert found_spec == specification + + +def test_pipeline_layer_pipeline_step_and_definition(): + pipeline_name = _uid("pipeline") + step_task = "task:clean" + definition_name = _uid("pipeline_def") + pipeline_id = f"pipeline:{pipeline_name}" + + pipeline = PipelineEntity(name=pipeline_name, tasks=[step_task], input=[], output=[]) + PipeKG.add_pipeline(pipeline) + PipeKG.add_pipeline_step(pipeline_name=pipeline_name, step_number=1, task_id=step_task) + PipeKG.add_pipeline_definition(name=definition_name, pipeline_id=pipeline_id) + + found_pipeline = PipeKG.find_pipeline(pipeline_name) + found_step = PipeKG.find_pipeline_step(pipeline_name, 1) + found_definition = PipeKG.find_pipeline_definition(definition_name) -model: RDFLibBackend = SYS_KG.backend + assert found_pipeline is not None + assert found_pipeline.name == pipeline_name + assert step_task in found_pipeline.tasks + assert found_step is not None + assert found_definition is not None -def test_task_entity(): - add_task(task1) - add_task(task2) - # print(model.get_rdflibgraph().serialize(format="turtle")) +def test_metrics_layer_add_and_find_metric(): + metric_name = _uid("metric") + metric = MetricEntity(name=metric_name, description="Accuracy metric", type="score") + PipeKG.add_metric(metric) -def test_task_result_entity(): - add_task_result(task_result1) - add_task_result(task_result2) + found_metric = PipeKG.find_metric(metric_name) - # print(model.get_rdflibgraph().serialize(format="turtle")) + assert found_metric is not None + assert found_metric.name == metric_name + assert found_metric.description == "Accuracy metric" + assert found_metric.type == "score" -def test_pipeline_entity(): - add_pipeline(pipeline) - # print(model.get_rdflibgraph().serialize(format="turtle")) +def test_run_layer_add_task_run(): + task_run = TaskRunEntity( + number=1, + name=_uid("task_run"), + status="success", + started_at=1.0, + ended_at=2.0, + input=[DataHandle(uri="file:///in.csv", type="text/csv")], + output=[DataHandle(uri="file:///out.csv", type="text/csv")], + executesTask="task:clean", + usesImplementation="impl:clean_v1", + hasParameterBinding=[], + ) -def test_pipeline_result_entity(): - add_pipeline_result(pipeline_result) - - print(model.get_rdflibgraph().serialize(format="turtle")) \ No newline at end of file + PipeKG.add_task_run(task_run) \ No newline at end of file diff --git a/src/kgpipe/test/common/test_task_category_catalog.py b/src/kgpipe/test/common/test_task_category_catalog.py new file mode 100644 index 0000000..605ac0f --- /dev/null +++ b/src/kgpipe/test/common/test_task_category_catalog.py @@ -0,0 +1,33 @@ +from kgpipe.common.models import TaskCategoryCatalog + + +def test_entity_resolution_children_include_expected_subtasks(): + children = TaskCategoryCatalog.get_children("EntityResolution") + assert "Blocking" in children + assert "Matching" in children + assert "EntityMatching" in children + assert "Clustering" in children + + +def test_subtask_relationships_for_entity_resolution(): + assert TaskCategoryCatalog.is_subtask_of("Blocking", "EntityResolution") + assert TaskCategoryCatalog.is_subtask_of("Matching", "EntityResolution") + assert TaskCategoryCatalog.is_subtask_of("Clustering", "EntityResolution") + assert not TaskCategoryCatalog.is_subtask_of("EntityResolution", "Blocking") + + +def test_ancestors_and_descendants_are_resolved(): + ancestors = TaskCategoryCatalog.get_ancestors("EntityMatching") + descendants = TaskCategoryCatalog.get_descendants("EntityResolution") + + assert ancestors[0] == "EntityResolution" + assert "TaskCategory" in ancestors + assert "Blocking" in descendants + assert "Clustering" in descendants + + +def test_register_custom_category_under_existing_parent(): + TaskCategoryCatalog.register("CandidateGeneration", parent="EntityResolution") + assert TaskCategoryCatalog.has("CandidateGeneration") + assert TaskCategoryCatalog.get_parent("CandidateGeneration") == "EntityResolution" + assert TaskCategoryCatalog.is_subtask_of("CandidateGeneration", "EntityResolution") diff --git a/src/kgpipe/test/common/test_task_model.py b/src/kgpipe/test/common/test_task_model.py new file mode 100644 index 0000000..cbe0e19 --- /dev/null +++ b/src/kgpipe/test/common/test_task_model.py @@ -0,0 +1,136 @@ +from pathlib import Path + +from kgpipe.common.models import Data, DataFormat, KgTask + + +def _write_output_task(inputs: dict[str, Data], outputs: dict[str, Data]) -> None: + _ = inputs["in"] + out_path = outputs["out"].path + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_text("generated") + + +def test_kgtask_run_success(tmp_path: Path): + in_file = tmp_path / "input.nt" + out_file = tmp_path / "output.nt" + in_file.write_text("seed") + + task = KgTask( + name="copy_like", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=_write_output_task, + ) + + report = task.run( + inputs=[Data(path=in_file, format=DataFormat.RDF_NTRIPLES)], + outputs=[Data(path=out_file, format=DataFormat.RDF_NTRIPLES)], + ) + + assert report.status == "success" + assert out_file.exists() + assert report.task_name == "copy_like" + assert len(report.inputs) == 1 + assert len(report.outputs) == 1 + + +def test_kgtask_run_failed_when_function_raises(tmp_path: Path): + def failing_task(_: dict[str, Data], __: dict[str, Data]) -> None: + raise RuntimeError("boom") + + in_file = tmp_path / "input.nt" + out_file = tmp_path / "output.nt" + in_file.write_text("seed") + + task = KgTask( + name="fails", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=failing_task, + ) + + report = task.run( + inputs=[Data(path=in_file, format=DataFormat.RDF_NTRIPLES)], + outputs=[Data(path=out_file, format=DataFormat.RDF_NTRIPLES)], + ) + + assert report.status == "failed" + assert report.error is not None + assert "boom" in report.error + + +def test_kgtask_run_skips_when_outputs_exist(tmp_path: Path): + called = {"count": 0} + + def should_not_run(_: dict[str, Data], __: dict[str, Data]) -> None: + called["count"] += 1 + + in_file = tmp_path / "input.nt" + out_file = tmp_path / "output.nt" + in_file.write_text("seed") + out_file.write_text("already-here") + + task = KgTask( + name="skip_if_present", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=should_not_run, + ) + + report = task.run( + inputs=[Data(path=in_file, format=DataFormat.RDF_NTRIPLES)], + outputs=[Data(path=out_file, format=DataFormat.RDF_NTRIPLES)], + ) + + assert report.status == "skipped" + assert called["count"] == 0 + + +def test_kgtask_stable_files_override_forces_run(tmp_path: Path): + called = {"count": 0} + out_file = tmp_path / "output.nt" + + def rewrite_output(_: dict[str, Data], outputs: dict[str, Data]) -> None: + called["count"] += 1 + outputs["out"].path.write_text("fresh") + + in_file = tmp_path / "input.nt" + in_file.write_text("seed") + out_file.write_text("stale") + + task = KgTask( + name="override_output", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=rewrite_output, + ) + + report = task.run( + inputs=[Data(path=in_file, format=DataFormat.RDF_NTRIPLES)], + outputs=[Data(path=out_file, format=DataFormat.RDF_NTRIPLES)], + stable_files_override=True, + ) + + assert report.status == "success" + assert called["count"] == 1 + assert out_file.read_text() == "fresh" + + +def test_kgtask_run_fails_for_missing_required_input(tmp_path: Path): + out_file = tmp_path / "output.nt" + + task = KgTask( + name="needs_input", + input_spec={"in": DataFormat.RDF_NTRIPLES}, + output_spec={"out": DataFormat.RDF_NTRIPLES}, + function=_write_output_task, + ) + + report = task.run( + inputs=[], + outputs=[Data(path=out_file, format=DataFormat.RDF_NTRIPLES)], + ) + + assert report.status == "failed" + assert report.error is not None + assert "Missing required inputs" in report.error diff --git a/src/kgpipe/test/evaluation/test_metric_config_template.py b/src/kgpipe/test/evaluation/test_metric_config_template.py new file mode 100644 index 0000000..7d971ef --- /dev/null +++ b/src/kgpipe/test/evaluation/test_metric_config_template.py @@ -0,0 +1,44 @@ +import yaml +from pathlib import Path + +from kgpipe.evaluation.aspects.reference import ReferenceConfig +from kgpipe.evaluation.util import get_metric_config_template, read_metric_config_yaml + + +def test_get_metric_config_template_for_reference_config(): + template_yaml = get_metric_config_template(ReferenceConfig) + template = yaml.safe_load(template_yaml) + + assert template["name"] is None + assert template["GT_MATCHES"] is None + assert template["GT_MATCHES_TARGET_DATASET"] is None + assert template["ENTITY_MATCH_THRESHOLD"] == 0.5 + assert template["RELATION_MATCH_THRESHOLD"] == 0.5 + assert template["VERIFIED_SOURCE_ENTITIES"] is None + assert template["REFERENCE_KG_PATH"] is None + assert template["EXPECTED_TEXT_LINKS"] is None + assert template["TE_LINK_THRESHOLD"] == 0.4 + assert template["SEED_KG_PATH"] is None + assert template["source_meta"] is None + assert template["dataset"] is None + assert template["JSON_EXPECTED_DIR"] is None + assert template["JSON_EXPECTED_RELATION_FILE"] is None + + +def test_metric_config_template_roundtrip_reference_config(tmp_path: Path): + template_yaml = get_metric_config_template(ReferenceConfig) + template = yaml.safe_load(template_yaml) + template["name"] = "reference-config-roundtrip" + template["GT_MATCHES"] = "/tmp/gt_matches.csv" + + config_path = tmp_path / "reference_config.yaml" + with open(config_path, "w") as f: + yaml.safe_dump(template, f, sort_keys=False) + + config = read_metric_config_yaml(config_path.as_posix(), ReferenceConfig) + + assert isinstance(config, ReferenceConfig) + assert config.name == "reference-config-roundtrip" + assert config.GT_MATCHES == Path("/tmp/gt_matches.csv") + assert config.ENTITY_MATCH_THRESHOLD == 0.5 + assert config.TE_LINK_THRESHOLD == 0.4 diff --git a/src/kgpipe/test/test_meta.py b/src/kgpipe/test/test_meta.py new file mode 100644 index 0000000..ee535b8 --- /dev/null +++ b/src/kgpipe/test/test_meta.py @@ -0,0 +1,25 @@ +from kgcore.api import KG +from kgcore.decorators.event import event + +def test_meta(): + kg = KG(backend='memory', name='test') + kg.create_entity(["Task"], props={"name": "test", "description": "test"}) + + + @event("Task", "create") + def task_created(e): + print(f"Task created: {e.id}") + + @event("Task", "update") + def task_updated(e): + print(f"Task updated: {e.id}") + + @event("Task", "delete") + def task_deleted(e): + print(f"Task deleted: {e.id}") + + task_created("e") + + es = kg.find_entities() + for e in es: + print(e) \ No newline at end of file diff --git a/src/kgpipe/test/test_meta_kg_query.py b/src/kgpipe/test/test_meta_kg_query.py new file mode 100644 index 0000000..5f5a0d4 --- /dev/null +++ b/src/kgpipe/test/test_meta_kg_query.py @@ -0,0 +1,106 @@ +import importlib.util +from pathlib import Path + + +def _load_query_module(): + module_path = Path(__file__).resolve().parents[2] / "kgpipe_view" / "meta_kg_query.py" + spec = importlib.util.spec_from_file_location("meta_kg_query", module_path) + module = importlib.util.module_from_spec(spec) + assert spec and spec.loader + spec.loader.exec_module(module) + return module + + +def test_query_tasks_implementations_maps_primary_results(monkeypatch): + module = _load_query_module() + + calls = [] + + def _fake_run_select(_endpoint_url, query): + calls.append(query) + return [ + { + "task": {"value": "http://example.org/kgp#TaskA"}, + "method": {"value": "http://example.org/kgp#MethodA"}, + "implementation": {"value": "http://example.org/kgp#ImplA"}, + "tool": {"value": "http://example.org/kgp#ToolA"}, + "runtime": {"value": "python"}, + "implementationVersion": {"value": "1.0.0"}, + "commandTemplate": {"value": "python run.py"}, + } + ] + + monkeypatch.setattr(module, "_run_select", _fake_run_select) + + frame = module.query_tasks_implementations("http://localhost:8890/sparql") + + assert len(calls) == 1 + assert frame.shape == (1, 7) + assert frame.loc[0, "task"] == "http://example.org/kgp#TaskA" + assert frame.loc[0, "implementation_version"] == "1.0.0" + + +def test_query_tasks_implementations_falls_back_when_primary_is_empty(monkeypatch): + module = _load_query_module() + + calls = [] + + def _fake_run_select(_endpoint_url, query): + calls.append(query) + if len(calls) == 1: + return [] + return [{"implementation": {"value": "http://example.org/kgp#ImplB"}}] + + monkeypatch.setattr(module, "_run_select", _fake_run_select) + + frame = module.query_tasks_implementations("http://localhost:8890/sparql") + + assert len(calls) == 2 + assert frame.shape == (1, 7) + assert frame.loc[0, "implementation"] == "http://example.org/kgp#ImplB" + assert frame.loc[0, "task"] == "" + + +def test_query_task_hierarchy_maps_primary_results(monkeypatch): + module = _load_query_module() + + calls = [] + + def _fake_run_select(_endpoint_url, query): + calls.append(query) + return [ + { + "task": {"value": "http://example.org/kgp#NormalizeTask"}, + "parentTask": {"value": "http://example.org/kgp#TransformTask"}, + } + ] + + monkeypatch.setattr(module, "_run_select", _fake_run_select) + + frame = module.query_task_hierarchy("http://localhost:8890/sparql") + + assert len(calls) == 1 + assert frame.shape == (1, 2) + assert frame.loc[0, "task"] == "http://example.org/kgp#NormalizeTask" + assert frame.loc[0, "parent_task"] == "http://example.org/kgp#TransformTask" + + +def test_query_task_hierarchy_falls_back_when_primary_is_empty(monkeypatch): + module = _load_query_module() + + calls = [] + + def _fake_run_select(_endpoint_url, query): + calls.append(query) + if len(calls) == 1: + return [] + return [{"task": {"value": "http://example.org/kgp#TrainTask"}}] + + monkeypatch.setattr(module, "_run_select", _fake_run_select) + + frame = module.query_task_hierarchy("http://localhost:8890/sparql") + + assert len(calls) == 2 + assert frame.shape == (1, 2) + assert frame.loc[0, "task"] == "http://example.org/kgp#TrainTask" + assert frame.loc[0, "parent_task"] == "" diff --git a/src/kgpipe/test/test_owl_to_mermaid.py b/src/kgpipe/test/test_owl_to_mermaid.py new file mode 100644 index 0000000..88ff02e --- /dev/null +++ b/src/kgpipe/test/test_owl_to_mermaid.py @@ -0,0 +1,51 @@ +import importlib.util +from pathlib import Path + + +def _load_converter_module(): + module_path = ( + Path(__file__).resolve().parents[2] / "kgpipe_view" / "owl_to_mermaid.py" + ) + spec = importlib.util.spec_from_file_location("owl_to_mermaid", module_path) + module = importlib.util.module_from_spec(spec) + assert spec and spec.loader + spec.loader.exec_module(module) + return module + + +def test_get_available_layers_includes_core_layer(): + module = _load_converter_module() + ttl_path = Path(__file__).resolve().parents[2] / "kgpipe_view" / "kgpipe.owl.ttl" + + layers = module.get_available_layers(ttl_path) + + assert "CoreLayer" in layers + assert "PipelineLayer" in layers + + +def test_convert_owl_ttl_to_mermaid_filters_by_layer(): + module = _load_converter_module() + ttl_path = Path(__file__).resolve().parents[2] / "kgpipe_view" / "kgpipe.owl.ttl" + + mermaid = module.convert_owl_ttl_to_mermaid(ttl_path, layer_filter="CoreLayer") + + assert "class Task" in mermaid + assert "class Method" in mermaid + assert "class Pipeline" not in mermaid + assert "class PipelineStep" not in mermaid + assert 'Method "0..*" --> "0..*" Task : realizesTask' in mermaid + assert 'Pipeline "0..*" --> "0..*" PipelineStep : hasStep' not in mermaid + + +def test_convert_owl_ttl_to_mermaid_filters_by_multiple_layers(): + module = _load_converter_module() + ttl_path = Path(__file__).resolve().parents[2] / "kgpipe_view" / "kgpipe.owl.ttl" + + mermaid = module.convert_owl_ttl_to_mermaid( + ttl_path, layer_filter=["CoreLayer", "PipelineLayer"] + ) + + assert "class Task" in mermaid + assert "class Pipeline" in mermaid + assert 'Pipeline "0..*" --> "0..*" PipelineStep : hasStep' in mermaid + assert "class Artifact" not in mermaid diff --git a/src/kgpipe_eval/__init__.py b/src/kgpipe_eval/__init__.py new file mode 100644 index 0000000..04e3f84 --- /dev/null +++ b/src/kgpipe_eval/__init__.py @@ -0,0 +1,2 @@ +# Refactor of kgpipe.evaluation to be a standalone package + diff --git a/src/kgpipe_eval/api.py b/src/kgpipe_eval/api.py new file mode 100644 index 0000000..36c821a --- /dev/null +++ b/src/kgpipe_eval/api.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import Any + + +# MetricConfig (rich, typed, input) +# ↓ +# computation +# ↓ +# MetricResult +# ├── measurements (results) +# └── metadata (flattened config + context) + +@dataclass(frozen=True) +class MetricConfig: + pass + +@dataclass(frozen=True) +class Measurement: + name: str + value: Any + unit: str | None = None + +@dataclass(frozen=True) +class MetricResult: + metric: "Metric" + measurements: list[Measurement] + summary: str | None = None + # TODO metadata/properties: dict[str, int | float | str | bool] = field(default_factory=dict) + +class Metric(ABC): + """ + Minimal metric interface for the `kgpipe eval-new` CLI. + + Metrics are instantiated (usually with default config) and then run via `compute(...)`. + """ + + key: str + description: str + + @abstractmethod + def compute(self, *args: Any, **kwargs: Any) -> MetricResult: ... + + +# --- + diff --git a/src/kgpipe_eval/config/manager.py b/src/kgpipe_eval/config/manager.py new file mode 100644 index 0000000..d7d5d81 --- /dev/null +++ b/src/kgpipe_eval/config/manager.py @@ -0,0 +1,334 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Any, Mapping +import re + +import yaml +from pydantic import BaseModel + +from kgpipe.common import KG +from kgpipe.common.model.data import DataFormat + +from kgpipe_eval.metrics.duplicates import DuplicateConfig +from kgpipe_eval.metrics.triple_alignment import TripleAlignmentConfig +from kgpipe_eval.metrics.consistency_violations import ConsistencyViolationsConfig +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig + + +MetricConfigModel = BaseModel +REQUIRED = "" + +_VAR_PATTERN = re.compile(r"^\$(\w+)$|^\$\{(\w+)\}$") + + +def _interpolate_vars(obj: Any, vars_map: Mapping[str, Any]) -> Any: + """ + Recursively interpolate simple $var / ${var} references inside YAML-loaded data. + + Only replaces when the *entire* string is a reference token. + """ + if isinstance(obj, str): + m = _VAR_PATTERN.match(obj.strip()) + if not m: + return obj + name = m.group(1) or m.group(2) + if name in vars_map: + return vars_map[name] + return obj + if isinstance(obj, list): + return [_interpolate_vars(v, vars_map) for v in obj] + if isinstance(obj, dict): + return {k: _interpolate_vars(v, vars_map) for k, v in obj.items()} + return obj + + +def _resolve_paths(obj: Any, *, base_dir: Path) -> Any: + """ + Recursively resolve relative paths for common config keys. + + - For keys ending with `_path` or `_kg_path`, if the value is a str/Path and + relative, make it absolute by joining with `base_dir`. + - For `reference_kg` when passed as str/Path, treat it as a path too. + """ + if isinstance(obj, list): + return [_resolve_paths(v, base_dir=base_dir) for v in obj] + if isinstance(obj, dict): + out: dict[str, Any] = {} + for k, v in obj.items(): + vv = _resolve_paths(v, base_dir=base_dir) + if isinstance(vv, (str, Path)): + if k == "reference_kg" or k.endswith("_path") or k.endswith("_kg_path"): + p = Path(vv) + if not p.is_absolute(): + vv = (base_dir / p).resolve() + out[k] = vv + return out + return obj + + +def _deep_merge_dict(base: Mapping[str, Any], override: Mapping[str, Any]) -> dict[str, Any]: + """ + Merge override into base recursively (override wins). + """ + out: dict[str, Any] = dict(base) + for k, v in override.items(): + if ( + k in out + and isinstance(out[k], Mapping) + and isinstance(v, Mapping) + ): + out[k] = _deep_merge_dict(out[k], v) + else: + out[k] = v + return out + + +def _kg_from_path(path: Path, *, name: str | None = None) -> KG: + """ + Build a minimal `kgpipe.common.KG` from a filesystem path. + + Notes: + - We infer `format` from the file suffix when possible, otherwise fall back to JSON. + - The KG object lazily parses the graph when `get_graph()` is called. + """ + suffix = path.suffix.lower().lstrip(".") + try: + fmt = DataFormat(suffix) + except Exception: + fmt = DataFormat.JSON + + return KG( + id=str(path), + name=(name or path.stem), + path=path, + format=fmt, + ) + + +def _resolve_entity_alignment_config( + metric_cfg: Mapping[str, Any], + named: Mapping[str, Mapping[str, Any]], +) -> dict[str, Any]: + """ + Resolve an entity alignment config from either: + - inline: `entity_alignment_config: {...}` + - ref: `entity_alignment_config_ref: name` + Optionally supports both; inline values override the referenced dict. + """ + inline = metric_cfg.get("entity_alignment_config") or {} + ref_name = metric_cfg.get("entity_alignment_config_ref") + if ref_name is None: + if not isinstance(inline, Mapping): + raise TypeError("`entity_alignment_config` must be a mapping if provided.") + return dict(inline) + + if not isinstance(ref_name, str) or not ref_name: + raise TypeError("`entity_alignment_config_ref` must be a non-empty string.") + if ref_name not in named: + raise KeyError(f"Unknown entity alignment config ref: {ref_name!r}") + + if not isinstance(inline, Mapping): + raise TypeError("`entity_alignment_config` must be a mapping if provided.") + return _deep_merge_dict(named[ref_name], inline) + + +def load_metric_configs(config_path: str | Path) -> dict[str, MetricConfigModel]: + """ + Load a single YAML file that defines metric configs and optional shared sub-configs. + + Expected YAML structure (minimal): + + ```yaml + entity_alignment_configs: + default: + method: label_embedding + verified_entities_path: path/to/entities.csv + entity_sim_threshold: 0.95 + + metrics: + entity_align: + entity_alignment_config_ref: default + + duplicates: + entity_alignment_config_ref: default + + triple_alignment: + reference_kg_path: path/to/reference.nt + entity_alignment_config_ref: default + value_sim_threshold: 0.5 + ``` + + Returned dict keys are metric keys (e.g. "duplicates") and values are instantiated + Pydantic config objects (e.g. `DuplicateConfig`). + """ + path = Path(config_path) + raw = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + if not isinstance(raw, Mapping): + raise TypeError("Top-level YAML must be a mapping/dict.") + + # Allow simple variable indirection like: + # reference_kg: test.ttl + # ... reference_kg: $reference_kg + vars_map = {k: v for k, v in raw.items() if isinstance(k, str)} + raw = _interpolate_vars(raw, vars_map) + raw = _resolve_paths(raw, base_dir=path.parent) + + named_entity_alignment: dict[str, dict[str, Any]] = {} + raw_named = raw.get("entity_alignment_configs") or {} + if raw_named: + if not isinstance(raw_named, Mapping): + raise TypeError("`entity_alignment_configs` must be a mapping/dict.") + for k, v in raw_named.items(): + if not isinstance(k, str) or not k: + raise TypeError("`entity_alignment_configs` keys must be non-empty strings.") + if not isinstance(v, Mapping): + raise TypeError(f"`entity_alignment_configs.{k}` must be a mapping/dict.") + named_entity_alignment[k] = dict(v) + + metrics_raw = raw.get("metrics") or {} + if not isinstance(metrics_raw, Mapping): + raise TypeError("`metrics` must be a mapping/dict.") + + out: dict[str, MetricConfigModel] = {} + for metric_key, metric_cfg_any in metrics_raw.items(): + if not isinstance(metric_key, str) or not metric_key: + raise TypeError("Metric keys in `metrics` must be non-empty strings.") + if metric_cfg_any is None: + metric_cfg: dict[str, Any] = {} + elif isinstance(metric_cfg_any, Mapping): + metric_cfg = dict(metric_cfg_any) + else: + raise TypeError(f"`metrics.{metric_key}` must be a mapping/dict.") + + # --- Metric-specific instantiation rules + if metric_key in {"entity_align", "entity_alignment"}: + entity_cfg_dict = _resolve_entity_alignment_config(metric_cfg, named_entity_alignment) + # Allow `reference_kg_path` convenience here too + if "reference_kg_path" in entity_cfg_dict and "reference_kg" not in entity_cfg_dict: + ref_path = Path(entity_cfg_dict.pop("reference_kg_path")) + entity_cfg_dict["reference_kg"] = _kg_from_path(ref_path) + # Backward compatible: accept `reference_kg: "/path/to/file.nt"` in YAML + if isinstance(entity_cfg_dict.get("reference_kg"), (str, Path)): + entity_cfg_dict["reference_kg"] = _kg_from_path(Path(entity_cfg_dict["reference_kg"])) + out[metric_key] = EntityAlignmentConfig.model_validate(entity_cfg_dict) + continue + + if metric_key in {"duplicates", "duplicate"}: + entity_cfg_dict = _resolve_entity_alignment_config(metric_cfg, named_entity_alignment) + if "reference_kg_path" in entity_cfg_dict and "reference_kg" not in entity_cfg_dict: + ref_path = Path(entity_cfg_dict.pop("reference_kg_path")) + entity_cfg_dict["reference_kg"] = _kg_from_path(ref_path) + if isinstance(entity_cfg_dict.get("reference_kg"), (str, Path)): + entity_cfg_dict["reference_kg"] = _kg_from_path(Path(entity_cfg_dict["reference_kg"])) + out[metric_key] = DuplicateConfig.model_validate( + { + "entity_alignment_config": EntityAlignmentConfig.model_validate(entity_cfg_dict), + } + ) + continue + + if metric_key in {"triple_alignment", "triple_align"}: + cfg_dict: dict[str, Any] = dict(metric_cfg) + entity_cfg_dict = _resolve_entity_alignment_config(metric_cfg, named_entity_alignment) + if "reference_kg_path" in entity_cfg_dict and "reference_kg" not in entity_cfg_dict: + ref_path = Path(entity_cfg_dict.pop("reference_kg_path")) + entity_cfg_dict["reference_kg"] = _kg_from_path(ref_path) + if isinstance(entity_cfg_dict.get("reference_kg"), (str, Path)): + entity_cfg_dict["reference_kg"] = _kg_from_path(Path(entity_cfg_dict["reference_kg"])) + cfg_dict["entity_alignment_config"] = EntityAlignmentConfig.model_validate(entity_cfg_dict) + + # Allow YAML to specify a path rather than an in-memory KG object + if "reference_kg_path" in cfg_dict and "reference_kg" not in cfg_dict: + ref_path = Path(cfg_dict.pop("reference_kg_path")) + cfg_dict["reference_kg"] = _kg_from_path(ref_path) + + out[metric_key] = TripleAlignmentConfig.model_validate(cfg_dict) + continue + + if metric_key in { + "consistency_violations", + "disjoint_domain", + "domain", + "range", + "relation_direction", + "datatype", + "datatype_format", + }: + cfg_dict = dict(metric_cfg) + if "reference_kg_path" in cfg_dict and "reference_kg" not in cfg_dict: + ref_path = Path(cfg_dict.pop("reference_kg_path")) + cfg_dict["reference_kg"] = _kg_from_path(ref_path) + if isinstance(cfg_dict.get("reference_kg"), (str, Path)): + cfg_dict["reference_kg"] = _kg_from_path(Path(cfg_dict["reference_kg"])) + out[metric_key] = ConsistencyViolationsConfig.model_validate(cfg_dict) + continue + + raise KeyError( + f"Unknown metric key {metric_key!r} in config. " + "Add it to `kgpipe_eval.config.manager.load_metric_configs`." + ) + + return out + + +def generate_default_config_dict() -> dict[str, Any]: + """ + Generate a complete default YAML config structure for all supported metric configs. + + This is intended as a *template* for users. Required values are filled with the + placeholder string `""`. + """ + # Shared sub-config defaults + entity_alignment_default = { + "method": "label_embedding", + # Prefer a path-based template: avoids embedding runtime `KG` objects into YAML. + "verified_entities_path": REQUIRED, + "verified_entities_delimiter": EntityAlignmentConfig.model_fields["verified_entities_delimiter"].default, + "entity_sim_threshold": EntityAlignmentConfig.model_fields["entity_sim_threshold"].default, + } + + return { + "entity_alignment_configs": { + "default": entity_alignment_default, + }, + "metrics": { + # Standalone metric uses EntityAlignmentConfig directly via a ref. + "entity_align": { + "entity_alignment_config_ref": "default", + }, + "duplicates": { + "entity_alignment_config_ref": "default", + }, + "triple_alignment": { + "reference_kg_path": REQUIRED, + "entity_alignment_config_ref": "default", + "value_sim_threshold": TripleAlignmentConfig.model_fields["value_sim_threshold"].default, + }, + # Consistency config currently requires both fields at type-level; + # template includes both so users can fill in one/both. + "consistency_violations": { + "reference_kg_path": REQUIRED, + "ontology_path": REQUIRED, + }, + }, + } + + +def generate_default_config_yaml() -> str: + """ + Return a YAML string (template) for `load_metric_configs`. + """ + cfg = generate_default_config_dict() + # Keep output stable and readable. + return yaml.safe_dump(cfg, sort_keys=False, default_flow_style=False) + + +def write_default_config_yaml(path: str | Path) -> Path: + """ + Write a default template YAML to disk and return the written path. + """ + out_path = Path(path) + out_path.write_text(generate_default_config_yaml(), encoding="utf-8") + return out_path + diff --git a/src/kgpipe_eval/evaluator.py b/src/kgpipe_eval/evaluator.py new file mode 100644 index 0000000..ec2f045 --- /dev/null +++ b/src/kgpipe_eval/evaluator.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +import inspect +from dataclasses import dataclass +from typing import Any, Dict, Iterable, List, Mapping, Sequence +import traceback + +from kgpipe_eval.api import Metric, MetricResult +from kgpipe_eval.utils.kg_utils import TripleGraph + + +def _metric_key(metric: Metric) -> str: + return getattr(metric, "key", metric.__class__.__name__) + + +@dataclass +class Evaluator: + """ + Execute multiple metrics against a KG and pass the right config (if any). + """ + + def run( + self, + kg: TripleGraph, + metrics: Sequence[Metric], + confs: Mapping[str, Any] | None = None, + ) -> List[MetricResult]: + confs = dict(confs or {}) + results: List[MetricResult] = [] + + for metric in metrics: + key = _metric_key(metric) + cfg = confs.get(key, confs.get(key.lower())) + + compute = getattr(metric, "compute", None) + if compute is None: + raise TypeError(f"Metric {key!r} has no compute() method") + + sig = inspect.signature(compute) + # Bound method: typically (kg) or (kg, config) + params = [ + p for p in sig.parameters.values() + if p.kind in (p.POSITIONAL_ONLY, p.POSITIONAL_OR_KEYWORD) + ] + + try: + if len(params) <= 1: + # compute(self) or compute(self, kg) -- call without config + res = compute(kg) if len(params) == 1 else compute() + else: + # compute(self, kg, config, ...) + if cfg is None: + raise KeyError( + f"Missing config for metric {key!r}. " + f"Provide `confs[{key!r}]`." + ) + res = compute(kg, cfg) + except Exception as e: + print(f"Failed running metric {key!r}: {e}") + print(traceback.format_exc()) + raise RuntimeError(f"Failed running metric {key!r}") from e + + if not isinstance(res, MetricResult): + raise TypeError(f"Metric {key!r} returned {type(res)!r}, expected MetricResult") + results.append(res) + + return results + diff --git a/src/kgpipe_eval/metrics/__init__.py b/src/kgpipe_eval/metrics/__init__.py new file mode 100644 index 0000000..fb29624 --- /dev/null +++ b/src/kgpipe_eval/metrics/__init__.py @@ -0,0 +1,132 @@ +from .statistics import CountMetric +from .triple_alignment import TripleAlignmentMetric +from .entity_alignment import EntityAlignmentMetric +from .duplicates import DuplicateMetric +from .consistency_violations import ( + DisjointDomainMetric, + DomainMetric, + RangeMetric, + RelationDirectionMetric, + DatatypeMetric, + DatatypeFormatMetric, +) + +__all__ = [ + "CountMetric", + "TripleAlignmentMetric", + "EntityAlignmentMetric", + "DuplicateMetric", + "DisjointDomainMetric", + "DomainMetric", + "RangeMetric", + "RelationDirectionMetric", + "DatatypeMetric", + "DatatypeFormatMetric", +] + +# @dataclass(frozen=True) +# class BinaryClassificationStats: +# tp: int +# fp: int +# tn: int +# fn: int + +# def recall(self) -> float: +# d = self.tp + self.fn +# return self.tp / d if d else 0.0 + +# def precision(self) -> float: +# d = self.tp + self.fp +# return self.tp / d if d else 0.0 + +# def f1(self) -> float: +# p = self.precision() +# r = self.recall() +# return 2 * p * r / (p + r) if (p + r) else 0.0 + +# def accuracy(self) -> float: +# d = self.tp + self.fp + self.tn + self.fn +# return (self.tp + self.tn) / d if d else 0.0 + +# def reference_binary_classification(kg: KgKg, config: MetricConfig) -> MetricResult: +# stats = BinaryClassificationStats(tp=10, fp=5, tn=15, fn=3) +# return MetricResult( +# metric_key="reference_binary_classification", +# summary="Reference comparison computed", +# measurements=[ +# Measurement("tp", stats.tp), +# Measurement("fp", stats.fp), +# Measurement("tn", stats.tn), +# Measurement("fn", stats.fn), +# Measurement("precision", stats.precision(), "ratio"), +# Measurement("recall", stats.recall(), "ratio"), +# Measurement("f1", stats.f1(), "ratio"), +# Measurement("accuracy", stats.accuracy(), "ratio"), +# ], +# ) + +# def graph_size(kg: KgKg, config: MetricConfig) -> MetricResult: +# size = 1532 +# return MetricResult( +# metric_key="graph_size", +# measurements=[ +# Measurement("triple_count", size, "triples") +# ], +# summary=f"Graph contains {size} triples", +# ) + +# def entity_duplication_rate(kg: KgKg, config: MetricConfig) -> MetricResult: +# duplicates = 7 +# total = 100 +# rate = duplicates / total if total else 0.0 +# return MetricResult( +# metric_key="entity_duplication_rate", +# measurements=[ +# Measurement("duplication_rate", rate, "ratio"), +# Measurement("duplicate_entities", duplicates, "entities"), +# Measurement("total_entities", total, "entities"), +# ], +# summary=f"Entity duplication rate: {rate:.2%}", +# ) +# --- + +# class BinaryClassifier(): +# tp: int +# fp: int +# tn: int +# fn: int + +# def recall(self) -> float: +# return self.tp / (self.tp + self.fn) + +# def precision(self) -> float: +# return self.tp / (self.tp + self.fp) + +# def f1(self) -> float: +# return 2 * self.precision() * self.recall() / (self.precision() + self.recall()) + +# def accuracy(self) -> float: +# return (self.tp + self.tn) / (self.tp + self.tn + self.fp + self.fn) + +# @lru_cache +# def compute_binary_classifier(kg: KgKg) -> BinaryClassifier: +# return BinaryClassifier(tp=10, fp=5, tn=15, fn=3) + + +# # Option 1 the metrics are recall, precision, f1, accuracy +# def reference_recall(kg: KgKg, reference: KgKg) -> KgMetricResult: +# binary_classifier = compute_binary_classifier(kg, reference) +# return KgMetricResult(summary=f"Reference recall: {binary_classifier.recall()}") + +# def reference_precision(kg: KgKg, reference: KgKg) -> KgMetricResult: +# binary_classifier = compute_binary_classifier(kg, reference) +# return KgMetricResult(summary=f"Reference precision: {binary_classifier.precision()}") + +# def reference_f1(kg: KgKg, reference: KgKg) -> KgMetricResult: +# binary_classifier = compute_binary_classifier(kg, reference) +# return KgMetricResult(summary=f"Reference F1: {binary_classifier.f1()}") + +# #Option 2 the metrics are Binary Classification which allows for more detailed analysis +# def reference_binary_classification(kg: KgKg, reference: KgKg) -> KgMetricResult: +# binary_classifier = compute_binary_classifier(kg, reference) +# return KgMetricResult(summary=f"Reference binary classification: {binary_classifier.tp}, {binary_classifier.fp}, {binary_classifier.tn}, {binary_classifier.fn}") \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/consistency_violations.py b/src/kgpipe_eval/metrics/consistency_violations.py new file mode 100644 index 0000000..7bbc1ae --- /dev/null +++ b/src/kgpipe_eval/metrics/consistency_violations.py @@ -0,0 +1,664 @@ +from kgpipe_eval.api import Metric, MetricResult, Measurement + +from pydantic import BaseModel, model_validator, ConfigDict +from kgpipe.common import KG +from pathlib import Path +from kgpipe_eval.utils.kg_utils import TripleGraph +from typing import Dict, Set, Optional + +from rdflib import URIRef, RDF, Literal, Graph, XSD +from rdflib.query import Result, ResultRow + +from kgcore.api.ontology import Ontology, OntologyUtil +from tqdm import tqdm + +def get_ontology_graph(ontology_path: Optional[Path], kg: KG) -> Graph: + if ontology_path is not None: + return Graph().parse(ontology_path) + elif kg is not None: + return kg.get_ontology_graph() + + +def enrich_type_information(graph: Graph, ontology: Ontology, type_property: URIRef = RDF.type) -> Graph: + type_dict = {} + + new_graph = Graph() + + for s, p, o in graph: + domain, range = ontology.get_domain_range(str(p)) + if domain and isinstance(s, URIRef): + if str(s) not in type_dict: + type_dict[str(s)] = [] + type_dict[str(s)].append(str(domain)) + if range and isinstance(o, URIRef): + if str(o) not in type_dict: + type_dict[str(o)] = [] + type_dict[str(o)].append(str(range)) + new_graph.add((s, p, o)) + + for uri, types in type_dict.items(): + for type in types: + new_graph.add((URIRef(uri), type_property, URIRef(type))) + return new_graph + +class ConsistencyViolationsConfig(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + reference_kg: Optional[KG] = None + ontology_path: Optional[Path] = None + + @model_validator(mode="after") + def _require_reference_kg_or_ontology_path(self): + if self.reference_kg is None and self.ontology_path is None: + raise ValueError("Provide either `reference_kg` or `ontology_path`.") + return self + +class DisjointDomainMetric(Metric): + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute disjoint domain score.""" + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + for s, p, o in ontology_graph.triples((None, None, None)): + graph.add((s, p, o)) + + # Get all disjoint domains + disjoint_domains_qr: Result = graph.query( + """ + SELECT DISTINCT ?subject + WHERE { + ?subject a ?disjointDomain1 . + ?subject a ?disjointDomain2 . + ?disjointDomain1 owl:disjointWith ?disjointDomain2 . + } + """ + ) + subjects_with_disjoint_domains = set([row["subject"] for row in disjoint_domains_qr if isinstance(row, ResultRow)]) + + subjects = set([str(s) for s in graph.subjects()]) + + return MetricResult( + metric=self, + measurements=[ + Measurement(name="subjects_with_disjoint_domains", value=len(subjects_with_disjoint_domains), unit="number"), + Measurement(name="subjects", value=len(subjects), unit="number"), + Measurement(name="normalized_score", value=1.0 - (len(subjects_with_disjoint_domains) / len(subjects)), unit="ratio"), + ], + summary=f"Number of subjects with disjoint domains: {len(subjects_with_disjoint_domains)}", + ) + +class DomainMetric(Metric): + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute incorrect relation domain score. + + TODO: check if this is correct for increment eval if namespace changes to former generic namespace not seed + """ + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + # disjoint class by class + disjoint_class_by_class : Dict[str, Set[str]] = {} + for class_ in ontology.classes: + if class_.disjointWith is not None: + disjoint_class_by_class[class_.uri] = class_.disjointWith + else: + disjoint_class_by_class[class_.uri] = set() + + + def is_subject_type(o, type): + # print(o, type) + if isinstance(o, URIRef): + types = [str(t) for _, _, t in graph.triples((o, RDF.type, None))] + return type in types and not any(str(other_type) in disjoint_class_by_class.get(str(type), set()) for other_type in types) + elif isinstance(o, Literal): + return o.datatype == type + else: + return False + + domain_by_property = {} + for property in ontology.properties: + if property.domain is not None: + domain_by_property[property.uri] = property.domain.uri + else: + print(f"Property {property.uri} has no domain") + domain_by_property[property.uri] = "TODO" + + incorrect_relation_domain = 0 + correct_relation_domain = 0 + + for s, p, o in graph.triples((None, None, None)): + if str(p) in domain_by_property: + if is_subject_type(s, domain_by_property[str(p)]): + correct_relation_domain += 1 + else: + incorrect_relation_domain += 1 + + if incorrect_relation_domain + correct_relation_domain > 0: + normalized_score = 1.0 - (incorrect_relation_domain / (incorrect_relation_domain + correct_relation_domain)) + else: + normalized_score = 0.0 + + return MetricResult( + metric=self, + measurements=[ + Measurement(name="incorrect_relation_domain", value=incorrect_relation_domain, unit="number"), + Measurement(name="correct_relation_domain", value=correct_relation_domain, unit="number"), + Measurement(name="normalized_score", value=normalized_score, unit="ratio"), + ], + summary=f"Number of incorrect relation domain: {incorrect_relation_domain}", + # name=self.name, + # value=incorrect_relation_domain, + # normalized_score=normalized_score, + # details={"incorrect_relation_domain": incorrect_relation_domain, "correct_relation_domain": correct_relation_domain}, + # aspect=self.aspect + ) + +class RangeMetric(Metric): + + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute incorrect relation range score.""" + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology : Ontology= OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + # disjoint class by class + disjoint_class_by_class : Dict[str, Set[str]] = {} + for class_ in ontology.classes: + if class_.disjointWith is not None: + disjoint_class_by_class[class_.uri] = class_.disjointWith + else: + disjoint_class_by_class[class_.uri] = set() + + def is_object_type(o, type): + # print(o, type) + if isinstance(o, URIRef): + types = [str(t) for s, p, t in graph.triples((o, RDF.type, None))] + # if str(type) not in types: + # print(f"Incorrect relation range {types} of {o} for property {p} with range {types}") + return str(type) in types and not any(str(other_type) in disjoint_class_by_class.get(str(type), set()) for other_type in types) + elif isinstance(o, Literal): + datatype = o.datatype + if not datatype: + datatype = str(XSD.string) + return str(datatype) == str(type) + else: + return False + + + range_by_property = {} + for property in ontology.properties: + if property.range is not None: + range_by_property[property.uri] = property.range.uri + else: + # print(f"Property {property.uri} has no range") + range_by_property[property.uri] = None + + incorrect_relation_range = 0 + correct_relation_range = 0 + + for s, p, o in graph.triples((None, None, None)): + if str(p) in range_by_property: + if is_object_type(o, range_by_property[str(p)]): + correct_relation_range += 1 + else: + # print(f"Incorrect relation range {o if isinstance(o, URIRef) else o.datatype} for property {p} with range {range_by_property[str(p)]}") + incorrect_relation_range += 1 + + normalized_score = 1.0 - (incorrect_relation_range / (incorrect_relation_range + correct_relation_range)) if incorrect_relation_range + correct_relation_range > 0 else 1.0 + """Compute incorrect relation range score.""" + return MetricResult( + metric=self, + measurements=[ + Measurement(name="incorrect_relation_range", value=incorrect_relation_range, unit="number"), + Measurement(name="correct_relation_range", value=correct_relation_range, unit="number"), + Measurement(name="normalized_score", value=normalized_score, unit="ratio"), + ], + summary=f"Number of incorrect relation range: {incorrect_relation_range}", + # name=self.name, + # value=incorrect_relation_range, + # normalized_score=normalized_score, + # details={"incorrect_relation_range": incorrect_relation_range, "correct_relation_range": correct_relation_range}, + # aspect=self.aspect + ) + +class RelationDirectionMetric(Metric): + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute incorrect relation direction score.""" + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + if len(ontology_graph) == 0: + ontology_graph = graph + print(f"INFO: ontology_graph is empty, using graph instead") + + # TODO use ontology implementation from framework + predicate_defs_sr = ontology_graph.query( + """ + SELECT DISTINCT ?predicate ?domain ?range + WHERE { + ?predicate rdfs:domain ?domain . + ?predicate rdfs:range ?range . + } + """ + ) + + # def check_type(uri, type): + # result = graph.query( + # """ + # SELECT ?uri + # WHERE { + # ?uri a ?type . + # } + # """, + # initBindings={"uri": uri, "type": type} + # ) + # return len(result) > 0 + + predicate_defs = {} + for row in predicate_defs_sr: + predicate_defs[str(row["predicate"])] = (str(row["domain"]), str(row["range"])) + + incorrect_relation_direction = 0 + correct_relation_direction = 0 + + entity_types = {} + for s, p, o in graph.triples((None, RDF.type, None)): + if str(s) not in entity_types: + entity_types[str(s)] = [] + entity_types[str(s)].append(str(o)) + + for s, p, o in tqdm(graph, desc="Checking relation direction"): + if str(s) not in entity_types: + continue + if str(p) in predicate_defs: + domain, range = predicate_defs[str(p)] + + if isinstance(o, URIRef): + if not str(s) in entity_types: + # print(f"Skipping s {s} because it is not in entity_types") + continue + if not str(o) in entity_types: + # print(f"Skipping o {o} because it is not in entity_types") + continue + if domain in entity_types[str(s)] and range in entity_types[str(o)]: + correct_relation_direction += 1 + if domain in entity_types[str(o)] and range in entity_types[str(s)]: + incorrect_relation_direction += 1 + + # print("incorrect_relation_direction", incorrect_relation_direction) + # print("correct_relation_direction", correct_relation_direction) + + if incorrect_relation_direction + correct_relation_direction > 0: + normalized_score = incorrect_relation_direction / (incorrect_relation_direction + correct_relation_direction) + normalized_score = 1.0 - normalized_score + else: + normalized_score = 0.0 + + return MetricResult( + metric=self, + measurements=[ + Measurement(name="incorrect_relation_direction", value=incorrect_relation_direction, unit="number"), + Measurement(name="correct_relation_direction", value=correct_relation_direction, unit="number"), + Measurement(name="normalized_score", value=normalized_score, unit="ratio"), + ], + summary=f"Number of incorrect relation direction: {incorrect_relation_direction}", + # name=self.name, + # value=incorrect_relation_direction, + # normalized_score=normalized_score, + # details={ + # "incorrect_relation_direction": incorrect_relation_direction, + # "correct_relation_direction": correct_relation_direction, + # "possible_relations": predicate_defs, + # "size_ontology_graph": len(ontology_graph) + # }, + # aspect=self.aspect + ) + +class DatatypeMetric(Metric): + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute incorrect datatype score.""" + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + def is_object_type(o, type): + # print(o, type) + if isinstance(o, URIRef): + types = [str(t) for s, p, t in graph.triples((o, RDF.type, None))] + # if str(type) not in types: + # print(f"Incorrect relation range {types} of {o} for property {p} with range {types}") + return str(type) in types + elif isinstance(o, Literal): + datatype = o.datatype + if not datatype: + datatype = str(XSD.string) + return str(datatype) == str(type) + else: + return False + + # def is_object_type(o, type): + # # print(o, type) + # if isinstance(o, URIRef): + # types = [str(t) for s, p, t in graph.triples((o, RDF.type, None))] + # return type in types + # elif isinstance(o, Literal): + # return str(o.datatype) == type + # else: + # return False + + range_by_property = {} + for property in ontology.properties: + if property.range is not None: + range_by_property[property.uri] = property.range.uri + else: + print(f"Property {property.uri} has no range") + range_by_property[property.uri] = "TODO" + + incorrect_datatype = 0 + correct_datatype = 0 + + for s, p, o in graph.triples((None, None, None)): + if str(p) in range_by_property: + if isinstance(o, Literal): + if not str(p) in range_by_property or is_object_type(o, range_by_property[str(p)]): + correct_datatype += 1 + else: + incorrect_datatype += 1 + # print(f"Incorrect datatype {o.datatype} for property {p} with range {range_by_property[str(p)]}") + + normalized_score = 1.0 - (incorrect_datatype / (incorrect_datatype + correct_datatype)) if incorrect_datatype + correct_datatype > 0 else 0.0 + + return MetricResult( + metric=self, + measurements=[ + Measurement(name="incorrect_datatype", value=incorrect_datatype, unit="number"), + Measurement(name="correct_datatype", value=correct_datatype, unit="number"), + Measurement(name="normalized_score", value=normalized_score, unit="ratio"), + ], + summary=f"Number of incorrect datatype: {incorrect_datatype}", + # name=self.name, + # value=incorrect_datatype, + # normalized_score=1.0 - (incorrect_datatype / (incorrect_datatype + correct_datatype)) if incorrect_datatype + correct_datatype > 0 else 0.0, + # details={"incorrect_datatype": incorrect_datatype, "correct_datatype": correct_datatype}, + # aspect=self.aspect + ) + +class DatatypeFormatMetric(Metric): + def compute(self, kg: TripleGraph, config: ConsistencyViolationsConfig): + """Compute incorrect datatype format score.""" + + from kgpipe.evaluation.aspects.func.datatype_validator import validate_datatype + + raw_graph: Graph = kg.get_graph() + ontology_graph: Graph = get_ontology_graph(config.ontology_path, config.reference_kg) + ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + graph = enrich_type_information(raw_graph, ontology) + + def is_object_type(o, type): + # print(o, type) + if isinstance(o, URIRef): + types = [str(t) for s, p, t in graph.triples((o, RDF.type, None))] + return type in types + elif isinstance(o, Literal): + return str(o.datatype) == type + else: + return False + + range_by_property = {} + for property in ontology.properties: + if property.range is not None: + range_by_property[property.uri] = property.range.uri + else: + print(f"Property {property.uri} has no range") + range_by_property[property.uri] = "TODO" + + incorrect_datatype = 0 + correct_datatype = 0 + + for s, p, o in graph.triples((None, None, None)): + if str(p) in range_by_property: + if isinstance(o, Literal): + if str(p) in range_by_property: + if validate_datatype(str(o), range_by_property[str(p)]): + # print(f"Correct datatype {o.datatype} for property {p} and value {o} with range {range_by_property[str(p)]}") + correct_datatype += 1 + else: + # print(f"Incorrect datatype {p} \'{o}\' {range_by_property[str(p)]}") + incorrect_datatype += 1 + else: + print(f"Property {p} has no range") + # if not str(p) in range_by_property: + # print(f"Property {p} has no range") + # # or validate_datatype(str(o), range_by_property[str(p)]): + # # print(f"Correct datatype {o.datatype} for property {p} and value {o} with range {range_by_property[str(p)]}") + # correct_datatype += 1 + # else: + # incorrect_datatype += 1 + + if incorrect_datatype + correct_datatype > 0: + normalized_score = 1.0 - (incorrect_datatype / (incorrect_datatype + correct_datatype)) + else: + normalized_score = 0.0 + + return MetricResult( + metric=self, + measurements=[ + Measurement(name="incorrect_datatype", value=incorrect_datatype, unit="number"), + Measurement(name="correct_datatype", value=correct_datatype, unit="number"), + Measurement(name="normalized_score", value=normalized_score, unit="ratio"), + ], + summary=f"Number of incorrect datatype: {incorrect_datatype}", + # name=self.name, + # value=incorrect_datatype, + # normalized_score=normalized_score, + # details={"incorrect_datatype": incorrect_datatype, "correct_datatype": correct_datatype}, + # aspect=self.aspect + ) + + +# @Registry.metric() +# class OntologyClassCoverageMetric(Metric): +# """Check if the KG has correct class coverage.""" +# def __init__(self): +# super().__init__( +# name="ontology_class_coverage", +# description="Check if the KG has correct class coverage", +# aspect=EvaluationAspect.SEMANTIC +# ) + +# def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: +# """Compute ontology class coverage score.""" + +# raw_graph: Graph = kg.get_graph() +# ontology_graph: Graph = kg.get_ontology_graph() +# ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) +# graph = enrich_type_information(raw_graph, ontology) + +# expected_classes = set([c.uri for c in ontology.classes if not c.uri.startswith(str(OWL))]) + +# found_classes = set(str(o) for s, p, o in graph.triples((None, RDF.type, None)) if not str(o).startswith(str(OWL))) + +# true_positive = len(expected_classes & found_classes) +# false_positive = len(found_classes - expected_classes) +# false_negative = len(expected_classes - found_classes) + +# precision = true_positive / (true_positive + false_positive) if true_positive + false_positive > 0 else 0.0 +# recall = true_positive / (true_positive + false_negative) if true_positive + false_negative > 0 else 0.0 +# f1_score = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0.0 + +# return MetricResult( +# name=self.name, +# value=true_positive, +# normalized_score=f1_score, +# details={"true_positive": true_positive, "false_positive": false_positive, "false_negative": false_negative}, +# aspect=self.aspect +# ) + +# @Registry.metric() +# class OntologyRelationCoverageMetric(Metric): +# """Check if the KG has correct relation coverage.""" +# def __init__(self): +# super().__init__( +# name="ontology_relation_coverage", +# description="Check if the KG has correct relation coverage", +# aspect=EvaluationAspect.SEMANTIC +# ) + +# def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: +# """Compute ontology relation coverage score.""" + +# raw_graph: Graph = kg.get_graph() +# ontology_graph: Graph = kg.get_ontology_graph() +# ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) +# graph = enrich_type_information(raw_graph, ontology) + +# NOT_FILTER: List[str] = [str(OWL), str(RDF), str(RDFS)] + +# expected_relations = set([r.uri for r in ontology.properties]) +# expected_relations = set([r for r in expected_relations if not any(filter(lambda x: r.startswith(x), NOT_FILTER))]) + +# # print(expected_relations) + +# found_relations = set(str(p) for _, p, _ in graph.triples((None, None, None))) +# def filter_relation(r): +# return any(filter(lambda x: r.startswith(x), NOT_FILTER)) +# found_relations = set([r for r in found_relations if not filter_relation(r)]) + +# # print(found_relations) + +# true_positive = len(expected_relations & found_relations) +# false_positive = len(found_relations - expected_relations) +# false_negative = len(expected_relations - found_relations) + +# precision = true_positive / (true_positive + false_positive) if true_positive + false_positive > 0 else 0.0 +# recall = true_positive / (true_positive + false_negative) if true_positive + false_negative > 0 else 0.0 +# f1_score = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0.0 + +# return MetricResult( +# name=self.name, +# value=true_positive, +# normalized_score=f1_score, +# details={"true_positive": true_positive, "false_positive": false_positive, "false_negative": false_negative, "missing": (expected_relations - found_relations)}, +# aspect=self.aspect +# ) + +# @Registry.metric() +# class OntologyPropertyCoverageMetric(Metric): +# """Check if the KG has correct property coverage.""" +# def __init__(self): +# super().__init__( +# name="ontology_property_coverage", +# description="Check if the KG has correct property coverage", +# aspect=EvaluationAspect.SEMANTIC +# ) + +# def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: +# """Compute ontology property coverage score.""" +# return MetricResult( +# name=self.name, +# value=0.0, +# normalized_score=1.0, +# details={"error": "Not implemented"}, +# aspect=self.aspect +# ) + +# @Registry.metric() +# class OntologyNamespaceCoverageMetric(Metric): +# """Check if the KG has correct namespace coverage.""" +# def __init__(self): +# super().__init__( +# name="ontology_namespace_coverage", +# description="Check if the KG has correct namespace coverage", +# aspect=EvaluationAspect.SEMANTIC +# ) + +# def compute(self, kg: KG, config: SemanticConfig, **kwargs) -> MetricResult: +# """Compute ontology namespace coverage score.""" + +# # graph = kg.get_graph() +# # ontology_graph = kg.get_ontology_graph() +# # if len(ontology_graph) == 0: +# # ontology_graph = graph + +# # ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + + +# return MetricResult( +# name=self.name, +# value=0.0, +# normalized_score=1.0, +# details={"error": "Not implemented"}, +# aspect=self.aspect +# ) + +# class OntologyClassCoverageMetric(): +# pass + +# class OntologyRelationCoverageMetric(): +# pass + +# class OntologyNamespaceCoverageMetric(): +# pass + +# Cardinality Metric + # """Compute incorrect relation cardinality score.""" + + # raw_graph: Graph = kg.get_graph() + # ontology_graph: Graph = kg.get_ontology_graph() + # ontology = OntologyUtil.load_ontology_from_graph(ontology_graph) + # graph = enrich_type_information(raw_graph, ontology) + # if len(ontology_graph) == 0: + # ontology_graph = graph + + # cardinality_by_property = {} + # property_cardinalities: Dict[str, Dict[str, int]] = defaultdict(lambda: defaultdict(int)) + # properties_in_graph = set() + + # for s, p, o in graph.triples((None, None, None)): + # properties_in_graph.add(str(p)) + + # for property in properties_in_graph: + # cardinality_by_property[property] = get_property_cardinality(ontology_graph, property) + + # # print(cardinality_by_property) + # # print(property_cardinalities) + + # for s, p, o in graph.triples((None, None, None)): + # if str(p) in cardinality_by_property: + # if str(s) in property_cardinalities[str(p)]: + # property_cardinalities[str(p)][str(s)] += 1 + # else: + # property_cardinalities[str(p)][str(s)] = 1 + + # incorrect_cardinality = 0 + # correct_cardinality = 0 + + # for property, cardinality in property_cardinalities.items(): + # min, max = cardinality_by_property[property] + # for subject, count in cardinality.items(): + # if count > max: + # incorrect_cardinality += 1 + # elif count < min: + # incorrect_cardinality += 1 + # else: + # correct_cardinality += 1 + + # return MetricResult( + # name=self.name, + # value=incorrect_cardinality, + # normalized_score=1.0 - (incorrect_cardinality / (incorrect_cardinality + correct_cardinality)) if incorrect_cardinality + correct_cardinality > 0 else 0.0, + # details={"incorrect_cardinality": incorrect_cardinality, "correct_cardinality": correct_cardinality}, + # aspect=self.aspect + # ) \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/duplicates.py b/src/kgpipe_eval/metrics/duplicates.py new file mode 100644 index 0000000..869f3ff --- /dev/null +++ b/src/kgpipe_eval/metrics/duplicates.py @@ -0,0 +1,71 @@ +from kgpipe_eval.utils.alignment_utils import EntityAlignment, align_entities_by_label_embedding, EntityAlignmentConfig +from kgpipe_eval.api import Metric, MetricResult, Measurement +from kgpipe_eval.utils.kg_utils import Term, TripleGraph + +from pydantic import BaseModel, ConfigDict +from kgpipe.common import KG +import numpy as np + +DEBUG = False + +class DuplicateConfig(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + entity_alignment_config: EntityAlignmentConfig + +class DuplicateMeasures(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + duplicates: int + total_references: int + already_matched_references: set[Term] + +def eval_duplicates(kg: TripleGraph, config: DuplicateConfig): + """ + checks expected & integrated source entity overlap using label embeddings + """ + + alignments : list[EntityAlignment] = align_entities_by_label_embedding(kg, config.entity_alignment_config) + + duplicates = set() + already_matched_references = set() + + for alignment in alignments: + if alignment.target in already_matched_references: + duplicates.add(alignment.target) + already_matched_references.add(alignment.target) + + if DEBUG: + print("Duplicates:") + for alignment in alignments: + if alignment.target in duplicates: + print(alignment.target, alignment.source, alignment.score) + + return duplicates + +class DuplicateMetric(Metric): + def compute(self, kg: TripleGraph, config: DuplicateConfig): + duplicates = eval_duplicates(kg, config) + entity_count = len(list(kg.entities())) + return MetricResult( + metric=self, + measurements=[ + Measurement(name="duplicates", value=len(duplicates), unit="number"), + Measurement(name="entity_count", value=entity_count, unit="number"), + Measurement(name="duplicates_ratio", value=len(duplicates) / entity_count, unit="percentage"), + ], + summary=f"Duplicates in the KG" + ) + +# find all duplicate entities in the KG +# using +# - reference KG +# - fuzzy matching +# - exact matching +# - semantic matching +# - clustering +# - ... +# return a list of duplicate entities +# return a list of duplicate entities with the matching score +# return a list of duplicate entities with the matching score and the matching type +# return a list of duplicate entities with the matching score and the matching type and the matching details +# return a list of duplicate entities with the matching score and the matching type and the matching details and the matching details +# return a list of duplicate entities with the matching score and the matching type and the matching details and the matching details and the matching details \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/entity_alignment.py b/src/kgpipe_eval/metrics/entity_alignment.py new file mode 100644 index 0000000..d6237cc --- /dev/null +++ b/src/kgpipe_eval/metrics/entity_alignment.py @@ -0,0 +1,163 @@ +from kgpipe.common import KG + +from kgpipe_eval.api import Metric, Measurement, MetricResult + +from kgpipe_eval.utils.measurement_utils import BCMeasurement +from kgpipe_eval.utils.alignment_utils import align_entities_by_label_embedding, EntityAlignmentConfig, load_entity_uri_label_type_pairs, get_entity_uri_label_typeset_pairs, get_entity_uri_label_type_pairs + +# Core Interface + +def eval_entity_alignment(kg: KG, config: EntityAlignmentConfig): + if config.method == "label_embedding": + alignments = eval_entity_alignment_by_label_embedding(kg, config) + elif config.method == "label_alias_embedding": + alignments = eval_entity_alignment_by_label_alias_embedding(kg, config) + elif config.method == "label_embedding_and_type": + alignments = eval_entity_alignment_by_label_embedding_and_type(kg, config) + elif config.method == "label_embedding_and_intersecting_type": + alignments = eval_entity_alignment_by_label_embedding_and_intersecting_type(kg, config) + else: + raise ValueError(f"Invalid method: {config.method}") + return alignments + +# Specific Implementations + +def eval_entity_alignment_by_label_embedding_and_type(kg: KG, config: EntityAlignmentConfig): + alignments = align_entities_by_label_embedding(kg, config) + + ref_entity_uri_label_type_pairs = load_entity_uri_label_type_pairs(config) + gen_entity_uri_label_type_pairs = list(get_entity_uri_label_type_pairs(kg, config.ignored_entities)) + + # print ref and gen pairs for testing + # print("--------------------------------") + # print("ref_entity_uri_label_type_pairs") + # for pair in ref_entity_uri_label_type_pairs: + # print(pair) + # print("--------------------------------") + # print("gen_entity_uri_label_type_pairs") + # for pair in gen_entity_uri_label_type_pairs: + # print(pair) + # print("--------------------------------") + # print("alignments") + # for alignment in alignments: + # print(alignment) + + ref_types = {pair.uri: pair.type for pair in ref_entity_uri_label_type_pairs if pair.type is not None} + # TODO gen_types can be multiple types, we need to handle this + gen_types = {pair.uri: pair.type for pair in gen_entity_uri_label_type_pairs if pair.type is not None} + + filtered_alignments = [] + for alignment in alignments: + if alignment.target in ref_types and alignment.source in gen_types: + if ref_types[alignment.target] == gen_types[alignment.source]: + filtered_alignments.append(alignment) + + ref_uris = set(pair.uri for pair in ref_entity_uri_label_type_pairs) + gen_uris = set(pair.uri for pair in gen_entity_uri_label_type_pairs) + aligned_gen_uris = set(alignment.target for alignment in filtered_alignments) + aligned_ref_uris = set(alignment.source for alignment in filtered_alignments) + + tp = len(ref_uris & aligned_gen_uris) # generated entities that are also in the reference + fp = len(gen_uris - aligned_ref_uris) # generated entities that are not in the reference + tn = 0 + fn = len(ref_uris - aligned_gen_uris) # missing generated entities that are in the reference + + return BCMeasurement( + tp=tp, + fp=fp, + tn=tn, + fn=fn + ) + +def eval_entity_alignment_by_label_embedding_and_intersecting_type(kg: KG, config: EntityAlignmentConfig): + # Debugging: print some information about the config + print("--------------------------------") + print("ignored_entities") + print(len(config.ignored_entities)) + print("--------------------------------") + + alignments = align_entities_by_label_embedding(kg, config) + + ref_entity_uri_label_type_pairs = load_entity_uri_label_type_pairs(config) + gen_entity_uri_label_type_pairs = list(get_entity_uri_label_typeset_pairs(kg, config.ignored_entities)) + + ref_types = {pair.uri: set([pair.type]) for pair in ref_entity_uri_label_type_pairs if pair.type is not None} + # TODO gen_types can be multiple types, we need to handle this + gen_types = {pair.uri: pair.type_set for pair in gen_entity_uri_label_type_pairs if pair.type_set is not None} + + filtered_alignments = [] + for alignment in alignments: + if alignment.target in ref_types and alignment.source in gen_types: + # Debugging: print the intersection of the reference and generated types + # print("---") + # print("alignment.target", alignment.target) + # print("alignment.source", alignment.source) + # print("ref_types[alignment.target]", ref_types[alignment.target]) + # print("gen_types[alignment.source]", gen_types[alignment.source]) + # print("intersection", ref_types[alignment.target] & gen_types[alignment.source]) + # print("---") + if len(ref_types[alignment.target] & gen_types[alignment.source]) > 0: + filtered_alignments.append(alignment) + + ref_uris = set(pair.uri for pair in ref_entity_uri_label_type_pairs) + gen_uris = set(pair.uri for pair in gen_entity_uri_label_type_pairs) + aligned_gen_uris = set(alignment.target for alignment in filtered_alignments) + aligned_ref_uris = set(alignment.source for alignment in filtered_alignments) + + tp = len(ref_uris & aligned_gen_uris) # generated entities that are also in the reference + fp = len(gen_uris - aligned_ref_uris) # generated entities that are not in the reference + tn = 0 + fn = len(ref_uris - aligned_gen_uris) # missing generated entities that are in the reference + + return BCMeasurement( + tp=tp, + fp=fp, + tn=tn, + fn=fn + ) + + +def eval_entity_alignment_by_label_embedding(kg: KG, config: EntityAlignmentConfig): + alignments = align_entities_by_label_embedding(kg, config) + + ref_entity_uri_label_type_pairs = load_entity_uri_label_type_pairs(config) + gen_entity_uri_label_type_pairs = list(get_entity_uri_label_type_pairs(kg)) + + ref_uris = set(pair.uri for pair in ref_entity_uri_label_type_pairs) + gen_uris = set(pair.uri for pair in gen_entity_uri_label_type_pairs) + aligned_gen_uris = set(alignment.target for alignment in alignments) + aligned_ref_uris = set(alignment.source for alignment in alignments) + + tp = len(ref_uris & aligned_gen_uris) # generated entities that are also in the reference + fp = len(gen_uris - aligned_ref_uris) # generated entities that are not in the reference + tn = 0 + fn = len(ref_uris - aligned_gen_uris) # missing generated entities that are in the reference + + return BCMeasurement( + tp=tp, + fp=fp, + tn=tn, + fn=fn + ) + +def eval_entity_alignment_by_label_alias_embedding(kg: KG, config: EntityAlignmentConfig): + raise NotImplementedError("Label alias embedding alignment is not implemented yet") + +# Metric Implementation + +class EntityAlignmentMetric(Metric): + def compute(self, kg: KG, config: EntityAlignmentConfig): + alignments: BCMeasurement = eval_entity_alignment(kg, config) + return MetricResult( + metric=self, + measurements=[ + Measurement(name="tp", value=alignments.tp, unit="number"), + Measurement(name="fp", value=alignments.fp, unit="number"), + Measurement(name="tn", value=alignments.tn, unit="number"), + Measurement(name="fn", value=alignments.fn, unit="number"), + Measurement(name="precision", value=alignments.precision(), unit="percentage"), + Measurement(name="recall", value=alignments.recall(), unit="percentage"), + Measurement(name="f1_score", value=alignments.f1_score(), unit="percentage"), + ], + summary=f"Entity alignment by {config.method}" + ) \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/llm_annotation.py b/src/kgpipe_eval/metrics/llm_annotation.py new file mode 100644 index 0000000..49b099a --- /dev/null +++ b/src/kgpipe_eval/metrics/llm_annotation.py @@ -0,0 +1,4 @@ +from kgpipe_eval.api import Metric + +class LLM_KgAccuracyMetric(Metric): + pass \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/statistics.py b/src/kgpipe_eval/metrics/statistics.py new file mode 100644 index 0000000..191776d --- /dev/null +++ b/src/kgpipe_eval/metrics/statistics.py @@ -0,0 +1,76 @@ +from kgpipe_eval.utils.kg_utils import TripleGraph +from kgpipe_eval.api import Metric, MetricResult, Measurement +from functools import lru_cache + +from pydantic import BaseModel +from typing import Mapping +from collections import defaultdict + +from rdflib import RDF, RDFS +from rdflib.term import URIRef, Literal + +class CountMeasures(BaseModel): + entity_count: int + triple_count: int + property_count: int + class_count: int + property_occurrence: Mapping[str, int] + class_occurrence: Mapping[str, int] + +# @lru_cache(maxsize=1) +def count_measures(kg: TripleGraph) -> CountMeasures: + + triple_count = 0 + subject_count = 0 # TODO misses shallow object entities + + class_occurrence = defaultdict(int) + property_occurrence = defaultdict(int) + + for _ in kg.subjects(): + subject_count += 1 + + for s, p, o in kg.triples((None, None, None)): + triple_count += 1 + if p == RDF.type: + class_occurrence[str(o)] += 1 + property_occurrence[str(p)] += 1 + + return CountMeasures( + entity_count=subject_count, + property_count=len(property_occurrence.keys()), + triple_count=triple_count, + class_count=len(class_occurrence.keys()), + class_occurrence=class_occurrence, + property_occurrence=property_occurrence, + ) + +class CountMetric(Metric): + key = "CountMetric" + description = "Counts triples/classes/properties (basic statistics)." + + def compute(self, kg: TripleGraph) -> MetricResult: + counts = count_measures(kg) + return MetricResult( + metric=self, + measurements=[ + Measurement(name="entity_count", value=counts.entity_count, unit="number"), + Measurement(name="triple_count", value=counts.triple_count, unit="number"), + Measurement(name="property_count", value=counts.property_count, unit="number"), + Measurement(name="class_count", value=counts.class_count, unit="number"), + Measurement(name="property_occurrence", value=counts.property_occurrence, unit="dictionary"), + Measurement(name="class_occurrence", value=counts.class_occurrence, unit="dictionary"), + ], + summary=f"Measures of entities, triples, properties, classes, property occurrences, and class occurrences" + ) + +class DegreeMetric(Metric): + # def compute(self, kg: TripleGraph) -> MetricResult: + # degrees = degree_measures(kg) + # return MetricResult( + # metric=self, + # measurements=[ + # Measurement(name="degree", value=degrees.degree, unit="number"), + # ], + # summary=f"Measures of degrees" + # ) + pass \ No newline at end of file diff --git a/src/kgpipe_eval/metrics/triple_alignment.py b/src/kgpipe_eval/metrics/triple_alignment.py new file mode 100644 index 0000000..5114ddb --- /dev/null +++ b/src/kgpipe_eval/metrics/triple_alignment.py @@ -0,0 +1,74 @@ +from pydantic import BaseModel, ConfigDict +from typing import Literal + +from kgpipe.common import KG +from kgpipe_eval.metrics.entity_alignment import EntityAlignmentConfig +from kgpipe_eval.utils.kg_utils import KgLike, KgManager, TripleGraph +from kgpipe_eval.utils.alignment_utils import align_triples_by_value_embedding +from kgpipe_eval.utils.measurement_utils import BCMeasurement +from kgpipe_eval.api import Measurement, Metric, MetricResult + +# measures precision, recall, f1 score, etc. + +class TripleAlignmentConfig(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + reference_kg: KgLike + method: Literal["value_embedding", "exact"] = "value_embedding" + entity_alignment_config: EntityAlignmentConfig + value_sim_threshold: float = 0.5 + cache_literal_embeddings: bool = False + cache_ref_literal_embeddings: bool = True + +def eval_triple_alignment(tg: TripleGraph, config: TripleAlignmentConfig): + if config.method == "value_embedding": + alignments = align_triples_by_value_embedding(tg, config) + elif config.method == "exact": + pass + # alignments = align_triples_by_exact_match(tg, config) + else: + raise ValueError(f"Invalid method: {config.method}") + + print("Triple alignments: ", len(alignments)) + + ref_tg = KgManager.load_kg(config.reference_kg) + ref_triples = set(ref_tg.triples((None, None, None))) + gen_triples = set(tg.triples((None, None, None))) + + aligned_ref_triples = set(a.target for a in alignments) + aligned_gen_triples = set(a.source for a in alignments) + + tp = len(aligned_ref_triples) # aligned reference triples + fp = len(gen_triples - aligned_gen_triples) # generated triples not aligned to any reference triple + tn = 0 + fn = len(ref_triples - aligned_ref_triples) # reference triples missing in generation + + return BCMeasurement(tp=tp, fp=fp, tn=tn, fn=fn) + +# def eval_triple_alignment_by_label_embedding(method: Literal["exact", "fuzzy", "semantic"] = "exact"): +# pass + + +# def eval_triple_alignment_by_label_embedding_soft_literals(method: Literal["exact", "fuzzy", "semantic"] = "exact"): +# pass + +class TripleAlignmentMetric(Metric): + + def compute(self, kg: KG, config: TripleAlignmentConfig): + m: BCMeasurement = eval_triple_alignment(kg, config) + return MetricResult( + metric=self, + measurements=[ + Measurement(name="tp", value=m.tp, unit="number"), + Measurement(name="fp", value=m.fp, unit="number"), + Measurement(name="tn", value=m.tn, unit="number"), + Measurement(name="fn", value=m.fn, unit="number"), + Measurement(name="precision", value=m.precision(), unit="percentage"), + Measurement(name="recall", value=m.recall(), unit="percentage"), + Measurement(name="f1_score", value=m.f1_score(), unit="percentage"), + ], + summary=f"Triple alignment by {config.method}", + ) + + +# Backward-compatibility alias (imported by `kgpipe_eval.metrics.__init__`). +# TripleAlignmentMetric = TripleAlignmentMetric \ No newline at end of file diff --git a/src/kgpipe_eval/test/__init__.py b/src/kgpipe_eval/test/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe_eval/test/examples.py b/src/kgpipe_eval/test/examples.py new file mode 100644 index 0000000..06ad14d --- /dev/null +++ b/src/kgpipe_eval/test/examples.py @@ -0,0 +1,257 @@ +SEED_TURTLE_TRIPLES = """ +@prefix : . +@prefix o: . +@prefix rdf: . +@prefix rdfs: . +@prefix xsd: . + +:store1 rdf:type o:BookStore ; + rdfs:label "Example Books (Downtown)"@en ; + :countryCode "US" ; + :hasInventory :itemA, :itemB, :itemC . + +:publisherHC rdf:type o:Publisher ; + rdfs:label "HarperCollins" ; + :countryCode "GB" . +""" +TEST_TURTLE_TRIPLES = """ +@prefix : . +@prefix o: . +@prefix rdf: . +@prefix rdfs: . +@prefix xsd: . + +# Entities designed to exercise alignment corner-cases: +# - multiple entities per type (Book/Author/Publisher/Store) +# - missing / extra attributes across graphs +# - literal variations (lang tags, datatypes, different lexical forms) +# - ambiguous labels (near-duplicates, casing differences) +# - multi-valued properties + +:store1 rdf:type o:BookStore ; + rdfs:label "Example Books (Downtown)"@en ; + :countryCode "US" ; + :hasInventory :itemA, :itemB, :itemC . + +:publisherHC rdf:type o:Publisher ; + rdfs:label "HarperCollins" ; + :countryCode "GB" . + +# different wrong type +:publisherPenguin rdf:type o:Author ; + rdfs:label "Penguin Books"@en ; + :countryCode "GB" . + +:authorTolkien rdf:type o:Author ; + rdfs:label "J. R. R. Tolkien" ; + :born "1892-01-03"^^xsd:date ; + :died "1973-09-02"^^xsd:date ; + :sameAs . + +:authorRowling rdf:type o:Author ; + rdfs:label "J.K. Rowling" ; + :born "1965-07-31"^^xsd:date . + +:itemA rdf:type o:Book ; + rdfs:label "The Hobbit"@en ; + :bookTitle "The Hobbit, or There and Back Again"@en ; + :bookAuthor :authorTolkien ; + :publisher :publisherHC ; + :isbn13 "9780261102217" ; + :pageCount "310"^^xsd:integer ; + :tags "fantasy", "classic" ; + :inSeries :seriesMiddleEarth . + +:itemB rdf:type o:Book ; + rdfs:label "The Hobbit (Illustrated)"@en ; + :bookTitle "The Hobbit"@en ; + :bookAuthor :authorTolkien ; + :publisher :publisherHC ; + :isbn13 "978-0-261-10221-7" ; # lexical variation + :pageCount 320 ; # integer without explicit datatype + :publicationYear "1997"^^xsd:gYear . + +:itemC rdf:type o:Book ; + rdfs:label "Harry Potter and the Philosopher's Stone"@en ; + :bookTitle "Harry Potter and the Philosopher's Stone"@en ; + :bookAuthor :authorRowling ; + :publisher :publisherPenguin ; + :isbn13 "9780747532699" ; + :pageCount "223"^^xsd:integer . + +# Same label, different type (common edge case for label-only alignment) +:hobbit rdf:type o:Film ; + rdfs:label "The Hobbit"@en ; + :releaseYear "2012"^^xsd:gYear . + +# Missing rdf:type but has label (edge case for type-aware matching) +:unknownEntity rdfs:label "HarperCollins" . + +:seriesMiddleEarth rdf:type o:Series ; + rdfs:label "Middle-earth Legendarium"@en . + +# false positive unexpected entity +:unexpectedEntity rdf:type o:Book ; + rdfs:label "Unexpected Entity"@en . +""" + +GENERATED_TURTLE_TRIPLES = """ +@prefix : . +@prefix o: . +@prefix rdf: . +@prefix rdfs: . +@prefix xsd: . + +# Entities designed to exercise alignment corner-cases: +# - multiple entities per type (Book/Author/Publisher/Store) +# - missing / extra attributes across graphs +# - literal variations (lang tags, datatypes, different lexical forms) +# - ambiguous labels (near-duplicates, casing differences) +# - multi-valued properties + +:store1 rdf:type o:BookStore ; + rdfs:label "Example Books (Downtown)"@en ; + :countryCode "US" ; + :hasInventory :itemA, :itemB, :itemC . + +:publisherHC rdf:type o:Publisher ; + rdfs:label "HarperCollins" ; + :countryCode "GB" . + +# different wrong type +:publisherPenguin rdf:type o:Author ; + rdfs:label "Penguin Books"@en ; + :countryCode "GB" . + +:authorTolkien rdf:type o:Author ; + rdfs:label "J. R. R. Tolkien" ; + :born "1892-01-03"^^xsd:date ; + :died "1973-09-02"^^xsd:date ; + :sameAs . + +:authorRowling rdf:type o:Author ; + rdfs:label "J.K. Rowling" ; + :born "1965-07-31"^^xsd:date . + +:itemA rdf:type o:Book ; + rdfs:label "The Hobbit"@en ; + :bookTitle "The Hobbit, or There and Back Again"@en ; + :bookAuthor :authorTolkien ; + :publisher :publisherHC ; + :isbn13 "9780261102217" ; + :pageCount "310"^^xsd:integer ; + :tags "fantasy", "classic" ; + :inSeries :seriesMiddleEarth . + +:itemB rdf:type o:Book ; + rdfs:label "The Hobbit (Illustrated)"@en ; + :bookTitle "The Hobbit"@en ; + :bookAuthor :authorTolkien ; + :publisher :publisherHC ; + :isbn13 "978-0-261-10221-7" ; # lexical variation + :pageCount 320 ; # integer without explicit datatype + :publicationYear "1997"^^xsd:gYear . + +:itemC rdf:type o:Book ; + rdfs:label "Harry Potter and the Philosopher's Stone"@en ; + :bookTitle "Harry Potter and the Philosopher's Stone"@en ; + :bookAuthor :authorRowling ; + :publisher :publisherPenguin ; + :isbn13 "9780747532699" ; + :pageCount "223"^^xsd:integer . + +# Same label, different type (common edge case for label-only alignment) +:hobbit rdf:type o:Film ; + rdfs:label "The Hobbit"@en ; + :releaseYear "2012"^^xsd:gYear . + +# Missing rdf:type but has label (edge case for type-aware matching) +:unknownEntity rdfs:label "HarperCollins" . + +:seriesMiddleEarth rdf:type o:Series ; + rdfs:label "Middle-earth Legendarium"@en . + +# false positive unexpected entity +:unexpectedEntity rdf:type o:Book ; + rdfs:label "Unexpected Entity"@en . +""" + +REFERENCE_TURTLE_TRIPLES = """ +@prefix : . +@prefix o: . +@prefix rdf: . +@prefix rdfs: . +@prefix xsd: . + +# Reference graph intentionally differs from TEST_TURTLE_TRIPLES: +# - different labels / casing / punctuation +# - extra / missing properties +# - alternate modeling (blank nodes, different predicates) +# - near-duplicate entities to test ambiguity + +:storeMain rdf:type o:BookStore ; + rdfs:label "Example Books - Downtown"@en ; + :countryCode "USA" ; # lexical variation + :hasInventory :refItemA, :refItemC . + +:publisherHC rdf:type o:Publisher ; + rdfs:label "Harper Collins"@en ; # spacing difference + :countryCode "UK" . + +:publisherPenguin rdf:type o:Publisher ; + rdfs:label "Penguin"@en ; + :countryCode "GB" . + +:authorTolkien rdf:type o:Author ; + rdfs:label "J.R.R. Tolkien" ; # punctuation difference + :born "1892-01-03"^^xsd:date ; + :sameAs ; + :nameParts [ :given "John" ; :middle "Ronald Reuel" ; :family "Tolkien" ] . + +:authorRowling rdf:type o:Author ; + rdfs:label "Joanne Rowling"@en ; # alias-ish label + :born "1965-07-31"^^xsd:date . + +:refItemA rdf:type o:Book ; + rdfs:label "The Hobbit"@en ; + :title "The Hobbit, or There and Back Again"@en ; # different predicate + :bookAuthor :authorTolkien ; + :publisher :publisherHC ; + :isbn13 "9780261102217" ; + :pageCount "310"^^xsd:integer ; + :tags "classic" . # missing one tag compared to test + +# Same-work but modeled as a separate edition entity +:refItemA_Edition1 rdf:type o:Edition ; + rdfs:label "The Hobbit (1st edition)"@en ; + :about :refItemA ; + :publicationYear "1937"^^xsd:gYear . + +:refItemC rdf:type o:Book ; + rdfs:label "Harry Potter and the Philosopher’s Stone"@en ; # curly apostrophe + :bookTitle "Harry Potter and the Philosopher's Stone"@en ; + :bookAuthor :authorRowling ; + :publisher :publisherPenguin ; + :isbn13 "9780747532699" ; + :pageCount "223"^^xsd:integer ; + :tags "fantasy"@en . + +# Near-duplicate label (to trigger ambiguity in label similarity) +:refItemC_US rdf:type o:Book ; + rdfs:label "Harry Potter and the Sorcerer's Stone"@en ; + :sameAs :refItemC . +""" + +VERIFIED_ENTITIES = """ +dataset,entity_id,entity_label,entity_type +test,http://example.org/reference_bookstore/itemA,The Hobbit,o:Book +test,http://example.org/reference_bookstore/itemB,The Hobbit (Illustrated),o:Book +test,http://example.org/reference_bookstore/itemC,Harry Potter and the Philosopher's Stone,o:Book +test,http://example.org/reference_bookstore/authorTolkien,J. R. R. Tolkien,o:Author +test,http://example.org/reference_bookstore/authorRowling,J.K. Rowling,o:Author +test,http://example.org/reference_bookstore/publisherHC,HarperCollins,o:Publisher +test,http://example.org/reference_bookstore/publisherPenguin,Penguin Books,o:Publisher +test,http://example.org/reference_bookstore/store1,Example Books (Downtown),o:BookStore +test,http://example.org/reference_bookstore/seriesMiddleEarth,Middle-earth Legendarium,o:Series +test,http://example.org/reference_bookstore/missingEntity,Gone with the Wind,o:Book +""" \ No newline at end of file diff --git a/src/kgpipe_eval/test/test_alignment_eval.py b/src/kgpipe_eval/test/test_alignment_eval.py new file mode 100644 index 0000000..9d10a93 --- /dev/null +++ b/src/kgpipe_eval/test/test_alignment_eval.py @@ -0,0 +1,62 @@ +import json + +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig +from kgpipe_eval.metrics.entity_alignment import EntityAlignmentMetric +from kgpipe_eval.metrics.triple_alignment import TripleAlignmentMetric, TripleAlignmentConfig +from kgpipe_eval.test.utils import get_test_kg, get_verified_entities_path, render_metric_result, get_reference_kg, get_generated_kg +from kgpipe_eval.utils.kg_utils import KgManager +from kgpipe_eval.api import MetricResult + + +def test_align_entities_by_label_embedding(): + config = EntityAlignmentConfig( + method="label_embedding", + reference_kg=None, + verified_entities_path=get_verified_entities_path(), + verified_entities_delimiter=",", + entity_sim_threshold=0.95 + ) + tg = KgManager.load_kg(get_test_kg()) + metric_result : MetricResult = EntityAlignmentMetric().compute(tg, config) + print(render_metric_result(metric_result)) + +def test_align_entities_by_label_embedding_and_type(): + config = EntityAlignmentConfig( + method="label_embedding_and_type", + reference_kg=None, + verified_entities_path=get_verified_entities_path(), + verified_entities_delimiter=",", + entity_sim_threshold=0.95 + ) + tg = KgManager.load_kg(get_test_kg()) + metric_result : MetricResult = EntityAlignmentMetric().compute(tg, config) + print(render_metric_result(metric_result)) + +def test_align_entities_by_label_embedding_and_type_ref_kg(): + config = EntityAlignmentConfig( + method="label_embedding", + reference_kg=get_reference_kg(), + verified_entities_path=None, + verified_entities_delimiter="\t", + entity_sim_threshold=0.95 + ) + tg = KgManager.load_kg(get_test_kg()) + metric_result : MetricResult = EntityAlignmentMetric().compute(tg, config) + print(render_metric_result(metric_result)) + +def test_align_triples_by_value_embedding(): + config = TripleAlignmentConfig( + reference_kg=get_reference_kg(), + entity_alignment_config=EntityAlignmentConfig( + method="label_embedding", + reference_kg=get_reference_kg(), + verified_entities_path=None, + verified_entities_delimiter="\t", + entity_sim_threshold=0.95 + ), + value_sim_threshold=0.5 + ) + tg = KgManager.load_kg(get_generated_kg()) + metric_result : MetricResult = TripleAlignmentMetric().compute(tg, config) + print(render_metric_result(metric_result)) + diff --git a/src/kgpipe_eval/test/test_config_manager.py b/src/kgpipe_eval/test/test_config_manager.py new file mode 100644 index 0000000..37af30d --- /dev/null +++ b/src/kgpipe_eval/test/test_config_manager.py @@ -0,0 +1,102 @@ +from __future__ import annotations + +from pathlib import Path + +from kgpipe_eval.config.manager import load_metric_configs, generate_default_config_dict +from kgpipe_eval.metrics.duplicates import DuplicateConfig +from kgpipe_eval.metrics.triple_alignment import TripleAlignmentConfig +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig + + +def test_load_metric_configs_resolves_entity_alignment_refs(tmp_path: Path) -> None: + cfg = tmp_path / "eval.yaml" + cfg.write_text( + """ +entity_alignment_configs: + default: + method: label_embedding + verified_entities_path: tmp_test_data/verified_entities.csv + verified_entities_delimiter: "," + entity_sim_threshold: 0.95 + +metrics: + entity_align: + entity_alignment_config_ref: default + + duplicates: + entity_alignment_config_ref: default + + triple_alignment: + reference_kg_path: tmp_test_data/reference.nt + entity_alignment_config_ref: default + value_sim_threshold: 0.6 +""".lstrip(), + encoding="utf-8", + ) + + loaded = load_metric_configs(cfg) + assert "entity_align" in loaded + assert "duplicates" in loaded + assert "triple_alignment" in loaded + + assert isinstance(loaded["entity_align"], EntityAlignmentConfig) + assert isinstance(loaded["duplicates"], DuplicateConfig) + assert isinstance(loaded["triple_alignment"], TripleAlignmentConfig) + + assert loaded["entity_align"].verified_entities_delimiter == "," + assert loaded["duplicates"].entity_alignment_config.verified_entities_delimiter == "," + assert loaded["triple_alignment"].entity_alignment_config.verified_entities_delimiter == "," + + # reference_kg is constructed from reference_kg_path + assert loaded["triple_alignment"].reference_kg.path.as_posix().endswith("tmp_test_data/reference.nt") + + +def test_generate_default_config_dict_has_all_sections() -> None: + cfg = generate_default_config_dict() + assert "entity_alignment_configs" in cfg + assert "metrics" in cfg + assert "default" in cfg["entity_alignment_configs"] + assert "verified_entities_path" in cfg["entity_alignment_configs"]["default"] + + metrics = cfg["metrics"] + assert "entity_align" in metrics + assert "duplicates" in metrics + assert "triple_alignment" in metrics + assert "consistency_violations" in metrics + + +def test_load_metric_configs_interpolates_vars_and_resolves_paths(tmp_path: Path) -> None: + # mirror the style used in experiments/examples/scripts/run_eval.yaml + cfg = tmp_path / "run_eval.yaml" + (tmp_path / "test.ttl").write_text( + """ +@prefix : . +@prefix rdfs: . +:a rdfs:label "A" . +""".lstrip(), + encoding="utf-8", + ) + cfg.write_text( + """ +reference_kg: test.ttl + +entity_alignment_configs: + default: + method: label_embedding + reference_kg: $reference_kg + entity_sim_threshold: 0.95 + +metrics: + duplicates: + entity_alignment_config_ref: default +""".lstrip(), + encoding="utf-8", + ) + + loaded = load_metric_configs(cfg) + assert isinstance(loaded["duplicates"], DuplicateConfig) + # reference_kg should be a KG whose path resolves relative to cfg location + kg = loaded["duplicates"].entity_alignment_config.reference_kg + assert kg is not None + assert kg.path == (tmp_path / "test.ttl").resolve() + diff --git a/src/kgpipe_eval/test/test_consistency_eval.py b/src/kgpipe_eval/test/test_consistency_eval.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe_eval/test/test_duplicates_eval.py b/src/kgpipe_eval/test/test_duplicates_eval.py new file mode 100644 index 0000000..1c21d89 --- /dev/null +++ b/src/kgpipe_eval/test/test_duplicates_eval.py @@ -0,0 +1,20 @@ +from kgpipe_eval.metrics.duplicates import DuplicateConfig +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig +from kgpipe_eval.test.utils import get_verified_entities_path +from kgpipe_eval.api import MetricResult +from kgpipe_eval.metrics.duplicates import DuplicateMetric +from kgpipe_eval.test.utils import get_test_kg, render_metric_result +from kgpipe_eval.utils.kg_utils import KgManager + +def test_duplicates_eval(): + config = DuplicateConfig( + entity_alignment_config=EntityAlignmentConfig( + method="label_embedding", + reference_kg=None, + verified_entities_path=get_verified_entities_path(), + verified_entities_delimiter=",", + entity_sim_threshold=0.95 + ) + ) + metric_result : MetricResult = DuplicateMetric().compute(KgManager.load_kg(get_test_kg()), config) + print(render_metric_result(metric_result)) diff --git a/src/kgpipe_eval/test/test_evaluator.py b/src/kgpipe_eval/test/test_evaluator.py new file mode 100644 index 0000000..2e2c03b --- /dev/null +++ b/src/kgpipe_eval/test/test_evaluator.py @@ -0,0 +1,32 @@ +from __future__ import annotations + +from kgpipe_eval.evaluator import Evaluator +from kgpipe_eval.metrics.statistics import CountMetric +from kgpipe_eval.metrics.duplicates import DuplicateMetric, DuplicateConfig +from kgpipe_eval.utils.alignment_utils import EntityAlignmentConfig +from kgpipe_eval.test.utils import get_test_kg, get_verified_entities_path +from kgpipe_eval.utils.kg_utils import KgManager + + +def test_evaluator_runs_metrics_with_and_without_config() -> None: + kg = KgManager.load_kg(get_test_kg()) + try: + dup_cfg = DuplicateConfig( + entity_alignment_config=EntityAlignmentConfig( + method="label_embedding", + verified_entities_path=get_verified_entities_path(), + verified_entities_delimiter=",", + entity_sim_threshold=0.95, + ) + ) + + metrics = [CountMetric(), DuplicateMetric()] + confs = {"DuplicateMetric": dup_cfg} + + results = Evaluator().run(kg=kg, metrics=metrics, confs=confs) + assert len(results) == 2 + assert results[0].metric.__class__.__name__ == "CountMetric" + assert results[1].metric.__class__.__name__ == "DuplicateMetric" + finally: + KgManager.unload_kg(kg) + diff --git a/src/kgpipe_eval/test/test_kg_utils.py b/src/kgpipe_eval/test/test_kg_utils.py new file mode 100644 index 0000000..39f10f0 --- /dev/null +++ b/src/kgpipe_eval/test/test_kg_utils.py @@ -0,0 +1,29 @@ +from kgpipe_eval.utils.kg_utils import KgManager +from kgpipe_eval.test.utils import get_test_kg, get_reference_kg +from pathlib import Path + +tmp_dir = Path("tmp_test_data") + +def test_substract_kg(): + # TODO test can be improved / cleaned up + kg = get_test_kg() + kg_graph = KgManager.load_kg(kg) + kg_path = kg.path + + # read kg + with open(kg_path, "r") as f: + triples = f.readlines() + sample_triples = triples[:10] + other_kg_path = tmp_dir / "other_kg.nt" + with open(other_kg_path, "w") as f: + f.write("\n".join(sample_triples)) + other_kg_graph = KgManager.load_kg(other_kg_path) + + substracted_kg_graph = KgManager.substract_kg(kg_graph, other_kg_graph) + len_kg_triples = len(list(kg_graph.triples((None, None, None)))) + len_other_kg_triples = len(list(other_kg_graph.triples((None, None, None)))) + len_substracted_kg_triples = len(list(substracted_kg_graph.triples((None, None, None)))) + # print(f"len_kg_triples: {len_kg_triples}") + # print(f"len_other_kg_triples: {len_other_kg_triples}") + # print(f"len_substracted_kg_triples: {len_substracted_kg_triples}") + assert len_substracted_kg_triples == len_kg_triples - len_other_kg_triples \ No newline at end of file diff --git a/src/kgpipe_eval/test/test_llm_eval.py b/src/kgpipe_eval/test/test_llm_eval.py new file mode 100644 index 0000000..b02ced2 --- /dev/null +++ b/src/kgpipe_eval/test/test_llm_eval.py @@ -0,0 +1,6 @@ +import pytest + +# @pytest.skip(reason="Long running test") +# def test_llm_eval(): +# pass + diff --git a/src/kgpipe_eval/test/test_metric_utils.py b/src/kgpipe_eval/test/test_metric_utils.py new file mode 100644 index 0000000..6ddc8c8 --- /dev/null +++ b/src/kgpipe_eval/test/test_metric_utils.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +import json +from pathlib import Path + +from kgpipe_eval.utils.metric_utils import eval_results_jsons_to_rows, write_eval_csv + + +def test_eval_results_json_to_rows_and_csv(tmp_path: Path) -> None: + # Create a fake output structure: //stage_1/eval_results.json + p = tmp_path / "rdf_a" / "stage_1" + p.mkdir(parents=True) + + (p / "eval_results.json").write_text( + json.dumps( + [ + { + "metric": "DuplicateMetric", + "summary": "Duplicates in the KG", + "measurements": [ + {"name": "duplicates", "value": 3, "unit": "number"}, + {"name": "entity_count", "value": 10, "unit": "number"}, + {"name": "duplicates_ratio", "value": 0.3, "unit": "percentage"}, + ], + } + ] + ) + ) + + allowlist = { + "DuplicateMetric": { + "duplicates": "number", + "entity_count": "number", + "duplicates_ratio": "percentage", + } + } + + rows = eval_results_jsons_to_rows([p / "eval_results.json"], allowlist=allowlist) + assert rows == [ + { + "pipeline": "rdf_a", + "stage": "stage_1", + "DuplicateMetric__duplicates__number": 3, + "DuplicateMetric__entity_count__number": 10, + "DuplicateMetric__duplicates_ratio__percentage": 0.3, + } + ] + + out_csv = tmp_path / "out.csv" + write_eval_csv([p / "eval_results.json"], out_path=out_csv, allowlist=allowlist) + txt = out_csv.read_text() + + # Header + one row, with stable columns including pipeline/stage and allowlist columns. + lines = [l for l in txt.splitlines() if l.strip()] + assert len(lines) == 2 + assert lines[0].split(",") == [ + "pipeline", + "stage", + "DuplicateMetric__duplicates__number", + "DuplicateMetric__duplicates_ratio__percentage", + "DuplicateMetric__entity_count__number", + ] + assert lines[1].split(",") == ["rdf_a", "stage_1", "3", "0.3", "10"] + diff --git a/src/kgpipe_eval/test/test_source_eval.py b/src/kgpipe_eval/test/test_source_eval.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe_eval/test/test_statistics_eval.py b/src/kgpipe_eval/test/test_statistics_eval.py new file mode 100644 index 0000000..67d803d --- /dev/null +++ b/src/kgpipe_eval/test/test_statistics_eval.py @@ -0,0 +1,8 @@ +from kgpipe_eval.metrics.statistics import CountMetric +from kgpipe_eval.test.utils import get_test_kg +from kgpipe_eval.utils.kg_utils import KgManager + +def test_count_metric(): + metric = CountMetric() + report = metric.compute(KgManager.load_kg(get_test_kg())) + print(report) \ No newline at end of file diff --git a/src/kgpipe_eval/test/utils.py b/src/kgpipe_eval/test/utils.py new file mode 100644 index 0000000..855b957 --- /dev/null +++ b/src/kgpipe_eval/test/utils.py @@ -0,0 +1,52 @@ +from pathlib import Path +from kgpipe.common import KG +from kgpipe.common.model.data import DataFormat +from kgpipe_eval.test.examples import * +from kgpipe_eval.api import MetricResult +from kgpipe_eval.utils.metric_utils import render_metric_result +from rdflib import Graph +import json +from collections.abc import Mapping, Sequence + +tmp_dir = Path("tmp_test_data") + +if not tmp_dir.exists(): + tmp_dir.mkdir(parents=True, exist_ok=True) + + +def get_test_kg(sample_size: int = -1) -> KG: + test_triples = TEST_TURTLE_TRIPLES + if sample_size > 0: + test_triples = test_triples[:sample_size] + # write test_triples to a file + g = Graph() + g.parse(data=test_triples, format="turtle") + g.serialize(destination=tmp_dir / "test.nt", format="ntriples") + return KG("test", name="test", path=tmp_dir / "test.nt", format=DataFormat.RDF_NTRIPLES) + +def get_generated_kg(sample_size: int = -1) -> KG: + generated_triples = GENERATED_TURTLE_TRIPLES + if sample_size > 0: + generated_triples = generated_triples[:sample_size] + # write generated_triples to a file + g = Graph() + g.parse(data=generated_triples, format="turtle") + g.serialize(destination=tmp_dir / "generated.nt", format="ntriples") + return KG("generated", name="generated", path=tmp_dir / "generated.nt", format=DataFormat.RDF_NTRIPLES) + +def get_reference_kg(sample_size: int = -1) -> KG: + reference_triples = REFERENCE_TURTLE_TRIPLES + if sample_size > 0: + reference_triples = reference_triples[:sample_size] + # write reference_triples to a file + g = Graph() + g.parse(data=reference_triples, format="turtle") + g.serialize(destination=tmp_dir / "reference.nt", format="ntriples") + return KG("reference", name="reference", path=tmp_dir / "reference.nt", format=DataFormat.RDF_NTRIPLES) + +def get_verified_entities_path() -> Path: + path = tmp_dir / "verified_entities.csv" + with open(path, "w") as f: + # Avoid a leading blank line which breaks csv.DictReader header parsing + f.write(VERIFIED_ENTITIES.lstrip().replace("o:", "http://example.org/ontology/")) + return path \ No newline at end of file diff --git a/src/kgpipe_eval/utils/__init__.py b/src/kgpipe_eval/utils/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/kgpipe_eval/utils/alignment_utils.py b/src/kgpipe_eval/utils/alignment_utils.py new file mode 100644 index 0000000..86774e4 --- /dev/null +++ b/src/kgpipe_eval/utils/alignment_utils.py @@ -0,0 +1,390 @@ +from transformers.models.t5gemma2.modeling_t5gemma2 import T5Gemma2ClassificationHead +from kgpipe.common import KG +from typing import TYPE_CHECKING, Literal, NamedTuple, Optional +from functools import lru_cache +from pydantic import BaseModel, ConfigDict, model_validator + +from kgpipe_eval.utils.kg_utils import TripleGraph, Term, Triple, KgLike, KgManager +from kgpipe.util.embeddings.st_emb import get_model + +from rdflib import RDFS, RDF +from rdflib.term import BNode +from rdflib.term import Literal as RdLiteral +from kgpipe.datasets.multipart_multisource import read_entities_csv, EntitiesRow +import numpy as np +from pathlib import Path +from tqdm import tqdm +from tqdm import tqdm +from typing import Set + +# TODO source entities csv to label only graph + +DEBUG = True + +class EntityAlignmentConfig(BaseModel): + model_config = ConfigDict(arbitrary_types_allowed=True) + method: Literal["label_embedding", "label_alias_embedding", "label_embedding_and_type", "label_embedding_and_intersecting_type"] = "label_embedding" + reference_kg: Optional[KgLike] = None + verified_entities_path: Optional[Path] = None + verified_entities_delimiter: str = "\t" + entity_sim_threshold: float = 0.95 + ignored_entities: Optional[Set[Term]] = None + + # value_sim_threshold: float = 0.5 + + @model_validator(mode="after") + def _require_reference_source(self): + if self.reference_kg is None and self.verified_entities_path is None: + raise ValueError("Provide either `reference_kg` or `verified_entities_path`.") + return self + + +EntityAlignment = NamedTuple("EntityAlignment", [("source", Term), ("target", Term), ("score", float)]) +TripleAlignment = NamedTuple("TripleAlignment", [("source", Triple), ("target", Triple)]) + +# Core alignment method interfaces + +@lru_cache(maxsize=1000) +def get_aligned_entities(kg: KG, reference_kg: KG, method: Literal["exact", "fuzzy", "semantic"] = "exact") -> list[EntityAlignment]: + return kg.entities.intersection(reference_kg.entities) + +def get_aligned_triples(kg: KG, reference_kg: KG, method: Literal["exact", "fuzzy", "semantic"] = "exact") -> list[TripleAlignment]: + return kg.triples.intersection(reference_kg.triples) + +# Helper methods + +# def get_entity_uri_label_pairs(triple_graph: TripleGraph) -> list[tuple[Term, Term]]: +# return [(s, label) for s, _, label in triple_graph.triples((None, RDFS.label, None))] + +UriLabelTypePair = NamedTuple("UriLabelTypePair", [("uri", Term), ("label", Term), ("type", Term)]) +UriLabelTypeSetPair = NamedTuple("UriLabelTypeSetPair", [("uri", Term), ("label", Term), ("type_set", set[Term])]) + +def get_entity_uri_label_type_pairs(kg: KG, ignored_entities: Optional[Set[Term]] = None) -> list[UriLabelTypePair]: + label_by_uri = {} + type_by_uri = {} + for s, p, o in kg.triples((None, RDFS.label, None)): + label_by_uri[str(s)] = str(o) + for s, p, o in kg.triples((None, RDF.type, None)): + type_by_uri[str(s)] = str(o) + for uri in label_by_uri: + if ignored_entities and str(uri) in ignored_entities: + continue + if uri in type_by_uri: + yield UriLabelTypePair(uri=uri, label=label_by_uri[uri], type=type_by_uri[uri]) + else: + yield UriLabelTypePair(uri=uri, label=label_by_uri[uri], type=None) + +def get_entity_uri_label_typeset_pairs(kg: KG, ignored_entities: Optional[Set[Term]] = None) -> list[UriLabelTypeSetPair]: + label_by_uri = {} + types_by_uri = {} + for s, p, o in kg.triples((None, RDFS.label, None)): + label_by_uri[str(s)] = str(o) + for s, p, o in kg.triples((None, RDF.type, None)): + if str(s) not in types_by_uri: + types_by_uri[str(s)] = set() + types_by_uri[str(s)].add(str(o)) + for uri in label_by_uri: + if ignored_entities and str(uri) in ignored_entities: + continue + if uri in types_by_uri: + yield UriLabelTypeSetPair(uri=uri, label=label_by_uri[uri], type_set=types_by_uri[uri]) + else: + yield UriLabelTypeSetPair(uri=uri, label=label_by_uri[uri], type_set=set()) + +def load_verified_entities(path: Path, delimiter: str = "\t") -> list[UriLabelTypePair]: + """ + """ + if path.name.endswith(".json"): + raise ValueError("JSON format not supported for verified entities") + elif path.name.endswith(".csv"): + return [UriLabelTypePair(uri=entity.entity_id, label=entity.entity_label, type=entity.entity_type) for entity in read_entities_csv(path=path, delimiter=delimiter)] + else: + raise ValueError(f"Unsupported file type: {path}") + +def load_entity_uri_label_type_pairs(config: EntityAlignmentConfig) -> list[UriLabelTypePair]: + if config.verified_entities_path is not None: + return load_verified_entities(config.verified_entities_path, delimiter=config.verified_entities_delimiter) + elif config.reference_kg is not None: + # `get_entity_uri_label_type_pairs` is a generator; downstream alignment uses indexing. + return list(get_entity_uri_label_type_pairs(KgManager.load_kg(config.reference_kg))) + else: + raise ValueError("No verified entities path or reference KG provided") + +# Specific alignment methods + +def align_entities_by_label_embedding(tg: TripleGraph, config: EntityAlignmentConfig) -> list[EntityAlignment]: + ref_entity_uri_label_type_pairs = load_entity_uri_label_type_pairs(config) + gen_entity_uri_label_type_pairs = list(get_entity_uri_label_type_pairs(tg, config.ignored_entities)) + # encode([]) yields shape (0,) which cannot matmul against (d, n_ref) + if not ref_entity_uri_label_type_pairs or not gen_entity_uri_label_type_pairs: + return [] + + model = get_model() + ref_labels = [pair.label for pair in ref_entity_uri_label_type_pairs] + gen_labels = [pair.label for pair in gen_entity_uri_label_type_pairs] + ref_labels_embeddings = model.encode(ref_labels, convert_to_numpy=True, show_progress_bar=False) + gen_labels_embeddings = model.encode(gen_labels, convert_to_numpy=True, show_progress_bar=False) + + sims = np.dot(gen_labels_embeddings, ref_labels_embeddings.T) + + alignments = [] + for i in range(sims.shape[0]): + best_j = np.argmax(sims[i]) + if sims[i][best_j] >= config.entity_sim_threshold: + alignments.append(EntityAlignment(source=gen_entity_uri_label_type_pairs[i].uri, target=ref_entity_uri_label_type_pairs[best_j].uri, score=sims[i][best_j])) + return alignments + +def align_by_label_alias_embedding(triple_graph: TripleGraph, model="", similarity="cosine", threshold=0.5): + pass + + +if TYPE_CHECKING: # avoid circular import at runtime + from kgpipe_eval.metrics.triple_alignment import TripleAlignmentConfig + + +def _is_literal(term: Term) -> bool: + return isinstance(term, RdLiteral) + + +def _literal_text(lit: RdLiteral) -> str: + # Prefer lexical form; fall back to python value string. + try: + return str(lit) + except Exception: + return str(lit.toPython()) + + +def align_triples_by_value_embedding(tg: TripleGraph, config: "TripleAlignmentConfig") -> list[TripleAlignment]: + """ + Align generated triples in `tg` to reference triples using: + - entity alignment (for URI/BNode subjects/objects) + - embedding similarity for literal object values (for same subject+predicate) + """ + ref_tg = KgManager.load_kg(config.reference_kg) + + # 0) Blank node mapping. + # + # rdflib assigns fresh IDs to BNodes on parse/load, so loading the "same" KG + # twice will not preserve BNode identifiers. We map BNodes by an outgoing-edge + # signature (predicate + object lexical form) to make exact-equal graphs align. + def _term_key(t: Term) -> str: + return str(t) + + def _bnode_signature(g: TripleGraph, b: BNode) -> tuple[tuple[str, str], ...]: + pairs: list[tuple[str, str]] = [] + for _, p, o in g.triples((b, None, None)): + if _is_literal(o): + ok = _literal_text(o) + else: + ok = _term_key(o) + pairs.append((_term_key(p), ok)) + pairs.sort() + return tuple(pairs) + + def _build_bnode_map(gen_g: TripleGraph, ref_g: TripleGraph) -> dict[str, Term]: + ref_by_sig: dict[tuple[tuple[str, str], ...], list[BNode]] = {} + for s, _, _ in ref_g.triples((None, None, None)): + if isinstance(s, BNode): + print(f"Ref bnode: {s}") + sig = _bnode_signature(ref_g, s) + ref_by_sig.setdefault(sig, []).append(s) + + gen_by_sig: dict[tuple[tuple[str, str], ...], list[BNode]] = {} + for s, _, _ in gen_g.triples((None, None, None)): + if isinstance(s, BNode): + print(f"Gen bnode: {s}") + sig = _bnode_signature(gen_g, s) + gen_by_sig.setdefault(sig, []).append(s) + + # Accept signature matches. If a signature occurs multiple times in both graphs, + # map deterministically by sorting node IDs and zipping. This makes identical KGs + # align even when they contain repeated blank-node structures. + out: dict[str, Term] = {} + for sig, gen_nodes in gen_by_sig.items(): + ref_nodes = ref_by_sig.get(sig, []) + if not ref_nodes: + continue + if len(gen_nodes) != len(ref_nodes): + continue + for gnode, rnode in zip(sorted(gen_nodes, key=_term_key), sorted(ref_nodes, key=_term_key)): + out[_term_key(gnode)] = rnode + return out + + gen_bnode_to_ref: dict[str, Term] = _build_bnode_map(tg, ref_tg) + + # 1) Entity alignments (generated -> reference) + ent_cfg = config.entity_alignment_config + if getattr(ent_cfg, "reference_kg", None) is None and getattr(ent_cfg, "verified_entities_path", None) is None: + # Ensure validator requirements are met; default to using the reference KG. + ent_cfg = ent_cfg.model_copy(update={"reference_kg": config.reference_kg}) + + entity_alignments = align_entities_by_label_embedding(tg, ent_cfg) + + if DEBUG: print("Entity alignments: ", len(entity_alignments)) + + def _as_term(t: Term | str) -> Term: + # Entity alignment currently carries string IDs; convert to rdflib Terms so + # aligned triples are comparable to `ref_tg.triples(...)` output. + if isinstance(t, str): + try: + from rdflib import URIRef + return URIRef(t) + except Exception: + # Fall back to raw string (will likely not match ref triples, but + # avoids crashing on non-URI identifiers). + return t # type: ignore[return-value] + return t + + gen_to_ref_entity: dict[str, Term] = {} + best_score_by_gen: dict[str, float] = {} + for a in entity_alignments: + gen_key = str(a.source) + if gen_key not in best_score_by_gen or a.score > best_score_by_gen[gen_key]: + best_score_by_gen[gen_key] = float(a.score) + gen_to_ref_entity[gen_key] = _as_term(a.target) + + # 2) Index generated triples, both raw and entity-mapped + mapped_gen_triples: list[tuple[Triple, Triple]] = [] # (raw_gen, mapped_to_ref_space) + gen_by_sp_literal: dict[tuple[Term, Term], list[tuple[Triple, str]]] = {} + gen_by_sp_entity: dict[tuple[Term, Term], set[Triple]] = {} + + if DEBUG: print("Gen by sp literal: ", len(gen_by_sp_literal)) + if DEBUG: print("Gen by sp entity: ", len(gen_by_sp_entity)) + + sp_iter = getattr(tg, "iter_sp_groups", None) + if callable(sp_iter): + sp_groups = sp_iter() + for s, p, os in sp_groups: + for o in os: + mapped_s = gen_to_ref_entity.get(str(s), gen_bnode_to_ref.get(str(s), s)) + mapped_o = gen_to_ref_entity.get(str(o), gen_bnode_to_ref.get(str(o), o)) if not _is_literal(o) else o + mapped = (mapped_s, p, mapped_o) + raw = (s, p, o) + mapped_gen_triples.append((raw, mapped)) + + # Normalize keys to string form to avoid rdflib Term vs string mismatches. + sp = (_term_key(mapped_s), _term_key(p)) + if _is_literal(o): + gen_by_sp_literal.setdefault(sp, []).append((raw, _literal_text(o))) + else: + gen_by_sp_entity.setdefault(sp, set()).add(raw) + else: + for s, p, o in tg.triples((None, None, None)): + mapped_s = gen_to_ref_entity.get(str(s), gen_bnode_to_ref.get(str(s), s)) + mapped_o = gen_to_ref_entity.get(str(o), gen_bnode_to_ref.get(str(o), o)) if not _is_literal(o) else o + mapped = (mapped_s, p, mapped_o) + raw = (s, p, o) + mapped_gen_triples.append((raw, mapped)) + + # Normalize keys to string form to avoid rdflib Term vs string mismatches. + sp = (_term_key(mapped_s), _term_key(p)) + if _is_literal(o): + gen_by_sp_literal.setdefault(sp, []).append((raw, _literal_text(o))) + else: + gen_by_sp_entity.setdefault(sp, set()).add(raw) + + if DEBUG: print("Mapped gen triples: ", len(mapped_gen_triples)) + + # 3) Prepare literal embedding caches (optional) + model = get_model() + alignments: list[TripleAlignment] = [] + + # Encode generated literal texts once, cache by text. + cache_gen_literals = bool(getattr(config, "cache_literal_embeddings", True)) + gen_lit_emb_by_text: dict[str, np.ndarray] = {} + if cache_gen_literals and gen_by_sp_literal: + unique_texts = sorted({txt for candidates in gen_by_sp_literal.values() for _, txt in candidates}) + if unique_texts: + emb = model.encode(unique_texts, convert_to_numpy=True, show_progress_bar=True) + gen_lit_emb_by_text = {t: emb[i : i + 1] for i, t in enumerate(unique_texts)} + + # Reference literal embedding cache (by text). + cache_ref_literals = bool(getattr(config, "cache_ref_literal_embeddings", True)) + ref_lit_emb_by_text: dict[str, np.ndarray] = {} + if cache_ref_literals: + unique_texts = sorted({_literal_text(ro) for _, _, ro in ref_tg.triples((None, None, None))}) + if unique_texts: + emb = model.encode(unique_texts, convert_to_numpy=True, show_progress_bar=True) + ref_lit_emb_by_text = {t: emb[i : i + 1] for i, t in enumerate(unique_texts)} + + def get_ref_literal_embedding(texts: list[str]) -> np.ndarray: + if cache_ref_literals and ref_lit_emb_by_text: + return np.concatenate([ref_lit_emb_by_text[t] for t in texts], axis=0) + else: + return model.encode(texts, convert_to_numpy=True, show_progress_bar=True) + + + def get_gen_literal_embedding(texts: list[str]) -> np.ndarray: + if cache_gen_literals and gen_lit_emb_by_text: + return np.concatenate([gen_lit_emb_by_text[t] for t in texts], axis=0) + else: + return model.encode(texts, convert_to_numpy=True, show_progress_bar=True) + + if DEBUG: print("gen_lit_emb_by_text: ", len(gen_lit_emb_by_text)) + if DEBUG: print("ref_lit_emb_by_text: ", len(ref_lit_emb_by_text)) + + from rdflib import Graph + gen_graph : Graph = tg._graph() + ref_graph : Graph = ref_tg._graph() + + if DEBUG: print("Gen graph: ", len(list(gen_graph.triples((None, None, None))))) + if DEBUG: print("Ref graph: ", len(list(ref_graph.triples((None, None, None))))) + + for gs, gp in tqdm(gen_graph.subject_predicates(unique=True), desc="Aligning triples by value embedding"): + # sp = (_term_key(gs), _term_key(gp)) + + # check for s mapping in reference space + ref_s = gen_to_ref_entity.get(str(gs), gen_bnode_to_ref.get(str(gs), gs)) + if ref_s is None: + continue # s is not mapped to reference space + + gen_objects = list(gen_graph.objects(gs, gp)) + gen_literal_objs = [o for o in gen_objects if _is_literal(o)] + + # IMPORTANT: query reference objects in reference-space subject + ref_objects = list(ref_graph.objects(ref_s, gp)) + ref_literal_objs = [o for o in ref_objects if _is_literal(o)] + + # print("gs: ", gs, "gp: ", gp) + # print("ref_s: ", ref_s) + # print("gen_literal_objs: ", len(gen_literal_objs)) + # print("ref_literal_objs: ", len(ref_literal_objs)) + # print("gen_objects: ", len(gen_objects)) + # print("ref_objects: ", len(ref_objects)) + + if len(gen_literal_objs) > 0 and len(ref_literal_objs) > 0: + + gen_object_texts = [_literal_text(o) for o in gen_literal_objs] + ref_object_texts = [_literal_text(o) for o in ref_literal_objs] + + gen_object_embeddings = get_gen_literal_embedding(gen_object_texts) + ref_object_embeddings = get_ref_literal_embedding(ref_object_texts) + + sims = np.dot(gen_object_embeddings, ref_object_embeddings.T) # shape (n_gen, n_ref) + best_flat = int(np.argmax(sims)) + best_i, best_j = np.unravel_index(best_flat, sims.shape) + + if float(sims[best_i, best_j]) >= float(config.value_sim_threshold): + alignments.append( + TripleAlignment( + source=(gs, gp, gen_literal_objs[best_i]), + target=(ref_s, gp, ref_literal_objs[best_j]), + ) + ) + + # get all non-literal objects mapped to reference space + gen_object_non_literal = [o for o in gen_objects if not _is_literal(o)] + ref_object_non_literal = [o for o in ref_objects if not _is_literal(o)] + + # find if any of the non-literal objects in the generated graph are mapped to the same object in the reference graph + for gen_obj in gen_object_non_literal: + ref_o = gen_to_ref_entity.get(str(gen_obj), gen_bnode_to_ref.get(str(gen_obj), gen_obj)) + if ref_o in ref_object_non_literal: + alignments.append( + TripleAlignment( + source=(gs, gp, gen_obj), + target=(ref_s, gp, ref_o), + ) + ) + + return alignments \ No newline at end of file diff --git a/src/kgpipe_eval/utils/annotation_utils.py b/src/kgpipe_eval/utils/annotation_utils.py new file mode 100644 index 0000000..c196a77 --- /dev/null +++ b/src/kgpipe_eval/utils/annotation_utils.py @@ -0,0 +1,51 @@ +from typing import Literal + +# Labels + +def get_labeled_entities(kg: KG, reference_kg: KG, method: Literal["exact", "fuzzy", "semantic"] = "exact") -> list[Entity]: + return kg.entities.intersection(reference_kg.entities) + +def get_labeled_triples(kg: KG, reference_kg: KG, method: Literal["exact", "fuzzy", "semantic"] = "exact") -> list[Triple]: + return kg.triples.intersection(reference_kg.triples) + + +def label_triples_with_llm(Triple): + """ +You are validating RDF triples. + +Task 1: +For each triple, decide whether it is: +- plausible in isolation +- implausible in isolation +- unclear + +Task 2: +Considering that all triples refer to the same subject node, decide whether the set is: +- coherent +- ambiguous +- conflated +- temporally inconsistent +- geographically inconsistent + +Task 3: +Explain which triples are mutually incompatible and why. +{ + "triple_labels": [ + { + "triple": ":Paris :locatedIn :France .", + "label": "plausible_in_isolation" + }, + { + "triple": ":Paris :population \"2,100,000\" .", + "label": "plausible_in_isolation" + }, + { + "triple": ":Paris :locatedIn :Texas .", + "label": "plausible_in_isolation" + } + ], + "entity_label": "conflated", + "graph_label": "contextually_incompatible", + "explanation": "The subject :Paris appears to merge Paris, France and Paris, Texas." +} + """ \ No newline at end of file diff --git a/src/kgpipe_eval/utils/entailment_utils.py b/src/kgpipe_eval/utils/entailment_utils.py new file mode 100644 index 0000000..19e8cb0 --- /dev/null +++ b/src/kgpipe_eval/utils/entailment_utils.py @@ -0,0 +1,7 @@ + + +def check_entailment(): + pass + +def check_entailment_by_llm(): + pass \ No newline at end of file diff --git a/src/kgpipe_eval/utils/kg_utils.py b/src/kgpipe_eval/utils/kg_utils.py new file mode 100644 index 0000000..9c304a0 --- /dev/null +++ b/src/kgpipe_eval/utils/kg_utils.py @@ -0,0 +1,207 @@ +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Protocol, Union, runtime_checkable, Optional, Tuple, Literal +from collections import defaultdict + +from rdflib import RDF, Graph, RDFS +from rdflib.term import Identifier, Literal, URIRef + +from kgpipe.common import KG + +KgLike = Union[KG, Graph, str, Path] + +Term = Union[Identifier, str, URIRef, Literal] + +Triple = tuple[Term, Term, Term] + +TriplePattern = Tuple[ + Optional[Term], Optional[Term], Optional[Term] +] + +@runtime_checkable +class TripleGraph(Protocol): + """ + TripleGraph is a protocol that defines the interface for a graph that can be used to evaluate metrics. + It is used to abstract the underlying graph implementation and allow for different graph implementations to be used. + + This is intentionally small: metrics should depend on *these* operations, + not on a specific in-memory representation (RDFLib Graph today; Spark later). + """ + + def triples(self, triple_pattern: TriplePattern) -> Iterable[Triple]: + pass + + def subjects(self) -> Iterable[Term]: + pass + + def entities(self) -> Iterable[Term]: + pass + + def labels(self, term: Term) -> Literal: + pass + + def types(self, term: Term) -> Iterable[Term]: + pass + + def close(self) -> None: + pass + + def cache(self) -> None: + pass + +# def iter_triples(self) -> Iterable[Triple]: +# """Iterate (s, p, o) triples.""" + +# @property +# def triples(self) -> frozenset[Triple]: +# """Materialized triple set (may be expensive).""" + +# @property +# def entities(self) -> frozenset[Term]: +# """All subjects/objects that are IRIs or blank nodes (no literals).""" + +# @property +# def relations(self) -> frozenset[Term]: +# """All predicates.""" + +# @property +# def classes(self) -> frozenset[Term]: +# """All classes used in rdf:type assertions.""" + +# @property +# def class_occurrences(self) -> Mapping[Term, int]: +# """Class → number of rdf:type occurrences.""" + +@dataclass(frozen=True) +class SparkTripleGraph(TripleGraph): + """ + KG backend that exposes evaluation-friendly views derived from a Spark DataFrame. + """ + # df: SparkDataFrame + + def triples(self, triple_pattern: TriplePattern) -> Iterable[Triple]: + # return self.df.filter(triple_pattern).collect() + pass + + def close(self) -> None: + pass + + def cache(self) -> None: + pass + +@dataclass(frozen=True) +class RdfLibTripleGraph(TripleGraph): + """ + KG backend that exposes evaluation-friendly views derived from an RDFLib `Graph`. + + Accepts: + - `kgpipe.common.KG` (uses `get_graph()`) + - an RDFLib `Graph` + - a path/str (parsed by RDFLib) + """ + kg: KgLike + + def _graph(self) -> Graph: + if isinstance(self.kg, Graph): + return self.kg + elif isinstance(self.kg, KG): + return self.kg.get_graph() + elif isinstance(self.kg, Path): + return Graph().parse(str(self.kg)) + elif isinstance(self.kg, str): + return Graph().parse(self.kg) + else: + raise ValueError(f"Unsupported KG type: {type(self.kg)}") + + def get_graph(self) -> Graph: + return self._graph() + + def get_ontology_graph(self) -> Graph: + if isinstance(self.kg, KG): + return self.kg.get_ontology_graph() + else: + raise ValueError(f"Unsupported KG type: {type(self.kg)}") + + def triples(self, triple_pattern: TriplePattern) -> Iterable[Triple]: + g = self._graph() + # RDFLib yields (s, p, o) as Identifiers + return g.triples(triple_pattern) + + def subjects(self) -> Iterable[Term]: + g = self._graph() + return g.subjects(unique=True) + + def iter_sp_groups(self) -> Iterable[tuple[Term, Term, list[Term]]]: + """Yield (s, p, [o1, o2, ...]) for all subjects/predicates.""" + g = self._graph() + by_sp: dict[tuple[Term, Term], list[Term]] = defaultdict(list) + for s, p, o in g.triples((None, None, None)): + by_sp[(s, p)].append(o) + for (s, p), objs in by_sp.items(): + yield (s, p, objs) + + def subject_predicate_pairs(self) -> Iterable[tuple[Term, Term]]: + """Yield (s, p) for all subjects/predicates.""" + g = self._graph() + return g.subject_predicates(unique=True) + + def objects(self, subject: Term, predicate: Term) -> Iterable[Term]: + """Yield (o) for all objects of (s, p).""" + g = self._graph() + return g.objects(subject, predicate) + + def entities(self) -> Iterable[Term]: + return self.subjects() # TODO inlcude objects that are not subjects + + def labels(self, term: Term) -> Literal: + g = self._graph() + return g.triples((term, RDFS.label, None)) + + def types(self, term: Term) -> Iterable[Term]: + g = self._graph() + return g.triples((term, RDF.type, None)) + +class KgManager: + """ + KgManager is a class that manages the loading and unloading of KGs. + It is used to abstract the underlying graph implementation and allow for different graph implementations to be used. + """ + + @staticmethod + def load_kg(kg: KgLike, backend: Literal["rdflib", "spark"] = "rdflib") -> TripleGraph: + if backend == "rdflib": + return RdfLibTripleGraph(kg=kg) + else: + raise ValueError(f"Unsupported backend: {backend}") + + @staticmethod + def load_kg_from_path(path: Path, backend: Literal["rdflib", "spark"] = "rdflib") -> TripleGraph: + if backend == "rdflib": + return RdfLibTripleGraph(kg=path) + else: + raise ValueError(f"Unsupported backend: {backend}") + + @staticmethod + def cache_kg(kg: TripleGraph) -> None: + kg.cache() + + @staticmethod + def unload_kg(kg: TripleGraph) -> None: + kg.close() + + + @staticmethod + def substract_kg(kg: TripleGraph, other_kg: TripleGraph) -> TripleGraph: + """ + Substract the other_kg from the kg. + """ + # TODO can be improved later by using a more efficient algorithm + triples = kg._graph().triples((None, None, None)) + other_triples = other_kg._graph() + new_graph = Graph() + for triple in triples: + if triple not in other_triples: + new_graph.add(triple) + return RdfLibTripleGraph(kg=new_graph) \ No newline at end of file diff --git a/src/kgpipe_eval/utils/measurement_utils.py b/src/kgpipe_eval/utils/measurement_utils.py new file mode 100644 index 0000000..065338b --- /dev/null +++ b/src/kgpipe_eval/utils/measurement_utils.py @@ -0,0 +1,47 @@ +from pydantic import BaseModel + +class BinaryClassificationMeasurement(BaseModel): + tp: int + fp: int + tn: int + fn: int + + def accuracy(self) -> float: + denom = (self.tp + self.tn + self.fp + self.fn) + return (self.tp + self.tn) / denom if denom else 0.0 + + def precision(self) -> float: + denom = (self.tp + self.fp) + return self.tp / denom if denom else 0.0 + + def recall(self) -> float: + denom = (self.tp + self.fn) + return self.tp / denom if denom else 0.0 + + def f1_score(self) -> float: + p = self.precision() + r = self.recall() + denom = (p + r) + return 2 * p * r / denom if denom else 0.0 + + def __str__(self): + return f"tp: {self.tp}, fp: {self.fp}, tn: {self.tn}, fn: {self.fn}, accuracy: {self.accuracy()}, precision: {self.precision()}, recall: {self.recall()}, f1_score: {self.f1_score()}" + + def to_dict(self) -> dict: + """ + Convenience export including derived measures. + + Note: do not override BaseModel internals like `__dict__`. + """ + return { + "tp": self.tp, + "fp": self.fp, + "tn": self.tn, + "fn": self.fn, + "accuracy": self.accuracy(), + "precision": self.precision(), + "recall": self.recall(), + "f1_score": self.f1_score(), + } + +BCMeasurement = BinaryClassificationMeasurement \ No newline at end of file diff --git a/src/kgpipe_eval/utils/metric_utils.py b/src/kgpipe_eval/utils/metric_utils.py new file mode 100644 index 0000000..71e0c71 --- /dev/null +++ b/src/kgpipe_eval/utils/metric_utils.py @@ -0,0 +1,197 @@ +from __future__ import annotations + +import csv +import json +from dataclasses import dataclass +from pathlib import Path +from collections.abc import Mapping, Sequence +from typing import Any, Iterable + +JsonValue = Any + + +@dataclass(frozen=True) +class MeasurementKey: + metric: str + measurement: str + unit: str + + +Allowlist = Mapping[str, Mapping[str, str]] + +from kgpipe_eval.api import MetricResult + + +def render_metric_result(metric_result: MetricResult, truncate: bool = False, truncate_value: int = 5) -> str: + """ + Render a MetricResult into a human-readable table-like string. + + This is intended for CLI/test output (not machine-parseable export). + """ + + def _metric_key(mr: MetricResult) -> str: + metric = mr.metric + return getattr(metric, "key", metric.__class__.__name__) + + def _fmt_value(v: Any) -> str: + if isinstance(v, float): + # stable, compact representation for test output + return f"{v:.6g}" + if isinstance(v, (int, bool)) or v is None: + return str(v) + if isinstance(v, str): + if truncate: + lines = v.splitlines()[:truncate_value] + return "\n".join(lines) + "\n..." + return v + if isinstance(v, Mapping): + rendered = json.dumps(v, indent=2, sort_keys=True, default=str) + if truncate: + return "\n".join(rendered.splitlines()[:truncate_value]) + "\n..." + return rendered + if isinstance(v, Sequence) and not isinstance(v, (str, bytes, bytearray)): + rendered = json.dumps(v, indent=2, sort_keys=True, default=str) + if truncate: + return "\n".join(rendered.splitlines()[:truncate_value]) + "\n..." + return rendered + return str(v) + + key = _metric_key(metric_result) + summary = metric_result.summary or "" + + ms = sorted(metric_result.measurements, key=lambda m: m.name) + name_w = max([len("measurement"), *(len(m.name) for m in ms)] or [len("measurement")]) + unit_w = max([len("unit"), *(len(m.unit or "") for m in ms)] or [len("unit")]) + + lines: list[str] = [] + lines.append("=" * 80) + lines.append(f"metric: {key}") + if summary: + lines.append(f"summary: {summary}") + if not ms: + lines.append("(no measurements)") + return "\n".join(lines) + + lines.append("") + lines.append(f"{'measurement':<{name_w}} {'value'}{' ' * max(1, 2)}{'unit':<{unit_w}}") + lines.append(f"{'-' * name_w} {'-' * 20} {'-' * unit_w}") + + for m in ms: + unit = m.unit or "" + rendered = _fmt_value(m.value) + rendered_lines = rendered.splitlines() or [""] + lines.append(f"{m.name:<{name_w}} {rendered_lines[0]:<20} {unit:<{unit_w}}") + for cont in rendered_lines[1:]: + lines.append(f"{'':<{name_w}} {cont}") + + return "\n".join(lines) + + +def parse_eval_results(path: Path) -> dict[MeasurementKey, JsonValue]: + """ + Parse a single `eval_results.json` and return a flattened mapping. + + Expected file schema (per entry): + - metric: str + - measurements: [{name: str, value: any-json, unit: str|null}, ...] + """ + raw = json.loads(path.read_text()) + if not isinstance(raw, list): + raise ValueError(f"{path} must contain a JSON list, got {type(raw).__name__}") + + out: dict[MeasurementKey, JsonValue] = {} + for entry in raw: + if not isinstance(entry, Mapping): + raise ValueError(f"{path} entries must be objects, got {type(entry).__name__}") + + metric = entry.get("metric") + if not isinstance(metric, str) or not metric: + raise ValueError(f"{path} entry missing 'metric' string") + + measurements = entry.get("measurements", []) + if not isinstance(measurements, list): + raise ValueError(f"{path} entry 'measurements' must be a list") + + for m in measurements: + if not isinstance(m, Mapping): + continue + name = m.get("name") + unit = m.get("unit") + if not isinstance(name, str) or not name: + continue + if unit is None: + unit = "" + if not isinstance(unit, str): + unit = str(unit) + out[MeasurementKey(metric=metric, measurement=name, unit=unit)] = m.get("value") + + return out + + +def allowlist_to_columns(allowlist: Allowlist) -> list[str]: + cols: list[str] = [] + for metric in sorted(allowlist.keys()): + for measurement in sorted(allowlist[metric].keys()): + unit = allowlist[metric][measurement] + cols.append(f"{metric}__{measurement}__{unit}") + return cols + + +def eval_results_jsons_to_rows( + paths: Sequence[Path], + *, + allowlist: Allowlist, +) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + + for path in paths: + if path.name != "eval_results.json": + raise ValueError(f"Expected eval_results.json file, got {path}") + stage_dir = path.parent + stage = stage_dir.name + if not stage.startswith("stage_"): + raise ValueError(f"Expected stage directory named stage_*, got {stage_dir}") + + pipeline_dir = stage_dir.parent + pipeline = pipeline_dir.name + if not pipeline: + raise ValueError(f"Could not derive pipeline name from {path}") + + flat = parse_eval_results(path) + + row: dict[str, Any] = {"pipeline": pipeline, "stage": stage} + for metric, measurements in allowlist.items(): + for measurement, unit in measurements.items(): + key = MeasurementKey(metric=metric, measurement=measurement, unit=unit) + col = f"{metric}__{measurement}__{unit}" + row[col] = flat.get(key, "") + + rows.append(row) + + return rows + + +def write_eval_csv( + paths: Sequence[Path], + *, + out_path: Path, + allowlist: Allowlist, + delimiter: str = ",", + round_ndigits: int | None = None, +) -> None: + rows = eval_results_jsons_to_rows(paths, allowlist=allowlist) + columns = ["pipeline", "stage", *allowlist_to_columns(allowlist)] + + out_path.parent.mkdir(parents=True, exist_ok=True) + with out_path.open("w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=columns, extrasaction="ignore", delimiter=delimiter) + writer.writeheader() + for r in rows: + # Ensure blanks for missing keys + row = {k: r.get(k, "") for k in columns} + if round_ndigits is not None: + for k, v in list(row.items()): + if isinstance(v, float): + row[k] = round(v, round_ndigits) + writer.writerow(row) + diff --git a/src/kgpipe_eval/utils/score_utils.py b/src/kgpipe_eval/utils/score_utils.py new file mode 100644 index 0000000..47ef814 --- /dev/null +++ b/src/kgpipe_eval/utils/score_utils.py @@ -0,0 +1,359 @@ +from __future__ import annotations + +import json +import math +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, List, Mapping, Sequence + +from kgpipe_eval.api import MetricResult +from kgpipe_eval.utils.metric_utils import MeasurementKey, parse_eval_results + +JsonMapping = Mapping[str, Any] +MeasurementLookup = Mapping[MeasurementKey, Any] + + +@dataclass(frozen=True) +class ResolvedMeasurement: + metric: str + measurement: str + value: float + weight: float = 1.0 + transform: str | None = None + + +@dataclass(frozen=True) +class SubgroupScore: + name: str + score: float + measurements: tuple[ResolvedMeasurement, ...] = () + + +@dataclass(frozen=True) +class AggregateScore: + final_score: float + subgroups: dict[str, SubgroupScore] = field(default_factory=dict) + + +_AGGREGATIONS = frozenset( + {"mean", "weighted_mean", "min", "max", "geometric_mean", "harmonic_mean", "product"} +) +_TRANSFORMS = frozenset({None, "identity", "invert", "one_minus"}) + + +def _as_float(value: Any, *, context: str) -> float: + if isinstance(value, bool): + raise ValueError(f"{context}: boolean values are not supported") + if isinstance(value, (int, float)): + return float(value) + raise ValueError(f"{context}: expected numeric value, got {type(value).__name__}") + + +def _apply_transform(value: float, transform: str | None) -> float: + if transform in (None, "identity"): + return value + if transform in ("invert", "one_minus"): + return 1.0 - value + raise ValueError(f"Unsupported transform: {transform!r}") + + +def _aggregate(values: Sequence[float], method: str, weights: Sequence[float] | None = None) -> float: + if not values: + raise ValueError("Cannot aggregate an empty list of values") + + if method == "mean": + return sum(values) / len(values) + + if method == "weighted_mean": + if weights is None: + raise ValueError("weighted_mean requires weights") + if len(weights) != len(values): + raise ValueError("weighted_mean requires one weight per value") + total_weight = sum(weights) + if total_weight <= 0: + raise ValueError("weighted_mean requires positive total weight") + return sum(v * w for v, w in zip(values, weights)) / total_weight + + if method == "min": + return min(values) + + if method == "max": + return max(values) + + if method == "product": + result = 1.0 + for value in values: + result *= value + return result + + if method == "geometric_mean": + if any(v < 0 for v in values): + raise ValueError("geometric_mean requires non-negative values") + if any(v == 0 for v in values): + return 0.0 + return math.exp(sum(math.log(v) for v in values) / len(values)) + + if method == "harmonic_mean": + if any(v < 0 for v in values): + raise ValueError("harmonic_mean requires non-negative values") + if any(v == 0 for v in values): + return 0.0 + return len(values) / sum(1.0 / v for v in values) + + raise ValueError(f"Unsupported aggregation method: {method!r}") + + +def _parse_measurement_ref(item: Any, *, subgroup: str) -> tuple[str, str, float, str | None]: + if isinstance(item, str): + if "." in item: + metric, measurement = item.split(".", 1) + elif ":" in item: + metric, measurement = item.split(":", 1) + else: + raise ValueError( + f"subgroup {subgroup!r}: measurement ref {item!r} must be " + "'MetricName.measurement' or 'MetricName:measurement'" + ) + return metric, measurement, 1.0, None + + if not isinstance(item, Mapping): + raise ValueError(f"subgroup {subgroup!r}: measurement item must be a mapping or string") + + metric = item.get("metric") + measurement = item.get("measurement") + if not isinstance(metric, str) or not metric: + raise ValueError(f"subgroup {subgroup!r}: measurement item missing 'metric'") + if not isinstance(measurement, str) or not measurement: + raise ValueError(f"subgroup {subgroup!r}: measurement item missing 'measurement'") + + weight = item.get("weight", 1.0) + transform = item.get("transform") + if not isinstance(weight, (int, float)): + raise ValueError(f"subgroup {subgroup!r}: weight for {metric}.{measurement} must be numeric") + if transform is not None and not isinstance(transform, str): + raise ValueError(f"subgroup {subgroup!r}: transform for {metric}.{measurement} must be a string") + if transform not in _TRANSFORMS: + raise ValueError(f"subgroup {subgroup!r}: unsupported transform {transform!r}") + + return metric, measurement, float(weight), transform + + +def _lookup_measurement( + measurements: MeasurementLookup, + *, + metric: str, + measurement: str, + subgroup: str, +) -> Any: + for key, value in measurements.items(): + if key.metric == metric and key.measurement == measurement: + return value + raise KeyError( + f"subgroup {subgroup!r}: measurement {metric}.{measurement} not found in eval results" + ) + + +def _resolve_subgroup( + name: str, + subgroup_cfg: JsonMapping, + measurements: MeasurementLookup, +) -> SubgroupScore: + if not isinstance(subgroup_cfg, Mapping): + raise ValueError(f"subgroup {name!r}: config must be an object") + + items = subgroup_cfg.get("measurements", subgroup_cfg.get("items", [])) + if not isinstance(items, list) or not items: + raise ValueError(f"subgroup {name!r}: 'measurements' must be a non-empty list") + + aggregation = subgroup_cfg.get("aggregation", subgroup_cfg.get("type", "mean")) + if not isinstance(aggregation, str) or aggregation not in _AGGREGATIONS: + raise ValueError(f"subgroup {name!r}: unsupported aggregation {aggregation!r}") + + default_transform = subgroup_cfg.get("transform") + if default_transform is not None and default_transform not in _TRANSFORMS: + raise ValueError(f"subgroup {name!r}: unsupported transform {default_transform!r}") + + resolved: list[ResolvedMeasurement] = [] + values: list[float] = [] + weights: list[float] = [] + + for item in items: + metric, measurement, weight, item_transform = _parse_measurement_ref(item, subgroup=name) + transform = item_transform if item_transform is not None else default_transform + raw_value = _lookup_measurement( + measurements, + metric=metric, + measurement=measurement, + subgroup=name, + ) + value = _apply_transform( + _as_float(raw_value, context=f"{name}.{metric}.{measurement}"), + transform, + ) + resolved.append( + ResolvedMeasurement( + metric=metric, + measurement=measurement, + value=value, + weight=weight, + transform=transform, + ) + ) + values.append(value) + weights.append(weight) + + score = _aggregate(values, aggregation, weights if aggregation == "weighted_mean" else None) + return SubgroupScore(name=name, score=score, measurements=tuple(resolved)) + + +def aggregate_scores( + measurements: MeasurementLookup | Sequence[Mapping[str, Any]] | Path | str, + config: JsonMapping, +) -> AggregateScore: + """ + Aggregate eval measurements into named subgroups, then into a final score. + + Config schema (dict / JSON): + + { + "subgroups": { + "coverage": { + "measurements": [ + {"metric": "EntityAlignmentMetric", "measurement": "recall"}, + {"metric": "TripleAlignmentMetric", "measurement": "recall", "weight": 2.0} + ], + "aggregation": "mean" + }, + "correctness": { + "measurements": [ + "EntityAlignmentMetric.precision", + "TripleAlignmentMetric.precision" + ], + "aggregation": "mean" + }, + "cleanliness": { + "measurements": [ + {"metric": "DuplicateMetric", "measurement": "duplicates_ratio", "transform": "invert"} + ], + "aggregation": "mean" + } + }, + "final": { + "aggregation": "weighted_mean", + "weights": { + "coverage": 0.4, + "correctness": 0.4, + "cleanliness": 0.2 + } + } + } + + Measurement refs may be objects or shorthand strings like ``MetricName.measurement``. + Supported subgroup/final aggregations: mean, weighted_mean, min, max, geometric_mean, + harmonic_mean, product. + Supported transforms: identity (default), invert / one_minus (``1 - value``). + """ + lookup = _coerce_measurement_lookup(measurements) + + subgroups_cfg = config.get("subgroups") + if not isinstance(subgroups_cfg, Mapping) or not subgroups_cfg: + raise ValueError("config must contain a non-empty 'subgroups' object") + + subgroup_scores: dict[str, SubgroupScore] = {} + for name, subgroup_cfg in subgroups_cfg.items(): + if not isinstance(name, str) or not name: + raise ValueError("subgroup names must be non-empty strings") + subgroup_scores[name] = _resolve_subgroup(name, subgroup_cfg, lookup) + + final_cfg = config.get("final", {}) + if not isinstance(final_cfg, Mapping): + raise ValueError("config 'final' must be an object") + + final_aggregation = final_cfg.get("aggregation", final_cfg.get("type", "weighted_mean")) + if not isinstance(final_aggregation, str) or final_aggregation not in _AGGREGATIONS: + raise ValueError(f"unsupported final aggregation {final_aggregation!r}") + + subgroup_names = list(subgroup_scores.keys()) + subgroup_values = [subgroup_scores[name].score for name in subgroup_names] + + final_weights_cfg = final_cfg.get("weights") + if final_aggregation == "weighted_mean": + if not isinstance(final_weights_cfg, Mapping): + raise ValueError("final weighted_mean requires a 'weights' object") + final_weights = [float(final_weights_cfg.get(name, 0.0)) for name in subgroup_names] + elif isinstance(final_weights_cfg, Mapping): + final_weights = [float(final_weights_cfg.get(name, 1.0)) for name in subgroup_names] + else: + final_weights = None + + final_score = _aggregate( + subgroup_values, + final_aggregation, + final_weights if final_aggregation == "weighted_mean" else None, + ) + return AggregateScore(final_score=final_score, subgroups=subgroup_scores) + + +def aggregate_scores_from_json( + eval_results_path: Path | str, + config: JsonMapping | Path | str, +) -> AggregateScore: + """Load eval results and config from JSON files and compute the aggregate score.""" + measurements = parse_eval_results(Path(eval_results_path)) + resolved_config = _coerce_config(config) + return aggregate_scores(measurements, resolved_config) + +def aggregate_scores_from_results( + results: List[MetricResult], + config: JsonMapping | Path | str, +) -> AggregateScore: + lookup: dict[MeasurementKey, Any] = {} + for result in results: + metric = getattr(result.metric, "key", result.metric.__class__.__name__) + for measurement in result.measurements: + lookup[ + MeasurementKey( + metric=metric, + measurement=measurement.name, + unit=measurement.unit or "", + ) + ] = measurement.value + resolved_config = _coerce_config(config) + return aggregate_scores(lookup, resolved_config) + +def _coerce_measurement_lookup( + measurements: MeasurementLookup | Sequence[Mapping[str, Any]] | Path | str, +) -> dict[MeasurementKey, Any]: + if isinstance(measurements, (str, Path)): + return parse_eval_results(Path(measurements)) + + if isinstance(measurements, Sequence) and not isinstance(measurements, (str, bytes, bytearray)): + if measurements and isinstance(measurements[0], Mapping) and "metric" in measurements[0]: + out: dict[MeasurementKey, Any] = {} + for entry in measurements: + if not isinstance(entry, Mapping): + continue + metric = entry.get("metric") + if not isinstance(metric, str): + continue + for m in entry.get("measurements", []): + if not isinstance(m, Mapping): + continue + name = m.get("name") + unit = m.get("unit") or "" + if isinstance(name, str) and name: + out[MeasurementKey(metric=metric, measurement=name, unit=str(unit))] = m.get("value") + return out + return dict(measurements) # type: ignore[arg-type] + + return dict(measurements) + + +def _coerce_config(config: JsonMapping | Path | str) -> JsonMapping: + if isinstance(config, (str, Path)): + path = Path(config) + loaded = json.loads(path.read_text()) + if not isinstance(loaded, Mapping): + raise ValueError(f"{path} must contain a JSON object") + return loaded + return config diff --git a/src/kgpipe_eval/utils/verbalize_utils.py b/src/kgpipe_eval/utils/verbalize_utils.py new file mode 100644 index 0000000..7b31cfc --- /dev/null +++ b/src/kgpipe_eval/utils/verbalize_utils.py @@ -0,0 +1,16 @@ +from kgpipe_eval.utils.kg_utils import Triple, TripleGraph, TriplePattern + +def verbalize_triple_simple(triple: Triple, TripleGraph) -> str: + """ + using label of subject, predicate, object to verbalize the triple + """ + return f"{triple[0]} {triple[1]} {triple[2]}" + +def verbalize_triples(triples: list[Triple]) -> list[str]: + pass + +def verbalize_triple_graph(triple_graph: TripleGraph) -> list[str]: + pass + +def verbalize_triple_graph_subject_groups(triple_graph: TripleGraph) -> list[list[str]]: + pass \ No newline at end of file diff --git a/src/kgpipe_llm/any_extraction.py b/src/kgpipe_llm/any_extraction.py new file mode 100644 index 0000000..9509030 --- /dev/null +++ b/src/kgpipe_llm/any_extraction.py @@ -0,0 +1,198 @@ +# Generalized variant of RDF triple generation + +from kgpipe.common import Registry, DataFormat, Data, TaskInput, TaskOutput +from kgpipe.common.model.configuration import ConfigurationDefinition, Parameter, ParameterType, ConfigurationProfile +from kgpipe_llm.common.snippets import generate_ontology_snippet_v3 +from kgcore.api.ontology import OntologyUtil +from pathlib import Path +from kgpipe_llm.common.core import LLMClient + +from shutil import RegistryError +from pydantic import BaseModel, AnyUrl + +# class OntologyGroundedSurfaceTriple(BaseModel): +# subject_label: str +# predicate_uri: AnyUrl +# object_label: str + +from pydantic import BaseModel, Field, AnyUrl + + +class SurfaceTriple(BaseModel): + subject: str = Field( + description="Surface-form subject label. Not a URI." + ) + predicate_uri: AnyUrl = Field( + description="Ontology property URI." + ) + object: str = Field( + description="Surface-form object label or literal. Not a URI." + ) + + +class SurfaceTripleExtractionResult(BaseModel): + triples: list[SurfaceTriple] + +# ontology-guided semantic triple extraction. +# surface semantic triples +# ontology-grounded surface triples + + +def get_ontology_grounded_surface_triples_prompt_template(ontology: str, input_data: str) -> str: + return """ +You are an ontology-guided semantic triple extraction system. + +Your task is to extract ontology-grounded surface triples from the provided input data. + +A valid triple has the form: + + + +Where: +- subject is a surface-form string, label, name, or textual identifier. +- predicate_uri is a URI from the provided ontology vocabulary. +- object is a surface-form string, label, value, literal, or textual identifier. +- subject and object MUST NOT be converted into URIs. +- predicate_uri MUST be selected only from the ontology vocabulary. +- Do not invent ontology properties. +- Do not invent facts not supported by the input. +- Prefer the most specific ontology property that correctly matches the input. +- If a relation or attribute is present in the input but cannot be mapped to the ontology, place it in unmapped_candidates. +- Preserve meaningful entity names as they appear in the input, normalizing only whitespace and obvious formatting artifacts. +- Extract both attributes and relations when they can be represented with an ontology property. +- Return only valid structured output matching the provided schema. + +Ontology vocabulary: + +{ontology} + +Input data: + +{input_data} + +Extraction guidance: +1. Identify named entities, records, rows, objects, or document subjects. +2. Identify attributes and relations expressed in the input. +3. Map each attribute or relation to the best matching ontology property URI. +4. Emit triples using string labels for subject and object. +5. Include evidence when possible. +6. Include confidence between 0.0 and 1.0. +7. Report unmapped relation or attribute candidates. +""".format(ontology=ontology, input_data=input_data) + +def extract_ontology_surface_triples(data: str, ontology: Path, client: LLMClient) -> SurfaceTripleExtractionResult: + + ontology_snippet = generate_ontology_snippet_v3(OntologyUtil.load_ontology_from_file(ontology)) + + prompt = get_ontology_grounded_surface_triples_prompt_template(ontology_snippet, data) + response = client.send_prompt(prompt, SurfaceTripleExtractionResult) + + return response + +@Registry.task( + input_spec={"input": DataFormat.ANY}, + output_spec={"output": DataFormat.RDF_NTRIPLES}, + description="Generate RDF triples for a schema", + config_spec=ConfigurationDefinition( + name="extract_ontology_surface_triples", + parameters=[ + Parameter( + name="ontology", + datatype=ParameterType.string, + description="The schema to generate RDF triples for" + ), + Parameter( + name="prompt_template", + datatype=ParameterType.string, + description="The prompt template to use for the LLM" + ), + ] + ) +) +def extract_ontology_surface_triples_task(input: TaskInput, output: TaskOutput, config: ConfigurationProfile): + pass + + + +# from typing import Any, Literal +# from pydantic import BaseModel, Field, AnyUrl + + +# class OntologyTerm(BaseModel): +# uri: AnyUrl = Field( +# description="The ontology URI identifying a class, attribute, or relation." +# ) +# label: str | None = Field( +# default=None, +# description="Optional human-readable label for the ontology term." +# ) +# description: str | None = Field( +# default=None, +# description="Optional description or definition of the ontology term." +# ) + + +# class OntologyGroundedSurfaceTriple(BaseModel): +# subject: str = Field( +# description="Surface-form name or label of the subject entity. This is not a URI." +# ) + +# predicate_uri: AnyUrl = Field( +# description="URI of the ontology property, attribute, or relation used as the predicate." +# ) + +# object: str = Field( +# description="Surface-form value, entity name, label, literal, or textual object. This is not a URI." +# ) + +# subject_type_uri: AnyUrl | None = Field( +# default=None, +# description="Optional ontology class URI for the subject, if inferable from the input and ontology." +# ) + +# object_type_uri: AnyUrl | None = Field( +# default=None, +# description="Optional ontology class URI for the object, if inferable from the input and ontology." +# ) + +# evidence: str | None = Field( +# default=None, +# description="Short quote or compact excerpt from the input that supports this triple." +# ) + +# confidence: float = Field( +# ge=0.0, +# le=1.0, +# description="Model confidence that the triple is correct and uses the appropriate ontology predicate." +# ) + + +# class TripleExtractionIssue(BaseModel): +# message: str = Field( +# description="Description of an ambiguity, missing ontology term, or extraction problem." +# ) + +# severity: Literal["info", "warning", "error"] = Field( +# description="Severity of the issue." +# ) + +# related_text: str | None = Field( +# default=None, +# description="Optional source text related to the issue." +# ) + + +# class OntologySurfaceTripleExtractionResult(BaseModel): +# triples: list[OntologyGroundedSurfaceTriple] = Field( +# description="Extracted ontology-grounded surface triples." +# ) + +# unmapped_candidates: list[str] = Field( +# default_factory=list, +# description="Candidate relations or attributes found in the input that could not be mapped to the ontology." +# ) + +# issues: list[TripleExtractionIssue] = Field( +# default_factory=list, +# description="Warnings or errors encountered during extraction." +# ) \ No newline at end of file diff --git a/src/kgpipe_llm/common/api_utils.py b/src/kgpipe_llm/common/api_utils.py index 2299af9..61ec431 100644 --- a/src/kgpipe_llm/common/api_utils.py +++ b/src/kgpipe_llm/common/api_utils.py @@ -1,373 +1,37 @@ -# specific LLM API utils (ollama, openai, etc.) -import json -import requests -from typing import Tuple, List -from typing import Optional, Dict, Any, Type -from pydantic import BaseModel -from enum import Enum -from tiktoken import encoding_for_model -import os - -TIMEOUT = 900 # 10 minutes - -# def schemadict_to_openai_tool( -# schema_dict: Dict[str, Any], -# model_name: str, -# *, -# name: Optional[str] = None, -# description: Optional[str] = None, -# additional_properties: bool = False, -# ) -> Dict[str, Any]: -# """ -# Convert a schema dictionary to an OpenAI 'tools' entry (function calling). -# """ - -# schema = schema_dict - -# # We want the object schema under "parameters" -# # Keep $defs so nested models/refs work. -# params: Dict[str, Any] = { -# "type": "object", -# "properties": schema.get("properties", {}), -# "required": schema.get("required", []), -# "additionalProperties": additional_properties, -# } -# if "$defs" in schema: -# params["$defs"] = schema["$defs"] - -# tool = { -# "type": "function", -# "function": { -# "name": name or model_name, -# "description": description or (model_name.__doc__ or "").strip() or f"{model_name} schema", -# "parameters": params, -# }, -# } -# return tool - -# def pydantic_to_openai_tool( -# model: Type[BaseModel], -# *, -# name: Optional[str] = None, -# description: Optional[str] = None, -# additional_properties: bool = False, -# ) -> Dict[str, Any]: -# """ -# Convert a Pydantic v2 model to an OpenAI 'tools' entry (function calling). -# """ -# # Pydantic v2 emits draft-2020-12 JSON Schema. OpenAI accepts schemas -# # that look like draft-07/2019-09 object schemas, including $defs/$ref. -# #schema = model.model_json_schema(ref_template="#/$defs/{model}") -# schema = model.model_json_schema() -# return schemadict_to_openai_tool(schema, model.__name__, name=name, description=description, additional_properties=additional_properties) - -from typing import Any, Dict, Optional, Type -from pydantic import BaseModel - -def schemadict_to_openai_tool( - schema_dict: Dict[str, Any], - *, - name: str, - description: Optional[str] = None, - additional_properties: Optional[bool] = None, -) -> Dict[str, Any]: - """ - Convert a schema dictionary to an OpenAI 'tools' entry (function calling). - Pass the schema through unchanged (array/object/etc.), only tweaking root-level keys. - """ - # Copy so we don't mutate the caller's schema - params = dict(schema_dict) - - # Titles are optional noise for tool schemas; drop them at root. - params.pop("title", None) - - # Only inject additionalProperties if the root is an object. - if additional_properties is not None and params.get("type") == "object": - params["additionalProperties"] = additional_properties - - tool = { - "type": "function", - "function": { - "name": name, - "description": (description or schema_dict.get("description") or f"{name} parameters").strip(), - "parameters": params, - }, - } - return tool - - -def pydantic_to_openai_tool( - model: Type[BaseModel], - *, - name: Optional[str] = None, - description: Optional[str] = None, - additional_properties: Optional[bool] = None, -) -> Dict[str, Any]: - """ - Convert a Pydantic v2 model to an OpenAI 'tools' entry (function calling). - Works for object models, RootModel[list[...]], unions, literals, etc. - """ - schema = model.model_json_schema() - resolved_name = name or model.__name__ - resolved_description = ( - description - or (model.__doc__ or "").strip() - or schema.get("description") - or f"{resolved_name} parameters" - ) - return schemadict_to_openai_tool( - schema, - name=resolved_name, - description=resolved_description, - additional_properties=additional_properties, - ) - - -# def schemadict_to_openai_tool( -# schema_dict: Dict[str, Any], -# *, -# name: str, -# description: Optional[str] = None, -# additional_properties: Optional[bool] = None, -# ) -> Dict[str, Any]: -# """ -# Convert a schema dictionary to an OpenAI 'tools' entry (function calling). -# """ -# params: Dict[str, Any] = { -# "type": "object", -# "properties": schema_dict.get("properties", {}), -# "required": schema_dict.get("required", []), -# } -# # Only set this if the caller asked to, otherwise leave Pydantic's default intact. -# if additional_properties is not None: -# params["additionalProperties"] = additional_properties - -# # Keep nested refs/defs -# if "$defs" in schema_dict: -# params["$defs"] = schema_dict["$defs"] - -# tool = { -# "type": "function", -# "function": { -# "name": name, -# "description": (description or schema_dict.get("description") or f"{name} parameters").strip(), -# "parameters": params, -# }, -# } -# return tool - - -# def pydantic_to_openai_tool( -# model: Type[BaseModel], -# *, -# name: Optional[str] = None, -# description: Optional[str] = None, -# additional_properties: Optional[bool] = None, -# ) -> Dict[str, Any]: -# """ -# Convert a Pydantic v2 model to an OpenAI 'tools' entry (function calling). -# """ -# # Pydantic v2 emits draft-2020-12 JSON Schema (with $defs). That's fine for OpenAI tools. -# schema = model.model_json_schema() - -# resolved_name = name or model.__name__ -# # Prefer explicit description → model docstring → schema description → fallback -# resolved_description = ( -# description -# or (model.__doc__ or "").strip() -# or schema.get("description") -# or f"{resolved_name} parameters" -# ) - -# return schemadict_to_openai_tool( -# schema, -# name=resolved_name, -# description=resolved_description, -# additional_properties=additional_properties, -# ) +"""Compatibility facade for API-specific LLM helpers. -def openai_call_with_json_out( - *, - endpoint_url: str, - api_key: str, - model_name: str, - user_content: str, - system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", - seed: str = "", - response_format: str = "json_object" -) -> Dict[str, Any]: +Provider implementations live under ``kgpipe_llm.common.apis``. +""" - payload = { - "model": model_name, - "messages": [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_content}, - ], +from __future__ import annotations - "temperature": 1 - } - - if response_format and response_format != "": - print(f"INFO: openai_call_with_json_out response_format json_object") - payload["response_format"] = { - "type": "json_object" - } - - if seed and seed != "": - payload["seed"] = seed - - headers = { - "Content-Type": "application/json", - "Authorization": f"Bearer {api_key}", - } - - resp = requests.post(endpoint_url, headers=headers, json=payload, timeout=TIMEOUT) - if resp.status_code != 200: - print(resp.content.decode("utf-8")) - resp.raise_for_status() - resp_data = resp.json() - - content = resp_data["choices"][0]["message"]["content"] - - try: - return json.loads(content) - except Exception as e: - print(f"Error parsing JSON: {e}") - return content - -def openai_call_with_tool( - *, - endpoint_url: str, - api_key: str, - model_name: str, - user_content: str, - pyd_model: Type[BaseModel] | Dict, - system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", - seed: str = "" -) -> Tuple[dict, BaseModel]: - if isinstance(pyd_model, dict): - print(f"INFO: openai_call_with_tool CUSTOM JSON SCHEMA") - tool = schemadict_to_openai_tool(pyd_model, name="CustomJsonSchema", additional_properties=True) - else: - print(f"INFO: openai_call_with_tool Pydantic model {pyd_model.__name__}") - tool = pydantic_to_openai_tool(pyd_model, additional_properties=False) - - payload = { - "model": model_name, - "messages": [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_content}, - ], - "tools": [tool], - # Force the model to call our function so we get structured JSON back - "tool_choice": {"type": "function", "function": {"name": tool["function"]["name"]}}, - "temperature": 1 - } - - if seed and seed != "": - payload["seed"] = seed - - headers = { - "Content-Type": "application/json", - "Authorization": f"Bearer {api_key}", - } - - resp = requests.post(endpoint_url, headers=headers, json=payload, timeout=TIMEOUT) - if resp.status_code != 200: - print(resp.content.decode("utf-8")) - resp.raise_for_status() - data = resp.json() - - # Extract tool call arguments - choice = data["choices"][0] - tool_calls = choice["message"].get("tool_calls", []) - if not tool_calls: - raise ValueError("Model did not return a tool call; check tool_choice or prompt.") - - args_json_str = tool_calls[0]["function"]["arguments"] - args = json.loads(args_json_str) - - # Validate using Pydantic - if isinstance(pyd_model, dict): - validated = args - else: - validated = pyd_model.model_validate(args) - return args, validated - - -def ollama_call( - *, - endpoint_url: str, - api_key: str, - schema_class: Type[BaseModel] | Dict | str, - model_name: str, - user_content: str, - system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", - seed: str = "" -) -> Dict[str, Any]: - payload = { - "model": model_name, - "prompt": user_content, - "stream": False, - } - - # If schema_class is a string, we want raw output - if isinstance(schema_class, str): - # For raw output, don't set format - pass - elif isinstance(schema_class, Dict): - payload["format"] = schema_class - else: - # For structured output, set the JSON schema format - payload["format"] = schema_class.model_json_schema() - - if system_prompt and system_prompt != "": - payload["system"] = system_prompt - - if seed and seed != "": - payload["seed"] = seed - - headers = { - "Content-Type": "application/json", - } +import os - if api_key and api_key != "": - headers["Authorization"] = f"Bearer {api_key}" - headers["X-API-Key"] = api_key +from tiktoken import encoding_for_model - try: - response = requests.post( - endpoint_url, - headers=headers, - json=payload, - timeout=300 - ) - - if response.status_code == 200: - raw_output = response.json()["response"] - if isinstance(schema_class, str): - return raw_output - elif isinstance(schema_class, Dict): - return json.loads(raw_output) - else: - parsed_output = json.loads(raw_output) - schema_class.model_validate(parsed_output) - return parsed_output - else: - print(f"Request failed: {response.status_code} - {response.text}") - return {} - - except Exception as e: - print(f"Error processing LLM response: {e}") - return {} +from .apis.ollama_comp import ollama_call +from .apis.openai_comp import ( + openai_call_with_json_out, + openai_call_with_tool, + pydantic_to_openai_tool, + schemadict_to_openai_tool, +) def get_token_count(text: str) -> int: - """ - Get the token count of a text string. - """ + """Return token count using the configured default GPT tokenizer.""" model_name = os.getenv("DEFAULT_LLM_MODEL_NAME", "gpt-5-mini") if not model_name.startswith("gpt"): model_name = "gpt-5-mini" encoding = encoding_for_model(model_name) - return len(encoding.encode(text)) \ No newline at end of file + return len(encoding.encode(text)) + + +__all__ = [ + "ollama_call", + "openai_call_with_json_out", + "openai_call_with_tool", + "pydantic_to_openai_tool", + "schemadict_to_openai_tool", + "get_token_count", +] \ No newline at end of file diff --git a/src/kgpipe_llm/common/apis/__init__.py b/src/kgpipe_llm/common/apis/__init__.py new file mode 100644 index 0000000..6504700 --- /dev/null +++ b/src/kgpipe_llm/common/apis/__init__.py @@ -0,0 +1,20 @@ +"""API-specific completion backends.""" + +from .ollama_comp import ollama_call +from .openai_comp import ( + openai_call_with_json_out, + openai_call_with_tool, + pydantic_to_openai_tool, + schemadict_to_openai_tool, +) +from .openwebui_comp import openwebui_call_with_json_out, openwebui_call_with_tool + +__all__ = [ + "ollama_call", + "openai_call_with_json_out", + "openai_call_with_tool", + "openwebui_call_with_json_out", + "openwebui_call_with_tool", + "pydantic_to_openai_tool", + "schemadict_to_openai_tool", +] diff --git a/src/kgpipe_llm/common/apis/ollama_comp.py b/src/kgpipe_llm/common/apis/ollama_comp.py index e69de29..e4733ee 100644 --- a/src/kgpipe_llm/common/apis/ollama_comp.py +++ b/src/kgpipe_llm/common/apis/ollama_comp.py @@ -0,0 +1,67 @@ +"""Ollama-compatible completion helper.""" + +from __future__ import annotations + +import json +from typing import Any, Dict, Type + +import requests +from pydantic import BaseModel + +TIMEOUT = 300 + + +def ollama_call( + *, + endpoint_url: str, + api_key: str, + schema_class: Type[BaseModel] | Dict[str, Any] | str, + model_name: str, + user_content: str, + system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", + seed: str = "", +) -> Dict[str, Any]: + payload: Dict[str, Any] = { + "model": model_name, + "prompt": user_content, + "stream": False, + } + + if isinstance(schema_class, str): + pass + elif isinstance(schema_class, dict): + payload["format"] = schema_class + else: + payload["format"] = schema_class.model_json_schema() + + if system_prompt: + payload["system"] = system_prompt + if seed: + payload["seed"] = seed + + headers = {"Content-Type": "application/json"} + if api_key: + headers["Authorization"] = f"Bearer {api_key}" + headers["X-API-Key"] = api_key + + try: + response = requests.post( + endpoint_url, + headers=headers, + json=payload, + timeout=TIMEOUT, + ) + if response.status_code != 200: + print(f"Request failed: {response.status_code} - {response.text}") + return {} + + raw_output = response.json()["response"] + if isinstance(schema_class, str): + return raw_output + parsed_output = json.loads(raw_output) + if not isinstance(schema_class, dict): + schema_class.model_validate(parsed_output) + return parsed_output + except Exception as exc: + print(f"Error processing LLM response: {exc}") + return {} diff --git a/src/kgpipe_llm/common/apis/openai_comp.py b/src/kgpipe_llm/common/apis/openai_comp.py index e69de29..d9bfb23 100644 --- a/src/kgpipe_llm/common/apis/openai_comp.py +++ b/src/kgpipe_llm/common/apis/openai_comp.py @@ -0,0 +1,203 @@ +"""OpenAI-compatible completion helpers with structured-output fallback.""" + +from __future__ import annotations + +import json +from typing import Any, Dict, Optional, Tuple, Type + +import requests +from pydantic import BaseModel + +TIMEOUT = 900 + + +def schemadict_to_openai_tool( + schema_dict: Dict[str, Any], + *, + name: str, + description: Optional[str] = None, + additional_properties: Optional[bool] = None, +) -> Dict[str, Any]: + """ + Convert a JSON schema dictionary into an OpenAI tool schema. + """ + params = dict(schema_dict) + params.pop("title", None) + if additional_properties is not None and params.get("type") == "object": + params["additionalProperties"] = additional_properties + return { + "type": "function", + "function": { + "name": name, + "description": (description or schema_dict.get("description") or f"{name} parameters").strip(), + "parameters": params, + }, + } + + +def pydantic_to_openai_tool( + model: Type[BaseModel], + *, + name: Optional[str] = None, + description: Optional[str] = None, + additional_properties: Optional[bool] = None, +) -> Dict[str, Any]: + """ + Convert a Pydantic model into an OpenAI tool schema. + """ + schema = model.model_json_schema() + resolved_name = name or model.__name__ + resolved_description = ( + description + or (model.__doc__ or "").strip() + or schema.get("description") + or f"{resolved_name} parameters" + ) + return schemadict_to_openai_tool( + schema, + name=resolved_name, + description=resolved_description, + additional_properties=additional_properties, + ) + + +def _build_headers(api_key: str) -> Dict[str, str]: + return { + "Content-Type": "application/json", + "Authorization": f"Bearer {api_key}", + } + + +def openai_call_with_json_out( + *, + endpoint_url: str, + api_key: str, + model_name: str, + user_content: str, + system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", + seed: str = "", + response_format: str = "json_object", +) -> Dict[str, Any]: + payload: Dict[str, Any] = { + "model": model_name, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_content}, + ], + "temperature": 1, + } + + if response_format: + payload["response_format"] = {"type": "json_object"} + if seed: + payload["seed"] = seed + + response = requests.post( + endpoint_url, + headers=_build_headers(api_key), + json=payload, + timeout=TIMEOUT, + ) + if response.status_code != 200: + print(response.content.decode("utf-8")) + response.raise_for_status() + content = response.json()["choices"][0]["message"]["content"] + try: + return json.loads(content) + except Exception as exc: + print(f"Error parsing JSON: {exc}") + return content + + +def _validate_or_passthrough( + args: Dict[str, Any], pyd_model: Type[BaseModel] | Dict[str, Any] +) -> BaseModel | Dict[str, Any]: + if isinstance(pyd_model, dict): + return args + return pyd_model.model_validate(args) + + +def _fallback_structured_output( + *, + endpoint_url: str, + api_key: str, + model_name: str, + user_content: str, + system_prompt: str, + seed: str, + pyd_model: Type[BaseModel] | Dict[str, Any], +) -> Tuple[Dict[str, Any], BaseModel | Dict[str, Any]]: + fallback_args = openai_call_with_json_out( + endpoint_url=endpoint_url, + api_key=api_key, + model_name=model_name, + user_content=user_content, + system_prompt=system_prompt, + seed=seed, + response_format="json_object", + ) + if not isinstance(fallback_args, dict): + raise ValueError("Fallback response_format=json_object did not return JSON object.") + return fallback_args, _validate_or_passthrough(fallback_args, pyd_model) + + +def openai_call_with_tool( + *, + endpoint_url: str, + api_key: str, + model_name: str, + user_content: str, + pyd_model: Type[BaseModel] | Dict[str, Any], + system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", + seed: str = "", +) -> Tuple[Dict[str, Any], BaseModel | Dict[str, Any]]: + if isinstance(pyd_model, dict): + tool = schemadict_to_openai_tool( + pyd_model, + name="CustomJsonSchema", + additional_properties=True, + ) + else: + tool = pydantic_to_openai_tool(pyd_model, additional_properties=False) + + payload: Dict[str, Any] = { + "model": model_name, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_content}, + ], + "tools": [tool], + "tool_choice": {"type": "function", "function": {"name": tool["function"]["name"]}}, + "temperature": 1, + } + if seed: + payload["seed"] = seed + + response = requests.post( + endpoint_url, + headers=_build_headers(api_key), + json=payload, + timeout=TIMEOUT, + ) + if response.status_code != 200: + print(response.content.decode("utf-8")) + response.raise_for_status() + data = response.json() + + try: + tool_calls = data["choices"][0]["message"].get("tool_calls", []) + if not tool_calls: + raise ValueError("No tool calls in response.") + args = json.loads(tool_calls[0]["function"]["arguments"]) + return args, _validate_or_passthrough(args, pyd_model) + except Exception as exc: + print(f"Tool-call structured parsing failed, using JSON fallback: {exc}") + return _fallback_structured_output( + endpoint_url=endpoint_url, + api_key=api_key, + model_name=model_name, + user_content=user_content, + system_prompt=system_prompt, + seed=seed, + pyd_model=pyd_model, + ) diff --git a/src/kgpipe_llm/common/apis/openwebui_comp.py b/src/kgpipe_llm/common/apis/openwebui_comp.py index e69de29..874948e 100644 --- a/src/kgpipe_llm/common/apis/openwebui_comp.py +++ b/src/kgpipe_llm/common/apis/openwebui_comp.py @@ -0,0 +1,45 @@ +"""OpenWebUI completion helpers. + +OpenWebUI often exposes OpenAI-compatible endpoints, so these wrappers delegate +to the OpenAI-compatible implementation while keeping a dedicated module. +""" + +from __future__ import annotations + +from typing import Any, Dict, Tuple, Type + +from pydantic import BaseModel + +from .openai_comp import ( + openai_call_with_json_out, + openai_call_with_tool, + pydantic_to_openai_tool, + schemadict_to_openai_tool, +) + + +def openwebui_call_with_json_out(**kwargs: Any) -> Dict[str, Any]: + """OpenWebUI wrapper for JSON-mode completions.""" + return openai_call_with_json_out(**kwargs) + + +def openwebui_call_with_tool( + *, + endpoint_url: str, + api_key: str, + model_name: str, + user_content: str, + pyd_model: Type[BaseModel] | Dict[str, Any], + system_prompt: str = "You are a careful JSON-LD KG engineering assistant.", + seed: str = "", +) -> Tuple[Dict[str, Any], BaseModel | Dict[str, Any]]: + """OpenWebUI wrapper for tool-calling structured output.""" + return openai_call_with_tool( + endpoint_url=endpoint_url, + api_key=api_key, + model_name=model_name, + user_content=user_content, + pyd_model=pyd_model, + system_prompt=system_prompt, + seed=seed, + ) diff --git a/src/kgpipe_llm/common/core.py b/src/kgpipe_llm/common/core.py index 90c8990..7c3af9f 100644 --- a/src/kgpipe_llm/common/core.py +++ b/src/kgpipe_llm/common/core.py @@ -2,146 +2,127 @@ Core LLM client and base task functionality for data integration tasks. """ -import requests -import json -from typing import Optional, TypeVar, Generic, Dict -from pydantic import BaseModel -from typing import AnyStr import os -from .api_utils import openai_call_with_tool, openai_call_with_json_out, ollama_call, get_token_count - -# Type variable for Pydantic models -T = TypeVar('T', bound=BaseModel) +from typing import Any, Dict, Generic, Optional, TypeVar, cast from dotenv import load_dotenv +from pydantic import BaseModel + +from .api_utils import get_token_count +from .apis.ollama_comp import ollama_call +from .apis.openai_comp import openai_call_with_json_out, openai_call_with_tool +from .apis.openwebui_comp import openwebui_call_with_json_out, openwebui_call_with_tool + load_dotenv() OPENAI_V1_COMPLETIONS_URL = "https://api.openai.com/v1/chat/completions" GPT_MODELS_EXTRA = ["o4-mini", "o1-mini", "o1-preview"] +OPENAI_LIKE_TYPES = {"openai", "openwebui"} + +T = TypeVar("T", bound=BaseModel) + + +def _infer_api_type(model_name: str, endpoint_url: str) -> str: + endpoint = (endpoint_url or "").lower() + if "localhost:11434" in endpoint or endpoint.endswith("/api/generate"): + return "ollama" + if "openwebui" in endpoint: + return "openwebui" + if model_name.startswith("gpt") or model_name in GPT_MODELS_EXTRA: + return "openai" + if endpoint.endswith("/v1/chat/completions"): + return "openai" + if endpoint.endswith("/api/chat/completions"): + return "openwebui" + return "ollama" + + +def _resolve_openai_like_endpoint(endpoint_url: str) -> str: + if endpoint_url and "chat/completions" in endpoint_url: + return endpoint_url + return OPENAI_V1_COMPLETIONS_URL + class LLMClient: - """Client for interacting with Ollama LLM API with structured output validation.""" - - def __init__(self, endpoint_url: str = "http://localhost:11434/api/generate", - model_name: str = "gemma3:27B", - token: str = "", - seed: str = ""): + """Client for interacting with LLM APIs with structured output validation.""" + + def __init__( + self, + endpoint_url: str = "http://localhost:11434/api/generate", + model_name: str = "gemma3:27B", + token: str = "", + seed: str = "", + api_type: Optional[str] = None, + ): self.endpoint_url = endpoint_url self.model_name = model_name self.token = token self.seed = seed - if model_name.startswith("gpt") or model_name in GPT_MODELS_EXTRA: - self.api_type = "openai" - else: - self.api_type = "ollama" - - - # def send_message(self, messages: list[dict], schema_class: type[T] | str, system_prompt: str = "") -> Optional[T] | str: - # """ - # Send a message to the LLM and validate the response against a Pydantic schema. - # """ - # payload = { - # "model": self.model_name, - # "messages": messages, - # "stream": False, - # } - - # if system_prompt and system_prompt != "": - # payload["system"] = system_prompt - - # if isinstance(schema_class, str): - # # For raw output, don't set format - # pass - # else: - # # For structured output, set the JSON schema format - # payload["format"] = schema_class.model_json_schema() - - # headers = { - # "Content-Type": "application/json", - # } - - # if self.token and self.token != "": - # headers["Authorization"] = f"Bearer {self.token}" - # headers["X-API-Key"] = self.token - - # try: - # response = requests.post( - # self.endpoint_url, - # headers=headers, - # json=payload, - # timeout=30 - # ) - # print(response.json()) - - # if response.status_code == 200: - # raw_output = response.json()["response"] - # if isinstance(schema_class, str): - # return raw_output - # else: - # parsed_output = json.loads(raw_output) - # result = schema_class.model_validate(parsed_output) - # return result - # else: - # print(f"Request failed: {response.status_code} - {response.text}") - # return None - - # except Exception as e: - # print(f"Error processing LLM response: {e}") - # return None - - - def send_prompt(self, prompt: str, schema_class: type[T] | str | Dict, system_prompt: str = "") -> Dict: + self.api_type = api_type or _infer_api_type(model_name, endpoint_url) + + def send_prompt( + self, + prompt: str, + schema_class: type[T] | str | Dict[str, Any], + system_prompt: str = "", + ) -> Any: """ Send a prompt to the LLM and validate the response against a Pydantic schema. - - Args: - prompt: The text prompt to send to the LLM - schema_class: The Pydantic model class to validate the response against, or str for raw output - - Returns: - Validated Pydantic model instance, raw string, or None if validation fails """ - print("INPUT_TOKEN_COUNT", get_token_count(prompt)) - if self.api_type == "openai": + if self.api_type in OPENAI_LIKE_TYPES: + endpoint = _resolve_openai_like_endpoint(self.endpoint_url) + json_call = ( + openwebui_call_with_json_out if self.api_type == "openwebui" else openai_call_with_json_out + ) + tool_call = openwebui_call_with_tool if self.api_type == "openwebui" else openai_call_with_tool + if isinstance(schema_class, str): - print(f"INFO: openai_call_with_json_out {type(schema_class)}") - return openai_call_with_json_out( - endpoint_url=OPENAI_V1_COMPLETIONS_URL, + print(f"INFO: {self.api_type}_call_with_json_out {type(schema_class)}") + return json_call( + endpoint_url=endpoint, api_key=self.token, model_name=self.model_name, user_content=prompt, system_prompt=system_prompt, seed=self.seed, - response_format=schema_class + response_format=schema_class, ) - else: - # special return type for openai - print(f"INFO: openai_call_with_tool {type(schema_class)}") - dict_val, model_val = openai_call_with_tool( - endpoint_url=OPENAI_V1_COMPLETIONS_URL, - api_key=self.token, - model_name=self.model_name, - user_content=prompt, - pyd_model=schema_class, - system_prompt=system_prompt, - seed=self.seed - ) - return dict_val - else: - print(f"INFO: ollama_call with {type(schema_class)}") - return ollama_call( - endpoint_url=self.endpoint_url, + print(f"INFO: {self.api_type}_call_with_tool {type(schema_class)}") + dict_val, model_val = tool_call( + endpoint_url=endpoint, api_key=self.token, model_name=self.model_name, user_content=prompt, - schema_class=schema_class, + pyd_model=schema_class, system_prompt=system_prompt, - seed=self.seed + seed=self.seed, ) + # Prefer returning the validated Pydantic instance when possible. + if isinstance(schema_class, type) and issubclass(schema_class, BaseModel): + if isinstance(model_val, BaseModel): + return model_val + if isinstance(dict_val, dict): + return cast(type[T], schema_class).model_validate(dict_val) + return dict_val + + print(f"INFO: ollama_call with {type(schema_class)}") + raw_val = ollama_call( + endpoint_url=self.endpoint_url, + api_key=self.token, + model_name=self.model_name, + user_content=prompt, + schema_class=schema_class, + system_prompt=system_prompt, + seed=self.seed, + ) + # Ollama path currently returns raw JSON; upgrade to a validated model when requested. + if isinstance(schema_class, type) and issubclass(schema_class, BaseModel) and isinstance(raw_val, dict): + return cast(type[T], schema_class).model_validate(raw_val) + return raw_val class BaseTask(Generic[T]): @@ -157,6 +138,7 @@ def execute(self, *args, **kwargs) -> Optional[T]: class LlmAPIConfig(BaseModel): """Configuration for an LLM API.""" + endpoint_url: str model_name: str ollama_token: Optional[str] @@ -164,36 +146,35 @@ class LlmAPIConfig(BaseModel): seed: str context_window: int - # def __init__(self, endpoint_url: str, model_name: str, ollama_token: str, openai_token: str): - # self.endpoint_url = endpoint_url - # self.model_name = model_name - # self.ollama_token = ollama_token - # self.openai_token = openai_token - def get_config_from_env() -> LlmAPIConfig: """Get the configuration for an LLM API from the environment.""" + opt_llm_endpoint_url = os.getenv("LLM_ENDPOINT_URL") + opt_llm_model_name = os.getenv("DEFAULT_LLM_MODEL_NAME", "gemma3:27B") + opt_ollama_token = os.getenv("OLLAMA_TOKEN") + opt_openai_token = os.getenv("OPENAI_TOKEN") + llm_seed = os.getenv("LLM_SEED", "") + opt_context_window = int(os.getenv("CONTEXT_WINDOW", 16384)) + + print( + "INFO: get_config_from_env", + opt_llm_endpoint_url, + opt_llm_model_name, + opt_ollama_token, + opt_openai_token, + llm_seed, + opt_context_window, + ) - OPT_LLM_ENDPOINT_URL = os.getenv("LLM_ENDPOINT_URL") - OPT_LLM_MODEL_NAME = os.getenv("DEFAULT_LLM_MODEL_NAME") - OPT_OLLAMA_TOKEN = os.getenv("OLLAMA_TOKEN") - OPT_OPENAI_TOKEN = os.getenv("OPENAI_TOKEN") - LLM_SEED = os.getenv("LLM_SEED", "") - OPT_CONTEXT_WINDOW = int(os.getenv("CONTEXT_WINDOW", 16384)) - - print(f"INFO: get_config_from_env {OPT_LLM_ENDPOINT_URL} {OPT_LLM_MODEL_NAME} {OPT_OLLAMA_TOKEN} {OPT_OPENAI_TOKEN} {LLM_SEED} {OPT_CONTEXT_WINDOW}") - - # TODO requires one token to be set - if OPT_LLM_ENDPOINT_URL is None or (OPT_LLM_MODEL_NAME is None and OPT_OLLAMA_TOKEN is None and OPT_OPENAI_TOKEN is None): - # raise ValueError("LLM_ENDPOINT_URL, LLM_MODEL_NAME, OLLAMA_TOKEN, and OPENAI_TOKEN must be set. Also, CONTEXT_WINDOW must be set.") - raise ValueError("LLM_ENDPOINT_URL, LLM_MODEL_NAME, OLLAMA_TOKEN, and OPENAI_TOKEN must be set. Also, CONTEXT_WINDOW must be set.") + if opt_llm_endpoint_url is None: + raise ValueError("LLM_ENDPOINT_URL must be set.") return LlmAPIConfig( - endpoint_url=OPT_LLM_ENDPOINT_URL, - model_name=OPT_LLM_MODEL_NAME, - ollama_token=OPT_OLLAMA_TOKEN, - openai_token=OPT_OPENAI_TOKEN, - seed=LLM_SEED, - context_window=OPT_CONTEXT_WINDOW, + endpoint_url=opt_llm_endpoint_url, + model_name=opt_llm_model_name, + ollama_token=opt_ollama_token, + openai_token=opt_openai_token, + seed=llm_seed, + context_window=opt_context_window, ) @@ -201,14 +182,17 @@ def get_client_from_env() -> LLMClient: """Get the client for an LLM API from the environment.""" config = get_config_from_env() print(f"INFO: get_client_from_env {config.model_name}") - if config.model_name.startswith("gpt") or config.model_name in GPT_MODELS_EXTRA: - api_type = "openai" - else: - api_type = "ollama" + api_type = _infer_api_type(config.model_name, config.endpoint_url) + token = config.ollama_token if api_type == "ollama" else config.openai_token print(f"INFO: get_client_from_env with {api_type}") return LLMClient( endpoint_url=config.endpoint_url, model_name=config.model_name, - token=config.ollama_token if api_type == "ollama" else config.openai_token, - seed=config.seed - ) \ No newline at end of file + token=token or "", + seed=config.seed, + api_type=api_type, + ) + + +# Backward compatibility for modules importing a shared default client. +default_client = LLMClient() diff --git a/src/kgpipe_llm/test/test_any_extraction.py b/src/kgpipe_llm/test/test_any_extraction.py new file mode 100644 index 0000000..ab7ded6 --- /dev/null +++ b/src/kgpipe_llm/test/test_any_extraction.py @@ -0,0 +1,24 @@ +from kgpipe_llm.any_extraction import extract_ontology_surface_triples +from pathlib import Path +from kgpipe_llm.common.core import LLMClient +import os + +TEXT=""" +Titanic is a 1997 American epic historical romance film written and directed by James Cameron. Incorporating both historical and fictional aspects, it is based on accounts of the sinking of RMS Titanic in 1912. Leonardo DiCaprio and Kate Winslet star as members of different social classes who fall in love during the ship's ill-fated maiden voyage. The ensemble cast includes Billy Zane, Kathy Bates, Frances Fisher, Bernard Hill, Jonathan Hyde, Danny Nucci, David Warner and Bill Paxton. Cameron's inspiration came from his fascination with shipwrecks. He felt a love story interspersed with human loss would be essential to convey the emotional impact of the disaster. Production began on September 1, 1995, when Cameron shot footage of the Titanic wreck. The modern scenes were shot on board the Shirshov Institute of Oceanology research vessel Akademik Mstislav Keldysh, which Cameron had used as a base when filming the wreck. Scale models, computer-generated imagery (CGI), and a reconstruction of the Titanic built at Baja Studios were used to recreate the sinking. Titanic was initially in development at 20th Century Fox, but delays and a mounting budget resulted in Fox partnering with Paramount Pictures for financial help. It was the most expensive film ever made at the time, with a production budget of $200 million. Filming took place from July 1996 to March 1997. Titanic premiered at the Tokyo International Film Festival on November 1, 1997, and was released in the United States on December 19. It was distributed by Paramount Pictures in the United States and Canada and by 20th Century Fox in other territories. It was praised for its visual effects, performances (particularly those of DiCaprio, Winslet, and Gloria Stuart), production values, direction, score, cinematography, story, and emotional depth. Among other awards, the film received fourteen nominations at the 70th Academy Awards and won eleven, including Best Picture and Best Director. In doing so, it tied both All About Eve (1950) for the record for the most Academy Award nominations, and Ben-Hur (1959) for the most Academy Awards won by a film, making Titanic the most successful individual film in Academy Award history (these records would be matched by 2016's La La Land and 2003's The Lord of the Rings: The Return of the King respectively, although the nomination record was surpassed by 2025's Sinners in 2026). With an initial worldwide gross of over $1.84 billion, Titanic was the first film to reach the billion-dollar mark (1993's Jurassic Park would later become the earliest-released film to achieve this feat, via subsequent re-releases), and was the highest-grossing film of all time until Cameron's next film, Avatar (2009), surpassed it in 2010. Income from the initial theatrical release, retail video, and soundtrack sales and US broadcast rights exceeded $3.2 billion. Releases pushed the worldwide theatrical total to $2.264 billion, making Titanic the second film to gross more than $2 billion worldwide after Avatar; as of 2023, it is the fourth-highest-grossing film. In 2017, the Library of Congress selected it for preservation in the United States National Film Registry as "culturally, historically, or aesthetically significant +""" + +API_KEY = os.getenv("OPENAI_API_KEY") +if not API_KEY: + raise ValueError("OPENAI_API_KEY is not set") + +model_name="o4-mini" +ontology_path = Path("/home/marvin/phd/data/moviekg/datasets/film_10k/ontology.ttl") + +def test_extract_ontology_surface_triples(): + client = LLMClient( + model_name=model_name, + token=API_KEY, + api_type="openai", + ) + result = extract_ontology_surface_triples(TEXT, ontology_path, client) + print(result.model_dump_json(indent=2)) \ No newline at end of file diff --git a/src/kgpipe_parameters/README.md b/src/kgpipe_parameters/README.md new file mode 100644 index 0000000..8619136 --- /dev/null +++ b/src/kgpipe_parameters/README.md @@ -0,0 +1,38 @@ +# KGpipe Parameters + +Subpackage to analyze and optimize paramters for data integration tasks + +1. Extract/Find configuration Parameters for a Task T and its implementations I +2. Match and cluster configuration parameters +3. Find best configuration parameters + + +## TODOs + +- [ ] Adding parameters to KgTask.run(file_input,file_output,parameters) +- [ ] Store extraction results in a structured way: provenance, assignment, descriptions +- [ ] Cluster paramters: same task (triple extract, cleaning, entity resolution) + +## Parameter Mining + +Methods to find parameter or settings for codeing libraries, CLI, or remote APIs(Http) + +Inputs +- api documentation +- Readmes +- command help output +- code files + +Methods +- regex +- llm + +## Clustering +... + +## Description +... + +## Optimization +... + diff --git a/src/kgpipe_parameters/__init__.py b/src/kgpipe_parameters/__init__.py new file mode 100644 index 0000000..1698efa --- /dev/null +++ b/src/kgpipe_parameters/__init__.py @@ -0,0 +1,46 @@ +""" +KGpipe Parameters subpackage for analyzing and optimizing parameters for data integration tasks. + +This package provides functionality to: +1. Extract/Find configuration Parameters for a Task T and its implementations I +2. Match and cluster configuration parameters +3. Find best configuration parameters +""" + +from .extraction import ( + ParameterMiner, + RawParameter, + ExtractionResult, + SourceType, + ExtractionMethod, + ReadmeDocExtractor, + LLMReadmeDocExtractor, +) + +from .clustering import ( + ParameterClusterer, + ParameterVector, + ParameterCluster, + ClusteringResult, +) + +from .visualization import ParameterVisualizer + +__all__ = [ + # Extraction + "ParameterMiner", + "RawParameter", + "ExtractionResult", + "SourceType", + "ExtractionMethod", + "ReadmeDocExtractor", + "LLMReadmeDocExtractor", + # Clustering + "ParameterClusterer", + "ParameterVector", + "ParameterCluster", + "ClusteringResult", + # Visualization + "ParameterVisualizer", +] + diff --git a/src/kgpipe_parameters/clustering/__init__.py b/src/kgpipe_parameters/clustering/__init__.py new file mode 100644 index 0000000..7698ade --- /dev/null +++ b/src/kgpipe_parameters/clustering/__init__.py @@ -0,0 +1,20 @@ +""" +Parameter clustering module. + +Groups similar parameters across tools using sentence-transformer embeddings +and agglomerative clustering so that common configuration knobs are surfaced. +""" + +from .models import ParameterVector, ParameterCluster, ClusteringResult +from .similarity import embed_parameters, cosine_similarity_matrix +from .clusterer import ParameterClusterer + +__all__ = [ + "ParameterVector", + "ParameterCluster", + "ClusteringResult", + "embed_parameters", + "cosine_similarity_matrix", + "ParameterClusterer", +] + diff --git a/src/kgpipe_parameters/clustering/clusterer.py b/src/kgpipe_parameters/clustering/clusterer.py new file mode 100644 index 0000000..bd7d62c --- /dev/null +++ b/src/kgpipe_parameters/clustering/clusterer.py @@ -0,0 +1,242 @@ +""" +Main clustering logic. + +Loads extracted parameters from experiment JSON output, embeds them with +sentence-transformers, and applies agglomerative clustering to surface +groups of similar configuration knobs across tools. +""" + +from __future__ import annotations + +import json +import logging +from collections import Counter +from pathlib import Path +from typing import Any, Dict, List, Optional + +import numpy as np + +from .models import ParameterVector, ParameterCluster, ClusteringResult +from .similarity import DEFAULT_MODEL_NAME, embed_parameters + +logger = logging.getLogger(__name__) + + +class ParameterClusterer: + """ + Cluster extracted parameters by semantic similarity. + + Typical usage:: + + clusterer = ParameterClusterer() + result = clusterer.cluster_from_output_dir(Path("output/")) + for c in result.cross_tool_clusters(): + print(c.label, c.tools, c.size()) + """ + + def __init__( + self, + model_name: str = DEFAULT_MODEL_NAME, + distance_threshold: float = 0.55, + min_cluster_size: int = 1, + ): + """ + Parameters + ---------- + model_name : str + Sentence-transformer model to use for embeddings. + distance_threshold : float + Maximum cosine *distance* (1 − similarity) at which two + parameters are still merged into the same cluster. + Lower → tighter clusters. ``0.55`` is a good starting + point for short technical phrases. + min_cluster_size : int + Drop clusters smaller than this after clustering. + """ + self.model_name = model_name + self.distance_threshold = distance_threshold + self.min_cluster_size = min_cluster_size + self._model = None # lazy-loaded + + # ------------------------------------------------------------------ + # Loading helpers + # ------------------------------------------------------------------ + + @staticmethod + def load_parameters_from_json(path: Path) -> List[ParameterVector]: + """ + Load parameters from one tool's JSON output file. + + Expected format: the JSON written by + ``ToolExtractionResult.to_dict()`` — a dict with a + ``"parameters"`` list and a ``"tool_name"`` string. + """ + with open(path) as f: + data = json.load(f) + + tool_name = data.get("tool_name", path.stem) + vectors: List[ParameterVector] = [] + + for p in data.get("parameters", []): + pv = ParameterVector( + name=p.get("name", ""), + tool_name=tool_name, + native_keys=p.get("native_keys", []), + description=p.get("description"), + type_hint=p.get("type_hint"), + default_value=p.get("default_value"), + required=p.get("required", False), + source_label=p.get("_source", ""), + ) + vectors.append(pv) + + return vectors + + def load_from_output_dir(self, output_dir: Path) -> List[ParameterVector]: + """ + Load parameters from *all* tool JSON files in *output_dir*. + + Skips files whose name starts with ``_`` (e.g. ``_summary.json``). + """ + all_params: List[ParameterVector] = [] + for json_file in sorted(output_dir.glob("*.json")): + if json_file.name.startswith("_"): + continue + try: + params = self.load_parameters_from_json(json_file) + logger.info( + "Loaded %d parameters from %s", len(params), json_file.name + ) + all_params.extend(params) + except Exception as e: + logger.warning("Failed to load %s: %s", json_file, e) + + logger.info("Total parameters loaded: %d", len(all_params)) + return all_params + + # ------------------------------------------------------------------ + # Clustering + # ------------------------------------------------------------------ + + def cluster(self, parameters: List[ParameterVector]) -> ClusteringResult: + """ + Embed and cluster a list of parameters. + + Returns a ``ClusteringResult`` with numbered clusters. + """ + if not parameters: + return ClusteringResult( + model_name=self.model_name, + distance_threshold=self.distance_threshold, + ) + + # 1. Compute embeddings + if self._model is None: + from sentence_transformers import SentenceTransformer + + self._model = SentenceTransformer(self.model_name) + + embeddings = embed_parameters( + parameters, model_name=self.model_name, model=self._model + ) + + # 2. Agglomerative clustering with cosine distance + n = len(parameters) + + if n == 1: + # AgglomerativeClustering requires ≥ 2 samples; short-circuit. + labels = np.array([0]) + else: + from sklearn.cluster import AgglomerativeClustering + + sim_matrix = embeddings @ embeddings.T + np.clip(sim_matrix, -1.0, 1.0, out=sim_matrix) + dist_matrix = 1.0 - sim_matrix + + clustering_model = AgglomerativeClustering( + n_clusters=None, + metric="precomputed", + linkage="average", + distance_threshold=self.distance_threshold, + ) + labels = clustering_model.fit_predict(dist_matrix) + + # 3. Build ParameterCluster objects + cluster_map: Dict[int, List[int]] = {} + for idx, label in enumerate(labels): + cluster_map.setdefault(int(label), []).append(idx) + + clusters: List[ParameterCluster] = [] + for cid, member_indices in sorted(cluster_map.items()): + members = [parameters[i] for i in member_indices] + if len(members) < self.min_cluster_size: + continue + + tools = sorted(set(m.tool_name for m in members)) + centroid = embeddings[member_indices].mean(axis=0) + + # Label = most common parameter name in the cluster + name_counts = Counter(m.name for m in members) + label_name = name_counts.most_common(1)[0][0] + + clusters.append( + ParameterCluster( + cluster_id=cid, + label=label_name, + members=members, + tools=tools, + centroid=centroid.tolist(), + ) + ) + + # Sort: cross-tool first, then by size descending + clusters.sort(key=lambda c: (-int(c.is_cross_tool()), -c.size())) + + return ClusteringResult( + n_parameters=len(parameters), + n_clusters=len(clusters), + distance_threshold=self.distance_threshold, + model_name=self.model_name, + clusters=clusters, + ) + + def cluster_from_output_dir(self, output_dir: Path) -> ClusteringResult: + """Convenience: load + cluster in one call.""" + params = self.load_from_output_dir(output_dir) + return self.cluster(params) + + # ------------------------------------------------------------------ + # Output helpers + # ------------------------------------------------------------------ + + @staticmethod + def save_result(result: ClusteringResult, path: Path) -> None: + """Save clustering result as JSON.""" + # Strip large embedding lists to keep the file readable + data = result.model_dump() + for cluster in data.get("clusters", []): + cluster.pop("centroid", None) + for member in cluster.get("members", []): + member.pop("embedding", None) + + with open(path, "w") as f: + json.dump(data, f, indent=2, default=str) + logger.info("Saved clustering result to %s", path) + + @staticmethod + def save_table(result: ClusteringResult, path: Path) -> None: + """Save a flat CSV parameter table from clustering results.""" + import csv + + rows = result.to_table_rows() + if not rows: + logger.warning("No rows to write to table") + return + + fieldnames = list(rows[0].keys()) + with open(path, "w", newline="") as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + writer.writeheader() + writer.writerows(rows) + logger.info("Saved parameter table (%d rows) to %s", len(rows), path) + diff --git a/src/kgpipe_parameters/clustering/models.py b/src/kgpipe_parameters/clustering/models.py new file mode 100644 index 0000000..5570d36 --- /dev/null +++ b/src/kgpipe_parameters/clustering/models.py @@ -0,0 +1,114 @@ +""" +Data models for parameter clustering results. +""" + +from typing import List, Optional, Dict, Any +from pydantic import BaseModel, ConfigDict, Field +import numpy as np + + +class ParameterVector(BaseModel): + """A parameter together with its embedding and origin metadata.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + name: str = Field(..., description="Normalized parameter name") + tool_name: str = Field(..., description="Tool this parameter belongs to") + native_keys: List[str] = Field(default_factory=list) + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Any] = None + required: bool = False + source_label: str = Field( + "", description="Human-readable source (e.g. 'cli', 'readme:README.md')" + ) + # Embedding stored as plain list for JSON serialisation; converted to + # numpy array for computation. + embedding: Optional[List[float]] = Field( + None, description="Sentence-transformer embedding" + ) + + # ------------------------------------------------------------------ + # Helpers + # ------------------------------------------------------------------ + def text_for_embedding(self) -> str: + """Build the text representation used for embedding computation.""" + parts = [self.name.replace("_", " ")] + if self.description: + parts.append(self.description) + if self.native_keys: + parts.append(" ".join(self.native_keys)) + if self.type_hint: + parts.append(f"type: {self.type_hint}") + return " | ".join(parts) + + +class ParameterCluster(BaseModel): + """A cluster of similar parameters found across one or more tools.""" + + cluster_id: int = Field(..., description="Numeric cluster identifier") + label: str = Field( + "", description="Human-readable label (e.g. most common parameter name)" + ) + members: List[ParameterVector] = Field(default_factory=list) + tools: List[str] = Field( + default_factory=list, + description="Distinct tool names represented in this cluster", + ) + centroid: Optional[List[float]] = Field( + None, description="Mean embedding of the cluster" + ) + + def size(self) -> int: + return len(self.members) + + def is_cross_tool(self) -> bool: + """Return True if parameters from more than one tool are in this cluster.""" + return len(self.tools) > 1 + + +class ClusteringResult(BaseModel): + """Container for an entire clustering run.""" + + n_parameters: int = Field(0, description="Total parameters fed to clustering") + n_clusters: int = Field(0, description="Number of clusters produced") + distance_threshold: float = Field( + 0.0, description="Distance threshold used for clustering" + ) + model_name: str = Field("", description="Sentence-transformer model used") + clusters: List[ParameterCluster] = Field(default_factory=list) + metadata: Dict[str, Any] = Field(default_factory=dict) + + # ------------------------------------------------------------------ + # Convenience helpers + # ------------------------------------------------------------------ + def cross_tool_clusters(self) -> List[ParameterCluster]: + """Return only clusters that span more than one tool.""" + return [c for c in self.clusters if c.is_cross_tool()] + + def to_table_rows(self) -> List[Dict[str, Any]]: + """ + Flatten clusters into a list of rows suitable for a pandas DataFrame + or CSV export. + """ + rows: List[Dict[str, Any]] = [] + for cluster in self.clusters: + for member in cluster.members: + rows.append( + { + "cluster_id": cluster.cluster_id, + "cluster_label": cluster.label, + "cluster_size": cluster.size(), + "cross_tool": cluster.is_cross_tool(), + "tool": member.tool_name, + "parameter": member.name, + "native_keys": ", ".join(member.native_keys), + "description": member.description or "", + "type_hint": member.type_hint or "", + "default_value": member.default_value, + "required": member.required, + "source": member.source_label, + } + ) + return rows + diff --git a/src/kgpipe_parameters/clustering/similarity.py b/src/kgpipe_parameters/clustering/similarity.py new file mode 100644 index 0000000..5c52e0c --- /dev/null +++ b/src/kgpipe_parameters/clustering/similarity.py @@ -0,0 +1,96 @@ +""" +Embedding computation and similarity helpers for parameter clustering. + +Uses sentence-transformers to encode parameter descriptions into dense +vectors, then provides numpy-based cosine-similarity utilities. +""" + +from __future__ import annotations + +import logging +from typing import List, Optional + +import numpy as np + +from .models import ParameterVector + +logger = logging.getLogger(__name__) + +# Default lightweight model; works well for short technical phrases. +DEFAULT_MODEL_NAME = "all-MiniLM-L6-v2" + + +def _load_model(model_name: str): + """Load a SentenceTransformer model (cached after first call).""" + from sentence_transformers import SentenceTransformer + + logger.info("Loading sentence-transformer model: %s", model_name) + return SentenceTransformer(model_name) + + +def embed_parameters( + parameters: List[ParameterVector], + model_name: str = DEFAULT_MODEL_NAME, + batch_size: int = 64, + model: Optional[object] = None, +) -> np.ndarray: + """ + Compute embeddings for a list of ParameterVectors. + + Each parameter's ``text_for_embedding()`` is encoded via the + sentence-transformer *model_name*. The resulting embeddings are + stored back into each ``ParameterVector.embedding`` field **and** + returned as a (N, D) numpy array. + + Parameters + ---------- + parameters : list[ParameterVector] + Parameters to embed. + model_name : str + HuggingFace model identifier. + batch_size : int + Encoding batch size. + model : optional + Pre-loaded SentenceTransformer instance (avoids reloading). + + Returns + ------- + np.ndarray + Shape ``(len(parameters), embedding_dim)``. + """ + if not parameters: + return np.empty((0, 0)) + + if model is None: + model = _load_model(model_name) + + texts = [p.text_for_embedding() for p in parameters] + embeddings = model.encode(texts, batch_size=batch_size, show_progress_bar=False) + embeddings = np.asarray(embeddings, dtype=np.float32) + + # Normalise to unit length so cosine similarity = dot product. + norms = np.linalg.norm(embeddings, axis=1, keepdims=True) + norms[norms == 0] = 1.0 + embeddings = embeddings / norms + + for pv, emb in zip(parameters, embeddings): + pv.embedding = emb.tolist() + + return embeddings + + +def cosine_similarity_matrix(embeddings: np.ndarray) -> np.ndarray: + """ + Compute the pair-wise cosine similarity matrix. + + If the embeddings are already L2-normalised (as ``embed_parameters`` + produces), this is simply ``embeddings @ embeddings.T``. + """ + if embeddings.size == 0: + return np.empty((0, 0)) + # Ensure unit vectors + norms = np.linalg.norm(embeddings, axis=1, keepdims=True) + norms[norms == 0] = 1.0 + normed = embeddings / norms + return normed @ normed.T + diff --git a/src/kgpipe_parameters/config_mapper.py b/src/kgpipe_parameters/config_mapper.py new file mode 100644 index 0000000..a9d854b --- /dev/null +++ b/src/kgpipe_parameters/config_mapper.py @@ -0,0 +1,20 @@ + +""" +Maps a GLOBAL configuration to a local Parameter of a task implementation. +""" + +from kgpipe.common.model.configuration import Parameter, ConfigurationProfile +from kgpipe.common.model.task import KgTask, Data, TaskInput, TaskOutput, KgTask + +class ConfigMapper: + def __init__(self, task: KgTask): + self.task = task + + def map_config(self, config: ConfigurationMapping): + return self.task.config + + + + +def example_task(i: TaskInput, o: TaskOutput, p: ConfigurationProfile): + pass \ No newline at end of file diff --git a/src/kgpipe_parameters/extraction/__init__.py b/src/kgpipe_parameters/extraction/__init__.py new file mode 100644 index 0000000..db5f283 --- /dev/null +++ b/src/kgpipe_parameters/extraction/__init__.py @@ -0,0 +1,75 @@ +""" +Parameter extraction module for mining configuration parameters from various sources. +""" + +from .param_miner import ParameterMiner +from .extractors import ( + CLIExtractor, + PythonLibExtractor, + HTTPAPIExtractor, + DockerExtractor, + ReadmeDocExtractor, + LLMCLIExtractor, + LLMPythonExtractor, + LLMHTTPExtractor, + LLMDockerExtractor, + LLMReadmeDocExtractor, +) +from .models import ( + RawParameter, + ExtractionResult, + SourceType, + ExtractionMethod, +) +from .base import ( + BaseExtractor, + RegexExtractor, + LLMExtractor, +) +from .utils import ( + to_parameter_model, + normalize_parameter_name, + parse_default_value, + infer_parameter_type, + extract_constraints, +) +from .chunk_filter import ( + score_chunk, + has_parameter_signals, + KEYWORD_SETS, +) + +__all__ = [ + # Main class + "ParameterMiner", + # Extractors + "CLIExtractor", + "PythonLibExtractor", + "HTTPAPIExtractor", + "DockerExtractor", + "ReadmeDocExtractor", + "LLMCLIExtractor", + "LLMPythonExtractor", + "LLMHTTPExtractor", + "LLMDockerExtractor", + "LLMReadmeDocExtractor", + # Base classes + "BaseExtractor", + "RegexExtractor", + "LLMExtractor", + # Models + "RawParameter", + "ExtractionResult", + "SourceType", + "ExtractionMethod", + # Utilities + "to_parameter_model", + "normalize_parameter_name", + "parse_default_value", + "infer_parameter_type", + "extract_constraints", + # Chunk filtering + "score_chunk", + "has_parameter_signals", + "KEYWORD_SETS", +] diff --git a/src/kgpipe_parameters/extraction/base.py b/src/kgpipe_parameters/extraction/base.py new file mode 100644 index 0000000..b9c492c --- /dev/null +++ b/src/kgpipe_parameters/extraction/base.py @@ -0,0 +1,85 @@ +""" +Base classes for parameter extractors. +""" + +from abc import ABC, abstractmethod +from typing import List, Optional +from .models import RawParameter, ExtractionResult, SourceType, ExtractionMethod + + +class BaseExtractor(ABC): + """Abstract base class for all parameter extractors.""" + + def __init__(self, source_type: SourceType): + self.source_type = source_type + + @abstractmethod + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """ + Extract parameters from the given source. + + Args: + source: Source content (text, file path, etc.) + tool_name: Optional name of the tool being analyzed + + Returns: + ExtractionResult containing extracted parameters + """ + pass + + +class RegexExtractor(BaseExtractor): + """Base class for regex-based parameter extraction.""" + + def __init__(self, source_type: SourceType, patterns: Optional[dict] = None): + super().__init__(source_type) + self.patterns = patterns or {} + + @abstractmethod + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using regex patterns.""" + pass + + def _apply_patterns(self, text: str) -> List[RawParameter]: + """ + Apply regex patterns to extract parameters. + Subclasses should override this with their specific pattern matching logic. + """ + return [] + + +class LLMExtractor(BaseExtractor): + """Base class for LLM-based parameter extraction.""" + + def __init__(self, source_type: SourceType, llm_client=None): + super().__init__(source_type) + self.llm_client = llm_client + if llm_client is None: + try: + from kgpipe_llm.common.core import LLMClient, get_client_from_env + self.llm_client = get_client_from_env() + except ImportError: + raise ImportError( + "LLM extraction requires kgpipe_llm. " + "Install it or provide an LLMClient instance." + ) + + @abstractmethod + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + pass + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + """ + Create a prompt for LLM extraction. + Subclasses should override this with their specific prompt template. + """ + return f"Extract configuration parameters from:\n\n{source}" + + def _parse_llm_response(self, response: dict) -> List[RawParameter]: + """ + Parse LLM response into RawParameter objects. + Subclasses should override this with their specific parsing logic. + """ + return [] + diff --git a/src/kgpipe_parameters/extraction/chunk_filter.py b/src/kgpipe_parameters/extraction/chunk_filter.py new file mode 100644 index 0000000..fc8feac --- /dev/null +++ b/src/kgpipe_parameters/extraction/chunk_filter.py @@ -0,0 +1,274 @@ +""" +Keyword-based chunk scoring for pre-filtering files before extraction. + +Counts parameter-signal keywords in a text chunk and returns a relevance +score. Files/chunks that score below a configurable threshold are skipped +entirely, preventing noise (e.g. Arabic segmenter scripts in CoreNLP) from +polluting both the regex and LLM extraction paths. + +No embeddings, no extra dependencies — pure keyword counting. +""" + +import re +from typing import Dict, List, Optional, Tuple + +__all__ = ["score_chunk", "has_parameter_signals", "KEYWORD_SETS"] + + +# ── Keyword sets per language / file-type ──────────────────────────────── + +_PYTHON_KEYWORDS: List[str] = [ + # argparse / click / typer + "argparse", + "add_argument", + "ArgumentParser", + "click.option", + "click.argument", + "click.command", + "typer.Option", + "typer.Argument", + # dataclass / pydantic + "@dataclass", + "Field(", + "BaseModel", + "BaseSettings", + # general config signals + "default=", + "default_factory", + "required=", + "choices=", + "type=", + "nargs=", + "help=", + "metavar=", + # plain constructor parameters (frameworks like valentine, etc.) + "def __init__(self,", + "self.__", + "self._", + # env vars + "os.environ", + "os.getenv", + "environ.get", + # configparser / yaml / json config + "configparser", + "ConfigParser", + "config.get", + "config[", + "yaml.load", + "yaml.safe_load", + "json.load", + # hydra / omegaconf + "@hydra.main", + "OmegaConf", + "DictConfig", +] + +_JAVA_KEYWORDS: List[str] = [ + # JCommander / picocli / commons-cli + "@Option", + "@Parameter", + "@CommandLine", + "@Command", + ".addOption(", + "Options(", + "new Option(", + "OptionBuilder", + "CommandLine", + # Java properties / config + "getProperty(", + "setProperty(", + "properties.get(", + "Properties", + ".properties", + "loadProperties", + "getConfig(", + "getString(", + "getInt(", + "getDouble(", + "getBoolean(", + # Spring + "@Value(", + "@ConfigurationProperties", + "@RequestParam", + "@PathVariable", + # general + "default:", + "DEFAULT_", + "CONFIG_", + "PARAM_", +] + +_PROPERTIES_KEYWORDS: List[str] = [ + # .properties files are inherently config + "=", + ":", +] + +_XML_KEYWORDS: List[str] = [ + " str: + """Guess the language/type from a file extension.""" + if not file_path: + return "generic" + # Handle Dockerfile* specially + lower = file_path.lower() + if "dockerfile" in lower or "docker-compose" in lower: + return "docker" + # Extension-based lookup + for ext, lang in _EXT_TO_LANG.items(): + if lower.endswith(ext): + return lang + return "generic" + + +def score_chunk( + text: str, + file_path: Optional[str] = None, + language: Optional[str] = None, +) -> Tuple[int, List[str]]: + """ + Score a text chunk by counting parameter-signal keyword hits. + + Args: + text: The text content to score. + file_path: Optional file path (used to auto-detect language). + language: Explicit language override (python, java, …). + If None, detected from *file_path*. + + Returns: + (score, matched_keywords) — score is the number of distinct keyword + matches found; matched_keywords lists which ones fired. + """ + if not text: + return 0, [] + + lang = language or _detect_language(file_path) + keywords = KEYWORD_SETS.get(lang, KEYWORD_SETS["generic"]) + + matched: List[str] = [] + for kw in keywords: + if kw in text: + matched.append(kw) + + return len(matched), matched + + +def has_parameter_signals( + text: str, + file_path: Optional[str] = None, + language: Optional[str] = None, + threshold: int = 2, +) -> bool: + """ + Return True if *text* contains at least *threshold* distinct + parameter-signal keywords. + + For .properties and .xml files the threshold is automatically lowered + to 1 because their content is inherently config-like. + + Args: + text: The text content to check. + file_path: Optional file path for language detection. + language: Explicit language override. + threshold: Minimum keyword hits required (default 2). + + Returns: + True if the chunk passes the keyword filter. + """ + lang = language or _detect_language(file_path) + + # .properties / .xml files are inherently config — lower bar. + # Java files with *any* annotation-style signal are worth inspecting. + if lang in ("properties", "xml", "java"): + threshold = min(threshold, 1) + + score, _ = score_chunk(text, file_path=file_path, language=lang) + return score >= threshold + diff --git a/src/kgpipe_parameters/extraction/extractors/__init__.py b/src/kgpipe_parameters/extraction/extractors/__init__.py new file mode 100644 index 0000000..c506246 --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/__init__.py @@ -0,0 +1,29 @@ +""" +Extractor implementations for different source types. +""" + +from .cli import CLIExtractor, LLMCLIExtractor +from .python_lib import PythonLibExtractor, LLMPythonExtractor +from .http_api import HTTPAPIExtractor, LLMHTTPExtractor +from .docker import DockerExtractor, LLMDockerExtractor +from .readme_doc import ReadmeDocExtractor, LLMReadmeDocExtractor + +__all__ = [ + # CLI + "CLIExtractor", + "LLMCLIExtractor", + # Python + "PythonLibExtractor", + "LLMPythonExtractor", + # HTTP API + "HTTPAPIExtractor", + "LLMHTTPExtractor", + # Docker + "DockerExtractor", + "LLMDockerExtractor", + # README / documentation + "ReadmeDocExtractor", + "LLMReadmeDocExtractor", +] + + diff --git a/src/kgpipe_parameters/extraction/extractors/cli.py b/src/kgpipe_parameters/extraction/extractors/cli.py new file mode 100644 index 0000000..82ee8ba --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/cli.py @@ -0,0 +1,193 @@ +""" +CLI parameter extraction from help output. +""" + +import re +from typing import Optional, Union + +from ..models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from ..base import RegexExtractor, LLMExtractor +from ..patterns import CLI_PATTERNS +from ..utils import normalize_parameter_name, parse_default_value + + +class CLIExtractor(RegexExtractor): + """Extract parameters from CLI help output.""" + + def __init__(self, use_llm: bool = False, llm_client=None): + super().__init__(SourceType.CLI, CLI_PATTERNS) + self.use_llm = use_llm + if use_llm: + self.llm_extractor = LLMCLIExtractor(llm_client) + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters from CLI help text.""" + if self.use_llm: + return self.llm_extractor.extract(source, tool_name) + + parameters = [] + errors = [] + + try: + lines = source.split('\n') + current_param = None + + for line in lines: + # Skip usage lines (they contain brackets and are not actual parameter descriptions) + if line.strip().startswith("usage:") or (line.strip().startswith("[") and "]" in line and "optional" not in line.lower() and "arguments" not in line.lower()): + continue + + # Match long flags: --param or --param=VALUE + long_match = CLI_PATTERNS["long_flag"].search(line) + if long_match: + param_name = long_match.group(1) + # Don't use group(2) from usage line - it's the placeholder, not default + default_val = None + + normalized = normalize_parameter_name(param_name) + native_keys = [f"--{param_name}"] + + # Check for short form on same line (but not -h from usage line) + short_match = CLI_PATTERNS["short_flag"].search(line) + if short_match and short_match.group(1) != 'h': # Skip -h help flag + native_keys.append(f"-{short_match.group(1)}") + + # Extract description - skip placeholder if present + # Pattern: --param PLACEHOLDER Description text + # We want to skip the PLACEHOLDER (uppercase word) if it exists + desc_match = re.search(rf"--{param_name}\s+(?:[A-Z_]+\s+)?(.+)", line) + if not desc_match: + # Fallback: just get everything after the flag + desc_match = re.search(r"--[^\s]+\s+(.+)", line) + description = desc_match.group(1).strip() if desc_match else None + + # Check if required + required = CLI_PATTERNS["required"].search(line) is not None + + # Extract default value from description line (not usage line) + default_match = CLI_PATTERNS["default_value"].search(line) + if default_match: + default_val = default_match.group(1).strip() + + # Extract type hint + type_match = CLI_PATTERNS["type_hint"].search(line) + type_hint = type_match.group(1) if type_match else None + + current_param = RawParameter( + name=normalized, + native_keys=native_keys, + description=description, + type_hint=type_hint, + default_value=parse_default_value(default_val) if default_val else None, + required=required, + source=line, + provenance={"line": lines.index(line) + 1} + ) + parameters.append(current_param) + + # Match short flags: -p + elif CLI_PATTERNS["short_flag"].search(line) and not long_match: + short_match = CLI_PATTERNS["short_flag"].search(line) + param_name = short_match.group(1) + normalized = normalize_parameter_name(param_name) + + current_param = RawParameter( + name=normalized, + native_keys=[f"-{param_name}"], + description=None, + source=line, + provenance={"line": lines.index(line) + 1} + ) + parameters.append(current_param) + + # If we have a current param, try to extract description from continuation lines + elif current_param and line.strip() and not line.strip().startswith('-'): + if not current_param.description: + current_param.description = line.strip() + else: + current_param.description += " " + line.strip() + + except Exception as e: + errors.append(f"Error extracting CLI parameters: {str(e)}") + + return ExtractionResult( + tool_name=tool_name or "unknown_cli_tool", + source_type=SourceType.CLI, + extraction_method=ExtractionMethod.REGEX, + parameters=parameters, + errors=errors + ) + + +class LLMCLIExtractor(LLMExtractor): + """LLM-based CLI parameter extraction.""" + + def __init__(self, llm_client=None): + super().__init__(SourceType.CLI, llm_client) + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + return f"""Extract all configuration parameters from the following CLI help output. +For each parameter, identify: +- Parameter name (normalized, without -- or -) +- Native keys/flags (--flag, -f, etc.) +- Description +- Type (if mentioned) +- Default value (if mentioned) +- Whether it's required or optional + +CLI Help Output: +{source} + +Return a JSON object with a 'parameters' array. Each parameter should have: name, native_keys, description, type_hint, default_value, required.""" + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + from pydantic import BaseModel + from typing import List as TypingList + + class ParameterSchema(BaseModel): + name: str + native_keys: TypingList[str] + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Union[str, int, float, bool]] = None + required: bool = False + + class ExtractionSchema(BaseModel): + parameters: TypingList[ParameterSchema] + + try: + prompt = self._create_prompt(source, tool_name) + response = self.llm_client.send_prompt(prompt, ExtractionSchema) + + parameters = [] + if "parameters" in response: + for param_data in response["parameters"]: + raw_param = RawParameter( + name=normalize_parameter_name(param_data["name"]), + native_keys=param_data.get("native_keys", []), + description=param_data.get("description"), + type_hint=param_data.get("type_hint"), + default_value=param_data.get("default_value"), + required=param_data.get("required", False), + source=source[:100], # First 100 chars + provenance={"method": "llm"} + ) + parameters.append(raw_param) + + return ExtractionResult( + tool_name=tool_name or "unknown_cli_tool", + source_type=SourceType.CLI, + extraction_method=ExtractionMethod.LLM, + parameters=parameters + ) + except Exception as e: + return ExtractionResult( + tool_name=tool_name or "unknown_cli_tool", + source_type=SourceType.CLI, + extraction_method=ExtractionMethod.LLM, + parameters=[], + errors=[f"LLM extraction failed: {str(e)}"] + ) + + diff --git a/src/kgpipe_parameters/extraction/extractors/docker.py b/src/kgpipe_parameters/extraction/extractors/docker.py new file mode 100644 index 0000000..e26b85a --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/docker.py @@ -0,0 +1,188 @@ +""" +Docker parameter extraction from Dockerfile and docker-compose.yml. +""" + +import yaml +from typing import List, Optional, Union + +from ..models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from ..base import RegexExtractor, LLMExtractor +from ..patterns import DOCKER_PATTERNS +from ..utils import normalize_parameter_name, parse_default_value + + +class DockerExtractor(RegexExtractor): + """Extract parameters from Docker configurations (Dockerfile, docker-compose.yml).""" + + def __init__(self, use_llm: bool = False, llm_client=None): + super().__init__(SourceType.DOCKER, DOCKER_PATTERNS) + self.use_llm = use_llm + if use_llm: + self.llm_extractor = LLMDockerExtractor(llm_client) + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters from Docker configuration.""" + if self.use_llm: + return self.llm_extractor.extract(source, tool_name) + + parameters = [] + errors = [] + + try: + # Check if it's a Dockerfile or docker-compose.yml + if "FROM" in source or "RUN" in source: + # Dockerfile + parameters.extend(self._extract_from_dockerfile(source)) + elif "version:" in source or "services:" in source: + # docker-compose.yml + try: + compose = yaml.safe_load(source) + parameters.extend(self._extract_from_compose(compose)) + except yaml.YAMLError: + parameters.extend(self._extract_from_dockerfile(source)) + else: + parameters.extend(self._extract_from_dockerfile(source)) + + except Exception as e: + errors.append(f"Error extracting Docker parameters: {str(e)}") + + return ExtractionResult( + tool_name=tool_name or "unknown_docker", + source_type=SourceType.DOCKER, + extraction_method=ExtractionMethod.REGEX, + parameters=parameters, + errors=errors + ) + + def _extract_from_dockerfile(self, source: str) -> List[RawParameter]: + """Extract ENV and ARG declarations from Dockerfile.""" + parameters = [] + lines = source.split('\n') + + for line in lines: + # ENV declarations + env_match = DOCKER_PATTERNS["env_declaration"].search(line) + if env_match: + var_name = env_match.group(1) + var_value = env_match.group(2) if env_match.group(2) else None + + parameters.append(RawParameter( + name=normalize_parameter_name(var_name), + native_keys=[var_name], + description=f"Environment variable: {var_name}", + default_value=parse_default_value(var_value) if var_value else None, + required=False, + source=line, + provenance={"type": "ENV", "line": lines.index(line) + 1} + )) + + # ARG declarations + arg_match = DOCKER_PATTERNS["arg_declaration"].search(line) + if arg_match: + var_name = arg_match.group(1) + var_value = arg_match.group(2) if arg_match.group(2) else None + + parameters.append(RawParameter( + name=normalize_parameter_name(var_name), + native_keys=[var_name], + description=f"Build argument: {var_name}", + default_value=parse_default_value(var_value) if var_value else None, + required=False, + source=line, + provenance={"type": "ARG", "line": lines.index(line) + 1} + )) + + return parameters + + def _extract_from_compose(self, compose: dict) -> List[RawParameter]: + """Extract environment variables from docker-compose.yml.""" + parameters = [] + + services = compose.get("services", {}) + for service_name, service_config in services.items(): + env = service_config.get("environment", {}) + if isinstance(env, dict): + for var_name, var_value in env.items(): + parameters.append(RawParameter( + name=normalize_parameter_name(var_name), + native_keys=[var_name], + description=f"Environment variable for service {service_name}", + default_value=parse_default_value(str(var_value)) if var_value else None, + required=False, + source=f"services.{service_name}.environment", + provenance={"service": service_name, "type": "environment"} + )) + + return parameters + + +class LLMDockerExtractor(LLMExtractor): + """LLM-based Docker parameter extraction.""" + + def __init__(self, llm_client=None): + super().__init__(SourceType.DOCKER, llm_client) + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + return f"""Extract all configuration parameters from the following Docker configuration. +Look for: +- ENV variables +- ARG build arguments +- Environment variables in docker-compose.yml +- Volume mounts and port mappings that could be parameterized + +Docker Configuration: +{source} + +Return a JSON object with a 'parameters' array. Each parameter should have: name, native_keys, description, type_hint, default_value, required.""" + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + from pydantic import BaseModel + from typing import List as TypingList + + class ParameterSchema(BaseModel): + name: str + native_keys: TypingList[str] + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Union[str, int, float, bool]] = None + required: bool = False + + class ExtractionSchema(BaseModel): + parameters: TypingList[ParameterSchema] + + try: + prompt = self._create_prompt(source, tool_name) + response = self.llm_client.send_prompt(prompt, ExtractionSchema) + + parameters = [] + if "parameters" in response: + for param_data in response["parameters"]: + raw_param = RawParameter( + name=normalize_parameter_name(param_data["name"]), + native_keys=param_data.get("native_keys", []), + description=param_data.get("description"), + type_hint=param_data.get("type_hint"), + default_value=param_data.get("default_value"), + required=param_data.get("required", False), + source=source[:200], + provenance={"method": "llm"} + ) + parameters.append(raw_param) + + return ExtractionResult( + tool_name=tool_name or "unknown_docker", + source_type=SourceType.DOCKER, + extraction_method=ExtractionMethod.LLM, + parameters=parameters + ) + except Exception as e: + return ExtractionResult( + tool_name=tool_name or "unknown_docker", + source_type=SourceType.DOCKER, + extraction_method=ExtractionMethod.LLM, + parameters=[], + errors=[f"LLM extraction failed: {str(e)}"] + ) + + diff --git a/src/kgpipe_parameters/extraction/extractors/http_api.py b/src/kgpipe_parameters/extraction/extractors/http_api.py new file mode 100644 index 0000000..35591ed --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/http_api.py @@ -0,0 +1,186 @@ +""" +HTTP API parameter extraction from OpenAPI/Swagger specs and documentation. +""" + +import json +import yaml +from typing import List, Optional, Union + +from ..models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from ..base import RegexExtractor, LLMExtractor +from ..utils import normalize_parameter_name + + +class HTTPAPIExtractor(RegexExtractor): + """Extract parameters from HTTP API documentation (OpenAPI, Swagger, etc.).""" + + def __init__(self, use_llm: bool = False, llm_client=None): + super().__init__(SourceType.HTTP_API, {}) + self.use_llm = use_llm + if use_llm: + self.llm_extractor = LLMHTTPExtractor(llm_client) + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters from API documentation.""" + if self.use_llm: + return self.llm_extractor.extract(source, tool_name) + + parameters = [] + errors = [] + + try: + # Try to parse as OpenAPI/Swagger spec + spec = None + try: + # Try JSON first + if source.strip().startswith('{'): + spec = json.loads(source) + else: + # Try YAML + spec = yaml.safe_load(source) + + # Check if it looks like OpenAPI/Swagger spec + if spec and isinstance(spec, dict) and ("openapi" in spec or "swagger" in spec or "paths" in spec): + parameters.extend(self._extract_from_openapi(spec)) + else: + # Not a valid spec, try regex-based extraction + parameters.extend(self._extract_from_docs(source)) + except (json.JSONDecodeError, yaml.YAMLError): + # If parsing fails, try regex-based extraction + parameters.extend(self._extract_from_docs(source)) + + except Exception as e: + errors.append(f"Error extracting API parameters: {str(e)}") + + return ExtractionResult( + tool_name=tool_name or "unknown_api", + source_type=SourceType.HTTP_API, + extraction_method=ExtractionMethod.REGEX, + parameters=parameters, + errors=errors + ) + + def _extract_from_openapi(self, spec: dict) -> List[RawParameter]: + """Extract parameters from OpenAPI specification.""" + parameters = [] + + # Extract from paths + paths = spec.get("paths", {}) + for path, methods in paths.items(): + for method, operation in methods.items(): + # Path parameters + for param in operation.get("parameters", []): + param_name = param.get("name", "") + param_schema = param.get("schema", {}) + + raw_param = RawParameter( + name=normalize_parameter_name(param_name), + native_keys=[param_name], + description=param.get("description"), + type_hint=param_schema.get("type"), + default_value=param_schema.get("default"), + required=param.get("required", False), + source=f"{method.upper()} {path}", + provenance={"location": "path", "method": method} + ) + parameters.append(raw_param) + + # Request body parameters + request_body = operation.get("requestBody", {}) + content = request_body.get("content", {}) + for content_type, schema_obj in content.items(): + schema = schema_obj.get("schema", {}) + if "properties" in schema: + for prop_name, prop_schema in schema["properties"].items(): + raw_param = RawParameter( + name=normalize_parameter_name(prop_name), + native_keys=[prop_name], + description=prop_schema.get("description"), + type_hint=prop_schema.get("type"), + default_value=prop_schema.get("default"), + required=prop_name in schema.get("required", []), + source=f"{method.upper()} {path} (body)", + provenance={"location": "body", "method": method} + ) + parameters.append(raw_param) + + return parameters + + def _extract_from_docs(self, source: str) -> List[RawParameter]: + """Extract parameters from unstructured API documentation.""" + parameters = [] + # Basic regex extraction for common patterns + # This is a simplified version - LLM would be better for complex docs + return parameters + + +class LLMHTTPExtractor(LLMExtractor): + """LLM-based HTTP API parameter extraction.""" + + def __init__(self, llm_client=None): + super().__init__(SourceType.HTTP_API, llm_client) + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + return f"""Extract all API parameters from the following API documentation or specification. +Look for: +- Query parameters +- Path parameters +- Request body parameters +- Header parameters + +API Documentation: +{source} + +Return a JSON object with a 'parameters' array. Each parameter should have: name, native_keys, description, type_hint, default_value, required.""" + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + from pydantic import BaseModel + from typing import List as TypingList + + class ParameterSchema(BaseModel): + name: str + native_keys: TypingList[str] + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Union[str, int, float, bool]] = None + required: bool = False + + class ExtractionSchema(BaseModel): + parameters: TypingList[ParameterSchema] + + try: + prompt = self._create_prompt(source, tool_name) + response = self.llm_client.send_prompt(prompt, ExtractionSchema) + + parameters = [] + if "parameters" in response: + for param_data in response["parameters"]: + raw_param = RawParameter( + name=normalize_parameter_name(param_data["name"]), + native_keys=param_data.get("native_keys", []), + description=param_data.get("description"), + type_hint=param_data.get("type_hint"), + default_value=param_data.get("default_value"), + required=param_data.get("required", False), + source=source[:200], + provenance={"method": "llm"} + ) + parameters.append(raw_param) + + return ExtractionResult( + tool_name=tool_name or "unknown_api", + source_type=SourceType.HTTP_API, + extraction_method=ExtractionMethod.LLM, + parameters=parameters + ) + except Exception as e: + return ExtractionResult( + tool_name=tool_name or "unknown_api", + source_type=SourceType.HTTP_API, + extraction_method=ExtractionMethod.LLM, + parameters=[], + errors=[f"LLM extraction failed: {str(e)}"] + ) + + diff --git a/src/kgpipe_parameters/extraction/extractors/python_lib.py b/src/kgpipe_parameters/extraction/extractors/python_lib.py new file mode 100644 index 0000000..bb62cef --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/python_lib.py @@ -0,0 +1,359 @@ +""" +Python library parameter extraction from source code. +""" + +import re +import ast +from typing import List, Optional, Union + +from ..models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from ..base import RegexExtractor, LLMExtractor +from ..patterns import PYTHON_PATTERNS +from ..utils import normalize_parameter_name, parse_default_value + + +class PythonLibExtractor(RegexExtractor): + """Extract parameters from Python code (functions, classes, docstrings).""" + + def __init__(self, use_llm: bool = False, llm_client=None): + super().__init__(SourceType.PYTHON_LIB, PYTHON_PATTERNS) + self.use_llm = use_llm + if use_llm: + self.llm_extractor = LLMPythonExtractor(llm_client) + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters from Python source code.""" + if self.use_llm: + return self.llm_extractor.extract(source, tool_name) + + parameters = [] + errors = [] + + try: + # Try to parse as Python AST + try: + tree = ast.parse(source) + parameters.extend(self._extract_from_ast(tree, source)) + except SyntaxError: + # If not valid Python, try regex-based extraction + parameters.extend(self._extract_from_regex(source)) + + except Exception as e: + errors.append(f"Error extracting Python parameters: {str(e)}") + + return ExtractionResult( + tool_name=tool_name or "unknown_python_lib", + source_type=SourceType.PYTHON_LIB, + extraction_method=ExtractionMethod.REGEX, + parameters=parameters, + errors=errors + ) + + # Type hints that almost certainly indicate I/O data, not configuration. + _IO_TYPE_HINTS = frozenset({ + "DataFrame", "pd.DataFrame", "pandas.DataFrame", + "ndarray", "np.ndarray", "numpy.ndarray", + "Series", "pd.Series", + "BaseTable", "BaseColumn", + "Table", "Column", + "Dataset", + "Pool", "Process", + "Iterator", "Generator", + "TextIO", "BinaryIO", "IO", + }) + + @classmethod + def _looks_like_io_param(cls, name: str, type_hint: Optional[str], has_default: bool) -> bool: + """ + Heuristic: return True if a function parameter is likely an I/O + argument rather than a tunable configuration knob. + + Rules: + - Parameters whose type hint is a known data type (DataFrame, ndarray, + BaseTable, etc.) are I/O. + - Required parameters (no default) of non-__init__ methods whose names + suggest data flow (source, target, input, output, data, table, path, + pool, etc.) are I/O. + """ + if type_hint: + # Check the raw type and any component of a composite hint + for io_type in cls._IO_TYPE_HINTS: + if io_type in type_hint: + return True + # Common I/O parameter name stems + io_name_hints = { + "source", "target", "input", "output", "data", + "table", "column", "pool", "file", "path", + "stream", "buffer", "reader", "writer", + } + normalized = name.lower().replace("_", "") + for h in io_name_hints: + if h in normalized: + # If it has a simple scalar default, it might still be config + if has_default: + return False + return True + return False + + def _extract_from_ast(self, tree: ast.AST, source: str) -> List[RawParameter]: + """Extract parameters from Python AST.""" + parameters = [] + extractor_cls = self # reference for nested class + + class ParameterVisitor(ast.NodeVisitor): + def __init__(self): + self.params = [] + self.source_lines = source.split('\n') + self._current_class = None + + def visit_ClassDef(self, node): + prev_class = self._current_class + self._current_class = node.name + + # Extract class-level attributes (dataclasses, Pydantic models, etc.) + for item in node.body: + if isinstance(item, ast.AnnAssign): + # Annotated assignment: name: type = default + if isinstance(item.target, ast.Name): + attr_name = item.target.id + + # Get type hint + type_hint = None + if item.annotation: + type_hint = ast.unparse(item.annotation) if hasattr(ast, 'unparse') else str(item.annotation) + + # Get default value + default_val = None + if item.value: + if hasattr(ast, 'unparse'): + default_val = ast.unparse(item.value) + else: + try: + default_val = ast.literal_eval(item.value) + except (ValueError, TypeError): + default_val = None + + param = RawParameter( + name=normalize_parameter_name(attr_name), + native_keys=[attr_name], + description=None, + type_hint=type_hint, + default_value=parse_default_value(str(default_val)) if default_val is not None else None, + required=default_val is None, + source=f"{node.name}.{attr_name}", + provenance={"class": node.name, "line": item.lineno if hasattr(item, 'lineno') else node.lineno} + ) + self.params.append(param) + elif isinstance(item, ast.Assign): + # Regular assignment: name = value (might be in dataclass) + for target in item.targets: + if isinstance(target, ast.Name): + attr_name = target.id + # Try to get value + default_val = None + if item.value: + try: + default_val = ast.literal_eval(item.value) + except (ValueError, TypeError): + default_val = None + + param = RawParameter( + name=normalize_parameter_name(attr_name), + native_keys=[attr_name], + description=None, + type_hint=None, + default_value=parse_default_value(str(default_val)) if default_val is not None else None, + required=False, + source=f"{node.name}.{attr_name}", + provenance={"class": node.name, "line": item.lineno} + ) + self.params.append(param) + + self.generic_visit(node) + self._current_class = prev_class + + def visit_FunctionDef(self, node): + is_init = node.name == '__init__' + is_method = self._current_class is not None + class_name = self._current_class + # For non-__init__ methods inside a class, only keep params + # that look like configuration (have defaults and don't look + # like I/O data arguments). + skip_io = is_method and not is_init + + for arg in node.args.args: + if arg.arg in ('self', 'cls'): + continue + + # Get type hint + type_hint = None + if arg.annotation: + type_hint = ast.unparse(arg.annotation) if hasattr(ast, 'unparse') else str(arg.annotation) + + # Get default value + default_val = None + default_idx = len(node.args.args) - len(node.args.defaults) + if arg in node.args.args[default_idx:]: + default_node = node.args.defaults[node.args.args[default_idx:].index(arg)] + if hasattr(ast, 'unparse'): + default_val = ast.unparse(default_node) + else: + default_val = ast.literal_eval(default_node) if isinstance(default_node, (ast.Constant, ast.Str, ast.Num)) else None + + has_default = default_val is not None + + # ── I/O filter for non-constructor methods ── + if skip_io and extractor_cls._looks_like_io_param(arg.arg, type_hint, has_default): + continue + + # For non-__init__ methods, skip required params that + # have no default — they're almost always data args. + if skip_io and not has_default: + continue + + # Extract docstring info (Sphinx :param: and numpydoc styles) + description = None + if ast.get_docstring(node): + docstring = ast.get_docstring(node) + # Sphinx style — :param name: description + sphinx_pat = re.compile( + rf":param\s+{re.escape(arg.arg)}:\s*(.+?)(?=\n|:param|$)", + re.MULTILINE, + ) + m = sphinx_pat.search(docstring) + if m: + description = m.group(1).strip() + else: + # Numpydoc style — + # name : type + # Description text + numpydoc_pat = re.compile( + rf"^\s*{re.escape(arg.arg)}\s*(?::.*)?$\n((?:[ \t]+.+\n?)+)", + re.MULTILINE, + ) + m = numpydoc_pat.search(docstring) + if m: + # Merge continuation lines and strip indent + desc_lines = [l.strip() for l in m.group(1).splitlines() if l.strip()] + description = " ".join(desc_lines) + + func_label = f"{class_name}.{node.name}" if class_name else node.name + param = RawParameter( + name=normalize_parameter_name(arg.arg), + native_keys=[arg.arg], + description=description, + type_hint=type_hint, + default_value=parse_default_value(default_val) if default_val else None, + required=default_val is None, + source=f"{func_label}()", + provenance={ + "function": node.name, + "class": class_name, + "is_constructor": is_init, + "line": node.lineno, + } + ) + self.params.append(param) + + self.generic_visit(node) + + visitor = ParameterVisitor() + visitor.visit(tree) + return visitor.params + + def _extract_from_regex(self, source: str) -> List[RawParameter]: + """Fallback regex-based extraction.""" + parameters = [] + + # Extract function parameters + func_pattern = re.compile(r"def\s+\w+\s*\(([^)]+)\)", re.MULTILINE) + for match in func_pattern.finditer(source): + params_str = match.group(1) + for param_match in PYTHON_PATTERNS["function_param"].finditer(params_str): + param_name = param_match.group(1) + type_hint = param_match.group(2).strip() if param_match.group(2) else None + default_val = param_match.group(3).strip() if param_match.group(3) else None + + parameters.append(RawParameter( + name=normalize_parameter_name(param_name), + native_keys=[param_name], + type_hint=type_hint, + default_value=parse_default_value(default_val) if default_val else None, + required=default_val is None, + source=match.group(0), + provenance={"method": "regex"} + )) + + return parameters + + +class LLMPythonExtractor(LLMExtractor): + """LLM-based Python parameter extraction.""" + + def __init__(self, llm_client=None): + super().__init__(SourceType.PYTHON_LIB, llm_client) + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + return f"""Extract all configuration parameters from the following Python code. +Look for: +- Function parameters with type hints and defaults +- Class attributes with type annotations +- Configuration classes (dataclasses, Pydantic models) +- Environment variables + +Python Code: +{source} + +Return a JSON object with a 'parameters' array. Each parameter should have: name, native_keys, description, type_hint, default_value, required.""" + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + from pydantic import BaseModel + from typing import List as TypingList + + class ParameterSchema(BaseModel): + name: str + native_keys: TypingList[str] + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Union[str, int, float, bool]] = None + required: bool = False + + class ExtractionSchema(BaseModel): + parameters: TypingList[ParameterSchema] + + try: + prompt = self._create_prompt(source, tool_name) + response = self.llm_client.send_prompt(prompt, ExtractionSchema) + + parameters = [] + if "parameters" in response: + for param_data in response["parameters"]: + raw_param = RawParameter( + name=normalize_parameter_name(param_data["name"]), + native_keys=param_data.get("native_keys", []), + description=param_data.get("description"), + type_hint=param_data.get("type_hint"), + default_value=param_data.get("default_value"), + required=param_data.get("required", False), + source=source[:200], # First 200 chars + provenance={"method": "llm"} + ) + parameters.append(raw_param) + + return ExtractionResult( + tool_name=tool_name or "unknown_python_lib", + source_type=SourceType.PYTHON_LIB, + extraction_method=ExtractionMethod.LLM, + parameters=parameters + ) + except Exception as e: + return ExtractionResult( + tool_name=tool_name or "unknown_python_lib", + source_type=SourceType.PYTHON_LIB, + extraction_method=ExtractionMethod.LLM, + parameters=[], + errors=[f"LLM extraction failed: {str(e)}"] + ) + + diff --git a/src/kgpipe_parameters/extraction/extractors/readme_doc.py b/src/kgpipe_parameters/extraction/extractors/readme_doc.py new file mode 100644 index 0000000..aca8392 --- /dev/null +++ b/src/kgpipe_parameters/extraction/extractors/readme_doc.py @@ -0,0 +1,320 @@ +""" +README / documentation parameter extraction. + +Extracts configuration parameters from README files, documentation pages, +and other unstructured markdown/text docs that describe tool usage. +""" + +import re +from typing import List, Optional, Set, Union + +from ..models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from ..base import RegexExtractor, LLMExtractor +from ..patterns import README_PATTERNS +from ..utils import normalize_parameter_name, parse_default_value + + +# Noise words that appear as flags/placeholders but are not real parameters +_NOISE_NAMES: Set[str] = { + "h", "help", "version", "v", "verbose", "quiet", "q", + "the", "a", "an", "is", "are", "was", "were", "be", + "to", "of", "in", "for", "on", "at", "by", "with", + "it", "its", "we", "our", "you", "your", + "e", "g", "i", "x", "s", +} + + +def _extract_code_blocks(text: str) -> List[str]: + """Return contents of fenced code blocks (``` … ```).""" + return re.findall(r"```[^\n]*\n(.*?)```", text, re.DOTALL) + + +class ReadmeDocExtractor(RegexExtractor): + """Extract parameters from README / documentation text (Markdown or plain text).""" + + def __init__(self, use_llm: bool = False, llm_client=None): + super().__init__(SourceType.README, README_PATTERNS) + self.use_llm = use_llm + if use_llm: + self.llm_extractor = LLMReadmeDocExtractor(llm_client) + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters from README / documentation text.""" + if self.use_llm: + return self.llm_extractor.extract(source, tool_name) + + parameters: List[RawParameter] = [] + errors: List[str] = [] + seen_names: Set[str] = set() + + try: + # --- 1. Parameters described in markdown list items --- + # e.g. - `threshold`: The matching threshold (default: 0.5) + for match in README_PATTERNS["list_param"].finditer(source): + name_raw = match.group(1) + description = match.group(2).strip() + normalized = normalize_parameter_name(name_raw) + if normalized in _NOISE_NAMES or len(normalized) < 2: + continue + if normalized in seen_names: + continue + seen_names.add(normalized) + + default_val = self._find_default(description) + type_hint = self._find_type_hint(description) + + parameters.append(RawParameter( + name=normalized, + native_keys=[name_raw], + description=description, + type_hint=type_hint, + default_value=parse_default_value(default_val) if default_val else None, + required=False, + source=match.group(0).strip(), + provenance={"method": "readme_list_param"}, + )) + + # --- 2. Parameters in markdown tables --- + for match in README_PATTERNS["table_param"].finditer(source): + name_raw = match.group(1).strip() + col2 = match.group(2).strip() + col3 = match.group(3).strip() + normalized = normalize_parameter_name(name_raw) + if normalized in _NOISE_NAMES or len(normalized) < 2: + continue + if normalized in seen_names: + continue + seen_names.add(normalized) + + # Heuristic: second column is often type, third is description + type_hint = col2 if col2 and len(col2) < 30 else None + description = col3 or col2 + + parameters.append(RawParameter( + name=normalized, + native_keys=[name_raw], + description=description if description else None, + type_hint=type_hint, + default_value=None, + required=False, + source=match.group(0).strip(), + provenance={"method": "readme_table"}, + )) + + # --- 3. Flags from code blocks --- + code_blocks = _extract_code_blocks(source) + for block in code_blocks: + for match in README_PATTERNS["code_block_flag"].finditer(block): + flag = match.group(1) + value = match.group(2) + normalized = normalize_parameter_name(flag) + if normalized in _NOISE_NAMES or len(normalized) < 2: + continue + if normalized in seen_names: + continue + seen_names.add(normalized) + + parameters.append(RawParameter( + name=normalized, + native_keys=[flag], + description=None, + type_hint=None, + default_value=parse_default_value(value) if value else None, + required=False, + source=block[:200].strip(), + provenance={"method": "readme_code_block"}, + )) + + # JVM-style flags + for jvm_match in README_PATTERNS["jvm_flag"].finditer(block): + flag = jvm_match.group(1) + normalized = normalize_parameter_name(flag) + if normalized in seen_names: + continue + seen_names.add(normalized) + + parameters.append(RawParameter( + name=normalized, + native_keys=[flag], + description=f"JVM flag: {flag}", + type_hint=None, + default_value=None, + required=False, + source=block[:200].strip(), + provenance={"method": "readme_jvm_flag"}, + )) + + # --- 4. Inline flags referenced with backticks --- + for match in README_PATTERNS["inline_flag"].finditer(source): + flag = match.group(1) + normalized = normalize_parameter_name(flag) + if normalized in _NOISE_NAMES or len(normalized) < 2: + continue + if normalized in seen_names: + continue + seen_names.add(normalized) + + # Try to find a surrounding sentence as description + start = max(0, match.start() - 120) + end = min(len(source), match.end() + 120) + context = source[start:end].replace("\n", " ").strip() + + parameters.append(RawParameter( + name=normalized, + native_keys=[flag], + description=context, + type_hint=None, + default_value=None, + required=False, + source=context, + provenance={"method": "readme_inline_flag"}, + )) + + # --- 5. Environment variable references --- + for match in README_PATTERNS["env_reference"].finditer(source): + var_name = match.group(1) + var_value = match.group(2) if match.lastindex >= 2 else None + normalized = normalize_parameter_name(var_name) + if normalized in seen_names or len(normalized) < 2: + continue + seen_names.add(normalized) + + parameters.append(RawParameter( + name=normalized, + native_keys=[var_name], + description=f"Environment variable: {var_name}", + type_hint=None, + default_value=parse_default_value(var_value) if var_value else None, + required=False, + source=match.group(0).strip(), + provenance={"method": "readme_env_var"}, + )) + + # --- 6. Placeholder parameters from usage lines --- + for match in README_PATTERNS["placeholder"].finditer(source): + name_raw = match.group(1) + normalized = normalize_parameter_name(name_raw) + if normalized in _NOISE_NAMES or len(normalized) < 2: + continue + if normalized in seen_names: + continue + seen_names.add(normalized) + + # Grab surrounding line as context + line_start = source.rfind("\n", 0, match.start()) + 1 + line_end = source.find("\n", match.end()) + if line_end == -1: + line_end = len(source) + context_line = source[line_start:line_end].strip() + + parameters.append(RawParameter( + name=normalized, + native_keys=[f"<{name_raw}>"], + description=context_line, + type_hint=None, + default_value=None, + required=True, # placeholders are usually required + source=context_line, + provenance={"method": "readme_placeholder"}, + )) + + except Exception as e: + errors.append(f"Error extracting README parameters: {str(e)}") + + return ExtractionResult( + tool_name=tool_name or "unknown_readme", + source_type=SourceType.README, + extraction_method=ExtractionMethod.REGEX, + parameters=parameters, + errors=errors, + ) + + # ------------------------------------------------------------------ + # helpers + # ------------------------------------------------------------------ + + @staticmethod + def _find_default(text: str) -> Optional[str]: + """Try to extract a default value from a description string.""" + m = re.search(r"default[=:]\s*[`\"']?([^`\"'\]),\s]+)", text, re.IGNORECASE) + return m.group(1) if m else None + + @staticmethod + def _find_type_hint(text: str) -> Optional[str]: + """Try to infer a type hint from a description string.""" + for token in ("int", "integer", "float", "number", "bool", "boolean", "string", "str", "path", "file"): + if re.search(rf"\b{token}\b", text, re.IGNORECASE): + return token + return None + + +class LLMReadmeDocExtractor(LLMExtractor): + """LLM-based README / documentation parameter extraction.""" + + def __init__(self, llm_client=None): + super().__init__(SourceType.README, llm_client) + + def _create_prompt(self, source: str, tool_name: Optional[str] = None) -> str: + return f"""Extract all configuration parameters from the following README / documentation text. +Look for: +- Command-line flags and options mentioned in usage examples +- Environment variables +- Configuration keys or settings +- Input/output paths that can be parameterized +- Any tunable values (thresholds, limits, memory sizes, etc.) + +Documentation: +{source} + +Return a JSON object with a 'parameters' array. Each parameter should have: name, native_keys, description, type_hint, default_value, required.""" + + def extract(self, source: str, tool_name: Optional[str] = None) -> ExtractionResult: + """Extract parameters using LLM.""" + from pydantic import BaseModel + from typing import List as TypingList + + class ParameterSchema(BaseModel): + name: str + native_keys: TypingList[str] + description: Optional[str] = None + type_hint: Optional[str] = None + default_value: Optional[Union[str, int, float, bool]] = None + required: bool = False + + class ExtractionSchema(BaseModel): + parameters: TypingList[ParameterSchema] + + try: + prompt = self._create_prompt(source, tool_name) + response = self.llm_client.send_prompt(prompt, ExtractionSchema) + + parameters = [] + if "parameters" in response: + for param_data in response["parameters"]: + raw_param = RawParameter( + name=normalize_parameter_name(param_data["name"]), + native_keys=param_data.get("native_keys", []), + description=param_data.get("description"), + type_hint=param_data.get("type_hint"), + default_value=param_data.get("default_value"), + required=param_data.get("required", False), + source=source[:200], + provenance={"method": "llm"}, + ) + parameters.append(raw_param) + + return ExtractionResult( + tool_name=tool_name or "unknown_readme", + source_type=SourceType.README, + extraction_method=ExtractionMethod.LLM, + parameters=parameters, + ) + except Exception as e: + return ExtractionResult( + tool_name=tool_name or "unknown_readme", + source_type=SourceType.README, + extraction_method=ExtractionMethod.LLM, + parameters=[], + errors=[f"LLM extraction failed: {str(e)}"], + ) + diff --git a/src/kgpipe_parameters/extraction/models.py b/src/kgpipe_parameters/extraction/models.py new file mode 100644 index 0000000..f209fe0 --- /dev/null +++ b/src/kgpipe_parameters/extraction/models.py @@ -0,0 +1,85 @@ +""" +Pydantic models for raw parameter extraction results. +""" + +from typing import List, Optional, Dict, Any, Union +from pydantic import BaseModel, Field, ConfigDict +from datetime import datetime +from enum import Enum + + +class SourceType(str, Enum): + """Types of sources for parameter extraction.""" + CLI = "cli" + PYTHON_LIB = "python_lib" + HTTP_API = "http_api" + DOCKER = "docker" + README = "readme" + UNKNOWN = "unknown" + + +class ExtractionMethod(str, Enum): + """Methods used for parameter extraction.""" + REGEX = "regex" + LLM = "llm" + AUTO = "auto" + + +class RawParameter(BaseModel): + """ + Intermediate representation of an extracted parameter. + This is the raw extraction result before conversion to Parameter model. + """ + name: str = Field(..., description="Normalized parameter name") + native_keys: List[str] = Field(default_factory=list, description="Original parameter names/flags from source") + description: Optional[str] = Field(None, description="Parameter description/documentation") + type_hint: Optional[str] = Field(None, description="Type hint or type name from source") + default_value: Optional[Union[str, int, float, bool]] = Field(None, description="Default value if present") + required: bool = Field(False, description="Whether parameter is required") + constraints: Dict[str, Any] = Field(default_factory=dict, description="Constraints like min, max, allowed_values") + source: str = Field(..., description="Source text or file path where parameter was found") + provenance: Dict[str, Any] = Field(default_factory=dict, description="Additional metadata about extraction") + + model_config = ConfigDict( + json_schema_extra={ + "example": { + "name": "threshold", + "native_keys": ["--threshold", "-t", "THRESHOLD"], + "description": "Matching threshold value", + "type_hint": "float", + "default_value": 0.5, + "required": False, + "constraints": {"minimum": 0.0, "maximum": 1.0}, + "source": "tool.py --help", + "provenance": {"line_number": 42, "extraction_method": "regex"} + } + } + ) + + +class ExtractionResult(BaseModel): + """ + Container for extracted parameters with metadata. + """ + tool_name: str = Field(..., description="Name of the tool/library being analyzed") + source_type: SourceType = Field(..., description="Type of source (CLI, Python, API, Docker)") + extraction_method: ExtractionMethod = Field(..., description="Method used for extraction") + parameters: List[RawParameter] = Field(default_factory=list, description="List of extracted parameters") + timestamp: datetime = Field(default_factory=datetime.now, description="When extraction was performed") + metadata: Dict[str, Any] = Field(default_factory=dict, description="Additional metadata about the extraction") + errors: List[str] = Field(default_factory=list, description="Any errors encountered during extraction") + + model_config = ConfigDict( + json_schema_extra={ + "example": { + "tool_name": "paris_matcher", + "source_type": "cli", + "extraction_method": "regex", + "parameters": [], + "timestamp": "2024-01-01T00:00:00", + "metadata": {"source_file": "paris --help"}, + "errors": [] + } + } + ) + diff --git a/src/kgpipe_parameters/extraction/param_miner.py b/src/kgpipe_parameters/extraction/param_miner.py new file mode 100644 index 0000000..bb209d0 --- /dev/null +++ b/src/kgpipe_parameters/extraction/param_miner.py @@ -0,0 +1,204 @@ +""" +Parameter mining/extraction from various sources (CLI, Python, HTTP APIs, Docker). + +This module provides the main ParameterMiner class for unified parameter extraction. +Individual extractors are implemented in the extractors/ submodule. +""" + +import ast +from pathlib import Path +from typing import Optional, Union + +from .models import RawParameter, ExtractionResult, SourceType, ExtractionMethod +from .extractors import ( + CLIExtractor, + PythonLibExtractor, + HTTPAPIExtractor, + DockerExtractor, + ReadmeDocExtractor, + LLMCLIExtractor, + LLMPythonExtractor, + LLMHTTPExtractor, + LLMDockerExtractor, + LLMReadmeDocExtractor, +) + +# Re-export extractors for backwards compatibility +__all__ = [ + "ParameterMiner", + "CLIExtractor", + "PythonLibExtractor", + "HTTPAPIExtractor", + "DockerExtractor", + "ReadmeDocExtractor", + "LLMCLIExtractor", + "LLMPythonExtractor", + "LLMHTTPExtractor", + "LLMDockerExtractor", + "LLMReadmeDocExtractor", +] + + +class ParameterMiner: + """ + Main class for parameter extraction from various sources. + Provides unified interface for extracting configuration parameters. + """ + + def __init__(self, llm_client=None): + """ + Initialize ParameterMiner. + + Args: + llm_client: Optional LLMClient instance for LLM-based extraction + """ + self.llm_client = llm_client + self.extractors = { + SourceType.CLI: CLIExtractor(use_llm=False), + SourceType.PYTHON_LIB: PythonLibExtractor(use_llm=False), + SourceType.HTTP_API: HTTPAPIExtractor(use_llm=False), + SourceType.DOCKER: DockerExtractor(use_llm=False), + SourceType.README: ReadmeDocExtractor(use_llm=False), + } + + def extract_parameters( + self, + source: Union[str, Path], + source_type: Optional[SourceType] = None, + method: ExtractionMethod = ExtractionMethod.AUTO, + tool_name: Optional[str] = None + ) -> ExtractionResult: + """ + Extract parameters from a source. + + Args: + source: Source content (text, file path, etc.) + source_type: Type of source (auto-detected if None) + method: Extraction method ('regex', 'llm', or 'auto') + tool_name: Optional name of the tool being analyzed + + Returns: + ExtractionResult containing extracted parameters + """ + # Read file if Path provided + if isinstance(source, Path): + source_path = source + source = source_path.read_text() + if not tool_name: + tool_name = source_path.stem + elif isinstance(source, str): + # Only treat as file path if it's a short string without newlines + # and actually exists as a file + if len(source) < 260 and '\n' not in source and Path(source).exists(): + source_path = Path(source) + source = source_path.read_text() + if not tool_name: + tool_name = source_path.stem + + # Auto-detect source type if not provided + if source_type is None: + source_type = self._detect_source_type(source) + + # Select extraction method + if method == ExtractionMethod.AUTO: + # Try regex first, fallback to LLM if available + try: + if source_type == SourceType.UNKNOWN: + # For unknown source types, try CLI extractor as fallback + extractor = self.extractors[SourceType.CLI] + else: + extractor = self.extractors[source_type] + result = extractor.extract(source, tool_name) + # If regex extraction found few/no parameters and LLM is available, try LLM + if len(result.parameters) == 0 and self.llm_client: + method = ExtractionMethod.LLM + else: + return result + except (KeyError, Exception): + if self.llm_client: + method = ExtractionMethod.LLM + else: + # Return empty result for unknown types + return ExtractionResult( + tool_name=tool_name or "unknown", + source_type=source_type, + extraction_method=ExtractionMethod.REGEX, + parameters=[], + errors=[f"No extractor available for source type: {source_type}"] + ) + + # Use LLM if requested or as fallback + if method == ExtractionMethod.LLM: + if not self.llm_client: + raise ValueError("LLM extraction requires an LLMClient instance") + + # Create LLM extractor for the source type + llm_extractors = { + SourceType.CLI: LLMCLIExtractor(self.llm_client), + SourceType.PYTHON_LIB: LLMPythonExtractor(self.llm_client), + SourceType.HTTP_API: LLMHTTPExtractor(self.llm_client), + SourceType.DOCKER: LLMDockerExtractor(self.llm_client), + SourceType.README: LLMReadmeDocExtractor(self.llm_client), + } + extractor = llm_extractors.get(source_type) + if extractor: + return extractor.extract(source, tool_name) + + # Use regex extractor + extractor = self.extractors[source_type] + return extractor.extract(source, tool_name) + + def _detect_source_type(self, source: str) -> SourceType: + """Auto-detect source type from content.""" + source_lower = source.lower() + + # Check for CLI help patterns + if any(x in source_lower for x in ["usage:", "options:", "--help", "arguments:"]): + return SourceType.CLI + + # Check for Python code + if any(x in source for x in ["def ", "class ", "import ", "@"]): + try: + ast.parse(source) + return SourceType.PYTHON_LIB + except SyntaxError: + pass + + # Check for OpenAPI/Swagger + if any(x in source for x in ['"openapi"', '"swagger"', "paths:", "components:"]): + return SourceType.HTTP_API + + # Check for Docker + if any(x in source for x in ["FROM ", "ENV ", "ARG ", "docker-compose", "services:"]): + return SourceType.DOCKER + + # Check for README / Markdown documentation + if any(x in source for x in ["# ", "## ", "```", "**", "[", "](", "---"]): + return SourceType.README + + return SourceType.UNKNOWN + + def to_parameter_model(self, raw_param: RawParameter): + """ + Convert RawParameter to Parameter model. + + Args: + raw_param: RawParameter instance + + Returns: + Parameter model instance + """ + from .utils import to_parameter_model + return to_parameter_model(raw_param) + + def to_json(self, result: ExtractionResult) -> str: + """ + Convert ExtractionResult to JSON string. + + Args: + result: ExtractionResult instance + + Returns: + JSON string representation + """ + return result.model_dump_json(indent=2) diff --git a/src/kgpipe_parameters/extraction/patterns.py b/src/kgpipe_parameters/extraction/patterns.py new file mode 100644 index 0000000..f1d985b --- /dev/null +++ b/src/kgpipe_parameters/extraction/patterns.py @@ -0,0 +1,156 @@ +""" +Regex patterns for parameter extraction from various sources. +""" + +import re +from typing import Dict, List, Tuple, Optional + + +# CLI argument patterns +CLI_PATTERNS = { + # Long form: --param, --param=VALUE, --param VALUE + "long_flag": re.compile(r"--([a-zA-Z][a-zA-Z0-9_-]*)(?:[=\s]+([^\s]+))?"), + # Short form: -p, -p VALUE, -pVALUE + "short_flag": re.compile(r"-([a-zA-Z])(?:\s+([^\s]+))?"), + # Combined: -p, --param + "combined_flag": re.compile(r"(-[a-zA-Z]|--[a-zA-Z][a-zA-Z0-9_-]+)"), + # Description lines (common in help output) + "description": re.compile(r"^\s+([^\s]+(?:\s+[^\s]+)*)\s+(.+)$"), + # Required/optional indicators + "required": re.compile(r"(required|mandatory|must)", re.IGNORECASE), + "optional": re.compile(r"(optional|\[optional\]|\[default)", re.IGNORECASE), + # Default values: [default: value], (default: value), default=value + # Match: (default: 0.5) -> capture "0.5", [default: value] -> capture "value", default=value -> capture "value" + # The pattern matches "default:" or "default=" and captures the value until closing bracket/paren or end + "default_value": re.compile(r"default[=:]\s*([^\])]+?)(?:\]|\)|$)", re.IGNORECASE), + # Type hints: , [str], (float) + "type_hint": re.compile(r"[<\[\(]([a-zA-Z]+)[>\]\)]"), +} + +# Python code patterns +PYTHON_PATTERNS = { + # Function parameter: param: type = default + "function_param": re.compile(r"(\w+)(?:\s*:\s*([^=]+))?(?:\s*=\s*([^,)]+))?"), + # Type hints: param: int, param: Optional[str] = None + "type_annotation": re.compile(r":\s*([^=,)]+)"), + # Default values in function signatures + "default_in_sig": re.compile(r"=\s*([^,)]+)"), + # Docstring parameter descriptions: :param name: description + "docstring_param": re.compile(r":param\s+(\w+):\s*(.+?)(?=\n|:param|$)", re.MULTILINE), + # Docstring type: :type name: type + "docstring_type": re.compile(r":type\s+(\w+):\s*([^\n]+)"), + # Class attributes with type hints + "class_attr": re.compile(r"(\w+)\s*:\s*([^=\n]+)(?:\s*=\s*([^\n]+))?"), + # Environment variable assignments: VAR = value + "env_var": re.compile(r"([A-Z_][A-Z0-9_]*)\s*=\s*(.+)"), +} + +# HTTP API patterns +API_PATTERNS = { + # Query parameters: ?param=value + "query_param": re.compile(r"[?&]([^=&]+)(?:=([^&]+))?"), + # Path parameters: /{param}/ + "path_param": re.compile(r"/\{([^}]+)\}/"), + # Header parameters: X-Header-Name: value + "header": re.compile(r"([A-Z][a-zA-Z0-9-]+):\s*(.+)"), + # JSON schema properties + "json_property": re.compile(r'"([^"]+)":\s*\{[^}]*"type":\s*"([^"]+)"'), + # OpenAPI parameter definitions + "openapi_param": re.compile(r'"([^"]+)":\s*\{[^}]*"in":\s*"([^"]+)"'), +} + +# Docker patterns +DOCKER_PATTERNS = { + # ENV variable: ENV VAR=value or ENV VAR value + "env_declaration": re.compile(r"ENV\s+([A-Z_][A-Z0-9_]*)(?:\s*=\s*|\s+)(.+)", re.IGNORECASE), + # ARG declaration: ARG VAR[=default] + "arg_declaration": re.compile(r"ARG\s+([A-Z_][A-Z0-9_]*)(?:\s*=\s*([^\s]+))?", re.IGNORECASE), + # Environment variable in docker-compose: VAR: value + "compose_env": re.compile(r"([A-Z_][A-Z0-9_]*)\s*:\s*(.+)"), + # Volume mounts: -v /host:/container + "volume_mount": re.compile(r"-v\s+([^:\s]+):([^:\s]+)"), + # Port mappings: -p HOST:CONTAINER + "port_mapping": re.compile(r"-p\s+(\d+):(\d+)"), +} + +# README / documentation patterns +README_PATTERNS = { + # Flags or options mentioned in code blocks or inline code: --param, -p + "inline_flag": re.compile(r"`(-{1,2}[a-zA-Z][a-zA-Z0-9_-]*)`"), + # Command-line invocations in code blocks: tool --param value + "code_block_flag": re.compile(r"(?:^|\s)(-{1,2}[a-zA-Z][a-zA-Z0-9_-]*)(?:\s+(\S+))?", re.MULTILINE), + # Environment variable references: $VAR, ${VAR}, ENV VAR, set VAR= + "env_reference": re.compile(r"(?:\$\{?|(?:set|export)\s+)([A-Z_][A-Z0-9_]*)(?:\}|=([^\s]+))?"), + # Config key-value in YAML/properties style: key: value or key = value + "config_kv": re.compile(r"^\s*([a-zA-Z_][a-zA-Z0-9_.]+)\s*[=:]\s*(.+)$", re.MULTILINE), + # JVM-style flags: -Xmx47000m, -XX:+UseG1GC + "jvm_flag": re.compile(r"(-X[a-z]+\d*[a-zA-Z]*|-XX:[+\-]?\w+(?:=\S+)?)"), + # Markdown table rows with parameter-like content: | param | type | description | + "table_param": re.compile(r"\|\s*`?([a-zA-Z_][a-zA-Z0-9_-]*)`?\s*\|([^|]*)\|([^|]*)\|"), + # Setting/configuration references: "set X to Y", "configure X as Y" + "setting_reference": re.compile( + r"(?:set|configure|specify|use)\s+[`\"']?([a-zA-Z_][a-zA-Z0-9_-]*)[`\"']?\s+(?:to|as|=)\s+[`\"']?([^\s,`\"']+)", + re.IGNORECASE, + ), + # Parameter descriptions in lists: - `param`: description or * param — description + "list_param": re.compile(r"^\s*[-*]\s+`([a-zA-Z_][a-zA-Z0-9_-]*)`[:\s]+(.+)$", re.MULTILINE), + # Placeholder patterns like , [param], {param} in usage lines + "placeholder": re.compile(r"<([a-zA-Z_][a-zA-Z0-9_]*)>"), +} + +# Common patterns for all sources +COMMON_PATTERNS = { + # Numeric constraints: min=0, max=100 + "min_max": re.compile(r"(?:min|minimum)[=:]\s*([0-9.]+).*(?:max|maximum)[=:]\s*([0-9.]+)", re.IGNORECASE), + # Allowed values: choices=[a, b, c] or enum: [a, b, c] + "allowed_values": re.compile(r"(?:choices|enum|options)[=:]\s*\[([^\]]+)\]", re.IGNORECASE), + # Boolean flags: true/false, yes/no, 1/0 + "boolean": re.compile(r"(true|false|yes|no|1|0)", re.IGNORECASE), + # Numeric types: int, float, number + "numeric": re.compile(r"(int|integer|float|number|double)", re.IGNORECASE), + # String types: str, string, text + "string": re.compile(r"(str|string|text)", re.IGNORECASE), +} + + +def get_patterns(source_type: str) -> Dict[str, re.Pattern]: + """ + Get regex patterns for a specific source type. + + Args: + source_type: One of 'cli', 'python', 'api', 'docker' + + Returns: + Dictionary of compiled regex patterns + """ + patterns_map = { + "cli": CLI_PATTERNS, + "python": PYTHON_PATTERNS, + "api": API_PATTERNS, + "docker": DOCKER_PATTERNS, + "readme": README_PATTERNS, + } + return patterns_map.get(source_type.lower(), {}) + + +def match_pattern(text: str, pattern: re.Pattern, group_names: Optional[List[str]] = None) -> List[Dict[str, str]]: + """ + Match a pattern against text and return structured results. + + Args: + text: Text to search + pattern: Compiled regex pattern + group_names: Optional names for capture groups + + Returns: + List of dictionaries with match information + """ + matches = [] + for match in pattern.finditer(text): + groups = match.groups() + if group_names and len(group_names) == len(groups): + matches.append(dict(zip(group_names, groups))) + else: + matches.append({"match": match.group(0), "groups": groups}) + return matches + diff --git a/src/kgpipe_parameters/extraction/utils.py b/src/kgpipe_parameters/extraction/utils.py new file mode 100644 index 0000000..1993bf2 --- /dev/null +++ b/src/kgpipe_parameters/extraction/utils.py @@ -0,0 +1,237 @@ +""" +Utility functions for parameter extraction and conversion. +""" + +import re +from typing import Optional, Union, List, Any, Dict +from .models import RawParameter +from kgpipe.common.model.configuration import Parameter, ParameterType + + +def infer_parameter_type(type_hint: Optional[str], default_value: Any = None) -> ParameterType: + """ + Infer ParameterType from type hint string or default value. + + Args: + type_hint: Type hint string (e.g., "int", "float", "str", "bool") + default_value: Default value to infer type from if type_hint is None + + Returns: + ParameterType enum value + """ + if type_hint: + type_hint_lower = type_hint.lower().strip() + + # Check for boolean + if any(x in type_hint_lower for x in ["bool", "boolean"]): + return ParameterType.boolean + + # Check for integer + if any(x in type_hint_lower for x in ["int", "integer"]): + return ParameterType.integer + + # Check for float/number + if any(x in type_hint_lower for x in ["float", "number", "double", "decimal"]): + return ParameterType.number + + # Check for array/list + if any(x in type_hint_lower for x in ["list", "array", "[]", "List"]): + return ParameterType.array + + # Check for object/dict + if any(x in type_hint_lower for x in ["dict", "object", "Dict", "{}"]): + return ParameterType.object + + # Check for enum + if "enum" in type_hint_lower or "choice" in type_hint_lower: + return ParameterType.enum + + # Infer from default value + if default_value is not None: + if isinstance(default_value, bool): + return ParameterType.boolean + elif isinstance(default_value, int): + return ParameterType.integer + elif isinstance(default_value, float): + return ParameterType.number + elif isinstance(default_value, list): + return ParameterType.array + elif isinstance(default_value, dict): + return ParameterType.object + + # Default to string + return ParameterType.string + + +def parse_default_value(value_str: Optional[str]) -> Optional[Union[str, int, float, bool]]: + """ + Parse a default value string into appropriate Python type. + + Args: + value_str: String representation of default value + + Returns: + Parsed value (str, int, float, or bool) or None + """ + if value_str is None: + return None + + value_str = value_str.strip().strip('"').strip("'") + + # Try integer first (before boolean, so "0" and "1" stay numeric) + try: + if value_str.isdigit() or (value_str.startswith("-") and value_str[1:].isdigit()): + return int(value_str) + except ValueError: + pass + + # Try boolean + if value_str.lower() in ["true", "false", "yes", "no"]: + return value_str.lower() in ["true", "yes"] + + # Try float + try: + return float(value_str) + except ValueError: + pass + + # Return as string + return value_str + + +def normalize_parameter_name(name: str) -> str: + """ + Normalize parameter name to a standard format. + + Args: + name: Original parameter name (may include --, -, etc.) + + Returns: + Normalized name (lowercase, underscores instead of hyphens) + """ + # Remove leading dashes and spaces + name = name.lstrip("-").lstrip() + + # Replace hyphens with underscores + name = name.replace("-", "_") + + # Convert to lowercase + name = name.lower() + + # Remove special characters except underscores + name = re.sub(r"[^a-z0-9_]", "", name) + + return name + + +def extract_constraints(description: Optional[str], type_hint: Optional[str] = None) -> Dict[str, Any]: + """ + Extract constraints (min, max, allowed_values) from description or type hint. + + Args: + description: Parameter description text + type_hint: Type hint string + + Returns: + Dictionary with constraint information + """ + constraints = {} + + if not description: + return constraints + + # Extract min/max values - try combined first, then separate + min_max_pattern = re.compile(r"(?:min|minimum)[=:]\s*([0-9.]+).*(?:max|maximum)[=:]\s*([0-9.]+)", re.IGNORECASE) + min_max_match = min_max_pattern.search(description) + if min_max_match: + constraints["minimum"] = float(min_max_match.group(1)) + constraints["maximum"] = float(min_max_match.group(2)) + + # Try separate min and max (even if combined pattern didn't match) + # More flexible pattern to handle "Minimum value: 10" or "min: 10" formats + min_pattern = re.compile(r"(?:min|minimum)(?:\s+value)?[=:]\s*([0-9.]+)", re.IGNORECASE) + max_pattern = re.compile(r"(?:max|maximum)(?:\s+value)?[=:]\s*([0-9.]+)", re.IGNORECASE) + min_match = min_pattern.search(description) + max_match = max_pattern.search(description) + if min_match and "minimum" not in constraints: + constraints["minimum"] = float(min_match.group(1)) + if max_match and "maximum" not in constraints: + constraints["maximum"] = float(max_match.group(1)) + + # Extract allowed values / choices + choices_pattern = re.compile(r"(?:choices|enum|options|allowed)[=:]\s*\[([^\]]+)\]", re.IGNORECASE) + choices_match = choices_pattern.search(description) + if choices_match: + choices_str = choices_match.group(1) + # Split by comma and clean up + choices = [c.strip().strip('"').strip("'") for c in choices_str.split(",")] + constraints["allowed_values"] = choices + + return constraints + + +def to_parameter_model(raw_param: RawParameter) -> Parameter: + """ + Convert a RawParameter to a Parameter model. + + Args: + raw_param: RawParameter instance + + Returns: + Parameter model instance + """ + # Infer parameter type + param_type = infer_parameter_type(raw_param.type_hint, raw_param.default_value) + + # Parse default value + default_val = raw_param.default_value + if isinstance(default_val, str): + default_val = parse_default_value(default_val) + + # Ensure default value matches the inferred type + if default_val is None: + # Set appropriate default based on type + if param_type == ParameterType.boolean: + default_val = False + elif param_type == ParameterType.integer: + default_val = 0 + elif param_type == ParameterType.number: + default_val = 0.0 + elif param_type == ParameterType.string: + default_val = "" + elif param_type == ParameterType.array: + default_val = [] + elif param_type == ParameterType.object: + default_val = {} + + # Extract constraints + constraints = extract_constraints(raw_param.description, raw_param.type_hint) + constraints.update(raw_param.constraints) + + # Get allowed values + allowed_values = constraints.get("allowed_values", []) + if allowed_values: + # Convert to appropriate types + typed_allowed = [] + for val in allowed_values: + parsed = parse_default_value(str(val)) + typed_allowed.append(parsed if parsed is not None else str(val)) + allowed_values = typed_allowed + + # Ensure native_keys includes the name + native_keys = list(raw_param.native_keys) + if raw_param.name not in native_keys: + native_keys.insert(0, raw_param.name) + + return Parameter( + name=raw_param.name, + native_keys=native_keys, + datatype=param_type, + default_value=default_val, + required=raw_param.required, + allowed_values=allowed_values, + minimum=constraints.get("minimum"), + maximum=constraints.get("maximum"), + unit=constraints.get("unit"), + ) + diff --git a/src/kgpipe_parameters/tests/__init__.py b/src/kgpipe_parameters/tests/__init__.py new file mode 100644 index 0000000..c5a603a --- /dev/null +++ b/src/kgpipe_parameters/tests/__init__.py @@ -0,0 +1,4 @@ +""" +Tests for parameter extraction module. +""" + diff --git a/src/kgpipe_parameters/tests/conftest.py b/src/kgpipe_parameters/tests/conftest.py new file mode 100644 index 0000000..5c4004f --- /dev/null +++ b/src/kgpipe_parameters/tests/conftest.py @@ -0,0 +1,131 @@ +""" +Pytest fixtures for parameter extraction tests. +""" + +import pytest +from pathlib import Path +from unittest.mock import Mock, MagicMock +from typing import Dict, Any + + +def get_test_data_path(relative_path: str) -> Path: + """Get path to test data file.""" + test_dir = Path(__file__).parent + path = test_dir / "test_data" / relative_path + if not path.exists(): + raise FileNotFoundError(f"Test data path {path} does not exist") + return path + + +@pytest.fixture +def test_data_dir(): + """Fixture for test data directory.""" + return Path(__file__).parent / "test_data" + + +@pytest.fixture +def cli_help_argparse(): + """Fixture for argparse CLI help text.""" + path = get_test_data_path("cli/argparse_help.txt") + return path.read_text() + + +@pytest.fixture +def cli_help_click(): + """Fixture for click CLI help text.""" + path = get_test_data_path("cli/click_help.txt") + return path.read_text() + + +@pytest.fixture +def cli_help_simple(): + """Fixture for simple CLI help text.""" + path = get_test_data_path("cli/simple_help.txt") + return path.read_text() + + +@pytest.fixture +def python_function_code(): + """Fixture for Python function code.""" + path = get_test_data_path("python/function_with_params.py") + return path.read_text() + + +@pytest.fixture +def python_dataclass_code(): + """Fixture for Python dataclass code.""" + path = get_test_data_path("python/dataclass_config.py") + return path.read_text() + + +@pytest.fixture +def python_pydantic_code(): + """Fixture for Python Pydantic model code.""" + path = get_test_data_path("python/pydantic_model.py") + return path.read_text() + + +@pytest.fixture +def openapi_spec(): + """Fixture for OpenAPI specification.""" + path = get_test_data_path("api/openapi_spec.yaml") + return path.read_text() + + +@pytest.fixture +def swagger_spec(): + """Fixture for Swagger specification.""" + path = get_test_data_path("api/swagger_spec.json") + return path.read_text() + + +@pytest.fixture +def dockerfile_content(): + """Fixture for Dockerfile content.""" + path = get_test_data_path("docker/Dockerfile") + return path.read_text() + + +@pytest.fixture +def docker_compose_content(): + """Fixture for docker-compose.yml content.""" + path = get_test_data_path("docker/docker-compose.yml") + return path.read_text() + + +@pytest.fixture +def readme_tool_doc(): + """Fixture for a tool README with configuration parameters.""" + path = get_test_data_path("readme/tool_readme.md") + return path.read_text() + + +@pytest.fixture +def readme_minimal(): + """Fixture for a minimal README.""" + path = get_test_data_path("readme/minimal_readme.md") + return path.read_text() + + +@pytest.fixture +def mock_llm_client(): + """Fixture for mocked LLM client.""" + mock_client = Mock() + + # Mock response structure + mock_response = { + "parameters": [ + { + "name": "threshold", + "native_keys": ["--threshold", "-t"], + "description": "Matching threshold", + "type_hint": "float", + "default_value": 0.5, + "required": False + } + ] + } + + mock_client.send_prompt = Mock(return_value=mock_response) + return mock_client + diff --git a/src/kgpipe_parameters/tests/test_chunk_filter.py b/src/kgpipe_parameters/tests/test_chunk_filter.py new file mode 100644 index 0000000..a8dc404 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_chunk_filter.py @@ -0,0 +1,263 @@ +""" +Tests for keyword-based chunk scoring / filtering. +""" + +import pytest + +from kgpipe_parameters.extraction.chunk_filter import ( + score_chunk, + has_parameter_signals, + KEYWORD_SETS, + _detect_language, +) + + +# ── Language detection ────────────────────────────────────────────────── + +class TestLanguageDetection: + """Tests for _detect_language helper.""" + + def test_python_extension(self): + assert _detect_language("src/foo/bar.py") == "python" + + def test_java_extension(self): + assert _detect_language("src/Main.java") == "java" + + def test_properties_extension(self): + assert _detect_language("conf/server.properties") == "properties" + + def test_xml_extension(self): + assert _detect_language("config.xml") == "xml" + + def test_dockerfile(self): + assert _detect_language("Dockerfile") == "docker" + assert _detect_language("docker-compose.yml") == "docker" + + def test_readme(self): + assert _detect_language("README.md") == "readme" + assert _detect_language("INSTALL.txt") == "readme" + + def test_unknown_defaults_to_generic(self): + assert _detect_language("random.xyz") == "generic" + assert _detect_language(None) == "generic" + + +# ── Scoring ───────────────────────────────────────────────────────────── + +class TestScoreChunk: + """Tests for score_chunk.""" + + def test_empty_text(self): + score, matched = score_chunk("") + assert score == 0 + assert matched == [] + + def test_python_argparse(self): + code = ''' +import argparse +parser = argparse.ArgumentParser() +parser.add_argument("--threshold", type=float, default=0.5, help="Matching threshold") +''' + score, matched = score_chunk(code, file_path="cli.py") + assert score >= 3 # argparse, add_argument, default=, type=, help= + assert "argparse" in matched + assert "add_argument" in matched + + def test_python_dataclass(self): + code = ''' +from dataclasses import dataclass, field + +@dataclass +class Config: + threshold: float = 0.5 + batch_size: int = Field(default=32) +''' + score, matched = score_chunk(code, file_path="config.py") + assert score >= 2 + assert "@dataclass" in matched + assert "Field(" in matched + + def test_python_no_signals(self): + code = ''' +def compute_arabic_segmenter(text): + tokens = text.split() + return [t for t in tokens if len(t) > 2] +''' + score, matched = score_chunk(code, file_path="segmenter.py") + assert score < 2 # No real parameter signals + + def test_java_option_annotation(self): + code = ''' +public class RunPARIS { + @Option(name = "-n", usage = "number of iterations") + int numIterations = 10; + + @Option(name = "-t", usage = "threshold") + double threshold = 0.5; +} +''' + score, matched = score_chunk(code, file_path="RunPARIS.java") + assert score >= 1 # @Option is the signal; Java threshold is 1 + assert "@Option" in matched + # The file still passes the filter (Java auto-lowers threshold to 1) + assert has_parameter_signals(code, file_path="RunPARIS.java") is True + + def test_java_properties_access(self): + code = ''' +Properties props = new Properties(); +props.load(new FileInputStream("config.properties")); +String value = props.getProperty("matchThreshold"); +int maxIter = Integer.parseInt(props.getProperty("maxIterations")); +''' + score, matched = score_chunk(code, file_path="Config.java") + assert score >= 2 + assert "getProperty(" in matched + assert "Properties" in matched + + def test_java_no_signals(self): + code = ''' +public class ArabicTokenizer { + public List tokenize(String text) { + return Arrays.asList(text.split(" ")); + } +} +''' + score, matched = score_chunk(code, file_path="ArabicTokenizer.java") + assert score < 2 + + def test_properties_file(self): + content = ''' +# Server configuration +server.port=8080 +matching.threshold=0.5 +max.iterations=100 +''' + score, matched = score_chunk(content, file_path="server.properties") + assert score >= 1 # .properties files have low bar + + def test_xml_config(self): + content = ''' + + + + +''' + score, matched = score_chunk(content, file_path="config.xml") + assert score >= 2 + assert "= 3 + assert "ENV " in matched + assert "ARG " in matched + assert "EXPOSE " in matched + + def test_readme_with_params(self): + content = ''' +# My Tool + +## Usage + +```bash +mytool --threshold 0.5 --output result.txt +``` + +## Configuration + +- `threshold`: Matching threshold (default: 0.5) +- `max_iter`: Maximum iterations (default: 100) +''' + score, matched = score_chunk(content, file_path="README.md") + assert score >= 3 + + def test_readme_no_params(self): + content = ''' +# My Project + +This is a library for natural language processing. + +## License + +MIT License +''' + score, matched = score_chunk(content, file_path="README.md") + # Very few or no config signals + assert score <= 2 + + def test_explicit_language_override(self): + code = "parser.add_argument('--foo')" + score, matched = score_chunk(code, language="python") + assert "add_argument" in matched + + +# ── has_parameter_signals ─────────────────────────────────────────────── + +class TestHasParameterSignals: + """Tests for the boolean filter function.""" + + def test_python_with_signals(self): + code = 'parser = argparse.ArgumentParser()\nparser.add_argument("--x", default=5)' + assert has_parameter_signals(code, file_path="cli.py") is True + + def test_python_without_signals(self): + code = "x = 1 + 2\nprint(x)" + assert has_parameter_signals(code, file_path="math.py") is False + + def test_threshold_override(self): + code = "argparse" + # With default threshold=2 this would fail (only 1 keyword) + assert has_parameter_signals(code, file_path="x.py", threshold=2) is False + # With threshold=1 it passes + assert has_parameter_signals(code, file_path="x.py", threshold=1) is True + + def test_properties_low_bar(self): + content = "key=value" + # .properties files auto-lower threshold to 1 + assert has_parameter_signals(content, file_path="app.properties") is True + + def test_xml_low_bar(self): + content = '' + assert has_parameter_signals(content, file_path="config.xml") is True + + def test_java_config_class_passes(self): + code = ''' +public class AppConfig { + @Option(name = "-t") + double threshold = DEFAULT_THRESHOLD; +} +''' + assert has_parameter_signals(code, file_path="AppConfig.java") is True + + def test_java_non_config_class_fails(self): + code = ''' +public class Utils { + public static String trim(String s) { + return s.trim(); + } +} +''' + assert has_parameter_signals(code, file_path="Utils.java") is False + + +# ── Keyword set sanity ────────────────────────────────────────────────── + +class TestKeywordSets: + """Sanity checks on the keyword dictionaries.""" + + def test_all_sets_non_empty(self): + for name, kws in KEYWORD_SETS.items(): + assert len(kws) > 0, f"Keyword set '{name}' is empty" + + def test_no_empty_keywords(self): + for name, kws in KEYWORD_SETS.items(): + for kw in kws: + assert kw.strip() != "", f"Empty keyword in set '{name}'" + diff --git a/src/kgpipe_parameters/tests/test_clustering.py b/src/kgpipe_parameters/tests/test_clustering.py new file mode 100644 index 0000000..1839ad8 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_clustering.py @@ -0,0 +1,324 @@ +""" +Tests for the parameter clustering module. +""" + +import json +import pytest +import numpy as np +from pathlib import Path +from unittest.mock import patch, MagicMock +from typing import List + +from kgpipe_parameters.clustering.models import ( + ParameterVector, + ParameterCluster, + ClusteringResult, +) +from kgpipe_parameters.clustering.similarity import ( + embed_parameters, + cosine_similarity_matrix, +) +from kgpipe_parameters.clustering.clusterer import ParameterClusterer + + +# ============================================================================ +# Fixtures +# ============================================================================ + + +def _make_param( + name: str, + tool: str, + description: str = "", + native_keys: List[str] | None = None, + type_hint: str | None = None, + default_value=None, +) -> ParameterVector: + return ParameterVector( + name=name, + tool_name=tool, + native_keys=native_keys or [], + description=description, + type_hint=type_hint, + default_value=default_value, + source_label=f"{tool}/source", + ) + + +@pytest.fixture +def sample_parameters() -> List[ParameterVector]: + """A small set of parameters from two fictitious tools.""" + return [ + # Tool A + _make_param("threshold", "tool_a", "Matching threshold value", ["--threshold", "-t"], "float", 0.5), + _make_param("max_iterations", "tool_a", "Maximum number of iterations", ["--max-iter"], "int", 100), + _make_param("output_dir", "tool_a", "Output directory path", ["--output", "-o"], "str"), + _make_param("batch_size", "tool_a", "Number of items per batch", ["--batch-size"], "int", 32), + # Tool B + _make_param("similarity_threshold", "tool_b", "Threshold for similarity matching", ["--sim-threshold"], "float", 0.7), + _make_param("iterations", "tool_b", "Number of iterations to run", ["--iterations", "-n"], "int", 50), + _make_param("output_path", "tool_b", "Path for output files", ["--output-path"], "str"), + _make_param("learning_rate", "tool_b", "Learning rate for optimizer", ["--lr"], "float", 0.001), + ] + + +@pytest.fixture +def mock_sentence_model(): + """A mock SentenceTransformer that returns deterministic embeddings.""" + model = MagicMock() + # Return embeddings designed so that similar parameters are closer. + # Each "encode" call gets a list of texts; we return a (N, 8) array + # seeded from the text hash so it is deterministic. + + def _encode(texts, batch_size=64, show_progress_bar=False): + rng = np.random.RandomState(42) + # Use a small embedding dim for testing speed + embs = [] + for t in texts: + seed = sum(ord(c) for c in t) % 2**31 + r = np.random.RandomState(seed) + embs.append(r.randn(8).astype(np.float32)) + return np.array(embs) + + model.encode = _encode + return model + + +# ============================================================================ +# ParameterVector tests +# ============================================================================ + + +class TestParameterVector: + def test_text_for_embedding_basic(self): + pv = _make_param("threshold", "t", "matching threshold", ["--threshold"]) + text = pv.text_for_embedding() + assert "threshold" in text + assert "matching threshold" in text + assert "--threshold" in text + + def test_text_for_embedding_minimal(self): + pv = _make_param("x", "t") + text = pv.text_for_embedding() + assert "x" in text + + +# ============================================================================ +# ParameterCluster tests +# ============================================================================ + + +class TestParameterCluster: + def test_size(self): + members = [_make_param("a", "t1"), _make_param("b", "t2")] + cluster = ParameterCluster(cluster_id=0, label="a", members=members, tools=["t1", "t2"]) + assert cluster.size() == 2 + + def test_is_cross_tool(self): + c1 = ParameterCluster(cluster_id=0, label="x", members=[], tools=["t1", "t2"]) + assert c1.is_cross_tool() + + c2 = ParameterCluster(cluster_id=1, label="x", members=[], tools=["t1"]) + assert not c2.is_cross_tool() + + +# ============================================================================ +# ClusteringResult tests +# ============================================================================ + + +class TestClusteringResult: + def test_cross_tool_clusters(self): + c1 = ParameterCluster(cluster_id=0, label="x", members=[], tools=["t1", "t2"]) + c2 = ParameterCluster(cluster_id=1, label="y", members=[], tools=["t1"]) + result = ClusteringResult(clusters=[c1, c2], n_clusters=2) + assert len(result.cross_tool_clusters()) == 1 + + def test_to_table_rows(self): + members = [_make_param("threshold", "t1"), _make_param("threshold", "t2")] + cluster = ParameterCluster(cluster_id=0, label="threshold", members=members, tools=["t1", "t2"]) + result = ClusteringResult(clusters=[cluster], n_clusters=1, n_parameters=2) + rows = result.to_table_rows() + assert len(rows) == 2 + assert rows[0]["cluster_label"] == "threshold" + assert rows[0]["tool"] == "t1" + assert rows[1]["tool"] == "t2" + + def test_to_table_rows_empty(self): + result = ClusteringResult() + assert result.to_table_rows() == [] + + +# ============================================================================ +# Similarity tests +# ============================================================================ + + +class TestSimilarity: + def test_embed_parameters(self, sample_parameters, mock_sentence_model): + embeddings = embed_parameters( + sample_parameters, model=mock_sentence_model + ) + assert embeddings.shape[0] == len(sample_parameters) + assert embeddings.shape[1] > 0 + # All embeddings should be stored back + for pv in sample_parameters: + assert pv.embedding is not None + assert len(pv.embedding) == embeddings.shape[1] + + def test_embed_parameters_empty(self, mock_sentence_model): + embeddings = embed_parameters([], model=mock_sentence_model) + assert embeddings.shape == (0, 0) + + def test_cosine_similarity_matrix_identity(self): + embs = np.eye(3, dtype=np.float32) + sim = cosine_similarity_matrix(embs) + np.testing.assert_allclose(sim, np.eye(3), atol=1e-5) + + def test_cosine_similarity_matrix_same_vector(self): + embs = np.ones((4, 5), dtype=np.float32) + sim = cosine_similarity_matrix(embs) + np.testing.assert_allclose(sim, np.ones((4, 4)), atol=1e-5) + + def test_cosine_similarity_matrix_empty(self): + embs = np.empty((0, 0)) + sim = cosine_similarity_matrix(embs) + assert sim.shape == (0, 0) + + +# ============================================================================ +# ParameterClusterer tests +# ============================================================================ + + +class TestParameterClusterer: + def test_cluster_basic(self, sample_parameters, mock_sentence_model): + """Clustering should produce at least one cluster.""" + clusterer = ParameterClusterer(distance_threshold=0.8) + clusterer._model = mock_sentence_model + + result = clusterer.cluster(sample_parameters) + assert result.n_parameters == len(sample_parameters) + assert result.n_clusters > 0 + # All parameters should be assigned to some cluster + total_members = sum(c.size() for c in result.clusters) + assert total_members == len(sample_parameters) + + def test_cluster_empty(self): + clusterer = ParameterClusterer() + result = clusterer.cluster([]) + assert result.n_parameters == 0 + assert result.n_clusters == 0 + + def test_cluster_single_param(self, mock_sentence_model): + clusterer = ParameterClusterer() + clusterer._model = mock_sentence_model + params = [_make_param("threshold", "tool_a", "test")] + result = clusterer.cluster(params) + assert result.n_parameters == 1 + assert result.n_clusters == 1 + + def test_load_parameters_from_json(self, tmp_path): + """Test loading parameters from a tool JSON output file.""" + data = { + "tool_name": "test_tool", + "parameters": [ + { + "name": "threshold", + "native_keys": ["--threshold"], + "description": "test", + "type_hint": "float", + "default_value": 0.5, + "required": False, + "_source": "cli", + }, + { + "name": "output", + "native_keys": ["--output"], + "description": "output path", + "_source": "cli", + }, + ], + } + json_file = tmp_path / "test_tool.json" + json_file.write_text(json.dumps(data)) + + params = ParameterClusterer.load_parameters_from_json(json_file) + assert len(params) == 2 + assert params[0].name == "threshold" + assert params[0].tool_name == "test_tool" + assert params[1].name == "output" + + def test_load_from_output_dir(self, tmp_path): + """Test loading from a directory with multiple tool files.""" + for tool_name in ["tool_a", "tool_b"]: + data = { + "tool_name": tool_name, + "parameters": [ + {"name": "param1", "native_keys": [], "_source": "cli"}, + ], + } + (tmp_path / f"{tool_name}.json").write_text(json.dumps(data)) + # Summary file should be skipped + (tmp_path / "_summary.json").write_text("{}") + + clusterer = ParameterClusterer() + params = clusterer.load_from_output_dir(tmp_path) + assert len(params) == 2 + tool_names = {p.tool_name for p in params} + assert tool_names == {"tool_a", "tool_b"} + + def test_save_result(self, tmp_path): + members = [_make_param("threshold", "t1")] + cluster = ParameterCluster(cluster_id=0, label="threshold", members=members, tools=["t1"]) + result = ClusteringResult(clusters=[cluster], n_clusters=1, n_parameters=1) + + out_path = tmp_path / "clusters.json" + ParameterClusterer.save_result(result, out_path) + assert out_path.exists() + + saved = json.loads(out_path.read_text()) + assert saved["n_clusters"] == 1 + assert len(saved["clusters"]) == 1 + + def test_save_table(self, tmp_path): + members = [ + _make_param("threshold", "t1", description="test"), + _make_param("threshold", "t2", description="test"), + ] + cluster = ParameterCluster(cluster_id=0, label="threshold", members=members, tools=["t1", "t2"]) + result = ClusteringResult(clusters=[cluster], n_clusters=1, n_parameters=2) + + csv_path = tmp_path / "table.csv" + ParameterClusterer.save_table(result, csv_path) + assert csv_path.exists() + + import csv + with open(csv_path) as f: + reader = csv.DictReader(f) + rows = list(reader) + assert len(rows) == 2 + assert rows[0]["parameter"] == "threshold" + + def test_cluster_from_output_dir(self, tmp_path, mock_sentence_model): + """Integration test: load → cluster from an output directory.""" + for tool_name, params in [ + ("tool_a", [ + {"name": "threshold", "native_keys": ["--threshold"], "description": "match threshold", "_source": "cli"}, + {"name": "output", "native_keys": ["--output"], "description": "output path", "_source": "cli"}, + ]), + ("tool_b", [ + {"name": "similarity_threshold", "native_keys": ["--sim-threshold"], "description": "threshold for similarity", "_source": "cli"}, + {"name": "output_dir", "native_keys": ["--output-dir"], "description": "directory for output", "_source": "cli"}, + ]), + ]: + data = {"tool_name": tool_name, "parameters": params} + (tmp_path / f"{tool_name}.json").write_text(json.dumps(data)) + + clusterer = ParameterClusterer(distance_threshold=0.8) + clusterer._model = mock_sentence_model + result = clusterer.cluster_from_output_dir(tmp_path) + + assert result.n_parameters == 4 + assert result.n_clusters > 0 + diff --git a/src/kgpipe_parameters/tests/test_data/api/openapi_spec.yaml b/src/kgpipe_parameters/tests/test_data/api/openapi_spec.yaml new file mode 100644 index 0000000..cc3b967 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/api/openapi_spec.yaml @@ -0,0 +1,42 @@ +openapi: 3.0.0 +info: + title: Matching API + version: 1.0.0 +paths: + /api/match: + post: + summary: Match entities + parameters: + - name: threshold + in: query + schema: + type: number + default: 0.5 + minimum: 0.0 + maximum: 1.0 + description: Matching threshold + - name: max_results + in: query + schema: + type: integer + default: 100 + description: Maximum number of results + requestBody: + content: + application/json: + schema: + type: object + required: + - input_file + properties: + input_file: + type: string + description: Input file path + output_file: + type: string + description: Output file path + verbose: + type: boolean + default: false + description: Enable verbose logging + diff --git a/src/kgpipe_parameters/tests/test_data/api/swagger_spec.json b/src/kgpipe_parameters/tests/test_data/api/swagger_spec.json new file mode 100644 index 0000000..899f978 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/api/swagger_spec.json @@ -0,0 +1,43 @@ +{ + "swagger": "2.0", + "info": { + "title": "Matching API", + "version": "1.0.0" + }, + "paths": { + "/api/match": { + "post": { + "parameters": [ + { + "name": "threshold", + "in": "query", + "type": "number", + "default": 0.5, + "minimum": 0.0, + "maximum": 1.0, + "description": "Matching threshold" + }, + { + "name": "input_file", + "in": "body", + "schema": { + "type": "object", + "required": ["input_file"], + "properties": { + "input_file": { + "type": "string", + "description": "Input file path" + }, + "output_file": { + "type": "string", + "description": "Output file path" + } + } + } + } + ] + } + } + } +} + diff --git a/src/kgpipe_parameters/tests/test_data/cli/argparse_help.txt b/src/kgpipe_parameters/tests/test_data/cli/argparse_help.txt new file mode 100644 index 0000000..e958f50 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/cli/argparse_help.txt @@ -0,0 +1,8 @@ +usage: tool.py [-h] [--threshold THRESHOLD] [--output OUTPUT] [--verbose] + +optional arguments: + -h, --help show this help message and exit + --threshold THRESHOLD Matching threshold (default: 0.5) + --output OUTPUT Output file path (required) + --verbose Enable verbose logging + diff --git a/src/kgpipe_parameters/tests/test_data/cli/click_help.txt b/src/kgpipe_parameters/tests/test_data/cli/click_help.txt new file mode 100644 index 0000000..8e9d948 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/cli/click_help.txt @@ -0,0 +1,8 @@ +Usage: tool.py [OPTIONS] + +Options: + --threshold FLOAT Matching threshold [default: 0.5] + --output TEXT Output file path (required) + --verbose Enable verbose logging + --help Show this message and exit. + diff --git a/src/kgpipe_parameters/tests/test_data/cli/simple_help.txt b/src/kgpipe_parameters/tests/test_data/cli/simple_help.txt new file mode 100644 index 0000000..bf078f9 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/cli/simple_help.txt @@ -0,0 +1,8 @@ +Usage: matcher [OPTIONS] + + --threshold VALUE Matching threshold (0.0-1.0) [default: 0.5] + --input FILE Input file path (required) + --output FILE Output file path + --max-results INT Maximum number of results [default: 100] + --verbose Enable verbose output + diff --git a/src/kgpipe_parameters/tests/test_data/docker/Dockerfile b/src/kgpipe_parameters/tests/test_data/docker/Dockerfile new file mode 100644 index 0000000..1a99c0b --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/docker/Dockerfile @@ -0,0 +1,14 @@ +FROM python:3.9 + +ARG BUILD_VERSION=latest +ARG THRESHOLD=0.5 + +ENV THRESHOLD=${THRESHOLD} +ENV OUTPUT_DIR=/output +ENV MAX_RESULTS=100 +ENV VERBOSE=false + +WORKDIR /app +COPY . . +CMD ["python", "app.py"] + diff --git a/src/kgpipe_parameters/tests/test_data/docker/docker-compose.yml b/src/kgpipe_parameters/tests/test_data/docker/docker-compose.yml new file mode 100644 index 0000000..3ecdc59 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/docker/docker-compose.yml @@ -0,0 +1,23 @@ +version: '3.8' + +services: + matcher: + image: matcher:latest + environment: + THRESHOLD: 0.5 + OUTPUT_DIR: /output + MAX_RESULTS: 100 + VERBOSE: "false" + volumes: + - ./data:/data + ports: + - "8080:8080" + + processor: + image: processor:latest + environment: + INPUT_DIR: /input + BATCH_SIZE: 50 + depends_on: + - matcher + diff --git a/src/kgpipe_parameters/tests/test_data/python/dataclass_config.py b/src/kgpipe_parameters/tests/test_data/python/dataclass_config.py new file mode 100644 index 0000000..716c2a1 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/python/dataclass_config.py @@ -0,0 +1,12 @@ +from dataclasses import dataclass +from typing import Optional + +@dataclass +class MatchingConfig: + """Configuration for matching operations.""" + threshold: float = 0.5 + input_file: str + output_file: Optional[str] = None + verbose: bool = False + max_results: int = 100 + diff --git a/src/kgpipe_parameters/tests/test_data/python/function_with_params.py b/src/kgpipe_parameters/tests/test_data/python/function_with_params.py new file mode 100644 index 0000000..be93fce --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/python/function_with_params.py @@ -0,0 +1,16 @@ +def process_data( + input_file: str, + threshold: float = 0.5, + verbose: bool = False, + max_results: int = 100 +) -> None: + """ + Process data with configurable parameters. + + :param input_file: Path to input file (required) + :param threshold: Matching threshold (default: 0.5, min: 0.0, max: 1.0) + :param verbose: Enable verbose logging + :param max_results: Maximum number of results (default: 100) + """ + pass + diff --git a/src/kgpipe_parameters/tests/test_data/python/pydantic_model.py b/src/kgpipe_parameters/tests/test_data/python/pydantic_model.py new file mode 100644 index 0000000..75a27bc --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/python/pydantic_model.py @@ -0,0 +1,11 @@ +from pydantic import BaseModel, Field +from typing import Optional + +class MatchingConfig(BaseModel): + """Configuration for matching operations.""" + threshold: float = Field(default=0.5, ge=0.0, le=1.0, description="Matching threshold") + input_file: str = Field(..., description="Input file path (required)") + output_file: Optional[str] = Field(default=None, description="Output file path") + verbose: bool = Field(default=False, description="Enable verbose logging") + max_results: int = Field(default=100, ge=1, description="Maximum number of results") + diff --git a/src/kgpipe_parameters/tests/test_data/readme/minimal_readme.md b/src/kgpipe_parameters/tests/test_data/readme/minimal_readme.md new file mode 100644 index 0000000..8f37aa4 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/readme/minimal_readme.md @@ -0,0 +1,12 @@ +# SimpleTool + +A minimal tool. + +## Usage + +``` +simpletool +``` + +Set `workers` to control parallelism. + diff --git a/src/kgpipe_parameters/tests/test_data/readme/tool_readme.md b/src/kgpipe_parameters/tests/test_data/readme/tool_readme.md new file mode 100644 index 0000000..ef86be5 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_data/readme/tool_readme.md @@ -0,0 +1,58 @@ +# EntityMatcher + +A tool for matching entities across knowledge graphs. + +## Installation + +```bash +pip install entity-matcher +``` + +## Usage + +```bash +entity-matcher --input data.nt --output results.tsv --threshold 0.8 --max-iter 10 +entity-matcher --format csv --verbose +``` + +## Configuration + +The following parameters can be set: + +- `threshold`: The matching threshold, a float between 0 and 1 (default: 0.5) +- `max_iter`: Maximum number of iterations (default: 10) +- `input`: Path to input knowledge base (required) +- `output`: Path to output results file (required) +- `format`: Output format, one of csv, tsv, json (default: tsv) +- `similarity_metric`: Similarity metric to use, e.g. jaccard, cosine (default: jaccard) + +## Advanced Configuration + +| Parameter | Type | Description | +|-----------|------|-------------| +| `batch_size` | int | Number of entities per batch | +| `num_threads` | int | Number of parallel threads | +| `cache_dir` | path | Directory for caching intermediate results | +| `log_level` | string | Logging level: DEBUG, INFO, WARNING, ERROR | + +## Environment Variables + +You can also configure via environment: + +```bash +export MATCHER_THRESHOLD=0.8 +export MATCHER_MAX_MEMORY=4096 +``` + +## Running with Java Backend + +For the Java backend, you may need to increase JVM memory: + +```bash +java -Xmx8192m -Xms2048m -jar entity-matcher.jar +``` + +## API + +See the [API documentation](docs/api.md) for details. + diff --git a/src/kgpipe_parameters/tests/test_paramters_extraction.py b/src/kgpipe_parameters/tests/test_paramters_extraction.py new file mode 100644 index 0000000..e98966b --- /dev/null +++ b/src/kgpipe_parameters/tests/test_paramters_extraction.py @@ -0,0 +1,599 @@ +""" +Comprehensive tests for parameter extraction module. +""" + +import pytest +from pathlib import Path +from unittest.mock import Mock, patch + +from kgpipe_parameters.extraction import ( + ParameterMiner, + CLIExtractor, + PythonLibExtractor, + HTTPAPIExtractor, + DockerExtractor, + ReadmeDocExtractor, + RawParameter, + ExtractionResult, + SourceType, + ExtractionMethod, +) +from kgpipe_parameters.extraction.utils import ( + normalize_parameter_name, + parse_default_value, + infer_parameter_type, + extract_constraints, + to_parameter_model, +) +from kgpipe.common.model.configuration import Parameter, ParameterType + + +# ============================================================================= +# Utility Function Tests +# ============================================================================= + +class TestUtils: + """Tests for utility functions.""" + + def test_normalize_parameter_name(self): + """Test parameter name normalization.""" + assert normalize_parameter_name("--threshold") == "threshold" + assert normalize_parameter_name("-t") == "t" + assert normalize_parameter_name("threshold") == "threshold" + assert normalize_parameter_name("THRESHOLD") == "threshold" + assert normalize_parameter_name("max-results") == "max_results" + assert normalize_parameter_name("camelCase") == "camelcase" + + def test_parse_default_value(self): + """Test parsing of default values.""" + assert parse_default_value("0.5") == 0.5 + assert parse_default_value("100") == 100 + assert parse_default_value("true") is True + assert parse_default_value("false") is False + assert parse_default_value("yes") is True + assert parse_default_value("no") is False + assert parse_default_value("hello") == "hello" + assert parse_default_value('"hello"') == "hello" + assert parse_default_value("'world'") == "world" + assert parse_default_value(None) is None + + def test_infer_parameter_type(self): + """Test type inference from type hints and default values.""" + # From type hints + assert infer_parameter_type("int") == ParameterType.integer + assert infer_parameter_type("float") == ParameterType.number + assert infer_parameter_type("str") == ParameterType.string + assert infer_parameter_type("bool") == ParameterType.boolean + assert infer_parameter_type("List[str]") == ParameterType.array + assert infer_parameter_type("Dict[str, Any]") == ParameterType.object + assert infer_parameter_type("enum") == ParameterType.enum + + # From default values + assert infer_parameter_type(None, 42) == ParameterType.integer + assert infer_parameter_type(None, 3.14) == ParameterType.number + assert infer_parameter_type(None, "text") == ParameterType.string + assert infer_parameter_type(None, True) == ParameterType.boolean + assert infer_parameter_type(None, []) == ParameterType.array + assert infer_parameter_type(None, {}) == ParameterType.object + + # Default to string + assert infer_parameter_type(None, None) == ParameterType.string + + def test_extract_constraints(self): + """Test constraint extraction from descriptions.""" + desc1 = "Threshold value (min: 0.0, max: 1.0)" + constraints1 = extract_constraints(desc1) + assert constraints1["minimum"] == 0.0 + assert constraints1["maximum"] == 1.0 + + desc2 = "Choices: [option1, option2, option3]" + constraints2 = extract_constraints(desc2) + assert "allowed_values" in constraints2 + assert len(constraints2["allowed_values"]) == 3 + + desc3 = "Minimum value: 10" + constraints3 = extract_constraints(desc3) + assert constraints3["minimum"] == 10.0 + + desc4 = "Maximum value: 100" + constraints4 = extract_constraints(desc4) + assert constraints4["maximum"] == 100.0 + + def test_to_parameter_model(self): + """Test conversion from RawParameter to Parameter model.""" + raw_param = RawParameter( + name="threshold", + native_keys=["--threshold", "-t"], + description="Matching threshold (min: 0.0, max: 1.0)", + type_hint="float", + default_value=0.5, + required=False, + source="test", + ) + + param = to_parameter_model(raw_param) + + assert isinstance(param, Parameter) + assert param.name == "threshold" + assert "--threshold" in param.native_keys + assert param.datatype == ParameterType.number + assert param.default_value == 0.5 + assert param.required is False + assert param.minimum == 0.0 + assert param.maximum == 1.0 + + +# ============================================================================= +# CLI Extractor Tests +# ============================================================================= + +class TestCLIExtractor: + """Tests for CLI parameter extraction.""" + + def test_cli_extractor_basic(self, cli_help_simple): + """Test basic CLI parameter extraction.""" + extractor = CLIExtractor() + result = extractor.extract(cli_help_simple, "matcher") + + assert isinstance(result, ExtractionResult) + assert result.source_type == SourceType.CLI + assert result.extraction_method == ExtractionMethod.REGEX + assert len(result.parameters) > 0 + + # Check that threshold parameter was extracted + threshold_params = [p for p in result.parameters if "threshold" in p.name] + assert len(threshold_params) > 0 + + def test_cli_extractor_with_defaults(self, cli_help_argparse): + """Test extraction of parameters with default values.""" + extractor = CLIExtractor() + result = extractor.extract(cli_help_argparse, "tool") + + # Find threshold parameter with default + threshold_params = [p for p in result.parameters if "threshold" in p.name] + if threshold_params: + param = threshold_params[0] + assert param.default_value == 0.5 or param.default_value == "0.5" + + def test_cli_extractor_required_flags(self, cli_help_argparse): + """Test detection of required vs optional parameters.""" + extractor = CLIExtractor() + result = extractor.extract(cli_help_argparse, "tool") + + # Check for required parameters + output_params = [p for p in result.parameters if "output" in p.name] + if output_params: + # Output is marked as required in the test data + param = output_params[0] + # The extractor should detect "required" in description + + def test_cli_extractor_multiple_flags(self, cli_help_click): + """Test extraction of both long and short flags.""" + extractor = CLIExtractor() + result = extractor.extract(cli_help_click, "tool") + + # Check that parameters have native_keys + for param in result.parameters: + assert len(param.native_keys) > 0 + + def test_cli_extractor_description(self, cli_help_simple): + """Test extraction of parameter descriptions.""" + extractor = CLIExtractor() + result = extractor.extract(cli_help_simple, "matcher") + + # Check that descriptions are extracted + params_with_desc = [p for p in result.parameters if p.description] + assert len(params_with_desc) > 0 + + +# ============================================================================= +# Python Extractor Tests +# ============================================================================= + +class TestPythonExtractor: + """Tests for Python parameter extraction.""" + + def test_python_extractor_function_params(self, python_function_code): + """Test extraction from function signatures.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_function_code, "process_data") + + assert isinstance(result, ExtractionResult) + assert result.source_type == SourceType.PYTHON_LIB + assert len(result.parameters) > 0 + + # Check for expected parameters + param_names = [p.name for p in result.parameters] + assert "input_file" in param_names or "inputfile" in param_names + assert "threshold" in param_names + + def test_python_extractor_type_hints(self, python_function_code): + """Test extraction of type hints.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_function_code, "process_data") + + # Check that type hints are extracted + params_with_types = [p for p in result.parameters if p.type_hint] + assert len(params_with_types) > 0 + + def test_python_extractor_docstrings(self, python_function_code): + """Test extraction from docstrings.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_function_code, "process_data") + + # Check that descriptions from docstrings are extracted + params_with_desc = [p for p in result.parameters if p.description] + assert len(params_with_desc) > 0 + + def test_python_extractor_dataclass(self, python_dataclass_code): + """Test extraction from dataclass attributes.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_dataclass_code, "MatchingConfig") + + assert len(result.parameters) > 0 + param_names = [p.name for p in result.parameters] + assert "threshold" in param_names or "input_file" in param_names + + def test_python_extractor_pydantic_model(self, python_pydantic_code): + """Test extraction from Pydantic models.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_pydantic_code, "MatchingConfig") + + assert len(result.parameters) > 0 + param_names = [p.name for p in result.parameters] + assert "threshold" in param_names or "input_file" in param_names + + def test_python_extractor_ast_parsing(self, python_function_code): + """Test AST-based extraction.""" + extractor = PythonLibExtractor() + result = extractor.extract(python_function_code, "process_data") + + # AST parsing should work for valid Python code + assert result.extraction_method == ExtractionMethod.REGEX + assert len(result.parameters) > 0 + + +# ============================================================================= +# HTTP API Extractor Tests +# ============================================================================= + +class TestHTTPAPIExtractor: + """Tests for HTTP API parameter extraction.""" + + def test_api_extractor_openapi_spec(self, openapi_spec): + """Test extraction from OpenAPI YAML.""" + extractor = HTTPAPIExtractor() + result = extractor.extract(openapi_spec, "matching_api") + + assert isinstance(result, ExtractionResult) + assert result.source_type == SourceType.HTTP_API + assert len(result.parameters) > 0 + + # Check for expected parameters + param_names = [p.name for p in result.parameters] + assert "threshold" in param_names or "input_file" in param_names + + def test_api_extractor_swagger_spec(self, swagger_spec): + """Test extraction from Swagger JSON.""" + extractor = HTTPAPIExtractor() + result = extractor.extract(swagger_spec, "matching_api") + + assert len(result.parameters) > 0 + param_names = [p.name for p in result.parameters] + assert "threshold" in param_names or "input_file" in param_names + + def test_api_extractor_path_params(self, openapi_spec): + """Test path parameter extraction.""" + extractor = HTTPAPIExtractor() + result = extractor.extract(openapi_spec, "matching_api") + + # OpenAPI spec has query params, not path params in our test data + # But we should still extract parameters + assert len(result.parameters) > 0 + + def test_api_extractor_query_params(self, openapi_spec): + """Test query parameter extraction.""" + extractor = HTTPAPIExtractor() + result = extractor.extract(openapi_spec, "matching_api") + + # Check for query parameters + query_params = [p for p in result.parameters if "threshold" in p.name or "max_results" in p.name] + assert len(query_params) > 0 + + def test_api_extractor_request_body(self, openapi_spec): + """Test request body parameter extraction.""" + extractor = HTTPAPIExtractor() + result = extractor.extract(openapi_spec, "matching_api") + + # Check for request body parameters + body_params = [p for p in result.parameters if "input_file" in p.name or "output_file" in p.name] + assert len(body_params) > 0 + + +# ============================================================================= +# Docker Extractor Tests +# ============================================================================= + +class TestDockerExtractor: + """Tests for Docker parameter extraction.""" + + def test_docker_extractor_env_vars(self, dockerfile_content): + """Test ENV variable extraction from Dockerfile.""" + extractor = DockerExtractor() + result = extractor.extract(dockerfile_content, "dockerfile") + + assert isinstance(result, ExtractionResult) + assert result.source_type == SourceType.DOCKER + assert len(result.parameters) > 0 + + # Check for ENV variables + env_params = [p for p in result.parameters if "THRESHOLD" in p.native_keys or "threshold" in p.name] + assert len(env_params) > 0 + + def test_docker_extractor_args(self, dockerfile_content): + """Test ARG extraction from Dockerfile.""" + extractor = DockerExtractor() + result = extractor.extract(dockerfile_content, "dockerfile") + + # Check for ARG declarations + arg_params = [p for p in result.parameters if "BUILD_VERSION" in p.native_keys or "build_version" in p.name] + assert len(arg_params) > 0 + + def test_docker_extractor_compose_env(self, docker_compose_content): + """Test environment variable extraction from docker-compose.yml.""" + extractor = DockerExtractor() + result = extractor.extract(docker_compose_content, "docker_compose") + + assert len(result.parameters) > 0 + + # Check for environment variables + env_params = [p for p in result.parameters if "THRESHOLD" in p.native_keys or "threshold" in p.name] + assert len(env_params) > 0 + + def test_docker_extractor_multiple_services(self, docker_compose_content): + """Test extraction from multiple services.""" + extractor = DockerExtractor() + result = extractor.extract(docker_compose_content, "docker_compose") + + # Should extract from both matcher and processor services + assert len(result.parameters) > 0 + + +# ============================================================================= +# README / Documentation Extractor Tests +# ============================================================================= + +class TestReadmeDocExtractor: + """Tests for README / documentation parameter extraction.""" + + def test_readme_extractor_list_params(self, readme_tool_doc): + """Test extraction of parameters from markdown list items.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + assert isinstance(result, ExtractionResult) + assert result.source_type == SourceType.README + assert result.extraction_method == ExtractionMethod.REGEX + assert len(result.parameters) > 0 + + param_names = [p.name for p in result.parameters] + assert "threshold" in param_names + assert "max_iter" in param_names + + def test_readme_extractor_table_params(self, readme_tool_doc): + """Test extraction of parameters from markdown tables.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + param_names = [p.name for p in result.parameters] + assert "batch_size" in param_names + assert "num_threads" in param_names + + def test_readme_extractor_env_vars(self, readme_tool_doc): + """Test extraction of environment variable references.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + param_names = [p.name for p in result.parameters] + assert "matcher_threshold" in param_names or "matcher_max_memory" in param_names + + def test_readme_extractor_placeholders(self, readme_tool_doc): + """Test extraction of placeholder parameters from usage lines.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + param_names = [p.name for p in result.parameters] + # , , from the Java usage line + assert "kb1" in param_names or "outputfolder" in param_names + + def test_readme_extractor_defaults(self, readme_tool_doc): + """Test that default values are extracted from descriptions.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + threshold_params = [p for p in result.parameters if p.name == "threshold"] + if threshold_params: + param = threshold_params[0] + assert param.default_value == 0.5 or param.default_value == "0.5" + + def test_readme_extractor_descriptions(self, readme_tool_doc): + """Test that descriptions are extracted.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_tool_doc, "entity_matcher") + + params_with_desc = [p for p in result.parameters if p.description] + assert len(params_with_desc) > 0 + + def test_readme_extractor_minimal(self, readme_minimal): + """Test extraction from a minimal README.""" + extractor = ReadmeDocExtractor() + result = extractor.extract(readme_minimal, "simple_tool") + + assert isinstance(result, ExtractionResult) + param_names = [p.name for p in result.parameters] + # Should find at least the and placeholders + assert "inputfile" in param_names or "outputfile" in param_names + + def test_readme_extractor_empty(self): + """Test handling of empty README.""" + extractor = ReadmeDocExtractor() + result = extractor.extract("", "test") + + assert isinstance(result, ExtractionResult) + assert len(result.parameters) == 0 + + +# ============================================================================= +# ParameterMiner Integration Tests +# ============================================================================= + +class TestParameterMiner: + """Integration tests for ParameterMiner.""" + + def test_parameter_miner_auto_detect_cli(self, cli_help_simple): + """Test auto-detection of CLI source.""" + miner = ParameterMiner() + result = miner.extract_parameters(cli_help_simple, method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.CLI + + def test_parameter_miner_auto_detect_python(self, python_function_code): + """Test auto-detection of Python source.""" + miner = ParameterMiner() + result = miner.extract_parameters(python_function_code, method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.PYTHON_LIB + + def test_parameter_miner_auto_detect_api(self, openapi_spec): + """Test auto-detection of API source.""" + miner = ParameterMiner() + result = miner.extract_parameters(openapi_spec, method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.HTTP_API + + def test_parameter_miner_auto_detect_docker(self, dockerfile_content): + """Test auto-detection of Docker source.""" + miner = ParameterMiner() + result = miner.extract_parameters(dockerfile_content, method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.DOCKER + + def test_parameter_miner_auto_detect_readme(self, readme_tool_doc): + """Test auto-detection of README source.""" + miner = ParameterMiner() + result = miner.extract_parameters(readme_tool_doc, method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.README + + def test_parameter_miner_file_path(self, test_data_dir): + """Test extraction from file path.""" + miner = ParameterMiner() + cli_file = test_data_dir / "cli" / "simple_help.txt" + result = miner.extract_parameters(str(cli_file), method=ExtractionMethod.AUTO) + + assert result.source_type == SourceType.CLI + assert result.tool_name == "simple_help" + + def test_parameter_miner_method_auto(self, cli_help_simple): + """Test auto method selection (regex → LLM fallback).""" + miner = ParameterMiner() + result = miner.extract_parameters(cli_help_simple, method=ExtractionMethod.AUTO) + + # Should use regex by default + assert result.extraction_method == ExtractionMethod.REGEX + + def test_parameter_miner_to_json(self, cli_help_simple): + """Test JSON output conversion.""" + miner = ParameterMiner() + result = miner.extract_parameters(cli_help_simple, method=ExtractionMethod.AUTO) + + json_output = miner.to_json(result) + assert isinstance(json_output, str) + assert "parameters" in json_output or '"parameters"' in json_output + + def test_parameter_miner_to_parameter_model(self, cli_help_simple): + """Test Parameter model conversion.""" + miner = ParameterMiner() + result = miner.extract_parameters(cli_help_simple, method=ExtractionMethod.AUTO) + + if result.parameters: + param_model = miner.to_parameter_model(result.parameters[0]) + assert isinstance(param_model, Parameter) + assert param_model.name is not None + assert param_model.datatype is not None + + +# ============================================================================= +# Error Handling Tests +# ============================================================================= + +class TestErrorHandling: + """Tests for error handling.""" + + def test_extractor_invalid_source(self): + """Test handling of invalid source content.""" + extractor = CLIExtractor() + result = extractor.extract("This is not valid CLI help", "test") + + # Should not crash, but may return empty or minimal results + assert isinstance(result, ExtractionResult) + + def test_extractor_empty_source(self): + """Test handling of empty source.""" + extractor = CLIExtractor() + result = extractor.extract("", "test") + + assert isinstance(result, ExtractionResult) + assert len(result.parameters) == 0 + + def test_extractor_malformed_spec(self): + """Test handling of malformed specifications.""" + extractor = HTTPAPIExtractor() + result = extractor.extract("{ invalid json }", "test") + + assert isinstance(result, ExtractionResult) + # Should handle gracefully, may have errors + assert len(result.errors) >= 0 + + def test_parameter_miner_unknown_source_type(self): + """Test handling of unknown source types.""" + miner = ParameterMiner() + result = miner.extract_parameters("Random text that doesn't match any pattern", method=ExtractionMethod.AUTO) + + assert isinstance(result, ExtractionResult) + # Should default to UNKNOWN or handle gracefully + assert result.source_type in [SourceType.UNKNOWN, SourceType.CLI, SourceType.PYTHON_LIB] + + +# ============================================================================= +# LLM Extractor Tests (Optional - Mock LLM) +# ============================================================================= + +class TestLLMExtractor: + """Tests for LLM-based extraction (with mocked LLM client).""" + + def test_llm_extractor_cli(self, cli_help_simple, mock_llm_client): + """Test LLM-based CLI extraction.""" + from kgpipe_parameters.extraction.param_miner import LLMCLIExtractor + + extractor = LLMCLIExtractor(mock_llm_client) + result = extractor.extract(cli_help_simple, "test_tool") + + assert isinstance(result, ExtractionResult) + assert result.extraction_method == ExtractionMethod.LLM + # Mock should return parameters + assert len(result.parameters) > 0 + + def test_llm_extractor_fallback(self, cli_help_simple, mock_llm_client): + """Test fallback from regex to LLM when regex fails.""" + miner = ParameterMiner(llm_client=mock_llm_client) + + # Use a source that regex might struggle with + result = miner.extract_parameters( + cli_help_simple, + method=ExtractionMethod.AUTO + ) + + # Should try regex first, but if it fails and LLM is available, use LLM + assert isinstance(result, ExtractionResult) + diff --git a/src/kgpipe_parameters/tests/test_visualization.py b/src/kgpipe_parameters/tests/test_visualization.py new file mode 100644 index 0000000..187b826 --- /dev/null +++ b/src/kgpipe_parameters/tests/test_visualization.py @@ -0,0 +1,179 @@ +""" +Tests for the parameter visualization module. +""" + +import json +import pytest +import numpy as np +from pathlib import Path +from typing import List + +from kgpipe_parameters.clustering.models import ( + ParameterVector, + ParameterCluster, + ClusteringResult, +) +from kgpipe_parameters.visualization import ParameterVisualizer + + +# ============================================================================ +# Helpers +# ============================================================================ + + +def _make_param( + name: str, + tool: str, + description: str = "", + embedding: List[float] | None = None, +) -> ParameterVector: + return ParameterVector( + name=name, + tool_name=tool, + description=description, + native_keys=[f"--{name}"], + source_label=f"{tool}/source", + embedding=embedding, + ) + + +def _random_embedding( + dim: int = 16, rng: np.random.Generator | None = None +) -> List[float]: + rng = rng or np.random.default_rng(42) + vec = rng.standard_normal(dim).astype(np.float32) + vec /= np.linalg.norm(vec) + return vec.tolist() + + +@pytest.fixture +def sample_clustering_result() -> ClusteringResult: + """A small synthetic clustering result for visualization tests.""" + rng = np.random.default_rng(0) + + # Cluster 0: cross-tool (threshold, 3 params, 2 tools) + c0_members = [ + _make_param( + "threshold", "tool_a", "Matching threshold", _random_embedding(rng=rng) + ), + _make_param( + "threshold", "tool_b", "Score threshold", _random_embedding(rng=rng) + ), + _make_param( + "similarity_threshold", + "tool_a", + "Similarity cutoff", + _random_embedding(rng=rng), + ), + ] + c0 = ParameterCluster( + cluster_id=0, + label="threshold", + members=c0_members, + tools=["tool_a", "tool_b"], + ) + + # Cluster 1: cross-tool (output, 2 params, 2 tools) + c1_members = [ + _make_param( + "output_dir", "tool_a", "Output directory", _random_embedding(rng=rng) + ), + _make_param( + "output_path", "tool_b", "Output file path", _random_embedding(rng=rng) + ), + ] + c1 = ParameterCluster( + cluster_id=1, + label="output_dir", + members=c1_members, + tools=["tool_a", "tool_b"], + ) + + # Cluster 2: single-tool (verbose, 2 params) + c2_members = [ + _make_param( + "verbose", "tool_a", "Verbosity level", _random_embedding(rng=rng) + ), + _make_param("debug", "tool_a", "Debug mode", _random_embedding(rng=rng)), + ] + c2 = ParameterCluster( + cluster_id=2, + label="verbose", + members=c2_members, + tools=["tool_a"], + ) + + return ClusteringResult( + n_parameters=7, + n_clusters=3, + distance_threshold=0.55, + model_name="test-model", + clusters=[c0, c1, c2], + ) + + +# ============================================================================ +# Tests +# ============================================================================ + + +class TestParameterVisualizer: + """Tests for ParameterVisualizer.""" + + def test_generate_all_creates_files(self, sample_clustering_result, tmp_path): + viz = ParameterVisualizer(sample_clustering_result, tmp_path) + paths = viz.generate_all() + assert len(paths) == 3 + for p in paths: + assert p.exists() + assert p.suffix == ".png" + + def test_plot_cluster_sizes(self, sample_clustering_result, tmp_path): + viz = ParameterVisualizer(sample_clustering_result, tmp_path) + path = viz.plot_cluster_sizes() + assert path.exists() + assert path.name == "_viz_cluster_sizes.png" + + def test_plot_tool_cluster_heatmap(self, sample_clustering_result, tmp_path): + viz = ParameterVisualizer(sample_clustering_result, tmp_path) + path = viz.plot_tool_cluster_heatmap() + assert path.exists() + assert path.name == "_viz_tool_heatmap.png" + + def test_plot_embedding_scatter(self, sample_clustering_result, tmp_path): + viz = ParameterVisualizer(sample_clustering_result, tmp_path) + path = viz.plot_embedding_scatter() + assert path.exists() + assert path.name == "_viz_embedding_scatter.png" + + def test_empty_result_returns_empty(self, tmp_path): + empty = ClusteringResult() + viz = ParameterVisualizer(empty, tmp_path) + paths = viz.generate_all() + assert paths == [] + + def test_from_clusters_json(self, sample_clustering_result, tmp_path): + # Write a JSON file + json_path = tmp_path / "_clusters.json" + data = sample_clustering_result.model_dump() + # Strip centroids/embeddings like the real save does + for c in data.get("clusters", []): + c.pop("centroid", None) + with open(json_path, "w") as f: + json.dump(data, f, default=str) + + viz = ParameterVisualizer.from_clusters_json(json_path) + assert viz.result.n_clusters == 3 + + def test_scatter_too_few_points(self, tmp_path): + """Scatter plot gracefully handles < 3 embedded parameters.""" + m = _make_param("x", "t", embedding=_random_embedding()) + c = ParameterCluster(cluster_id=0, label="x", members=[m], tools=["t"]) + result = ClusteringResult(n_parameters=1, n_clusters=1, clusters=[c]) + viz = ParameterVisualizer(result, tmp_path) + path = viz.plot_embedding_scatter() + assert path.exists() + + + + diff --git a/src/kgpipe_parameters/visualization/__init__.py b/src/kgpipe_parameters/visualization/__init__.py new file mode 100644 index 0000000..0bb436b --- /dev/null +++ b/src/kgpipe_parameters/visualization/__init__.py @@ -0,0 +1,9 @@ +"""Visualization module for parameter clustering results.""" + +from .kgpipe_parameter_explorer import ParameterVisualizer + +__all__ = ["ParameterVisualizer"] + + + + diff --git a/src/kgpipe_parameters/visualization/kgpipe_parameter_explorer.py b/src/kgpipe_parameters/visualization/kgpipe_parameter_explorer.py new file mode 100644 index 0000000..f03e086 --- /dev/null +++ b/src/kgpipe_parameters/visualization/kgpipe_parameter_explorer.py @@ -0,0 +1,257 @@ +""" +Visualization of parameter clustering results. + +Produces static plots (PNG) summarising how extracted parameters group +across tools: + - cluster size distribution + - tool × cluster heatmap (cross-tool clusters) + - 2-D embedding scatter (PCA, coloured by tool) +""" + +from __future__ import annotations + +import logging +from pathlib import Path +from typing import Optional + +import numpy as np +import matplotlib + +matplotlib.use("Agg") # non-interactive backend +import matplotlib.pyplot as plt +import seaborn as sns + +from ..clustering.models import ClusteringResult + +logger = logging.getLogger(__name__) + +# Consistent style +sns.set_theme(style="whitegrid", font_scale=0.9) + + +class ParameterVisualizer: + """Generate static visualizations from a ``ClusteringResult``.""" + + def __init__(self, result: ClusteringResult, output_dir: Path): + self.result = result + self.output_dir = Path(output_dir) + self.output_dir.mkdir(parents=True, exist_ok=True) + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def generate_all(self) -> list[Path]: + """Run every visualization and return list of saved file paths.""" + paths: list[Path] = [] + if not self.result.clusters: + logger.warning("No clusters to visualize") + return paths + + paths.append(self.plot_cluster_sizes()) + paths.append(self.plot_tool_cluster_heatmap()) + paths.append(self.plot_embedding_scatter()) + logger.info( + "Generated %d visualization(s) in %s", len(paths), self.output_dir + ) + return paths + + # ------------------------------------------------------------------ + # Individual plots + # ------------------------------------------------------------------ + + def plot_cluster_sizes( + self, filename: str = "_viz_cluster_sizes.png" + ) -> Path: + """Horizontal bar chart of cluster sizes (top-30).""" + clusters = sorted(self.result.clusters, key=lambda c: -c.size())[:30] + labels = [ + f"[{c.cluster_id}] {c.label}" + (" ★" if c.is_cross_tool() else "") + for c in clusters + ] + sizes = [c.size() for c in clusters] + colors = [ + "#4c72b0" if c.is_cross_tool() else "#c0c0c0" for c in clusters + ] + + fig, ax = plt.subplots(figsize=(8, max(4, len(labels) * 0.35))) + ax.barh(range(len(labels)), sizes, color=colors) + ax.set_yticks(range(len(labels))) + ax.set_yticklabels(labels) + ax.invert_yaxis() + ax.set_xlabel("Number of parameters") + ax.set_title( + f"Cluster sizes (top {len(clusters)} of {self.result.n_clusters})" + ) + # Legend for cross-tool marker + from matplotlib.patches import Patch + + ax.legend( + handles=[ + Patch(facecolor="#4c72b0", label="Cross-tool"), + Patch(facecolor="#c0c0c0", label="Single tool"), + ], + loc="lower right", + ) + fig.tight_layout() + path = self.output_dir / filename + fig.savefig(path, dpi=150) + plt.close(fig) + logger.info("Saved cluster size chart to %s", path) + return path + + def plot_tool_cluster_heatmap( + self, filename: str = "_viz_tool_heatmap.png" + ) -> Path: + """Heatmap of tools × clusters (cross-tool clusters only).""" + import pandas as pd + + cross = self.result.cross_tool_clusters() + if not cross: + # Fall back to top-20 clusters if no cross-tool clusters + cross = sorted(self.result.clusters, key=lambda c: -c.size())[:20] + + all_tools = sorted( + {m.tool_name for c in cross for m in c.members} + ) + cluster_labels = [f"[{c.cluster_id}] {c.label}" for c in cross] + + matrix = np.zeros((len(all_tools), len(cross)), dtype=int) + for j, c in enumerate(cross): + for m in c.members: + i = all_tools.index(m.tool_name) + matrix[i, j] += 1 + + df = pd.DataFrame(matrix, index=all_tools, columns=cluster_labels) + + fig, ax = plt.subplots( + figsize=(max(6, len(cross) * 0.6), max(3, len(all_tools) * 0.5)) + ) + sns.heatmap( + df, + annot=True, + fmt="d", + cmap="YlOrRd", + linewidths=0.5, + ax=ax, + ) + ax.set_title("Parameters per tool × cluster (cross-tool clusters)") + ax.set_ylabel("Tool") + ax.set_xlabel("Cluster") + plt.xticks(rotation=45, ha="right") + fig.tight_layout() + path = self.output_dir / filename + fig.savefig(path, dpi=150) + plt.close(fig) + logger.info("Saved tool×cluster heatmap to %s", path) + return path + + def plot_embedding_scatter( + self, filename: str = "_viz_embedding_scatter.png" + ) -> Path: + """2-D PCA scatter of parameter embeddings, coloured by tool.""" + from sklearn.decomposition import PCA + + # Collect all members across clusters + all_members = [m for c in self.result.clusters for m in c.members] + cluster_for_member = [ + c.cluster_id for c in self.result.clusters for m in c.members + ] + + # Check if embeddings are present; if not, recompute them + has_embeddings = any(m.embedding is not None for m in all_members) + if not has_embeddings and all_members: + logger.info("Embeddings not in clustering result — recomputing") + from ..clustering.similarity import embed_parameters + + embed_parameters(all_members) + + # Collect embeddings and metadata + embeddings = [] + tools = [] + names = [] + cluster_ids = [] + for cid, m in zip(cluster_for_member, all_members): + if m.embedding is not None: + embeddings.append(m.embedding) + tools.append(m.tool_name) + names.append(m.name) + cluster_ids.append(cid) + + if len(embeddings) < 3: + # Not enough points for meaningful 2-D projection + logger.warning( + "Too few embedded parameters (%d) for scatter plot", + len(embeddings), + ) + fig, ax = plt.subplots() + ax.text( + 0.5, + 0.5, + "Too few parameters for scatter plot", + ha="center", + va="center", + transform=ax.transAxes, + ) + path = self.output_dir / filename + fig.savefig(path, dpi=150) + plt.close(fig) + return path + + X = np.array(embeddings, dtype=np.float32) + pca = PCA(n_components=2, random_state=42) + X_2d = pca.fit_transform(X) + + unique_tools = sorted(set(tools)) + palette = sns.color_palette("husl", len(unique_tools)) + tool_to_color = dict(zip(unique_tools, palette)) + + fig, ax = plt.subplots(figsize=(10, 7)) + for tool in unique_tools: + mask = [t == tool for t in tools] + pts = X_2d[mask] + ax.scatter( + pts[:, 0], + pts[:, 1], + label=tool, + color=tool_to_color[tool], + alpha=0.65, + s=30, + edgecolors="white", + linewidth=0.3, + ) + + ax.set_xlabel(f"PC1 ({pca.explained_variance_ratio_[0]:.1%} var)") + ax.set_ylabel(f"PC2 ({pca.explained_variance_ratio_[1]:.1%} var)") + ax.set_title( + f"Parameter embeddings — {self.result.n_parameters} params, " + f"{self.result.n_clusters} clusters" + ) + ax.legend(title="Tool", bbox_to_anchor=(1.02, 1), loc="upper left") + fig.tight_layout() + path = self.output_dir / filename + fig.savefig(path, dpi=150, bbox_inches="tight") + plt.close(fig) + logger.info("Saved embedding scatter to %s", path) + return path + + # ------------------------------------------------------------------ + # Alternate constructor from JSON file + # ------------------------------------------------------------------ + + @classmethod + def from_clusters_json( + cls, json_path: Path, output_dir: Optional[Path] = None + ) -> "ParameterVisualizer": + """ + Create a visualizer from a ``_clusters.json`` file. + + If *output_dir* is not given, plots are saved next to the JSON file. + """ + import json + + with open(json_path) as f: + data = json.load(f) + + result = ClusteringResult.model_validate(data) + return cls(result, output_dir or json_path.parent) diff --git a/src/kgpipe_tasks/entity_resolution/fusion/union.py b/src/kgpipe_tasks/entity_resolution/fusion/union.py index 3af3e01..f31f49d 100644 --- a/src/kgpipe_tasks/entity_resolution/fusion/union.py +++ b/src/kgpipe_tasks/entity_resolution/fusion/union.py @@ -7,8 +7,9 @@ import json from kgpipe.common.registry import Registry import os -from kgcore.model.ontology import OntologyUtil -from kgpipe.execution.config import SOURCE_NAMESPACE, TARGET_ONTOLOGY_NAMESPACE, TARGET_RESOURCE_NAMESPACE + +from kgcore.api.ontology import OntologyUtil +from kgpipe.common.config import SOURCE_NAMESPACE, TARGET_ONTOLOGY_NAMESPACE, TARGET_RESOURCE_NAMESPACE def fuse_rdf_files(f1,f2,er): diff --git a/src/kgpipe_view/__init__.py b/src/kgpipe_view/__init__.py new file mode 100644 index 0000000..388a278 --- /dev/null +++ b/src/kgpipe_view/__init__.py @@ -0,0 +1 @@ +# Package marker for kgpipe_view modules. diff --git a/src/kgpipe_view/diagram_tab.py b/src/kgpipe_view/diagram_tab.py new file mode 100644 index 0000000..f1112e1 --- /dev/null +++ b/src/kgpipe_view/diagram_tab.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +from pathlib import Path + +import streamlit as st + +try: + from kgpipe_view.owl_to_mermaid import convert_and_write_mermaid, get_available_layers + from kgpipe_view.ui_common import render_mermaid +except ModuleNotFoundError: + from owl_to_mermaid import convert_and_write_mermaid, get_available_layers + from ui_common import render_mermaid + + +def render_diagram_tab(ttl_path: Path, mermaid_path: Path) -> None: + try: + layer_options = get_available_layers(ttl_path) + selected_layers = st.multiselect( + "Layers", + options=layer_options, + default=layer_options, + key="layer-filter", + ) + mermaid_code = convert_and_write_mermaid( + ttl_path=ttl_path, + output_path=mermaid_path, + layer_filter=selected_layers, + ) + except Exception as exc: # pragma: no cover - UI fallback path + st.error(f"Failed to convert `{ttl_path.name}` to Mermaid: {exc}") + return + + st.success( + f"Generated Mermaid from `{ttl_path.name}` and saved `{mermaid_path.name}`." + ) + if selected_layers: + st.caption(f"Current layer filter: `{', '.join(selected_layers)}`") + else: + st.caption("Current layer filter: `none`") + render_mermaid(mermaid_code) + with st.expander("Show Mermaid source"): + st.code(mermaid_code, language="mermaid") diff --git a/src/kgpipe_view/evaluations_tab.py b/src/kgpipe_view/evaluations_tab.py new file mode 100644 index 0000000..593057a --- /dev/null +++ b/src/kgpipe_view/evaluations_tab.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +import streamlit as st + +try: + from kgpipe_view.meta_kg_query import query_kg_data +except ModuleNotFoundError: + from meta_kg_query import query_kg_data + + +def render_evaluations_tab(endpoint_url: str) -> None: + st.subheader("Evaluations") + st.caption("Shows evaluation relations under `kgp:Evaluation`, including standalone evaluation nodes.") + + if st.button("Load evaluation hierarchy"): + try: + evaluation_hierarchy_df = query_kg_data(endpoint_url) + except Exception as exc: # pragma: no cover - network dependent path + st.error(f"Could not query `{endpoint_url}`: {exc}") + else: + if evaluation_hierarchy_df.empty: + st.info("No `kgp:Evaluation` subclass hierarchy returned by the endpoint.") + else: + st.dataframe(evaluation_hierarchy_df, use_container_width=True) diff --git a/src/kgpipe_view/kgpipe.owl.ttl b/src/kgpipe_view/kgpipe.owl.ttl new file mode 100644 index 0000000..a05c816 --- /dev/null +++ b/src/kgpipe_view/kgpipe.owl.ttl @@ -0,0 +1,285 @@ +@prefix : . +@prefix rdf: . +@prefix rdfs: . +@prefix owl: . +@prefix xsd: . + +:kgp a owl:Ontology . + +################################################################# +# Classes +################################################################# + +:Task a owl:Class, :CoreLayer . +:Method a owl:Class, :CoreLayer . +:Tool a owl:Class, :CoreLayer . +#:FrameworkTool a owl:Class ; rdfs:subClassOf :Tool . + +:Implementation a owl:Class, :CoreLayer . + +#:Interface a owl:Class, :CoreLayer . +#:CLIInterface a owl:Class ; rdfs:subClassOf :Interface . +#:RESTInterface a owl:Class ; rdfs:subClassOf :Interface . +#:LibraryAPIInterface a owl:Class ; rdfs:subClassOf :Interface . + +:Pipeline a owl:Class, :PipelineLayer . +:PipelineStep a owl:Class, :PipelineLayer . +:PipelineDefinition a owl:Class, :PipelineLayer . + +:TaskRun a owl:Class, :RunLayer . +:PipelineRun a owl:Class, :RunLayer . + +:DataArtifact a owl:Class, :DataLayer . +:DataDataArtifactSpec a owl:Class, :DataLayer . +:DataDataArtifactType a owl:Class, :DataLayer . +#:Schema a owl:Class, :DataLayer . + +:Parameter a owl:Class, :ParameterLayer . +:ParameterBinding a owl:Class, :ParameterLayer . + +################################################################# +# Object Properties +################################################################# + +### Task decomposition +:hasSubtask a owl:ObjectProperty ; + rdfs:domain :Task ; + rdfs:range :Task . + +### Semantics: method / tool / implementation +:realizesTask a owl:ObjectProperty ; + rdfs:domain :Method ; + rdfs:range :Task . + +:providesMethod a owl:ObjectProperty ; + rdfs:domain :Tool ; + rdfs:range :Method . + +:supportsTask a owl:ObjectProperty ; + rdfs:domain :Tool ; + rdfs:range :Task . + +:usesTool a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :Tool . + +:implementsMethod a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :Method . + +:hasInterface a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :Interface . + +### Pipeline structure +:hasStep a owl:ObjectProperty ; + rdfs:domain :Pipeline ; + rdfs:range :PipelineStep . + +:stepTask a owl:ObjectProperty ; + rdfs:domain :PipelineStep ; + rdfs:range :Task . + +:stepMethod a owl:ObjectProperty ; + rdfs:domain :PipelineStep ; + rdfs:range :Method . + +:nextStep a owl:ObjectProperty ; + rdfs:domain :PipelineStep ; + rdfs:range :PipelineStep . + +:definesPipeline a owl:ObjectProperty ; + rdfs:domain :PipelineDefinition ; + rdfs:range :Pipeline . + +:definedInTool a owl:ObjectProperty ; + rdfs:domain :PipelineDefinition ; + rdfs:range :Tool . + +:hasSourceDataArtifact a owl:ObjectProperty ; + rdfs:domain :PipelineDefinition ; + rdfs:range :DataArtifact . + +### Execution / runs +:executesTask a owl:ObjectProperty ; + rdfs:domain :TaskRun ; + rdfs:range :Task . + +:usesImplementation a owl:ObjectProperty ; + rdfs:domain :TaskRun ; + rdfs:range :Implementation . + +:runsPipeline a owl:ObjectProperty ; + rdfs:domain :PipelineRun ; + rdfs:range :Pipeline . + +:usesPipelineDefinition a owl:ObjectProperty ; + rdfs:domain :PipelineRun ; + rdfs:range :PipelineDefinition . + +:hasTaskRun a owl:ObjectProperty ; + rdfs:domain :PipelineRun ; + rdfs:range :TaskRun . + +### Data flow (runtime) +:hasInputDataArtifact a owl:ObjectProperty ; + rdfs:domain :TaskRun ; + rdfs:range :DataArtifact . + +:hasOutputDataArtifact a owl:ObjectProperty ; + rdfs:domain :TaskRun ; + rdfs:range :DataArtifact . + +### Data flow typing (design-time) +:expectsInputSpec a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :DataDataArtifactSpec . + +:producesOutputSpec a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :DataDataArtifactSpec . + +:requiresType a owl:ObjectProperty ; + rdfs:domain :DataDataArtifactSpec ; + rdfs:range :DataDataArtifactType . + +### DataArtifact typing / schema +:hasDataDataArtifactType a owl:ObjectProperty ; + rdfs:domain :DataArtifact ; + rdfs:range :DataDataArtifactType . + +#:conformsToSchema a owl:ObjectProperty ; +# rdfs:domain :DataArtifact ; +# rdfs:range :Schema . + +### Parameters +:hasParameter a owl:ObjectProperty ; + rdfs:domain :Implementation ; + rdfs:range :Parameter . + +:hasParameterBinding a owl:ObjectProperty ; + rdfs:domain :TaskRun ; + rdfs:range :ParameterBinding . + +:bindsParameter a owl:ObjectProperty ; + rdfs:domain :ParameterBinding ; + rdfs:range :Parameter . + +################################################################# +# Datatype Properties +################################################################# + +### Implementation +:implementationName a owl:DatatypeProperty ; + rdfs:domain :Implementation ; + rdfs:range xsd:string . + +:commandTemplate a owl:DatatypeProperty ; + rdfs:domain :Implementation ; + rdfs:range xsd:string . + +:runtime a owl:DatatypeProperty ; + rdfs:domain :Implementation ; + rdfs:range xsd:string . + +:implementationVersion a owl:DatatypeProperty ; + rdfs:domain :Implementation ; + rdfs:range xsd:string . + +### Method +:methodName a owl:DatatypeProperty ; + rdfs:domain :Method ; + rdfs:range xsd:string . + +### Tool +:toolVersion a owl:DatatypeProperty ; + rdfs:domain :Tool ; + rdfs:range xsd:string . + +:toolName a owl:DatatypeProperty ; + rdfs:domain :Tool ; + rdfs:range xsd:string . + +:toolPage a owl:DatatypeProperty ; + rdfs:domain :Tool ; + rdfs:range xsd:string . + +:toolName a owl:DatatypeProperty ; + rdfs:domain :Tool ; + rdfs:range xsd:string . + +### DataArtifact +:location a owl:DatatypeProperty ; + rdfs:domain :DataArtifact ; + rdfs:range xsd:anyURI . + +### DataDataArtifactSpec +:dataType a owl:DatatypeProperty ; + rdfs:domain :DataDataArtifactSpec ; + rdfs:range xsd:string . + +### DataDataArtifactType +:dataFormat a owl:DatatypeProperty ; + rdfs:domain :DataDataArtifactType ; + rdfs:range xsd:string . + +:dataSchema a owl:DatatypeProperty ; + rdfs:domain :DataDataArtifactType ; + rdfs:range xsd:string . + +### Parameter +:paramName a owl:DatatypeProperty ; + rdfs:domain :Parameter ; + rdfs:range xsd:string . + +:paramDescription a owl:DatatypeProperty ; + rdfs:domain :Parameter ; + rdfs:range xsd:string . + +:paramDataType a owl:DatatypeProperty ; + rdfs:domain :Parameter ; + rdfs:range xsd:string . + +:defaultValue a owl:DatatypeProperty ; + rdfs:domain :Parameter ; + rdfs:range xsd:string . + +### ParameterBinding +:value a owl:DatatypeProperty ; + rdfs:domain :ParameterBinding ; + rdfs:range xsd:string . + +### TaskRun +:startedAt a owl:DatatypeProperty ; + rdfs:domain :TaskRun ; + rdfs:range xsd:dateTime . + +:endedAt a owl:DatatypeProperty ; + rdfs:domain :TaskRun ; + rdfs:range xsd:dateTime . + +:status a owl:DatatypeProperty ; + rdfs:domain :TaskRun ; + rdfs:range xsd:string . + +:exitCode a owl:DatatypeProperty ; + rdfs:domain :TaskRun ; + rdfs:range xsd:integer . + +:logPath a owl:DatatypeProperty ; + rdfs:domain :TaskRun ; + rdfs:range xsd:string . + +### PipelineRun +:pipelineStartedAt a owl:DatatypeProperty ; + rdfs:domain :PipelineRun ; + rdfs:range xsd:dateTime . + +:pipelineEndedAt a owl:DatatypeProperty ; + rdfs:domain :PipelineRun ; + rdfs:range xsd:dateTime . + +:pipelineStatus a owl:DatatypeProperty ; + rdfs:domain :PipelineRun ; + rdfs:range xsd:string . + diff --git a/src/kgpipe_view/kgpipe_view.py b/src/kgpipe_view/kgpipe_view.py index 7a990c3..b76ee3a 100644 --- a/src/kgpipe_view/kgpipe_view.py +++ b/src/kgpipe_view/kgpipe_view.py @@ -1,85 +1,42 @@ -from turtle import back -from streamlit import table, title, text_input, button, write -import streamlit as st -from kgpipe.common.registry import Registry -import kgpipe_tasks.tasks -import sqlite3 -import graphviz - -title("KGpipe View") -st.set_page_config(layout="wide") - -from streamlit_cytoscapejs import st_cytoscapejs - -elements = [ - {"data": {"id": "one", "label": "Node 1"}, "position": {"x": 0, "y": 0}}, - {"data": {"id": "two", "label": "Node 2"}, "position": {"x": 100, "y": 0}}, - {"data": {"source": "one", "target": "two", "label": "Edge from Node1 to Node2"}}, -] -stylesheet = [ - {"selector": "node", "style": {"width": 20, "height": 20, "shape": "rectangle"}}, - {"selector": "edge", "style": {"width": 10}}, -] +from __future__ import annotations -clicked_elements = st_cytoscapejs(elements, stylesheet, width=1000, height=1000) +from pathlib import Path -if clicked_elements is not None: - st.write(clicked_elements) - -# # wide streamlit view -# wide_view = True - -# from kgpipe.common.systemgraph import backend +import streamlit as st -# def sparql(query: str): -# qr = backend.query_sparql(query) -# bindings = qr["results"]["bindings"] -# results = [] -# for binding in bindings: -# keys = binding.keys() -# row = {} -# for key in keys: -# row[key] = binding[key]["value"] -# results.append(row) -# return results +try: + from kgpipe_view.diagram_tab import render_diagram_tab + from kgpipe_view.evaluations_tab import render_evaluations_tab + from kgpipe_view.pipelines_tab import render_pipelines_tab + from kgpipe_view.tasks_tab import render_tasks_tab +except ModuleNotFoundError: + # Support direct script execution via: streamlit run src/kgpipe_view/kgpipe_view.py + import importlib -# # create sqlite3 database -# conn = sqlite3.connect("kgpipe_view.db") -# cursor = conn.cursor() -# cursor.execute("CREATE TABLE IF NOT EXISTS queries (id INTEGER PRIMARY KEY AUTOINCREMENT, query TEXT)") -# conn.commit() + render_diagram_tab = importlib.import_module("diagram_tab").render_diagram_tab + render_evaluations_tab = importlib.import_module("evaluations_tab").render_evaluations_tab + render_pipelines_tab = importlib.import_module("pipelines_tab").render_pipelines_tab + render_tasks_tab = importlib.import_module("tasks_tab").render_tasks_tab -# def save_query(query: str): -# cursor.execute("INSERT INTO queries (query) VALUES (?)", (query,)) -# conn.commit() -# def get_queries(): -# cursor.execute("SELECT * FROM queries") -# return cursor.fetchall() +st.set_page_config(page_title="KGpipe View", layout="wide") +st.title("KGpipe View") +st.caption("Explore the KGpipe meta knowledge graph rendered from Owl/Turtle.") -# queries = get_queries() -# # drop down menu for queries -# query_dropdown = st.selectbox("Queries", [q[1] for q in queries]) +base_dir = Path(__file__).resolve().parent +ttl_path = base_dir / "kgpipe.owl.ttl" +mermaid_path = base_dir / "kgpipe.owl.mmd" -# # query field -# query = text_input("SELECT * { ?s ?p ?o . } LIMIT 10", value=query_dropdown) -# if button("Execute"): -# query_result = sparql(query) -# table(query_result) +diagram_tab, tasks_tab, pipelines_tab, evaluations_tab = st.tabs(["Ontology Diagram", "Tasks", "Pipelines", "Evaluations"]) -# # save query button -# if button("Save Query"): -# save_query(query) -# queries = get_queries() -# query_dropdown = st.selectbox("Queries", [q[1] for q in queries]) +with diagram_tab: + render_diagram_tab(ttl_path=ttl_path, mermaid_path=mermaid_path) +with tasks_tab: + endpoint_url = render_tasks_tab() -# def graph_visualization(query_result: list): -# graph = graphviz.Digraph() -# for row in query_result: -# graph.edge(row["s"], row["o"]) -# return graph +with pipelines_tab: + render_pipelines_tab(endpoint_url=endpoint_url) -# # graph visualization -# graph = graph_visualization(query_result) -# st.graphviz_chart(graph) \ No newline at end of file +with evaluations_tab: + render_evaluations_tab(endpoint_url=endpoint_url) \ No newline at end of file diff --git a/src/kgpipe_view/meta_kg_query.py b/src/kgpipe_view/meta_kg_query.py new file mode 100644 index 0000000..3e3c398 --- /dev/null +++ b/src/kgpipe_view/meta_kg_query.py @@ -0,0 +1,309 @@ +from __future__ import annotations + +from typing import Any + +import pandas as pd + + +PRIMARY_QUERY = """ +PREFIX kgp: + +SELECT ?task ?method ?implementation ?tool ?runtime ?implementationVersion ?commandTemplate +WHERE { + ?implementation a kgp:Implementation . + OPTIONAL { ?implementation kgp:implementsMethod ?method . } + OPTIONAL { ?implementation kgp:usesTool ?tool . } + OPTIONAL { ?implementation kgp:runtime ?runtime . } + OPTIONAL { ?implementation kgp:implementationVersion ?implementationVersion . } + OPTIONAL { ?implementation kgp:commandTemplate ?commandTemplate . } + OPTIONAL { ?method kgp:realizesTask ?task . } +} +ORDER BY ?task ?implementation +""" + + +TASK_HIERARCHY_PRIMARY_QUERY = """ +PREFIX kgp: +PREFIX rdfs: +PREFIX owl: + +SELECT DISTINCT ?task ?parentTask +WHERE { + { + ?task a kgp:Task . + } + UNION + { + ?task a owl:Class . + ?task rdfs:subClassOf+ kgp:Task . + FILTER(?task != kgp:Task) + } + UNION + { + ?method kgp:realizesTask ?task . + } + FILTER(isIRI(?task)) + OPTIONAL { + ?task rdfs:subClassOf ?parentTask . + ?parentTask rdfs:subClassOf* kgp:Task . + FILTER(?parentTask != owl:Thing) + } +} +ORDER BY ?task ?parentTask +""" + + +TASK_HIERARCHY_FALLBACK_QUERY = """ +PREFIX rdfs: + +SELECT DISTINCT ?task ?parentTask +WHERE { + { + ?task a ?taskType . + FILTER(STRENDS(STR(?taskType), "Task")) + } + UNION + { + ?task a ?classType . + FILTER(STRENDS(STR(?classType), "Class")) + ?task rdfs:subClassOf+ ?taskRoot . + FILTER(STRENDS(STR(?taskRoot), "Task")) + FILTER(?task != ?taskRoot) + } + UNION + { + ?method ?realizesTaskPredicate ?task . + FILTER(STRENDS(STR(?realizesTaskPredicate), "realizesTask")) + } + FILTER(isIRI(?task)) + OPTIONAL { + ?task rdfs:subClassOf ?parentTask . + FILTER(STRENDS(STR(?parentTask), "Task")) + } +} +ORDER BY ?task ?parentTask +""" + + +FALLBACK_QUERY = """ +SELECT ?task ?method ?implementation ?tool ?runtime ?implementationVersion ?commandTemplate +WHERE { + ?implementation a ?implementationType . + FILTER(STRENDS(STR(?implementationType), "Implementation")) + + OPTIONAL { + ?implementation ?implementsMethodPredicate ?method . + FILTER(STRENDS(STR(?implementsMethodPredicate), "implementsMethod")) + } + OPTIONAL { + ?method ?realizesTaskPredicate ?task . + FILTER(STRENDS(STR(?realizesTaskPredicate), "realizesTask")) + } + OPTIONAL { + ?implementation ?usesToolPredicate ?tool . + FILTER(STRENDS(STR(?usesToolPredicate), "usesTool")) + } + OPTIONAL { + ?implementation ?runtimePredicate ?runtime . + FILTER(STRENDS(STR(?runtimePredicate), "runtime")) + } + OPTIONAL { + ?implementation ?implementationVersionPredicate ?implementationVersion . + FILTER(STRENDS(STR(?implementationVersionPredicate), "implementationVersion")) + } + OPTIONAL { + ?implementation ?commandTemplatePredicate ?commandTemplate . + FILTER(STRENDS(STR(?commandTemplatePredicate), "commandTemplate")) + } +} +ORDER BY ?task ?implementation +""" + +PIPELINE_RUN_QUERY = """ +PREFIX kgp: + +SELECT DISTINCT ?pipelineRun +WHERE { + ?pipelineRun a kgp:PipelineRun . +} +""" + +KG_DATA_QUERY = """ +PREFIX kgp: + +SELECT DISTINCT ?kgData +WHERE { + VALUES ?format { + ".nt" + ".ttl" + ".rdf" + ".jsonld" + } + ?kgData a kgp:Data . + ?kgData ?format . +} +""" + +TASK_IO_SPECS_QUERY = """ +PREFIX kgp: + +SELECT DISTINCT ?implementation ?ioType ?format +WHERE { + ?implementation a kgp:Implementation . + ?implementation ?ioPredicate ?dataNode . + ?dataNode ?formatPredicate ?format . + FILTER(isIRI(?implementation)) + FILTER(isIRI(?dataNode)) + FILTER( + STRENDS(STR(?ioPredicate), "input") + || STRENDS(STR(?ioPredicate), "output") + ) + FILTER(STRENDS(STR(?formatPredicate), "format")) + BIND( + IF(STRENDS(STR(?ioPredicate), "input"), "input", "output") + AS ?ioType + ) +} +ORDER BY ?implementation ?ioType ?format +""" + +TASK_IO_SPECS_FALLBACK_QUERY = """ +SELECT DISTINCT ?implementation ?ioType ?format +WHERE { + ?implementation a ?implementationType . + FILTER(STRENDS(STR(?implementationType), "Implementation")) + ?implementation ?ioPredicate ?dataNode . + ?dataNode ?formatPredicate ?format . + FILTER(isIRI(?implementation)) + FILTER(isIRI(?dataNode)) + FILTER( + STRENDS(STR(?ioPredicate), "input") + || STRENDS(STR(?ioPredicate), "output") + ) + FILTER(STRENDS(STR(?formatPredicate), "format")) + BIND( + IF(STRENDS(STR(?ioPredicate), "input"), "input", "output") + AS ?ioType + ) +} +ORDER BY ?implementation ?ioType ?format +""" + + +def _run_select(endpoint_url: str, query: str) -> list[dict[str, Any]]: + from SPARQLWrapper import JSON, SPARQLWrapper + + client = SPARQLWrapper(endpoint_url) + client.setQuery(query) + client.setReturnFormat(JSON) + result = client.query().convert() + return result.get("results", {}).get("bindings", []) + + +def _cell(binding: dict[str, Any], key: str) -> str: + item = binding.get(key) + if not item: + return "" + return str(item.get("value", "")) + + +def _to_task_rows(bindings: list[dict[str, Any]]) -> list[dict[str, str]]: + rows: list[dict[str, str]] = [] + for binding in bindings: + rows.append( + { + "task": _cell(binding, "task"), + "method": _cell(binding, "method"), + "implementation": _cell(binding, "implementation"), + "tool": _cell(binding, "tool"), + "runtime": _cell(binding, "runtime"), + "implementation_version": _cell(binding, "implementationVersion"), + "command_template": _cell(binding, "commandTemplate"), + } + ) + return rows + + +def _to_task_hierarchy_rows(bindings: list[dict[str, Any]]) -> list[dict[str, str]]: + rows: list[dict[str, str]] = [] + for binding in bindings: + rows.append( + { + "task": _cell(binding, "task"), + "parent_task": _cell(binding, "parentTask"), + } + ) + return rows + +def _to_pipeline_run_rows(bindings: list[dict[str, Any]]) -> list[dict[str, str]]: + rows: list[dict[str, str]] = [] + for binding in bindings: + rows.append( + { + "pipeline_run": _cell(binding, "pipelineRun"), + } + ) + print(rows) + return rows + +def _to_kg_data_rows(bindings: list[dict[str, Any]]) -> list[dict[str, str]]: + rows: list[dict[str, str]] = [] + for binding in bindings: + rows.append( + { + "kg_data": _cell(binding, "kgData"), + } + ) + return rows + +def _to_task_io_rows(bindings: list[dict[str, Any]]) -> list[dict[str, str]]: + rows: list[dict[str, str]] = [] + for binding in bindings: + rows.append( + { + "implementation": _cell(binding, "implementation"), + "io_type": _cell(binding, "ioType"), + "format": _cell(binding, "format"), + } + ) + return rows + + +def query_tasks_implementations(endpoint_url: str) -> pd.DataFrame: + bindings = _run_select(endpoint_url, PRIMARY_QUERY) + if not bindings: + bindings = _run_select(endpoint_url, FALLBACK_QUERY) + rows = _to_task_rows(bindings) + return pd.DataFrame(rows) + + +def query_task_hierarchy(endpoint_url: str) -> pd.DataFrame: + bindings = _run_select(endpoint_url, TASK_HIERARCHY_PRIMARY_QUERY) + if not bindings: + bindings = _run_select(endpoint_url, TASK_HIERARCHY_FALLBACK_QUERY) + rows = _to_task_hierarchy_rows(bindings) + return pd.DataFrame(rows) + +def query_pipeline_hierarchy(endpoint_url: str) -> pd.DataFrame: + bindings = _run_select(endpoint_url, PIPELINE_RUN_QUERY) + rows = _to_pipeline_run_rows(bindings) + return pd.DataFrame(rows) + +def query_evaluation_hierarchy(endpoint_url: str) -> pd.DataFrame: + # TODO: Implement evaluation hierarchy query + # bindings = _run_select(endpoint_url, EVALUATION_HIERARCHY_QUERY) + # rows = _to_evaluation_hierarchy_rows(bindings) + # return pd.DataFrame(rows) + return pd.DataFrame([]) + +def query_kg_data(endpoint_url: str) -> pd.DataFrame: + bindings = _run_select(endpoint_url, KG_DATA_QUERY) + rows = _to_kg_data_rows(bindings) + return pd.DataFrame(rows) + +def query_task_io_specs(endpoint_url: str) -> pd.DataFrame: + bindings = _run_select(endpoint_url, TASK_IO_SPECS_QUERY) + if not bindings: + bindings = _run_select(endpoint_url, TASK_IO_SPECS_FALLBACK_QUERY) + rows = _to_task_io_rows(bindings) + return pd.DataFrame(rows) \ No newline at end of file diff --git a/src/kgpipe_view/owl_to_mermaid.py b/src/kgpipe_view/owl_to_mermaid.py new file mode 100644 index 0000000..af9f10b --- /dev/null +++ b/src/kgpipe_view/owl_to_mermaid.py @@ -0,0 +1,138 @@ +from __future__ import annotations + +from collections import defaultdict +from pathlib import Path +from typing import Iterable, Optional + +from rdflib import Graph +from rdflib.namespace import OWL, RDF, RDFS + + +def _local_name(uri: object) -> str: + text = str(uri) + if "#" in text: + return text.rsplit("#", maxsplit=1)[-1] + if "/" in text: + return text.rsplit("/", maxsplit=1)[-1] + return text + + +def _load_graph(ttl_path: Path) -> Graph: + graph = Graph() + graph.parse(ttl_path, format="turtle") + return graph + + +def get_available_layers(ttl_path: Path) -> list[str]: + graph = _load_graph(ttl_path) + layer_names: set[str] = set() + for class_node in graph.subjects(RDF.type, OWL.Class): + for class_type in graph.objects(class_node, RDF.type): + if class_type != OWL.Class: + layer_names.add(_local_name(class_type)) + return sorted(layer_names) + + +def _normalize_layer_filter(layer_filter: Optional[str | Iterable[str]]) -> Optional[set[str]]: + if layer_filter is None: + return None + if isinstance(layer_filter, str): + return {layer_filter} + normalized = {layer for layer in layer_filter if layer} + return normalized or None + + +def _filtered_class_names( + graph: Graph, layer_filter: Optional[str | Iterable[str]] +) -> set[str]: + all_classes = {_local_name(node) for node in graph.subjects(RDF.type, OWL.Class)} + selected_layers = _normalize_layer_filter(layer_filter) + if not selected_layers: + return all_classes + + selected: set[str] = set() + for class_node in graph.subjects(RDF.type, OWL.Class): + class_types = {_local_name(node) for node in graph.objects(class_node, RDF.type)} + if class_types.intersection(selected_layers): + selected.add(_local_name(class_node)) + return selected + + +def convert_owl_ttl_to_mermaid( + ttl_path: Path, layer_filter: Optional[str | Iterable[str]] = None +) -> str: + graph = _load_graph(ttl_path) + selected_classes = _filtered_class_names(graph, layer_filter) + + classes = sorted(selected_classes) + object_property_nodes = sorted( + set(graph.subjects(RDF.type, OWL.ObjectProperty)), key=lambda node: _local_name(node) + ) + datatype_property_nodes = sorted( + set(graph.subjects(RDF.type, OWL.DatatypeProperty)), + key=lambda node: _local_name(node), + ) + + domain_map: dict[str, list[str]] = defaultdict(list) + range_map: dict[str, list[str]] = defaultdict(list) + for prop_node in object_property_nodes + datatype_property_nodes: + prop_name = _local_name(prop_node) + for domain in graph.objects(prop_node, RDFS.domain): + domain_map[prop_name].append(_local_name(domain)) + for value_range in graph.objects(prop_node, RDFS.range): + range_map[prop_name].append(_local_name(value_range)) + + lines: list[str] = ["classDiagram", "direction LR", ""] + + for class_name in classes: + lines.append(f"class {class_name}") + + subclass_lines: list[str] = [] + for child, _, parent in graph.triples((None, RDFS.subClassOf, None)): + child_name = _local_name(child) + parent_name = _local_name(parent) + if child_name not in selected_classes or parent_name not in selected_classes: + continue + subclass_lines.append(f"{parent_name} <|-- {child_name}") + if subclass_lines: + lines.extend(["", "%% Inheritance", *sorted(set(subclass_lines))]) + + relation_lines: list[str] = [] + for prop in sorted(_local_name(node) for node in object_property_nodes): + for domain in domain_map.get(prop, []): + for value_range in range_map.get(prop, []): + if domain not in selected_classes or value_range not in selected_classes: + continue + relation_lines.append( + f'{domain} "0..*" --> "0..*" {value_range} : {prop}' + ) + if relation_lines: + lines.extend(["", "%% Object properties", *sorted(set(relation_lines))]) + + datatype_map: dict[str, list[str]] = defaultdict(list) + for prop in sorted(_local_name(node) for node in datatype_property_nodes): + for domain in domain_map.get(prop, []): + if domain not in selected_classes: + continue + value_ranges = range_map.get(prop, ["string"]) + for value_range in value_ranges: + datatype_map[domain].append(f" +{value_range} {prop}") + + if datatype_map: + lines.extend(["", "%% Datatype properties"]) + for domain in sorted(datatype_map): + lines.append(f"class {domain} {{") + lines.extend(sorted(set(datatype_map[domain]))) + lines.append("}") + + return "\n".join(lines) + "\n" + + +def convert_and_write_mermaid( + ttl_path: Path, + output_path: Path, + layer_filter: Optional[str | Iterable[str]] = None, +) -> str: + mermaid = convert_owl_ttl_to_mermaid(ttl_path, layer_filter=layer_filter) + output_path.write_text(mermaid, encoding="utf-8") + return mermaid diff --git a/src/kgpipe_view/pipelines_tab.py b/src/kgpipe_view/pipelines_tab.py new file mode 100644 index 0000000..78e92b1 --- /dev/null +++ b/src/kgpipe_view/pipelines_tab.py @@ -0,0 +1,323 @@ +from __future__ import annotations + +import json +from uuid import uuid4 + +import streamlit as st + +from kgpipe.common.systemgraph import PipeKG +try: + from kgpipe_view.meta_kg_query import query_pipeline_hierarchy + from kgpipe_view.ui_common import render_mermaid +except ModuleNotFoundError: + from meta_kg_query import query_pipeline_hierarchy + from ui_common import render_mermaid + + +def _entity_to_task_label(entity) -> str: + """Best-effort conversion from implementation-like object to task label.""" + name = getattr(entity, "name", "") or "" + return _task_label_to_base_name(str(name)) + + +def _task_label_to_base_name(label: str) -> str: + base = str(label or "").strip() + if base.endswith("Impl"): + return base[:-4] + return base + + +@st.cache_data(show_spinner=False) +def _get_task_io_specs() -> dict[str, dict[str, set[str]]]: + return PipeKG().list_task_io_specs() + + +def _shared_formats( + source_task_name: str, + target_task_name: str, + specs: dict[str, dict[str, set[str]]], +) -> set[str]: + source_outputs = specs.get(_task_label_to_base_name(source_task_name), {}).get("outputs", set()) + target_inputs = specs.get(_task_label_to_base_name(target_task_name), {}).get("inputs", set()) + if not source_outputs or not target_inputs: + return {"*"} + return source_outputs.intersection(target_inputs) + + +def _edge_label(formats: set[str]) -> str: + if not formats: + return "" + if formats == {"*"}: + return "any" + return ", ".join(sorted(formats)) + + +def _task_io_summary(task_name: str, specs: dict[str, dict[str, set[str]]]) -> str: + task_spec = specs.get(_task_label_to_base_name(task_name), {}) + inputs = sorted(task_spec.get("inputs", set())) + outputs = sorted(task_spec.get("outputs", set())) + in_text = ", ".join(inputs) if inputs else "-" + out_text = ", ".join(outputs) if outputs else "-" + return f"in: {in_text} | out: {out_text}" + + +def _task_options_from_implementations(implementations: list) -> list[str]: + labels = {_entity_to_task_label(entity) for entity in implementations} + return sorted(label for label in labels if label) + + +def _pipeline_to_mermaid( + pipeline_nodes: list[dict[str, str]], + pipeline_edges: list[dict[str, object]], +) -> str: + if not pipeline_nodes: + return "flowchart LR\n empty[Empty pipeline]" + + def _safe_node_id(raw: str) -> str: + return "n_" + "".join(ch if ch.isalnum() else "_" for ch in raw) + + node_ids = {node["id"] for node in pipeline_nodes} + lines = ["flowchart LR"] + for node in pipeline_nodes: + node_id = _safe_node_id(node["id"]) + label = node["name"].replace(chr(34), chr(39)) + lines.append(f'{node_id}["{label}"]') + + for edge in pipeline_edges: + source = str(edge.get("from", "")) + target = str(edge.get("to", "")) + if source not in node_ids or target not in node_ids or source == target: + continue + src = _safe_node_id(source) + dst = _safe_node_id(target) + formats = set(edge.get("formats", [])) + label = _edge_label(formats) + if label: + lines.append(f'{src} -->|{label}| {dst}') + else: + lines.append(f"{src} --> {dst}") + return "\n".join(lines) + + +def _get_tasks() -> list: + return PipeKG().list_taskImplementations() + + +def render_pipelines_tab(endpoint_url: str) -> None: + st.subheader("Pipelines") + st.caption("Shows pipeline relations under `kgp:Pipeline`, including standalone pipeline nodes.") + + if st.button("Load pipeline hierarchy"): + try: + pipeline_hierarchy_df = query_pipeline_hierarchy(endpoint_url) + except Exception as exc: # pragma: no cover - network dependent path + st.error(f"Could not query `{endpoint_url}`: {exc}") + else: + if pipeline_hierarchy_df.empty: + st.info("No `kgp:Pipeline` subclass hierarchy returned by the endpoint.") + else: + st.dataframe(pipeline_hierarchy_df, use_container_width=True) + + st.divider() + st.subheader("Pipeline builder") + st.caption("Compose a DAG pipeline with reusable outputs and parallel branches.") + + if "pipeline_nodes" not in st.session_state: + legacy_steps = st.session_state.get("pipeline_steps", []) + migrated_nodes: list[dict[str, str]] = [] + if legacy_steps and isinstance(legacy_steps[0], str): + migrated_nodes = [ + {"id": f"task-{uuid4().hex[:8]}", "name": task_name} + for task_name in legacy_steps + ] + elif legacy_steps: + migrated_nodes = legacy_steps + st.session_state.pipeline_nodes = migrated_nodes + st.session_state.pipeline_edges = [] + for idx in range(len(migrated_nodes) - 1): + st.session_state.pipeline_edges.append( + { + "from": migrated_nodes[idx]["id"], + "to": migrated_nodes[idx + 1]["id"], + "formats": [], + } + ) + st.session_state.pop("pipeline_steps", None) + if "pipeline_edges" not in st.session_state: + st.session_state.pipeline_edges = [] + + implementations = _get_tasks() + task_options = _task_options_from_implementations(implementations) + task_specs = _get_task_io_specs() + + pipeline_nodes = st.session_state.pipeline_nodes + pipeline_edges = st.session_state.pipeline_edges + + def _node_label(node: dict[str, str]) -> str: + return f'{node["name"]} ({node["id"][-6:]})' + + if not task_options: + st.info("No task implementations found in PipeKG yet.") + else: + task_option_labels = { + task_name: f"{task_name} [{_task_io_summary(task_name, task_specs)}]" + for task_name in task_options + } + selected_task_name = st.selectbox( + "Task to add", + options=task_options, + key="pipeline_builder_selected_task", + format_func=lambda task_name: task_option_labels.get(task_name, task_name), + ) + st.caption(f"Selected task spec: `{_task_io_summary(selected_task_name, task_specs)}`") + + selected_source_id = None + if pipeline_nodes: + source_options = [{"label": "No dependency (new branch/root)", "id": None}] + for node in pipeline_nodes: + formats = _shared_formats(node["name"], selected_task_name, task_specs) + if formats: + source_options.append( + { + "label": f'{_node_label(node)} [{_edge_label(formats)}]', + "id": node["id"], + } + ) + + source_label = st.selectbox( + "Connect new task from", + options=[opt["label"] for opt in source_options], + key="pipeline_builder_selected_source", + help="Pick an upstream task output to reuse, or create a root branch with no dependency.", + ) + source_lookup = {opt["label"]: opt["id"] for opt in source_options} + selected_source_id = source_lookup[source_label] + else: + st.caption("First task creates the first root in the pipeline.") + + add_col, remove_col, clear_col = st.columns(3) + with add_col: + if st.button("Add task node", key="pipeline_builder_add_task"): + new_id = f"task-{uuid4().hex[:8]}" + st.session_state.pipeline_nodes.append({"id": new_id, "name": selected_task_name}) + if selected_source_id is not None: + source_node = next( + (node for node in st.session_state.pipeline_nodes if node["id"] == selected_source_id), + None, + ) + formats = set() + if source_node is not None: + formats = _shared_formats(source_node["name"], selected_task_name, task_specs) + st.session_state.pipeline_edges.append( + { + "from": selected_source_id, + "to": new_id, + "formats": sorted(formats), + } + ) + st.rerun() + with remove_col: + if st.button("Remove last task node", key="pipeline_builder_remove_last"): + if st.session_state.pipeline_nodes: + removed = st.session_state.pipeline_nodes.pop() + removed_id = removed["id"] + st.session_state.pipeline_edges = [ + edge + for edge in st.session_state.pipeline_edges + if edge.get("from") != removed_id and edge.get("to") != removed_id + ] + st.rerun() + with clear_col: + if st.button("Clear pipeline", key="pipeline_builder_clear"): + st.session_state.pipeline_nodes = [] + st.session_state.pipeline_edges = [] + st.session_state.pop("pipeline_builder_selected_task", None) + st.session_state.pop("pipeline_builder_selected_source", None) + st.rerun() + + if len(st.session_state.pipeline_nodes) >= 2: + st.markdown("**Connect existing tasks**") + id_to_node = {node["id"]: node for node in st.session_state.pipeline_nodes} + source_node_id = st.selectbox( + "From task", + options=[node["id"] for node in st.session_state.pipeline_nodes], + format_func=lambda node_id: _node_label(id_to_node[node_id]), + key="pipeline_builder_connect_source", + ) + existing_pairs = { + (edge.get("from"), edge.get("to")) for edge in st.session_state.pipeline_edges + } + target_candidates: list[tuple[str, str]] = [] + for node in st.session_state.pipeline_nodes: + if node["id"] == source_node_id: + continue + if (source_node_id, node["id"]) in existing_pairs: + continue + source_node = id_to_node[source_node_id] + formats = _shared_formats(source_node["name"], node["name"], task_specs) + if formats: + target_candidates.append( + (node["id"], f'{_node_label(node)} [{_edge_label(formats)}]') + ) + + if target_candidates: + target_label = st.selectbox( + "To task", + options=[label for _, label in target_candidates], + key="pipeline_builder_connect_target", + ) + target_lookup = {label: node_id for node_id, label in target_candidates} + target_node_id = target_lookup[target_label] + if st.button("Add dependency edge", key="pipeline_builder_add_edge"): + source_node = id_to_node[source_node_id] + target_node = id_to_node[target_node_id] + formats = _shared_formats(source_node["name"], target_node["name"], task_specs) + st.session_state.pipeline_edges.append( + { + "from": source_node_id, + "to": target_node_id, + "formats": sorted(formats), + } + ) + st.rerun() + else: + st.caption("No additional compatible target task found for this source.") + + if pipeline_nodes: + st.caption("Current pipeline") + st.write(", ".join(_node_label(node) for node in pipeline_nodes)) + st.markdown("**Pipeline graph**") + render_mermaid(_pipeline_to_mermaid(pipeline_nodes, pipeline_edges), height=300) + + incoming: dict[str, int] = {node["id"]: 0 for node in pipeline_nodes} + outgoing: dict[str, int] = {node["id"]: 0 for node in pipeline_nodes} + for edge in pipeline_edges: + source = edge.get("from") + target = edge.get("to") + if source in outgoing: + outgoing[source] += 1 + if target in incoming: + incoming[target] += 1 + + id_to_node = {node["id"]: node for node in pipeline_nodes} + roots = [id_to_node[node_id] for node_id, count in incoming.items() if count == 0] + leaves = [id_to_node[node_id] for node_id, count in outgoing.items() if count == 0] + + st.markdown("**Loose ends**") + roots_text = ", ".join(_node_label(node) for node in roots) if roots else "none" + leaves_text = ", ".join(_node_label(node) for node in leaves) if leaves else "none" + st.caption(f"Open inputs (roots): {roots_text}") + st.caption(f"Open outputs (leaves): {leaves_text}") + + st.code( + json.dumps( + { + "nodes": pipeline_nodes, + "edges": pipeline_edges, + }, + indent=2, + ), + language="json", + ) + else: + st.info("Your pipeline is empty. Add one or more task nodes to start building.") diff --git a/src/kgpipe_view/tasks_tab.py b/src/kgpipe_view/tasks_tab.py new file mode 100644 index 0000000..b8525ce --- /dev/null +++ b/src/kgpipe_view/tasks_tab.py @@ -0,0 +1,43 @@ +from __future__ import annotations + +import streamlit as st + +try: + from kgpipe_view.meta_kg_query import query_task_hierarchy, query_tasks_implementations +except ModuleNotFoundError: + from meta_kg_query import query_task_hierarchy, query_tasks_implementations + + +def render_tasks_tab() -> str: + endpoint_url = st.text_input( + "Meta KG SPARQL endpoint", + value="http://localhost:8890/sparql", + help="SPARQL endpoint for the live meta knowledge graph.", + ) + if st.button("Load task implementations", type="primary"): + try: + task_implementation_df = query_tasks_implementations(endpoint_url) + except Exception as exc: # pragma: no cover - network dependent path + st.error(f"Could not query `{endpoint_url}`: {exc}") + else: + if task_implementation_df.empty: + st.info("No task-implementation mappings returned by the endpoint.") + else: + st.dataframe(task_implementation_df, use_container_width=True) + + st.divider() + st.subheader("Task hierarchy") + st.caption("Shows subclass relations under `kgp:Task`, including standalone task nodes.") + + if st.button("Load task hierarchy"): + try: + task_hierarchy_df = query_task_hierarchy(endpoint_url) + except Exception as exc: # pragma: no cover - network dependent path + st.error(f"Could not query `{endpoint_url}`: {exc}") + else: + if task_hierarchy_df.empty: + st.info("No `kgp:Task` subclass hierarchy returned by the endpoint.") + else: + st.dataframe(task_hierarchy_df, use_container_width=True) + + return endpoint_url diff --git a/src/kgpipe_view/ui_common.py b/src/kgpipe_view/ui_common.py new file mode 100644 index 0000000..e0fa70d --- /dev/null +++ b/src/kgpipe_view/ui_common.py @@ -0,0 +1,34 @@ +from __future__ import annotations + +import json + +import streamlit.components.v1 as components + + +def render_mermaid(mermaid_text: str, height: int = 900) -> None: + """Render Mermaid source in Streamlit using Mermaid JS.""" + mermaid_json = json.dumps(mermaid_text) + html = f""" + +
+
+
+ + """ + components.html(html, height=height, scrolling=True) diff --git a/uv.lock b/uv.lock new file mode 100644 index 0000000..127684f --- /dev/null +++ b/uv.lock @@ -0,0 +1,3316 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'win32' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'win32' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'emscripten' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda'", +] +conflicts = [[ + { package = "kgpipe", extra = "cpu" }, + { package = "kgpipe", extra = "cuda" }, +]] + +[[package]] +name = "altair" +version = "6.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "jinja2" }, + { name = "jsonschema" }, + { name = "narwhals" }, + { name = "packaging" }, + { name = "typing-extensions", marker = "python_full_version < '3.15' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/3a/1e/365a9144db3254f86f1b974660b9ede1e9a38c9dc0730e4a9b1192eec5d6/altair-6.1.0.tar.gz", hash = "sha256:dda699216cf85b040d968ae5a569ad45957616811e38760a85e5118269daca67", size = 765519, upload-time = "2026-04-21T13:08:46.44Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/63/5dacc8d8306c715088b897a479e551bc0779fd2f0f26c97fec5e36542b4e/altair-6.1.0-py3-none-any.whl", hash = "sha256:fdf5fd939512e5b2fc4441c82dfd2635e706defbd037db0ac429ef5ddce66c3b", size = 796996, upload-time = "2026-04-21T13:08:48.549Z" }, +] + +[[package]] +name = "annotated-doc" +version = "0.0.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/57/ba/046ceea27344560984e26a590f90bc7f4a75b06701f653222458922b558c/annotated_doc-0.0.4.tar.gz", hash = "sha256:fbcda96e87e9c92ad167c2e53839e57503ecfda18804ea28102353485033faa4", size = 7288, upload-time = "2025-11-10T22:07:42.062Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/d3/26bf1008eb3d2daa8ef4cacc7f3bfdc11818d111f7e2d0201bc6e3b49d45/annotated_doc-0.0.4-py3-none-any.whl", hash = "sha256:571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320", size = 5303, upload-time = "2025-11-10T22:07:40.673Z" }, +] + +[[package]] +name = "annotated-types" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ee/67/531ea369ba64dcff5ec9c3402f9f51bf748cec26dde048a2f973a4eea7f5/annotated_types-0.7.0.tar.gz", hash = "sha256:aff07c09a53a08bc8cfccb9c85b05f1aa9a2a6f23728d790723543408344ce89", size = 16081, upload-time = "2024-05-20T21:33:25.928Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl", hash = "sha256:1f02e8b43a8fbbc3f3e0d4f0f4bfc8131bcb4eebe8849b8e5c773f3a1c582a53", size = 13643, upload-time = "2024-05-20T21:33:24.1Z" }, +] + +[[package]] +name = "anyio" +version = "4.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "idna" }, + { name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/14/2c5dd9f512b66549ae92767a9c7b330ae88e1932ca57876909410251fe13/anyio-4.13.0.tar.gz", hash = "sha256:334b70e641fd2221c1505b3890c69882fe4a2df910cba14d97019b90b24439dc", size = 231622, upload-time = "2026-03-24T12:59:09.671Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/da/42/e921fccf5015463e32a3cf6ee7f980a6ed0f395ceeaa45060b61d86486c2/anyio-4.13.0-py3-none-any.whl", hash = "sha256:08b310f9e24a9594186fd75b4f73f4a4152069e3853f1ed8bfbf58369f4ad708", size = 114353, upload-time = "2026-03-24T12:59:08.246Z" }, +] + +[[package]] +name = "attrs" +version = "26.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, +] + +[[package]] +name = "babel" +version = "2.18.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7d/b2/51899539b6ceeeb420d40ed3cd4b7a40519404f9baf3d4ac99dc413a834b/babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d", size = 9959554, upload-time = "2026-02-01T12:30:56.078Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/77/f5/21d2de20e8b8b0408f0681956ca2c69f1320a3848ac50e6e7f39c6159675/babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35", size = 10196845, upload-time = "2026-02-01T12:30:53.445Z" }, +] + +[[package]] +name = "backrefs" +version = "7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5e/a7/a7dd63622beef68cc0d3c3c36d472e143dd95443d5ebf14cd1a5b4dfbf11/backrefs-7.0.tar.gz", hash = "sha256:4989bb9e1e99eb23647c7160ed51fb21d0b41b5d200f2d3017da41e023097e82", size = 7012453, upload-time = "2026-04-28T16:28:04.215Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d4/39/39a31d7eae729ea14ed10c3ccef79371197177b9355a86cb3525709e8502/backrefs-7.0-py310-none-any.whl", hash = "sha256:b57cd227ea556b0aed3dc9b8da4628db4eabc0402c6d7fcfc69283a93955f7e9", size = 380824, upload-time = "2026-04-28T16:27:55.647Z" }, + { url = "https://files.pythonhosted.org/packages/c9/b5/9302644225ba7dfa934a2ff2b9c7bb85701313a90dddb3dfaf693fa5bae2/backrefs-7.0-py311-none-any.whl", hash = "sha256:a0fa7360c63509e9e077e174ef4e6d3c21c8db94189b9d957289ae6d794b9475", size = 392626, upload-time = "2026-04-28T16:27:57.42Z" }, + { url = "https://files.pythonhosted.org/packages/36/da/87912ddec6e06feffbaa3d7aa18fc6352bee2e8f1fee185d7d1690f8f4e8/backrefs-7.0-py312-none-any.whl", hash = "sha256:ca42ce6a49ace3d75684dfa9937f3373902a63284ecb385ce36d15e5dcb41c12", size = 398537, upload-time = "2026-04-28T16:27:58.913Z" }, + { url = "https://files.pythonhosted.org/packages/00/bb/90ba423612b6aa0adccc6b1874bcd4a9b44b660c0c16f346611e00f64ac3/backrefs-7.0-py313-none-any.whl", hash = "sha256:f2c52955d631b9e1ac4cd56209f0a3a946d592b98e7790e77699339ae01c102a", size = 400491, upload-time = "2026-04-28T16:28:00.928Z" }, + { url = "https://files.pythonhosted.org/packages/3e/5c/fb93d3092640a24dfb7bd7727a24016d7c01774ca013e60efd3f683c8002/backrefs-7.0-py314-none-any.whl", hash = "sha256:a6448b28180e3ca01134c9cf09dcebafad8531072e09903c5451748a05f24bc9", size = 412349, upload-time = "2026-04-28T16:28:02.412Z" }, +] + +[[package]] +name = "black" +version = "26.3.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "mypy-extensions" }, + { name = "packaging" }, + { name = "pathspec" }, + { name = "platformdirs" }, + { name = "pytokens" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e1/c5/61175d618685d42b005847464b8fb4743a67b1b8fdb75e50e5a96c31a27a/black-26.3.1.tar.gz", hash = "sha256:2c50f5063a9641c7eed7795014ba37b0f5fa227f3d408b968936e24bc0566b07", size = 666155, upload-time = "2026-03-12T03:36:03.593Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/f8/da5eae4fc75e78e6dceb60624e1b9662ab00d6b452996046dfa9b8a6025b/black-26.3.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b5e6f89631eb88a7302d416594a32faeee9fb8fb848290da9d0a5f2903519fc1", size = 1895920, upload-time = "2026-03-12T03:40:13.921Z" }, + { url = "https://files.pythonhosted.org/packages/2c/9f/04e6f26534da2e1629b2b48255c264cabf5eedc5141d04516d9d68a24111/black-26.3.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:41cd2012d35b47d589cb8a16faf8a32ef7a336f56356babd9fcf70939ad1897f", size = 1718499, upload-time = "2026-03-12T03:40:15.239Z" }, + { url = "https://files.pythonhosted.org/packages/04/91/a5935b2a63e31b331060c4a9fdb5a6c725840858c599032a6f3aac94055f/black-26.3.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f76ff19ec5297dd8e66eb64deda23631e642c9393ab592826fd4bdc97a4bce7", size = 1794994, upload-time = "2026-03-12T03:40:17.124Z" }, + { url = "https://files.pythonhosted.org/packages/e7/0a/86e462cdd311a3c2a8ece708d22aba17d0b2a0d5348ca34b40cdcbea512e/black-26.3.1-cp312-cp312-win_amd64.whl", hash = "sha256:ddb113db38838eb9f043623ba274cfaf7d51d5b0c22ecb30afe58b1bb8322983", size = 1420867, upload-time = "2026-03-12T03:40:18.83Z" }, + { url = "https://files.pythonhosted.org/packages/5b/e5/22515a19cb7eaee3440325a6b0d95d2c0e88dd180cb011b12ae488e031d1/black-26.3.1-cp312-cp312-win_arm64.whl", hash = "sha256:dfdd51fc3e64ea4f35873d1b3fb25326773d55d2329ff8449139ebaad7357efb", size = 1230124, upload-time = "2026-03-12T03:40:20.425Z" }, + { url = "https://files.pythonhosted.org/packages/f5/77/5728052a3c0450c53d9bb3945c4c46b91baa62b2cafab6801411b6271e45/black-26.3.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:855822d90f884905362f602880ed8b5df1b7e3ee7d0db2502d4388a954cc8c54", size = 1895034, upload-time = "2026-03-12T03:40:21.813Z" }, + { url = "https://files.pythonhosted.org/packages/52/73/7cae55fdfdfbe9d19e9a8d25d145018965fe2079fa908101c3733b0c55a0/black-26.3.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8a33d657f3276328ce00e4d37fe70361e1ec7614da5d7b6e78de5426cb56332f", size = 1718503, upload-time = "2026-03-12T03:40:23.666Z" }, + { url = "https://files.pythonhosted.org/packages/e1/87/af89ad449e8254fdbc74654e6467e3c9381b61472cc532ee350d28cfdafb/black-26.3.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f1cd08e99d2f9317292a311dfe578fd2a24b15dbce97792f9c4d752275c1fa56", size = 1793557, upload-time = "2026-03-12T03:40:25.497Z" }, + { url = "https://files.pythonhosted.org/packages/43/10/d6c06a791d8124b843bf325ab4ac7d2f5b98731dff84d6064eafd687ded1/black-26.3.1-cp313-cp313-win_amd64.whl", hash = "sha256:c7e72339f841b5a237ff14f7d3880ddd0fc7f98a1199e8c4327f9a4f478c1839", size = 1422766, upload-time = "2026-03-12T03:40:27.14Z" }, + { url = "https://files.pythonhosted.org/packages/59/4f/40a582c015f2d841ac24fed6390bd68f0fc896069ff3a886317959c9daf8/black-26.3.1-cp313-cp313-win_arm64.whl", hash = "sha256:afc622538b430aa4c8c853f7f63bc582b3b8030fd8c80b70fb5fa5b834e575c2", size = 1232140, upload-time = "2026-03-12T03:40:28.882Z" }, + { url = "https://files.pythonhosted.org/packages/d5/da/e36e27c9cebc1311b7579210df6f1c86e50f2d7143ae4fcf8a5017dc8809/black-26.3.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2d6bfaf7fd0993b420bed691f20f9492d53ce9a2bcccea4b797d34e947318a78", size = 1889234, upload-time = "2026-03-12T03:40:30.964Z" }, + { url = "https://files.pythonhosted.org/packages/0e/7b/9871acf393f64a5fa33668c19350ca87177b181f44bb3d0c33b2d534f22c/black-26.3.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:f89f2ab047c76a9c03f78d0d66ca519e389519902fa27e7a91117ef7611c0568", size = 1720522, upload-time = "2026-03-12T03:40:32.346Z" }, + { url = "https://files.pythonhosted.org/packages/03/87/e766c7f2e90c07fb7586cc787c9ae6462b1eedab390191f2b7fc7f6170a9/black-26.3.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b07fc0dab849d24a80a29cfab8d8a19187d1c4685d8a5e6385a5ce323c1f015f", size = 1787824, upload-time = "2026-03-12T03:40:33.636Z" }, + { url = "https://files.pythonhosted.org/packages/ac/94/2424338fb2d1875e9e83eed4c8e9c67f6905ec25afd826a911aea2b02535/black-26.3.1-cp314-cp314-win_amd64.whl", hash = "sha256:0126ae5b7c09957da2bdbd91a9ba1207453feada9e9fe51992848658c6c8e01c", size = 1445855, upload-time = "2026-03-12T03:40:35.442Z" }, + { url = "https://files.pythonhosted.org/packages/86/43/0c3338bd928afb8ee7471f1a4eec3bdbe2245ccb4a646092a222e8669840/black-26.3.1-cp314-cp314-win_arm64.whl", hash = "sha256:92c0ec1f2cc149551a2b7b47efc32c866406b6891b0ee4625e95967c8f4acfb1", size = 1258109, upload-time = "2026-03-12T03:40:36.832Z" }, + { url = "https://files.pythonhosted.org/packages/8e/0d/52d98722666d6fc6c3dd4c76df339501d6efd40e0ff95e6186a7b7f0befd/black-26.3.1-py3-none-any.whl", hash = "sha256:2bd5aa94fc267d38bb21a70d7410a89f1a1d318841855f698746f8e7f51acd1b", size = 207542, upload-time = "2026-03-12T03:36:01.668Z" }, +] + +[[package]] +name = "blinker" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/28/9b3f50ce0e048515135495f198351908d99540d69bfdc8c1d15b73dc55ce/blinker-1.9.0.tar.gz", hash = "sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf", size = 22460, upload-time = "2024-11-08T17:25:47.436Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl", hash = "sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc", size = 8458, upload-time = "2024-11-08T17:25:46.184Z" }, +] + +[[package]] +name = "cachetools" +version = "7.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ff/e2/85f227594656000ff4d8adadae91a21f536d4a84c6c716a86bd6685874be/cachetools-7.1.1.tar.gz", hash = "sha256:27bdf856d68fd3c71c26c01b5edc312124ed427524d1ddb31aa2b7746fe20d4b", size = 40202, upload-time = "2026-05-03T20:00:29.391Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bf/0f/f897abe4ea0a8c408ae65c8c83bffab4936ad65d6032d4fb4cd35bbdc3ee/cachetools-7.1.1-py3-none-any.whl", hash = "sha256:0335cd7a0952d2b22327441fb0628139e234c565559eeb91a8a4ac7551c5353d", size = 16775, upload-time = "2026-05-03T20:00:27.857Z" }, +] + +[[package]] +name = "certifi" +version = "2026.4.22" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/25/ee/6caf7a40c36a1220410afe15a1cc64993a1f864871f698c0f93acb72842a/certifi-2026.4.22.tar.gz", hash = "sha256:8d455352a37b71bf76a79caa83a3d6c25afee4a385d632127b6afb3963f1c580", size = 137077, upload-time = "2026-04-22T11:26:11.191Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/22/30/7cd8fdcdfbc5b869528b079bfb76dcdf6056b1a2097a662e5e8c04f42965/certifi-2026.4.22-py3-none-any.whl", hash = "sha256:3cb2210c8f88ba2318d29b0388d1023c8492ff72ecdde4ebdaddbb13a31b1c4a", size = 135707, upload-time = "2026-04-22T11:26:09.372Z" }, +] + +[[package]] +name = "charset-normalizer" +version = "3.4.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e7/a1/67fe25fac3c7642725500a3f6cfe5821ad557c3abb11c9d20d12c7008d3e/charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5", size = 144271, upload-time = "2026-04-02T09:28:39.342Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0c/eb/4fc8d0a7110eb5fc9cc161723a34a8a6c200ce3b4fbf681bc86feee22308/charset_normalizer-3.4.7-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:eca9705049ad3c7345d574e3510665cb2cf844c2f2dcfe675332677f081cbd46", size = 311328, upload-time = "2026-04-02T09:26:24.331Z" }, + { url = "https://files.pythonhosted.org/packages/f8/e3/0fadc706008ac9d7b9b5be6dc767c05f9d3e5df51744ce4cc9605de7b9f4/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6178f72c5508bfc5fd446a5905e698c6212932f25bcdd4b47a757a50605a90e2", size = 208061, upload-time = "2026-04-02T09:26:25.568Z" }, + { url = "https://files.pythonhosted.org/packages/42/f0/3dd1045c47f4a4604df85ec18ad093912ae1344ac706993aff91d38773a2/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e1421b502d83040e6d7fb2fb18dff63957f720da3d77b2fbd3187ceb63755d7b", size = 229031, upload-time = "2026-04-02T09:26:26.865Z" }, + { url = "https://files.pythonhosted.org/packages/dc/67/675a46eb016118a2fbde5a277a5d15f4f69d5f3f5f338e5ee2f8948fcf43/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:edac0f1ab77644605be2cbba52e6b7f630731fc42b34cb0f634be1a6eface56a", size = 225239, upload-time = "2026-04-02T09:26:28.044Z" }, + { url = "https://files.pythonhosted.org/packages/4b/f8/d0118a2f5f23b02cd166fa385c60f9b0d4f9194f574e2b31cef350ad7223/charset_normalizer-3.4.7-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5649fd1c7bade02f320a462fdefd0b4bd3ce036065836d4f42e0de958038e116", size = 216589, upload-time = "2026-04-02T09:26:29.239Z" }, + { url = "https://files.pythonhosted.org/packages/b1/f1/6d2b0b261b6c4ceef0fcb0d17a01cc5bc53586c2d4796fa04b5c540bc13d/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:203104ed3e428044fd943bc4bf45fa73c0730391f9621e37fe39ecf477b128cb", size = 202733, upload-time = "2026-04-02T09:26:30.5Z" }, + { url = "https://files.pythonhosted.org/packages/6f/c0/7b1f943f7e87cc3db9626ba17807d042c38645f0a1d4415c7a14afb5591f/charset_normalizer-3.4.7-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:298930cec56029e05497a76988377cbd7457ba864beeea92ad7e844fe74cd1f1", size = 212652, upload-time = "2026-04-02T09:26:31.709Z" }, + { url = "https://files.pythonhosted.org/packages/38/dd/5a9ab159fe45c6e72079398f277b7d2b523e7f716acc489726115a910097/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:708838739abf24b2ceb208d0e22403dd018faeef86ddac04319a62ae884c4f15", size = 211229, upload-time = "2026-04-02T09:26:33.282Z" }, + { url = "https://files.pythonhosted.org/packages/d5/ff/531a1cad5ca855d1c1a8b69cb71abfd6d85c0291580146fda7c82857caa1/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:0f7eb884681e3938906ed0434f20c63046eacd0111c4ba96f27b76084cd679f5", size = 203552, upload-time = "2026-04-02T09:26:34.845Z" }, + { url = "https://files.pythonhosted.org/packages/c1/4c/a5fb52d528a8ca41f7598cb619409ece30a169fbdf9cdce592e53b46c3a6/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:4dc1e73c36828f982bfe79fadf5919923f8a6f4df2860804db9a98c48824ce8d", size = 230806, upload-time = "2026-04-02T09:26:36.152Z" }, + { url = "https://files.pythonhosted.org/packages/59/7a/071feed8124111a32b316b33ae4de83d36923039ef8cf48120266844285b/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:aed52fea0513bac0ccde438c188c8a471c4e0f457c2dd20cdbf6ea7a450046c7", size = 212316, upload-time = "2026-04-02T09:26:37.672Z" }, + { url = "https://files.pythonhosted.org/packages/fd/35/f7dba3994312d7ba508e041eaac39a36b120f32d4c8662b8814dab876431/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:fea24543955a6a729c45a73fe90e08c743f0b3334bbf3201e6c4bc1b0c7fa464", size = 227274, upload-time = "2026-04-02T09:26:38.93Z" }, + { url = "https://files.pythonhosted.org/packages/8a/2d/a572df5c9204ab7688ec1edc895a73ebded3b023bb07364710b05dd1c9be/charset_normalizer-3.4.7-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bb6d88045545b26da47aa879dd4a89a71d1dce0f0e549b1abcb31dfe4a8eac49", size = 218468, upload-time = "2026-04-02T09:26:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/86/eb/890922a8b03a568ca2f336c36585a4713c55d4d67bf0f0c78924be6315ca/charset_normalizer-3.4.7-cp312-cp312-win32.whl", hash = "sha256:2257141f39fe65a3fdf38aeccae4b953e5f3b3324f4ff0daf9f15b8518666a2c", size = 148460, upload-time = "2026-04-02T09:26:41.416Z" }, + { url = "https://files.pythonhosted.org/packages/35/d9/0e7dffa06c5ab081f75b1b786f0aefc88365825dfcd0ac544bdb7b2b6853/charset_normalizer-3.4.7-cp312-cp312-win_amd64.whl", hash = "sha256:5ed6ab538499c8644b8a3e18debabcd7ce684f3fa91cf867521a7a0279cab2d6", size = 159330, upload-time = "2026-04-02T09:26:42.554Z" }, + { url = "https://files.pythonhosted.org/packages/9e/5d/481bcc2a7c88ea6b0878c299547843b2521ccbc40980cb406267088bc701/charset_normalizer-3.4.7-cp312-cp312-win_arm64.whl", hash = "sha256:56be790f86bfb2c98fb742ce566dfb4816e5a83384616ab59c49e0604d49c51d", size = 147828, upload-time = "2026-04-02T09:26:44.075Z" }, + { url = "https://files.pythonhosted.org/packages/c1/3b/66777e39d3ae1ddc77ee606be4ec6d8cbd4c801f65e5a1b6f2b11b8346dd/charset_normalizer-3.4.7-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:f496c9c3cc02230093d8330875c4c3cdfc3b73612a5fd921c65d39cbcef08063", size = 309627, upload-time = "2026-04-02T09:26:45.198Z" }, + { url = "https://files.pythonhosted.org/packages/2e/4e/b7f84e617b4854ade48a1b7915c8ccfadeba444d2a18c291f696e37f0d3b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea948db76d31190bf08bd371623927ee1339d5f2a0b4b1b4a4439a65298703c", size = 207008, upload-time = "2026-04-02T09:26:46.824Z" }, + { url = "https://files.pythonhosted.org/packages/c4/bb/ec73c0257c9e11b268f018f068f5d00aa0ef8c8b09f7753ebd5f2880e248/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a277ab8928b9f299723bc1a2dabb1265911b1a76341f90a510368ca44ad9ab66", size = 228303, upload-time = "2026-04-02T09:26:48.397Z" }, + { url = "https://files.pythonhosted.org/packages/85/fb/32d1f5033484494619f701e719429c69b766bfc4dbc61aa9e9c8c166528b/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3bec022aec2c514d9cf199522a802bd007cd588ab17ab2525f20f9c34d067c18", size = 224282, upload-time = "2026-04-02T09:26:49.684Z" }, + { url = "https://files.pythonhosted.org/packages/fa/07/330e3a0dda4c404d6da83b327270906e9654a24f6c546dc886a0eb0ffb23/charset_normalizer-3.4.7-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e044c39e41b92c845bc815e5ae4230804e8e7bc29e399b0437d64222d92809dd", size = 215595, upload-time = "2026-04-02T09:26:50.915Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7c/fc890655786e423f02556e0216d4b8c6bcb6bdfa890160dc66bf52dee468/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:f495a1652cf3fbab2eb0639776dad966c2fb874d79d87ca07f9d5f059b8bd215", size = 201986, upload-time = "2026-04-02T09:26:52.197Z" }, + { url = "https://files.pythonhosted.org/packages/d8/97/bfb18b3db2aed3b90cf54dc292ad79fdd5ad65c4eae454099475cbeadd0d/charset_normalizer-3.4.7-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e712b419df8ba5e42b226c510472b37bd57b38e897d3eca5e8cfd410a29fa859", size = 211711, upload-time = "2026-04-02T09:26:53.49Z" }, + { url = "https://files.pythonhosted.org/packages/6f/a5/a581c13798546a7fd557c82614a5c65a13df2157e9ad6373166d2a3e645d/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7804338df6fcc08105c7745f1502ba68d900f45fd770d5bdd5288ddccb8a42d8", size = 210036, upload-time = "2026-04-02T09:26:54.975Z" }, + { url = "https://files.pythonhosted.org/packages/8c/bf/b3ab5bcb478e4193d517644b0fb2bf5497fbceeaa7a1bc0f4d5b50953861/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:481551899c856c704d58119b5025793fa6730adda3571971af568f66d2424bb5", size = 202998, upload-time = "2026-04-02T09:26:56.303Z" }, + { url = "https://files.pythonhosted.org/packages/e7/4e/23efd79b65d314fa320ec6017b4b5834d5c12a58ba4610aa353af2e2f577/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:f59099f9b66f0d7145115e6f80dd8b1d847176df89b234a5a6b3f00437aa0832", size = 230056, upload-time = "2026-04-02T09:26:57.554Z" }, + { url = "https://files.pythonhosted.org/packages/b9/9f/1e1941bc3f0e01df116e68dc37a55c4d249df5e6fa77f008841aef68264f/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f59ad4c0e8f6bba240a9bb85504faa1ab438237199d4cce5f622761507b8f6a6", size = 211537, upload-time = "2026-04-02T09:26:58.843Z" }, + { url = "https://files.pythonhosted.org/packages/80/0f/088cbb3020d44428964a6c97fe1edfb1b9550396bf6d278330281e8b709c/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:3dedcc22d73ec993f42055eff4fcfed9318d1eeb9a6606c55892a26964964e48", size = 226176, upload-time = "2026-04-02T09:27:00.437Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9f/130394f9bbe06f4f63e22641d32fc9b202b7e251c9aef4db044324dac493/charset_normalizer-3.4.7-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:64f02c6841d7d83f832cd97ccf8eb8a906d06eb95d5276069175c696b024b60a", size = 217723, upload-time = "2026-04-02T09:27:02.021Z" }, + { url = "https://files.pythonhosted.org/packages/73/55/c469897448a06e49f8fa03f6caae97074fde823f432a98f979cc42b90e69/charset_normalizer-3.4.7-cp313-cp313-win32.whl", hash = "sha256:4042d5c8f957e15221d423ba781e85d553722fc4113f523f2feb7b188cc34c5e", size = 148085, upload-time = "2026-04-02T09:27:03.192Z" }, + { url = "https://files.pythonhosted.org/packages/5d/78/1b74c5bbb3f99b77a1715c91b3e0b5bdb6fe302d95ace4f5b1bec37b0167/charset_normalizer-3.4.7-cp313-cp313-win_amd64.whl", hash = "sha256:3946fa46a0cf3e4c8cb1cc52f56bb536310d34f25f01ca9b6c16afa767dab110", size = 158819, upload-time = "2026-04-02T09:27:04.454Z" }, + { url = "https://files.pythonhosted.org/packages/68/86/46bd42279d323deb8687c4a5a811fd548cb7d1de10cf6535d099877a9a9f/charset_normalizer-3.4.7-cp313-cp313-win_arm64.whl", hash = "sha256:80d04837f55fc81da168b98de4f4b797ef007fc8a79ab71c6ec9bc4dd662b15b", size = 147915, upload-time = "2026-04-02T09:27:05.971Z" }, + { url = "https://files.pythonhosted.org/packages/97/c8/c67cb8c70e19ef1960b97b22ed2a1567711de46c4ddf19799923adc836c2/charset_normalizer-3.4.7-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:c36c333c39be2dbca264d7803333c896ab8fa7d4d6f0ab7edb7dfd7aea6e98c0", size = 309234, upload-time = "2026-04-02T09:27:07.194Z" }, + { url = "https://files.pythonhosted.org/packages/99/85/c091fdee33f20de70d6c8b522743b6f831a2f1cd3ff86de4c6a827c48a76/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1c2aed2e5e41f24ea8ef1590b8e848a79b56f3a5564a65ceec43c9d692dc7d8a", size = 208042, upload-time = "2026-04-02T09:27:08.749Z" }, + { url = "https://files.pythonhosted.org/packages/87/1c/ab2ce611b984d2fd5d86a5a8a19c1ae26acac6bad967da4967562c75114d/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:54523e136b8948060c0fa0bc7b1b50c32c186f2fceee897a495406bb6e311d2b", size = 228706, upload-time = "2026-04-02T09:27:09.951Z" }, + { url = "https://files.pythonhosted.org/packages/a8/29/2b1d2cb00bf085f59d29eb773ce58ec2d325430f8c216804a0a5cd83cbca/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:715479b9a2802ecac752a3b0efa2b0b60285cf962ee38414211abdfccc233b41", size = 224727, upload-time = "2026-04-02T09:27:11.175Z" }, + { url = "https://files.pythonhosted.org/packages/47/5c/032c2d5a07fe4d4855fea851209cca2b6f03ebeb6d4e3afdb3358386a684/charset_normalizer-3.4.7-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bd6c2a1c7573c64738d716488d2cdd3c00e340e4835707d8fdb8dc1a66ef164e", size = 215882, upload-time = "2026-04-02T09:27:12.446Z" }, + { url = "https://files.pythonhosted.org/packages/2c/c2/356065d5a8b78ed04499cae5f339f091946a6a74f91e03476c33f0ab7100/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:c45e9440fb78f8ddabcf714b68f936737a121355bf59f3907f4e17721b9d1aae", size = 200860, upload-time = "2026-04-02T09:27:13.721Z" }, + { url = "https://files.pythonhosted.org/packages/0c/cd/a32a84217ced5039f53b29f460962abb2d4420def55afabe45b1c3c7483d/charset_normalizer-3.4.7-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3534e7dcbdcf757da6b85a0bbf5b6868786d5982dd959b065e65481644817a18", size = 211564, upload-time = "2026-04-02T09:27:15.272Z" }, + { url = "https://files.pythonhosted.org/packages/44/86/58e6f13ce26cc3b8f4a36b94a0f22ae2f00a72534520f4ae6857c4b81f89/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e8ac484bf18ce6975760921bb6148041faa8fef0547200386ea0b52b5d27bf7b", size = 211276, upload-time = "2026-04-02T09:27:16.834Z" }, + { url = "https://files.pythonhosted.org/packages/8f/fe/d17c32dc72e17e155e06883efa84514ca375f8a528ba2546bee73fc4df81/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:a5fe03b42827c13cdccd08e6c0247b6a6d4b5e3cdc53fd1749f5896adcdc2356", size = 201238, upload-time = "2026-04-02T09:27:18.229Z" }, + { url = "https://files.pythonhosted.org/packages/6a/29/f33daa50b06525a237451cdb6c69da366c381a3dadcd833fa5676bc468b3/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2d6eb928e13016cea4f1f21d1e10c1cebd5a421bc57ddf5b1142ae3f86824fab", size = 230189, upload-time = "2026-04-02T09:27:19.445Z" }, + { url = "https://files.pythonhosted.org/packages/b6/6e/52c84015394a6a0bdcd435210a7e944c5f94ea1055f5cc5d56c5fe368e7b/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e74327fb75de8986940def6e8dee4f127cc9752bee7355bb323cc5b2659b6d46", size = 211352, upload-time = "2026-04-02T09:27:20.79Z" }, + { url = "https://files.pythonhosted.org/packages/8c/d7/4353be581b373033fb9198bf1da3cf8f09c1082561e8e922aa7b39bf9fe8/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d6038d37043bced98a66e68d3aa2b6a35505dc01328cd65217cefe82f25def44", size = 227024, upload-time = "2026-04-02T09:27:22.063Z" }, + { url = "https://files.pythonhosted.org/packages/30/45/99d18aa925bd1740098ccd3060e238e21115fffbfdcb8f3ece837d0ace6c/charset_normalizer-3.4.7-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7579e913a5339fb8fa133f6bbcfd8e6749696206cf05acdbdca71a1b436d8e72", size = 217869, upload-time = "2026-04-02T09:27:23.486Z" }, + { url = "https://files.pythonhosted.org/packages/5c/05/5ee478aa53f4bb7996482153d4bfe1b89e0f087f0ab6b294fcf92d595873/charset_normalizer-3.4.7-cp314-cp314-win32.whl", hash = "sha256:5b77459df20e08151cd6f8b9ef8ef1f961ef73d85c21a555c7eed5b79410ec10", size = 148541, upload-time = "2026-04-02T09:27:25.146Z" }, + { url = "https://files.pythonhosted.org/packages/48/77/72dcb0921b2ce86420b2d79d454c7022bf5be40202a2a07906b9f2a35c97/charset_normalizer-3.4.7-cp314-cp314-win_amd64.whl", hash = "sha256:92a0a01ead5e668468e952e4238cccd7c537364eb7d851ab144ab6627dbbe12f", size = 159634, upload-time = "2026-04-02T09:27:26.642Z" }, + { url = "https://files.pythonhosted.org/packages/c6/a3/c2369911cd72f02386e4e340770f6e158c7980267da16af8f668217abaa0/charset_normalizer-3.4.7-cp314-cp314-win_arm64.whl", hash = "sha256:67f6279d125ca0046a7fd386d01b311c6363844deac3e5b069b514ba3e63c246", size = 148384, upload-time = "2026-04-02T09:27:28.271Z" }, + { url = "https://files.pythonhosted.org/packages/94/09/7e8a7f73d24dba1f0035fbbf014d2c36828fc1bf9c88f84093e57d315935/charset_normalizer-3.4.7-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:effc3f449787117233702311a1b7d8f59cba9ced946ba727bdc329ec69028e24", size = 330133, upload-time = "2026-04-02T09:27:29.474Z" }, + { url = "https://files.pythonhosted.org/packages/8d/da/96975ddb11f8e977f706f45cddd8540fd8242f71ecdb5d18a80723dcf62c/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fbccdc05410c9ee21bbf16a35f4c1d16123dcdeb8a1d38f33654fa21d0234f79", size = 216257, upload-time = "2026-04-02T09:27:30.793Z" }, + { url = "https://files.pythonhosted.org/packages/e5/e8/1d63bf8ef2d388e95c64b2098f45f84758f6d102a087552da1485912637b/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:733784b6d6def852c814bce5f318d25da2ee65dd4839a0718641c696e09a2960", size = 234851, upload-time = "2026-04-02T09:27:32.44Z" }, + { url = "https://files.pythonhosted.org/packages/9b/40/e5ff04233e70da2681fa43969ad6f66ca5611d7e669be0246c4c7aaf6dc8/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a89c23ef8d2c6b27fd200a42aa4ac72786e7c60d40efdc76e6011260b6e949c4", size = 233393, upload-time = "2026-04-02T09:27:34.03Z" }, + { url = "https://files.pythonhosted.org/packages/be/c1/06c6c49d5a5450f76899992f1ee40b41d076aee9279b49cf9974d2f313d5/charset_normalizer-3.4.7-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c114670c45346afedc0d947faf3c7f701051d2518b943679c8ff88befe14f8e", size = 223251, upload-time = "2026-04-02T09:27:35.369Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f2ff16fb050946169e3e1f82134d107e5d4ae72647ec8a1b1446c148480f/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:a180c5e59792af262bf263b21a3c49353f25945d8d9f70628e73de370d55e1e1", size = 206609, upload-time = "2026-04-02T09:27:36.661Z" }, + { url = "https://files.pythonhosted.org/packages/69/d5/a527c0cd8d64d2eab7459784fb4169a0ac76e5a6fc5237337982fd61347e/charset_normalizer-3.4.7-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3c9a494bc5ec77d43cea229c4f6db1e4d8fe7e1bbffa8b6f0f0032430ff8ab44", size = 220014, upload-time = "2026-04-02T09:27:38.019Z" }, + { url = "https://files.pythonhosted.org/packages/7e/80/8a7b8104a3e203074dc9aa2c613d4b726c0e136bad1cc734594b02867972/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8d828b6667a32a728a1ad1d93957cdf37489c57b97ae6c4de2860fa749b8fc1e", size = 218979, upload-time = "2026-04-02T09:27:39.37Z" }, + { url = "https://files.pythonhosted.org/packages/02/9a/b759b503d507f375b2b5c153e4d2ee0a75aa215b7f2489cf314f4541f2c0/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf1493cd8607bec4d8a7b9b004e699fcf8f9103a9284cc94962cb73d20f9d4a3", size = 209238, upload-time = "2026-04-02T09:27:40.722Z" }, + { url = "https://files.pythonhosted.org/packages/c2/4e/0f3f5d47b86bdb79256e7290b26ac847a2832d9a4033f7eb2cd4bcf4bb5b/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0c96c3b819b5c3e9e165495db84d41914d6894d55181d2d108cc1a69bfc9cce0", size = 236110, upload-time = "2026-04-02T09:27:42.33Z" }, + { url = "https://files.pythonhosted.org/packages/96/23/bce28734eb3ed2c91dcf93abeb8a5cf393a7b2749725030bb630e554fdd8/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:752a45dc4a6934060b3b0dab47e04edc3326575f82be64bc4fc293914566503e", size = 219824, upload-time = "2026-04-02T09:27:43.924Z" }, + { url = "https://files.pythonhosted.org/packages/2c/6f/6e897c6984cc4d41af319b077f2f600fc8214eb2fe2d6bcb79141b882400/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:8778f0c7a52e56f75d12dae53ae320fae900a8b9b4164b981b9c5ce059cd1fcb", size = 233103, upload-time = "2026-04-02T09:27:45.348Z" }, + { url = "https://files.pythonhosted.org/packages/76/22/ef7bd0fe480a0ae9b656189ec00744b60933f68b4f42a7bb06589f6f576a/charset_normalizer-3.4.7-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ce3412fbe1e31eb81ea42f4169ed94861c56e643189e1e75f0041f3fe7020abe", size = 225194, upload-time = "2026-04-02T09:27:46.706Z" }, + { url = "https://files.pythonhosted.org/packages/c5/a7/0e0ab3e0b5bc1219bd80a6a0d4d72ca74d9250cb2382b7c699c147e06017/charset_normalizer-3.4.7-cp314-cp314t-win32.whl", hash = "sha256:c03a41a8784091e67a39648f70c5f97b5b6a37f216896d44d2cdcb82615339a0", size = 159827, upload-time = "2026-04-02T09:27:48.053Z" }, + { url = "https://files.pythonhosted.org/packages/7a/1d/29d32e0fb40864b1f878c7f5a0b343ae676c6e2b271a2d55cc3a152391da/charset_normalizer-3.4.7-cp314-cp314t-win_amd64.whl", hash = "sha256:03853ed82eeebbce3c2abfdbc98c96dc205f32a79627688ac9a27370ea61a49c", size = 174168, upload-time = "2026-04-02T09:27:49.795Z" }, + { url = "https://files.pythonhosted.org/packages/de/32/d92444ad05c7a6e41fb2036749777c163baf7a0301a040cb672d6b2b1ae9/charset_normalizer-3.4.7-cp314-cp314t-win_arm64.whl", hash = "sha256:c35abb8bfff0185efac5878da64c45dafd2b37fb0383add1be155a763c1f083d", size = 153018, upload-time = "2026-04-02T09:27:51.116Z" }, + { url = "https://files.pythonhosted.org/packages/db/8f/61959034484a4a7c527811f4721e75d02d653a35afb0b6054474d8185d4c/charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d", size = 61958, upload-time = "2026-04-02T09:28:37.794Z" }, +] + +[[package]] +name = "click" +version = "8.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bb/63/f9e1ea081ce35720d8b92acde70daaedace594dc93b693c869e0d5910718/click-8.3.3.tar.gz", hash = "sha256:398329ad4837b2ff7cbe1dd166a4c0f8900c3ca3a218de04466f38f6497f18a2", size = 328061, upload-time = "2026-04-22T15:11:27.506Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ae/44/c1221527f6a71a01ec6fbad7fa78f1d50dfa02217385cf0fa3eec7087d59/click-8.3.3-py3-none-any.whl", hash = "sha256:a2bf429bb3033c89fa4936ffb35d5cb471e3719e1f3c8a7c3fff0b8314305613", size = 110502, upload-time = "2026-04-22T15:11:25.044Z" }, +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "contourpy" +version = "1.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/58/01/1253e6698a07380cd31a736d248a3f2a50a7c88779a1813da27503cadc2a/contourpy-1.3.3.tar.gz", hash = "sha256:083e12155b210502d0bca491432bb04d56dc3432f95a979b429f2848c3dbe880", size = 13466174, upload-time = "2025-07-26T12:03:12.549Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/be/45/adfee365d9ea3d853550b2e735f9d66366701c65db7855cd07621732ccfc/contourpy-1.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b08a32ea2f8e42cf1d4be3169a98dd4be32bafe4f22b6c4cb4ba810fa9e5d2cb", size = 293419, upload-time = "2025-07-26T12:01:21.16Z" }, + { url = "https://files.pythonhosted.org/packages/53/3e/405b59cfa13021a56bba395a6b3aca8cec012b45bf177b0eaf7a202cde2c/contourpy-1.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:556dba8fb6f5d8742f2923fe9457dbdd51e1049c4a43fd3986a0b14a1d815fc6", size = 273979, upload-time = "2025-07-26T12:01:22.448Z" }, + { url = "https://files.pythonhosted.org/packages/d4/1c/a12359b9b2ca3a845e8f7f9ac08bdf776114eb931392fcad91743e2ea17b/contourpy-1.3.3-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92d9abc807cf7d0e047b95ca5d957cf4792fcd04e920ca70d48add15c1a90ea7", size = 332653, upload-time = "2025-07-26T12:01:24.155Z" }, + { url = "https://files.pythonhosted.org/packages/63/12/897aeebfb475b7748ea67b61e045accdfcf0d971f8a588b67108ed7f5512/contourpy-1.3.3-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2e8faa0ed68cb29af51edd8e24798bb661eac3bd9f65420c1887b6ca89987c8", size = 379536, upload-time = "2025-07-26T12:01:25.91Z" }, + { url = "https://files.pythonhosted.org/packages/43/8a/a8c584b82deb248930ce069e71576fc09bd7174bbd35183b7943fb1064fd/contourpy-1.3.3-cp312-cp312-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:626d60935cf668e70a5ce6ff184fd713e9683fb458898e4249b63be9e28286ea", size = 384397, upload-time = "2025-07-26T12:01:27.152Z" }, + { url = "https://files.pythonhosted.org/packages/cc/8f/ec6289987824b29529d0dfda0d74a07cec60e54b9c92f3c9da4c0ac732de/contourpy-1.3.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d00e655fcef08aba35ec9610536bfe90267d7ab5ba944f7032549c55a146da1", size = 362601, upload-time = "2025-07-26T12:01:28.808Z" }, + { url = "https://files.pythonhosted.org/packages/05/0a/a3fe3be3ee2dceb3e615ebb4df97ae6f3828aa915d3e10549ce016302bd1/contourpy-1.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:451e71b5a7d597379ef572de31eeb909a87246974d960049a9848c3bc6c41bf7", size = 1331288, upload-time = "2025-07-26T12:01:31.198Z" }, + { url = "https://files.pythonhosted.org/packages/33/1d/acad9bd4e97f13f3e2b18a3977fe1b4a37ecf3d38d815333980c6c72e963/contourpy-1.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:459c1f020cd59fcfe6650180678a9993932d80d44ccde1fa1868977438f0b411", size = 1403386, upload-time = "2025-07-26T12:01:33.947Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8f/5847f44a7fddf859704217a99a23a4f6417b10e5ab1256a179264561540e/contourpy-1.3.3-cp312-cp312-win32.whl", hash = "sha256:023b44101dfe49d7d53932be418477dba359649246075c996866106da069af69", size = 185018, upload-time = "2025-07-26T12:01:35.64Z" }, + { url = "https://files.pythonhosted.org/packages/19/e8/6026ed58a64563186a9ee3f29f41261fd1828f527dd93d33b60feca63352/contourpy-1.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:8153b8bfc11e1e4d75bcb0bff1db232f9e10b274e0929de9d608027e0d34ff8b", size = 226567, upload-time = "2025-07-26T12:01:36.804Z" }, + { url = "https://files.pythonhosted.org/packages/d1/e2/f05240d2c39a1ed228d8328a78b6f44cd695f7ef47beb3e684cf93604f86/contourpy-1.3.3-cp312-cp312-win_arm64.whl", hash = "sha256:07ce5ed73ecdc4a03ffe3e1b3e3c1166db35ae7584be76f65dbbe28a7791b0cc", size = 193655, upload-time = "2025-07-26T12:01:37.999Z" }, + { url = "https://files.pythonhosted.org/packages/68/35/0167aad910bbdb9599272bd96d01a9ec6852f36b9455cf2ca67bd4cc2d23/contourpy-1.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:177fb367556747a686509d6fef71d221a4b198a3905fe824430e5ea0fda54eb5", size = 293257, upload-time = "2025-07-26T12:01:39.367Z" }, + { url = "https://files.pythonhosted.org/packages/96/e4/7adcd9c8362745b2210728f209bfbcf7d91ba868a2c5f40d8b58f54c509b/contourpy-1.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:d002b6f00d73d69333dac9d0b8d5e84d9724ff9ef044fd63c5986e62b7c9e1b1", size = 274034, upload-time = "2025-07-26T12:01:40.645Z" }, + { url = "https://files.pythonhosted.org/packages/73/23/90e31ceeed1de63058a02cb04b12f2de4b40e3bef5e082a7c18d9c8ae281/contourpy-1.3.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:348ac1f5d4f1d66d3322420f01d42e43122f43616e0f194fc1c9f5d830c5b286", size = 334672, upload-time = "2025-07-26T12:01:41.942Z" }, + { url = "https://files.pythonhosted.org/packages/ed/93/b43d8acbe67392e659e1d984700e79eb67e2acb2bd7f62012b583a7f1b55/contourpy-1.3.3-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:655456777ff65c2c548b7c454af9c6f33f16c8884f11083244b5819cc214f1b5", size = 381234, upload-time = "2025-07-26T12:01:43.499Z" }, + { url = "https://files.pythonhosted.org/packages/46/3b/bec82a3ea06f66711520f75a40c8fc0b113b2a75edb36aa633eb11c4f50f/contourpy-1.3.3-cp313-cp313-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:644a6853d15b2512d67881586bd03f462c7ab755db95f16f14d7e238f2852c67", size = 385169, upload-time = "2025-07-26T12:01:45.219Z" }, + { url = "https://files.pythonhosted.org/packages/4b/32/e0f13a1c5b0f8572d0ec6ae2f6c677b7991fafd95da523159c19eff0696a/contourpy-1.3.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4debd64f124ca62069f313a9cb86656ff087786016d76927ae2cf37846b006c9", size = 362859, upload-time = "2025-07-26T12:01:46.519Z" }, + { url = "https://files.pythonhosted.org/packages/33/71/e2a7945b7de4e58af42d708a219f3b2f4cff7386e6b6ab0a0fa0033c49a9/contourpy-1.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a15459b0f4615b00bbd1e91f1b9e19b7e63aea7483d03d804186f278c0af2659", size = 1332062, upload-time = "2025-07-26T12:01:48.964Z" }, + { url = "https://files.pythonhosted.org/packages/12/fc/4e87ac754220ccc0e807284f88e943d6d43b43843614f0a8afa469801db0/contourpy-1.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ca0fdcd73925568ca027e0b17ab07aad764be4706d0a925b89227e447d9737b7", size = 1403932, upload-time = "2025-07-26T12:01:51.979Z" }, + { url = "https://files.pythonhosted.org/packages/a6/2e/adc197a37443f934594112222ac1aa7dc9a98faf9c3842884df9a9d8751d/contourpy-1.3.3-cp313-cp313-win32.whl", hash = "sha256:b20c7c9a3bf701366556e1b1984ed2d0cedf999903c51311417cf5f591d8c78d", size = 185024, upload-time = "2025-07-26T12:01:53.245Z" }, + { url = "https://files.pythonhosted.org/packages/18/0b/0098c214843213759692cc638fce7de5c289200a830e5035d1791d7a2338/contourpy-1.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:1cadd8b8969f060ba45ed7c1b714fe69185812ab43bd6b86a9123fe8f99c3263", size = 226578, upload-time = "2025-07-26T12:01:54.422Z" }, + { url = "https://files.pythonhosted.org/packages/8a/9a/2f6024a0c5995243cd63afdeb3651c984f0d2bc727fd98066d40e141ad73/contourpy-1.3.3-cp313-cp313-win_arm64.whl", hash = "sha256:fd914713266421b7536de2bfa8181aa8c699432b6763a0ea64195ebe28bff6a9", size = 193524, upload-time = "2025-07-26T12:01:55.73Z" }, + { url = "https://files.pythonhosted.org/packages/c0/b3/f8a1a86bd3298513f500e5b1f5fd92b69896449f6cab6a146a5d52715479/contourpy-1.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:88df9880d507169449d434c293467418b9f6cbe82edd19284aa0409e7fdb933d", size = 306730, upload-time = "2025-07-26T12:01:57.051Z" }, + { url = "https://files.pythonhosted.org/packages/3f/11/4780db94ae62fc0c2053909b65dc3246bd7cecfc4f8a20d957ad43aa4ad8/contourpy-1.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:d06bb1f751ba5d417047db62bca3c8fde202b8c11fb50742ab3ab962c81e8216", size = 287897, upload-time = "2025-07-26T12:01:58.663Z" }, + { url = "https://files.pythonhosted.org/packages/ae/15/e59f5f3ffdd6f3d4daa3e47114c53daabcb18574a26c21f03dc9e4e42ff0/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e4e6b05a45525357e382909a4c1600444e2a45b4795163d3b22669285591c1ae", size = 326751, upload-time = "2025-07-26T12:02:00.343Z" }, + { url = "https://files.pythonhosted.org/packages/0f/81/03b45cfad088e4770b1dcf72ea78d3802d04200009fb364d18a493857210/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ab3074b48c4e2cf1a960e6bbeb7f04566bf36b1861d5c9d4d8ac04b82e38ba20", size = 375486, upload-time = "2025-07-26T12:02:02.128Z" }, + { url = "https://files.pythonhosted.org/packages/0c/ba/49923366492ffbdd4486e970d421b289a670ae8cf539c1ea9a09822b371a/contourpy-1.3.3-cp313-cp313t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6c3d53c796f8647d6deb1abe867daeb66dcc8a97e8455efa729516b997b8ed99", size = 388106, upload-time = "2025-07-26T12:02:03.615Z" }, + { url = "https://files.pythonhosted.org/packages/9f/52/5b00ea89525f8f143651f9f03a0df371d3cbd2fccd21ca9b768c7a6500c2/contourpy-1.3.3-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:50ed930df7289ff2a8d7afeb9603f8289e5704755c7e5c3bbd929c90c817164b", size = 352548, upload-time = "2025-07-26T12:02:05.165Z" }, + { url = "https://files.pythonhosted.org/packages/32/1d/a209ec1a3a3452d490f6b14dd92e72280c99ae3d1e73da74f8277d4ee08f/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:4feffb6537d64b84877da813a5c30f1422ea5739566abf0bd18065ac040e120a", size = 1322297, upload-time = "2025-07-26T12:02:07.379Z" }, + { url = "https://files.pythonhosted.org/packages/bc/9e/46f0e8ebdd884ca0e8877e46a3f4e633f6c9c8c4f3f6e72be3fe075994aa/contourpy-1.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2b7e9480ffe2b0cd2e787e4df64270e3a0440d9db8dc823312e2c940c167df7e", size = 1391023, upload-time = "2025-07-26T12:02:10.171Z" }, + { url = "https://files.pythonhosted.org/packages/b9/70/f308384a3ae9cd2209e0849f33c913f658d3326900d0ff5d378d6a1422d2/contourpy-1.3.3-cp313-cp313t-win32.whl", hash = "sha256:283edd842a01e3dcd435b1c5116798d661378d83d36d337b8dde1d16a5fc9ba3", size = 196157, upload-time = "2025-07-26T12:02:11.488Z" }, + { url = "https://files.pythonhosted.org/packages/b2/dd/880f890a6663b84d9e34a6f88cded89d78f0091e0045a284427cb6b18521/contourpy-1.3.3-cp313-cp313t-win_amd64.whl", hash = "sha256:87acf5963fc2b34825e5b6b048f40e3635dd547f590b04d2ab317c2619ef7ae8", size = 240570, upload-time = "2025-07-26T12:02:12.754Z" }, + { url = "https://files.pythonhosted.org/packages/80/99/2adc7d8ffead633234817ef8e9a87115c8a11927a94478f6bb3d3f4d4f7d/contourpy-1.3.3-cp313-cp313t-win_arm64.whl", hash = "sha256:3c30273eb2a55024ff31ba7d052dde990d7d8e5450f4bbb6e913558b3d6c2301", size = 199713, upload-time = "2025-07-26T12:02:14.4Z" }, + { url = "https://files.pythonhosted.org/packages/72/8b/4546f3ab60f78c514ffb7d01a0bd743f90de36f0019d1be84d0a708a580a/contourpy-1.3.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fde6c716d51c04b1c25d0b90364d0be954624a0ee9d60e23e850e8d48353d07a", size = 292189, upload-time = "2025-07-26T12:02:16.095Z" }, + { url = "https://files.pythonhosted.org/packages/fd/e1/3542a9cb596cadd76fcef413f19c79216e002623158befe6daa03dbfa88c/contourpy-1.3.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:cbedb772ed74ff5be440fa8eee9bd49f64f6e3fc09436d9c7d8f1c287b121d77", size = 273251, upload-time = "2025-07-26T12:02:17.524Z" }, + { url = "https://files.pythonhosted.org/packages/b1/71/f93e1e9471d189f79d0ce2497007731c1e6bf9ef6d1d61b911430c3db4e5/contourpy-1.3.3-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:22e9b1bd7a9b1d652cd77388465dc358dafcd2e217d35552424aa4f996f524f5", size = 335810, upload-time = "2025-07-26T12:02:18.9Z" }, + { url = "https://files.pythonhosted.org/packages/91/f9/e35f4c1c93f9275d4e38681a80506b5510e9327350c51f8d4a5a724d178c/contourpy-1.3.3-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a22738912262aa3e254e4f3cb079a95a67132fc5a063890e224393596902f5a4", size = 382871, upload-time = "2025-07-26T12:02:20.418Z" }, + { url = "https://files.pythonhosted.org/packages/b5/71/47b512f936f66a0a900d81c396a7e60d73419868fba959c61efed7a8ab46/contourpy-1.3.3-cp314-cp314-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:afe5a512f31ee6bd7d0dda52ec9864c984ca3d66664444f2d72e0dc4eb832e36", size = 386264, upload-time = "2025-07-26T12:02:21.916Z" }, + { url = "https://files.pythonhosted.org/packages/04/5f/9ff93450ba96b09c7c2b3f81c94de31c89f92292f1380261bd7195bea4ea/contourpy-1.3.3-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f64836de09927cba6f79dcd00fdd7d5329f3fccc633468507079c829ca4db4e3", size = 363819, upload-time = "2025-07-26T12:02:23.759Z" }, + { url = "https://files.pythonhosted.org/packages/3e/a6/0b185d4cc480ee494945cde102cb0149ae830b5fa17bf855b95f2e70ad13/contourpy-1.3.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1fd43c3be4c8e5fd6e4f2baeae35ae18176cf2e5cced681cca908addf1cdd53b", size = 1333650, upload-time = "2025-07-26T12:02:26.181Z" }, + { url = "https://files.pythonhosted.org/packages/43/d7/afdc95580ca56f30fbcd3060250f66cedbde69b4547028863abd8aa3b47e/contourpy-1.3.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6afc576f7b33cf00996e5c1102dc2a8f7cc89e39c0b55df93a0b78c1bd992b36", size = 1404833, upload-time = "2025-07-26T12:02:28.782Z" }, + { url = "https://files.pythonhosted.org/packages/e2/e2/366af18a6d386f41132a48f033cbd2102e9b0cf6345d35ff0826cd984566/contourpy-1.3.3-cp314-cp314-win32.whl", hash = "sha256:66c8a43a4f7b8df8b71ee1840e4211a3c8d93b214b213f590e18a1beca458f7d", size = 189692, upload-time = "2025-07-26T12:02:30.128Z" }, + { url = "https://files.pythonhosted.org/packages/7d/c2/57f54b03d0f22d4044b8afb9ca0e184f8b1afd57b4f735c2fa70883dc601/contourpy-1.3.3-cp314-cp314-win_amd64.whl", hash = "sha256:cf9022ef053f2694e31d630feaacb21ea24224be1c3ad0520b13d844274614fd", size = 232424, upload-time = "2025-07-26T12:02:31.395Z" }, + { url = "https://files.pythonhosted.org/packages/18/79/a9416650df9b525737ab521aa181ccc42d56016d2123ddcb7b58e926a42c/contourpy-1.3.3-cp314-cp314-win_arm64.whl", hash = "sha256:95b181891b4c71de4bb404c6621e7e2390745f887f2a026b2d99e92c17892339", size = 198300, upload-time = "2025-07-26T12:02:32.956Z" }, + { url = "https://files.pythonhosted.org/packages/1f/42/38c159a7d0f2b7b9c04c64ab317042bb6952b713ba875c1681529a2932fe/contourpy-1.3.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:33c82d0138c0a062380332c861387650c82e4cf1747aaa6938b9b6516762e772", size = 306769, upload-time = "2025-07-26T12:02:34.2Z" }, + { url = "https://files.pythonhosted.org/packages/c3/6c/26a8205f24bca10974e77460de68d3d7c63e282e23782f1239f226fcae6f/contourpy-1.3.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ea37e7b45949df430fe649e5de8351c423430046a2af20b1c1961cae3afcda77", size = 287892, upload-time = "2025-07-26T12:02:35.807Z" }, + { url = "https://files.pythonhosted.org/packages/66/06/8a475c8ab718ebfd7925661747dbb3c3ee9c82ac834ccb3570be49d129f4/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d304906ecc71672e9c89e87c4675dc5c2645e1f4269a5063b99b0bb29f232d13", size = 326748, upload-time = "2025-07-26T12:02:37.193Z" }, + { url = "https://files.pythonhosted.org/packages/b4/a3/c5ca9f010a44c223f098fccd8b158bb1cb287378a31ac141f04730dc49be/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ca658cd1a680a5c9ea96dc61cdbae1e85c8f25849843aa799dfd3cb370ad4fbe", size = 375554, upload-time = "2025-07-26T12:02:38.894Z" }, + { url = "https://files.pythonhosted.org/packages/80/5b/68bd33ae63fac658a4145088c1e894405e07584a316738710b636c6d0333/contourpy-1.3.3-cp314-cp314t-manylinux_2_26_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ab2fd90904c503739a75b7c8c5c01160130ba67944a7b77bbf36ef8054576e7f", size = 388118, upload-time = "2025-07-26T12:02:40.642Z" }, + { url = "https://files.pythonhosted.org/packages/40/52/4c285a6435940ae25d7410a6c36bda5145839bc3f0beb20c707cda18b9d2/contourpy-1.3.3-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b7301b89040075c30e5768810bc96a8e8d78085b47d8be6e4c3f5a0b4ed478a0", size = 352555, upload-time = "2025-07-26T12:02:42.25Z" }, + { url = "https://files.pythonhosted.org/packages/24/ee/3e81e1dd174f5c7fefe50e85d0892de05ca4e26ef1c9a59c2a57e43b865a/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:2a2a8b627d5cc6b7c41a4beff6c5ad5eb848c88255fda4a8745f7e901b32d8e4", size = 1322295, upload-time = "2025-07-26T12:02:44.668Z" }, + { url = "https://files.pythonhosted.org/packages/3c/b2/6d913d4d04e14379de429057cd169e5e00f6c2af3bb13e1710bcbdb5da12/contourpy-1.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:fd6ec6be509c787f1caf6b247f0b1ca598bef13f4ddeaa126b7658215529ba0f", size = 1391027, upload-time = "2025-07-26T12:02:47.09Z" }, + { url = "https://files.pythonhosted.org/packages/93/8a/68a4ec5c55a2971213d29a9374913f7e9f18581945a7a31d1a39b5d2dfe5/contourpy-1.3.3-cp314-cp314t-win32.whl", hash = "sha256:e74a9a0f5e3fff48fb5a7f2fd2b9b70a3fe014a67522f79b7cca4c0c7e43c9ae", size = 202428, upload-time = "2025-07-26T12:02:48.691Z" }, + { url = "https://files.pythonhosted.org/packages/fa/96/fd9f641ffedc4fa3ace923af73b9d07e869496c9cc7a459103e6e978992f/contourpy-1.3.3-cp314-cp314t-win_amd64.whl", hash = "sha256:13b68d6a62db8eafaebb8039218921399baf6e47bf85006fd8529f2a08ef33fc", size = 250331, upload-time = "2025-07-26T12:02:50.137Z" }, + { url = "https://files.pythonhosted.org/packages/ae/8c/469afb6465b853afff216f9528ffda78a915ff880ed58813ba4faf4ba0b6/contourpy-1.3.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b7448cb5a725bb1e35ce88771b86fba35ef418952474492cf7c764059933ff8b", size = 203831, upload-time = "2025-07-26T12:02:51.449Z" }, +] + +[[package]] +name = "coverage" +version = "7.14.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/23/7f/d0720730a397a999ffc0fd3f5bebef347338e3a47b727da66fbb228e2ff2/coverage-7.14.0.tar.gz", hash = "sha256:057a6af2f160a85384cde4ab36f0d2777bae1057bae255f95413cdd382aa5c74", size = 919489, upload-time = "2026-05-10T18:02:31.397Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/09/1e/2f996b2c8415cbb6f54b0f5ec1ee850c96d7911961afb4fc05f4a89d8c58/coverage-7.14.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7ffd19fc8aed057fd686a17a4935eef5f9859d69208f96310e893e64b9b6ccf5", size = 219967, upload-time = "2026-05-10T18:00:13.756Z" }, + { url = "https://files.pythonhosted.org/packages/34/23/35c7aea1274aef7525bdd2dc92f710bdde6d11652239d71d1ec450067939/coverage-7.14.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:829994cfe1aeb773ca27bf246d4badc1e764893e3bfb98fff820fcecd1ca4662", size = 220329, upload-time = "2026-05-10T18:00:15.264Z" }, + { url = "https://files.pythonhosted.org/packages/75/cf/a8f4b43a16e194b0261257ad28ded5853ec052570afef4a84e1d81189f3b/coverage-7.14.0-cp312-cp312-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:b4f07cf7edcb7ec39431a5074d7ea83b29a9f71fcfc494f0f40af4e65180420f", size = 251839, upload-time = "2026-05-10T18:00:17.16Z" }, + { url = "https://files.pythonhosted.org/packages/69/ff/6699e7b71e60d3049eb2bdcbc95ee3f35707b2b0e48f32e9e63d3ce30c08/coverage-7.14.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:ca3d9cf2c32b521bd9518385608787fa86f38daf993695307531822c3430ed67", size = 254576, upload-time = "2026-05-10T18:00:18.829Z" }, + { url = "https://files.pythonhosted.org/packages/22/ec/c936d495fcd67f48f03a9c4ad3297ff80d1f222a5df3980f15b34c186c21/coverage-7.14.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:92af52828e7f29d827346b0294e5a0853fa206db77db0395b282918d41e28db9", size = 255690, upload-time = "2026-05-10T18:00:20.648Z" }, + { url = "https://files.pythonhosted.org/packages/5c/42/5af63f636cc62a4a2b1b3ba9146f6ee6f53a35a50d5cefc54d5670f60999/coverage-7.14.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7b2bb6c9d7e769360d0f20a0f219603fd64f0c8f97de17ab25853261602be0fb", size = 257949, upload-time = "2026-05-10T18:00:22.28Z" }, + { url = "https://files.pythonhosted.org/packages/26/d3/a225317bd2012132a27e1176d51660b826f99bb975876463c44ea0d7ee5a/coverage-7.14.0-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:1c9ed6ef99f88fb8c14aa8e2bf8eb0fe55fa2edfea68f8675d78741df1a5ac0e", size = 252242, upload-time = "2026-05-10T18:00:24.076Z" }, + { url = "https://files.pythonhosted.org/packages/f1/7f/9e65495298c3ea414742998539c37d048b5e81cc818fb1828cc6b51d10bf/coverage-7.14.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8231ade007f37959fbf58acc677f26b922c02eda6f0428ea307da0fd39681bf3", size = 253608, upload-time = "2026-05-10T18:00:25.588Z" }, + { url = "https://files.pythonhosted.org/packages/94/46/1522b524a35bdad22b2b8c4f9d32d0a104b524726ec380b2db68db1746f5/coverage-7.14.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:d8b013632cc1ce1d09dbe4f32667b4d320ec2f54fc326ebeffcd0b0bcc2bb6c4", size = 251753, upload-time = "2026-05-10T18:00:27.104Z" }, + { url = "https://files.pythonhosted.org/packages/f3/e9/cdf00d38817742c541ade405e115a3f7bf36e6f2a8b99d4f209861b85a2d/coverage-7.14.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1733198802d71ec4c524f322e2867ee05c62e9e75df86bdca545407a221827d1", size = 255823, upload-time = "2026-05-10T18:00:29.038Z" }, + { url = "https://files.pythonhosted.org/packages/38/fc/5e7877cf5f902d08a17ff1c532511476d87e1bea355bd5028cb97f902e79/coverage-7.14.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:72a305291fa8ee01332f1aaf38b348ca34097f6aa0b0ef627eef2837e57bbba5", size = 251323, upload-time = "2026-05-10T18:00:30.647Z" }, + { url = "https://files.pythonhosted.org/packages/18/9d/50f05a72dff8487464fdd4178dda5daed642a060e60afb644e3d45123559/coverage-7.14.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fcaba850dd317c65423a9d63d88f9573c53b00354d6dd95724576cc98a131595", size = 253197, upload-time = "2026-05-10T18:00:32.211Z" }, + { url = "https://files.pythonhosted.org/packages/00/3f/6f61ffe6439df266c3cf60f5c99cfaa21103d0210d706a42fc6c30683ff8/coverage-7.14.0-cp312-cp312-win32.whl", hash = "sha256:5ac83957a80d0701310e96d8bec68cdcf4f90a7674b7d13f15a344315b41ab27", size = 222515, upload-time = "2026-05-10T18:00:33.717Z" }, + { url = "https://files.pythonhosted.org/packages/85/19/93853133df2cb371083285ef6a93982a0173e7a233b0f61373ba9fd30eb2/coverage-7.14.0-cp312-cp312-win_amd64.whl", hash = "sha256:70390b0da32cb90b501953716302906e8bcce087cb283e70d8c97729f22e92b2", size = 223324, upload-time = "2026-05-10T18:00:35.172Z" }, + { url = "https://files.pythonhosted.org/packages/74/18/9f7fe62f659f24b7a82a0be56bf94c1bd0a89e0ae7ab4c668f6e82404294/coverage-7.14.0-cp312-cp312-win_arm64.whl", hash = "sha256:91b993743d959b8be85b4abf9d5478216a69329c321efe5be0433c1a841d691d", size = 221944, upload-time = "2026-05-10T18:00:37.014Z" }, + { url = "https://files.pythonhosted.org/packages/6b/76/b7c66ee3c66e1b0f9d894c8125983aa0c03fb2336f2fd16559f9c966157f/coverage-7.14.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f2bbb8254370eb4c628ff3d6fa8a7f74ddc40565394d4f7ab791d1fe568e37ef", size = 219990, upload-time = "2026-05-10T18:00:38.887Z" }, + { url = "https://files.pythonhosted.org/packages/b3/af/e567cbad5ba69c013a50146dfa886dc7193361fda77521f51274ff620e1b/coverage-7.14.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:23b81107f46d3f21d0cbce30664fcec0f5d9f585638a67081750f99738f6bf66", size = 220365, upload-time = "2026-05-10T18:00:40.864Z" }, + { url = "https://files.pythonhosted.org/packages/44/6f/9ad575d505b4d805b254febc8a5b338a2efe278f8786e56ff1cb8413f9c3/coverage-7.14.0-cp313-cp313-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:22a7e06a5f11a757cdfe79018e9095f9f69ae283c5cd8123774c788deec8717b", size = 251363, upload-time = "2026-05-10T18:00:42.489Z" }, + { url = "https://files.pythonhosted.org/packages/6f/5f/b5370068b2f57787454592ed7dcd1002f0f1703b7db1fa30f6a325a4ca6e/coverage-7.14.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9d1aa57a1dc8e05bdc42e81c5d671d849577aeedf279f4c449d6d286f9ed88ca", size = 253961, upload-time = "2026-05-10T18:00:44.079Z" }, + { url = "https://files.pythonhosted.org/packages/29/1e/51adf17738976e8f2b85ddef7b7aa12a0838b056c92f175941d8862767c1/coverage-7.14.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:90c1a51bcfddf645b3bb7ec333d9e94393a8e94f55642380fa8a9a5a9e636cb7", size = 255193, upload-time = "2026-05-10T18:00:45.623Z" }, + { url = "https://files.pythonhosted.org/packages/9e/7b/5bfd7ac1df3b881c2ac7a5cbc99c7609e6296c402f5ef587cd81c6f355b3/coverage-7.14.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a841fae2fadcae4f438d43b6ccc4aac2ad609f47cdb6cfdce60cbb3fe5ca7bc2", size = 257326, upload-time = "2026-05-10T18:00:47.173Z" }, + { url = "https://files.pythonhosted.org/packages/7d/38/1d37d316b174fad3843a1d76dbdfe4398771c9ecd0515935dd9ece9cd627/coverage-7.14.0-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c79d2319cabef1fe8e86df73371126931550804738f78ad7d31e3aad85a67367", size = 251582, upload-time = "2026-05-10T18:00:49.152Z" }, + { url = "https://files.pythonhosted.org/packages/34/46/746704f95980ba220214e1a41e18cec5aea80a898eaa53c51bf2d645ff36/coverage-7.14.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1b23b0c6f0b1db6ad769b7050c8b641c0bf215ded26c1816955b17b7f26edfa9", size = 253325, upload-time = "2026-05-10T18:00:51.252Z" }, + { url = "https://files.pythonhosted.org/packages/e1/b9/bbe87206d9687b192352f893797825b5f5b15ecd3aa9c68fbff0c074d77b/coverage-7.14.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:55d3089079ce181a4566b1065ab28d2575eb76d8ac8f81f4fcda2bf037fee087", size = 251291, upload-time = "2026-05-10T18:00:52.816Z" }, + { url = "https://files.pythonhosted.org/packages/46/57/b8cdb12ac0d73ef0243218bd5e22c9df8f92edab8018213a86aec67c5324/coverage-7.14.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:49c005cba1e2f9677fb2845dcdf9a2e72a52a17d63e8231aaaae35d9f50215ef", size = 255448, upload-time = "2026-05-10T18:00:54.548Z" }, + { url = "https://files.pythonhosted.org/packages/1f/d4/5002019538b2036ce3c84340f54d2fd5100d55b0a6b0894eee56128d03c7/coverage-7.14.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:9117377b823daa28aa8635fbb08cda1cd6be3d7143257345459559aeef852d52", size = 251110, upload-time = "2026-05-10T18:00:56.122Z" }, + { url = "https://files.pythonhosted.org/packages/37/53/20c5009477660f084e6ed60bc02a91894b8e234e617e86ecfd9aaf78e27b/coverage-7.14.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7b79d646cf46d5cf9a9f40281d4441df5849e445726e369006d2b117710b33fe", size = 252885, upload-time = "2026-05-10T18:00:57.967Z" }, + { url = "https://files.pythonhosted.org/packages/ae/ab/3cf6427ac9c1f1db747dbb1ce71dde47984876d4c2cfd018a3fef0a78d4d/coverage-7.14.0-cp313-cp313-win32.whl", hash = "sha256:fb609b3658479e33f9516d46f1a89dbb9b6c261366e3a11844a96ec487533dae", size = 222539, upload-time = "2026-05-10T18:00:59.581Z" }, + { url = "https://files.pythonhosted.org/packages/8f/b8/9228523e80321c2cb4880d1f589bc0171f2f71432c35118ad04dc01decce/coverage-7.14.0-cp313-cp313-win_amd64.whl", hash = "sha256:0773d8329cf32b6fd222e4b52622c61fe8d503eb966cfc8d3c3c10c96266d50e", size = 223344, upload-time = "2026-05-10T18:01:01.531Z" }, + { url = "https://files.pythonhosted.org/packages/a3/99/118daa192f95e3a6cb2740100fbf8797cda1734b4134ef0b5d501a7fa8f3/coverage-7.14.0-cp313-cp313-win_arm64.whl", hash = "sha256:b4e26a0f1b696faf283bffe5b8569e44e336c582439df5d53281ab89ee0cba96", size = 221966, upload-time = "2026-05-10T18:01:03.16Z" }, + { url = "https://files.pythonhosted.org/packages/e6/f1/a46cc0c013be170216253184a32366d7cbdb9252feaec866b05c2d12a894/coverage-7.14.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:953f521ca9445300397e65fda3dca58b2dbd68fee983777420b57ac3c77e9f90", size = 220679, upload-time = "2026-05-10T18:01:05.058Z" }, + { url = "https://files.pythonhosted.org/packages/64/8c/9c30a3d311a34177fa432995be7fbfc64477d8bac5630bd38055b1c9b424/coverage-7.14.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:98af83fd65ae24b1fdd03aaead967a9f523bcd2f1aab2d4f3ffda65bb568a6f1", size = 221033, upload-time = "2026-05-10T18:01:07.002Z" }, + { url = "https://files.pythonhosted.org/packages/9a/cd/3fb5e06c3badefd0c1b47e2044fdca67f8220a4ec2e7fcfb476aa0a67c6c/coverage-7.14.0-cp313-cp313t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:668b92e6958c4db7cf92e81caac328dfbbdbb215db2850ad28f0cbe1eea0bfbd", size = 262333, upload-time = "2026-05-10T18:01:08.903Z" }, + { url = "https://files.pythonhosted.org/packages/a8/e6/fbc322325c7294d3e22c1ad6b79e45d0806b25228c8e5842aed6d8169aa7/coverage-7.14.0-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9fbd898551762dea00d3fef2b1c4f99afd2c6a3ff952ea07d60a9bd5ed4f34bc", size = 264410, upload-time = "2026-05-10T18:01:10.531Z" }, + { url = "https://files.pythonhosted.org/packages/08/92/c497b264bec1673c47cc77e26f760fcda4654cabf1f39546d1a23a3b8c35/coverage-7.14.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:68af363c07ecd8d4b7d4043d85cb376d7d227eceb54e5323ee45da73dbd3e426", size = 266836, upload-time = "2026-05-10T18:01:12.19Z" }, + { url = "https://files.pythonhosted.org/packages/78/fc/045da320987f401af5d2815d351e8aa799aec859f60e29f445e3089eeedb/coverage-7.14.0-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:6e57054a583da8ac55edf24117ea4c9133032cfc4cf72aa2d48c1e5d4b52f899", size = 267974, upload-time = "2026-05-10T18:01:13.926Z" }, + { url = "https://files.pythonhosted.org/packages/1b/ae/227b1e379497fb7a4fc3286e620f80c8a1e7cec66d45695a01639eb1af65/coverage-7.14.0-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:cc3499459bbcdd51a65b64c35ab7ed2764eaf3cba826e0df3f1d7fe2e102b70b", size = 261578, upload-time = "2026-05-10T18:01:15.564Z" }, + { url = "https://files.pythonhosted.org/packages/a0/f5/3570342900f2acea31d33ff1590c5d8bac1a8e1a2e1c6d34a5d5e61de681/coverage-7.14.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:45899ec2138a4346ed34d601dedf5076fb74edf2d1dd9dc76a78e82397edee90", size = 264394, upload-time = "2026-05-10T18:01:17.607Z" }, + { url = "https://files.pythonhosted.org/packages/16/29/de1bbc01c935b28f89b1dc3db85b011c055e843a8e5e3b83141c3f80af7f/coverage-7.14.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:8767486808c436f05b23ab98eb963fb29185e32a9357a166971685cb3459900f", size = 262022, upload-time = "2026-05-10T18:01:19.304Z" }, + { url = "https://files.pythonhosted.org/packages/35/95/f53890b0bf2fc10ab168e05d38869215e73ca24c4cb521c3bb0eb62fe16b/coverage-7.14.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:a3b5ddfd6aa7ddad53ee3edb231e88a2151507a43229b7d71b953916deca127d", size = 265732, upload-time = "2026-05-10T18:01:21.494Z" }, + { url = "https://files.pythonhosted.org/packages/ed/ea/c919e259081dd2bdf0e43b87209709ba7ec2e4117c2a7f5185379c43463c/coverage-7.14.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:63df0fe568e698e1045792399f8ab6da3a6c2dce3182813fb92afa2641087b47", size = 260921, upload-time = "2026-05-10T18:01:23.533Z" }, + { url = "https://files.pythonhosted.org/packages/1a/2c/c2831889705a81dc5d1c6ca12e4d8e9b95dfc146d153488a6c0ea685d28e/coverage-7.14.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:827d6397dbd95144939b18f89edf31f63e1f99633e8d5f32f22ba8bdda567477", size = 263109, upload-time = "2026-05-10T18:01:25.165Z" }, + { url = "https://files.pythonhosted.org/packages/5a/a9/2fcae5003cac3d63fe344d2166243c2756935f48420863c5272b240d550b/coverage-7.14.0-cp313-cp313t-win32.whl", hash = "sha256:7bf43e000d24012599b879791cff41589af90674722421ef11b11a5431920bab", size = 223212, upload-time = "2026-05-10T18:01:27.157Z" }, + { url = "https://files.pythonhosted.org/packages/3f/bb/18e94d7b14b9b398164197114a587a04ab7c9fdbe1d237eef57311c5e883/coverage-7.14.0-cp313-cp313t-win_amd64.whl", hash = "sha256:3f5549365af25d770e06b1f8f5682d9a5637d06eb494db91c6fa75d3950cc917", size = 224272, upload-time = "2026-05-10T18:01:29.107Z" }, + { url = "https://files.pythonhosted.org/packages/db/56/4f14fad782b035c81c4ffd09159e7103d42bb1d93ac8496d04b90a11b7da/coverage-7.14.0-cp313-cp313t-win_arm64.whl", hash = "sha256:6d160217ec6fe890f16ad3a9531761589443749e448f91986c972714fad361c8", size = 222530, upload-time = "2026-05-10T18:01:31.151Z" }, + { url = "https://files.pythonhosted.org/packages/1c/18/b9a6586d73992807c26f9a5f274131be3d76b56b18a82b9392e2a25d2e45/coverage-7.14.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:9aed9fa983514ca032790f3fe0d1c0e42ca7e16b42432af1706b50a9a46bef5d", size = 220036, upload-time = "2026-05-10T18:01:33.057Z" }, + { url = "https://files.pythonhosted.org/packages/f3/9b/4165a1d56ddc302a0e2d518fd9d412a4fd0b57562618c78c5f21c57194f5/coverage-7.14.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ba3b8390db29296dbbf49e91b6fe08f990743a90c8f447ba4c2ffc29670dfa63", size = 220368, upload-time = "2026-05-10T18:01:34.705Z" }, + { url = "https://files.pythonhosted.org/packages/69/aa/c12e52a5ba148d9995229d557e3be6e554fe469addc0e9241b2f0956d8ea/coverage-7.14.0-cp314-cp314-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:3a5d8e876dfa2f102e970b183863d6dedd023d3c0eeca1fe7a9787bc5f28b212", size = 251417, upload-time = "2026-05-10T18:01:36.949Z" }, + { url = "https://files.pythonhosted.org/packages/d7/51/ec641c26e6dca1b25a7d2035ba6ecb7c884ef1a100a9e42fbe4ce4405139/coverage-7.14.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:5ebb8f4614a3787d567e610bbfdf96a4798dd69a1afb1bd8ad228d4111fe6ff3", size = 253924, upload-time = "2026-05-10T18:01:38.985Z" }, + { url = "https://files.pythonhosted.org/packages/33/c4/59c3de0bd1b538824173fd518fed51c1ce740ca5ed68e74545983f4053a9/coverage-7.14.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b9bf47223dd8db3d4c4b2e443b02bace480d428f0822c3f991600448a176c97", size = 255269, upload-time = "2026-05-10T18:01:40.957Z" }, + { url = "https://files.pythonhosted.org/packages/7b/a9/36dfa153a62040296f6e7febfdb20a5720622f6ef5a81a41e8237b9a5344/coverage-7.14.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:3485a836550b303d006d57cc06e3d5afaabc642c77050b7c985a97b13e3776b8", size = 257583, upload-time = "2026-05-10T18:01:42.607Z" }, + { url = "https://files.pythonhosted.org/packages/26/7b/cc2c048d4114d9ab1c2409e9ee365e5ae10736df6dffcfc9444effa6c708/coverage-7.14.0-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:3e7e88110bae996d199d1693ca8ec3fd52441d426401ae963437598667b4c5eb", size = 251434, upload-time = "2026-05-10T18:01:44.537Z" }, + { url = "https://files.pythonhosted.org/packages/ee/df/6770eaa576e604575e9a78055313250faef5faa84bd6f71a39fece519c43/coverage-7.14.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:15228a6800ce7bdf1b74800595e56db7138cecb338fdbf044806e10dcf182dfe", size = 253280, upload-time = "2026-05-10T18:01:46.175Z" }, + { url = "https://files.pythonhosted.org/packages/ad/9e/1c0264514a3f98259a6d64765a397b2c8373e3ba59ee722a4802d3ec0c61/coverage-7.14.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:9d26ac7f5398bafc5b57421ad994e8a4749e8a7a0e62d05ec7d53014d5963bfa", size = 251241, upload-time = "2026-05-10T18:01:48.732Z" }, + { url = "https://files.pythonhosted.org/packages/64/16/4efdf3e3c4079cdbf0ece56a2fea872df9e8a3e15a13a0af4400e1075944/coverage-7.14.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:2fb73254ff43c911c967a899e1359bc5049b4b115d6e8fbdde4937d0a2246cd5", size = 255516, upload-time = "2026-05-10T18:01:50.819Z" }, + { url = "https://files.pythonhosted.org/packages/93/69/b1de96346603881b3d1bc8d6447c83200e1c9700ffbaff926ba01ff5724c/coverage-7.14.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:454a380af72c6adada298ed270d38c7a391288198dbfb8467f786f588751a90c", size = 251059, upload-time = "2026-05-10T18:01:52.773Z" }, + { url = "https://files.pythonhosted.org/packages/a4/66/2881853e0363a5e0a724d1103e53650795367471b6afb234f8b49e713bc6/coverage-7.14.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:65c86fb646d2bd2972e96bd1a8b45817ed907cee68655d6295fe7ec031d04cca", size = 252716, upload-time = "2026-05-10T18:01:54.506Z" }, + { url = "https://files.pythonhosted.org/packages/55/5c/0d3305d002c41dcde873dbe456491e663dc55152ca526b630b5c47efd62f/coverage-7.14.0-cp314-cp314-win32.whl", hash = "sha256:6a6516b02a6101398e19a3f44820f69bab2590697f7def4331f668b14adaf828", size = 222788, upload-time = "2026-05-10T18:01:56.487Z" }, + { url = "https://files.pythonhosted.org/packages/f9/58/6e1b8f52fdc3184b47dc5037f5070d83a3d11042db1594b02d2a44d786c8/coverage-7.14.0-cp314-cp314-win_amd64.whl", hash = "sha256:45e0f79d8351fa76e256716df91eab12890d32678b9590df7ae1042e4bd4cf5d", size = 223600, upload-time = "2026-05-10T18:01:58.497Z" }, + { url = "https://files.pythonhosted.org/packages/00/70/a18c408e674bc26281cadaedc7351f929bd2094e191e4b15271c30b084cc/coverage-7.14.0-cp314-cp314-win_arm64.whl", hash = "sha256:4b899594a8b2d81e5cc064a0d7f9cac2081fed91049456cae7676787e41549c9", size = 222168, upload-time = "2026-05-10T18:02:00.411Z" }, + { url = "https://files.pythonhosted.org/packages/3d/89/2681f071d238b62aff8dfc2ab44fc24cfdb38d1c01f391a80522ff5d3a16/coverage-7.14.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f580f8c80acd94ac72e863efe2cab791d8c38d153e0b463b92dfa000d5c84cd1", size = 220766, upload-time = "2026-05-10T18:02:02.313Z" }, + { url = "https://files.pythonhosted.org/packages/bd/c7/c987babafd9207ffa1995e1ef1f9b26762cf4963aa768a66b6f0501e4616/coverage-7.14.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a2bd259c442cd43c49b30fbafc51776eb19ea396faf159d26a83e6a0a5f13b0c", size = 221035, upload-time = "2026-05-10T18:02:04.017Z" }, + { url = "https://files.pythonhosted.org/packages/5a/e9/d6a5ac3b333088143d6fc877d398a9a674dc03124a2f776e131f03864823/coverage-7.14.0-cp314-cp314t-manylinux1_i686.manylinux_2_28_i686.manylinux_2_5_i686.whl", hash = "sha256:a706b908dfa85538863504c624b237a3cc34232bf403c057414ebfdb3b4d9f84", size = 262405, upload-time = "2026-05-10T18:02:05.915Z" }, + { url = "https://files.pythonhosted.org/packages/38/b1/e70838d29a7c08e22d44398a46db90815bbcbf28de06992bd9210d1a8d8e/coverage-7.14.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7333cd944ee4393b9b3d3c1b598c936d4fc8d70573a4c7dacfec5590dd50e436", size = 264530, upload-time = "2026-05-10T18:02:07.582Z" }, + { url = "https://files.pythonhosted.org/packages/6b/73/5c31ef97763288d03d9995152b96d5475b527c63d91c84b01caea894b83a/coverage-7.14.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0f162bc9a15b82d947b02651b0c7e1609d6f7a8735ca330cfadec8481dd97d5a", size = 266932, upload-time = "2026-05-10T18:02:09.401Z" }, + { url = "https://files.pythonhosted.org/packages/e1/76/dd56d80f29c5f05b4d76f7e7c6d47cafacae017189c75c5759d24f9ff0cc/coverage-7.14.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:362cb78e01a5dc82009d88004cf60f2e6b6d6fcbfdec05b05af73b0abf40118f", size = 268062, upload-time = "2026-05-10T18:02:11.399Z" }, + { url = "https://files.pythonhosted.org/packages/6e/c7/27ba85cd5b95614f159ff93ebff1901584a8d192e2e5e24c4943a7453f59/coverage-7.14.0-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:acebd068fca5512c3a6fde9c045f901613478781a73f0e82b307b214daef23fb", size = 261504, upload-time = "2026-05-10T18:02:13.257Z" }, + { url = "https://files.pythonhosted.org/packages/13/2e/e8149f60ab5d5684c6eee881bdf34b127115cddbb958b196768dd9d63473/coverage-7.14.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:29fe3da551dface75deb2ccbf87b6b66e2e7ef38f6d89050b428be94afff3490", size = 264398, upload-time = "2026-05-10T18:02:15.063Z" }, + { url = "https://files.pythonhosted.org/packages/d9/7f/1261b025285323225f4b4abffa5a643649dfd67e25ddca7ebcbdea3b7cb3/coverage-7.14.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:b4cc4fce8672fffcb09b0eafc167b396b3ba53c4a7230f54b7aaffbf6c835fa9", size = 262000, upload-time = "2026-05-10T18:02:16.756Z" }, + { url = "https://files.pythonhosted.org/packages/d3/dc/829c54f60b9d08389439c00f813c752781c496fc5788c78d8006db4b4f2b/coverage-7.14.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:5d4a51aad8ba8bdcd2b8bd8f03d4aca19693fa2327a3470e4718a25b03481020", size = 265732, upload-time = "2026-05-10T18:02:18.817Z" }, + { url = "https://files.pythonhosted.org/packages/ed/b0/70bd1419941652fa062689cba9c3eeafb8f5e6fbb890bce41c3bdda5dbd6/coverage-7.14.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:9f323af3e1e4f68b60b7b247e37b8515563a61375518fa59de1af48ba28a3db6", size = 260847, upload-time = "2026-05-10T18:02:20.528Z" }, + { url = "https://files.pythonhosted.org/packages/f2/73/be40b2390656c654d35ea0015ea7ba3d945769cf80790ad5e0bb2d56d2ba/coverage-7.14.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1a0abc7342ea9711c469dd8b821c6c311e6bc6aac1442e5fbd6b27fae0a8f3db", size = 263166, upload-time = "2026-05-10T18:02:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/29/55/4a643f712fcf7cf2881f8ec1e0ccb7b164aff3108f69b51801246c8799f2/coverage-7.14.0-cp314-cp314t-win32.whl", hash = "sha256:a9f864ef57b7172e2db87a096642dd51e179e085ab6b2c371c29e885f65c8fb2", size = 223573, upload-time = "2026-05-10T18:02:24.11Z" }, + { url = "https://files.pythonhosted.org/packages/27/96/3acae5da0953be042c0b4dea6d6789d2f080701c77b88e44d5bd41b9219b/coverage-7.14.0-cp314-cp314t-win_amd64.whl", hash = "sha256:29943e552fdc08e082eb51400fb2f58e118a83b5542bd06531214e084399b644", size = 224680, upload-time = "2026-05-10T18:02:25.896Z" }, + { url = "https://files.pythonhosted.org/packages/93/3d/6ab5d2dd8325d838737c6f8d83d62eb6230e0d70b87b51b57bbfd08fa767/coverage-7.14.0-cp314-cp314t-win_arm64.whl", hash = "sha256:742a73ea621953b012f2c4c2219b512180dd84489acf5b1596b0aafc55b9100b", size = 222703, upload-time = "2026-05-10T18:02:27.822Z" }, + { url = "https://files.pythonhosted.org/packages/61/e8/cb8e80d6f9f55b99588625062822bf946cf03ed06315df4bd8397f5632a1/coverage-7.14.0-py3-none-any.whl", hash = "sha256:8de5b61163aee3d05c8a2beab6f47913df7981dad1baf82c414d99158c286ab1", size = 211764, upload-time = "2026-05-10T18:02:29.538Z" }, +] + +[[package]] +name = "cuda-bindings" +version = "13.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cuda-pathfinder", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/52/c8/b2589d68acf7e3d63e2be330b84bc25712e97ed799affbca7edd7eae25d6/cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e865447abfb83d6a98ad5130ed3c70b1fc295ae3eeee39fd07b4ddb0671b6788", size = 5722404, upload-time = "2026-03-11T00:12:44.041Z" }, + { url = "https://files.pythonhosted.org/packages/1f/92/f899f7bbb5617bb65ec52a6eac1e9a1447a86b916c4194f8a5001b8cde0c/cuda_bindings-13.2.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46d8776a55d6d5da9dd6e9858fba2efcda2abe6743871dee47dd06eb8cb6d955", size = 6320619, upload-time = "2026-03-11T00:12:45.939Z" }, + { url = "https://files.pythonhosted.org/packages/bb/a5/d7f01a415e134546248cef612adad8153c9f1eb10ec79505a7cd8294370b/cuda_bindings-13.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:45815daeb595bf3b405c52671a2542b1f8e9329f3b029494acbfcc74aeaa1f2d", size = 5840830, upload-time = "2026-03-11T00:12:48.43Z" }, + { url = "https://files.pythonhosted.org/packages/df/93/eef988860a3ca985f82c4f3174fc0cdd94e07331ba9a92e8e064c260337f/cuda_bindings-13.2.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6629ca2df6f795b784752409bcaedbd22a7a651b74b56a165ebc0c9dcbd504d0", size = 5614610, upload-time = "2026-03-11T00:12:50.337Z" }, + { url = "https://files.pythonhosted.org/packages/18/23/6db3aba46864aee357ab2415135b3fe3da7e9f1fa0221fa2a86a5968099c/cuda_bindings-13.2.0-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7dca0da053d3b4cc4869eff49c61c03f3c5dbaa0bcd712317a358d5b8f3f385d", size = 6149914, upload-time = "2026-03-11T00:12:52.374Z" }, + { url = "https://files.pythonhosted.org/packages/c4/84/d3b6220b51cbc02ca14db7387e97445126b4ff5125aaa6c5dd7dcb75e679/cuda_bindings-13.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:8cebe3ce4aeeca5af9c490e175f76c4b569bbf4a35a62294b777bc77bf7ac4d8", size = 5796512, upload-time = "2026-03-11T00:12:54.483Z" }, + { url = "https://files.pythonhosted.org/packages/c0/87/87a014f045b77c6de5c8527b0757fe644417b184e5367db977236a141602/cuda_bindings-13.2.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a6464b30f46692d6c7f65d4a0e0450d81dd29de3afc1bb515653973d01c2cd6e", size = 5685673, upload-time = "2026-03-11T00:12:56.371Z" }, + { url = "https://files.pythonhosted.org/packages/ee/5e/c0fe77a73aaefd3fff25ffaccaac69c5a63eafdf8b9a4c476626ef0ac703/cuda_bindings-13.2.0-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f4af9f3e1be603fa12d5ad6cfca7844c9d230befa9792b5abdf7dd79979c3626", size = 6191386, upload-time = "2026-03-11T00:12:58.965Z" }, + { url = "https://files.pythonhosted.org/packages/e3/73/98bcb069778fe420226db75aff54b5dd6c3ecfd0912edabab723326e80b7/cuda_bindings-13.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:bd658bb5c0e55b7b3e5dd0ed509c6addb298c665db26a9bfba35e1e626000ba2", size = 5938605, upload-time = "2026-03-11T00:13:01.639Z" }, + { url = "https://files.pythonhosted.org/packages/5f/58/ed2c3b39c8dd5f96aa7a4abef0d47a73932c7a988e30f5fa428f00ed0da1/cuda_bindings-13.2.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df850a1ff8ce1b3385257b08e47b70e959932f5f432d0a4e46a355962b4e4771", size = 5507469, upload-time = "2026-03-11T00:13:04.063Z" }, + { url = "https://files.pythonhosted.org/packages/1f/01/0c941b112ceeb21439b05895eace78ca1aa2eaaf695c8521a068fd9b4c00/cuda_bindings-13.2.0-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8a16384c6494e5485f39314b0b4afb04bee48d49edb16d5d8593fd35bbd231b", size = 6059693, upload-time = "2026-03-11T00:13:06.003Z" }, + { url = "https://files.pythonhosted.org/packages/52/49/4e01cc06447d39476e138d1b1adec8d35c0d04eccd2c8d69befc08cd66e8/cuda_bindings-13.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6ccf14e0c1def3b7200100aafff3a9f7e210ecb6e409329e92dcf6cd2c00d5c7", size = 6662637, upload-time = "2026-03-11T00:13:07.881Z" }, +] + +[[package]] +name = "cuda-pathfinder" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/d0/c177e29701cf1d3008d7d2b16b5fc626592ce13bd535f8795c5f57187e0e/cuda_pathfinder-1.5.4-py3-none-any.whl", hash = "sha256:9563d3175ce1828531acf4b94e1c1c7d67208c347ca002493e2654878b26f4b7", size = 51657, upload-time = "2026-04-27T22:42:07.712Z" }, +] + +[[package]] +name = "cuda-toolkit" +version = "13.0.2" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/57/b2/453099f5f3b698d7d0eab38916aac44c7f76229f451709e2eb9db6615dcd/cuda_toolkit-13.0.2-py2.py3-none-any.whl", hash = "sha256:b198824cf2f54003f50d64ada3a0f184b42ca0846c1c94192fa269ecd97a66eb", size = 2364, upload-time = "2025-12-19T23:24:07.328Z" }, +] + +[package.optional-dependencies] +cudart = [ + { name = "nvidia-cuda-runtime", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cufft = [ + { name = "nvidia-cufft", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cufile = [ + { name = "nvidia-cufile", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cupti = [ + { name = "nvidia-cuda-cupti", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +curand = [ + { name = "nvidia-curand", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cusolver = [ + { name = "nvidia-cusolver", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cusparse = [ + { name = "nvidia-cusparse", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +nvjitlink = [ + { name = "nvidia-nvjitlink", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +nvrtc = [ + { name = "nvidia-cuda-nvrtc", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +nvtx = [ + { name = "nvidia-nvtx", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] + +[[package]] +name = "cycler" +version = "0.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a9/95/a3dbbb5028f35eafb79008e7522a75244477d2838f38cbb722248dabc2a8/cycler-0.12.1.tar.gz", hash = "sha256:88bb128f02ba341da8ef447245a9e138fae777f6a23943da4540077d3601eb1c", size = 7615, upload-time = "2023-10-07T05:32:18.335Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/05/c19819d5e3d95294a6f5947fb9b9629efb316b96de511b418c53d245aae6/cycler-0.12.1-py3-none-any.whl", hash = "sha256:85cef7cff222d8644161529808465972e51340599459b8ac3ccbac5a854e0d30", size = 8321, upload-time = "2023-10-07T05:32:16.783Z" }, +] + +[[package]] +name = "docker" +version = "7.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pywin32", marker = "sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "requests" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/91/9b/4a2ea29aeba62471211598dac5d96825bb49348fa07e906ea930394a83ce/docker-7.1.0.tar.gz", hash = "sha256:ad8c70e6e3f8926cb8a92619b832b4ea5299e2831c14284663184e200546fa6c", size = 117834, upload-time = "2024-05-23T11:13:57.216Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e3/26/57c6fb270950d476074c087527a558ccb6f4436657314bfb6cdf484114c4/docker-7.1.0-py3-none-any.whl", hash = "sha256:c96b93b7f0a746f9e77d325bcfb87422a3d8bd4f03136ae8a85b37f1898d5fc0", size = 147774, upload-time = "2024-05-23T11:13:55.01Z" }, +] + +[[package]] +name = "dotenv" +version = "0.9.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dotenv" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/b2/b7/545d2c10c1fc15e48653c91efde329a790f2eecfbbf2bd16003b5db2bab0/dotenv-0.9.9-py2.py3-none-any.whl", hash = "sha256:29cf74a087b31dafdb5a446b6d7e11cbce8ed2741540e2339c69fbef92c94ce9", size = 1892, upload-time = "2025-02-19T22:15:01.647Z" }, +] + +[[package]] +name = "fastapi" +version = "0.136.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "pydantic" }, + { name = "starlette" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5d/45/c130091c2dfa061bbfe3150f2a5091ef1adf149f2a8d2ae769ecaf6e99a2/fastapi-0.136.1.tar.gz", hash = "sha256:7af665ad7acfa0a3baf8983d393b6b471b9da10ede59c60045f49fbc89a0fa7f", size = 397448, upload-time = "2026-04-23T16:49:44.046Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/ff/2e4eca3ade2c22fe1dea7043b8ee9dabe47753349eb1b56a202de8af6349/fastapi-0.136.1-py3-none-any.whl", hash = "sha256:a6e9d7eeada96c93a4d69cb03836b44fa34e2854accb7244a1ece36cd4781c3f", size = 117683, upload-time = "2026-04-23T16:49:42.437Z" }, +] + +[[package]] +name = "filelock" +version = "3.29.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b5/fe/997687a931ab51049acce6fa1f23e8f01216374ea81374ddee763c493db5/filelock-3.29.0.tar.gz", hash = "sha256:69974355e960702e789734cb4871f884ea6fe50bd8404051a3530bc07809cf90", size = 57571, upload-time = "2026-04-19T15:39:10.068Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/47/dd9a212ef6e343a6857485ffe25bba537304f1913bdbed446a23f7f592e1/filelock-3.29.0-py3-none-any.whl", hash = "sha256:96f5f6344709aa1572bbf631c640e4ebeeb519e08da902c39a001882f30ac258", size = 39812, upload-time = "2026-04-19T15:39:08.752Z" }, +] + +[[package]] +name = "fonttools" +version = "4.62.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9a/08/7012b00a9a5874311b639c3920270c36ee0c445b69d9989a85e5c92ebcb0/fonttools-4.62.1.tar.gz", hash = "sha256:e54c75fd6041f1122476776880f7c3c3295ffa31962dc6ebe2543c00dca58b5d", size = 3580737, upload-time = "2026-03-13T13:54:25.52Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/47/d4/dbacced3953544b9a93088cc10ef2b596d348c983d5c67a404fa41ec51ba/fonttools-4.62.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:90365821debbd7db678809c7491ca4acd1e0779b9624cdc6ddaf1f31992bf974", size = 2870219, upload-time = "2026-03-13T13:52:53.664Z" }, + { url = "https://files.pythonhosted.org/packages/66/9e/a769c8e99b81e5a87ab7e5e7236684de4e96246aae17274e5347d11ebd78/fonttools-4.62.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:12859ff0b47dd20f110804c3e0d0970f7b832f561630cd879969011541a464a9", size = 2414891, upload-time = "2026-03-13T13:52:56.493Z" }, + { url = "https://files.pythonhosted.org/packages/69/64/f19a9e3911968c37e1e620e14dfc5778299e1474f72f4e57c5ec771d9489/fonttools-4.62.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9c125ffa00c3d9003cdaaf7f2c79e6e535628093e14b5de1dccb08859b680936", size = 5033197, upload-time = "2026-03-13T13:52:59.179Z" }, + { url = "https://files.pythonhosted.org/packages/9b/8a/99c8b3c3888c5c474c08dbfd7c8899786de9604b727fcefb055b42c84bba/fonttools-4.62.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:149f7d84afca659d1a97e39a4778794a2f83bf344c5ee5134e09995086cc2392", size = 4988768, upload-time = "2026-03-13T13:53:02.761Z" }, + { url = "https://files.pythonhosted.org/packages/d1/c6/0f904540d3e6ab463c1243a0d803504826a11604c72dd58c2949796a1762/fonttools-4.62.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0aa72c43a601cfa9273bb1ae0518f1acadc01ee181a6fc60cd758d7fdadffc04", size = 4971512, upload-time = "2026-03-13T13:53:05.678Z" }, + { url = "https://files.pythonhosted.org/packages/29/0b/5cbef6588dc9bd6b5c9ad6a4d5a8ca384d0cea089da31711bbeb4f9654a6/fonttools-4.62.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:19177c8d96c7c36359266e571c5173bcee9157b59cfc8cb0153c5673dc5a3a7d", size = 5122723, upload-time = "2026-03-13T13:53:08.662Z" }, + { url = "https://files.pythonhosted.org/packages/4a/47/b3a5342d381595ef439adec67848bed561ab7fdb1019fa522e82101b7d9c/fonttools-4.62.1-cp312-cp312-win32.whl", hash = "sha256:a24decd24d60744ee8b4679d38e88b8303d86772053afc29b19d23bb8207803c", size = 2281278, upload-time = "2026-03-13T13:53:10.998Z" }, + { url = "https://files.pythonhosted.org/packages/28/b1/0c2ab56a16f409c6c8a68816e6af707827ad5d629634691ff60a52879792/fonttools-4.62.1-cp312-cp312-win_amd64.whl", hash = "sha256:9e7863e10b3de72376280b515d35b14f5eeed639d1aa7824f4cf06779ec65e42", size = 2331414, upload-time = "2026-03-13T13:53:13.992Z" }, + { url = "https://files.pythonhosted.org/packages/3b/56/6f389de21c49555553d6a5aeed5ac9767631497ac836c4f076273d15bd72/fonttools-4.62.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:c22b1014017111c401469e3acc5433e6acf6ebcc6aa9efb538a533c800971c79", size = 2865155, upload-time = "2026-03-13T13:53:16.132Z" }, + { url = "https://files.pythonhosted.org/packages/03/c5/0e3966edd5ec668d41dfe418787726752bc07e2f5fd8c8f208615e61fa89/fonttools-4.62.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:68959f5fc58ed4599b44aad161c2837477d7f35f5f79402d97439974faebfebe", size = 2412802, upload-time = "2026-03-13T13:53:18.878Z" }, + { url = "https://files.pythonhosted.org/packages/52/94/e6ac4b44026de7786fe46e3bfa0c87e51d5d70a841054065d49cd62bb909/fonttools-4.62.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef46db46c9447103b8f3ff91e8ba009d5fe181b1920a83757a5762551e32bb68", size = 5013926, upload-time = "2026-03-13T13:53:21.379Z" }, + { url = "https://files.pythonhosted.org/packages/e2/98/8b1e801939839d405f1f122e7d175cebe9aeb4e114f95bfc45e3152af9a7/fonttools-4.62.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6706d1cb1d5e6251a97ad3c1b9347505c5615c112e66047abbef0f8545fa30d1", size = 4964575, upload-time = "2026-03-13T13:53:23.857Z" }, + { url = "https://files.pythonhosted.org/packages/46/76/7d051671e938b1881670528fec69cc4044315edd71a229c7fd712eaa5119/fonttools-4.62.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2e7abd2b1e11736f58c1de27819e1955a53267c21732e78243fa2fa2e5c1e069", size = 4953693, upload-time = "2026-03-13T13:53:26.569Z" }, + { url = "https://files.pythonhosted.org/packages/1f/ae/b41f8628ec0be3c1b934fc12b84f4576a5c646119db4d3bdd76a217c90b5/fonttools-4.62.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:403d28ce06ebfc547fbcb0cb8b7f7cc2f7a2d3e1a67ba9a34b14632df9e080f9", size = 5094920, upload-time = "2026-03-13T13:53:29.329Z" }, + { url = "https://files.pythonhosted.org/packages/f2/f6/53a1e9469331a23dcc400970a27a4caa3d9f6edbf5baab0260285238b884/fonttools-4.62.1-cp313-cp313-win32.whl", hash = "sha256:93c316e0f5301b2adbe6a5f658634307c096fd5aae60a5b3412e4f3e1728ab24", size = 2279928, upload-time = "2026-03-13T13:53:32.352Z" }, + { url = "https://files.pythonhosted.org/packages/38/60/35186529de1db3c01f5ad625bde07c1f576305eab6d86bbda4c58445f721/fonttools-4.62.1-cp313-cp313-win_amd64.whl", hash = "sha256:7aa21ff53e28a9c2157acbc44e5b401149d3c9178107130e82d74ceb500e5056", size = 2330514, upload-time = "2026-03-13T13:53:34.991Z" }, + { url = "https://files.pythonhosted.org/packages/36/f0/2888cdac391807d68d90dcb16ef858ddc1b5309bfc6966195a459dd326e2/fonttools-4.62.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:fa1d16210b6b10a826d71bed68dd9ec24a9e218d5a5e2797f37c573e7ec215ca", size = 2864442, upload-time = "2026-03-13T13:53:37.509Z" }, + { url = "https://files.pythonhosted.org/packages/4b/b2/e521803081f8dc35990816b82da6360fa668a21b44da4b53fc9e77efcd62/fonttools-4.62.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:aa69d10ed420d8121118e628ad47d86e4caa79ba37f968597b958f6cceab7eca", size = 2410901, upload-time = "2026-03-13T13:53:40.55Z" }, + { url = "https://files.pythonhosted.org/packages/00/a4/8c3511ff06e53110039358dbbdc1a65d72157a054638387aa2ada300a8b8/fonttools-4.62.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bd13b7999d59c5eb1c2b442eb2d0c427cb517a0b7a1f5798fc5c9e003f5ff782", size = 4999608, upload-time = "2026-03-13T13:53:42.798Z" }, + { url = "https://files.pythonhosted.org/packages/28/63/cd0c3b26afe60995a5295f37c246a93d454023726c3261cfbb3559969bb9/fonttools-4.62.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8d337fdd49a79b0d51c4da87bc38169d21c3abbf0c1aa9367eff5c6656fb6dae", size = 4912726, upload-time = "2026-03-13T13:53:45.405Z" }, + { url = "https://files.pythonhosted.org/packages/70/b9/ac677cb07c24c685cf34f64e140617d58789d67a3dd524164b63648c6114/fonttools-4.62.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d241cdc4a67b5431c6d7f115fdf63335222414995e3a1df1a41e1182acd4bcc7", size = 4951422, upload-time = "2026-03-13T13:53:48.326Z" }, + { url = "https://files.pythonhosted.org/packages/e6/10/11c08419a14b85b7ca9a9faca321accccc8842dd9e0b1c8a72908de05945/fonttools-4.62.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c05557a78f8fa514da0f869556eeda40887a8abc77c76ee3f74cf241778afd5a", size = 5060979, upload-time = "2026-03-13T13:53:51.366Z" }, + { url = "https://files.pythonhosted.org/packages/4e/3c/12eea4a4cf054e7ab058ed5ceada43b46809fce2bf319017c4d63ae55bb4/fonttools-4.62.1-cp314-cp314-win32.whl", hash = "sha256:49a445d2f544ce4a69338694cad575ba97b9a75fff02720da0882d1a73f12800", size = 2283733, upload-time = "2026-03-13T13:53:53.606Z" }, + { url = "https://files.pythonhosted.org/packages/6b/67/74b070029043186b5dd13462c958cb7c7f811be0d2e634309d9a1ffb1505/fonttools-4.62.1-cp314-cp314-win_amd64.whl", hash = "sha256:1eecc128c86c552fb963fe846ca4e011b1be053728f798185a1687502f6d398e", size = 2335663, upload-time = "2026-03-13T13:53:56.23Z" }, + { url = "https://files.pythonhosted.org/packages/42/c5/4d2ed3ca6e33617fc5624467da353337f06e7f637707478903c785bd8e20/fonttools-4.62.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:1596aeaddf7f78e21e68293c011316a25267b3effdaccaf4d59bc9159d681b82", size = 2947288, upload-time = "2026-03-13T13:53:59.397Z" }, + { url = "https://files.pythonhosted.org/packages/1f/e9/7ab11ddfda48ed0f89b13380e5595ba572619c27077be0b2c447a63ff351/fonttools-4.62.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:8f8fca95d3bb3208f59626a4b0ea6e526ee51f5a8ad5d91821c165903e8d9260", size = 2449023, upload-time = "2026-03-13T13:54:01.642Z" }, + { url = "https://files.pythonhosted.org/packages/b2/10/a800fa090b5e8819942e54e19b55fc7c21fe14a08757c3aa3ca8db358939/fonttools-4.62.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee91628c08e76f77b533d65feb3fbe6d9dad699f95be51cf0d022db94089cdc4", size = 5137599, upload-time = "2026-03-13T13:54:04.495Z" }, + { url = "https://files.pythonhosted.org/packages/37/dc/8ccd45033fffd74deb6912fa1ca524643f584b94c87a16036855b498a1ed/fonttools-4.62.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5f37df1cac61d906e7b836abe356bc2f34c99d4477467755c216b72aa3dc748b", size = 4920933, upload-time = "2026-03-13T13:54:07.557Z" }, + { url = "https://files.pythonhosted.org/packages/99/eb/e618adefb839598d25ac8136cd577925d6c513dc0d931d93b8af956210f0/fonttools-4.62.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:92bb00a947e666169c99b43753c4305fc95a890a60ef3aeb2a6963e07902cc87", size = 5016232, upload-time = "2026-03-13T13:54:10.611Z" }, + { url = "https://files.pythonhosted.org/packages/d9/5f/9b5c9bfaa8ec82def8d8168c4f13615990d6ce5996fe52bd49bfb5e05134/fonttools-4.62.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:bdfe592802ef939a0e33106ea4a318eeb17822c7ee168c290273cbd5fabd746c", size = 5042987, upload-time = "2026-03-13T13:54:13.569Z" }, + { url = "https://files.pythonhosted.org/packages/90/aa/dfbbe24c6a6afc5c203d90cc0343e24bcbb09e76d67c4d6eef8c2558d7ba/fonttools-4.62.1-cp314-cp314t-win32.whl", hash = "sha256:b820fcb92d4655513d8402d5b219f94481c4443d825b4372c75a2072aa4b357a", size = 2348021, upload-time = "2026-03-13T13:54:16.98Z" }, + { url = "https://files.pythonhosted.org/packages/13/6f/ae9c4e4dd417948407b680855c2c7790efb52add6009aaecff1e3bc50e8e/fonttools-4.62.1-cp314-cp314t-win_amd64.whl", hash = "sha256:59b372b4f0e113d3746b88985f1c796e7bf830dd54b28374cd85c2b8acd7583e", size = 2414147, upload-time = "2026-03-13T13:54:19.416Z" }, + { url = "https://files.pythonhosted.org/packages/fd/ba/56147c165442cc5ba7e82ecf301c9a68353cede498185869e6e02b4c264f/fonttools-4.62.1-py3-none-any.whl", hash = "sha256:7487782e2113861f4ddcc07c3436450659e3caa5e470b27dc2177cade2d8e7fd", size = 1152647, upload-time = "2026-03-13T13:54:22.735Z" }, +] + +[[package]] +name = "fsspec" +version = "2026.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d5/8d/1c51c094345df128ca4a990d633fe1a0ff28726c9e6b3c41ba65087bba1d/fsspec-2026.4.0.tar.gz", hash = "sha256:301d8ac70ae90ef3ad05dcf94d6c3754a097f9b5fe4667d2787aa359ec7df7e4", size = 312760, upload-time = "2026-04-29T20:42:38.635Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d5/0c/043d5e551459da400957a1395e0febbf771446ff34291afcbe3d8be2a279/fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2", size = 203402, upload-time = "2026-04-29T20:42:36.842Z" }, +] + +[[package]] +name = "ghp-import" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/d9/29/d40217cbe2f6b1359e00c6c307bb3fc876ba74068cbab3dde77f03ca0dc4/ghp-import-2.1.0.tar.gz", hash = "sha256:9c535c4c61193c2df8871222567d7fd7e5014d835f97dc7b7439069e2413d343", size = 10943, upload-time = "2022-05-02T15:47:16.11Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f7/ec/67fbef5d497f86283db54c22eec6f6140243aae73265799baaaa19cd17fb/ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619", size = 11034, upload-time = "2022-05-02T15:47:14.552Z" }, +] + +[[package]] +name = "gitdb" +version = "4.0.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "smmap" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" }, +] + +[[package]] +name = "gitpython" +version = "3.1.50" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "gitdb" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/33/f6/354ae6491228b5eb40e10d89c4d13c651fe1cf7556e35ebdded50cff57ce/gitpython-3.1.50.tar.gz", hash = "sha256:80da2d12504d52e1f998772dc5baf6e553f8d2fcfe1fcc226c9d9a2ee3372dcc", size = 219798, upload-time = "2026-05-06T04:01:26.571Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/7a/1c6e3562dfd8950adbb11ffbc65d21e7c89d01a6e4f137fa981056de25c5/gitpython-3.1.50-py3-none-any.whl", hash = "sha256:d352abe2908d07355014abdd21ddf798c2a961469239afec4962e9da884858f9", size = 212507, upload-time = "2026-05-06T04:01:23.799Z" }, +] + +[[package]] +name = "griffelib" +version = "2.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9d/82/74f4a3310cdabfbb10da554c3a672847f1ed33c6f61dd472681ce7f1fe67/griffelib-2.0.2.tar.gz", hash = "sha256:3cf20b3bc470e83763ffbf236e0076b1211bac1bc67de13daf494640f2de707e", size = 166461, upload-time = "2026-03-27T11:34:51.091Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/8c/c9138d881c79aa0ea9ed83cbd58d5ca75624378b38cee225dcf5c42cc91f/griffelib-2.0.2-py3-none-any.whl", hash = "sha256:925c857658fb1ba40c0772c37acbc2ab650bd794d9c1b9726922e36ea4117ea1", size = 142357, upload-time = "2026-03-27T11:34:46.275Z" }, +] + +[[package]] +name = "h11" +version = "0.16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" }, +] + +[[package]] +name = "hf-xet" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/74/d8/5c06fc76461418326a7decf8367480c35be11a41fd938633929c60a9ec6b/hf_xet-1.5.0.tar.gz", hash = "sha256:e0fb0a34d9f406eed88233e829a67ec016bec5af19e480eac65a233ea289a948", size = 837196, upload-time = "2026-05-06T06:18:15.583Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/68/9b/6912c99070915a4f28119e3c5b52a9abd1eec0ad5cb293b8c967a0c6f5a2/hf_xet-1.5.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:7d70fe2ce97b9db73b9c9b9c81fe3693640aec83416a966c446afea54acfae3c", size = 4023383, upload-time = "2026-05-06T06:17:53.947Z" }, + { url = "https://files.pythonhosted.org/packages/0f/6d/9563cfde59b5d8128a9c7ec972a087f4c782e4f7bac5a85234edfd5d5e49/hf_xet-1.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:73a0dae8c71de3b0633a45c73f4a4a5ed09e94b43441d82981a781d4f12baa42", size = 3792751, upload-time = "2026-05-06T06:17:51.791Z" }, + { url = "https://files.pythonhosted.org/packages/07/a5/ed5a0cf35b49a0571af5a8f53416dad1877a718c021c9937c3a53cb45781/hf_xet-1.5.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a60290ec57e9b71767fba7c3645ddafdd0759974b540441510c629c6db6db24a", size = 4456058, upload-time = "2026-05-06T06:17:40.735Z" }, + { url = "https://files.pythonhosted.org/packages/60/fb/3ae8bf2a7a37a4197d0195d7247fd25b3952e15cb8a599e285dfaa6f52b3/hf_xet-1.5.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:e5de0f6deada0dada870bb376a11bcd1f08abf3a968a6d118f33e72d1b1eb480", size = 4250783, upload-time = "2026-05-06T06:17:38.412Z" }, + { url = "https://files.pythonhosted.org/packages/a2/9b/8bae40d4d91525085137196e84eb0ed49cf65b5e96e5c3ecdadd8bd0fac2/hf_xet-1.5.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c799d49f1a5544a0ef7591c0ee75e0d6b93d6f56dc7a4979f59f7518d2872216", size = 4445594, upload-time = "2026-05-06T06:18:04.219Z" }, + { url = "https://files.pythonhosted.org/packages/13/59/c74efbbd4e8728172b2cc72a2bc014d2947a4b7bdced932fbd3f5da1a4e5/hf_xet-1.5.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:2baea1b0b989e5c152fe81425f7745ddc8901280ba3d97c98d8cdece7b706c60", size = 4663995, upload-time = "2026-05-06T06:18:06.1Z" }, + { url = "https://files.pythonhosted.org/packages/73/32/8e1e0410af64cda9b139d1dcebdc993a8ff9c8c7c0e2696ae356d75ccc0d/hf_xet-1.5.0-cp313-cp313t-win_amd64.whl", hash = "sha256:526345b3ed45f374f6317349df489167606736c876241ba984105afe7fd4839d", size = 3966608, upload-time = "2026-05-06T06:18:19.74Z" }, + { url = "https://files.pythonhosted.org/packages/fc/34/a8febc8f4edbea8b3e21b02ebc8b628679b84ba7e45cde624a7736b51500/hf_xet-1.5.0-cp313-cp313t-win_arm64.whl", hash = "sha256:786d28e2eb8315d5035544b9d137b4a842d600c434bb91bf7d0d953cce906ad4", size = 3796946, upload-time = "2026-05-06T06:18:17.568Z" }, + { url = "https://files.pythonhosted.org/packages/2a/20/8fc8996afe5815fa1a6be8e9e5c02f24500f409d599e905800d498a4e14d/hf_xet-1.5.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:872d5601e6deea30d15865ede55d29eac6daf5a534ab417b99b6ef6b076dd96c", size = 4023495, upload-time = "2026-05-06T06:18:01.94Z" }, + { url = "https://files.pythonhosted.org/packages/32/6a/93d84463c00cecb561a7508aa6303e35ee2894294eac14245526924415fe/hf_xet-1.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:9929561f5abf4581c8ea79587881dfef6b8abb2a0d8a51915936fc2a614f4e73", size = 3792731, upload-time = "2026-05-06T06:18:00.021Z" }, + { url = "https://files.pythonhosted.org/packages/9d/5a/8ec8e0c863b382d00b3c2e2af6ded6b06371be617144a625903a6d562f4b/hf_xet-1.5.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f7b7bbae318e583a86fb21e5a4a175d6721d628a2874f4bd022d0e660c32a682", size = 4456738, upload-time = "2026-05-06T06:17:49.574Z" }, + { url = "https://files.pythonhosted.org/packages/c5/ca/f7effa1a67717da2bcc6b6c28f71c6ca648c77acaec4e2c32f40cbe16d85/hf_xet-1.5.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:cf7b2dc6f31a4ea754bb50f74cde482dcf5d366d184076d8530b9872787f3761", size = 4251622, upload-time = "2026-05-06T06:17:47.096Z" }, + { url = "https://files.pythonhosted.org/packages/65/f2/19247dba3e231cf77dec59ddfb878f00057635ff773d099c9b59d37812c3/hf_xet-1.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8dbcbab554c9ef158ef2c991545c3e970ddd8cc7acdcd0a78c5a41095dab4ded", size = 4445667, upload-time = "2026-05-06T06:18:11.983Z" }, + { url = "https://files.pythonhosted.org/packages/7f/64/6f116801a3bcfb6f59f5c251f48cadc47ea54026441c4a385079286a94fa/hf_xet-1.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5906bf7718d3636dc13402914736abe723492cb730f744834f5f5b67d3a12702", size = 4664619, upload-time = "2026-05-06T06:18:13.771Z" }, + { url = "https://files.pythonhosted.org/packages/5c/e8/069542d37946ed08669b127e1496fa99e78196d71de8d41eda5e9f1b7a58/hf_xet-1.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:5f3dc2248fc01cc0a00cd392ab497f1ca373fcbc7e3f2da1f452480b384e839e", size = 3966802, upload-time = "2026-05-06T06:18:28.162Z" }, + { url = "https://files.pythonhosted.org/packages/f9/91/fc6fdec27b14d04e88c386ac0a0129732b53fa23f7c4a78f4b83a039c567/hf_xet-1.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:b285cea1b5bab46b758772716ba8d6854a1a0310fed1c249d678a8b38601e5a0", size = 3797168, upload-time = "2026-05-06T06:18:26.287Z" }, + { url = "https://files.pythonhosted.org/packages/3d/fb/69ff198a82cae7eb1a69fb84d93b3a3e4816564d76817fe541ddc96874eb/hf_xet-1.5.0-cp37-abi3-macosx_10_12_x86_64.whl", hash = "sha256:dad0dc84e941b8ba3c860659fe1fdc35c049d47cce293f003287757e971a8f56", size = 4030814, upload-time = "2026-05-06T06:17:57.933Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ff/edcc2b40162bef3ff78e14ab637e5f3b89243d6aee72f5949d3bb6a5af83/hf_xet-1.5.0-cp37-abi3-macosx_11_0_arm64.whl", hash = "sha256:fd6e5a9b0fdac4ed03ed45ef79254a655b1aaab514a02202617fbf643f5fdf7a", size = 3798444, upload-time = "2026-05-06T06:17:55.79Z" }, + { url = "https://files.pythonhosted.org/packages/49/4d/103f76b04310e5e57656696cc184690d20c466af0bca3ca88f8c8ea5d4f3/hf_xet-1.5.0-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3531b1823a0e6d77d80f9ed15ca0e00f0d115094f8ac033d5cae88f4564cc949", size = 4465986, upload-time = "2026-05-06T06:17:44.886Z" }, + { url = "https://files.pythonhosted.org/packages/c4/a2/546f47f464737b3edbab6f8ddb57f2599b93d2cbb66f06abb475ccb48651/hf_xet-1.5.0-cp37-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:9a0ee58cd18d5ea799f7ed11290bbccbe56bdd8b1d97ca74b9cc49a3945d7a3b", size = 4259865, upload-time = "2026-05-06T06:17:42.639Z" }, + { url = "https://files.pythonhosted.org/packages/95/7f/1be593c1f28613be2e196473481cd81bfc5910795e30a34e8f744f6cac4f/hf_xet-1.5.0-cp37-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1e60df5a42e9bed8628b6416af2cba4cba57ae9f02de226a06b020d98e1aab18", size = 4459835, upload-time = "2026-05-06T06:18:08.026Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b2/703569fc881f3284487e68cda7b42179978480da3c438042a6bbbb4a671c/hf_xet-1.5.0-cp37-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:4b35549ce62601b84da4ff9b24d970032ace3d4430f52d91bcbb26c901d6c690", size = 4672414, upload-time = "2026-05-06T06:18:09.864Z" }, + { url = "https://files.pythonhosted.org/packages/af/37/1b6def445c567286b50aa3b33828158e135b1be44938dde59f11382a500c/hf_xet-1.5.0-cp37-abi3-win_amd64.whl", hash = "sha256:2806c7c17b4d23f8d88f7c4814f838c3b6150773fe339c20af23e1cfaf2797e4", size = 3977238, upload-time = "2026-05-06T06:18:23.621Z" }, + { url = "https://files.pythonhosted.org/packages/62/94/3b66b148778ee100dcfd69c2ca22b57b41b44d3063ceec934f209e9184ce/hf_xet-1.5.0-cp37-abi3-win_arm64.whl", hash = "sha256:b6c9df403040248c76d808d3e047d64db2d923bae593eb244c41e425cf6cd7be", size = 3806916, upload-time = "2026-05-06T06:18:21.7Z" }, +] + +[[package]] +name = "httpcore" +version = "1.0.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" }, +] + +[[package]] +name = "httptools" +version = "0.7.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b5/46/120a669232c7bdedb9d52d4aeae7e6c7dfe151e99dc70802e2fc7a5e1993/httptools-0.7.1.tar.gz", hash = "sha256:abd72556974f8e7c74a259655924a717a2365b236c882c3f6f8a45fe94703ac9", size = 258961, upload-time = "2025-10-10T03:55:08.559Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/53/7f/403e5d787dc4942316e515e949b0c8a013d84078a915910e9f391ba9b3ed/httptools-0.7.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:38e0c83a2ea9746ebbd643bdfb521b9aa4a91703e2cd705c20443405d2fd16a5", size = 206280, upload-time = "2025-10-10T03:54:39.274Z" }, + { url = "https://files.pythonhosted.org/packages/2a/0d/7f3fd28e2ce311ccc998c388dd1c53b18120fda3b70ebb022b135dc9839b/httptools-0.7.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f25bbaf1235e27704f1a7b86cd3304eabc04f569c828101d94a0e605ef7205a5", size = 110004, upload-time = "2025-10-10T03:54:40.403Z" }, + { url = "https://files.pythonhosted.org/packages/84/a6/b3965e1e146ef5762870bbe76117876ceba51a201e18cc31f5703e454596/httptools-0.7.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:2c15f37ef679ab9ecc06bfc4e6e8628c32a8e4b305459de7cf6785acd57e4d03", size = 517655, upload-time = "2025-10-10T03:54:41.347Z" }, + { url = "https://files.pythonhosted.org/packages/11/7d/71fee6f1844e6fa378f2eddde6c3e41ce3a1fb4b2d81118dd544e3441ec0/httptools-0.7.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7fe6e96090df46b36ccfaf746f03034e5ab723162bc51b0a4cf58305324036f2", size = 511440, upload-time = "2025-10-10T03:54:42.452Z" }, + { url = "https://files.pythonhosted.org/packages/22/a5/079d216712a4f3ffa24af4a0381b108aa9c45b7a5cc6eb141f81726b1823/httptools-0.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f72fdbae2dbc6e68b8239defb48e6a5937b12218e6ffc2c7846cc37befa84362", size = 495186, upload-time = "2025-10-10T03:54:43.937Z" }, + { url = "https://files.pythonhosted.org/packages/e9/9e/025ad7b65278745dee3bd0ebf9314934c4592560878308a6121f7f812084/httptools-0.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e99c7b90a29fd82fea9ef57943d501a16f3404d7b9ee81799d41639bdaae412c", size = 499192, upload-time = "2025-10-10T03:54:45.003Z" }, + { url = "https://files.pythonhosted.org/packages/6d/de/40a8f202b987d43afc4d54689600ff03ce65680ede2f31df348d7f368b8f/httptools-0.7.1-cp312-cp312-win_amd64.whl", hash = "sha256:3e14f530fefa7499334a79b0cf7e7cd2992870eb893526fb097d51b4f2d0f321", size = 86694, upload-time = "2025-10-10T03:54:45.923Z" }, + { url = "https://files.pythonhosted.org/packages/09/8f/c77b1fcbfd262d422f12da02feb0d218fa228d52485b77b953832105bb90/httptools-0.7.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:6babce6cfa2a99545c60bfef8bee0cc0545413cb0018f617c8059a30ad985de3", size = 202889, upload-time = "2025-10-10T03:54:47.089Z" }, + { url = "https://files.pythonhosted.org/packages/0a/1a/22887f53602feaa066354867bc49a68fc295c2293433177ee90870a7d517/httptools-0.7.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:601b7628de7504077dd3dcb3791c6b8694bbd967148a6d1f01806509254fb1ca", size = 108180, upload-time = "2025-10-10T03:54:48.052Z" }, + { url = "https://files.pythonhosted.org/packages/32/6a/6aaa91937f0010d288d3d124ca2946d48d60c3a5ee7ca62afe870e3ea011/httptools-0.7.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:04c6c0e6c5fb0739c5b8a9eb046d298650a0ff38cf42537fc372b28dc7e4472c", size = 478596, upload-time = "2025-10-10T03:54:48.919Z" }, + { url = "https://files.pythonhosted.org/packages/6d/70/023d7ce117993107be88d2cbca566a7c1323ccbaf0af7eabf2064fe356f6/httptools-0.7.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:69d4f9705c405ae3ee83d6a12283dc9feba8cc6aaec671b412917e644ab4fa66", size = 473268, upload-time = "2025-10-10T03:54:49.993Z" }, + { url = "https://files.pythonhosted.org/packages/32/4d/9dd616c38da088e3f436e9a616e1d0cc66544b8cdac405cc4e81c8679fc7/httptools-0.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:44c8f4347d4b31269c8a9205d8a5ee2df5322b09bbbd30f8f862185bb6b05346", size = 455517, upload-time = "2025-10-10T03:54:51.066Z" }, + { url = "https://files.pythonhosted.org/packages/1d/3a/a6c595c310b7df958e739aae88724e24f9246a514d909547778d776799be/httptools-0.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:465275d76db4d554918aba40bf1cbebe324670f3dfc979eaffaa5d108e2ed650", size = 458337, upload-time = "2025-10-10T03:54:52.196Z" }, + { url = "https://files.pythonhosted.org/packages/fd/82/88e8d6d2c51edc1cc391b6e044c6c435b6aebe97b1abc33db1b0b24cd582/httptools-0.7.1-cp313-cp313-win_amd64.whl", hash = "sha256:322d00c2068d125bd570f7bf78b2d367dad02b919d8581d7476d8b75b294e3e6", size = 85743, upload-time = "2025-10-10T03:54:53.448Z" }, + { url = "https://files.pythonhosted.org/packages/34/50/9d095fcbb6de2d523e027a2f304d4551855c2f46e0b82befd718b8b20056/httptools-0.7.1-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:c08fe65728b8d70b6923ce31e3956f859d5e1e8548e6f22ec520a962c6757270", size = 203619, upload-time = "2025-10-10T03:54:54.321Z" }, + { url = "https://files.pythonhosted.org/packages/07/f0/89720dc5139ae54b03f861b5e2c55a37dba9a5da7d51e1e824a1f343627f/httptools-0.7.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:7aea2e3c3953521c3c51106ee11487a910d45586e351202474d45472db7d72d3", size = 108714, upload-time = "2025-10-10T03:54:55.163Z" }, + { url = "https://files.pythonhosted.org/packages/b3/cb/eea88506f191fb552c11787c23f9a405f4c7b0c5799bf73f2249cd4f5228/httptools-0.7.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:0e68b8582f4ea9166be62926077a3334064d422cf08ab87d8b74664f8e9058e1", size = 472909, upload-time = "2025-10-10T03:54:56.056Z" }, + { url = "https://files.pythonhosted.org/packages/e0/4a/a548bdfae6369c0d078bab5769f7b66f17f1bfaa6fa28f81d6be6959066b/httptools-0.7.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:df091cf961a3be783d6aebae963cc9b71e00d57fa6f149025075217bc6a55a7b", size = 470831, upload-time = "2025-10-10T03:54:57.219Z" }, + { url = "https://files.pythonhosted.org/packages/4d/31/14df99e1c43bd132eec921c2e7e11cda7852f65619bc0fc5bdc2d0cb126c/httptools-0.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f084813239e1eb403ddacd06a30de3d3e09a9b76e7894dcda2b22f8a726e9c60", size = 452631, upload-time = "2025-10-10T03:54:58.219Z" }, + { url = "https://files.pythonhosted.org/packages/22/d2/b7e131f7be8d854d48cb6d048113c30f9a46dca0c9a8b08fcb3fcd588cdc/httptools-0.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:7347714368fb2b335e9063bc2b96f2f87a9ceffcd9758ac295f8bbcd3ffbc0ca", size = 452910, upload-time = "2025-10-10T03:54:59.366Z" }, + { url = "https://files.pythonhosted.org/packages/53/cf/878f3b91e4e6e011eff6d1fa9ca39f7eb17d19c9d7971b04873734112f30/httptools-0.7.1-cp314-cp314-win_amd64.whl", hash = "sha256:cfabda2a5bb85aa2a904ce06d974a3f30fb36cc63d7feaddec05d2050acede96", size = 88205, upload-time = "2025-10-10T03:55:00.389Z" }, +] + +[[package]] +name = "httpx" +version = "0.28.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "certifi" }, + { name = "httpcore" }, + { name = "idna" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" }, +] + +[[package]] +name = "huggingface-hub" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "filelock" }, + { name = "fsspec" }, + { name = "hf-xet", marker = "platform_machine == 'AMD64' or platform_machine == 'aarch64' or platform_machine == 'amd64' or platform_machine == 'arm64' or platform_machine == 'x86_64' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "httpx" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "tqdm" }, + { name = "typer" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/39/40/43109e943fd718b0ccd0cd61eb4f1c347df22bf81f5874c6f22adf44bcff/huggingface_hub-1.14.0.tar.gz", hash = "sha256:d6d2c9cd6be1d02ae9ec6672d5587d10a427f377db688e82528f426a041622c2", size = 782365, upload-time = "2026-05-06T14:14:34.278Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/89/a5/33b49ba7bea7c41bb37f74ec0f8beea0831e052330196633fe2c77516ea6/huggingface_hub-1.14.0-py3-none-any.whl", hash = "sha256:efe075535c62e130b30e836b138e13785f6f043d1f0539e0a39aa411a99e90b8", size = 661479, upload-time = "2026-05-06T14:14:32.029Z" }, +] + +[[package]] +name = "idna" +version = "3.15" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/77/7b3966d0b9d1d31a36ddf1746926a11dface89a83409bf1483f0237aa758/idna-3.15.tar.gz", hash = "sha256:ca962446ea538f7092a95e057da437618e886f4d349216d2b1e294abfdb65fdc", size = 199245, upload-time = "2026-05-12T22:45:57.011Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d2/23/408243171aa9aaba178d3e2559159c24c1171a641aa83b67bdd3394ead8e/idna-3.15-py3-none-any.whl", hash = "sha256:048adeaf8c2d788c40fee287673ccaa74c24ffd8dcf09ffa555a2fbb59f10ac8", size = 72340, upload-time = "2026-05-12T22:45:55.733Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "itsdangerous" +version = "2.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9c/cb/8ac0172223afbccb63986cc25049b154ecfb5e85932587206f42317be31d/itsdangerous-2.2.0.tar.gz", hash = "sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173", size = 54410, upload-time = "2024-04-16T21:28:15.614Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl", hash = "sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef", size = 16234, upload-time = "2024-04-16T21:28:14.499Z" }, +] + +[[package]] +name = "jinja2" +version = "3.1.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d", size = 245115, upload-time = "2025-03-05T20:05:02.478Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67", size = 134899, upload-time = "2025-03-05T20:05:00.369Z" }, +] + +[[package]] +name = "joblib" +version = "1.5.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/41/f2/d34e8b3a08a9cc79a50b2208a93dce981fe615b64d5a4d4abee421d898df/joblib-1.5.3.tar.gz", hash = "sha256:8561a3269e6801106863fd0d6d84bb737be9e7631e33aaed3fb9ce5953688da3", size = 331603, upload-time = "2025-12-15T08:41:46.427Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl", hash = "sha256:5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713", size = 309071, upload-time = "2025-12-15T08:41:44.973Z" }, +] + +[[package]] +name = "jsonpath-ng" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/32/58/250751940d75c8019659e15482d548a4aa3b6ce122c515102a4bfdac50e3/jsonpath_ng-1.8.0.tar.gz", hash = "sha256:54252968134b5e549ea5b872f1df1168bd7defe1a52fed5a358c194e1943ddc3", size = 74513, upload-time = "2026-02-24T14:42:06.182Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/99/33c7d78a3fb70d545fd5411ac67a651c81602cc09c9cf0df383733f068c5/jsonpath_ng-1.8.0-py3-none-any.whl", hash = "sha256:b8dde192f8af58d646fc031fac9c99fe4d00326afc4148f1f043c601a8cfe138", size = 67844, upload-time = "2026-02-28T00:53:19.637Z" }, +] + +[[package]] +name = "jsonschema" +version = "4.26.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "jsonschema-specifications" }, + { name = "referencing" }, + { name = "rpds-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" }, +] + +[[package]] +name = "jsonschema-specifications" +version = "2025.9.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "referencing" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, +] + +[[package]] +name = "kgcore" +version = "0.1.0" +source = { git = "https://github.com/Vehnem/kgcore.git#62aad8b1937fe613f4b28372d142961fbfe78547" } +dependencies = [ + { name = "click" }, + { name = "pydantic" }, + { name = "pytest" }, + { name = "python-dotenv" }, + { name = "rdflib" }, +] + +[[package]] +name = "kgpipe" +version = "0.7.0" +source = { editable = "." } +dependencies = [ + { name = "click" }, + { name = "docker" }, + { name = "dotenv" }, + { name = "fastapi" }, + { name = "jsonpath-ng" }, + { name = "kgcore" }, + { name = "matplotlib" }, + { name = "networkx" }, + { name = "pandas" }, + { name = "pulp" }, + { name = "pydantic" }, + { name = "pytest" }, + { name = "pyyaml" }, + { name = "rdflib" }, + { name = "redis" }, + { name = "rich" }, + { name = "scipy" }, + { name = "seaborn" }, + { name = "sparqlwrapper" }, + { name = "streamlit-elements" }, + { name = "tiktoken" }, + { name = "tqdm" }, + { name = "uvicorn" }, +] + +[package.optional-dependencies] +cpu = [ + { name = "torch", version = "2.12.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torch", version = "2.12.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torchaudio", version = "2.11.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torchaudio", version = "2.11.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torchvision", version = "0.27.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torchvision", version = "0.27.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +cuda = [ + { name = "torch", version = "2.12.0+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" } }, + { name = "torchaudio", version = "2.11.0+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" } }, + { name = "torchvision", version = "0.27.0+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" } }, +] +dev = [ + { name = "black" }, + { name = "pytest" }, + { name = "pytest-cov" }, + { name = "pytest-mock" }, + { name = "ruff" }, +] +docs = [ + { name = "mkdocs-material" }, + { name = "mkdocstrings", extra = ["python"] }, +] +ml = [ + { name = "sentence-transformers" }, + { name = "transformers" }, +] + +[package.metadata] +requires-dist = [ + { name = "black", marker = "extra == 'dev'" }, + { name = "click", specifier = ">=8.0" }, + { name = "docker", specifier = ">=7.0.0" }, + { name = "dotenv", specifier = ">=0.9.9" }, + { name = "fastapi", specifier = ">=0.135.1" }, + { name = "jsonpath-ng", specifier = ">=1.7.0" }, + { name = "kgcore", git = "https://github.com/Vehnem/kgcore.git" }, + { name = "matplotlib", specifier = ">=3.5.0" }, + { name = "mkdocs-material", marker = "extra == 'docs'" }, + { name = "mkdocstrings", extras = ["python"], marker = "extra == 'docs'" }, + { name = "networkx", specifier = ">=2.8.0" }, + { name = "pandas", specifier = ">=1.5.0" }, + { name = "pulp", specifier = ">=3.3.0" }, + { name = "pydantic", specifier = ">=2.0.0" }, + { name = "pytest", specifier = ">=8.4.2" }, + { name = "pytest", marker = "extra == 'dev'" }, + { name = "pytest-cov", marker = "extra == 'dev'" }, + { name = "pytest-mock", marker = "extra == 'dev'" }, + { name = "pyyaml", specifier = ">=6.0" }, + { name = "rdflib", specifier = ">=6.0.0" }, + { name = "redis", specifier = ">=7.0.0" }, + { name = "rich", specifier = ">=13.0.0" }, + { name = "ruff", marker = "extra == 'dev'" }, + { name = "scipy", specifier = ">=1.16.2" }, + { name = "seaborn", specifier = ">=0.13.2" }, + { name = "sentence-transformers", marker = "extra == 'ml'", specifier = ">=4.1.0" }, + { name = "sparqlwrapper", specifier = ">=2.0.0" }, + { name = "streamlit-elements", specifier = ">=0.1.0" }, + { name = "tiktoken", specifier = ">=0.11.0" }, + { name = "torch", marker = "extra == 'cpu'", index = "https://download.pytorch.org/whl/cpu", conflict = { package = "kgpipe", extra = "cpu" } }, + { name = "torch", marker = "extra == 'cuda'", index = "https://download.pytorch.org/whl/cu130", conflict = { package = "kgpipe", extra = "cuda" } }, + { name = "torchaudio", marker = "extra == 'cpu'", index = "https://download.pytorch.org/whl/cpu", conflict = { package = "kgpipe", extra = "cpu" } }, + { name = "torchaudio", marker = "extra == 'cuda'", index = "https://download.pytorch.org/whl/cu130", conflict = { package = "kgpipe", extra = "cuda" } }, + { name = "torchvision", marker = "extra == 'cpu'", index = "https://download.pytorch.org/whl/cpu", conflict = { package = "kgpipe", extra = "cpu" } }, + { name = "torchvision", marker = "extra == 'cuda'", index = "https://download.pytorch.org/whl/cu130", conflict = { package = "kgpipe", extra = "cuda" } }, + { name = "tqdm", specifier = ">=4.67.1" }, + { name = "transformers", marker = "extra == 'ml'", specifier = ">=4.50.0" }, + { name = "uvicorn", specifier = ">=0.41.0" }, +] +provides-extras = ["dev", "docs", "cpu", "cuda", "ml"] + +[[package]] +name = "kiwisolver" +version = "1.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d0/67/9c61eccb13f0bdca9307614e782fec49ffdde0f7a2314935d489fa93cd9c/kiwisolver-1.5.0.tar.gz", hash = "sha256:d4193f3d9dc3f6f79aaed0e5637f45d98850ebf01f7ca20e69457f3e8946b66a", size = 103482, upload-time = "2026-03-09T13:15:53.382Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4d/b2/818b74ebea34dabe6d0c51cb1c572e046730e64844da6ed646d5298c40ce/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:4e9750bc21b886308024f8a54ccb9a2cc38ac9fa813bf4348434e3d54f337ff9", size = 123158, upload-time = "2026-03-09T13:13:23.127Z" }, + { url = "https://files.pythonhosted.org/packages/bf/d9/405320f8077e8e1c5c4bd6adc45e1e6edf6d727b6da7f2e2533cf58bff71/kiwisolver-1.5.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:72ec46b7eba5b395e0a7b63025490d3214c11013f4aacb4f5e8d6c3041829588", size = 66388, upload-time = "2026-03-09T13:13:24.765Z" }, + { url = "https://files.pythonhosted.org/packages/99/9f/795fedf35634f746151ca8839d05681ceb6287fbed6cc1c9bf235f7887c2/kiwisolver-1.5.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ed3a984b31da7481b103f68776f7128a89ef26ed40f4dc41a2223cda7fb24819", size = 64068, upload-time = "2026-03-09T13:13:25.878Z" }, + { url = "https://files.pythonhosted.org/packages/c4/13/680c54afe3e65767bed7ec1a15571e1a2f1257128733851ade24abcefbcc/kiwisolver-1.5.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb5136fb5352d3f422df33f0c879a1b0c204004324150cc3b5e3c4f310c9049f", size = 1477934, upload-time = "2026-03-09T13:13:27.166Z" }, + { url = "https://files.pythonhosted.org/packages/c8/2f/cebfcdb60fd6a9b0f6b47a9337198bcbad6fbe15e68189b7011fd914911f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b2af221f268f5af85e776a73d62b0845fc8baf8ef0abfae79d29c77d0e776aaf", size = 1278537, upload-time = "2026-03-09T13:13:28.707Z" }, + { url = "https://files.pythonhosted.org/packages/f2/0d/9b782923aada3fafb1d6b84e13121954515c669b18af0c26e7d21f579855/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b0f172dc8ffaccb8522d7c5d899de00133f2f1ca7b0a49b7da98e901de87bf2d", size = 1296685, upload-time = "2026-03-09T13:13:30.528Z" }, + { url = "https://files.pythonhosted.org/packages/27/70/83241b6634b04fe44e892688d5208332bde130f38e610c0418f9ede47ded/kiwisolver-1.5.0-cp312-cp312-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:6ab8ba9152203feec73758dad83af9a0bbe05001eb4639e547207c40cfb52083", size = 1346024, upload-time = "2026-03-09T13:13:32.818Z" }, + { url = "https://files.pythonhosted.org/packages/e4/db/30ed226fb271ae1a6431fc0fe0edffb2efe23cadb01e798caeb9f2ceae8f/kiwisolver-1.5.0-cp312-cp312-manylinux_2_39_riscv64.whl", hash = "sha256:cdee07c4d7f6d72008d3f73b9bf027f4e11550224c7c50d8df1ae4a37c1402a6", size = 987241, upload-time = "2026-03-09T13:13:34.435Z" }, + { url = "https://files.pythonhosted.org/packages/ec/bd/c314595208e4c9587652d50959ead9e461995389664e490f4dce7ff0f782/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7c60d3c9b06fb23bd9c6139281ccbdc384297579ae037f08ae90c69f6845c0b1", size = 2227742, upload-time = "2026-03-09T13:13:36.4Z" }, + { url = "https://files.pythonhosted.org/packages/c1/43/0499cec932d935229b5543d073c2b87c9c22846aab48881e9d8d6e742a2d/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:e315e5ec90d88e140f57696ff85b484ff68bb311e36f2c414aa4286293e6dee0", size = 2323966, upload-time = "2026-03-09T13:13:38.204Z" }, + { url = "https://files.pythonhosted.org/packages/3d/6f/79b0d760907965acfd9d61826a3d41f8f093c538f55cd2633d3f0db269f6/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:1465387ac63576c3e125e5337a6892b9e99e0627d52317f3ca79e6930d889d15", size = 1977417, upload-time = "2026-03-09T13:13:39.966Z" }, + { url = "https://files.pythonhosted.org/packages/ab/31/01d0537c41cb75a551a438c3c7a80d0c60d60b81f694dac83dd436aec0d0/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:530a3fd64c87cffa844d4b6b9768774763d9caa299e9b75d8eca6a4423b31314", size = 2491238, upload-time = "2026-03-09T13:13:41.698Z" }, + { url = "https://files.pythonhosted.org/packages/e4/34/8aefdd0be9cfd00a44509251ba864f5caf2991e36772e61c408007e7f417/kiwisolver-1.5.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1d9daea4ea6b9be74fe2f01f7fbade8d6ffab263e781274cffca0dba9be9eec9", size = 2294947, upload-time = "2026-03-09T13:13:43.343Z" }, + { url = "https://files.pythonhosted.org/packages/ad/cf/0348374369ca588f8fe9c338fae49fa4e16eeb10ffb3d012f23a54578a9e/kiwisolver-1.5.0-cp312-cp312-win_amd64.whl", hash = "sha256:f18c2d9782259a6dc132fdc7a63c168cbc74b35284b6d75c673958982a378384", size = 73569, upload-time = "2026-03-09T13:13:45.792Z" }, + { url = "https://files.pythonhosted.org/packages/28/26/192b26196e2316e2bd29deef67e37cdf9870d9af8e085e521afff0fed526/kiwisolver-1.5.0-cp312-cp312-win_arm64.whl", hash = "sha256:f7c7553b13f69c1b29a5bde08ddc6d9d0c8bfb84f9ed01c30db25944aeb852a7", size = 64997, upload-time = "2026-03-09T13:13:46.878Z" }, + { url = "https://files.pythonhosted.org/packages/9d/69/024d6711d5ba575aa65d5538042e99964104e97fa153a9f10bc369182bc2/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:fd40bb9cd0891c4c3cb1ddf83f8bbfa15731a248fdc8162669405451e2724b09", size = 123166, upload-time = "2026-03-09T13:13:48.032Z" }, + { url = "https://files.pythonhosted.org/packages/ce/48/adbb40df306f587054a348831220812b9b1d787aff714cfbc8556e38fccd/kiwisolver-1.5.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c0e1403fd7c26d77c1f03e096dc58a5c726503fa0db0456678b8668f76f521e3", size = 66395, upload-time = "2026-03-09T13:13:49.365Z" }, + { url = "https://files.pythonhosted.org/packages/a8/3a/d0a972b34e1c63e2409413104216cd1caa02c5a37cb668d1687d466c1c45/kiwisolver-1.5.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:dda366d548e89a90d88a86c692377d18d8bd64b39c1fb2b92cb31370e2896bbd", size = 64065, upload-time = "2026-03-09T13:13:50.562Z" }, + { url = "https://files.pythonhosted.org/packages/2b/0a/7b98e1e119878a27ba8618ca1e18b14f992ff1eda40f47bccccf4de44121/kiwisolver-1.5.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:332b4f0145c30b5f5ad9374881133e5aa64320428a57c2c2b61e9d891a51c2f3", size = 1477903, upload-time = "2026-03-09T13:13:52.084Z" }, + { url = "https://files.pythonhosted.org/packages/18/d8/55638d89ffd27799d5cc3d8aa28e12f4ce7a64d67b285114dbedc8ea4136/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0c50b89ffd3e1a911c69a1dd3de7173c0cd10b130f56222e57898683841e4f96", size = 1278751, upload-time = "2026-03-09T13:13:54.673Z" }, + { url = "https://files.pythonhosted.org/packages/b8/97/b4c8d0d18421ecceba20ad8701358453b88e32414e6f6950b5a4bad54e65/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4db576bb8c3ef9365f8b40fe0f671644de6736ae2c27a2c62d7d8a1b4329f099", size = 1296793, upload-time = "2026-03-09T13:13:56.287Z" }, + { url = "https://files.pythonhosted.org/packages/c4/10/f862f94b6389d8957448ec9df59450b81bec4abb318805375c401a1e6892/kiwisolver-1.5.0-cp313-cp313-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:0b85aad90cea8ac6797a53b5d5f2e967334fa4d1149f031c4537569972596cb8", size = 1346041, upload-time = "2026-03-09T13:13:58.269Z" }, + { url = "https://files.pythonhosted.org/packages/a3/6a/f1650af35821eaf09de398ec0bc2aefc8f211f0cda50204c9f1673741ba9/kiwisolver-1.5.0-cp313-cp313-manylinux_2_39_riscv64.whl", hash = "sha256:d36ca54cb4c6c4686f7cbb7b817f66f5911c12ddb519450bbe86707155028f87", size = 987292, upload-time = "2026-03-09T13:13:59.871Z" }, + { url = "https://files.pythonhosted.org/packages/de/19/d7fb82984b9238115fe629c915007be608ebd23dc8629703d917dbfaffd4/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:38f4a703656f493b0ad185211ccfca7f0386120f022066b018eb5296d8613e23", size = 2227865, upload-time = "2026-03-09T13:14:01.401Z" }, + { url = "https://files.pythonhosted.org/packages/7f/b9/46b7f386589fd222dac9e9de9c956ce5bcefe2ee73b4e79891381dda8654/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3ac2360e93cb41be81121755c6462cff3beaa9967188c866e5fce5cf13170859", size = 2324369, upload-time = "2026-03-09T13:14:02.972Z" }, + { url = "https://files.pythonhosted.org/packages/92/8b/95e237cf3d9c642960153c769ddcbe278f182c8affb20cecc1cc983e7cc5/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:c95cab08d1965db3d84a121f1c7ce7479bdd4072c9b3dafd8fecce48a2e6b902", size = 1977989, upload-time = "2026-03-09T13:14:04.503Z" }, + { url = "https://files.pythonhosted.org/packages/1b/95/980c9df53501892784997820136c01f62bc1865e31b82b9560f980c0e649/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:fc20894c3d21194d8041a28b65622d5b86db786da6e3cfe73f0c762951a61167", size = 2491645, upload-time = "2026-03-09T13:14:06.106Z" }, + { url = "https://files.pythonhosted.org/packages/cb/32/900647fd0840abebe1561792c6b31e6a7c0e278fc3973d30572a965ca14c/kiwisolver-1.5.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:7a32f72973f0f950c1920475d5c5ea3d971b81b6f0ec53b8d0a956cc965f22e0", size = 2295237, upload-time = "2026-03-09T13:14:08.891Z" }, + { url = "https://files.pythonhosted.org/packages/be/8a/be60e3bbcf513cc5a50f4a3e88e1dcecebb79c1ad607a7222877becaa101/kiwisolver-1.5.0-cp313-cp313-win_amd64.whl", hash = "sha256:0bf3acf1419fa93064a4c2189ac0b58e3be7872bf6ee6177b0d4c63dc4cea276", size = 73573, upload-time = "2026-03-09T13:14:12.327Z" }, + { url = "https://files.pythonhosted.org/packages/4d/d2/64be2e429eb4fca7f7e1c52a91b12663aeaf25de3895e5cca0f47ef2a8d0/kiwisolver-1.5.0-cp313-cp313-win_arm64.whl", hash = "sha256:fa8eb9ecdb7efb0b226acec134e0d709e87a909fa4971a54c0c4f6e88635484c", size = 64998, upload-time = "2026-03-09T13:14:13.469Z" }, + { url = "https://files.pythonhosted.org/packages/b0/69/ce68dd0c85755ae2de490bf015b62f2cea5f6b14ff00a463f9d0774449ff/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:db485b3847d182b908b483b2ed133c66d88d49cacf98fd278fadafe11b4478d1", size = 125700, upload-time = "2026-03-09T13:14:14.636Z" }, + { url = "https://files.pythonhosted.org/packages/74/aa/937aac021cf9d4349990d47eb319309a51355ed1dbdc9c077cdc9224cb11/kiwisolver-1.5.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:be12f931839a3bdfe28b584db0e640a65a8bcbc24560ae3fdb025a449b3d754e", size = 67537, upload-time = "2026-03-09T13:14:15.808Z" }, + { url = "https://files.pythonhosted.org/packages/ee/20/3a87fbece2c40ad0f6f0aefa93542559159c5f99831d596050e8afae7a9f/kiwisolver-1.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:16b85d37c2cbb3253226d26e64663f755d88a03439a9c47df6246b35defbdfb7", size = 65514, upload-time = "2026-03-09T13:14:18.035Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7f/f943879cda9007c45e1f7dba216d705c3a18d6b35830e488b6c6a4e7cdf0/kiwisolver-1.5.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4432b835675f0ea7414aab3d37d119f7226d24869b7a829caeab49ebda407b0c", size = 1584848, upload-time = "2026-03-09T13:14:19.745Z" }, + { url = "https://files.pythonhosted.org/packages/37/f8/4d4f85cc1870c127c88d950913370dd76138482161cd07eabbc450deff01/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b0feb50971481a2cc44d94e88bdb02cdd497618252ae226b8eb1201b957e368", size = 1391542, upload-time = "2026-03-09T13:14:21.54Z" }, + { url = "https://files.pythonhosted.org/packages/04/0b/65dd2916c84d252b244bd405303220f729e7c17c9d7d33dca6feeff9ffc4/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:56fa888f10d0f367155e76ce849fa1166fc9730d13bd2d65a2aa13b6f5424489", size = 1404447, upload-time = "2026-03-09T13:14:23.205Z" }, + { url = "https://files.pythonhosted.org/packages/39/5c/2606a373247babce9b1d056c03a04b65f3cf5290a8eac5d7bdead0a17e21/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:940dda65d5e764406b9fb92761cbf462e4e63f712ab60ed98f70552e496f3bf1", size = 1455918, upload-time = "2026-03-09T13:14:24.74Z" }, + { url = "https://files.pythonhosted.org/packages/d5/d1/c6078b5756670658e9192a2ef11e939c92918833d2745f85cd14a6004bdf/kiwisolver-1.5.0-cp313-cp313t-manylinux_2_39_riscv64.whl", hash = "sha256:89fc958c702ee9a745e4700378f5d23fddbc46ff89e8fdbf5395c24d5c1452a3", size = 1072856, upload-time = "2026-03-09T13:14:26.597Z" }, + { url = "https://files.pythonhosted.org/packages/cb/c8/7def6ddf16eb2b3741d8b172bdaa9af882b03c78e9b0772975408801fa63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9027d773c4ff81487181a925945743413f6069634d0b122d0b37684ccf4f1e18", size = 2333580, upload-time = "2026-03-09T13:14:28.237Z" }, + { url = "https://files.pythonhosted.org/packages/9e/87/2ac1fce0eb1e616fcd3c35caa23e665e9b1948bb984f4764790924594128/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:5b233ea3e165e43e35dba1d2b8ecc21cf070b45b65ae17dd2747d2713d942021", size = 2423018, upload-time = "2026-03-09T13:14:30.018Z" }, + { url = "https://files.pythonhosted.org/packages/67/13/c6700ccc6cc218716bfcda4935e4b2997039869b4ad8a94f364c5a3b8e63/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ce9bf03dad3b46408c08649c6fbd6ca28a9fce0eb32fdfffa6775a13103b5310", size = 2062804, upload-time = "2026-03-09T13:14:32.888Z" }, + { url = "https://files.pythonhosted.org/packages/1b/bd/877056304626943ff0f1f44c08f584300c199b887cb3176cd7e34f1515f1/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:fc4d3f1fb9ca0ae9f97b095963bc6326f1dbfd3779d6679a1e016b9baaa153d3", size = 2597482, upload-time = "2026-03-09T13:14:34.971Z" }, + { url = "https://files.pythonhosted.org/packages/75/19/c60626c47bf0f8ac5dcf72c6c98e266d714f2fbbfd50cf6dab5ede3aaa50/kiwisolver-1.5.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:f443b4825c50a51ee68585522ab4a1d1257fac65896f282b4c6763337ac9f5d2", size = 2394328, upload-time = "2026-03-09T13:14:36.816Z" }, + { url = "https://files.pythonhosted.org/packages/47/84/6a6d5e5bb8273756c27b7d810d47f7ef2f1f9b9fd23c9ee9a3f8c75c9cef/kiwisolver-1.5.0-cp313-cp313t-win_arm64.whl", hash = "sha256:893ff3a711d1b515ba9da14ee090519bad4610ed1962fbe298a434e8c5f8db53", size = 68410, upload-time = "2026-03-09T13:14:38.695Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d7/060f45052f2a01ad5762c8fdecd6d7a752b43400dc29ff75cd47225a40fd/kiwisolver-1.5.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8df31fe574b8b3993cc61764f40941111b25c2d9fea13d3ce24a49907cd2d615", size = 123231, upload-time = "2026-03-09T13:14:41.323Z" }, + { url = "https://files.pythonhosted.org/packages/c2/a7/78da680eadd06ff35edef6ef68a1ad273bad3e2a0936c9a885103230aece/kiwisolver-1.5.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:1d49a49ac4cbfb7c1375301cd1ec90169dfeae55ff84710d782260ce77a75a02", size = 66489, upload-time = "2026-03-09T13:14:42.534Z" }, + { url = "https://files.pythonhosted.org/packages/49/b2/97980f3ad4fae37dd7fe31626e2bf75fbf8bdf5d303950ec1fab39a12da8/kiwisolver-1.5.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0cbe94b69b819209a62cb27bdfa5dc2a8977d8de2f89dfd97ba4f53ed3af754e", size = 64063, upload-time = "2026-03-09T13:14:44.759Z" }, + { url = "https://files.pythonhosted.org/packages/e7/f9/b06c934a6aa8bc91f566bd2a214fd04c30506c2d9e2b6b171953216a65b6/kiwisolver-1.5.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80aa065ffd378ff784822a6d7c3212f2d5f5e9c3589614b5c228b311fd3063ac", size = 1475913, upload-time = "2026-03-09T13:14:46.247Z" }, + { url = "https://files.pythonhosted.org/packages/6b/f0/f768ae564a710135630672981231320bc403cf9152b5596ec5289de0f106/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e7f886f47ab881692f278ae901039a234e4025a68e6dfab514263a0b1c4ae05", size = 1282782, upload-time = "2026-03-09T13:14:48.458Z" }, + { url = "https://files.pythonhosted.org/packages/e2/9f/1de7aad00697325f05238a5f2eafbd487fb637cc27a558b5367a5f37fb7f/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5060731cc3ed12ca3a8b57acd4aeca5bbc2f49216dd0bec1650a1acd89486bcd", size = 1300815, upload-time = "2026-03-09T13:14:50.721Z" }, + { url = "https://files.pythonhosted.org/packages/5a/c2/297f25141d2e468e0ce7f7a7b92e0cf8918143a0cbd3422c1ad627e85a06/kiwisolver-1.5.0-cp314-cp314-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:7a4aa69609f40fce3cbc3f87b2061f042eee32f94b8f11db707b66a26461591a", size = 1347925, upload-time = "2026-03-09T13:14:52.304Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d3/f4c73a02eb41520c47610207b21afa8cdd18fdbf64ffd94674ae21c4812d/kiwisolver-1.5.0-cp314-cp314-manylinux_2_39_riscv64.whl", hash = "sha256:d168fda2dbff7b9b5f38e693182d792a938c31db4dac3a80a4888de603c99554", size = 991322, upload-time = "2026-03-09T13:14:54.637Z" }, + { url = "https://files.pythonhosted.org/packages/7b/46/d3f2efef7732fcda98d22bf4ad5d3d71d545167a852ca710a494f4c15343/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:413b820229730d358efd838ecbab79902fe97094565fdc80ddb6b0a18c18a581", size = 2232857, upload-time = "2026-03-09T13:14:56.471Z" }, + { url = "https://files.pythonhosted.org/packages/3f/ec/2d9756bf2b6d26ae4349b8d3662fb3993f16d80c1f971c179ce862b9dbae/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:5124d1ea754509b09e53738ec185584cc609aae4a3b510aaf4ed6aa047ef9303", size = 2329376, upload-time = "2026-03-09T13:14:58.072Z" }, + { url = "https://files.pythonhosted.org/packages/8f/9f/876a0a0f2260f1bde92e002b3019a5fabc35e0939c7d945e0fa66185eb20/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e4415a8db000bf49a6dd1c478bf70062eaacff0f462b92b0ba68791a905861f9", size = 1982549, upload-time = "2026-03-09T13:14:59.668Z" }, + { url = "https://files.pythonhosted.org/packages/6c/4f/ba3624dfac23a64d54ac4179832860cb537c1b0af06024936e82ca4154a0/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:d618fd27420381a4f6044faa71f46d8bfd911bd077c555f7138ed88729bfbe79", size = 2494680, upload-time = "2026-03-09T13:15:01.364Z" }, + { url = "https://files.pythonhosted.org/packages/39/b7/97716b190ab98911b20d10bf92eca469121ec483b8ce0edd314f51bc85af/kiwisolver-1.5.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5092eb5b1172947f57d6ea7d89b2f29650414e4293c47707eb499ec07a0ac796", size = 2297905, upload-time = "2026-03-09T13:15:03.925Z" }, + { url = "https://files.pythonhosted.org/packages/a3/36/4e551e8aa55c9188bca9abb5096805edbf7431072b76e2298e34fd3a3008/kiwisolver-1.5.0-cp314-cp314-win_amd64.whl", hash = "sha256:d76e2d8c75051d58177e762164d2e9ab92886534e3a12e795f103524f221dd8e", size = 75086, upload-time = "2026-03-09T13:15:07.775Z" }, + { url = "https://files.pythonhosted.org/packages/70/15/9b90f7df0e31a003c71649cf66ef61c3c1b862f48c81007fa2383c8bd8d7/kiwisolver-1.5.0-cp314-cp314-win_arm64.whl", hash = "sha256:fa6248cd194edff41d7ea9425ced8ca3a6f838bfb295f6f1d6e6bb694a8518df", size = 66577, upload-time = "2026-03-09T13:15:09.139Z" }, + { url = "https://files.pythonhosted.org/packages/17/01/7dc8c5443ff42b38e72731643ed7cf1ed9bf01691ae5cdca98501999ed83/kiwisolver-1.5.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:d1ffeb80b5676463d7a7d56acbe8e37a20ce725570e09549fe738e02ca6b7e1e", size = 125794, upload-time = "2026-03-09T13:15:10.525Z" }, + { url = "https://files.pythonhosted.org/packages/46/8a/b4ebe46ebaac6a303417fab10c2e165c557ddaff558f9699d302b256bc53/kiwisolver-1.5.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:bc4d8e252f532ab46a1de9349e2d27b91fce46736a9eedaa37beaca66f574ed4", size = 67646, upload-time = "2026-03-09T13:15:12.016Z" }, + { url = "https://files.pythonhosted.org/packages/60/35/10a844afc5f19d6f567359bf4789e26661755a2f36200d5d1ed8ad0126e5/kiwisolver-1.5.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:6783e069732715ad0c3ce96dbf21dbc2235ab0593f2baf6338101f70371f4028", size = 65511, upload-time = "2026-03-09T13:15:13.311Z" }, + { url = "https://files.pythonhosted.org/packages/f8/8a/685b297052dd041dcebce8e8787b58923b6e78acc6115a0dc9189011c44b/kiwisolver-1.5.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e7c4c09a490dc4d4a7f8cbee56c606a320f9dc28cf92a7157a39d1ce7676a657", size = 1584858, upload-time = "2026-03-09T13:15:15.103Z" }, + { url = "https://files.pythonhosted.org/packages/9e/80/04865e3d4638ac5bddec28908916df4a3075b8c6cc101786a96803188b96/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2a075bd7bd19c70cf67c8badfa36cf7c5d8de3c9ddb8420c51e10d9c50e94920", size = 1392539, upload-time = "2026-03-09T13:15:16.661Z" }, + { url = "https://files.pythonhosted.org/packages/ba/01/77a19cacc0893fa13fafa46d1bba06fb4dc2360b3292baf4b56d8e067b24/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:bdd3e53429ff02aa319ba59dfe4ceeec345bf46cf180ec2cf6fd5b942e7975e9", size = 1405310, upload-time = "2026-03-09T13:15:18.229Z" }, + { url = "https://files.pythonhosted.org/packages/53/39/bcaf5d0cca50e604cfa9b4e3ae1d64b50ca1ae5b754122396084599ef903/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_24_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3cdcb35dc9d807259c981a85531048ede628eabcffb3239adf3d17463518992d", size = 1456244, upload-time = "2026-03-09T13:15:20.444Z" }, + { url = "https://files.pythonhosted.org/packages/d0/7a/72c187abc6975f6978c3e39b7cf67aeb8b3c0a8f9790aa7fd412855e9e1f/kiwisolver-1.5.0-cp314-cp314t-manylinux_2_39_riscv64.whl", hash = "sha256:70d593af6a6ca332d1df73d519fddb5148edb15cd90d5f0155e3746a6d4fcc65", size = 1073154, upload-time = "2026-03-09T13:15:22.039Z" }, + { url = "https://files.pythonhosted.org/packages/c7/ca/cf5b25783ebbd59143b4371ed0c8428a278abe68d6d0104b01865b1bbd0f/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:377815a8616074cabbf3f53354e1d040c35815a134e01d7614b7692e4bf8acfa", size = 2334377, upload-time = "2026-03-09T13:15:23.741Z" }, + { url = "https://files.pythonhosted.org/packages/4a/e5/b1f492adc516796e88751282276745340e2a72dcd0d36cf7173e0daf3210/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:0255a027391d52944eae1dbb5d4cc5903f57092f3674e8e544cdd2622826b3f0", size = 2425288, upload-time = "2026-03-09T13:15:25.789Z" }, + { url = "https://files.pythonhosted.org/packages/e6/e5/9b21fbe91a61b8f409d74a26498706e97a48008bfcd1864373d32a6ba31c/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:012b1eb16e28718fa782b5e61dc6f2da1f0792ca73bd05d54de6cb9561665fc9", size = 2063158, upload-time = "2026-03-09T13:15:27.63Z" }, + { url = "https://files.pythonhosted.org/packages/b1/02/83f47986138310f95ea95531f851b2a62227c11cbc3e690ae1374fe49f0f/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:0e3aafb33aed7479377e5e9a82e9d4bf87063741fc99fc7ae48b0f16e32bdd6f", size = 2597260, upload-time = "2026-03-09T13:15:29.421Z" }, + { url = "https://files.pythonhosted.org/packages/07/18/43a5f24608d8c313dd189cf838c8e68d75b115567c6279de7796197cfb6a/kiwisolver-1.5.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:e7a116ae737f0000343218c4edf5bd45893bfeaff0993c0b215d7124c9f77646", size = 2394403, upload-time = "2026-03-09T13:15:31.517Z" }, + { url = "https://files.pythonhosted.org/packages/3b/b5/98222136d839b8afabcaa943b09bd05888c2d36355b7e448550211d1fca4/kiwisolver-1.5.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1dd9b0b119a350976a6d781e7278ec7aca0b201e1a9e2d23d9804afecb6ca681", size = 79687, upload-time = "2026-03-09T13:15:33.204Z" }, + { url = "https://files.pythonhosted.org/packages/99/a2/ca7dc962848040befed12732dff6acae7fb3c4f6fc4272b3f6c9a30b8713/kiwisolver-1.5.0-cp314-cp314t-win_arm64.whl", hash = "sha256:58f812017cd2985c21fbffb4864d59174d4903dd66fa23815e74bbc7a0e2dd57", size = 70032, upload-time = "2026-03-09T13:15:34.411Z" }, + { url = "https://files.pythonhosted.org/packages/1c/fa/2910df836372d8761bb6eff7d8bdcb1613b5c2e03f260efe7abe34d388a7/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_10_13_x86_64.whl", hash = "sha256:5ae8e62c147495b01a0f4765c878e9bfdf843412446a247e28df59936e99e797", size = 130262, upload-time = "2026-03-09T13:15:35.629Z" }, + { url = "https://files.pythonhosted.org/packages/0f/41/c5f71f9f00aabcc71fee8b7475e3f64747282580c2fe748961ba29b18385/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:f6764a4ccab3078db14a632420930f6186058750df066b8ea2a7106df91d3203", size = 138036, upload-time = "2026-03-09T13:15:36.894Z" }, + { url = "https://files.pythonhosted.org/packages/fa/06/7399a607f434119c6e1fdc8ec89a8d51ccccadf3341dee4ead6bd14caaf5/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c31c13da98624f957b0fb1b5bae5383b2333c2c3f6793d9825dd5ce79b525cb7", size = 194295, upload-time = "2026-03-09T13:15:38.22Z" }, + { url = "https://files.pythonhosted.org/packages/b5/91/53255615acd2a1eaca307ede3c90eb550bae9c94581f8c00081b6b1c8f44/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:1f1489f769582498610e015a8ef2d36f28f505ab3096d0e16b4858a9ec214f57", size = 75987, upload-time = "2026-03-09T13:15:39.65Z" }, +] + +[[package]] +name = "markdown" +version = "3.10.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2b/f4/69fa6ed85ae003c2378ffa8f6d2e3234662abd02c10d216c0ba96081a238/markdown-3.10.2.tar.gz", hash = "sha256:994d51325d25ad8aa7ce4ebaec003febcce822c3f8c911e3b17c52f7f589f950", size = 368805, upload-time = "2026-02-09T14:57:26.942Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/de/1f/77fa3081e4f66ca3576c896ae5d31c3002ac6607f9747d2e3aa49227e464/markdown-3.10.2-py3-none-any.whl", hash = "sha256:e91464b71ae3ee7afd3017d9f358ef0baf158fd9a298db92f1d4761133824c36", size = 108180, upload-time = "2026-02-09T14:57:25.787Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[[package]] +name = "markupsafe" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7e/99/7690b6d4034fffd95959cbe0c02de8deb3098cc577c67bb6a24fe5d7caa7/markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698", size = 80313, upload-time = "2025-09-27T18:37:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/72/147da192e38635ada20e0a2e1a51cf8823d2119ce8883f7053879c2199b5/markupsafe-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:d53197da72cc091b024dd97249dfc7794d6a56530370992a5e1a08983ad9230e", size = 11615, upload-time = "2025-09-27T18:36:30.854Z" }, + { url = "https://files.pythonhosted.org/packages/9a/81/7e4e08678a1f98521201c3079f77db69fb552acd56067661f8c2f534a718/markupsafe-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:1872df69a4de6aead3491198eaf13810b565bdbeec3ae2dc8780f14458ec73ce", size = 12020, upload-time = "2025-09-27T18:36:31.971Z" }, + { url = "https://files.pythonhosted.org/packages/1e/2c/799f4742efc39633a1b54a92eec4082e4f815314869865d876824c257c1e/markupsafe-3.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a7e8ae81ae39e62a41ec302f972ba6ae23a5c5396c8e60113e9066ef893da0d", size = 24332, upload-time = "2025-09-27T18:36:32.813Z" }, + { url = "https://files.pythonhosted.org/packages/3c/2e/8d0c2ab90a8c1d9a24f0399058ab8519a3279d1bd4289511d74e909f060e/markupsafe-3.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d6dd0be5b5b189d31db7cda48b91d7e0a9795f31430b7f271219ab30f1d3ac9d", size = 22947, upload-time = "2025-09-27T18:36:33.86Z" }, + { url = "https://files.pythonhosted.org/packages/2c/54/887f3092a85238093a0b2154bd629c89444f395618842e8b0c41783898ea/markupsafe-3.0.3-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:94c6f0bb423f739146aec64595853541634bde58b2135f27f61c1ffd1cd4d16a", size = 21962, upload-time = "2025-09-27T18:36:35.099Z" }, + { url = "https://files.pythonhosted.org/packages/c9/2f/336b8c7b6f4a4d95e91119dc8521402461b74a485558d8f238a68312f11c/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:be8813b57049a7dc738189df53d69395eba14fb99345e0a5994914a3864c8a4b", size = 23760, upload-time = "2025-09-27T18:36:36.001Z" }, + { url = "https://files.pythonhosted.org/packages/32/43/67935f2b7e4982ffb50a4d169b724d74b62a3964bc1a9a527f5ac4f1ee2b/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:83891d0e9fb81a825d9a6d61e3f07550ca70a076484292a70fde82c4b807286f", size = 21529, upload-time = "2025-09-27T18:36:36.906Z" }, + { url = "https://files.pythonhosted.org/packages/89/e0/4486f11e51bbba8b0c041098859e869e304d1c261e59244baa3d295d47b7/markupsafe-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:77f0643abe7495da77fb436f50f8dab76dbc6e5fd25d39589a0f1fe6548bfa2b", size = 23015, upload-time = "2025-09-27T18:36:37.868Z" }, + { url = "https://files.pythonhosted.org/packages/2f/e1/78ee7a023dac597a5825441ebd17170785a9dab23de95d2c7508ade94e0e/markupsafe-3.0.3-cp312-cp312-win32.whl", hash = "sha256:d88b440e37a16e651bda4c7c2b930eb586fd15ca7406cb39e211fcff3bf3017d", size = 14540, upload-time = "2025-09-27T18:36:38.761Z" }, + { url = "https://files.pythonhosted.org/packages/aa/5b/bec5aa9bbbb2c946ca2733ef9c4ca91c91b6a24580193e891b5f7dbe8e1e/markupsafe-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:26a5784ded40c9e318cfc2bdb30fe164bdb8665ded9cd64d500a34fb42067b1c", size = 15105, upload-time = "2025-09-27T18:36:39.701Z" }, + { url = "https://files.pythonhosted.org/packages/e5/f1/216fc1bbfd74011693a4fd837e7026152e89c4bcf3e77b6692fba9923123/markupsafe-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:35add3b638a5d900e807944a078b51922212fb3dedb01633a8defc4b01a3c85f", size = 13906, upload-time = "2025-09-27T18:36:40.689Z" }, + { url = "https://files.pythonhosted.org/packages/38/2f/907b9c7bbba283e68f20259574b13d005c121a0fa4c175f9bed27c4597ff/markupsafe-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e1cf1972137e83c5d4c136c43ced9ac51d0e124706ee1c8aa8532c1287fa8795", size = 11622, upload-time = "2025-09-27T18:36:41.777Z" }, + { url = "https://files.pythonhosted.org/packages/9c/d9/5f7756922cdd676869eca1c4e3c0cd0df60ed30199ffd775e319089cb3ed/markupsafe-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:116bb52f642a37c115f517494ea5feb03889e04df47eeff5b130b1808ce7c219", size = 12029, upload-time = "2025-09-27T18:36:43.257Z" }, + { url = "https://files.pythonhosted.org/packages/00/07/575a68c754943058c78f30db02ee03a64b3c638586fba6a6dd56830b30a3/markupsafe-3.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:133a43e73a802c5562be9bbcd03d090aa5a1fe899db609c29e8c8d815c5f6de6", size = 24374, upload-time = "2025-09-27T18:36:44.508Z" }, + { url = "https://files.pythonhosted.org/packages/a9/21/9b05698b46f218fc0e118e1f8168395c65c8a2c750ae2bab54fc4bd4e0e8/markupsafe-3.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ccfcd093f13f0f0b7fdd0f198b90053bf7b2f02a3927a30e63f3ccc9df56b676", size = 22980, upload-time = "2025-09-27T18:36:45.385Z" }, + { url = "https://files.pythonhosted.org/packages/7f/71/544260864f893f18b6827315b988c146b559391e6e7e8f7252839b1b846a/markupsafe-3.0.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:509fa21c6deb7a7a273d629cf5ec029bc209d1a51178615ddf718f5918992ab9", size = 21990, upload-time = "2025-09-27T18:36:46.916Z" }, + { url = "https://files.pythonhosted.org/packages/c2/28/b50fc2f74d1ad761af2f5dcce7492648b983d00a65b8c0e0cb457c82ebbe/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:a4afe79fb3de0b7097d81da19090f4df4f8d3a2b3adaa8764138aac2e44f3af1", size = 23784, upload-time = "2025-09-27T18:36:47.884Z" }, + { url = "https://files.pythonhosted.org/packages/ed/76/104b2aa106a208da8b17a2fb72e033a5a9d7073c68f7e508b94916ed47a9/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:795e7751525cae078558e679d646ae45574b47ed6e7771863fcc079a6171a0fc", size = 21588, upload-time = "2025-09-27T18:36:48.82Z" }, + { url = "https://files.pythonhosted.org/packages/b5/99/16a5eb2d140087ebd97180d95249b00a03aa87e29cc224056274f2e45fd6/markupsafe-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8485f406a96febb5140bfeca44a73e3ce5116b2501ac54fe953e488fb1d03b12", size = 23041, upload-time = "2025-09-27T18:36:49.797Z" }, + { url = "https://files.pythonhosted.org/packages/19/bc/e7140ed90c5d61d77cea142eed9f9c303f4c4806f60a1044c13e3f1471d0/markupsafe-3.0.3-cp313-cp313-win32.whl", hash = "sha256:bdd37121970bfd8be76c5fb069c7751683bdf373db1ed6c010162b2a130248ed", size = 14543, upload-time = "2025-09-27T18:36:51.584Z" }, + { url = "https://files.pythonhosted.org/packages/05/73/c4abe620b841b6b791f2edc248f556900667a5a1cf023a6646967ae98335/markupsafe-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:9a1abfdc021a164803f4d485104931fb8f8c1efd55bc6b748d2f5774e78b62c5", size = 15113, upload-time = "2025-09-27T18:36:52.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/3a/fa34a0f7cfef23cf9500d68cb7c32dd64ffd58a12b09225fb03dd37d5b80/markupsafe-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:7e68f88e5b8799aa49c85cd116c932a1ac15caaa3f5db09087854d218359e485", size = 13911, upload-time = "2025-09-27T18:36:53.513Z" }, + { url = "https://files.pythonhosted.org/packages/e4/d7/e05cd7efe43a88a17a37b3ae96e79a19e846f3f456fe79c57ca61356ef01/markupsafe-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:218551f6df4868a8d527e3062d0fb968682fe92054e89978594c28e642c43a73", size = 11658, upload-time = "2025-09-27T18:36:54.819Z" }, + { url = "https://files.pythonhosted.org/packages/99/9e/e412117548182ce2148bdeacdda3bb494260c0b0184360fe0d56389b523b/markupsafe-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:3524b778fe5cfb3452a09d31e7b5adefeea8c5be1d43c4f810ba09f2ceb29d37", size = 12066, upload-time = "2025-09-27T18:36:55.714Z" }, + { url = "https://files.pythonhosted.org/packages/bc/e6/fa0ffcda717ef64a5108eaa7b4f5ed28d56122c9a6d70ab8b72f9f715c80/markupsafe-3.0.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4e885a3d1efa2eadc93c894a21770e4bc67899e3543680313b09f139e149ab19", size = 25639, upload-time = "2025-09-27T18:36:56.908Z" }, + { url = "https://files.pythonhosted.org/packages/96/ec/2102e881fe9d25fc16cb4b25d5f5cde50970967ffa5dddafdb771237062d/markupsafe-3.0.3-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8709b08f4a89aa7586de0aadc8da56180242ee0ada3999749b183aa23df95025", size = 23569, upload-time = "2025-09-27T18:36:57.913Z" }, + { url = "https://files.pythonhosted.org/packages/4b/30/6f2fce1f1f205fc9323255b216ca8a235b15860c34b6798f810f05828e32/markupsafe-3.0.3-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b8512a91625c9b3da6f127803b166b629725e68af71f8184ae7e7d54686a56d6", size = 23284, upload-time = "2025-09-27T18:36:58.833Z" }, + { url = "https://files.pythonhosted.org/packages/58/47/4a0ccea4ab9f5dcb6f79c0236d954acb382202721e704223a8aafa38b5c8/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9b79b7a16f7fedff2495d684f2b59b0457c3b493778c9eed31111be64d58279f", size = 24801, upload-time = "2025-09-27T18:36:59.739Z" }, + { url = "https://files.pythonhosted.org/packages/6a/70/3780e9b72180b6fecb83a4814d84c3bf4b4ae4bf0b19c27196104149734c/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:12c63dfb4a98206f045aa9563db46507995f7ef6d83b2f68eda65c307c6829eb", size = 22769, upload-time = "2025-09-27T18:37:00.719Z" }, + { url = "https://files.pythonhosted.org/packages/98/c5/c03c7f4125180fc215220c035beac6b9cb684bc7a067c84fc69414d315f5/markupsafe-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8f71bc33915be5186016f675cd83a1e08523649b0e33efdb898db577ef5bb009", size = 23642, upload-time = "2025-09-27T18:37:01.673Z" }, + { url = "https://files.pythonhosted.org/packages/80/d6/2d1b89f6ca4bff1036499b1e29a1d02d282259f3681540e16563f27ebc23/markupsafe-3.0.3-cp313-cp313t-win32.whl", hash = "sha256:69c0b73548bc525c8cb9a251cddf1931d1db4d2258e9599c28c07ef3580ef354", size = 14612, upload-time = "2025-09-27T18:37:02.639Z" }, + { url = "https://files.pythonhosted.org/packages/2b/98/e48a4bfba0a0ffcf9925fe2d69240bfaa19c6f7507b8cd09c70684a53c1e/markupsafe-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:1b4b79e8ebf6b55351f0d91fe80f893b4743f104bff22e90697db1590e47a218", size = 15200, upload-time = "2025-09-27T18:37:03.582Z" }, + { url = "https://files.pythonhosted.org/packages/0e/72/e3cc540f351f316e9ed0f092757459afbc595824ca724cbc5a5d4263713f/markupsafe-3.0.3-cp313-cp313t-win_arm64.whl", hash = "sha256:ad2cf8aa28b8c020ab2fc8287b0f823d0a7d8630784c31e9ee5edea20f406287", size = 13973, upload-time = "2025-09-27T18:37:04.929Z" }, + { url = "https://files.pythonhosted.org/packages/33/8a/8e42d4838cd89b7dde187011e97fe6c3af66d8c044997d2183fbd6d31352/markupsafe-3.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaa9599de571d72e2daf60164784109f19978b327a3910d3e9de8c97b5b70cfe", size = 11619, upload-time = "2025-09-27T18:37:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/b5/64/7660f8a4a8e53c924d0fa05dc3a55c9cee10bbd82b11c5afb27d44b096ce/markupsafe-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:c47a551199eb8eb2121d4f0f15ae0f923d31350ab9280078d1e5f12b249e0026", size = 12029, upload-time = "2025-09-27T18:37:07.213Z" }, + { url = "https://files.pythonhosted.org/packages/da/ef/e648bfd021127bef5fa12e1720ffed0c6cbb8310c8d9bea7266337ff06de/markupsafe-3.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f34c41761022dd093b4b6896d4810782ffbabe30f2d443ff5f083e0cbbb8c737", size = 24408, upload-time = "2025-09-27T18:37:09.572Z" }, + { url = "https://files.pythonhosted.org/packages/41/3c/a36c2450754618e62008bf7435ccb0f88053e07592e6028a34776213d877/markupsafe-3.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:457a69a9577064c05a97c41f4e65148652db078a3a509039e64d3467b9e7ef97", size = 23005, upload-time = "2025-09-27T18:37:10.58Z" }, + { url = "https://files.pythonhosted.org/packages/bc/20/b7fdf89a8456b099837cd1dc21974632a02a999ec9bf7ca3e490aacd98e7/markupsafe-3.0.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e8afc3f2ccfa24215f8cb28dcf43f0113ac3c37c2f0f0806d8c70e4228c5cf4d", size = 22048, upload-time = "2025-09-27T18:37:11.547Z" }, + { url = "https://files.pythonhosted.org/packages/9a/a7/591f592afdc734f47db08a75793a55d7fbcc6902a723ae4cfbab61010cc5/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:ec15a59cf5af7be74194f7ab02d0f59a62bdcf1a537677ce67a2537c9b87fcda", size = 23821, upload-time = "2025-09-27T18:37:12.48Z" }, + { url = "https://files.pythonhosted.org/packages/7d/33/45b24e4f44195b26521bc6f1a82197118f74df348556594bd2262bda1038/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:0eb9ff8191e8498cca014656ae6b8d61f39da5f95b488805da4bb029cccbfbaf", size = 21606, upload-time = "2025-09-27T18:37:13.485Z" }, + { url = "https://files.pythonhosted.org/packages/ff/0e/53dfaca23a69fbfbbf17a4b64072090e70717344c52eaaaa9c5ddff1e5f0/markupsafe-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:2713baf880df847f2bece4230d4d094280f4e67b1e813eec43b4c0e144a34ffe", size = 23043, upload-time = "2025-09-27T18:37:14.408Z" }, + { url = "https://files.pythonhosted.org/packages/46/11/f333a06fc16236d5238bfe74daccbca41459dcd8d1fa952e8fbd5dccfb70/markupsafe-3.0.3-cp314-cp314-win32.whl", hash = "sha256:729586769a26dbceff69f7a7dbbf59ab6572b99d94576a5592625d5b411576b9", size = 14747, upload-time = "2025-09-27T18:37:15.36Z" }, + { url = "https://files.pythonhosted.org/packages/28/52/182836104b33b444e400b14f797212f720cbc9ed6ba34c800639d154e821/markupsafe-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:bdc919ead48f234740ad807933cdf545180bfbe9342c2bb451556db2ed958581", size = 15341, upload-time = "2025-09-27T18:37:16.496Z" }, + { url = "https://files.pythonhosted.org/packages/6f/18/acf23e91bd94fd7b3031558b1f013adfa21a8e407a3fdb32745538730382/markupsafe-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:5a7d5dc5140555cf21a6fefbdbf8723f06fcd2f63ef108f2854de715e4422cb4", size = 14073, upload-time = "2025-09-27T18:37:17.476Z" }, + { url = "https://files.pythonhosted.org/packages/3c/f0/57689aa4076e1b43b15fdfa646b04653969d50cf30c32a102762be2485da/markupsafe-3.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:1353ef0c1b138e1907ae78e2f6c63ff67501122006b0f9abad68fda5f4ffc6ab", size = 11661, upload-time = "2025-09-27T18:37:18.453Z" }, + { url = "https://files.pythonhosted.org/packages/89/c3/2e67a7ca217c6912985ec766c6393b636fb0c2344443ff9d91404dc4c79f/markupsafe-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1085e7fbddd3be5f89cc898938f42c0b3c711fdcb37d75221de2666af647c175", size = 12069, upload-time = "2025-09-27T18:37:19.332Z" }, + { url = "https://files.pythonhosted.org/packages/f0/00/be561dce4e6ca66b15276e184ce4b8aec61fe83662cce2f7d72bd3249d28/markupsafe-3.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1b52b4fb9df4eb9ae465f8d0c228a00624de2334f216f178a995ccdcf82c4634", size = 25670, upload-time = "2025-09-27T18:37:20.245Z" }, + { url = "https://files.pythonhosted.org/packages/50/09/c419f6f5a92e5fadde27efd190eca90f05e1261b10dbd8cbcb39cd8ea1dc/markupsafe-3.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fed51ac40f757d41b7c48425901843666a6677e3e8eb0abcff09e4ba6e664f50", size = 23598, upload-time = "2025-09-27T18:37:21.177Z" }, + { url = "https://files.pythonhosted.org/packages/22/44/a0681611106e0b2921b3033fc19bc53323e0b50bc70cffdd19f7d679bb66/markupsafe-3.0.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f190daf01f13c72eac4efd5c430a8de82489d9cff23c364c3ea822545032993e", size = 23261, upload-time = "2025-09-27T18:37:22.167Z" }, + { url = "https://files.pythonhosted.org/packages/5f/57/1b0b3f100259dc9fffe780cfb60d4be71375510e435efec3d116b6436d43/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e56b7d45a839a697b5eb268c82a71bd8c7f6c94d6fd50c3d577fa39a9f1409f5", size = 24835, upload-time = "2025-09-27T18:37:23.296Z" }, + { url = "https://files.pythonhosted.org/packages/26/6a/4bf6d0c97c4920f1597cc14dd720705eca0bf7c787aebc6bb4d1bead5388/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f3e98bb3798ead92273dc0e5fd0f31ade220f59a266ffd8a4f6065e0a3ce0523", size = 22733, upload-time = "2025-09-27T18:37:24.237Z" }, + { url = "https://files.pythonhosted.org/packages/14/c7/ca723101509b518797fedc2fdf79ba57f886b4aca8a7d31857ba3ee8281f/markupsafe-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5678211cb9333a6468fb8d8be0305520aa073f50d17f089b5b4b477ea6e67fdc", size = 23672, upload-time = "2025-09-27T18:37:25.271Z" }, + { url = "https://files.pythonhosted.org/packages/fb/df/5bd7a48c256faecd1d36edc13133e51397e41b73bb77e1a69deab746ebac/markupsafe-3.0.3-cp314-cp314t-win32.whl", hash = "sha256:915c04ba3851909ce68ccc2b8e2cd691618c4dc4c4232fb7982bca3f41fd8c3d", size = 14819, upload-time = "2025-09-27T18:37:26.285Z" }, + { url = "https://files.pythonhosted.org/packages/1a/8a/0402ba61a2f16038b48b39bccca271134be00c5c9f0f623208399333c448/markupsafe-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4faffd047e07c38848ce017e8725090413cd80cbc23d86e55c587bf979e579c9", size = 15426, upload-time = "2025-09-27T18:37:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/70/bc/6f1c2f612465f5fa89b95bead1f44dcb607670fd42891d8fdcd5d039f4f4/markupsafe-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:32001d6a8fc98c8cb5c947787c5d08b0a50663d139f1305bac5885d98d9b40fa", size = 14146, upload-time = "2025-09-27T18:37:28.327Z" }, +] + +[[package]] +name = "matplotlib" +version = "3.10.9" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "contourpy" }, + { name = "cycler" }, + { name = "fonttools" }, + { name = "kiwisolver" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pillow" }, + { name = "pyparsing" }, + { name = "python-dateutil" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/63/1b/4be5be87d43d327a0cf4de1a56e86f7f84c89312452406cf122efe2839e6/matplotlib-3.10.9.tar.gz", hash = "sha256:fd66508e8c6877d98e586654b608a0456db8d7e8a546eb1e2600efd957302358", size = 34811233, upload-time = "2026-04-24T00:14:13.539Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/35/c6/5581e26c72233ebb2a2a6fed2d24fb7c66b4700120b813f51b0555acf0b6/matplotlib-3.10.9-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f0c3c28d9fbcc1fe7a03be236d73430cf6409c41fb2383a7ac52fe932b072cb1", size = 8319908, upload-time = "2026-04-24T00:12:21.323Z" }, + { url = "https://files.pythonhosted.org/packages/b7/18/4880dd762e40cd360c1bf06e890c5a97b997e91cb324602b1a19950ad5ce/matplotlib-3.10.9-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:41cb28c2bd769aa3e98322c6ab09854cbcc52ab69d2759d681bba3e327b2b320", size = 8216016, upload-time = "2026-04-24T00:12:23.4Z" }, + { url = "https://files.pythonhosted.org/packages/32/91/d024616abdba99e83120e07a20658976f6a343646710760c4a51df126029/matplotlib-3.10.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ae20801130378b82d647ff5047c07316295b68dc054ca6b3c13519d0ea624285", size = 8789336, upload-time = "2026-04-24T00:12:26.096Z" }, + { url = "https://files.pythonhosted.org/packages/5c/04/030a2f61ef2158f5e4c259487a92ac877732499fb33d871585d89e03c42d/matplotlib-3.10.9-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6c63ebcd8b4b169eb2f5c200552ae6b8be8999a005b6b507ed76fb8d7d674fe2", size = 9604602, upload-time = "2026-04-24T00:12:29.052Z" }, + { url = "https://files.pythonhosted.org/packages/fc/c2/541e4d09d87bb6b5830fc28b4c887a9a8cf4e1c6cee698a8c05552ae2003/matplotlib-3.10.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:d75d11c949914165976c621b2324f9ef162af7ebf4b057ddf95dd1dba7e5edcf", size = 9670966, upload-time = "2026-04-24T00:12:32.131Z" }, + { url = "https://files.pythonhosted.org/packages/04/a1/4571fc46e7702de8d0c2dc54ad1b2f8e29328dea3ee90831181f7353d93c/matplotlib-3.10.9-cp312-cp312-win_amd64.whl", hash = "sha256:d091f9d758b34aaaaa6331d13574bf01891d903b3dec59bfff458ef7551de5d6", size = 8217462, upload-time = "2026-04-24T00:12:35.226Z" }, + { url = "https://files.pythonhosted.org/packages/4b/d0/2269edb12aa30c13c8bcc9382892e39943ce1d28aab4ec296e0381798e81/matplotlib-3.10.9-cp312-cp312-win_arm64.whl", hash = "sha256:10cc5ce06d10231c36f40e875f3c7e8050362a4ee8f0ee5d29a6b3277d57bb42", size = 8136688, upload-time = "2026-04-24T00:12:37.442Z" }, + { url = "https://files.pythonhosted.org/packages/aa/d3/8d4f6afbecb49fc04e060a57c0fce39ea51cc163a6bd87303ccd698e4fa6/matplotlib-3.10.9-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:b580440f1ff81a0e34122051a3dfabb7e4b7f9e380629929bde0eff9af72165f", size = 8320331, upload-time = "2026-04-24T00:12:39.688Z" }, + { url = "https://files.pythonhosted.org/packages/63/d9/9e14bc7564bf92d5ffa801ae5fac819ce74b925dfb55e3ebde61a3bbad3e/matplotlib-3.10.9-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:b1b745c489cd1a77a0dc1120a05dc87af9798faebc913601feb8c73d89bf2d1e", size = 8216461, upload-time = "2026-04-24T00:12:42.494Z" }, + { url = "https://files.pythonhosted.org/packages/8a/17/4402d0d14ccf1dfc70932600b68097fbbf9c898a4871d2cbbe79c7801a32/matplotlib-3.10.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8f3bcac1ca5ed000a6f4337d47ba67dfddf37ed6a46c15fd7f014997f7bf865f", size = 8790091, upload-time = "2026-04-24T00:12:44.789Z" }, + { url = "https://files.pythonhosted.org/packages/3e/0b/322aeec06dd9b91411f92028b37d447342770a24392aa4813e317064dad5/matplotlib-3.10.9-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a8d66a55def891c33147ba3ba9bfcabf0b526a43764c818acbb4525e5ed0838", size = 9605027, upload-time = "2026-04-24T00:12:47.583Z" }, + { url = "https://files.pythonhosted.org/packages/74/88/5f13482f55e7b00bcfc09838b093c2456e1379978d2a146844aae05350ad/matplotlib-3.10.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:d843374407c4017a6403b59c6c81606773d136f3259d5b6da3131bc814542cc2", size = 9671269, upload-time = "2026-04-24T00:12:50.878Z" }, + { url = "https://files.pythonhosted.org/packages/c5/e0/0840fd2f93da988ec660b8ad1984abe9f25d2aed22a5e394ff1c68c88307/matplotlib-3.10.9-cp313-cp313-win_amd64.whl", hash = "sha256:f4399f64b3e94cd500195490972ae1ee81170df1636fa15364d157d5bdd7b921", size = 8217588, upload-time = "2026-04-24T00:12:53.784Z" }, + { url = "https://files.pythonhosted.org/packages/47/b9/d706d06dd605c49b9f83a2aed8c13e3e5db70697d7a80b7e3d7915de6b17/matplotlib-3.10.9-cp313-cp313-win_arm64.whl", hash = "sha256:ba7b3b8ef09eab7df0e86e9ae086faa433efbfbdb46afcb3aa16aabf779469a8", size = 8136913, upload-time = "2026-04-24T00:12:56.501Z" }, + { url = "https://files.pythonhosted.org/packages/9b/45/6e32d96978264c8ca8c4b1010adb955a1a49cfaf314e212bbc8908f04a61/matplotlib-3.10.9-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:09218df8a93712bd6ea133e83a153c755448cf7868316c531cffcc43f69d1cc9", size = 8368019, upload-time = "2026-04-24T00:12:58.896Z" }, + { url = "https://files.pythonhosted.org/packages/86/0a/c8e3d3bba245f0f7fc424937f8ff7ef77291a36af3edb97ccd78aa93d84f/matplotlib-3.10.9-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:82368699727bfb7b0182e1aa13082e3c08e092fa1a25d3e1fd92405bff96f6d4", size = 8264645, upload-time = "2026-04-24T00:13:01.406Z" }, + { url = "https://files.pythonhosted.org/packages/3d/aa/5bf5a14fe4fed73a4209a155606f8096ff797aad89c6c35179026571133e/matplotlib-3.10.9-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3225f4e1edcb8c86c884ddf79ebe20ecd0a67d30188f279897554ccd8fded4dc", size = 8802194, upload-time = "2026-04-24T00:13:03.702Z" }, + { url = "https://files.pythonhosted.org/packages/dd/5e/b4be852d6bba6fd15893fadf91ff26ae49cb91aac789e95dde9d342e664f/matplotlib-3.10.9-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de2445a0c6690d21b7eb6ce071cebad6d40a2e9bdf10d039074a96ba19797b99", size = 9622684, upload-time = "2026-04-24T00:13:06.647Z" }, + { url = "https://files.pythonhosted.org/packages/4c/3d/ed428c971139112ef730f62770654d609467346d09d4b62617e1afd68a5a/matplotlib-3.10.9-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:b2b9516251cb89ff618d757daec0e2ed1bf21248013844a853d87ef85ab3081d", size = 9680790, upload-time = "2026-04-24T00:13:10.009Z" }, + { url = "https://files.pythonhosted.org/packages/e7/09/052e884aaf2b985c63cb79f715f1d5b6a3eaa7de78f6a52b9dbc077d5b53/matplotlib-3.10.9-cp313-cp313t-win_amd64.whl", hash = "sha256:e9fae004b941b23ff2edcf1567a857ed77bafc8086ffa258190462328434faf8", size = 8287571, upload-time = "2026-04-24T00:13:13.087Z" }, + { url = "https://files.pythonhosted.org/packages/f4/38/ae27288e788c35a4250491422f3db7750366fc8c97d6f36fbdecfc1f5518/matplotlib-3.10.9-cp313-cp313t-win_arm64.whl", hash = "sha256:6b63d9c7c769b88ab81e10dc86e4e0607cf56817b9f9e6cf24b2a5f1693b8e38", size = 8188292, upload-time = "2026-04-24T00:13:15.546Z" }, + { url = "https://files.pythonhosted.org/packages/d6/e6/3bd8afd04949f02eabc1c17115ea5255e19cacd4d06fc5abdde4eeb0052c/matplotlib-3.10.9-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:172db52c9e683f5d12eaf57f0f54834190e12581fe1cc2a19595a8f5acb4e77d", size = 8321276, upload-time = "2026-04-24T00:13:18.318Z" }, + { url = "https://files.pythonhosted.org/packages/41/86/86231232fff41c9f8e4a1a7d7a597d349a02527109c3af7d618366122139/matplotlib-3.10.9-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:97e35e8d39ccc85859095e01a53847432ba9a53ddf7986f7a54a11b73d0e143f", size = 8218218, upload-time = "2026-04-24T00:13:20.974Z" }, + { url = "https://files.pythonhosted.org/packages/85/8f/becc9722cafc64f5d2eb0b7c1bf5f585271c618a45dbd8fabeb021f898b6/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aba1615dabe83188e19d4f75a253c6a08423e04c1425e64039f800050a69de6b", size = 9608145, upload-time = "2026-04-24T00:13:23.228Z" }, + { url = "https://files.pythonhosted.org/packages/32/5d/f7e914f7d9325abff4057cee62c0fa70263683189f774473cbfb534cd13b/matplotlib-3.10.9-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:34cf8167e023ad956c15f36302911d5406bd99a9862c1a8499ea6f7c0e015dc2", size = 9885085, upload-time = "2026-04-24T00:13:25.849Z" }, + { url = "https://files.pythonhosted.org/packages/a5/fd/fa69f2221534e80cc5772ac2b7d222011a2acafc2ec7216d5dd174c864ae/matplotlib-3.10.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:59476c6d29d612b8e9bb6ce8c5b631be6ba8f9e3a2421f22a02b192c7dd28716", size = 9672358, upload-time = "2026-04-24T00:13:28.906Z" }, + { url = "https://files.pythonhosted.org/packages/ab/1a/5a4f747a8b271cbb024946d2dd3c913ab5032ba430626f8c3528ada96b4b/matplotlib-3.10.9-cp314-cp314-win_amd64.whl", hash = "sha256:336b9acc64d309063126edcdaca00db9373af3c476bb94388fe9c5a53ad13e6f", size = 8349970, upload-time = "2026-04-24T00:13:31.904Z" }, + { url = "https://files.pythonhosted.org/packages/64/dc/95d60ecaefe30680a154b52ea96ab4b0dab547f1fd6aa12f5fb655e89cae/matplotlib-3.10.9-cp314-cp314-win_arm64.whl", hash = "sha256:2dc9477819ffd78ad12a20df1d9d6a6bd4fec6aaa9072681465fddca052f1456", size = 8272785, upload-time = "2026-04-24T00:13:34.511Z" }, + { url = "https://files.pythonhosted.org/packages/70/a0/005d68bc8b8418300ce6591f18586910a8526806e2ab663933d9f20a41e9/matplotlib-3.10.9-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:da4e09638420548f31c354032a6250e473c68e5a4e96899b4844cf39ddea23fe", size = 8367999, upload-time = "2026-04-24T00:13:36.962Z" }, + { url = "https://files.pythonhosted.org/packages/22/05/1236cc9290be70b2498af20ca348add76e3fffe7f67b477db5133a84f3ea/matplotlib-3.10.9-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:345f6f68ecc8da0ca56fad2ea08fde1a115eda530079eca185d50a7bc3e146c6", size = 8264543, upload-time = "2026-04-24T00:13:39.851Z" }, + { url = "https://files.pythonhosted.org/packages/cd/c2/071f5a5ff6c5bd63aaaf2f45c811d9bf2ced94bde188d9e1a519e21d0cba/matplotlib-3.10.9-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4edcfbd8565339aa62f1cd4012f7180926fdbe71850f7b0d3c379c175cd6b66c", size = 9622800, upload-time = "2026-04-24T00:13:42.296Z" }, + { url = "https://files.pythonhosted.org/packages/95/57/da7d1f10a85624b9e7db68e069dd94e58dc41dbf9463c5921632ecbe3661/matplotlib-3.10.9-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6be157fe17fc37cb95ac1d7374cf717ce9259616edec911a78d9d26dae8522d4", size = 9888561, upload-time = "2026-04-24T00:13:45.026Z" }, + { url = "https://files.pythonhosted.org/packages/67/b2/ef8d6bb59b0edb6c16c968b70f548aa13b54348972def5aa6ac85df67145/matplotlib-3.10.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:4e42042d54db34fda4e95a7bd3e5789c2a995d2dad3eb8850232ee534092fbbf", size = 9680884, upload-time = "2026-04-24T00:13:48.066Z" }, + { url = "https://files.pythonhosted.org/packages/61/1c/d21bfeb9931881ebe96bcfcff27c7ae4b160ae0ec291a714c42641a56d75/matplotlib-3.10.9-cp314-cp314t-win_amd64.whl", hash = "sha256:c27df8b3848f32a83d1767566595e43cfaa4460380974da06f4279a7ec143c39", size = 8432333, upload-time = "2026-04-24T00:13:51.008Z" }, + { url = "https://files.pythonhosted.org/packages/78/23/92493c3e6e1b635ccfff146f7b99e674808787915420373ac399283764c2/matplotlib-3.10.9-cp314-cp314t-win_arm64.whl", hash = "sha256:a49f1eadc84ca85fd72fa4e89e70e61bf86452df6f971af04b12c60761a0772c", size = 8324785, upload-time = "2026-04-24T00:13:53.633Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "mergedeep" +version = "1.3.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/3a/41/580bb4006e3ed0361b8151a01d324fb03f420815446c7def45d02f74c270/mergedeep-1.3.4.tar.gz", hash = "sha256:0096d52e9dad9939c3d975a774666af186eda617e6ca84df4c94dec30004f2a8", size = 4661, upload-time = "2021-02-05T18:55:30.623Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307", size = 6354, upload-time = "2021-02-05T18:55:29.583Z" }, +] + +[[package]] +name = "mkdocs" +version = "1.6.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "ghp-import" }, + { name = "jinja2" }, + { name = "markdown" }, + { name = "markupsafe" }, + { name = "mergedeep" }, + { name = "mkdocs-get-deps" }, + { name = "packaging" }, + { name = "pathspec" }, + { name = "pyyaml" }, + { name = "pyyaml-env-tag" }, + { name = "watchdog" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/bc/c6/bbd4f061bd16b378247f12953ffcb04786a618ce5e904b8c5a01a0309061/mkdocs-1.6.1.tar.gz", hash = "sha256:7b432f01d928c084353ab39c57282f29f92136665bdd6abf7c1ec8d822ef86f2", size = 3889159, upload-time = "2024-08-30T12:24:06.899Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/22/5b/dbc6a8cddc9cfa9c4971d59fb12bb8d42e161b7e7f8cc89e49137c5b279c/mkdocs-1.6.1-py3-none-any.whl", hash = "sha256:db91759624d1647f3f34aa0c3f327dd2601beae39a366d6e064c03468d35c20e", size = 3864451, upload-time = "2024-08-30T12:24:05.054Z" }, +] + +[[package]] +name = "mkdocs-autorefs" +version = "1.4.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown" }, + { name = "markupsafe" }, + { name = "mkdocs" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/52/c0/f641843de3f612a6b48253f39244165acff36657a91cc903633d456ae1ac/mkdocs_autorefs-1.4.4.tar.gz", hash = "sha256:d54a284f27a7346b9c38f1f852177940c222da508e66edc816a0fa55fc6da197", size = 56588, upload-time = "2026-02-10T15:23:55.105Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/de/a3e710469772c6a89595fc52816da05c1e164b4c866a89e3cb82fb1b67c5/mkdocs_autorefs-1.4.4-py3-none-any.whl", hash = "sha256:834ef5408d827071ad1bc69e0f39704fa34c7fc05bc8e1c72b227dfdc5c76089", size = 25530, upload-time = "2026-02-10T15:23:53.817Z" }, +] + +[[package]] +name = "mkdocs-get-deps" +version = "0.2.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mergedeep" }, + { name = "platformdirs" }, + { name = "pyyaml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ce/25/b3cccb187655b9393572bde9b09261d267c3bf2f2cdabe347673be5976a6/mkdocs_get_deps-0.2.2.tar.gz", hash = "sha256:8ee8d5f316cdbbb2834bc1df6e69c08fe769a83e040060de26d3c19fad3599a1", size = 11047, upload-time = "2026-03-10T02:46:33.632Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/29/744136411e785c4b0b744d5413e56555265939ab3a104c6a4b719dad33fd/mkdocs_get_deps-0.2.2-py3-none-any.whl", hash = "sha256:e7878cbeac04860b8b5e0ca31d3abad3df9411a75a32cde82f8e44b6c16ff650", size = 9555, upload-time = "2026-03-10T02:46:32.256Z" }, +] + +[[package]] +name = "mkdocs-material" +version = "9.7.6" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "babel" }, + { name = "backrefs" }, + { name = "colorama" }, + { name = "jinja2" }, + { name = "markdown" }, + { name = "mkdocs" }, + { name = "mkdocs-material-extensions" }, + { name = "paginate" }, + { name = "pygments" }, + { name = "pymdown-extensions" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/45/29/6d2bcf41ae40802c4beda2432396fff97b8456fb496371d1bc7aad6512ec/mkdocs_material-9.7.6.tar.gz", hash = "sha256:00bdde50574f776d328b1862fe65daeaf581ec309bd150f7bff345a098c64a69", size = 4097959, upload-time = "2026-03-19T15:41:58.161Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/01/bc663630c510822c95c47a66af9fa7a443c295b47d5f041e5e6ae62ef659/mkdocs_material-9.7.6-py3-none-any.whl", hash = "sha256:71b84353921b8ea1ba84fe11c50912cc512da8fe0881038fcc9a0761c0e635ba", size = 9305470, upload-time = "2026-03-19T15:41:55.217Z" }, +] + +[[package]] +name = "mkdocs-material-extensions" +version = "1.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/79/9b/9b4c96d6593b2a541e1cb8b34899a6d021d208bb357042823d4d2cabdbe7/mkdocs_material_extensions-1.3.1.tar.gz", hash = "sha256:10c9511cea88f568257f960358a467d12b970e1f7b2c0e5fb2bb48cab1928443", size = 11847, upload-time = "2023-11-22T19:09:45.208Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5b/54/662a4743aa81d9582ee9339d4ffa3c8fd40a4965e033d77b9da9774d3960/mkdocs_material_extensions-1.3.1-py3-none-any.whl", hash = "sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31", size = 8728, upload-time = "2023-11-22T19:09:43.465Z" }, +] + +[[package]] +name = "mkdocstrings" +version = "1.0.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "jinja2" }, + { name = "markdown" }, + { name = "markupsafe" }, + { name = "mkdocs" }, + { name = "mkdocs-autorefs" }, + { name = "pymdown-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1d/5d/f888d4d3eb31359b327bc9b17a212d6ef03fe0b0682fbb3fc2cb849fb12b/mkdocstrings-1.0.4.tar.gz", hash = "sha256:3969a6515b77db65fd097b53c1b7aa4ae840bd71a2ee62a6a3e89503446d7172", size = 100088, upload-time = "2026-04-15T09:16:53.376Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6e/94/be70f8ee9c45f2f62b39a1f0e9303bc20e138a8f3b8e50ffd89498e177e1/mkdocstrings-1.0.4-py3-none-any.whl", hash = "sha256:63464b4b29053514f32a1dbbf604e52876d5e638111b0c295ab7ed3cac73ca9b", size = 35560, upload-time = "2026-04-15T09:16:51.436Z" }, +] + +[package.optional-dependencies] +python = [ + { name = "mkdocstrings-python" }, +] + +[[package]] +name = "mkdocstrings-python" +version = "2.0.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "griffelib" }, + { name = "mkdocs-autorefs" }, + { name = "mkdocstrings" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a4/b4/5fed370d8ebd96e4e399460a7146ae989263f16588b05a6facd6dbd51e60/mkdocstrings_python-2.0.4.tar.gz", hash = "sha256:58c73c5d358e64e9b1673447663f4a2f8a8941e392e225fc0a0c893758cc452f", size = 199219, upload-time = "2026-06-05T08:13:01.819Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5e/e3/00ec594aef5f55522e6d373bc2ac53e53a8f5e9ae32f2d6854b0de4270f3/mkdocstrings_python-2.0.4-py3-none-any.whl", hash = "sha256:fd87c173e1e719a85997b6d4f852cdc55f36710e0ed08da3a7bd9abe79c9db00", size = 104790, upload-time = "2026-06-05T08:13:00.393Z" }, +] + +[[package]] +name = "mpmath" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106, upload-time = "2023-03-07T16:47:11.061Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" }, +] + +[[package]] +name = "mypy-extensions" +version = "1.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/a2/6e/371856a3fb9d31ca8dac321cda606860fa4548858c0cc45d9d1d4ca2628b/mypy_extensions-1.1.0.tar.gz", hash = "sha256:52e68efc3284861e772bbcd66823fde5ae21fd2fdb51c62a211403730b916558", size = 6343, upload-time = "2025-04-22T14:54:24.164Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, +] + +[[package]] +name = "narwhals" +version = "2.21.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/2d/0e/3ad61eb87088cc4932e0d851531fa82f845a6230b68b091a0e298cc7e537/narwhals-2.21.0.tar.gz", hash = "sha256:7c6e7f50528e62b7a967dd864d7e117d2955d38d4f730653ce46a9861358e2dc", size = 633083, upload-time = "2026-05-08T12:29:02.587Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c7/e1/68c2256b69a314eba133673377ba9118c356f6342a0c02b61de449cf2bf2/narwhals-2.21.0-py3-none-any.whl", hash = "sha256:1e6617d0fca68ae1fda29e5397c4eaacd3ffc9fffe6bcd6ded0c690475e853be", size = 451943, upload-time = "2026-05-08T12:29:01.058Z" }, +] + +[[package]] +name = "networkx" +version = "3.6.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509", size = 2517025, upload-time = "2025-12-08T17:02:39.908Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, +] + +[[package]] +name = "numpy" +version = "2.4.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/9f/b8cef5bffa569759033adda9481211426f12f53299629b410340795c2514/numpy-2.4.4.tar.gz", hash = "sha256:2d390634c5182175533585cc89f3608a4682ccb173cc9bb940b2881c8d6f8fa0", size = 20731587, upload-time = "2026-03-29T13:22:01.298Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/28/05/32396bec30fb2263770ee910142f49c1476d08e8ad41abf8403806b520ce/numpy-2.4.4-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:15716cfef24d3a9762e3acdf87e27f58dc823d1348f765bbea6bef8c639bfa1b", size = 16689272, upload-time = "2026-03-29T13:18:49.223Z" }, + { url = "https://files.pythonhosted.org/packages/c5/f3/a983d28637bfcd763a9c7aafdb6d5c0ebf3d487d1e1459ffdb57e2f01117/numpy-2.4.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:23cbfd4c17357c81021f21540da84ee282b9c8fba38a03b7b9d09ba6b951421e", size = 14699573, upload-time = "2026-03-29T13:18:52.629Z" }, + { url = "https://files.pythonhosted.org/packages/9b/fd/e5ecca1e78c05106d98028114f5c00d3eddb41207686b2b7de3e477b0e22/numpy-2.4.4-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:8b3b60bb7cba2c8c81837661c488637eee696f59a877788a396d33150c35d842", size = 5204782, upload-time = "2026-03-29T13:18:55.579Z" }, + { url = "https://files.pythonhosted.org/packages/de/2f/702a4594413c1a8632092beae8aba00f1d67947389369b3777aed783fdca/numpy-2.4.4-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:e4a010c27ff6f210ff4c6ef34394cd61470d01014439b192ec22552ee867f2a8", size = 6552038, upload-time = "2026-03-29T13:18:57.769Z" }, + { url = "https://files.pythonhosted.org/packages/7f/37/eed308a8f56cba4d1fdf467a4fc67ef4ff4bf1c888f5fc980481890104b1/numpy-2.4.4-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f9e75681b59ddaa5e659898085ae0eaea229d054f2ac0c7e563a62205a700121", size = 15670666, upload-time = "2026-03-29T13:19:00.341Z" }, + { url = "https://files.pythonhosted.org/packages/0a/0d/0e3ecece05b7a7e87ab9fb587855548da437a061326fff64a223b6dcb78a/numpy-2.4.4-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:81f4a14bee47aec54f883e0cad2d73986640c1590eb9bfaaba7ad17394481e6e", size = 16645480, upload-time = "2026-03-29T13:19:03.63Z" }, + { url = "https://files.pythonhosted.org/packages/34/49/f2312c154b82a286758ee2f1743336d50651f8b5195db18cdb63675ff649/numpy-2.4.4-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:62d6b0f03b694173f9fcb1fb317f7222fd0b0b103e784c6549f5e53a27718c44", size = 17020036, upload-time = "2026-03-29T13:19:07.428Z" }, + { url = "https://files.pythonhosted.org/packages/7b/e9/736d17bd77f1b0ec4f9901aaec129c00d59f5d84d5e79bba540ef12c2330/numpy-2.4.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fbc356aae7adf9e6336d336b9c8111d390a05df88f1805573ebb0807bd06fd1d", size = 18368643, upload-time = "2026-03-29T13:19:10.775Z" }, + { url = "https://files.pythonhosted.org/packages/63/f6/d417977c5f519b17c8a5c3bc9e8304b0908b0e21136fe43bf628a1343914/numpy-2.4.4-cp312-cp312-win32.whl", hash = "sha256:0d35aea54ad1d420c812bfa0385c71cd7cc5bcf7c65fed95fc2cd02fe8c79827", size = 5961117, upload-time = "2026-03-29T13:19:13.464Z" }, + { url = "https://files.pythonhosted.org/packages/2d/5b/e1deebf88ff431b01b7406ca3583ab2bbb90972bbe1c568732e49c844f7e/numpy-2.4.4-cp312-cp312-win_amd64.whl", hash = "sha256:b5f0362dc928a6ecd9db58868fca5e48485205e3855957bdedea308f8672ea4a", size = 12320584, upload-time = "2026-03-29T13:19:16.155Z" }, + { url = "https://files.pythonhosted.org/packages/58/89/e4e856ac82a68c3ed64486a544977d0e7bdd18b8da75b78a577ca31c4395/numpy-2.4.4-cp312-cp312-win_arm64.whl", hash = "sha256:846300f379b5b12cc769334464656bc882e0735d27d9726568bc932fdc49d5ec", size = 10221450, upload-time = "2026-03-29T13:19:18.994Z" }, + { url = "https://files.pythonhosted.org/packages/14/1d/d0a583ce4fefcc3308806a749a536c201ed6b5ad6e1322e227ee4848979d/numpy-2.4.4-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:08f2e31ed5e6f04b118e49821397f12767934cfdd12a1ce86a058f91e004ee50", size = 16684933, upload-time = "2026-03-29T13:19:22.47Z" }, + { url = "https://files.pythonhosted.org/packages/c1/62/2b7a48fbb745d344742c0277f01286dead15f3f68e4f359fbfcf7b48f70f/numpy-2.4.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e823b8b6edc81e747526f70f71a9c0a07ac4e7ad13020aa736bb7c9d67196115", size = 14694532, upload-time = "2026-03-29T13:19:25.581Z" }, + { url = "https://files.pythonhosted.org/packages/e5/87/499737bfba066b4a3bebff24a8f1c5b2dee410b209bc6668c9be692580f0/numpy-2.4.4-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:4a19d9dba1a76618dd86b164d608566f393f8ec6ac7c44f0cc879011c45e65af", size = 5199661, upload-time = "2026-03-29T13:19:28.31Z" }, + { url = "https://files.pythonhosted.org/packages/cd/da/464d551604320d1491bc345efed99b4b7034143a85787aab78d5691d5a0e/numpy-2.4.4-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:d2a8490669bfe99a233298348acc2d824d496dee0e66e31b66a6022c2ad74a5c", size = 6547539, upload-time = "2026-03-29T13:19:30.97Z" }, + { url = "https://files.pythonhosted.org/packages/7d/90/8d23e3b0dafd024bf31bdec225b3bb5c2dbfa6912f8a53b8659f21216cbf/numpy-2.4.4-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:45dbed2ab436a9e826e302fcdcbe9133f9b0006e5af7168afb8963a6520da103", size = 15668806, upload-time = "2026-03-29T13:19:33.887Z" }, + { url = "https://files.pythonhosted.org/packages/d1/73/a9d864e42a01896bb5974475438f16086be9ba1f0d19d0bb7a07427c4a8b/numpy-2.4.4-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c901b15172510173f5cb310eae652908340f8dede90fff9e3bf6c0d8dfd92f83", size = 16632682, upload-time = "2026-03-29T13:19:37.336Z" }, + { url = "https://files.pythonhosted.org/packages/34/fb/14570d65c3bde4e202a031210475ae9cde9b7686a2e7dc97ee67d2833b35/numpy-2.4.4-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:99d838547ace2c4aace6c4f76e879ddfe02bb58a80c1549928477862b7a6d6ed", size = 17019810, upload-time = "2026-03-29T13:19:40.963Z" }, + { url = "https://files.pythonhosted.org/packages/8a/77/2ba9d87081fd41f6d640c83f26fb7351e536b7ce6dd9061b6af5904e8e46/numpy-2.4.4-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0aec54fd785890ecca25a6003fd9a5aed47ad607bbac5cd64f836ad8666f4959", size = 18357394, upload-time = "2026-03-29T13:19:44.859Z" }, + { url = "https://files.pythonhosted.org/packages/a2/23/52666c9a41708b0853fa3b1a12c90da38c507a3074883823126d4e9d5b30/numpy-2.4.4-cp313-cp313-win32.whl", hash = "sha256:07077278157d02f65c43b1b26a3886bce886f95d20aabd11f87932750dfb14ed", size = 5959556, upload-time = "2026-03-29T13:19:47.661Z" }, + { url = "https://files.pythonhosted.org/packages/57/fb/48649b4971cde70d817cf97a2a2fdc0b4d8308569f1dd2f2611959d2e0cf/numpy-2.4.4-cp313-cp313-win_amd64.whl", hash = "sha256:5c70f1cc1c4efbe316a572e2d8b9b9cc44e89b95f79ca3331553fbb63716e2bf", size = 12317311, upload-time = "2026-03-29T13:19:50.67Z" }, + { url = "https://files.pythonhosted.org/packages/ba/d8/11490cddd564eb4de97b4579ef6bfe6a736cc07e94c1598590ae25415e01/numpy-2.4.4-cp313-cp313-win_arm64.whl", hash = "sha256:ef4059d6e5152fa1a39f888e344c73fdc926e1b2dd58c771d67b0acfbf2aa67d", size = 10222060, upload-time = "2026-03-29T13:19:54.229Z" }, + { url = "https://files.pythonhosted.org/packages/99/5d/dab4339177a905aad3e2221c915b35202f1ec30d750dd2e5e9d9a72b804b/numpy-2.4.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4bbc7f303d125971f60ec0aaad5e12c62d0d2c925f0ab1273debd0e4ba37aba5", size = 14822302, upload-time = "2026-03-29T13:19:57.585Z" }, + { url = "https://files.pythonhosted.org/packages/eb/e4/0564a65e7d3d97562ed6f9b0fd0fb0a6f559ee444092f105938b50043876/numpy-2.4.4-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:4d6d57903571f86180eb98f8f0c839fa9ebbfb031356d87f1361be91e433f5b7", size = 5327407, upload-time = "2026-03-29T13:20:00.601Z" }, + { url = "https://files.pythonhosted.org/packages/29/8d/35a3a6ce5ad371afa58b4700f1c820f8f279948cca32524e0a695b0ded83/numpy-2.4.4-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:4636de7fd195197b7535f231b5de9e4b36d2c440b6e566d2e4e4746e6af0ca93", size = 6647631, upload-time = "2026-03-29T13:20:02.855Z" }, + { url = "https://files.pythonhosted.org/packages/f4/da/477731acbd5a58a946c736edfdabb2ac5b34c3d08d1ba1a7b437fa0884df/numpy-2.4.4-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ad2e2ef14e0b04e544ea2fa0a36463f847f113d314aa02e5b402fdf910ef309e", size = 15727691, upload-time = "2026-03-29T13:20:06.004Z" }, + { url = "https://files.pythonhosted.org/packages/e6/db/338535d9b152beabeb511579598418ba0212ce77cf9718edd70262cc4370/numpy-2.4.4-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5a285b3b96f951841799528cd1f4f01cd70e7e0204b4abebac9463eecfcf2a40", size = 16681241, upload-time = "2026-03-29T13:20:09.417Z" }, + { url = "https://files.pythonhosted.org/packages/e2/a9/ad248e8f58beb7a0219b413c9c7d8151c5d285f7f946c3e26695bdbbe2df/numpy-2.4.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f8474c4241bc18b750be2abea9d7a9ec84f46ef861dbacf86a4f6e043401f79e", size = 17085767, upload-time = "2026-03-29T13:20:13.126Z" }, + { url = "https://files.pythonhosted.org/packages/b5/1a/3b88ccd3694681356f70da841630e4725a7264d6a885c8d442a697e1146b/numpy-2.4.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4e874c976154687c1f71715b034739b45c7711bec81db01914770373d125e392", size = 18403169, upload-time = "2026-03-29T13:20:17.096Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c9/fcfd5d0639222c6eac7f304829b04892ef51c96a75d479214d77e3ce6e33/numpy-2.4.4-cp313-cp313t-win32.whl", hash = "sha256:9c585a1790d5436a5374bac930dad6ed244c046ed91b2b2a3634eb2971d21008", size = 6083477, upload-time = "2026-03-29T13:20:20.195Z" }, + { url = "https://files.pythonhosted.org/packages/d5/e3/3938a61d1c538aaec8ed6fd6323f57b0c2d2d2219512434c5c878db76553/numpy-2.4.4-cp313-cp313t-win_amd64.whl", hash = "sha256:93e15038125dc1e5345d9b5b68aa7f996ec33b98118d18c6ca0d0b7d6198b7e8", size = 12457487, upload-time = "2026-03-29T13:20:22.946Z" }, + { url = "https://files.pythonhosted.org/packages/97/6a/7e345032cc60501721ef94e0e30b60f6b0bd601f9174ebd36389a2b86d40/numpy-2.4.4-cp313-cp313t-win_arm64.whl", hash = "sha256:0dfd3f9d3adbe2920b68b5cd3d51444e13a10792ec7154cd0a2f6e74d4ab3233", size = 10292002, upload-time = "2026-03-29T13:20:25.909Z" }, + { url = "https://files.pythonhosted.org/packages/6e/06/c54062f85f673dd5c04cbe2f14c3acb8c8b95e3384869bb8cc9bff8cb9df/numpy-2.4.4-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:f169b9a863d34f5d11b8698ead99febeaa17a13ca044961aa8e2662a6c7766a0", size = 16684353, upload-time = "2026-03-29T13:20:29.504Z" }, + { url = "https://files.pythonhosted.org/packages/4c/39/8a320264a84404c74cc7e79715de85d6130fa07a0898f67fb5cd5bd79908/numpy-2.4.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:2483e4584a1cb3092da4470b38866634bafb223cbcd551ee047633fd2584599a", size = 14704914, upload-time = "2026-03-29T13:20:33.547Z" }, + { url = "https://files.pythonhosted.org/packages/91/fb/287076b2614e1d1044235f50f03748f31fa287e3dbe6abeb35cdfa351eca/numpy-2.4.4-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:2d19e6e2095506d1736b7d80595e0f252d76b89f5e715c35e06e937679ea7d7a", size = 5210005, upload-time = "2026-03-29T13:20:36.45Z" }, + { url = "https://files.pythonhosted.org/packages/63/eb/fcc338595309910de6ecabfcef2419a9ce24399680bfb149421fa2df1280/numpy-2.4.4-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:6a246d5914aa1c820c9443ddcee9c02bec3e203b0c080349533fae17727dfd1b", size = 6544974, upload-time = "2026-03-29T13:20:39.014Z" }, + { url = "https://files.pythonhosted.org/packages/44/5d/e7e9044032a716cdfaa3fba27a8e874bf1c5f1912a1ddd4ed071bf8a14a6/numpy-2.4.4-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:989824e9faf85f96ec9c7761cd8d29c531ad857bfa1daa930cba85baaecf1a9a", size = 15684591, upload-time = "2026-03-29T13:20:42.146Z" }, + { url = "https://files.pythonhosted.org/packages/98/7c/21252050676612625449b4807d6b695b9ce8a7c9e1c197ee6216c8a65c7c/numpy-2.4.4-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27a8d92cd10f1382a67d7cf4db7ce18341b66438bdd9f691d7b0e48d104c2a9d", size = 16637700, upload-time = "2026-03-29T13:20:46.204Z" }, + { url = "https://files.pythonhosted.org/packages/b1/29/56d2bbef9465db24ef25393383d761a1af4f446a1df9b8cded4fe3a5a5d7/numpy-2.4.4-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e44319a2953c738205bf3354537979eaa3998ed673395b964c1176083dd46252", size = 17035781, upload-time = "2026-03-29T13:20:50.242Z" }, + { url = "https://files.pythonhosted.org/packages/e3/2b/a35a6d7589d21f44cea7d0a98de5ddcbb3d421b2622a5c96b1edf18707c3/numpy-2.4.4-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e892aff75639bbef0d2a2cfd55535510df26ff92f63c92cd84ef8d4ba5a5557f", size = 18362959, upload-time = "2026-03-29T13:20:54.019Z" }, + { url = "https://files.pythonhosted.org/packages/64/c9/d52ec581f2390e0f5f85cbfd80fb83d965fc15e9f0e1aec2195faa142cde/numpy-2.4.4-cp314-cp314-win32.whl", hash = "sha256:1378871da56ca8943c2ba674530924bb8ca40cd228358a3b5f302ad60cf875fc", size = 6008768, upload-time = "2026-03-29T13:20:56.912Z" }, + { url = "https://files.pythonhosted.org/packages/fa/22/4cc31a62a6c7b74a8730e31a4274c5dc80e005751e277a2ce38e675e4923/numpy-2.4.4-cp314-cp314-win_amd64.whl", hash = "sha256:715d1c092715954784bc79e1174fc2a90093dc4dc84ea15eb14dad8abdcdeb74", size = 12449181, upload-time = "2026-03-29T13:20:59.548Z" }, + { url = "https://files.pythonhosted.org/packages/70/2e/14cda6f4d8e396c612d1bf97f22958e92148801d7e4f110cabebdc0eef4b/numpy-2.4.4-cp314-cp314-win_arm64.whl", hash = "sha256:2c194dd721e54ecad9ad387c1d35e63dce5c4450c6dc7dd5611283dda239aabb", size = 10496035, upload-time = "2026-03-29T13:21:02.524Z" }, + { url = "https://files.pythonhosted.org/packages/b1/e8/8fed8c8d848d7ecea092dc3469643f9d10bc3a134a815a3b033da1d2039b/numpy-2.4.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:2aa0613a5177c264ff5921051a5719d20095ea586ca88cc802c5c218d1c67d3e", size = 14824958, upload-time = "2026-03-29T13:21:05.671Z" }, + { url = "https://files.pythonhosted.org/packages/05/1a/d8007a5138c179c2bf33ef44503e83d70434d2642877ee8fbb230e7c0548/numpy-2.4.4-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:42c16925aa5a02362f986765f9ebabf20de75cdefdca827d14315c568dcab113", size = 5330020, upload-time = "2026-03-29T13:21:08.635Z" }, + { url = "https://files.pythonhosted.org/packages/99/64/ffb99ac6ae93faf117bcbd5c7ba48a7f45364a33e8e458545d3633615dda/numpy-2.4.4-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:874f200b2a981c647340f841730fc3a2b54c9d940566a3c4149099591e2c4c3d", size = 6650758, upload-time = "2026-03-29T13:21:10.949Z" }, + { url = "https://files.pythonhosted.org/packages/6e/6e/795cc078b78a384052e73b2f6281ff7a700e9bf53bcce2ee579d4f6dd879/numpy-2.4.4-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c9b39d38a9bd2ae1becd7eac1303d031c5c110ad31f2b319c6e7d98b135c934d", size = 15729948, upload-time = "2026-03-29T13:21:14.047Z" }, + { url = "https://files.pythonhosted.org/packages/5f/86/2acbda8cc2af5f3d7bfc791192863b9e3e19674da7b5e533fded124d1299/numpy-2.4.4-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b268594bccac7d7cf5844c7732e3f20c50921d94e36d7ec9b79e9857694b1b2f", size = 16679325, upload-time = "2026-03-29T13:21:17.561Z" }, + { url = "https://files.pythonhosted.org/packages/bc/59/cafd83018f4aa55e0ac6fa92aa066c0a1877b77a615ceff1711c260ffae8/numpy-2.4.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ac6b31e35612a26483e20750126d30d0941f949426974cace8e6b5c58a3657b0", size = 17084883, upload-time = "2026-03-29T13:21:21.106Z" }, + { url = "https://files.pythonhosted.org/packages/f0/85/a42548db84e65ece46ab2caea3d3f78b416a47af387fcbb47ec28e660dc2/numpy-2.4.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:8e3ed142f2728df44263aaf5fb1f5b0b99f4070c553a0d7f033be65338329150", size = 18403474, upload-time = "2026-03-29T13:21:24.828Z" }, + { url = "https://files.pythonhosted.org/packages/ed/ad/483d9e262f4b831000062e5d8a45e342166ec8aaa1195264982bca267e62/numpy-2.4.4-cp314-cp314t-win32.whl", hash = "sha256:dddbbd259598d7240b18c9d87c56a9d2fb3b02fe266f49a7c101532e78c1d871", size = 6155500, upload-time = "2026-03-29T13:21:28.205Z" }, + { url = "https://files.pythonhosted.org/packages/c7/03/2fc4e14c7bd4ff2964b74ba90ecb8552540b6315f201df70f137faa5c589/numpy-2.4.4-cp314-cp314t-win_amd64.whl", hash = "sha256:a7164afb23be6e37ad90b2f10426149fd75aee07ca55653d2aa41e66c4ef697e", size = 12637755, upload-time = "2026-03-29T13:21:31.107Z" }, + { url = "https://files.pythonhosted.org/packages/58/78/548fb8e07b1a341746bfbecb32f2c268470f45fa028aacdbd10d9bc73aab/numpy-2.4.4-cp314-cp314t-win_arm64.whl", hash = "sha256:ba203255017337d39f89bdd58417f03c4426f12beed0440cfd933cb15f8669c7", size = 10566643, upload-time = "2026-03-29T13:21:34.339Z" }, +] + +[[package]] +name = "nvidia-cublas" +version = "13.1.1.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cuda-nvrtc", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/a7/a1/0bd24ee8c8d03adac032fd2909426a00c88f8c57961b1277ded97f91119f/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b7a210458267ac818974c53038fbec2e969d5c99f305ab15c72522fa9f001dd5", size = 542848918, upload-time = "2026-04-08T18:46:22.985Z" }, + { url = "https://files.pythonhosted.org/packages/3b/cd/154ca20c38269e05eff77c1464e6c1da89f50a6390b565e9d82e06bc11e1/nvidia_cublas-13.1.1.3-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:37936a16db8fe4ac1f065c2139360608a543a09275cb1a1af612e08cfa065436", size = 423138758, upload-time = "2026-04-08T18:46:58.655Z" }, + { url = "https://files.pythonhosted.org/packages/45/9e/2f562daf80eb8f7a685fb7bea4fda71f6048e4f359d6fdd1b6e70206cb2f/nvidia_cublas-13.1.1.3-py3-none-win_amd64.whl", hash = "sha256:b6cdce694e47ff6aadf0a69df1cab6628d696f5ff56e8d16af50309d855fa20f", size = 404358158, upload-time = "2026-04-08T18:47:26.987Z" }, +] + +[[package]] +name = "nvidia-cuda-cupti" +version = "13.0.85" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2a/2a/80353b103fc20ce05ef51e928daed4b6015db4aaa9162ed0997090fe2250/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_aarch64.whl", hash = "sha256:796bd679890ee55fb14a94629b698b6db54bcfd833d391d5e94017dd9d7d3151", size = 10310827, upload-time = "2025-09-04T08:26:42.012Z" }, + { url = "https://files.pythonhosted.org/packages/33/6d/737d164b4837a9bbd202f5ae3078975f0525a55730fe871d8ed4e3b952b0/nvidia_cuda_cupti-13.0.85-py3-none-manylinux_2_25_x86_64.whl", hash = "sha256:4eb01c08e859bf924d222250d2e8f8b8ff6d3db4721288cf35d14252a4d933c8", size = 10715597, upload-time = "2025-09-04T08:26:51.312Z" }, + { url = "https://files.pythonhosted.org/packages/ad/df/b74b10025c1205695c5676373f2edd3e87a7202cc62ead0dfbc373b0f6ea/nvidia_cuda_cupti-13.0.85-py3-none-win_amd64.whl", hash = "sha256:683f58d301548deeefcb8f6fac1b8d907691b9d8b18eccab417f51e362102f00", size = 7736776, upload-time = "2025-09-04T08:38:08.38Z" }, +] + +[[package]] +name = "nvidia-cuda-nvrtc" +version = "13.0.88" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c3/68/483a78f5e8f31b08fb1bb671559968c0ca3a065ac7acabfc7cee55214fd6/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:ad9b6d2ead2435f11cbb6868809d2adeeee302e9bb94bcf0539c7a40d80e8575", size = 90215200, upload-time = "2025-09-04T08:28:44.204Z" }, + { url = "https://files.pythonhosted.org/packages/b7/dc/6bb80850e0b7edd6588d560758f17e0550893a1feaf436807d64d2da040f/nvidia_cuda_nvrtc-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d27f20a0ca67a4bb34268a5e951033496c5b74870b868bacd046b1b8e0c3267b", size = 43015449, upload-time = "2025-09-04T08:28:20.239Z" }, + { url = "https://files.pythonhosted.org/packages/4a/af/345fedb9f4c76c84ab4fa445b36bd4048a4d9db60e6bc76b4f913ff4b852/nvidia_cuda_nvrtc-13.0.88-py3-none-win_amd64.whl", hash = "sha256:6bcd4e7f8e205cbe644f5a98f2f799bef9556fefc89dd786e79a16312ce49872", size = 76807835, upload-time = "2025-09-04T08:39:15.274Z" }, +] + +[[package]] +name = "nvidia-cuda-runtime" +version = "13.0.96" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/87/4f/17d7b9b8e285199c58ce28e31b5c5bbaa4d8271af06a89b6405258245de2/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ef9bcbe90493a2b9d810e43d249adb3d02e98dd30200d86607d8d02687c43f55", size = 2261060, upload-time = "2025-10-09T08:55:15.78Z" }, + { url = "https://files.pythonhosted.org/packages/2e/24/d1558f3b68b1d26e706813b1d10aa1d785e4698c425af8db8edc3dced472/nvidia_cuda_runtime-13.0.96-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7f82250d7782aa23b6cfe765ecc7db554bd3c2870c43f3d1821f1d18aebf0548", size = 2243632, upload-time = "2025-10-09T08:55:36.117Z" }, + { url = "https://files.pythonhosted.org/packages/b7/94/6b867483bec07da24ffa32736c79fabb94ef3a7af4d787a9d4a974868576/nvidia_cuda_runtime-13.0.96-py3-none-win_amd64.whl", hash = "sha256:f79298c8a098cec150a597c8eba58ecdab96e3bdc4b9bc4f9983635031740492", size = 2927037, upload-time = "2025-10-09T09:04:23.782Z" }, +] + +[[package]] +name = "nvidia-cudnn-cu13" +version = "9.20.0.48" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/c5/83384d846b2fd17c44bd499b36c75a45ed4f095fbbb2252294e89cea5c5c/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:e31454ae00094b0c55319d9d15b6fa2fc50a9e1c0f5c8c80fb75258234e731e1", size = 444574296, upload-time = "2026-03-09T19:28:27.751Z" }, + { url = "https://files.pythonhosted.org/packages/6e/5e/edb9c0ae051602c3ccaffe424256463636d639e27d7f302dde9975ef9e7a/nvidia_cudnn_cu13-9.20.0.48-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0c45dd8eeb50b603f07995b1b300c62ffe6a1980482b82b3bcf94a4ca9d49304", size = 366173588, upload-time = "2026-03-09T19:29:34.474Z" }, + { url = "https://files.pythonhosted.org/packages/78/39/21507455b1bca8b5702a9e9fc6ce73735f216f558dac2c9ede58e4d456b8/nvidia_cudnn_cu13-9.20.0.48-py3-none-win_amd64.whl", hash = "sha256:af8139732b99c0118be65ea5aac97f0d46018f8c552889e49d2fb0c6261a4a24", size = 350712614, upload-time = "2026-03-09T19:31:11.398Z" }, +] + +[[package]] +name = "nvidia-cufft" +version = "12.0.0.61" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/8b/ae/f417a75c0259e85c1d2f83ca4e960289a5f814ed0cea74d18c353d3e989d/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2708c852ef8cd89d1d2068bdbece0aa188813a0c934db3779b9b1faa8442e5f5", size = 214053554, upload-time = "2025-09-04T08:31:38.196Z" }, + { url = "https://files.pythonhosted.org/packages/a8/2f/7b57e29836ea8714f81e9898409196f47d772d5ddedddf1592eadb8ab743/nvidia_cufft-12.0.0.61-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6c44f692dce8fd5ffd3e3df134b6cdb9c2f72d99cf40b62c32dde45eea9ddad3", size = 214085489, upload-time = "2025-09-04T08:31:56.044Z" }, + { url = "https://files.pythonhosted.org/packages/85/b2/f8af21a2ed1beed337a6a02c5a28aeb85441f4d578ec3d529543c775ea4b/nvidia_cufft-12.0.0.61-py3-none-win_amd64.whl", hash = "sha256:2abce5b39d2f5ae12730fb7e5db6696533e36c26e2d3e8fd1750bdd2853364eb", size = 213342123, upload-time = "2025-09-04T08:40:51.145Z" }, +] + +[[package]] +name = "nvidia-cufile" +version = "1.15.1.6" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/70/4f193de89a48b71714e74602ee14d04e4019ad36a5a9f20c425776e72cd6/nvidia_cufile-1.15.1.6-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:08a3ecefae5a01c7f5117351c64f17c7c62efa5fffdbe24fc7d298da19cd0b44", size = 1223672, upload-time = "2025-09-04T08:32:22.779Z" }, + { url = "https://files.pythonhosted.org/packages/ab/73/cc4a14c9813a8a0d509417cf5f4bdaba76e924d58beb9864f5a7baceefbf/nvidia_cufile-1.15.1.6-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:bdc0deedc61f548bddf7733bdc216456c2fdb101d020e1ab4b88d232d5e2f6d1", size = 1136992, upload-time = "2025-09-04T08:32:14.119Z" }, +] + +[[package]] +name = "nvidia-curand" +version = "10.4.0.35" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/72/7c2ae24fb6b63a32e6ae5d241cc65263ea18d08802aaae087d9f013335a2/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:133df5a7509c3e292aaa2b477afd0194f06ce4ea24d714d616ff36439cee349a", size = 61962106, upload-time = "2025-08-04T10:21:41.128Z" }, + { url = "https://files.pythonhosted.org/packages/a5/9f/be0a41ca4a4917abf5cb9ae0daff1a6060cc5de950aec0396de9f3b52bc5/nvidia_curand-10.4.0.35-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:1aee33a5da6e1db083fe2b90082def8915f30f3248d5896bcec36a579d941bfc", size = 59544258, upload-time = "2025-08-04T10:22:03.992Z" }, + { url = "https://files.pythonhosted.org/packages/99/27/72103153b1ffc00e09fdc40ac970235343dcd1ea8bd762e84d2d73219ffa/nvidia_curand-10.4.0.35-py3-none-win_amd64.whl", hash = "sha256:65b1710aa6961d326b411e314b374290904c5ddf41dc3f766ebc3f1d7d4ca69f", size = 55242481, upload-time = "2025-08-04T10:30:41.831Z" }, +] + +[[package]] +name = "nvidia-cusolver" +version = "12.0.4.66" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cusparse", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-nvjitlink", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/c8/c3/b30c9e935fc01e3da443ec0116ed1b2a009bb867f5324d3f2d7e533e776b/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:02c2457eaa9e39de20f880f4bd8820e6a1cfb9f9a34f820eb12a155aa5bc92d2", size = 223467760, upload-time = "2025-09-04T08:33:04.222Z" }, + { url = "https://files.pythonhosted.org/packages/5f/67/cba3777620cdacb99102da4042883709c41c709f4b6323c10781a9c3aa34/nvidia_cusolver-12.0.4.66-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:0a759da5dea5c0ea10fd307de75cdeb59e7ea4fcb8add0924859b944babf1112", size = 200941980, upload-time = "2025-09-04T08:33:22.767Z" }, + { url = "https://files.pythonhosted.org/packages/99/ef/332a0101260ca78a1daef046bf0b06199e8ed4dac1d2aa698289c358169c/nvidia_cusolver-12.0.4.66-py3-none-win_amd64.whl", hash = "sha256:16515bd33a8e76bb54d024cfa068fa68d30e80fc34b9e1090813ea9362e0cb65", size = 193551444, upload-time = "2025-09-04T08:41:46.813Z" }, +] + +[[package]] +name = "nvidia-cusparse" +version = "12.6.3.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/f8/94/5c26f33738ae35276672f12615a64bd008ed5be6d1ebcb23579285d960a9/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:80bcc4662f23f1054ee334a15c72b8940402975e0eab63178fc7e670aa59472c", size = 162155568, upload-time = "2025-09-04T08:33:42.864Z" }, + { url = "https://files.pythonhosted.org/packages/fa/18/623c77619c31d62efd55302939756966f3ecc8d724a14dab2b75f1508850/nvidia_cusparse-12.6.3.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2b3c89c88d01ee0e477cb7f82ef60a11a4bcd57b6b87c33f789350b59759360b", size = 145942937, upload-time = "2025-09-04T08:33:58.029Z" }, + { url = "https://files.pythonhosted.org/packages/02/b0/b043d6f3480f102f885cf87fc3ffd3edcb5e23b855025a50e2ef4d059185/nvidia_cusparse-12.6.3.3-py3-none-win_amd64.whl", hash = "sha256:cbcf42feb737bd7ec15b4c0a63e62351886bd3f975027b8815d7f720a2b5ea79", size = 143783033, upload-time = "2025-09-04T08:42:12.391Z" }, +] + +[[package]] +name = "nvidia-cusparselt-cu13" +version = "0.8.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/46/e1/cdc1797eadf82d3a9a575a19b33fdc871a97edbec42c00b5b5e914f4aff4/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_aarch64.whl", hash = "sha256:4dca476c50bf4780d46cd0bfbd82e2bc10a08e4fef7950917ce8d7578d22a23f", size = 221051344, upload-time = "2025-09-05T18:49:51.289Z" }, + { url = "https://files.pythonhosted.org/packages/34/7d/2661f2fb3ac4302f3a246f5fc030213ac60c1fe0bce84f9783dbd831dbb7/nvidia_cusparselt_cu13-0.8.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:786ce87568c303fadb5afcc7102d454cd3040d75f6f8626f5db460d1871f4dd0", size = 170148586, upload-time = "2025-09-05T18:50:50.248Z" }, + { url = "https://files.pythonhosted.org/packages/31/83/f3647ce26916c94a6ca4ff1810623e2c405cff2dea6e78d29516b2514df9/nvidia_cusparselt_cu13-0.8.1-py3-none-win_amd64.whl", hash = "sha256:dccbd362f91a7b9024d1f55ee9f548ac065027ff15d8c8b0db889ab3a8f31215", size = 156885108, upload-time = "2025-09-05T18:51:35.958Z" }, +] + +[[package]] +name = "nvidia-nccl-cu13" +version = "2.29.7" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/72/0d/daf50d44177ee0cbc7ff0a0c91eb5ff676c82be42f9a970bc7597f440c3a/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_aarch64.whl", hash = "sha256:674a12383e3c38a1bcccae7d4f3633b37852230b6047883cb2f4c2d1b36d9bf5", size = 206014712, upload-time = "2026-03-03T05:34:20.843Z" }, + { url = "https://files.pythonhosted.org/packages/67/f4/58e4e91b6919367c7aafb8e36fce9aad1a3047e536bf7e2fd560927d3a4c/nvidia_nccl_cu13-2.29.7-py3-none-manylinux_2_18_x86_64.whl", hash = "sha256:edd81538446786ec3b73972543e53bb43bcaf0bfc8ef76cb679fcc390ffe136d", size = 205976000, upload-time = "2026-03-03T05:36:24.472Z" }, +] + +[[package]] +name = "nvidia-nvjitlink" +version = "13.0.88" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/7a/123e033aaff487c77107195fa5a2b8686795ca537935a24efae476c41f05/nvidia_nvjitlink-13.0.88-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:13a74f429e23b921c1109976abefacc69835f2f433ebd323d3946e11d804e47b", size = 40713933, upload-time = "2025-09-04T08:35:43.553Z" }, + { url = "https://files.pythonhosted.org/packages/ab/2c/93c5250e64df4f894f1cbb397c6fd71f79813f9fd79d7cd61de3f97b3c2d/nvidia_nvjitlink-13.0.88-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e931536ccc7d467a98ba1d8b89ff7fa7f1fa3b13f2b0069118cd7f47bff07d0c", size = 38768748, upload-time = "2025-09-04T08:35:20.008Z" }, + { url = "https://files.pythonhosted.org/packages/e4/01/07530b0e37546231052e30234540289c42eaffa486f1a34a87fed340157b/nvidia_nvjitlink-13.0.88-py3-none-win_amd64.whl", hash = "sha256:634e96e3da9ef845ae744097a1f289238ecf946ce0b82e93cdce14b9782e682f", size = 36035115, upload-time = "2025-09-04T08:43:03.001Z" }, +] + +[[package]] +name = "nvidia-nvshmem-cu13" +version = "3.4.5" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/0f/05cc9c720236dcd2db9c1ab97fff629e96821be2e63103569da0c9b72f19/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6dc2a197f38e5d0376ad52cd1a2a3617d3cdc150fd5966f4aee9bcebb1d68fe9", size = 60215947, upload-time = "2025-09-06T00:32:20.022Z" }, + { url = "https://files.pythonhosted.org/packages/3c/35/a9bf80a609e74e3b000fef598933235c908fcefcef9026042b8e6dfde2a9/nvidia_nvshmem_cu13-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:290f0a2ee94c9f3687a02502f3b9299a9f9fe826e6d0287ee18482e78d495b80", size = 60412546, upload-time = "2025-09-06T00:32:41.564Z" }, +] + +[[package]] +name = "nvidia-nvtx" +version = "13.0.85" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/f3/d86c845465a2723ad7e1e5c36dcd75ddb82898b3f53be47ebd429fb2fa5d/nvidia_nvtx-13.0.85-py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:4936d1d6780fbe68db454f5e72a42ff64d1fd6397df9f363ae786930fd5c1cd4", size = 148047, upload-time = "2025-09-04T08:29:01.761Z" }, + { url = "https://files.pythonhosted.org/packages/a8/64/3708a90d1ebe202ffdeb7185f878a3c84d15c2b2c31858da2ce0583e2def/nvidia_nvtx-13.0.85-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb7780edb6b14107373c835bf8b72e7a178bac7367e23da7acb108f973f157a6", size = 148878, upload-time = "2025-09-04T08:28:53.627Z" }, + { url = "https://files.pythonhosted.org/packages/d2/50/0e2220f8620a177de994211186ffc5bfa9f2ce1e1282797f8f90096f9f88/nvidia_nvtx-13.0.85-py3-none-win_amd64.whl", hash = "sha256:d66ea44254dd3c6eacc300047af6e1288d2269dd072b417e0adffbf479e18519", size = 137066, upload-time = "2025-09-04T08:39:25.649Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "paginate" +version = "0.5.7" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ec/46/68dde5b6bc00c1296ec6466ab27dddede6aec9af1b99090e1107091b3b84/paginate-0.5.7.tar.gz", hash = "sha256:22bd083ab41e1a8b4f3690544afb2c60c25e5c9a63a30fa2f483f6c60c8e5945", size = 19252, upload-time = "2024-08-25T14:17:24.139Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/90/96/04b8e52da071d28f5e21a805b19cb9390aa17a47462ac87f5e2696b9566d/paginate-0.5.7-py2.py3-none-any.whl", hash = "sha256:b885e2af73abcf01d9559fd5216b57ef722f8c42affbb63942377668e35c7591", size = 13746, upload-time = "2024-08-25T14:17:22.55Z" }, +] + +[[package]] +name = "pandas" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f8/87/4341c6252d1c47b08768c3d25ac487362bf403f0313ddae4a2a26c9b1b4c/pandas-3.0.3.tar.gz", hash = "sha256:696a4a00a2a2a35d4e5deb3fc946641b96c944f02230e4f76137fe35d806c4fc", size = 4651414, upload-time = "2026-05-11T18:54:29.21Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/f1/392f8c5bfc16f66a0d2d41561c01627c228fe7ed2a0d056ef11315042570/pandas-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fed2ff7fd9779120e388e285fc029bd5cf9490cdd2e4166a9ee22c0e49a9ab09", size = 10357846, upload-time = "2026-05-11T18:52:36.143Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3d/b16412745651e855f357e5e66930248688378853a6e2698a214e331fba1f/pandas-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b168fc218fd80a6cbdbdbc1a97ddc7889ed057d7eb45f50d866ceab5f39904c4", size = 9899550, upload-time = "2026-05-11T18:52:38.976Z" }, + { url = "https://files.pythonhosted.org/packages/31/a8/fa2535168fffcedf67f4f6de28d2dd903a747ca7c8ea6989451aaeb3a92f/pandas-3.0.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0383c72c75cdcca61a9e116e611143902dbfd08bff356829c2f6d1cf40a9ca8c", size = 10412965, upload-time = "2026-05-11T18:52:41.915Z" }, + { url = "https://files.pythonhosted.org/packages/65/b6/09b01cdbc15224e2850365192d17b7bdebb8bdbd8780ed221fcdf0d9a515/pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6dc0b3fd2169c9157deed50b4d519553a3655c8c6a96027136d654592be973a9", size = 10894600, upload-time = "2026-05-11T18:52:45.02Z" }, + { url = "https://files.pythonhosted.org/packages/c9/a4/2eb28f2fccb4ced4a2c79ab2a5dee9ade1ebf44922ebad6fea158c9f95d4/pandas-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7e65d5407dc0b394f509699650e4a2ec01c0514f21850f453fa60f3be79a5dbf", size = 11422824, upload-time = "2026-05-11T18:52:48.058Z" }, + { url = "https://files.pythonhosted.org/packages/f8/45/830bb57f533a4604b355e07edcb8ea18cf88b5f94e5fca92f27052d7c597/pandas-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f8894dc474d648fe7b6ff0ca9b0bd73950d19952bc1a6534540762c5d79d305c", size = 11950889, upload-time = "2026-05-11T18:52:50.905Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c5/fc1b368f303087d20e8c9bf3d6ceb186263cfac0ade735cd938538bea839/pandas-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:c7be265b62cef88e253a941e4698604973736dcfe242fdb5198f0f7bc473cdcc", size = 9755463, upload-time = "2026-05-11T18:52:53.386Z" }, + { url = "https://files.pythonhosted.org/packages/86/bd/fda8f9705b1b09c6ebe14bfc0fa0e4ec8584d54ea673628f157ff55131af/pandas-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:557409bc4178e70ee8d9ddb494798e51ebf6ea59330f6be22c51bab2a7db6c49", size = 9066158, upload-time = "2026-05-11T18:52:56.038Z" }, + { url = "https://files.pythonhosted.org/packages/c5/90/62d8302883c44308c477e222c3daf7c813a34c8e96985882fbd53d964352/pandas-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:67b3b64c11910cfa29f4e94a14d3bff9ee693b6fc76055e7cad549cee0aec5fa", size = 10331071, upload-time = "2026-05-11T18:52:58.838Z" }, + { url = "https://files.pythonhosted.org/packages/7f/ae/6a6493c783a101f165e4356953ba3c74d6f77f0042fa7d753da9dfbb640c/pandas-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:39436b377d56d2a2e52d0395bdbee171f01068e99af5250509aceeb929f765c7", size = 9875690, upload-time = "2026-05-11T18:53:01.431Z" }, + { url = "https://files.pythonhosted.org/packages/62/7c/5df8e9f56c69a2769fbe9382a5ef8f2658c007e376434e1e2cbb57ad895f/pandas-3.0.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d4be06d68f9ddcfc645b87534911da79a8fbffc7573c80e0edcf42a5020624d8", size = 10381634, upload-time = "2026-05-11T18:53:04.393Z" }, + { url = "https://files.pythonhosted.org/packages/99/68/1237369725aa617bb358263d535803e3053fdbc593513ec5ed9c9896b5b6/pandas-3.0.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a4eeb6830daf35a71cc09649bd823e2b542dac246cdee9614c6e4bd65028cd6a", size = 10891243, upload-time = "2026-05-11T18:53:07.643Z" }, + { url = "https://files.pythonhosted.org/packages/25/93/77d108e8af7222b4a503ebde0e30215b1c2e4f8e53a526431890f22d5586/pandas-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1928e07221f82db493cd4af1e23c1bfca524a19a4699887975bff68f49a72bfb", size = 11388659, upload-time = "2026-05-11T18:53:10.634Z" }, + { url = "https://files.pythonhosted.org/packages/d0/bd/eff5b4399f332ac386c853f6cd2bd3fa2ca0061b9f36ecd9c4d7c4265649/pandas-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51b1fe551acb77dac643c6fda86084d8d446c10fe64b06a9cc29c4cc8540e7f2", size = 11942880, upload-time = "2026-05-11T18:53:13.536Z" }, + { url = "https://files.pythonhosted.org/packages/2c/20/559ace4200982c3887d0b86bfd0d856a2143ef8ddab63cc07934951a964c/pandas-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:a82d532a3351d435432cd913edbccaf8b8e01d4dd0e5ced5a8d2e8ecd94c7e44", size = 9757091, upload-time = "2026-05-11T18:53:16.306Z" }, + { url = "https://files.pythonhosted.org/packages/3a/66/69055a09fe200f29f922a3eeec4804611900b95f52d932ece3393c3c0c19/pandas-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:275c14e0fce14a2ec20eee474aecd305478ea3c1e6f6a9d8fe219a165542717e", size = 9057282, upload-time = "2026-05-11T18:53:18.768Z" }, + { url = "https://files.pythonhosted.org/packages/57/0e/efe801b0e6811e8e650cd21b7f2608e30f08a7067e2bf6e8752b0d56ee3c/pandas-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:46997386d528eb40376ecd6b033cf4a8a1e5282580f68f43de875b78cba2199d", size = 10767016, upload-time = "2026-05-11T18:53:21.227Z" }, + { url = "https://files.pythonhosted.org/packages/ea/dc/eb55135a1d5f0f0519f28da1f609a206d2cad1f9c35c32d51e38dd7261ae/pandas-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:261e308dfb22448384b7580cf719d2f998fe2966c92893c3e77d14008af1f066", size = 10420210, upload-time = "2026-05-11T18:53:23.982Z" }, + { url = "https://files.pythonhosted.org/packages/c6/3e/b1d5d955ce33ffecb407465a60bc32769d74fcf68224b7ae67ae11d4dea4/pandas-3.0.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dd1a5d1def6a46002e964510bdc67c368aa0951df5d1d9f8365336f5a1f490cd", size = 10336126, upload-time = "2026-05-11T18:53:26.731Z" }, + { url = "https://files.pythonhosted.org/packages/f5/76/a01261711ab60a22d71b862f0de20e4c504bf80457270ad8cb42110f6abc/pandas-3.0.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d72828c20c6d6e83e1e22a6a3b47b326b71664112fa9705dcbccfd7a39b62085", size = 10728051, upload-time = "2026-05-11T18:53:29.125Z" }, + { url = "https://files.pythonhosted.org/packages/e9/21/ea191195e587b18cf682e97f433f81b2d0fbe341380e80a3e0d6e4403c8e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d26cbe1fcfc12e8fd900e2454163e466b2d3af84f7c75481df7683ffc073d870", size = 11350796, upload-time = "2026-05-11T18:53:32.056Z" }, + { url = "https://files.pythonhosted.org/packages/64/69/f0eaaf54939f0e8c6768fd06be9af2cef9b36048b96dfb9e1b2c685a807e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:3e91cec1879ada0624fc3dc9953c5cbd60208e59c0db28f540c5d6d47502422f", size = 11799741, upload-time = "2026-05-11T18:53:34.985Z" }, + { url = "https://files.pythonhosted.org/packages/45/a4/865e0e510cae5fc2194de4db28be638952de942571ba9125934fd9c01d47/pandas-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:08d789b41f87e0905880e293cedf6197ce71fe67cc081358b1e148a491b9bd13", size = 10499958, upload-time = "2026-05-11T18:53:37.857Z" }, + { url = "https://files.pythonhosted.org/packages/86/54/effdcc3c0ff7a08037889200e148ebe94c16c4f653be078c7b3675955df1/pandas-3.0.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3650109c0f22879df8bd6179ab9ee3d7f1d1d4e7e0094a3f0032d9f51e2e64ac", size = 10336065, upload-time = "2026-05-11T18:53:41.099Z" }, + { url = "https://files.pythonhosted.org/packages/68/10/bf2d6738d72748b961a3751ab89522d58c54efc36a8e1a12161216cd45cf/pandas-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:bab900348131a7db1f69a7309ef141fd5680f1487094193bcbbb61791573bf8f", size = 9926101, upload-time = "2026-05-11T18:53:43.515Z" }, + { url = "https://files.pythonhosted.org/packages/ae/e9/e35cf11c8a136e757b956f5f0efdcaa50aecde85ea055f1898dfc68262f3/pandas-3.0.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ba7e08b9ac1d54569cd1e256e3668975ed624d6826f7b68df0342b012007bddb", size = 10457553, upload-time = "2026-05-11T18:53:46.394Z" }, + { url = "https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d71c63ae4ebdbf70209742096f1fc46a83a0613c99d4b23766cced9ff8cd62a", size = 10914065, upload-time = "2026-05-11T18:53:49.134Z" }, + { url = "https://files.pythonhosted.org/packages/c4/c2/1ef644445fcd72e3627bceec77e3560636f87ddce4ed841afe76b83b5bf9/pandas-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e3a2ec42c98ffa2565a67e08e218d06d72576d758d90facb7c00805194d8f360", size = 11459188, upload-time = "2026-05-11T18:53:52.527Z" }, + { url = "https://files.pythonhosted.org/packages/7e/49/4d8d4f42cbc9c4adc7a1870f269c02cbd6cd40d059622c06fb298addcbad/pandas-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:335f62418ed562cfc3c49e9e196375c28b729dcef8543abf4f9438e381bf3c76", size = 11982966, upload-time = "2026-05-11T18:53:55.043Z" }, + { url = "https://files.pythonhosted.org/packages/38/55/792619469bab9882d8bbd5865d45a72f6478762d04a9af4bf0d08c503e95/pandas-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:3c20a521bbb85902f79f7270c80a59e1b5452d96d170c034f207181870f97ac5", size = 9876755, upload-time = "2026-05-11T18:53:58.067Z" }, + { url = "https://files.pythonhosted.org/packages/2a/af/33c469653b0ba03b50c3a98192d4c07f0c75c66b263ceb097fce0ee97d31/pandas-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:a2d2dff8a04f3917b55ab3910c32990f8ddf7eceba114947838cefa976a68977", size = 9198658, upload-time = "2026-05-11T18:54:00.733Z" }, + { url = "https://files.pythonhosted.org/packages/a2/fa/b8c257bd76b8bd060c3a9151c1fca05e9b9c5e3af5d0f549c0356f6d143d/pandas-3.0.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:0d589105b3c14645af1738ff279b2995102d8f7a03b0a66dc8d95550eb513e04", size = 10787242, upload-time = "2026-05-11T18:54:03.564Z" }, + { url = "https://files.pythonhosted.org/packages/54/eb/f19206ffb0bf1919002969aa448b4702c6594845156a6f8050674855aac3/pandas-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:13fc1e853d9e04743d11ba75a985ccbc2a317fe07d8af61e445a6fd24dacd6a6", size = 10436369, upload-time = "2026-05-11T18:54:06.311Z" }, + { url = "https://files.pythonhosted.org/packages/fd/24/c7c39fb4fe22b71a0c2d78bf0c585c600092d85f94f086d2b3b2f6ca27e2/pandas-3.0.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:819959dab7bbd0049c15623fbac4e29a191b9528160a61fb1032242d8ced2d9c", size = 10358306, upload-time = "2026-05-11T18:54:09.085Z" }, + { url = "https://files.pythonhosted.org/packages/16/ec/dd2a9eb7fa1204df88c0864164e35b228ac581062ac612ba0a67fd812e4c/pandas-3.0.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:60ae316d3fd75d1858d450d0db0103ea2be3e7d4a95ec2f064f7e2ae63f7b028", size = 10758394, upload-time = "2026-05-11T18:54:11.956Z" }, + { url = "https://files.pythonhosted.org/packages/95/6e/00c61ea8e85b4f6d8d35e11852a1a4998fc7fafc91c6a602d1cc9c972d64/pandas-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bd3a518890b400d32f9023722dc9a9a5c969f00b415419a3c06c043f09bb5d7d", size = 11375717, upload-time = "2026-05-11T18:54:14.539Z" }, + { url = "https://files.pythonhosted.org/packages/31/89/8fc1c268969fac43688d65fd92e67df24bd128d53cb4d2eee534cd307399/pandas-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9c39be2d709d01fa972a0cabc522389fceca4f3969332ba25a7d6c5802cf976a", size = 11828897, upload-time = "2026-05-11T18:54:17.146Z" }, + { url = "https://files.pythonhosted.org/packages/56/3b/e7d20dea247a3e6dc0bd8a6953854afbedc03951def4e7371e05e7263e25/pandas-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4db8c527972a821cf5286b40ccc57642a39bc62e62022b42f99f8a67fca8c3a1", size = 10900855, upload-time = "2026-05-11T18:54:19.72Z" }, + { url = "https://files.pythonhosted.org/packages/0f/54/68a0978d1ef8502b8492099beaa6e7a0c1b32e3b5d4f677f5810cb08711c/pandas-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b2c95f8bfc1ee412bf482605d7bfd30c12d1d26bd59fdd91efeef1d4718decb1", size = 9466464, upload-time = "2026-05-11T18:54:22.754Z" }, +] + +[[package]] +name = "pathspec" +version = "1.1.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/5a/82/42f767fc1c1143d6fd36efb827202a2d997a375e160a71eb2888a925aac1/pathspec-1.1.1.tar.gz", hash = "sha256:17db5ecd524104a120e173814c90367a96a98d07c45b2e10c2f3919fff91bf5a", size = 135180, upload-time = "2026-04-27T01:46:08.907Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/d9/7fb5aa316bc299258e68c73ba3bddbc499654a07f151cba08f6153988714/pathspec-1.1.1-py3-none-any.whl", hash = "sha256:a00ce642f577bf7f473932318056212bc4f8bfdf53128c78bbd5af0b9b20b189", size = 57328, upload-time = "2026-04-27T01:46:07.06Z" }, +] + +[[package]] +name = "pillow" +version = "12.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/8c/21/c2bcdd5906101a30244eaffc1b6e6ce71a31bd0742a01eb89e660ebfac2d/pillow-12.2.0.tar.gz", hash = "sha256:a830b1a40919539d07806aa58e1b114df53ddd43213d9c8b75847eee6c0182b5", size = 46987819, upload-time = "2026-04-01T14:46:17.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/58/be/7482c8a5ebebbc6470b3eb791812fff7d5e0216c2be3827b30b8bb6603ed/pillow-12.2.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2d192a155bbcec180f8564f693e6fd9bccff5a7af9b32e2e4bf8c9c69dbad6b5", size = 5308279, upload-time = "2026-04-01T14:43:13.246Z" }, + { url = "https://files.pythonhosted.org/packages/d8/95/0a351b9289c2b5cbde0bacd4a83ebc44023e835490a727b2a3bd60ddc0f4/pillow-12.2.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f3f40b3c5a968281fd507d519e444c35f0ff171237f4fdde090dd60699458421", size = 4695490, upload-time = "2026-04-01T14:43:15.584Z" }, + { url = "https://files.pythonhosted.org/packages/de/af/4e8e6869cbed569d43c416fad3dc4ecb944cb5d9492defaed89ddd6fe871/pillow-12.2.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:03e7e372d5240cc23e9f07deca4d775c0817bffc641b01e9c3af208dbd300987", size = 6284462, upload-time = "2026-04-01T14:43:18.268Z" }, + { url = "https://files.pythonhosted.org/packages/e9/9e/c05e19657fd57841e476be1ab46c4d501bffbadbafdc31a6d665f8b737b6/pillow-12.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b86024e52a1b269467a802258c25521e6d742349d760728092e1bc2d135b4d76", size = 8094744, upload-time = "2026-04-01T14:43:20.716Z" }, + { url = "https://files.pythonhosted.org/packages/2b/54/1789c455ed10176066b6e7e6da1b01e50e36f94ba584dc68d9eebfe9156d/pillow-12.2.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7371b48c4fa448d20d2714c9a1f775a81155050d383333e0a6c15b1123dda005", size = 6398371, upload-time = "2026-04-01T14:43:23.443Z" }, + { url = "https://files.pythonhosted.org/packages/43/e3/fdc657359e919462369869f1c9f0e973f353f9a9ee295a39b1fea8ee1a77/pillow-12.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:62f5409336adb0663b7caa0da5c7d9e7bdbaae9ce761d34669420c2a801b2780", size = 7087215, upload-time = "2026-04-01T14:43:26.758Z" }, + { url = "https://files.pythonhosted.org/packages/8b/f8/2f6825e441d5b1959d2ca5adec984210f1ec086435b0ed5f52c19b3b8a6e/pillow-12.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:01afa7cf67f74f09523699b4e88c73fb55c13346d212a59a2db1f86b0a63e8c5", size = 6509783, upload-time = "2026-04-01T14:43:29.56Z" }, + { url = "https://files.pythonhosted.org/packages/67/f9/029a27095ad20f854f9dba026b3ea6428548316e057e6fc3545409e86651/pillow-12.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fc3d34d4a8fbec3e88a79b92e5465e0f9b842b628675850d860b8bd300b159f5", size = 7212112, upload-time = "2026-04-01T14:43:32.091Z" }, + { url = "https://files.pythonhosted.org/packages/be/42/025cfe05d1be22dbfdb4f264fe9de1ccda83f66e4fc3aac94748e784af04/pillow-12.2.0-cp312-cp312-win32.whl", hash = "sha256:58f62cc0f00fd29e64b29f4fd923ffdb3859c9f9e6105bfc37ba1d08994e8940", size = 6378489, upload-time = "2026-04-01T14:43:34.601Z" }, + { url = "https://files.pythonhosted.org/packages/5d/7b/25a221d2c761c6a8ae21bfa3874988ff2583e19cf8a27bf2fee358df7942/pillow-12.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:7f84204dee22a783350679a0333981df803dac21a0190d706a50475e361c93f5", size = 7084129, upload-time = "2026-04-01T14:43:37.213Z" }, + { url = "https://files.pythonhosted.org/packages/10/e1/542a474affab20fd4a0f1836cb234e8493519da6b76899e30bcc5d990b8b/pillow-12.2.0-cp312-cp312-win_arm64.whl", hash = "sha256:af73337013e0b3b46f175e79492d96845b16126ddf79c438d7ea7ff27783a414", size = 2463612, upload-time = "2026-04-01T14:43:39.421Z" }, + { url = "https://files.pythonhosted.org/packages/4a/01/53d10cf0dbad820a8db274d259a37ba50b88b24768ddccec07355382d5ad/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:8297651f5b5679c19968abefd6bb84d95fe30ef712eb1b2d9b2d31ca61267f4c", size = 4100837, upload-time = "2026-04-01T14:43:41.506Z" }, + { url = "https://files.pythonhosted.org/packages/0f/98/f3a6657ecb698c937f6c76ee564882945f29b79bad496abcba0e84659ec5/pillow-12.2.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:50d8520da2a6ce0af445fa6d648c4273c3eeefbc32d7ce049f22e8b5c3daecc2", size = 4176528, upload-time = "2026-04-01T14:43:43.773Z" }, + { url = "https://files.pythonhosted.org/packages/69/bc/8986948f05e3ea490b8442ea1c1d4d990b24a7e43d8a51b2c7d8b1dced36/pillow-12.2.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:766cef22385fa1091258ad7e6216792b156dc16d8d3fa607e7545b2b72061f1c", size = 3640401, upload-time = "2026-04-01T14:43:45.87Z" }, + { url = "https://files.pythonhosted.org/packages/34/46/6c717baadcd62bc8ed51d238d521ab651eaa74838291bda1f86fe1f864c9/pillow-12.2.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5d2fd0fa6b5d9d1de415060363433f28da8b1526c1c129020435e186794b3795", size = 5308094, upload-time = "2026-04-01T14:43:48.438Z" }, + { url = "https://files.pythonhosted.org/packages/71/43/905a14a8b17fdb1ccb58d282454490662d2cb89a6bfec26af6d3520da5ec/pillow-12.2.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:56b25336f502b6ed02e889f4ece894a72612fe885889a6e8c4c80239ff6e5f5f", size = 4695402, upload-time = "2026-04-01T14:43:51.292Z" }, + { url = "https://files.pythonhosted.org/packages/73/dd/42107efcb777b16fa0393317eac58f5b5cf30e8392e266e76e51cff28c3d/pillow-12.2.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f1c943e96e85df3d3478f7b691f229887e143f81fedab9b20205349ab04d73ed", size = 6280005, upload-time = "2026-04-01T14:43:54.242Z" }, + { url = "https://files.pythonhosted.org/packages/a8/68/b93e09e5e8549019e61acf49f65b1a8530765a7f812c77a7461bca7e4494/pillow-12.2.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:03f6fab9219220f041c74aeaa2939ff0062bd5c364ba9ce037197f4c6d498cd9", size = 8090669, upload-time = "2026-04-01T14:43:57.335Z" }, + { url = "https://files.pythonhosted.org/packages/4b/6e/3ccb54ce8ec4ddd1accd2d89004308b7b0b21c4ac3d20fa70af4760a4330/pillow-12.2.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5cdfebd752ec52bf5bb4e35d9c64b40826bc5b40a13df7c3cda20a2c03a0f5ed", size = 6395194, upload-time = "2026-04-01T14:43:59.864Z" }, + { url = "https://files.pythonhosted.org/packages/67/ee/21d4e8536afd1a328f01b359b4d3997b291ffd35a237c877b331c1c3b71c/pillow-12.2.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eedf4b74eda2b5a4b2b2fb4c006d6295df3bf29e459e198c90ea48e130dc75c3", size = 7082423, upload-time = "2026-04-01T14:44:02.74Z" }, + { url = "https://files.pythonhosted.org/packages/78/5f/e9f86ab0146464e8c133fe85df987ed9e77e08b29d8d35f9f9f4d6f917ba/pillow-12.2.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:00a2865911330191c0b818c59103b58a5e697cae67042366970a6b6f1b20b7f9", size = 6505667, upload-time = "2026-04-01T14:44:05.381Z" }, + { url = "https://files.pythonhosted.org/packages/ed/1e/409007f56a2fdce61584fd3acbc2bbc259857d555196cedcadc68c015c82/pillow-12.2.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1e1757442ed87f4912397c6d35a0db6a7b52592156014706f17658ff58bbf795", size = 7208580, upload-time = "2026-04-01T14:44:08.39Z" }, + { url = "https://files.pythonhosted.org/packages/23/c4/7349421080b12fb35414607b8871e9534546c128a11965fd4a7002ccfbee/pillow-12.2.0-cp313-cp313-win32.whl", hash = "sha256:144748b3af2d1b358d41286056d0003f47cb339b8c43a9ea42f5fea4d8c66b6e", size = 6375896, upload-time = "2026-04-01T14:44:11.197Z" }, + { url = "https://files.pythonhosted.org/packages/3f/82/8a3739a5e470b3c6cbb1d21d315800d8e16bff503d1f16b03a4ec3212786/pillow-12.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:390ede346628ccc626e5730107cde16c42d3836b89662a115a921f28440e6a3b", size = 7081266, upload-time = "2026-04-01T14:44:13.947Z" }, + { url = "https://files.pythonhosted.org/packages/c3/25/f968f618a062574294592f668218f8af564830ccebdd1fa6200f598e65c5/pillow-12.2.0-cp313-cp313-win_arm64.whl", hash = "sha256:8023abc91fba39036dbce14a7d6535632f99c0b857807cbbbf21ecc9f4717f06", size = 2463508, upload-time = "2026-04-01T14:44:16.312Z" }, + { url = "https://files.pythonhosted.org/packages/4d/a4/b342930964e3cb4dce5038ae34b0eab4653334995336cd486c5a8c25a00c/pillow-12.2.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:042db20a421b9bafecc4b84a8b6e444686bd9d836c7fd24542db3e7df7baad9b", size = 5309927, upload-time = "2026-04-01T14:44:18.89Z" }, + { url = "https://files.pythonhosted.org/packages/9f/de/23198e0a65a9cf06123f5435a5d95cea62a635697f8f03d134d3f3a96151/pillow-12.2.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:dd025009355c926a84a612fecf58bb315a3f6814b17ead51a8e48d3823d9087f", size = 4698624, upload-time = "2026-04-01T14:44:21.115Z" }, + { url = "https://files.pythonhosted.org/packages/01/a6/1265e977f17d93ea37aa28aa81bad4fa597933879fac2520d24e021c8da3/pillow-12.2.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:88ddbc66737e277852913bd1e07c150cc7bb124539f94c4e2df5344494e0a612", size = 6321252, upload-time = "2026-04-01T14:44:23.663Z" }, + { url = "https://files.pythonhosted.org/packages/3c/83/5982eb4a285967baa70340320be9f88e57665a387e3a53a7f0db8231a0cd/pillow-12.2.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d362d1878f00c142b7e1a16e6e5e780f02be8195123f164edf7eddd911eefe7c", size = 8126550, upload-time = "2026-04-01T14:44:26.772Z" }, + { url = "https://files.pythonhosted.org/packages/4e/48/6ffc514adce69f6050d0753b1a18fd920fce8cac87620d5a31231b04bfc5/pillow-12.2.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c727a6d53cb0018aadd8018c2b938376af27914a68a492f59dfcaca650d5eea", size = 6433114, upload-time = "2026-04-01T14:44:29.615Z" }, + { url = "https://files.pythonhosted.org/packages/36/a3/f9a77144231fb8d40ee27107b4463e205fa4677e2ca2548e14da5cf18dce/pillow-12.2.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:efd8c21c98c5cc60653bcb311bef2ce0401642b7ce9d09e03a7da87c878289d4", size = 7115667, upload-time = "2026-04-01T14:44:32.773Z" }, + { url = "https://files.pythonhosted.org/packages/c1/fc/ac4ee3041e7d5a565e1c4fd72a113f03b6394cc72ab7089d27608f8aaccb/pillow-12.2.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:9f08483a632889536b8139663db60f6724bfcb443c96f1b18855860d7d5c0fd4", size = 6538966, upload-time = "2026-04-01T14:44:35.252Z" }, + { url = "https://files.pythonhosted.org/packages/c0/a8/27fb307055087f3668f6d0a8ccb636e7431d56ed0750e07a60547b1e083e/pillow-12.2.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dac8d77255a37e81a2efcbd1fc05f1c15ee82200e6c240d7e127e25e365c39ea", size = 7238241, upload-time = "2026-04-01T14:44:37.875Z" }, + { url = "https://files.pythonhosted.org/packages/ad/4b/926ab182c07fccae9fcb120043464e1ff1564775ec8864f21a0ebce6ac25/pillow-12.2.0-cp313-cp313t-win32.whl", hash = "sha256:ee3120ae9dff32f121610bb08e4313be87e03efeadfc6c0d18f89127e24d0c24", size = 6379592, upload-time = "2026-04-01T14:44:40.336Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c4/f9e476451a098181b30050cc4c9a3556b64c02cf6497ea421ac047e89e4b/pillow-12.2.0-cp313-cp313t-win_amd64.whl", hash = "sha256:325ca0528c6788d2a6c3d40e3568639398137346c3d6e66bb61db96b96511c98", size = 7085542, upload-time = "2026-04-01T14:44:43.251Z" }, + { url = "https://files.pythonhosted.org/packages/00/a4/285f12aeacbe2d6dc36c407dfbbe9e96d4a80b0fb710a337f6d2ad978c75/pillow-12.2.0-cp313-cp313t-win_arm64.whl", hash = "sha256:2e5a76d03a6c6dcef67edabda7a52494afa4035021a79c8558e14af25313d453", size = 2465765, upload-time = "2026-04-01T14:44:45.996Z" }, + { url = "https://files.pythonhosted.org/packages/bf/98/4595daa2365416a86cb0d495248a393dfc84e96d62ad080c8546256cb9c0/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:3adc9215e8be0448ed6e814966ecf3d9952f0ea40eb14e89a102b87f450660d8", size = 4100848, upload-time = "2026-04-01T14:44:48.48Z" }, + { url = "https://files.pythonhosted.org/packages/0b/79/40184d464cf89f6663e18dfcf7ca21aae2491fff1a16127681bf1fa9b8cf/pillow-12.2.0-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:6a9adfc6d24b10f89588096364cc726174118c62130c817c2837c60cf08a392b", size = 4176515, upload-time = "2026-04-01T14:44:51.353Z" }, + { url = "https://files.pythonhosted.org/packages/b0/63/703f86fd4c422a9cf722833670f4f71418fb116b2853ff7da722ea43f184/pillow-12.2.0-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:6a6e67ea2e6feda684ed370f9a1c52e7a243631c025ba42149a2cc5934dec295", size = 3640159, upload-time = "2026-04-01T14:44:53.588Z" }, + { url = "https://files.pythonhosted.org/packages/71/e0/fb22f797187d0be2270f83500aab851536101b254bfa1eae10795709d283/pillow-12.2.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:2bb4a8d594eacdfc59d9e5ad972aa8afdd48d584ffd5f13a937a664c3e7db0ed", size = 5312185, upload-time = "2026-04-01T14:44:56.039Z" }, + { url = "https://files.pythonhosted.org/packages/ba/8c/1a9e46228571de18f8e28f16fabdfc20212a5d019f3e3303452b3f0a580d/pillow-12.2.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:80b2da48193b2f33ed0c32c38140f9d3186583ce7d516526d462645fd98660ae", size = 4695386, upload-time = "2026-04-01T14:44:58.663Z" }, + { url = "https://files.pythonhosted.org/packages/70/62/98f6b7f0c88b9addd0e87c217ded307b36be024d4ff8869a812b241d1345/pillow-12.2.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:22db17c68434de69d8ecfc2fe821569195c0c373b25cccb9cbdacf2c6e53c601", size = 6280384, upload-time = "2026-04-01T14:45:01.5Z" }, + { url = "https://files.pythonhosted.org/packages/5e/03/688747d2e91cfbe0e64f316cd2e8005698f76ada3130d0194664174fa5de/pillow-12.2.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7b14cc0106cd9aecda615dd6903840a058b4700fcb817687d0ee4fc8b6e389be", size = 8091599, upload-time = "2026-04-01T14:45:04.5Z" }, + { url = "https://files.pythonhosted.org/packages/f6/35/577e22b936fcdd66537329b33af0b4ccfefaeabd8aec04b266528cddb33c/pillow-12.2.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8cbeb542b2ebc6fcdacabf8aca8c1a97c9b3ad3927d46b8723f9d4f033288a0f", size = 6396021, upload-time = "2026-04-01T14:45:07.117Z" }, + { url = "https://files.pythonhosted.org/packages/11/8d/d2532ad2a603ca2b93ad9f5135732124e57811d0168155852f37fbce2458/pillow-12.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4bfd07bc812fbd20395212969e41931001fd59eb55a60658b0e5710872e95286", size = 7083360, upload-time = "2026-04-01T14:45:09.763Z" }, + { url = "https://files.pythonhosted.org/packages/5e/26/d325f9f56c7e039034897e7380e9cc202b1e368bfd04d4cbe6a441f02885/pillow-12.2.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9aba9a17b623ef750a4d11b742cbafffeb48a869821252b30ee21b5e91392c50", size = 6507628, upload-time = "2026-04-01T14:45:12.378Z" }, + { url = "https://files.pythonhosted.org/packages/5f/f7/769d5632ffb0988f1c5e7660b3e731e30f7f8ec4318e94d0a5d674eb65a4/pillow-12.2.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:deede7c263feb25dba4e82ea23058a235dcc2fe1f6021025dc71f2b618e26104", size = 7209321, upload-time = "2026-04-01T14:45:15.122Z" }, + { url = "https://files.pythonhosted.org/packages/6a/7a/c253e3c645cd47f1aceea6a8bacdba9991bf45bb7dfe927f7c893e89c93c/pillow-12.2.0-cp314-cp314-win32.whl", hash = "sha256:632ff19b2778e43162304d50da0181ce24ac5bb8180122cbe1bf4673428328c7", size = 6479723, upload-time = "2026-04-01T14:45:17.797Z" }, + { url = "https://files.pythonhosted.org/packages/cd/8b/601e6566b957ca50e28725cb6c355c59c2c8609751efbecd980db44e0349/pillow-12.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:4e6c62e9d237e9b65fac06857d511e90d8461a32adcc1b9065ea0c0fa3a28150", size = 7217400, upload-time = "2026-04-01T14:45:20.529Z" }, + { url = "https://files.pythonhosted.org/packages/d6/94/220e46c73065c3e2951bb91c11a1fb636c8c9ad427ac3ce7d7f3359b9b2f/pillow-12.2.0-cp314-cp314-win_arm64.whl", hash = "sha256:b1c1fbd8a5a1af3412a0810d060a78b5136ec0836c8a4ef9aa11807f2a22f4e1", size = 2554835, upload-time = "2026-04-01T14:45:23.162Z" }, + { url = "https://files.pythonhosted.org/packages/b6/ab/1b426a3974cb0e7da5c29ccff4807871d48110933a57207b5a676cccc155/pillow-12.2.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:57850958fe9c751670e49b2cecf6294acc99e562531f4bd317fa5ddee2068463", size = 5314225, upload-time = "2026-04-01T14:45:25.637Z" }, + { url = "https://files.pythonhosted.org/packages/19/1e/dce46f371be2438eecfee2a1960ee2a243bbe5e961890146d2dee1ff0f12/pillow-12.2.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:d5d38f1411c0ed9f97bcb49b7bd59b6b7c314e0e27420e34d99d844b9ce3b6f3", size = 4698541, upload-time = "2026-04-01T14:45:28.355Z" }, + { url = "https://files.pythonhosted.org/packages/55/c3/7fbecf70adb3a0c33b77a300dc52e424dc22ad8cdc06557a2e49523b703d/pillow-12.2.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5c0a9f29ca8e79f09de89293f82fc9b0270bb4af1d58bc98f540cc4aedf03166", size = 6322251, upload-time = "2026-04-01T14:45:30.924Z" }, + { url = "https://files.pythonhosted.org/packages/1c/3c/7fbc17cfb7e4fe0ef1642e0abc17fc6c94c9f7a16be41498e12e2ba60408/pillow-12.2.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1610dd6c61621ae1cf811bef44d77e149ce3f7b95afe66a4512f8c59f25d9ebe", size = 8127807, upload-time = "2026-04-01T14:45:33.908Z" }, + { url = "https://files.pythonhosted.org/packages/ff/c3/a8ae14d6defd2e448493ff512fae903b1e9bd40b72efb6ec55ce0048c8ce/pillow-12.2.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0a34329707af4f73cf1782a36cd2289c0368880654a2c11f027bcee9052d35dd", size = 6433935, upload-time = "2026-04-01T14:45:36.623Z" }, + { url = "https://files.pythonhosted.org/packages/6e/32/2880fb3a074847ac159d8f902cb43278a61e85f681661e7419e6596803ed/pillow-12.2.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8e9c4f5b3c546fa3458a29ab22646c1c6c787ea8f5ef51300e5a60300736905e", size = 7116720, upload-time = "2026-04-01T14:45:39.258Z" }, + { url = "https://files.pythonhosted.org/packages/46/87/495cc9c30e0129501643f24d320076f4cc54f718341df18cc70ec94c44e1/pillow-12.2.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:fb043ee2f06b41473269765c2feae53fc2e2fbf96e5e22ca94fb5ad677856f06", size = 6540498, upload-time = "2026-04-01T14:45:41.879Z" }, + { url = "https://files.pythonhosted.org/packages/18/53/773f5edca692009d883a72211b60fdaf8871cbef075eaa9d577f0a2f989e/pillow-12.2.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f278f034eb75b4e8a13a54a876cc4a5ab39173d2cdd93a638e1b467fc545ac43", size = 7239413, upload-time = "2026-04-01T14:45:44.705Z" }, + { url = "https://files.pythonhosted.org/packages/c9/e4/4b64a97d71b2a83158134abbb2f5bd3f8a2ea691361282f010998f339ec7/pillow-12.2.0-cp314-cp314t-win32.whl", hash = "sha256:6bb77b2dcb06b20f9f4b4a8454caa581cd4dd0643a08bacf821216a16d9c8354", size = 6482084, upload-time = "2026-04-01T14:45:47.568Z" }, + { url = "https://files.pythonhosted.org/packages/ba/13/306d275efd3a3453f72114b7431c877d10b1154014c1ebbedd067770d629/pillow-12.2.0-cp314-cp314t-win_amd64.whl", hash = "sha256:6562ace0d3fb5f20ed7290f1f929cae41b25ae29528f2af1722966a0a02e2aa1", size = 7225152, upload-time = "2026-04-01T14:45:50.032Z" }, + { url = "https://files.pythonhosted.org/packages/ff/6e/cf826fae916b8658848d7b9f38d88da6396895c676e8086fc0988073aaf8/pillow-12.2.0-cp314-cp314t-win_arm64.whl", hash = "sha256:aa88ccfe4e32d362816319ed727a004423aab09c5cea43c01a4b435643fa34eb", size = 2556579, upload-time = "2026-04-01T14:45:52.529Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.9.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9f/4a/0883b8e3802965322523f0b200ecf33d31f10991d0401162f4b23c698b42/platformdirs-4.9.6.tar.gz", hash = "sha256:3bfa75b0ad0db84096ae777218481852c0ebc6c727b3168c1b9e0118e458cf0a", size = 29400, upload-time = "2026-04-09T00:04:10.812Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/75/a6/a0a304dc33b49145b21f4808d763822111e67d1c3a32b524a1baf947b6e1/platformdirs-4.9.6-py3-none-any.whl", hash = "sha256:e61adb1d5e5cb3441b4b7710bea7e4c12250ca49439228cc1021c00dcfac0917", size = 21348, upload-time = "2026-04-09T00:04:09.463Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "protobuf" +version = "7.34.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/6b/6b/a0e95cad1ad7cc3f2c6821fcab91671bd5b78bd42afb357bb4765f29bc41/protobuf-7.34.1.tar.gz", hash = "sha256:9ce42245e704cc5027be797c1db1eb93184d44d1cdd71811fb2d9b25ad541280", size = 454708, upload-time = "2026-03-20T17:34:47.036Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/11/3325d41e6ee15bf1125654301211247b042563bcc898784351252549a8ad/protobuf-7.34.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:d8b2cc79c4d8f62b293ad9b11ec3aebce9af481fa73e64556969f7345ebf9fc7", size = 429247, upload-time = "2026-03-20T17:34:37.024Z" }, + { url = "https://files.pythonhosted.org/packages/eb/9d/aa69df2724ff63efa6f72307b483ce0827f4347cc6d6df24b59e26659fef/protobuf-7.34.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:5185e0e948d07abe94bb76ec9b8416b604cfe5da6f871d67aad30cbf24c3110b", size = 325753, upload-time = "2026-03-20T17:34:38.751Z" }, + { url = "https://files.pythonhosted.org/packages/92/e8/d174c91fd48e50101943f042b09af9029064810b734e4160bbe282fa1caa/protobuf-7.34.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:403b093a6e28a960372b44e5eb081775c9b056e816a8029c61231743d63f881a", size = 340198, upload-time = "2026-03-20T17:34:39.871Z" }, + { url = "https://files.pythonhosted.org/packages/53/1b/3b431694a4dc6d37b9f653f0c64b0a0d9ec074ee810710c0c3da21d67ba7/protobuf-7.34.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:8ff40ce8cd688f7265326b38d5a1bed9bfdf5e6723d49961432f83e21d5713e4", size = 324267, upload-time = "2026-03-20T17:34:41.1Z" }, + { url = "https://files.pythonhosted.org/packages/85/29/64de04a0ac142fb685fd09999bc3d337943fb386f3a0ec57f92fd8203f97/protobuf-7.34.1-cp310-abi3-win32.whl", hash = "sha256:34b84ce27680df7cca9f231043ada0daa55d0c44a2ddfaa58ec1d0d89d8bf60a", size = 426628, upload-time = "2026-03-20T17:34:42.536Z" }, + { url = "https://files.pythonhosted.org/packages/4d/87/cb5e585192a22b8bd457df5a2c16a75ea0db9674c3a0a39fc9347d84e075/protobuf-7.34.1-cp310-abi3-win_amd64.whl", hash = "sha256:e97b55646e6ce5cbb0954a8c28cd39a5869b59090dfaa7df4598a7fba869468c", size = 437901, upload-time = "2026-03-20T17:34:44.112Z" }, + { url = "https://files.pythonhosted.org/packages/88/95/608f665226bca68b736b79e457fded9a2a38c4f4379a4a7614303d9db3bc/protobuf-7.34.1-py3-none-any.whl", hash = "sha256:bb3812cd53aefea2b028ef42bd780f5b96407247f20c6ef7c679807e9d188f11", size = 170715, upload-time = "2026-03-20T17:34:45.384Z" }, +] + +[[package]] +name = "pulp" +version = "3.3.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/a8/6e63330798761e0c903091786681651168fda7365f029b8b5d66160861f2/pulp-3.3.1.tar.gz", hash = "sha256:a9ec237a56981b11c2096e8ba6bb72006833410ba5b400aa257426f85df5e293", size = 16304830, upload-time = "2026-05-05T12:25:43.123Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9f/23/5a77fe2b50d962213338ae0fdd9832960186ebc423388fff1a56680e5114/pulp-3.3.1-py3-none-any.whl", hash = "sha256:45aa73db3368eb13b156564e092784c8fa0c1feefa64c2afb0410d9dc0bb5cd9", size = 16390866, upload-time = "2026-05-05T12:25:39.83Z" }, +] + +[[package]] +name = "pyarrow" +version = "24.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/91/13/13e1069b351bdc3881266e11147ffccf687505dbb0ea74036237f5d454a5/pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83", size = 1180261, upload-time = "2026-04-21T10:51:25.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/a9/9686d9f07837f91f775e8932659192e02c74f9d8920524b480b85212cc68/pyarrow-24.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:6233c9ed9ab9d1db47de57d9753256d9dcffbf42db341576099f0fd9f6bf4810", size = 34981559, upload-time = "2026-04-21T10:47:22.17Z" }, + { url = "https://files.pythonhosted.org/packages/80/b6/0ddf0e9b6ead3474ab087ae598c76b031fc45532bf6a63f3a553440fb258/pyarrow-24.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:f7616236ec1bc2b15bfdec22a71ab38851c86f8f05ff64f379e1278cf20c634a", size = 36663654, upload-time = "2026-04-21T10:47:28.315Z" }, + { url = "https://files.pythonhosted.org/packages/7c/3b/926382efe8ce27ba729071d3566ade6dfb86bdf112f366000196b2f5780a/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:1617043b99bd33e5318ae18eb2919af09c71322ef1ca46566cdafc6e6712fb66", size = 45679394, upload-time = "2026-04-21T10:47:34.821Z" }, + { url = "https://files.pythonhosted.org/packages/b3/7a/829f7d9dfd37c207206081d6dad474d81dde29952401f07f2ba507814818/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6165461f55ef6314f026de6638d661188e3455d3ec49834556a0ebbdbace18bb", size = 48863122, upload-time = "2026-04-21T10:47:42.056Z" }, + { url = "https://files.pythonhosted.org/packages/5f/e8/f88ce625fe8babaae64e8db2d417c7653adb3019b08aae85c5ed787dc816/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3b13dedfe76a0ad2d1d859b0811b53827a4e9d93a0bcb05cf59333ab4980cc7e", size = 49376032, upload-time = "2026-04-21T10:47:48.967Z" }, + { url = "https://files.pythonhosted.org/packages/36/7a/82c363caa145fff88fb475da50d3bf52bb024f61917be5424c3392eaf878/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:25ea65d868eb04015cd18e6df2fbe98f07e5bda2abefabcb88fce39a947716f6", size = 51929490, upload-time = "2026-04-21T10:47:55.981Z" }, + { url = "https://files.pythonhosted.org/packages/66/1c/e3e72c8014ad2743ca64a701652c733cc5cbcee15c0463a32a8c55518d9e/pyarrow-24.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:295f0a7f2e242dabd513737cf076007dc5b2d59237e3eca37b05c0c6446f3826", size = 27355660, upload-time = "2026-04-21T10:48:01.718Z" }, + { url = "https://files.pythonhosted.org/packages/6f/d3/a1abf004482026ddc17f4503db227787fa3cfe41ec5091ff20e4fea55e57/pyarrow-24.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:02b001b3ed4723caa44f6cd1af2d5c86aa2cf9971dacc2ffa55b21237713dfba", size = 34976759, upload-time = "2026-04-21T10:48:07.258Z" }, + { url = "https://files.pythonhosted.org/packages/4f/4a/34f0a36d28a2dd32225301b79daad44e243dc1a2bb77d43b60749be255c4/pyarrow-24.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:04920d6a71aabd08a0417709efce97d45ea8e6fb733d9ca9ecffb13c67839f68", size = 36658471, upload-time = "2026-04-21T10:48:13.347Z" }, + { url = "https://files.pythonhosted.org/packages/1f/78/543b94712ae8bb1a6023bcc1acf1a740fbff8286747c289cd9468fced2a5/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:a964266397740257f16f7bb2e4f08a0c81454004beab8ff59dd531b73610e9f2", size = 45675981, upload-time = "2026-04-21T10:48:20.201Z" }, + { url = "https://files.pythonhosted.org/packages/84/9f/8fb7c222b100d314137fa40ec050de56cd8c6d957d1cfff685ce72f15b17/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6f066b179d68c413374294bc1735f68475457c933258df594443bb9d88ddc2a0", size = 48859172, upload-time = "2026-04-21T10:48:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a7/d3/1ea72538e6c8b3b475ed78d1049a2c518e655761ea50fe1171fc855fcab7/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1183baeb14c5f587b1ec52831e665718ce632caab84b7cd6b85fd44f96114495", size = 49385733, upload-time = "2026-04-21T10:48:34.7Z" }, + { url = "https://files.pythonhosted.org/packages/c3/be/c3d8b06a1ba35f2260f8e1f771abbee7d5e345c0937aab90675706b1690a/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:806f24b4085453c197a5078218d1ee08783ebbba271badd153d1ae22a3ee804f", size = 51934335, upload-time = "2026-04-21T10:48:42.099Z" }, + { url = "https://files.pythonhosted.org/packages/9c/62/89e07a1e7329d2cde3e3c6994ba0839a24977a2beda8be6005ea3d860b99/pyarrow-24.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:e4505fc6583f7b05ab854934896bcac8253b04ac1171a77dfb73efef92076d91", size = 27271748, upload-time = "2026-04-21T10:49:42.532Z" }, + { url = "https://files.pythonhosted.org/packages/17/1a/cff3a59f80b5b1658549d46611b67163f65e0664431c076ad728bf9d5af4/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:1a4e45017efbf115032e4475ee876d525e0e36c742214fbe405332480ecd6275", size = 35238554, upload-time = "2026-04-21T10:48:48.526Z" }, + { url = "https://files.pythonhosted.org/packages/a8/99/cce0f42a327bfef2c420fb6078a3eb834826e5d6697bf3009fe11d2ad051/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:7986f1fa71cee060ad00758bcc79d3a93bab8559bf978fab9e53472a2e25a17b", size = 36782301, upload-time = "2026-04-21T10:48:55.181Z" }, + { url = "https://files.pythonhosted.org/packages/2a/66/8e560d5ff6793ca29aca213c53eec0dd482dd46cb93b2819e5aab52e4252/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:d3e0b61e8efb24ed38898e5cdc5fffa9124be480008d401a1f8071500494ae42", size = 45721929, upload-time = "2026-04-21T10:49:03.676Z" }, + { url = "https://files.pythonhosted.org/packages/27/0c/a26e25505d030716e078d9f16eb74973cbf0b33b672884e9f9da1c83b871/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:55a3bc1e3df3b5567b7d27ef551b2283f0c68a5e86f1cd56abc569da4f31335b", size = 48825365, upload-time = "2026-04-21T10:49:11.714Z" }, + { url = "https://files.pythonhosted.org/packages/5f/eb/771f9ecb0c65e73fe9dccdd1717901b9594f08c4515d000c7c62df573811/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:641f795b361874ac9da5294f8f443dfdbee355cf2bd9e3b8d97aaac2306b9b37", size = 49451819, upload-time = "2026-04-21T10:49:21.474Z" }, + { url = "https://files.pythonhosted.org/packages/48/da/61ae89a88732f5a785646f3ec6125dbb640fa98a540eb2b9889caa561403/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8adc8e6ce5fccf5dc707046ae4914fd537def529709cc0d285d37a7f9cd442ca", size = 51909252, upload-time = "2026-04-21T10:49:31.164Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1a/8dd5cafab7b66573fa91c03d06d213356ad4edd71813aa75e08ce2b3a844/pyarrow-24.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:9b18371ad2f44044b81a8d23bc2d8a9b6a6226dca775e8e16cfee640473d6c5d", size = 27388127, upload-time = "2026-04-21T10:49:37.334Z" }, + { url = "https://files.pythonhosted.org/packages/ad/80/d022a34ff05d2cbedd8ccf841fc1f532ecfa9eb5ed1711b56d0e0ea71fc9/pyarrow-24.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:1cc9057f0319e26333b357e17f3c2c022f1a83739b48a88b25bfd5fa2dc18838", size = 35007997, upload-time = "2026-04-21T10:49:48.796Z" }, + { url = "https://files.pythonhosted.org/packages/1a/ff/f01485fda6f4e5d441afb8dd5e7681e4db18826c1e271852f5d3957d6a80/pyarrow-24.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e6f1278ee4785b6db21229374a1c9e54ec7c549de5d1efc9630b6207de7e170b", size = 36678720, upload-time = "2026-04-21T10:49:55.858Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c2/2d2d5fea814237923f71b36495211f20b43a1576f9a4d6da7e751a64ec6f/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:adbbedc55506cbdabb830890444fb856bfb0060c46c6f8026c6c2f2cf86ae795", size = 45741852, upload-time = "2026-04-21T10:50:04.624Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3a/28ba9c1c1ebdbb5f1b94dfebb46f207e52e6a554b7fe4132540fde29a3a0/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ae8a1145af31d903fa9bb166824d7abe9b4681a000b0159c9fb99c11bc11ad26", size = 48889852, upload-time = "2026-04-21T10:50:12.293Z" }, + { url = "https://files.pythonhosted.org/packages/df/51/4a389acfd31dca009f8fb82d7f510bb4130f2b3a8e18cf00194d0687d8ac/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d7027eba1df3b2069e2e8d80f644fa0918b68c46432af3d088ddd390d063ecde", size = 49445207, upload-time = "2026-04-21T10:50:20.677Z" }, + { url = "https://files.pythonhosted.org/packages/19/4b/0bab2b23d2ae901b1b9a03c0efd4b2d070256f8ce3fc43f6e58c167b2081/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e56a1ffe9bf7b727432b89104cc0849c21582949dd7bdcb34f17b2001a351a76", size = 51954117, upload-time = "2026-04-21T10:50:29.14Z" }, + { url = "https://files.pythonhosted.org/packages/29/88/f4e9145da0417b3d2c12035a8492b35ff4a3dbc653e614fcfb51d9dedb38/pyarrow-24.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:38be1808cdd068605b787e6ca9119b27eb275a0234e50212c3492331680c3b1e", size = 28001155, upload-time = "2026-04-21T10:51:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/79/4f/46a49a63f43526da895b1a45bbb51d5baf8e4d77159f8528fc3e5490007f/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:418e48ce50a45a6a6c73c454677203a9c75c966cb1e92ca3370959185f197a05", size = 35250387, upload-time = "2026-04-21T10:50:35.552Z" }, + { url = "https://files.pythonhosted.org/packages/a0/da/d5e0cd5ef00796922404806d5f00325cdadc3441ce2c13fe7115f2df9a64/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:2f16197705a230a78270cdd4ea8a1d57e86b2fdcbc34a1f6aebc72e65c986f9a", size = 36797102, upload-time = "2026-04-21T10:50:42.417Z" }, + { url = "https://files.pythonhosted.org/packages/34/c7/5904145b0a593a05236c882933d439b5720f0a145381179063722fbfc123/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:fb24ac194bfc5e86839d7dcd52092ee31e5fe6733fe11f5e3b06ef0812b20072", size = 45745118, upload-time = "2026-04-21T10:50:49.324Z" }, + { url = "https://files.pythonhosted.org/packages/13/d3/cca42fe166d1c6e4d5b80e530b7949104d10e17508a90ae202dac205ce2a/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:9700ebd9a51f5895ce75ff4ac4b3c47a7d4b42bc618be8e713e5d56bacf5f931", size = 48844765, upload-time = "2026-04-21T10:50:55.579Z" }, + { url = "https://files.pythonhosted.org/packages/b0/49/942c3b79878ba928324d1e17c274ed84581db8c0a749b24bcf4cbdf15bd3/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d8ddd2768da81d3ee08cfea9b597f4abb4e8e1dc8ae7e204b608d23a0d3ab699", size = 49471890, upload-time = "2026-04-21T10:51:02.439Z" }, + { url = "https://files.pythonhosted.org/packages/76/97/ff71431000a75d84135a1ace5ca4ba11726a231a8007bbb320a4c54075d5/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:61a3d7eaa97a14768b542f3d284dc6400dd2470d9f080708b13cd46b6ae18136", size = 51932250, upload-time = "2026-04-21T10:51:10.576Z" }, + { url = "https://files.pythonhosted.org/packages/51/be/6f79d55816d5c22557cf27533543d5d70dfe692adfbee4b99f2760674f38/pyarrow-24.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:c91d00057f23b8d353039520dc3a6c09d8608164c692e9f59a175a42b2ae0c19", size = 28131282, upload-time = "2026-04-21T10:51:16.815Z" }, +] + +[[package]] +name = "pydantic" +version = "2.13.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-types" }, + { name = "pydantic-core" }, + { name = "typing-extensions" }, + { name = "typing-inspection" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/18/a5/b60d21ac674192f8ab0ba4e9fd860690f9b4a6e51ca5df118733b487d8d6/pydantic-2.13.4.tar.gz", hash = "sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6", size = 844775, upload-time = "2026-05-06T13:43:05.343Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fd/7b/122376b1fd3c62c1ed9dc80c931ace4844b3c55407b6fb2d199377c9736f/pydantic-2.13.4-py3-none-any.whl", hash = "sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba", size = 472262, upload-time = "2026-05-06T13:43:02.641Z" }, +] + +[[package]] +name = "pydantic-core" +version = "2.46.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9d/56/921726b776ace8d8f5db44c4ef961006580d91dc52b803c489fafd1aa249/pydantic_core-2.46.4.tar.gz", hash = "sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1", size = 471464, upload-time = "2026-05-06T13:37:06.98Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/8c/af022f0af448d7747c5154288d46b5f2bc5f17366eaa0e23e9aa04d59f3b/pydantic_core-2.46.4-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2", size = 2106158, upload-time = "2026-05-06T13:38:57.215Z" }, + { url = "https://files.pythonhosted.org/packages/19/95/6195171e385007300f0f5574592e467c568becce2d937a0b6804f218bc49/pydantic_core-2.46.4-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f", size = 1951724, upload-time = "2026-05-06T13:37:02.697Z" }, + { url = "https://files.pythonhosted.org/packages/8e/bc/f47d1ff9cbb1620e1b5b697eef06010035735f07820180e74178226b27b3/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7", size = 1975742, upload-time = "2026-05-06T13:37:09.448Z" }, + { url = "https://files.pythonhosted.org/packages/5b/11/9b9a5b0306345664a2da6410877af6e8082481b5884b3ddd78d47c6013ce/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7", size = 2052418, upload-time = "2026-05-06T13:37:38.234Z" }, + { url = "https://files.pythonhosted.org/packages/f1/b7/a65fec226f5d78fc39f4a13c4cc0c768c22b113438f60c14adc9d2865038/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712", size = 2232274, upload-time = "2026-05-06T13:38:27.753Z" }, + { url = "https://files.pythonhosted.org/packages/68/f0/92039db98b907ef49269a8271f67db9cb78ae2fc68062ef7e4e77adb5f61/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4", size = 2309940, upload-time = "2026-05-06T13:38:05.353Z" }, + { url = "https://files.pythonhosted.org/packages/5f/97/2aab507d3d00ca626e8e57c1eac6a79e4e5fbcc63eb99733ff55d1717f65/pydantic_core-2.46.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce", size = 2094516, upload-time = "2026-05-06T13:39:10.577Z" }, + { url = "https://files.pythonhosted.org/packages/22/37/a8aca44d40d737dde2bc05b3c6c07dff0de07ce6f82e9f3167aeaf4d5dea/pydantic_core-2.46.4-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987", size = 2136854, upload-time = "2026-05-06T13:40:22.59Z" }, + { url = "https://files.pythonhosted.org/packages/24/99/fcef1b79238c06a8cbec70819ac722ba76e02bc8ada9b0fd66eba40da01b/pydantic_core-2.46.4-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b", size = 2180306, upload-time = "2026-05-06T13:40:10.666Z" }, + { url = "https://files.pythonhosted.org/packages/ae/6c/fc44000918855b42779d007ae63b0532794739027b2f417321cddbc44f6a/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458", size = 2190044, upload-time = "2026-05-06T13:40:43.231Z" }, + { url = "https://files.pythonhosted.org/packages/6b/65/d9cadc9f1920d7a127ad2edba16c1db7916e59719285cd6c94600b0080ba/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_armv7l.whl", hash = "sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b", size = 2329133, upload-time = "2026-05-06T13:39:57.365Z" }, + { url = "https://files.pythonhosted.org/packages/d0/cf/c873d91679f3a30bcf5e7ac280ce5573483e72295307685120d0d5ad3416/pydantic_core-2.46.4-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c", size = 2374464, upload-time = "2026-05-06T13:38:06.976Z" }, + { url = "https://files.pythonhosted.org/packages/47/bd/6f2fc8188f31bf10590f1e98e7b306336161fac930a8c514cd7bd828c7dc/pydantic_core-2.46.4-cp312-cp312-win32.whl", hash = "sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894", size = 1974823, upload-time = "2026-05-06T13:40:47.985Z" }, + { url = "https://files.pythonhosted.org/packages/40/8c/985c1d41ea1107c2534abd9870e4ed5c8e7669b5c308297835c001e7a1c4/pydantic_core-2.46.4-cp312-cp312-win_amd64.whl", hash = "sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89", size = 2072919, upload-time = "2026-05-06T13:39:21.153Z" }, + { url = "https://files.pythonhosted.org/packages/c4/ba/f463d006e0c47373ca7ec5e1a261c59dc01ef4d62b2657af925fb0deee3a/pydantic_core-2.46.4-cp312-cp312-win_arm64.whl", hash = "sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a", size = 2027604, upload-time = "2026-05-06T13:39:03.753Z" }, + { url = "https://files.pythonhosted.org/packages/51/a2/5d30b469c5267a17b39dec53208222f76a8d351dfac4af661888c5aee77d/pydantic_core-2.46.4-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008", size = 2106306, upload-time = "2026-05-06T13:37:48.029Z" }, + { url = "https://files.pythonhosted.org/packages/c1/81/4fa520eaffa8bd7d1525e644cd6d39e7d60b1592bc5b516693c7340b50f1/pydantic_core-2.46.4-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4", size = 1951906, upload-time = "2026-05-06T13:37:17.012Z" }, + { url = "https://files.pythonhosted.org/packages/03/d5/fd02da45b659668b05923b17ba3a0100a0a3d5541e3bd8fcc4ecb711309e/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76", size = 1976802, upload-time = "2026-05-06T13:37:35.113Z" }, + { url = "https://files.pythonhosted.org/packages/21/f2/95727e1368be3d3ed485eaab7adbd7dda408f33f7a36e8b48e0144002b91/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3", size = 2052446, upload-time = "2026-05-06T13:37:12.313Z" }, + { url = "https://files.pythonhosted.org/packages/9c/86/5d99feea3f77c7234b8718075b23db11532773c1a0dbd9b9490215dc2eeb/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76", size = 2232757, upload-time = "2026-05-06T13:39:01.149Z" }, + { url = "https://files.pythonhosted.org/packages/d2/3a/508ac615935ef7588cf6d9e9b91309fdc2da751af865e02a9098de88258c/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4", size = 2309275, upload-time = "2026-05-06T13:37:41.406Z" }, + { url = "https://files.pythonhosted.org/packages/07/f8/41db9de19d7987d6b04715a02b3b40aea467000275d9d758ffaa31af7d50/pydantic_core-2.46.4-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a", size = 2094467, upload-time = "2026-05-06T13:39:18.847Z" }, + { url = "https://files.pythonhosted.org/packages/2c/e2/f35033184cb11d0052daf4416e8e10a502ea2ac006fc4f459aee872727d1/pydantic_core-2.46.4-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262", size = 2134417, upload-time = "2026-05-06T13:40:17.944Z" }, + { url = "https://files.pythonhosted.org/packages/7e/7b/6ceeb1cc90e193862f444ebe373d8fdf613f0a82572dde03fb10734c6c71/pydantic_core-2.46.4-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e", size = 2179782, upload-time = "2026-05-06T13:40:32.618Z" }, + { url = "https://files.pythonhosted.org/packages/5a/f2/c8d7773ede6af08036423a00ae0ceffce266c3c52a096c435d68c896083f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd", size = 2188782, upload-time = "2026-05-06T13:36:51.018Z" }, + { url = "https://files.pythonhosted.org/packages/59/31/0c864784e31f09f05cdd87606f08923b9c9e7f6e51dd27f20f62f975ce9f/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_armv7l.whl", hash = "sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be", size = 2328334, upload-time = "2026-05-06T13:40:37.764Z" }, + { url = "https://files.pythonhosted.org/packages/c2/eb/4f6c8a41efa30baa755590f4141abf3a8c370fab610915733e74134a7270/pydantic_core-2.46.4-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d", size = 2372986, upload-time = "2026-05-06T13:39:34.152Z" }, + { url = "https://files.pythonhosted.org/packages/5b/24/b375a480d53113860c299764bfe9f349a3dc9108b3adc0d7f0d786492ebf/pydantic_core-2.46.4-cp313-cp313-win32.whl", hash = "sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb", size = 1973693, upload-time = "2026-05-06T13:37:55.072Z" }, + { url = "https://files.pythonhosted.org/packages/7e/e8/cff247591966f2d22ec8c003cd7587e27b7ba7b81ab2fb888e3ab75dc285/pydantic_core-2.46.4-cp313-cp313-win_amd64.whl", hash = "sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292", size = 2071819, upload-time = "2026-05-06T13:38:49.139Z" }, + { url = "https://files.pythonhosted.org/packages/c6/1a/f4aee670d5670e9e148e0c82c7db98d780be566c6e6a97ee8035528ca0b3/pydantic_core-2.46.4-cp313-cp313-win_arm64.whl", hash = "sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d", size = 2027411, upload-time = "2026-05-06T13:40:45.796Z" }, + { url = "https://files.pythonhosted.org/packages/8d/74/228a26ddad29c6672b805d9fd78e8d251cd04004fa7eed0e622096cd0250/pydantic_core-2.46.4-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb", size = 2102079, upload-time = "2026-05-06T13:38:41.019Z" }, + { url = "https://files.pythonhosted.org/packages/ad/1f/8970b150a4b4365623ae00fc88603491f763c627311ae8031e3111356d6e/pydantic_core-2.46.4-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462", size = 1952179, upload-time = "2026-05-06T13:36:59.812Z" }, + { url = "https://files.pythonhosted.org/packages/95/30/5211a831ae054928054b2f79731661087a2bc5c01e825c672b3a4a8f1b3e/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9", size = 1978926, upload-time = "2026-05-06T13:37:39.933Z" }, + { url = "https://files.pythonhosted.org/packages/57/e9/689668733b1eb67adeef047db3c2e8788fcf65a7fd9c9e2b46b7744fe245/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4", size = 2046785, upload-time = "2026-05-06T13:38:01.995Z" }, + { url = "https://files.pythonhosted.org/packages/60/d9/6715260422ff50a2109878fd24d948a6c3446bb2664f34ee78cd972b3acd/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914", size = 2228733, upload-time = "2026-05-06T13:40:50.371Z" }, + { url = "https://files.pythonhosted.org/packages/18/ae/fdb2f64316afca925640f8e70bb1a564b0ec2721c1389e25b8eb4bf9a299/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28", size = 2307534, upload-time = "2026-05-06T13:37:21.531Z" }, + { url = "https://files.pythonhosted.org/packages/89/1d/8eff589b45bb8190a9d12c49cfad0f176a5cbd1534908a6b5125e2886239/pydantic_core-2.46.4-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b", size = 2099732, upload-time = "2026-05-06T13:39:31.942Z" }, + { url = "https://files.pythonhosted.org/packages/06/d5/ee5a3366637fee41dee51a1fc91562dcf12ddbc68fda34e6b253da2324bb/pydantic_core-2.46.4-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c", size = 2129627, upload-time = "2026-05-06T13:37:25.033Z" }, + { url = "https://files.pythonhosted.org/packages/94/33/2414be571d2c6a6c4d08be21f9292b6d3fdb08949a97b6dfe985017821db/pydantic_core-2.46.4-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb", size = 2179141, upload-time = "2026-05-06T13:37:14.046Z" }, + { url = "https://files.pythonhosted.org/packages/7b/79/7daa95be995be0eecc4cf75064cb33f9bbbfe3fe0158caf2f0d4a996a5c7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_aarch64.whl", hash = "sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898", size = 2184325, upload-time = "2026-05-06T13:36:53.615Z" }, + { url = "https://files.pythonhosted.org/packages/9f/cb/d0a382f5c0de8a222dc61c65348e0ce831b1f68e0a018450d31c2cace3a5/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_armv7l.whl", hash = "sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e", size = 2323990, upload-time = "2026-05-06T13:40:29.971Z" }, + { url = "https://files.pythonhosted.org/packages/05/db/d9ba624cc4a5aced1598e88c04fdbd8310c8a69b9d38b9a3d39ce3a61ed7/pydantic_core-2.46.4-cp314-cp314-musllinux_1_1_x86_64.whl", hash = "sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519", size = 2369978, upload-time = "2026-05-06T13:37:23.027Z" }, + { url = "https://files.pythonhosted.org/packages/f2/20/d15df15ba918c423461905802bfd2981c3af0bfa0e40d05e13edbfa48bc3/pydantic_core-2.46.4-cp314-cp314-win32.whl", hash = "sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4", size = 1966354, upload-time = "2026-05-06T13:38:03.499Z" }, + { url = "https://files.pythonhosted.org/packages/fc/b6/6b8de4c0a7d7ab3004c439c80c5c1e0a3e8d78bbae19379b01960383d9e5/pydantic_core-2.46.4-cp314-cp314-win_amd64.whl", hash = "sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac", size = 2072238, upload-time = "2026-05-06T13:39:40.807Z" }, + { url = "https://files.pythonhosted.org/packages/32/36/51eb763beec1f4cf59b1db243a7dcc39cbb41230f050a09b9d69faaf0a48/pydantic_core-2.46.4-cp314-cp314-win_arm64.whl", hash = "sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a", size = 2018251, upload-time = "2026-05-06T13:37:26.72Z" }, + { url = "https://files.pythonhosted.org/packages/e8/91/855af51d625b23aa987116a19e231d2aaef9c4a415273ddc189b79a45fee/pydantic_core-2.46.4-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0", size = 2099593, upload-time = "2026-05-06T13:39:47.682Z" }, + { url = "https://files.pythonhosted.org/packages/fb/1b/8784a54c65edb5f49f0a14d6977cf1b209bba85a4c77445b255c2de58ab3/pydantic_core-2.46.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d", size = 1935226, upload-time = "2026-05-06T13:40:40.428Z" }, + { url = "https://files.pythonhosted.org/packages/e8/e7/1955d28d1afc56dd4b3ad7cc0cf39df1b9852964cf16e5d13912756d6d6b/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b", size = 1974605, upload-time = "2026-05-06T13:37:32.029Z" }, + { url = "https://files.pythonhosted.org/packages/93/e2/3fedbf0ba7a22850e6e9fd78117f1c0f10f950182344d8a6c535d468fdd8/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000", size = 2030777, upload-time = "2026-05-06T13:38:55.239Z" }, + { url = "https://files.pythonhosted.org/packages/f8/61/46be275fcaaba0b4f5b9669dd852267ce1ff616592dccf7a7845588df091/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e", size = 2236641, upload-time = "2026-05-06T13:37:08.096Z" }, + { url = "https://files.pythonhosted.org/packages/60/db/12e93e46a8bac9988be3c016860f83293daea8c716c029c9ace279036f2f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd", size = 2286404, upload-time = "2026-05-06T13:40:20.221Z" }, + { url = "https://files.pythonhosted.org/packages/e2/4a/4d8b19008f38d31c53b8219cfedc2e3d5de5fe99d90076b7e767de29274f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3", size = 2109219, upload-time = "2026-05-06T13:38:12.153Z" }, + { url = "https://files.pythonhosted.org/packages/88/70/3cbc40978fefb7bb09c6708d40d4ad1a5d70fd7213c3d17f971de868ec1f/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7", size = 2110594, upload-time = "2026-05-06T13:40:02.971Z" }, + { url = "https://files.pythonhosted.org/packages/9d/20/b8d36736216e29491125531685b2f9e61aa5b4b2599893f8268551da3338/pydantic_core-2.46.4-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff", size = 2159542, upload-time = "2026-05-06T13:39:27.506Z" }, + { url = "https://files.pythonhosted.org/packages/1d/a2/367df868eb584dacf6bf82a389272406d7178e301c4ac82545ab98bc2dd9/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_aarch64.whl", hash = "sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424", size = 2168146, upload-time = "2026-05-06T13:38:31.93Z" }, + { url = "https://files.pythonhosted.org/packages/c1/b8/4460f77f7e201893f649a29ab355dddd3beee8a97bcb1a320db414f9a06e/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_armv7l.whl", hash = "sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6", size = 2306309, upload-time = "2026-05-06T13:37:44.717Z" }, + { url = "https://files.pythonhosted.org/packages/64/c4/be2639293acd87dc8ddbcec41a73cee9b2ebf996fe6d892a1a74e88ad3f7/pydantic_core-2.46.4-cp314-cp314t-musllinux_1_1_x86_64.whl", hash = "sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565", size = 2369736, upload-time = "2026-05-06T13:37:05.645Z" }, + { url = "https://files.pythonhosted.org/packages/30/a6/9f9f380dbb301f67023bf8f707aaa75daadf84f7152d95c410fd7e81d994/pydantic_core-2.46.4-cp314-cp314t-win32.whl", hash = "sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02", size = 1955575, upload-time = "2026-05-06T13:38:51.116Z" }, + { url = "https://files.pythonhosted.org/packages/40/1f/f1eb9eb350e795d1af8586289746f5c5677d16043040d63710e22abc43c9/pydantic_core-2.46.4-cp314-cp314t-win_amd64.whl", hash = "sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5", size = 2051624, upload-time = "2026-05-06T13:38:21.672Z" }, + { url = "https://files.pythonhosted.org/packages/f6/d2/42dd53d0a85c27606f316d3aa5d2869c4e8470a5ed6dec30e4a1abe19192/pydantic_core-2.46.4-cp314-cp314t-win_arm64.whl", hash = "sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596", size = 2017325, upload-time = "2026-05-06T13:40:52.723Z" }, + { url = "https://files.pythonhosted.org/packages/9d/1d/8987ad40f65ae1432753072f214fb5c74fe47ffbd0698bb9cbbb585664f8/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_10_12_x86_64.whl", hash = "sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7", size = 2095527, upload-time = "2026-05-06T13:39:52.283Z" }, + { url = "https://files.pythonhosted.org/packages/64/d3/84c282a7eee1d3ac4c0377546ef5a1ea436ce26840d9ac3b7ed54a377507/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-macosx_11_0_arm64.whl", hash = "sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df", size = 1936024, upload-time = "2026-05-06T13:40:15.671Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ca/eac61596cdeb4d7e174d3dc0bd8a6238f14f75f97a24e7b7db4c7e7340a0/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526", size = 1990696, upload-time = "2026-05-06T13:38:34.717Z" }, + { url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" }, +] + +[[package]] +name = "pydeck" +version = "0.9.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "jinja2" }, + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/eb/df/4e9e7f20f8034a37c6571c93809f6d22388c39978c98d174d656c1a18fd2/pydeck-0.9.2.tar.gz", hash = "sha256:c10d9035e81ead6385264cac8d19402471f6866a15ca1f7df1400f52142bcf87", size = 5849672, upload-time = "2026-04-16T18:30:30.089Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/88/24/b30ee7d723100fd822de1bb4c0adea62f3419884a75a536f35f355d1e7c0/pydeck-0.9.2-py2.py3-none-any.whl", hash = "sha256:8213dfeacc5f6bfe6825f61c8ee34e3850e8a31fc43924379ec98edb34a75b25", size = 11305615, upload-time = "2026-04-16T18:30:28.133Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pymdown-extensions" +version = "10.21.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown" }, + { name = "pyyaml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" }, +] + +[[package]] +name = "pyparsing" +version = "3.3.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f3/91/9c6ee907786a473bf81c5f53cf703ba0957b23ab84c264080fb5a450416f/pyparsing-3.3.2.tar.gz", hash = "sha256:c777f4d763f140633dcb6d8a3eda953bf7a214dc4eff598413c070bcdc117cbc", size = 6851574, upload-time = "2026-01-21T03:57:59.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/bd/c038d7cc38edc1aa5bf91ab8068b63d4308c66c4c8bb3cbba7dfbc049f9c/pyparsing-3.3.2-py3-none-any.whl", hash = "sha256:850ba148bd908d7e2411587e247a1e4f0327839c40e2e5e6d05a007ecc69911d", size = 122781, upload-time = "2026-01-21T03:57:55.912Z" }, +] + +[[package]] +name = "pytest" +version = "9.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7d/0d/549bd94f1a0a402dc8cf64563a117c0f3765662e2e668477624baeec44d5/pytest-9.0.3.tar.gz", hash = "sha256:b86ada508af81d19edeb213c681b1d48246c1a91d304c6c81a427674c17eb91c", size = 1572165, upload-time = "2026-04-07T17:16:18.027Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d4/24/a372aaf5c9b7208e7112038812994107bc65a84cd00e0354a88c2c77a617/pytest-9.0.3-py3-none-any.whl", hash = "sha256:2c5efc453d45394fdd706ade797c0a81091eccd1d6e4bccfcd476e2b8e0ab5d9", size = 375249, upload-time = "2026-04-07T17:16:16.13Z" }, +] + +[[package]] +name = "pytest-cov" +version = "7.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "coverage" }, + { name = "pluggy" }, + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b1/51/a849f96e117386044471c8ec2bd6cfebacda285da9525c9106aeb28da671/pytest_cov-7.1.0.tar.gz", hash = "sha256:30674f2b5f6351aa09702a9c8c364f6a01c27aae0c1366ae8016160d1efc56b2", size = 55592, upload-time = "2026-03-21T20:11:16.284Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" }, +] + +[[package]] +name = "pytest-mock" +version = "3.15.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pytest" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/68/14/eb014d26be205d38ad5ad20d9a80f7d201472e08167f0bb4361e251084a9/pytest_mock-3.15.1.tar.gz", hash = "sha256:1849a238f6f396da19762269de72cb1814ab44416fa73a8686deac10b0d87a0f", size = 34036, upload-time = "2025-09-16T16:37:27.081Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/5a/cc/06253936f4a7fa2e0f48dfe6d851d9c56df896a9ab09ac019d70b760619c/pytest_mock-3.15.1-py3-none-any.whl", hash = "sha256:0a25e2eb88fe5168d535041d09a4529a188176ae608a6d249ee65abc0949630d", size = 10095, upload-time = "2025-09-16T16:37:25.734Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "python-dotenv" +version = "1.2.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/ed/0301aeeac3e5353ef3d94b6ec08bbcabd04a72018415dcb29e588514bba8/python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3", size = 50135, upload-time = "2026-03-01T16:00:26.196Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/d7/1959b9648791274998a9c3526f6d0ec8fd2233e4d4acce81bbae76b44b2a/python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a", size = 22101, upload-time = "2026-03-01T16:00:25.09Z" }, +] + +[[package]] +name = "python-multipart" +version = "0.0.28" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/82/54/a85eb421fbdd5007bc5af39d0f4ed9fa609e0fedbfdc2adcf0b34526870e/python_multipart-0.0.28.tar.gz", hash = "sha256:8550da197eac0f7ab748961fc9509b999fa2662ea25cef857f05249f6893c0f8", size = 45314, upload-time = "2026-05-10T11:05:16.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f3/a2/43bbc5860b5034e2af4ef99a0e04d726ff329c43e192ef3abaa8d7ecfce5/python_multipart-0.0.28-py3-none-any.whl", hash = "sha256:10faac07eb966c3f48dc415f9dee46c04cb10d58d30a35677db8027c825ed9b6", size = 29438, upload-time = "2026-05-10T11:05:15.052Z" }, +] + +[[package]] +name = "pytokens" +version = "0.4.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b6/34/b4e015b99031667a7b960f888889c5bd34ef585c85e1cb56a594b92836ac/pytokens-0.4.1.tar.gz", hash = "sha256:292052fe80923aae2260c073f822ceba21f3872ced9a68bb7953b348e561179a", size = 23015, upload-time = "2026-01-30T01:03:45.924Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/5d/e44573011401fb82e9d51e97f1290ceb377800fb4eed650b96f4753b499c/pytokens-0.4.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:140709331e846b728475786df8aeb27d24f48cbcf7bcd449f8de75cae7a45083", size = 160663, upload-time = "2026-01-30T01:03:06.473Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e6/5bbc3019f8e6f21d09c41f8b8654536117e5e211a85d89212d59cbdab381/pytokens-0.4.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d6c4268598f762bc8e91f5dbf2ab2f61f7b95bdc07953b602db879b3c8c18e1", size = 255626, upload-time = "2026-01-30T01:03:08.177Z" }, + { url = "https://files.pythonhosted.org/packages/bf/3c/2d5297d82286f6f3d92770289fd439956b201c0a4fc7e72efb9b2293758e/pytokens-0.4.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:24afde1f53d95348b5a0eb19488661147285ca4dd7ed752bbc3e1c6242a304d1", size = 269779, upload-time = "2026-01-30T01:03:09.756Z" }, + { url = "https://files.pythonhosted.org/packages/20/01/7436e9ad693cebda0551203e0bf28f7669976c60ad07d6402098208476de/pytokens-0.4.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:5ad948d085ed6c16413eb5fec6b3e02fa00dc29a2534f088d3302c47eb59adf9", size = 268076, upload-time = "2026-01-30T01:03:10.957Z" }, + { url = "https://files.pythonhosted.org/packages/2e/df/533c82a3c752ba13ae7ef238b7f8cdd272cf1475f03c63ac6cf3fcfb00b6/pytokens-0.4.1-cp312-cp312-win_amd64.whl", hash = "sha256:3f901fe783e06e48e8cbdc82d631fca8f118333798193e026a50ce1b3757ea68", size = 103552, upload-time = "2026-01-30T01:03:12.066Z" }, + { url = "https://files.pythonhosted.org/packages/cb/dc/08b1a080372afda3cceb4f3c0a7ba2bde9d6a5241f1edb02a22a019ee147/pytokens-0.4.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8bdb9d0ce90cbf99c525e75a2fa415144fd570a1ba987380190e8b786bc6ef9b", size = 160720, upload-time = "2026-01-30T01:03:13.843Z" }, + { url = "https://files.pythonhosted.org/packages/64/0c/41ea22205da480837a700e395507e6a24425151dfb7ead73343d6e2d7ffe/pytokens-0.4.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5502408cab1cb18e128570f8d598981c68a50d0cbd7c61312a90507cd3a1276f", size = 254204, upload-time = "2026-01-30T01:03:14.886Z" }, + { url = "https://files.pythonhosted.org/packages/e0/d2/afe5c7f8607018beb99971489dbb846508f1b8f351fcefc225fcf4b2adc0/pytokens-0.4.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:29d1d8fb1030af4d231789959f21821ab6325e463f0503a61d204343c9b355d1", size = 268423, upload-time = "2026-01-30T01:03:15.936Z" }, + { url = "https://files.pythonhosted.org/packages/68/d4/00ffdbd370410c04e9591da9220a68dc1693ef7499173eb3e30d06e05ed1/pytokens-0.4.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:970b08dd6b86058b6dc07efe9e98414f5102974716232d10f32ff39701e841c4", size = 266859, upload-time = "2026-01-30T01:03:17.458Z" }, + { url = "https://files.pythonhosted.org/packages/a7/c9/c3161313b4ca0c601eeefabd3d3b576edaa9afdefd32da97210700e47652/pytokens-0.4.1-cp313-cp313-win_amd64.whl", hash = "sha256:9bd7d7f544d362576be74f9d5901a22f317efc20046efe2034dced238cbbfe78", size = 103520, upload-time = "2026-01-30T01:03:18.652Z" }, + { url = "https://files.pythonhosted.org/packages/8f/a7/b470f672e6fc5fee0a01d9e75005a0e617e162381974213a945fcd274843/pytokens-0.4.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:4a14d5f5fc78ce85e426aa159489e2d5961acf0e47575e08f35584009178e321", size = 160821, upload-time = "2026-01-30T01:03:19.684Z" }, + { url = "https://files.pythonhosted.org/packages/80/98/e83a36fe8d170c911f864bfded690d2542bfcfacb9c649d11a9e6eb9dc41/pytokens-0.4.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f50fd18543be72da51dd505e2ed20d2228c74e0464e4262e4899797803d7fa", size = 254263, upload-time = "2026-01-30T01:03:20.834Z" }, + { url = "https://files.pythonhosted.org/packages/0f/95/70d7041273890f9f97a24234c00b746e8da86df462620194cef1d411ddeb/pytokens-0.4.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dc74c035f9bfca0255c1af77ddd2d6ae8419012805453e4b0e7513e17904545d", size = 268071, upload-time = "2026-01-30T01:03:21.888Z" }, + { url = "https://files.pythonhosted.org/packages/da/79/76e6d09ae19c99404656d7db9c35dfd20f2086f3eb6ecb496b5b31163bad/pytokens-0.4.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f66a6bbe741bd431f6d741e617e0f39ec7257ca1f89089593479347cc4d13324", size = 271716, upload-time = "2026-01-30T01:03:23.633Z" }, + { url = "https://files.pythonhosted.org/packages/79/37/482e55fa1602e0a7ff012661d8c946bafdc05e480ea5a32f4f7e336d4aa9/pytokens-0.4.1-cp314-cp314-win_amd64.whl", hash = "sha256:b35d7e5ad269804f6697727702da3c517bb8a5228afa450ab0fa787732055fc9", size = 104539, upload-time = "2026-01-30T01:03:24.788Z" }, + { url = "https://files.pythonhosted.org/packages/30/e8/20e7db907c23f3d63b0be3b8a4fd1927f6da2395f5bcc7f72242bb963dfe/pytokens-0.4.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8fcb9ba3709ff77e77f1c7022ff11d13553f3c30299a9fe246a166903e9091eb", size = 168474, upload-time = "2026-01-30T01:03:26.428Z" }, + { url = "https://files.pythonhosted.org/packages/d6/81/88a95ee9fafdd8f5f3452107748fd04c24930d500b9aba9738f3ade642cc/pytokens-0.4.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:79fc6b8699564e1f9b521582c35435f1bd32dd06822322ec44afdeba666d8cb3", size = 290473, upload-time = "2026-01-30T01:03:27.415Z" }, + { url = "https://files.pythonhosted.org/packages/cf/35/3aa899645e29b6375b4aed9f8d21df219e7c958c4c186b465e42ee0a06bf/pytokens-0.4.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d31b97b3de0f61571a124a00ffe9a81fb9939146c122c11060725bd5aea79975", size = 303485, upload-time = "2026-01-30T01:03:28.558Z" }, + { url = "https://files.pythonhosted.org/packages/52/a0/07907b6ff512674d9b201859f7d212298c44933633c946703a20c25e9d81/pytokens-0.4.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:967cf6e3fd4adf7de8fc73cd3043754ae79c36475c1c11d514fc72cf5490094a", size = 306698, upload-time = "2026-01-30T01:03:29.653Z" }, + { url = "https://files.pythonhosted.org/packages/39/2a/cbbf9250020a4a8dd53ba83a46c097b69e5eb49dd14e708f496f548c6612/pytokens-0.4.1-cp314-cp314t-win_amd64.whl", hash = "sha256:584c80c24b078eec1e227079d56dc22ff755e0ba8654d8383b2c549107528918", size = 116287, upload-time = "2026-01-30T01:03:30.912Z" }, + { url = "https://files.pythonhosted.org/packages/c6/78/397db326746f0a342855b81216ae1f0a32965deccfd7c830a2dbc66d2483/pytokens-0.4.1-py3-none-any.whl", hash = "sha256:26cef14744a8385f35d0e095dc8b3a7583f6c953c2e3d269c7f82484bf5ad2de", size = 13729, upload-time = "2026-01-30T01:03:45.029Z" }, +] + +[[package]] +name = "pywin32" +version = "311" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e7/ab/01ea1943d4eba0f850c3c61e78e8dd59757ff815ff3ccd0a84de5f541f42/pywin32-311-cp312-cp312-win32.whl", hash = "sha256:750ec6e621af2b948540032557b10a2d43b0cee2ae9758c54154d711cc852d31", size = 8706543, upload-time = "2025-07-14T20:13:20.765Z" }, + { url = "https://files.pythonhosted.org/packages/d1/a8/a0e8d07d4d051ec7502cd58b291ec98dcc0c3fff027caad0470b72cfcc2f/pywin32-311-cp312-cp312-win_amd64.whl", hash = "sha256:b8c095edad5c211ff31c05223658e71bf7116daa0ecf3ad85f3201ea3190d067", size = 9495040, upload-time = "2025-07-14T20:13:22.543Z" }, + { url = "https://files.pythonhosted.org/packages/ba/3a/2ae996277b4b50f17d61f0603efd8253cb2d79cc7ae159468007b586396d/pywin32-311-cp312-cp312-win_arm64.whl", hash = "sha256:e286f46a9a39c4a18b319c28f59b61de793654af2f395c102b4f819e584b5852", size = 8710102, upload-time = "2025-07-14T20:13:24.682Z" }, + { url = "https://files.pythonhosted.org/packages/a5/be/3fd5de0979fcb3994bfee0d65ed8ca9506a8a1260651b86174f6a86f52b3/pywin32-311-cp313-cp313-win32.whl", hash = "sha256:f95ba5a847cba10dd8c4d8fefa9f2a6cf283b8b88ed6178fa8a6c1ab16054d0d", size = 8705700, upload-time = "2025-07-14T20:13:26.471Z" }, + { url = "https://files.pythonhosted.org/packages/e3/28/e0a1909523c6890208295a29e05c2adb2126364e289826c0a8bc7297bd5c/pywin32-311-cp313-cp313-win_amd64.whl", hash = "sha256:718a38f7e5b058e76aee1c56ddd06908116d35147e133427e59a3983f703a20d", size = 9494700, upload-time = "2025-07-14T20:13:28.243Z" }, + { url = "https://files.pythonhosted.org/packages/04/bf/90339ac0f55726dce7d794e6d79a18a91265bdf3aa70b6b9ca52f35e022a/pywin32-311-cp313-cp313-win_arm64.whl", hash = "sha256:7b4075d959648406202d92a2310cb990fea19b535c7f4a78d3f5e10b926eeb8a", size = 8709318, upload-time = "2025-07-14T20:13:30.348Z" }, + { url = "https://files.pythonhosted.org/packages/c9/31/097f2e132c4f16d99a22bfb777e0fd88bd8e1c634304e102f313af69ace5/pywin32-311-cp314-cp314-win32.whl", hash = "sha256:b7a2c10b93f8986666d0c803ee19b5990885872a7de910fc460f9b0c2fbf92ee", size = 8840714, upload-time = "2025-07-14T20:13:32.449Z" }, + { url = "https://files.pythonhosted.org/packages/90/4b/07c77d8ba0e01349358082713400435347df8426208171ce297da32c313d/pywin32-311-cp314-cp314-win_amd64.whl", hash = "sha256:3aca44c046bd2ed8c90de9cb8427f581c479e594e99b5c0bb19b29c10fd6cb87", size = 9656800, upload-time = "2025-07-14T20:13:34.312Z" }, + { url = "https://files.pythonhosted.org/packages/c0/d2/21af5c535501a7233e734b8af901574572da66fcc254cb35d0609c9080dd/pywin32-311-cp314-cp314-win_arm64.whl", hash = "sha256:a508e2d9025764a8270f93111a970e1d0fbfc33f4153b388bb649b7eec4f9b42", size = 8932540, upload-time = "2025-07-14T20:13:36.379Z" }, +] + +[[package]] +name = "pyyaml" +version = "6.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/05/8e/961c0007c59b8dd7729d542c61a4d537767a59645b82a0b521206e1e25c2/pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f", size = 130960, upload-time = "2025-09-25T21:33:16.546Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/33/422b98d2195232ca1826284a76852ad5a86fe23e31b009c9886b2d0fb8b2/pyyaml-6.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196", size = 182063, upload-time = "2025-09-25T21:32:11.445Z" }, + { url = "https://files.pythonhosted.org/packages/89/a0/6cf41a19a1f2f3feab0e9c0b74134aa2ce6849093d5517a0c550fe37a648/pyyaml-6.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0", size = 173973, upload-time = "2025-09-25T21:32:12.492Z" }, + { url = "https://files.pythonhosted.org/packages/ed/23/7a778b6bd0b9a8039df8b1b1d80e2e2ad78aa04171592c8a5c43a56a6af4/pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28", size = 775116, upload-time = "2025-09-25T21:32:13.652Z" }, + { url = "https://files.pythonhosted.org/packages/65/30/d7353c338e12baef4ecc1b09e877c1970bd3382789c159b4f89d6a70dc09/pyyaml-6.0.3-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c", size = 844011, upload-time = "2025-09-25T21:32:15.21Z" }, + { url = "https://files.pythonhosted.org/packages/8b/9d/b3589d3877982d4f2329302ef98a8026e7f4443c765c46cfecc8858c6b4b/pyyaml-6.0.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc", size = 807870, upload-time = "2025-09-25T21:32:16.431Z" }, + { url = "https://files.pythonhosted.org/packages/05/c0/b3be26a015601b822b97d9149ff8cb5ead58c66f981e04fedf4e762f4bd4/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e", size = 761089, upload-time = "2025-09-25T21:32:17.56Z" }, + { url = "https://files.pythonhosted.org/packages/be/8e/98435a21d1d4b46590d5459a22d88128103f8da4c2d4cb8f14f2a96504e1/pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea", size = 790181, upload-time = "2025-09-25T21:32:18.834Z" }, + { url = "https://files.pythonhosted.org/packages/74/93/7baea19427dcfbe1e5a372d81473250b379f04b1bd3c4c5ff825e2327202/pyyaml-6.0.3-cp312-cp312-win32.whl", hash = "sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5", size = 137658, upload-time = "2025-09-25T21:32:20.209Z" }, + { url = "https://files.pythonhosted.org/packages/86/bf/899e81e4cce32febab4fb42bb97dcdf66bc135272882d1987881a4b519e9/pyyaml-6.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b", size = 154003, upload-time = "2025-09-25T21:32:21.167Z" }, + { url = "https://files.pythonhosted.org/packages/1a/08/67bd04656199bbb51dbed1439b7f27601dfb576fb864099c7ef0c3e55531/pyyaml-6.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd", size = 140344, upload-time = "2025-09-25T21:32:22.617Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/0fd08f8192109f7169db964b5707a2f1e8b745d4e239b784a5a1dd80d1db/pyyaml-6.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8", size = 181669, upload-time = "2025-09-25T21:32:23.673Z" }, + { url = "https://files.pythonhosted.org/packages/b1/16/95309993f1d3748cd644e02e38b75d50cbc0d9561d21f390a76242ce073f/pyyaml-6.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1", size = 173252, upload-time = "2025-09-25T21:32:25.149Z" }, + { url = "https://files.pythonhosted.org/packages/50/31/b20f376d3f810b9b2371e72ef5adb33879b25edb7a6d072cb7ca0c486398/pyyaml-6.0.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c", size = 767081, upload-time = "2025-09-25T21:32:26.575Z" }, + { url = "https://files.pythonhosted.org/packages/49/1e/a55ca81e949270d5d4432fbbd19dfea5321eda7c41a849d443dc92fd1ff7/pyyaml-6.0.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5", size = 841159, upload-time = "2025-09-25T21:32:27.727Z" }, + { url = "https://files.pythonhosted.org/packages/74/27/e5b8f34d02d9995b80abcef563ea1f8b56d20134d8f4e5e81733b1feceb2/pyyaml-6.0.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6", size = 801626, upload-time = "2025-09-25T21:32:28.878Z" }, + { url = "https://files.pythonhosted.org/packages/f9/11/ba845c23988798f40e52ba45f34849aa8a1f2d4af4b798588010792ebad6/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6", size = 753613, upload-time = "2025-09-25T21:32:30.178Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e0/7966e1a7bfc0a45bf0a7fb6b98ea03fc9b8d84fa7f2229e9659680b69ee3/pyyaml-6.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be", size = 794115, upload-time = "2025-09-25T21:32:31.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/94/980b50a6531b3019e45ddeada0626d45fa85cbe22300844a7983285bed3b/pyyaml-6.0.3-cp313-cp313-win32.whl", hash = "sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26", size = 137427, upload-time = "2025-09-25T21:32:32.58Z" }, + { url = "https://files.pythonhosted.org/packages/97/c9/39d5b874e8b28845e4ec2202b5da735d0199dbe5b8fb85f91398814a9a46/pyyaml-6.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c", size = 154090, upload-time = "2025-09-25T21:32:33.659Z" }, + { url = "https://files.pythonhosted.org/packages/73/e8/2bdf3ca2090f68bb3d75b44da7bbc71843b19c9f2b9cb9b0f4ab7a5a4329/pyyaml-6.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb", size = 140246, upload-time = "2025-09-25T21:32:34.663Z" }, + { url = "https://files.pythonhosted.org/packages/9d/8c/f4bd7f6465179953d3ac9bc44ac1a8a3e6122cf8ada906b4f96c60172d43/pyyaml-6.0.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac", size = 181814, upload-time = "2025-09-25T21:32:35.712Z" }, + { url = "https://files.pythonhosted.org/packages/bd/9c/4d95bb87eb2063d20db7b60faa3840c1b18025517ae857371c4dd55a6b3a/pyyaml-6.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310", size = 173809, upload-time = "2025-09-25T21:32:36.789Z" }, + { url = "https://files.pythonhosted.org/packages/92/b5/47e807c2623074914e29dabd16cbbdd4bf5e9b2db9f8090fa64411fc5382/pyyaml-6.0.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7", size = 766454, upload-time = "2025-09-25T21:32:37.966Z" }, + { url = "https://files.pythonhosted.org/packages/02/9e/e5e9b168be58564121efb3de6859c452fccde0ab093d8438905899a3a483/pyyaml-6.0.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788", size = 836355, upload-time = "2025-09-25T21:32:39.178Z" }, + { url = "https://files.pythonhosted.org/packages/88/f9/16491d7ed2a919954993e48aa941b200f38040928474c9e85ea9e64222c3/pyyaml-6.0.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5", size = 794175, upload-time = "2025-09-25T21:32:40.865Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3f/5989debef34dc6397317802b527dbbafb2b4760878a53d4166579111411e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764", size = 755228, upload-time = "2025-09-25T21:32:42.084Z" }, + { url = "https://files.pythonhosted.org/packages/d7/ce/af88a49043cd2e265be63d083fc75b27b6ed062f5f9fd6cdc223ad62f03e/pyyaml-6.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35", size = 789194, upload-time = "2025-09-25T21:32:43.362Z" }, + { url = "https://files.pythonhosted.org/packages/23/20/bb6982b26a40bb43951265ba29d4c246ef0ff59c9fdcdf0ed04e0687de4d/pyyaml-6.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac", size = 156429, upload-time = "2025-09-25T21:32:57.844Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f4/a4541072bb9422c8a883ab55255f918fa378ecf083f5b85e87fc2b4eda1b/pyyaml-6.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3", size = 143912, upload-time = "2025-09-25T21:32:59.247Z" }, + { url = "https://files.pythonhosted.org/packages/7c/f9/07dd09ae774e4616edf6cda684ee78f97777bdd15847253637a6f052a62f/pyyaml-6.0.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3", size = 189108, upload-time = "2025-09-25T21:32:44.377Z" }, + { url = "https://files.pythonhosted.org/packages/4e/78/8d08c9fb7ce09ad8c38ad533c1191cf27f7ae1effe5bb9400a46d9437fcf/pyyaml-6.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba", size = 183641, upload-time = "2025-09-25T21:32:45.407Z" }, + { url = "https://files.pythonhosted.org/packages/7b/5b/3babb19104a46945cf816d047db2788bcaf8c94527a805610b0289a01c6b/pyyaml-6.0.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c", size = 831901, upload-time = "2025-09-25T21:32:48.83Z" }, + { url = "https://files.pythonhosted.org/packages/8b/cc/dff0684d8dc44da4d22a13f35f073d558c268780ce3c6ba1b87055bb0b87/pyyaml-6.0.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702", size = 861132, upload-time = "2025-09-25T21:32:50.149Z" }, + { url = "https://files.pythonhosted.org/packages/b1/5e/f77dc6b9036943e285ba76b49e118d9ea929885becb0a29ba8a7c75e29fe/pyyaml-6.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c", size = 839261, upload-time = "2025-09-25T21:32:51.808Z" }, + { url = "https://files.pythonhosted.org/packages/ce/88/a9db1376aa2a228197c58b37302f284b5617f56a5d959fd1763fb1675ce6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065", size = 805272, upload-time = "2025-09-25T21:32:52.941Z" }, + { url = "https://files.pythonhosted.org/packages/da/92/1446574745d74df0c92e6aa4a7b0b3130706a4142b2d1a5869f2eaa423c6/pyyaml-6.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65", size = 829923, upload-time = "2025-09-25T21:32:54.537Z" }, + { url = "https://files.pythonhosted.org/packages/f0/7a/1c7270340330e575b92f397352af856a8c06f230aa3e76f86b39d01b416a/pyyaml-6.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9", size = 174062, upload-time = "2025-09-25T21:32:55.767Z" }, + { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, +] + +[[package]] +name = "pyyaml-env-tag" +version = "1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyyaml" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/eb/2e/79c822141bfd05a853236b504869ebc6b70159afc570e1d5a20641782eaa/pyyaml_env_tag-1.1.tar.gz", hash = "sha256:2eb38b75a2d21ee0475d6d97ec19c63287a7e140231e4214969d0eac923cd7ff", size = 5737, upload-time = "2025-05-13T15:24:01.64Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/11/432f32f8097b03e3cd5fe57e88efb685d964e2e5178a48ed61e841f7fdce/pyyaml_env_tag-1.1-py3-none-any.whl", hash = "sha256:17109e1a528561e32f026364712fee1264bc2ea6715120891174ed1b980d2e04", size = 4722, upload-time = "2025-05-13T15:23:59.629Z" }, +] + +[[package]] +name = "rdflib" +version = "7.6.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "pyparsing" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/98/f5/18bb77b7af9526add0c727a3b2048959847dc5fb030913e2918bf384fec3/rdflib-7.6.0.tar.gz", hash = "sha256:6c831288d5e4a5a7ece85d0ccde9877d512a3d0f02d7c06455d00d6d0ea379df", size = 4943826, upload-time = "2026-02-13T07:15:55.938Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/c2/6604a71269e0c1bd75656d5a001432d16f2cc5b8c057140ec797155c295e/rdflib-7.6.0-py3-none-any.whl", hash = "sha256:30c0a3ebf4c0e09215f066be7246794b6492e054e782d7ac2a34c9f70a15e0dd", size = 615416, upload-time = "2026-02-13T07:15:46.487Z" }, +] + +[[package]] +name = "redis" +version = "7.4.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/7b/7f/3759b1d0d72b7c92f0d70ffd9dc962b7b7b5ee74e135f9d7d8ab06b8a318/redis-7.4.0.tar.gz", hash = "sha256:64a6ea7bf567ad43c964d2c30d82853f8df927c5c9017766c55a1d1ed95d18ad", size = 4943913, upload-time = "2026-03-24T09:14:37.53Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/74/3a/95deec7db1eb53979973ebd156f3369a72732208d1391cd2e5d127062a32/redis-7.4.0-py3-none-any.whl", hash = "sha256:a9c74a5c893a5ef8455a5adb793a31bb70feb821c86eccb62eebef5a19c429ec", size = 409772, upload-time = "2026-03-24T09:14:35.968Z" }, +] + +[[package]] +name = "referencing" +version = "0.37.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "attrs" }, + { name = "rpds-py" }, + { name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, +] + +[[package]] +name = "regex" +version = "2026.5.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/dc/0e/49aee608ad09480e7fd276898c99ec6192985fa331abe4eb3a986094490b/regex-2026.5.9.tar.gz", hash = "sha256:a8234aa23ec39894bfe4a3f1b85616a7032481964a13ac6fc9f10de4f6fca270", size = 416074, upload-time = "2026-05-09T23:15:19.37Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/50/9b/6550044bc44e17c84d312c031c2ec42fbdb6a4ec4e29093be3a172d08772/regex-2026.5.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:57eeeb05db7979413dec5438f2db21d7ecbba787cde7a711df1a6f6df672aa06", size = 490451, upload-time = "2026-05-09T23:12:34.72Z" }, + { url = "https://files.pythonhosted.org/packages/1e/95/fc7ba4303b5a0f92446a12ee6778ef2c6c799233f5060042a31bf390cfe9/regex-2026.5.9-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:398c521292f4c7fb807001dcd54694d3a1fcafc179a36ad9cc56f98df85930b6", size = 292112, upload-time = "2026-05-09T23:12:36.285Z" }, + { url = "https://files.pythonhosted.org/packages/54/4b/ee27938d1b2c443e89a9a10e00d2d19aa5ee300cd3d61140644e93bb083e/regex-2026.5.9-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f7a7c26137296beba7784de6eba69c6a93a63ccebc385e4962fe67e267a91225", size = 289599, upload-time = "2026-05-09T23:12:38.089Z" }, + { url = "https://files.pythonhosted.org/packages/d8/dd/ba103dc19614e25f3880800ca67ce093d6e21b325d72b8383c7bf906e9fa/regex-2026.5.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6441cc660d76107934a09c22167200839a0e89604a6297f78a974e66e931d2c0", size = 796732, upload-time = "2026-05-09T23:12:40.062Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e7/f035b4fd858b050b0080bf302968dc0f59ba34e391872d54936758e6844e/regex-2026.5.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:91328f1c23d47595ca3ef0a7557fa129c5a23404b775c770697d2f35b33e0107", size = 865440, upload-time = "2026-05-09T23:12:42.059Z" }, + { url = "https://files.pythonhosted.org/packages/0a/51/8cd301ecc899aea28124357f729f4272f44de7806fc7ca02490bfbe253e8/regex-2026.5.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:93a7860539414dddaefba2b40f8771765ae17949d4c7182b876ce429e11a8309", size = 912329, upload-time = "2026-05-09T23:12:44.373Z" }, + { url = "https://files.pythonhosted.org/packages/cc/1e/3fbe2fa1e8cebd62f3bb7d3321cff1640aca2e240b51d9bd624aad949260/regex-2026.5.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd2810d22146b6d838acc5ec15602cb6b47920aa4e33015df3868eedfd20bab8", size = 801239, upload-time = "2026-05-09T23:12:46.268Z" }, + { url = "https://files.pythonhosted.org/packages/17/2f/6f6008682bf2cf98040a0d3153a8e557b6ab728d7713d045cee4ce544ab8/regex-2026.5.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:daff2bdbaf1d23e52fdff7c0b7bc2048b68f978df6a4d107ac981f94caef2e66", size = 777054, upload-time = "2026-05-09T23:12:48.051Z" }, + { url = "https://files.pythonhosted.org/packages/19/2b/eee0d20a6842ba04df4b8847a920b57ef56853f14ef85405473e586b605a/regex-2026.5.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4eeb011098fcb77af513dcef521a3dbecbf8849b1e38940759d293b7a93f5026", size = 785098, upload-time = "2026-05-09T23:12:49.851Z" }, + { url = "https://files.pythonhosted.org/packages/4a/98/6fc1e6410feefb92159edaed5041992bfe390e8d26c721865434acbca558/regex-2026.5.9-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:ea9c8ecfa1b73c73b626534d6626e5340d429630943672b8480724f44e84b962", size = 860095, upload-time = "2026-05-09T23:12:51.666Z" }, + { url = "https://files.pythonhosted.org/packages/18/a3/bd855e0f2cb1a978ecf6fa6bb69632dd9c3f6ea3b81cde62fde14c9daec7/regex-2026.5.9-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:cd2846168eb9ee3c513902bc8225409cb1caab31d04728b145171fa1625d9621", size = 765762, upload-time = "2026-05-09T23:12:53.413Z" }, + { url = "https://files.pythonhosted.org/packages/dc/66/0ae8c092e60b14c79d24f8e0b7f0aea5bfbffdcab00b5483d13404d3c3a5/regex-2026.5.9-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:39617fb0cde9c0e6306dc70e3bfc096f3da793219879f7ae7aa341a69fbdcf6d", size = 852100, upload-time = "2026-05-09T23:12:55.256Z" }, + { url = "https://files.pythonhosted.org/packages/21/de/8dfde60fc1b21c946a893ba273403b72617edb261370cb1087099a83f088/regex-2026.5.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fd03c4f0e33280d15cae17159b899245d6b7c53d21def19b263b39655061f5ce", size = 789479, upload-time = "2026-05-09T23:12:57.573Z" }, + { url = "https://files.pythonhosted.org/packages/c3/1c/bdcc98f9a4af4fdd166c74941174619ccff4726d3ce32faa8e9a2ecd38dd/regex-2026.5.9-cp312-cp312-win32.whl", hash = "sha256:164eba9b755ea6f244b0d881196fbc1fac09714e9782c9e2732b813142033c8e", size = 266699, upload-time = "2026-05-09T23:12:59.14Z" }, + { url = "https://files.pythonhosted.org/packages/78/87/240d36864f9e48ace85f72e79ced97ceb7f27ce87739a947dcb834b4e6bc/regex-2026.5.9-cp312-cp312-win_amd64.whl", hash = "sha256:86f40a5d6444db30a125c9c9177e6b25dad981cbc37451fd838f145e6edac92e", size = 277783, upload-time = "2026-05-09T23:13:00.789Z" }, + { url = "https://files.pythonhosted.org/packages/4f/b5/7b30f312b0669dff5beebe5b0989dc2d1a312b1a44fab852199c387a5b96/regex-2026.5.9-cp312-cp312-win_arm64.whl", hash = "sha256:96f5f58b54a063d7ea9dca08e1cf57bfe10499c4d579ee672da284f57f5f0070", size = 270513, upload-time = "2026-05-09T23:13:02.426Z" }, + { url = "https://files.pythonhosted.org/packages/aa/da/797e91ecec6f84135da778ddce78c20e0af5d2a15c26f87a81bc3eadb6db/regex-2026.5.9-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:d626b84406444b165fc0ba981604edea39f0588ff1f92baa23fe50799ea9afdb", size = 490303, upload-time = "2026-05-09T23:13:04.382Z" }, + { url = "https://files.pythonhosted.org/packages/44/da/bf30abaaa737b58f4a4b8c4a03659e02fd92092c822e0197ed9e0daab917/regex-2026.5.9-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d7bdc0ab8f3dd7e1b4f9ab88634e13374669db86bb3c72e8292f07ae313f539f", size = 292019, upload-time = "2026-05-09T23:13:06.022Z" }, + { url = "https://files.pythonhosted.org/packages/2d/e7/d0eaf5713828417b9e5648cf81fa9bacd4961f6ab98c380c2034f8716e35/regex-2026.5.9-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a8820737949116ffff55fe18f9fc644530063ba6ebfcb8314239416e78f1347c", size = 289468, upload-time = "2026-05-09T23:13:08.214Z" }, + { url = "https://files.pythonhosted.org/packages/d3/9b/b3fdd62b003baa1a9b593cd8c8699c9651c2e80cc21a5c715707983c42d7/regex-2026.5.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aa0fbdbac82cb3e4450d0ccde7d7a35607f4cb2dd9fba4b8b69bfaf8c9fa6aed", size = 796749, upload-time = "2026-05-09T23:13:10.573Z" }, + { url = "https://files.pythonhosted.org/packages/d4/30/66ab84588765f5b4b271a9ca09ef7ce2b87caa95176ec3d2ad65d7bc4902/regex-2026.5.9-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:57e8915c7986aa33d25e4d3629cef711cd2863f2961b10409f0c04cb8b7d9020", size = 865445, upload-time = "2026-05-09T23:13:12.523Z" }, + { url = "https://files.pythonhosted.org/packages/1a/89/f05169e8588aac365f35ffc7f3bc3184f095ef4cfded7cfaa3c7fd5dbd89/regex-2026.5.9-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:508f56a89ba9cb26e4168cbc37dbd60a28d82430a9e18ad1d25fe0883c314ca2", size = 912322, upload-time = "2026-05-09T23:13:14.281Z" }, + { url = "https://files.pythonhosted.org/packages/30/e1/c93444052cf41581f3c884ab3fb5823daf0992f11cd4388d4275ca610558/regex-2026.5.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b6d189041f15691cfa2b6c4290448ec221244d225b3f5fe9e7771b34ffcdf6e2", size = 801269, upload-time = "2026-05-09T23:13:16.569Z" }, + { url = "https://files.pythonhosted.org/packages/50/fe/0cf96b882f540e62e8b9956599798203d599c44cf4c77917ca27400ff69b/regex-2026.5.9-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e82db382b44d0111b22601c509c89f64434816c9e0eef9d1989cda8cc6ff1c04", size = 777085, upload-time = "2026-05-09T23:13:18.675Z" }, + { url = "https://files.pythonhosted.org/packages/23/5c/d78d4924e7fc875557b9e9b768423925fdfaac5549d06da7810019a9bd26/regex-2026.5.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2acfb48634f64996b57f90f39afa692ff362162722581921fe92239a59960f3c", size = 785153, upload-time = "2026-05-09T23:13:20.525Z" }, + { url = "https://files.pythonhosted.org/packages/bf/e0/5214774090e7b4524dcea3e3c4aa74141d43043f8beb49c1599db1c8b53a/regex-2026.5.9-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:d29eebfc9525db68cad3c97eedd7f754fa265aa5cd0cf4f863b2421e1b48fc9f", size = 860164, upload-time = "2026-05-09T23:13:22.263Z" }, + { url = "https://files.pythonhosted.org/packages/6e/e1/4a57a83350319b1271f0d7a249b8672513ed928b237a741631270de6caea/regex-2026.5.9-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:debb893095e944091c16e641a6e33c1b0f4cb61ab945ec5afbf53ce7068834d8", size = 765731, upload-time = "2026-05-09T23:13:24.277Z" }, + { url = "https://files.pythonhosted.org/packages/12/f4/499e74a20c156fc75836ee04a72a38d1a063978f600937f9760467beb1b0/regex-2026.5.9-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:d659eee77986549c9ea45b861c7567e44d6287c3dc9a4565478853f7b9fe2ff6", size = 852062, upload-time = "2026-05-09T23:13:26.125Z" }, + { url = "https://files.pythonhosted.org/packages/5b/92/7eebc0d0a01e78629695f342ba17e0deaff8fb45e79cc0d7b98287da6e3e/regex-2026.5.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2efa205e6d98b24d1f3ab395c11aa15cdf10935bca283d0285e0499c284fba21", size = 789577, upload-time = "2026-05-09T23:13:27.814Z" }, + { url = "https://files.pythonhosted.org/packages/05/a4/018e71f7d2ad48c1ebe6d3ae0026f9b7cb4802fd15c7cc02fdf724355102/regex-2026.5.9-cp313-cp313-win32.whl", hash = "sha256:f3844f134e834076677dd369976e9f5068679fcb8e50102fdf6b7ac96a3ec127", size = 266691, upload-time = "2026-05-09T23:13:29.549Z" }, + { url = "https://files.pythonhosted.org/packages/e6/1d/861a93719fb9ee7dbfc3761b3797b7a3e112a5d42c6129459d2d741be9b5/regex-2026.5.9-cp313-cp313-win_amd64.whl", hash = "sha256:3527bb4942d2c14552155406cdedd906567456821848aed1cb4933a391bf5eca", size = 277747, upload-time = "2026-05-09T23:13:31.859Z" }, + { url = "https://files.pythonhosted.org/packages/d9/c6/0a2436ae4da1ba76e51cb98943c6838a9a721faa40ebe2dce07694ae34e3/regex-2026.5.9-cp313-cp313-win_arm64.whl", hash = "sha256:56a33f191f17d8c417f99945ebdc1e691d3af9605d86ec68c7e54a57e3e17af6", size = 270500, upload-time = "2026-05-09T23:13:33.525Z" }, + { url = "https://files.pythonhosted.org/packages/e8/e9/d21346f7b60ed58789371358ed66b09d00f832e1bd7c06e55d9da5679882/regex-2026.5.9-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:01f28d868834624c934b8d2e0aa1c8341337e37831f4a012f18a5afcba4cbaf3", size = 494172, upload-time = "2026-05-09T23:13:35.935Z" }, + { url = "https://files.pythonhosted.org/packages/c4/43/fd1177a2032037c681baecdb3422ee4e1424aec4e4f470ef47793d325274/regex-2026.5.9-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:48036f6374aaa79eb3b754ec29c61d1c6b1606749d705a13f8854fa2539671f6", size = 293952, upload-time = "2026-05-09T23:13:38.307Z" }, + { url = "https://files.pythonhosted.org/packages/f2/7d/9fbf919768368d3f8a4f6c692cf2aa61e482b2b81ec6a298ace4cbf02480/regex-2026.5.9-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b96350aa424e79d4fd6b567b344dcbe2b2d6bfc48dfe7717587e1fa6d43da6ff", size = 292314, upload-time = "2026-05-09T23:13:40.353Z" }, + { url = "https://files.pythonhosted.org/packages/e2/6c/e41bfeecb589716843e7c4df09ba46ff2a42961457afece19059d85caeef/regex-2026.5.9-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8f3af7a4903c5c04a11a196a5aa75cdd7dd3f8508132f9fb3259d9f5908e3b88", size = 811681, upload-time = "2026-05-09T23:13:42.543Z" }, + { url = "https://files.pythonhosted.org/packages/87/83/a5c1c525fba0aa656e88ad0face0b1829788ef4c2fb6b26df58aa1151b84/regex-2026.5.9-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7e87577720152d2caae19fe2baaf1f8d5ca12091e9e229f03915c37d1e4b9178", size = 871135, upload-time = "2026-05-09T23:13:44.326Z" }, + { url = "https://files.pythonhosted.org/packages/18/d4/80882e799e440dd878b0979cbebf8fa4d54624a332c83037c7a701649e3f/regex-2026.5.9-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c8b9b9d294cfea3cd19c718ade7cc93492b2c4991abd9a68d0b3477ae6d8e100", size = 917265, upload-time = "2026-05-09T23:13:47.295Z" }, + { url = "https://files.pythonhosted.org/packages/ae/ff/8db60211e2286e396aad7dc7725356c502bff0901ea05bd6cdc2e1a042b9/regex-2026.5.9-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:728d8bfd28a8845c8b6bc5dc7ce010453d206396786c0765c2740cb65f37791e", size = 816311, upload-time = "2026-05-09T23:13:49.885Z" }, + { url = "https://files.pythonhosted.org/packages/4c/47/742ef579c61730f8d268e5cf1f9ce0e37e2ea041ad0f5644724f2378e463/regex-2026.5.9-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7e30b874d341fac767d7df5a0870540541c2c054b80cfaac116e8d367a8a7ff2", size = 785498, upload-time = "2026-05-09T23:13:52.25Z" }, + { url = "https://files.pythonhosted.org/packages/7f/ab/cb0999802dcb0fb95b1ab005e8d4163d8afdd67efc2cb6b6630ac13f8cb1/regex-2026.5.9-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:fd190e88a895a8901325fad284a3f74ea52b1da8525b76cc811fa9b1edf0ce2b", size = 801348, upload-time = "2026-05-09T23:13:54.127Z" }, + { url = "https://files.pythonhosted.org/packages/7d/62/8ca59a24c55bc34d166eefaf3717bd77772f329fdbf984d86581e0a3571c/regex-2026.5.9-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:8e76e8161ad00694cfce6767d5dea860c6391ac5b83e5c3a39661e696f11fc7e", size = 866493, upload-time = "2026-05-09T23:13:56.067Z" }, + { url = "https://files.pythonhosted.org/packages/8d/3d/30f2ae62cef3278bb5bb821f467277a55fb73f01032cf85997e15e8289a8/regex-2026.5.9-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ddda5340e6c01a293027dd46232fa79eaff1b48058ce7a98f572b6445b088041", size = 772811, upload-time = "2026-05-09T23:13:57.867Z" }, + { url = "https://files.pythonhosted.org/packages/d8/ae/7d2089bcd78ad0c0161bc684339df50032acb438a7bd3305e7ddb1193cec/regex-2026.5.9-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:205109e96b3cf5adf8f4cd62bedde9487feb282b9497a3535451e5a24cd706a0", size = 856584, upload-time = "2026-05-09T23:13:59.679Z" }, + { url = "https://files.pythonhosted.org/packages/a9/29/92ff47f75990131ea4f24ba17819e5a9d141e10819807e09addd73409af6/regex-2026.5.9-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dfbe4579b9f08036aa7d101d1835437a20783574ac66327e6b29b4018a138081", size = 803453, upload-time = "2026-05-09T23:14:01.978Z" }, + { url = "https://files.pythonhosted.org/packages/04/99/eff29f1037dcab36702c9ee5d6858cf1ce2336ea8ea2987f64245b99ea5e/regex-2026.5.9-cp313-cp313t-win32.whl", hash = "sha256:ed2c9e8068b614c574d8d30e543d617cf5379b0535d46f97ef00e904745a08b5", size = 269951, upload-time = "2026-05-09T23:14:03.661Z" }, + { url = "https://files.pythonhosted.org/packages/0e/9d/8870b8981d27b22cda77bb26a5ac7ebfa9c7d9e0dea195a834a82380e748/regex-2026.5.9-cp313-cp313t-win_amd64.whl", hash = "sha256:b46b0f094dc1d3b90356c85a0bd2c9bafc4a6a190b9d6f8ddd5a033b6e088ed4", size = 281240, upload-time = "2026-05-09T23:14:05.56Z" }, + { url = "https://files.pythonhosted.org/packages/72/b1/3379415e8f135c13ac551353397cc4fe97b4978f3cac73c5fcbcded548b8/regex-2026.5.9-cp313-cp313t-win_arm64.whl", hash = "sha256:872acc074bd29ffc9913ecdfedf6ea77502312ca44a4aa0d3779089c6069d8de", size = 272383, upload-time = "2026-05-09T23:14:07.843Z" }, + { url = "https://files.pythonhosted.org/packages/13/3e/9c3cd292d8808b3645a2ce517e200179b6d0e903f176300bd8b542e14de5/regex-2026.5.9-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:1bd7587a2948b4085195d5a3374eaf4a425dc3e55784c038175355ecf3bbbf8a", size = 490376, upload-time = "2026-05-09T23:14:09.64Z" }, + { url = "https://files.pythonhosted.org/packages/60/70/d43ee8a2ca0a8b68d167f21658b85520ac0574617c7f320367c5047f7556/regex-2026.5.9-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:dea2e88e1cce4522496cce630e11e67b98b7076620bc4336c3f674bc21a375f4", size = 291964, upload-time = "2026-05-09T23:14:11.424Z" }, + { url = "https://files.pythonhosted.org/packages/21/91/9d50b433828d8e74196904e168a43abf1e6e88b2a15d47ed742456720c37/regex-2026.5.9-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:2099f7e7ff7b6aa3192312650a56e91cc091e49d50b04e4f6f8b6e28b3b27f1c", size = 289682, upload-time = "2026-05-09T23:14:13.123Z" }, + { url = "https://files.pythonhosted.org/packages/3e/d2/b835e3cafbb9d977736912436259ff551d60919f7d7b3d37d46659c63564/regex-2026.5.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecd353045824e4477562a2ac718c25799cdaaa41f7aa925a806a8a3e6848a5b9", size = 796996, upload-time = "2026-05-09T23:14:14.923Z" }, + { url = "https://files.pythonhosted.org/packages/2c/a6/9f992d00019166b9de01c546dd4549bc679f2a68df11b877740b0760b7c2/regex-2026.5.9-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:65c8c8c37377794bd5b2f3ebe51919042bf17aec802e23c833d89782ed0c78af", size = 866089, upload-time = "2026-05-09T23:14:17.757Z" }, + { url = "https://files.pythonhosted.org/packages/e0/08/4d32af657e049b19cb62b02e46e38fe1518797bfb2203ee93a510b21b0dc/regex-2026.5.9-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5b73ab8afcf66c622db143d1c6fda4e58e4d537ee4f125229ad47b1ab80f34c0", size = 911530, upload-time = "2026-05-09T23:14:20.353Z" }, + { url = "https://files.pythonhosted.org/packages/d9/27/2af43dd1dc201d1fecefda64a45f4ad0995855b92724f795a777b402ee69/regex-2026.5.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0de5cf193997384ed2ca6f1cd4f78055b255d93d82d5a8cd6ba0d11c10b167e4", size = 800643, upload-time = "2026-05-09T23:14:22.265Z" }, + { url = "https://files.pythonhosted.org/packages/a4/dd/23a249047013b5321d4a60c4d2437462086f601b061776a525e5fba2a59f/regex-2026.5.9-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d641a8c9a61618047796d572a39a79b26167b0411d2c3031937b2fe2d081e2cf", size = 777223, upload-time = "2026-05-09T23:14:24.179Z" }, + { url = "https://files.pythonhosted.org/packages/94/6a/e85ed9538cd19586d0465076a4578a12e093ce776d15f3f8ce92733a8dd6/regex-2026.5.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:24b2355ef5cc9aa5b8f07d17704face1c166fdcc2290fa7bd6e6c925655a8346", size = 785760, upload-time = "2026-05-09T23:14:26.065Z" }, + { url = "https://files.pythonhosted.org/packages/2a/c4/f25473209438638e947c55f9156fd8f236f74169229028cc99116380868e/regex-2026.5.9-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:a24852d3c29ad9e47593593d8a247c44ccc3d0548ef12c822d6ed0810affe676", size = 860891, upload-time = "2026-05-09T23:14:28.17Z" }, + { url = "https://files.pythonhosted.org/packages/f9/f7/f4f86e3c74419c37370e91f150ae0c2ef7d34b2e0e4cdd5da046a02e4022/regex-2026.5.9-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:916714069da19329ef7de197dcbc77bb3104145c7c2c864dbfbe318f46b88b14", size = 765891, upload-time = "2026-05-09T23:14:30.06Z" }, + { url = "https://files.pythonhosted.org/packages/26/70/704d8e13765939146b1cd0ef4e2feb71d7929727d2290f026eed10095955/regex-2026.5.9-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:fa411799ca8da32a8d38d020a88faa5b6f91657d284761352940ecf9f7c3bbdd", size = 851380, upload-time = "2026-05-09T23:14:32.123Z" }, + { url = "https://files.pythonhosted.org/packages/26/29/1a13582a8460038edc38e49f64ceb0dd7c60f5caba77571f4bf6601965d9/regex-2026.5.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1e6da47d679b7010ef27556b6e0f99771b744936db1792a10ceac6547ae1503e", size = 789350, upload-time = "2026-05-09T23:14:34.799Z" }, + { url = "https://files.pythonhosted.org/packages/73/56/3dcafe34fc72e271d62ad9a291801e88a1457bb251c132f15fcc2e5aad1a/regex-2026.5.9-cp314-cp314-win32.whl", hash = "sha256:98bd73080e8756255137e1bd3f3f00295bbc5aa383c0e0f973920e9134d7c4ad", size = 272130, upload-time = "2026-05-09T23:14:36.729Z" }, + { url = "https://files.pythonhosted.org/packages/d0/9c/02eebf0be95efe416c664db7fb8b6b05b7a0b06a7544f2884f2558b0526f/regex-2026.5.9-cp314-cp314-win_amd64.whl", hash = "sha256:ff8d372ac2acdc048d1c19916f27ee61bc5722728458ba6ca5052f2c72d51763", size = 280999, upload-time = "2026-05-09T23:14:39.126Z" }, + { url = "https://files.pythonhosted.org/packages/70/5a/1dd1abee76cb7a846a0bcf42fdc87e5720c3c33c24f3e37814310a513d9f/regex-2026.5.9-cp314-cp314-win_arm64.whl", hash = "sha256:e1d93bf647916292e8edcec150c07ddf3dc50179ccaf770c04a7f9e452155372", size = 273500, upload-time = "2026-05-09T23:14:41.059Z" }, + { url = "https://files.pythonhosted.org/packages/86/c1/c5f619b0057a7965cb78ec559c1d7a45ce8c99a35bea95483d64959a93d9/regex-2026.5.9-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:83d0ee4a57d1c87cb549e195ec300b8f0ec3a82eba66d835e4e2ed8634fe4499", size = 494269, upload-time = "2026-05-09T23:14:42.869Z" }, + { url = "https://files.pythonhosted.org/packages/05/2c/5d01f1aee33de4bbe60c8452945bfc8477ca7c5ae4450f6bfe711036cb36/regex-2026.5.9-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:d3d7eb5c9a7f6df82ed3cfac9beb93882a5cbcb5b8b157b56cb2b3b276574ac1", size = 293954, upload-time = "2026-05-09T23:14:44.822Z" }, + { url = "https://files.pythonhosted.org/packages/7a/fe/e8988b2ae2108c6ef71bd4aa8d87fbe257976dd0810e826cd75f701c68b6/regex-2026.5.9-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:075160bf16658e16d35233300b8453aac25de4cbea808d22348b6979668e924d", size = 292405, upload-time = "2026-05-09T23:14:47.211Z" }, + { url = "https://files.pythonhosted.org/packages/79/34/d2b0937faa7859263f7f0a3c6b103a1296306be6952dc173d0154e9a2f49/regex-2026.5.9-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:45375819235558a4ff1c4971dc32881f022613abdb180128f5cb4768c1765a1c", size = 811855, upload-time = "2026-05-09T23:14:49.21Z" }, + { url = "https://files.pythonhosted.org/packages/80/fe/daf53a47457a8486db66c66c01ceb9c2303eecee3f87197f1e77eb1a736d/regex-2026.5.9-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ead4b163ac30a29574510cd4b3e2e985ac5290c05fc7095557d6a5f403fc31b5", size = 871189, upload-time = "2026-05-09T23:14:51.555Z" }, + { url = "https://files.pythonhosted.org/packages/1c/75/058fc4470cbfbf57d800aff1a0022b929a3f9fa553ee10a0cdf2070eb31f/regex-2026.5.9-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8c6e4218fbdfbcd4f6c19efca40930d24a621bf4b48cb76bc6640543bd28ef20", size = 917485, upload-time = "2026-05-09T23:14:53.633Z" }, + { url = "https://files.pythonhosted.org/packages/88/e7/179cfda3a28bc843b5c6cfe7f79f23489c791ed95f151083803660878432/regex-2026.5.9-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6351571c8a42b505eb555c0dc47d740d0fb66977dc142919eea6f4325b7c56a0", size = 816369, upload-time = "2026-05-09T23:14:56.198Z" }, + { url = "https://files.pythonhosted.org/packages/41/90/6f0cc422071688266d344fca8462d787cba0a2c144acb25721f9a61ec265/regex-2026.5.9-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:002205cafd2a9e78c6290c7d1df277bf3277b3b7a30e0b4bb0dac2e2e3f7cb2d", size = 785869, upload-time = "2026-05-09T23:14:58.602Z" }, + { url = "https://files.pythonhosted.org/packages/02/67/a31f1760f09c27b251ef39e9beb541f462cf977381d067faa764c2c0e393/regex-2026.5.9-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8abd33fef90b2a9efac5557d6033ca82d1195ed3a15fea5af15ba7b463c6a63b", size = 801427, upload-time = "2026-05-09T23:15:00.642Z" }, + { url = "https://files.pythonhosted.org/packages/e3/c4/1a80654597b6bc1e1ea0494824c31200e8a956abe290afae9b19a166a148/regex-2026.5.9-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:31037c82eccb44b7ea2e9e221d7c01429430e989a1f4b91ea5a855f6017b509a", size = 866482, upload-time = "2026-05-09T23:15:03.384Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/960724e06482c08466ff5611e242e86f80062949cdf6b4b9cc317b9dd93d/regex-2026.5.9-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5604dfd046dc37eca90250fc3be938b076c8059fa772ac0ed6f499b0f0fb0415", size = 773022, upload-time = "2026-05-09T23:15:05.625Z" }, + { url = "https://files.pythonhosted.org/packages/50/a8/a9979c3e7918280e93159ebcab5ef1a65116dd4f3bd6091be0eae4a126e8/regex-2026.5.9-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:0e1b1b4e496afbb24f4a62aba855ee4f88f25578927697b340702e48c9ee6bc2", size = 856642, upload-time = "2026-05-09T23:15:07.966Z" }, + { url = "https://files.pythonhosted.org/packages/fe/d4/a9b732f2f0072c0ab12227483abb24fffcb9f73f8a2b203df0a6d0434735/regex-2026.5.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:be3372b9df6ddecff6486d37e19095a7b4973137caf5512407a89f4455361f41", size = 803552, upload-time = "2026-05-09T23:15:10.215Z" }, + { url = "https://files.pythonhosted.org/packages/d5/fe/1b3113817447a1d4155e4ac76d2e072f42c0bcba2f43fa8a0e756ea2cd91/regex-2026.5.9-cp314-cp314t-win32.whl", hash = "sha256:3ddd90103f9e5c471c49c7852ecc1fe27c7e45eb99e977aefe7caa4e779f4f58", size = 275746, upload-time = "2026-05-09T23:15:12.609Z" }, + { url = "https://files.pythonhosted.org/packages/92/73/93d42045302636c91f2e5ef588b65b84b01428f28ec77de256b1dfdfbe5c/regex-2026.5.9-cp314-cp314t-win_amd64.whl", hash = "sha256:ca518ed29c46eecba6010b15f1b9a479314d2de409536e71b6a13aa04e3b8a77", size = 285685, upload-time = "2026-05-09T23:15:15.086Z" }, + { url = "https://files.pythonhosted.org/packages/da/80/35b4c33c804a165a7f55289afda3ea9e3eb6d15800341a2d66455c0f1f30/regex-2026.5.9-cp314-cp314t-win_arm64.whl", hash = "sha256:5e41809d2683fcde7d5a8c87a6567ba1fb1ce0de9f31bff578de00a4b2d76daa", size = 275713, upload-time = "2026-05-09T23:15:16.98Z" }, +] + +[[package]] +name = "requests" +version = "2.34.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "certifi" }, + { name = "charset-normalizer" }, + { name = "idna" }, + { name = "urllib3" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/43/b8/7a707d60fea4c49094e40262cc0e2ca6c768cca21587e34d3f705afec47e/requests-2.34.0.tar.gz", hash = "sha256:7d62fe92f50eb82c529b0916bb445afa1531a566fc8f35ffdc64446e771b856a", size = 142436, upload-time = "2026-05-11T19:29:51.717Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ef/e6/e300fce5fe83c30520607a015dabd985df3251e188d234bfe9492e17a389/requests-2.34.0-py3-none-any.whl", hash = "sha256:917520a21b767485ce7c588f4ebb917c436b24a31231b44228715eaeb5a52c60", size = 73021, upload-time = "2026-05-11T19:29:49.923Z" }, +] + +[[package]] +name = "rich" +version = "15.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, +] + +[[package]] +name = "rpds-py" +version = "0.30.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/20/af/3f2f423103f1113b36230496629986e0ef7e199d2aa8392452b484b38ced/rpds_py-0.30.0.tar.gz", hash = "sha256:dd8ff7cf90014af0c0f787eea34794ebf6415242ee1d6fa91eaba725cc441e84", size = 69469, upload-time = "2025-11-30T20:24:38.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/03/e7/98a2f4ac921d82f33e03f3835f5bf3a4a40aa1bfdc57975e74a97b2b4bdd/rpds_py-0.30.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a161f20d9a43006833cd7068375a94d035714d73a172b681d8881820600abfad", size = 375086, upload-time = "2025-11-30T20:22:17.93Z" }, + { url = "https://files.pythonhosted.org/packages/4d/a1/bca7fd3d452b272e13335db8d6b0b3ecde0f90ad6f16f3328c6fb150c889/rpds_py-0.30.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6abc8880d9d036ecaafe709079969f56e876fcf107f7a8e9920ba6d5a3878d05", size = 359053, upload-time = "2025-11-30T20:22:19.297Z" }, + { url = "https://files.pythonhosted.org/packages/65/1c/ae157e83a6357eceff62ba7e52113e3ec4834a84cfe07fa4b0757a7d105f/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ca28829ae5f5d569bb62a79512c842a03a12576375d5ece7d2cadf8abe96ec28", size = 390763, upload-time = "2025-11-30T20:22:21.661Z" }, + { url = "https://files.pythonhosted.org/packages/d4/36/eb2eb8515e2ad24c0bd43c3ee9cd74c33f7ca6430755ccdb240fd3144c44/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a1010ed9524c73b94d15919ca4d41d8780980e1765babf85f9a2f90d247153dd", size = 408951, upload-time = "2025-11-30T20:22:23.408Z" }, + { url = "https://files.pythonhosted.org/packages/d6/65/ad8dc1784a331fabbd740ef6f71ce2198c7ed0890dab595adb9ea2d775a1/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f8d1736cfb49381ba528cd5baa46f82fdc65c06e843dab24dd70b63d09121b3f", size = 514622, upload-time = "2025-11-30T20:22:25.16Z" }, + { url = "https://files.pythonhosted.org/packages/63/8e/0cfa7ae158e15e143fe03993b5bcd743a59f541f5952e1546b1ac1b5fd45/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d948b135c4693daff7bc2dcfc4ec57237a29bd37e60c2fabf5aff2bbacf3e2f1", size = 414492, upload-time = "2025-11-30T20:22:26.505Z" }, + { url = "https://files.pythonhosted.org/packages/60/1b/6f8f29f3f995c7ffdde46a626ddccd7c63aefc0efae881dc13b6e5d5bb16/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47f236970bccb2233267d89173d3ad2703cd36a0e2a6e92d0560d333871a3d23", size = 394080, upload-time = "2025-11-30T20:22:27.934Z" }, + { url = "https://files.pythonhosted.org/packages/6d/d5/a266341051a7a3ca2f4b750a3aa4abc986378431fc2da508c5034d081b70/rpds_py-0.30.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:2e6ecb5a5bcacf59c3f912155044479af1d0b6681280048b338b28e364aca1f6", size = 408680, upload-time = "2025-11-30T20:22:29.341Z" }, + { url = "https://files.pythonhosted.org/packages/10/3b/71b725851df9ab7a7a4e33cf36d241933da66040d195a84781f49c50490c/rpds_py-0.30.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a8fa71a2e078c527c3e9dc9fc5a98c9db40bcc8a92b4e8858e36d329f8684b51", size = 423589, upload-time = "2025-11-30T20:22:31.469Z" }, + { url = "https://files.pythonhosted.org/packages/00/2b/e59e58c544dc9bd8bd8384ecdb8ea91f6727f0e37a7131baeff8d6f51661/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:73c67f2db7bc334e518d097c6d1e6fed021bbc9b7d678d6cc433478365d1d5f5", size = 573289, upload-time = "2025-11-30T20:22:32.997Z" }, + { url = "https://files.pythonhosted.org/packages/da/3e/a18e6f5b460893172a7d6a680e86d3b6bc87a54c1f0b03446a3c8c7b588f/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5ba103fb455be00f3b1c2076c9d4264bfcb037c976167a6047ed82f23153f02e", size = 599737, upload-time = "2025-11-30T20:22:34.419Z" }, + { url = "https://files.pythonhosted.org/packages/5c/e2/714694e4b87b85a18e2c243614974413c60aa107fd815b8cbc42b873d1d7/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7cee9c752c0364588353e627da8a7e808a66873672bcb5f52890c33fd965b394", size = 563120, upload-time = "2025-11-30T20:22:35.903Z" }, + { url = "https://files.pythonhosted.org/packages/6f/ab/d5d5e3bcedb0a77f4f613706b750e50a5a3ba1c15ccd3665ecc636c968fd/rpds_py-0.30.0-cp312-cp312-win32.whl", hash = "sha256:1ab5b83dbcf55acc8b08fc62b796ef672c457b17dbd7820a11d6c52c06839bdf", size = 223782, upload-time = "2025-11-30T20:22:37.271Z" }, + { url = "https://files.pythonhosted.org/packages/39/3b/f786af9957306fdc38a74cef405b7b93180f481fb48453a114bb6465744a/rpds_py-0.30.0-cp312-cp312-win_amd64.whl", hash = "sha256:a090322ca841abd453d43456ac34db46e8b05fd9b3b4ac0c78bcde8b089f959b", size = 240463, upload-time = "2025-11-30T20:22:39.021Z" }, + { url = "https://files.pythonhosted.org/packages/f3/d2/b91dc748126c1559042cfe41990deb92c4ee3e2b415f6b5234969ffaf0cc/rpds_py-0.30.0-cp312-cp312-win_arm64.whl", hash = "sha256:669b1805bd639dd2989b281be2cfd951c6121b65e729d9b843e9639ef1fd555e", size = 230868, upload-time = "2025-11-30T20:22:40.493Z" }, + { url = "https://files.pythonhosted.org/packages/ed/dc/d61221eb88ff410de3c49143407f6f3147acf2538c86f2ab7ce65ae7d5f9/rpds_py-0.30.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:f83424d738204d9770830d35290ff3273fbb02b41f919870479fab14b9d303b2", size = 374887, upload-time = "2025-11-30T20:22:41.812Z" }, + { url = "https://files.pythonhosted.org/packages/fd/32/55fb50ae104061dbc564ef15cc43c013dc4a9f4527a1f4d99baddf56fe5f/rpds_py-0.30.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e7536cd91353c5273434b4e003cbda89034d67e7710eab8761fd918ec6c69cf8", size = 358904, upload-time = "2025-11-30T20:22:43.479Z" }, + { url = "https://files.pythonhosted.org/packages/58/70/faed8186300e3b9bdd138d0273109784eea2396c68458ed580f885dfe7ad/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2771c6c15973347f50fece41fc447c054b7ac2ae0502388ce3b6738cd366e3d4", size = 389945, upload-time = "2025-11-30T20:22:44.819Z" }, + { url = "https://files.pythonhosted.org/packages/bd/a8/073cac3ed2c6387df38f71296d002ab43496a96b92c823e76f46b8af0543/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0a59119fc6e3f460315fe9d08149f8102aa322299deaa5cab5b40092345c2136", size = 407783, upload-time = "2025-11-30T20:22:46.103Z" }, + { url = "https://files.pythonhosted.org/packages/77/57/5999eb8c58671f1c11eba084115e77a8899d6e694d2a18f69f0ba471ec8b/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:76fec018282b4ead0364022e3c54b60bf368b9d926877957a8624b58419169b7", size = 515021, upload-time = "2025-11-30T20:22:47.458Z" }, + { url = "https://files.pythonhosted.org/packages/e0/af/5ab4833eadc36c0a8ed2bc5c0de0493c04f6c06de223170bd0798ff98ced/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:692bef75a5525db97318e8cd061542b5a79812d711ea03dbc1f6f8dbb0c5f0d2", size = 414589, upload-time = "2025-11-30T20:22:48.872Z" }, + { url = "https://files.pythonhosted.org/packages/b7/de/f7192e12b21b9e9a68a6d0f249b4af3fdcdff8418be0767a627564afa1f1/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9027da1ce107104c50c81383cae773ef5c24d296dd11c99e2629dbd7967a20c6", size = 394025, upload-time = "2025-11-30T20:22:50.196Z" }, + { url = "https://files.pythonhosted.org/packages/91/c4/fc70cd0249496493500e7cc2de87504f5aa6509de1e88623431fec76d4b6/rpds_py-0.30.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:9cf69cdda1f5968a30a359aba2f7f9aa648a9ce4b580d6826437f2b291cfc86e", size = 408895, upload-time = "2025-11-30T20:22:51.87Z" }, + { url = "https://files.pythonhosted.org/packages/58/95/d9275b05ab96556fefff73a385813eb66032e4c99f411d0795372d9abcea/rpds_py-0.30.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a4796a717bf12b9da9d3ad002519a86063dcac8988b030e405704ef7d74d2d9d", size = 422799, upload-time = "2025-11-30T20:22:53.341Z" }, + { url = "https://files.pythonhosted.org/packages/06/c1/3088fc04b6624eb12a57eb814f0d4997a44b0d208d6cace713033ff1a6ba/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5d4c2aa7c50ad4728a094ebd5eb46c452e9cb7edbfdb18f9e1221f597a73e1e7", size = 572731, upload-time = "2025-11-30T20:22:54.778Z" }, + { url = "https://files.pythonhosted.org/packages/d8/42/c612a833183b39774e8ac8fecae81263a68b9583ee343db33ab571a7ce55/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba81a9203d07805435eb06f536d95a266c21e5b2dfbf6517748ca40c98d19e31", size = 599027, upload-time = "2025-11-30T20:22:56.212Z" }, + { url = "https://files.pythonhosted.org/packages/5f/60/525a50f45b01d70005403ae0e25f43c0384369ad24ffe46e8d9068b50086/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:945dccface01af02675628334f7cf49c2af4c1c904748efc5cf7bbdf0b579f95", size = 563020, upload-time = "2025-11-30T20:22:58.2Z" }, + { url = "https://files.pythonhosted.org/packages/0b/5d/47c4655e9bcd5ca907148535c10e7d489044243cc9941c16ed7cd53be91d/rpds_py-0.30.0-cp313-cp313-win32.whl", hash = "sha256:b40fb160a2db369a194cb27943582b38f79fc4887291417685f3ad693c5a1d5d", size = 223139, upload-time = "2025-11-30T20:23:00.209Z" }, + { url = "https://files.pythonhosted.org/packages/f2/e1/485132437d20aa4d3e1d8b3fb5a5e65aa8139f1e097080c2a8443201742c/rpds_py-0.30.0-cp313-cp313-win_amd64.whl", hash = "sha256:806f36b1b605e2d6a72716f321f20036b9489d29c51c91f4dd29a3e3afb73b15", size = 240224, upload-time = "2025-11-30T20:23:02.008Z" }, + { url = "https://files.pythonhosted.org/packages/24/95/ffd128ed1146a153d928617b0ef673960130be0009c77d8fbf0abe306713/rpds_py-0.30.0-cp313-cp313-win_arm64.whl", hash = "sha256:d96c2086587c7c30d44f31f42eae4eac89b60dabbac18c7669be3700f13c3ce1", size = 230645, upload-time = "2025-11-30T20:23:03.43Z" }, + { url = "https://files.pythonhosted.org/packages/ff/1b/b10de890a0def2a319a2626334a7f0ae388215eb60914dbac8a3bae54435/rpds_py-0.30.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:eb0b93f2e5c2189ee831ee43f156ed34e2a89a78a66b98cadad955972548be5a", size = 364443, upload-time = "2025-11-30T20:23:04.878Z" }, + { url = "https://files.pythonhosted.org/packages/0d/bf/27e39f5971dc4f305a4fb9c672ca06f290f7c4e261c568f3dea16a410d47/rpds_py-0.30.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:922e10f31f303c7c920da8981051ff6d8c1a56207dbdf330d9047f6d30b70e5e", size = 353375, upload-time = "2025-11-30T20:23:06.342Z" }, + { url = "https://files.pythonhosted.org/packages/40/58/442ada3bba6e8e6615fc00483135c14a7538d2ffac30e2d933ccf6852232/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cdc62c8286ba9bf7f47befdcea13ea0e26bf294bda99758fd90535cbaf408000", size = 383850, upload-time = "2025-11-30T20:23:07.825Z" }, + { url = "https://files.pythonhosted.org/packages/14/14/f59b0127409a33c6ef6f5c1ebd5ad8e32d7861c9c7adfa9a624fc3889f6c/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:47f9a91efc418b54fb8190a6b4aa7813a23fb79c51f4bb84e418f5476c38b8db", size = 392812, upload-time = "2025-11-30T20:23:09.228Z" }, + { url = "https://files.pythonhosted.org/packages/b3/66/e0be3e162ac299b3a22527e8913767d869e6cc75c46bd844aa43fb81ab62/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1f3587eb9b17f3789ad50824084fa6f81921bbf9a795826570bda82cb3ed91f2", size = 517841, upload-time = "2025-11-30T20:23:11.186Z" }, + { url = "https://files.pythonhosted.org/packages/3d/55/fa3b9cf31d0c963ecf1ba777f7cf4b2a2c976795ac430d24a1f43d25a6ba/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:39c02563fc592411c2c61d26b6c5fe1e51eaa44a75aa2c8735ca88b0d9599daa", size = 408149, upload-time = "2025-11-30T20:23:12.864Z" }, + { url = "https://files.pythonhosted.org/packages/60/ca/780cf3b1a32b18c0f05c441958d3758f02544f1d613abf9488cd78876378/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:51a1234d8febafdfd33a42d97da7a43f5dcb120c1060e352a3fbc0c6d36e2083", size = 383843, upload-time = "2025-11-30T20:23:14.638Z" }, + { url = "https://files.pythonhosted.org/packages/82/86/d5f2e04f2aa6247c613da0c1dd87fcd08fa17107e858193566048a1e2f0a/rpds_py-0.30.0-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:eb2c4071ab598733724c08221091e8d80e89064cd472819285a9ab0f24bcedb9", size = 396507, upload-time = "2025-11-30T20:23:16.105Z" }, + { url = "https://files.pythonhosted.org/packages/4b/9a/453255d2f769fe44e07ea9785c8347edaf867f7026872e76c1ad9f7bed92/rpds_py-0.30.0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6bdfdb946967d816e6adf9a3d8201bfad269c67efe6cefd7093ef959683c8de0", size = 414949, upload-time = "2025-11-30T20:23:17.539Z" }, + { url = "https://files.pythonhosted.org/packages/a3/31/622a86cdc0c45d6df0e9ccb6becdba5074735e7033c20e401a6d9d0e2ca0/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c77afbd5f5250bf27bf516c7c4a016813eb2d3e116139aed0096940c5982da94", size = 565790, upload-time = "2025-11-30T20:23:19.029Z" }, + { url = "https://files.pythonhosted.org/packages/1c/5d/15bbf0fb4a3f58a3b1c67855ec1efcc4ceaef4e86644665fff03e1b66d8d/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:61046904275472a76c8c90c9ccee9013d70a6d0f73eecefd38c1ae7c39045a08", size = 590217, upload-time = "2025-11-30T20:23:20.885Z" }, + { url = "https://files.pythonhosted.org/packages/6d/61/21b8c41f68e60c8cc3b2e25644f0e3681926020f11d06ab0b78e3c6bbff1/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4c5f36a861bc4b7da6516dbdf302c55313afa09b81931e8280361a4f6c9a2d27", size = 555806, upload-time = "2025-11-30T20:23:22.488Z" }, + { url = "https://files.pythonhosted.org/packages/f9/39/7e067bb06c31de48de3eb200f9fc7c58982a4d3db44b07e73963e10d3be9/rpds_py-0.30.0-cp313-cp313t-win32.whl", hash = "sha256:3d4a69de7a3e50ffc214ae16d79d8fbb0922972da0356dcf4d0fdca2878559c6", size = 211341, upload-time = "2025-11-30T20:23:24.449Z" }, + { url = "https://files.pythonhosted.org/packages/0a/4d/222ef0b46443cf4cf46764d9c630f3fe4abaa7245be9417e56e9f52b8f65/rpds_py-0.30.0-cp313-cp313t-win_amd64.whl", hash = "sha256:f14fc5df50a716f7ece6a80b6c78bb35ea2ca47c499e422aa4463455dd96d56d", size = 225768, upload-time = "2025-11-30T20:23:25.908Z" }, + { url = "https://files.pythonhosted.org/packages/86/81/dad16382ebbd3d0e0328776d8fd7ca94220e4fa0798d1dc5e7da48cb3201/rpds_py-0.30.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:68f19c879420aa08f61203801423f6cd5ac5f0ac4ac82a2368a9fcd6a9a075e0", size = 362099, upload-time = "2025-11-30T20:23:27.316Z" }, + { url = "https://files.pythonhosted.org/packages/2b/60/19f7884db5d5603edf3c6bce35408f45ad3e97e10007df0e17dd57af18f8/rpds_py-0.30.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ec7c4490c672c1a0389d319b3a9cfcd098dcdc4783991553c332a15acf7249be", size = 353192, upload-time = "2025-11-30T20:23:29.151Z" }, + { url = "https://files.pythonhosted.org/packages/bf/c4/76eb0e1e72d1a9c4703c69607cec123c29028bff28ce41588792417098ac/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f251c812357a3fed308d684a5079ddfb9d933860fc6de89f2b7ab00da481e65f", size = 384080, upload-time = "2025-11-30T20:23:30.785Z" }, + { url = "https://files.pythonhosted.org/packages/72/87/87ea665e92f3298d1b26d78814721dc39ed8d2c74b86e83348d6b48a6f31/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ac98b175585ecf4c0348fd7b29c3864bda53b805c773cbf7bfdaffc8070c976f", size = 394841, upload-time = "2025-11-30T20:23:32.209Z" }, + { url = "https://files.pythonhosted.org/packages/77/ad/7783a89ca0587c15dcbf139b4a8364a872a25f861bdb88ed99f9b0dec985/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3e62880792319dbeb7eb866547f2e35973289e7d5696c6e295476448f5b63c87", size = 516670, upload-time = "2025-11-30T20:23:33.742Z" }, + { url = "https://files.pythonhosted.org/packages/5b/3c/2882bdac942bd2172f3da574eab16f309ae10a3925644e969536553cb4ee/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4e7fc54e0900ab35d041b0601431b0a0eb495f0851a0639b6ef90f7741b39a18", size = 408005, upload-time = "2025-11-30T20:23:35.253Z" }, + { url = "https://files.pythonhosted.org/packages/ce/81/9a91c0111ce1758c92516a3e44776920b579d9a7c09b2b06b642d4de3f0f/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47e77dc9822d3ad616c3d5759ea5631a75e5809d5a28707744ef79d7a1bcfcad", size = 382112, upload-time = "2025-11-30T20:23:36.842Z" }, + { url = "https://files.pythonhosted.org/packages/cf/8e/1da49d4a107027e5fbc64daeab96a0706361a2918da10cb41769244b805d/rpds_py-0.30.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:b4dc1a6ff022ff85ecafef7979a2c6eb423430e05f1165d6688234e62ba99a07", size = 399049, upload-time = "2025-11-30T20:23:38.343Z" }, + { url = "https://files.pythonhosted.org/packages/df/5a/7ee239b1aa48a127570ec03becbb29c9d5a9eb092febbd1699d567cae859/rpds_py-0.30.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:4559c972db3a360808309e06a74628b95eaccbf961c335c8fe0d590cf587456f", size = 415661, upload-time = "2025-11-30T20:23:40.263Z" }, + { url = "https://files.pythonhosted.org/packages/70/ea/caa143cf6b772f823bc7929a45da1fa83569ee49b11d18d0ada7f5ee6fd6/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:0ed177ed9bded28f8deb6ab40c183cd1192aa0de40c12f38be4d59cd33cb5c65", size = 565606, upload-time = "2025-11-30T20:23:42.186Z" }, + { url = "https://files.pythonhosted.org/packages/64/91/ac20ba2d69303f961ad8cf55bf7dbdb4763f627291ba3d0d7d67333cced9/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ad1fa8db769b76ea911cb4e10f049d80bf518c104f15b3edb2371cc65375c46f", size = 591126, upload-time = "2025-11-30T20:23:44.086Z" }, + { url = "https://files.pythonhosted.org/packages/21/20/7ff5f3c8b00c8a95f75985128c26ba44503fb35b8e0259d812766ea966c7/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:46e83c697b1f1c72b50e5ee5adb4353eef7406fb3f2043d64c33f20ad1c2fc53", size = 553371, upload-time = "2025-11-30T20:23:46.004Z" }, + { url = "https://files.pythonhosted.org/packages/72/c7/81dadd7b27c8ee391c132a6b192111ca58d866577ce2d9b0ca157552cce0/rpds_py-0.30.0-cp314-cp314-win32.whl", hash = "sha256:ee454b2a007d57363c2dfd5b6ca4a5d7e2c518938f8ed3b706e37e5d470801ed", size = 215298, upload-time = "2025-11-30T20:23:47.696Z" }, + { url = "https://files.pythonhosted.org/packages/3e/d2/1aaac33287e8cfb07aab2e6b8ac1deca62f6f65411344f1433c55e6f3eb8/rpds_py-0.30.0-cp314-cp314-win_amd64.whl", hash = "sha256:95f0802447ac2d10bcc69f6dc28fe95fdf17940367b21d34e34c737870758950", size = 228604, upload-time = "2025-11-30T20:23:49.501Z" }, + { url = "https://files.pythonhosted.org/packages/e8/95/ab005315818cc519ad074cb7784dae60d939163108bd2b394e60dc7b5461/rpds_py-0.30.0-cp314-cp314-win_arm64.whl", hash = "sha256:613aa4771c99f03346e54c3f038e4cc574ac09a3ddfb0e8878487335e96dead6", size = 222391, upload-time = "2025-11-30T20:23:50.96Z" }, + { url = "https://files.pythonhosted.org/packages/9e/68/154fe0194d83b973cdedcdcc88947a2752411165930182ae41d983dcefa6/rpds_py-0.30.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:7e6ecfcb62edfd632e56983964e6884851786443739dbfe3582947e87274f7cb", size = 364868, upload-time = "2025-11-30T20:23:52.494Z" }, + { url = "https://files.pythonhosted.org/packages/83/69/8bbc8b07ec854d92a8b75668c24d2abcb1719ebf890f5604c61c9369a16f/rpds_py-0.30.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a1d0bc22a7cdc173fedebb73ef81e07faef93692b8c1ad3733b67e31e1b6e1b8", size = 353747, upload-time = "2025-11-30T20:23:54.036Z" }, + { url = "https://files.pythonhosted.org/packages/ab/00/ba2e50183dbd9abcce9497fa5149c62b4ff3e22d338a30d690f9af970561/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d08f00679177226c4cb8c5265012eea897c8ca3b93f429e546600c971bcbae7", size = 383795, upload-time = "2025-11-30T20:23:55.556Z" }, + { url = "https://files.pythonhosted.org/packages/05/6f/86f0272b84926bcb0e4c972262f54223e8ecc556b3224d281e6598fc9268/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5965af57d5848192c13534f90f9dd16464f3c37aaf166cc1da1cae1fd5a34898", size = 393330, upload-time = "2025-11-30T20:23:57.033Z" }, + { url = "https://files.pythonhosted.org/packages/cb/e9/0e02bb2e6dc63d212641da45df2b0bf29699d01715913e0d0f017ee29438/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9a4e86e34e9ab6b667c27f3211ca48f73dba7cd3d90f8d5b11be56e5dbc3fb4e", size = 518194, upload-time = "2025-11-30T20:23:58.637Z" }, + { url = "https://files.pythonhosted.org/packages/ee/ca/be7bca14cf21513bdf9c0606aba17d1f389ea2b6987035eb4f62bd923f25/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e5d3e6b26f2c785d65cc25ef1e5267ccbe1b069c5c21b8cc724efee290554419", size = 408340, upload-time = "2025-11-30T20:24:00.2Z" }, + { url = "https://files.pythonhosted.org/packages/c2/c7/736e00ebf39ed81d75544c0da6ef7b0998f8201b369acf842f9a90dc8fce/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:626a7433c34566535b6e56a1b39a7b17ba961e97ce3b80ec62e6f1312c025551", size = 383765, upload-time = "2025-11-30T20:24:01.759Z" }, + { url = "https://files.pythonhosted.org/packages/4a/3f/da50dfde9956aaf365c4adc9533b100008ed31aea635f2b8d7b627e25b49/rpds_py-0.30.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:acd7eb3f4471577b9b5a41baf02a978e8bdeb08b4b355273994f8b87032000a8", size = 396834, upload-time = "2025-11-30T20:24:03.687Z" }, + { url = "https://files.pythonhosted.org/packages/4e/00/34bcc2565b6020eab2623349efbdec810676ad571995911f1abdae62a3a0/rpds_py-0.30.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fe5fa731a1fa8a0a56b0977413f8cacac1768dad38d16b3a296712709476fbd5", size = 415470, upload-time = "2025-11-30T20:24:05.232Z" }, + { url = "https://files.pythonhosted.org/packages/8c/28/882e72b5b3e6f718d5453bd4d0d9cf8df36fddeb4ddbbab17869d5868616/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:74a3243a411126362712ee1524dfc90c650a503502f135d54d1b352bd01f2404", size = 565630, upload-time = "2025-11-30T20:24:06.878Z" }, + { url = "https://files.pythonhosted.org/packages/3b/97/04a65539c17692de5b85c6e293520fd01317fd878ea1995f0367d4532fb1/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:3e8eeb0544f2eb0d2581774be4c3410356eba189529a6b3e36bbbf9696175856", size = 591148, upload-time = "2025-11-30T20:24:08.445Z" }, + { url = "https://files.pythonhosted.org/packages/85/70/92482ccffb96f5441aab93e26c4d66489eb599efdcf96fad90c14bbfb976/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dbd936cde57abfee19ab3213cf9c26be06d60750e60a8e4dd85d1ab12c8b1f40", size = 556030, upload-time = "2025-11-30T20:24:10.956Z" }, + { url = "https://files.pythonhosted.org/packages/20/53/7c7e784abfa500a2b6b583b147ee4bb5a2b3747a9166bab52fec4b5b5e7d/rpds_py-0.30.0-cp314-cp314t-win32.whl", hash = "sha256:dc824125c72246d924f7f796b4f63c1e9dc810c7d9e2355864b3c3a73d59ade0", size = 211570, upload-time = "2025-11-30T20:24:12.735Z" }, + { url = "https://files.pythonhosted.org/packages/d0/02/fa464cdfbe6b26e0600b62c528b72d8608f5cc49f96b8d6e38c95d60c676/rpds_py-0.30.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27f4b0e92de5bfbc6f86e43959e6edd1425c33b5e69aab0984a72047f2bcf1e3", size = 226532, upload-time = "2025-11-30T20:24:14.634Z" }, +] + +[[package]] +name = "ruff" +version = "0.15.12" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/99/43/3291f1cc9106f4c63bdce7a8d0df5047fe8422a75b091c16b5e9355e0b11/ruff-0.15.12.tar.gz", hash = "sha256:ecea26adb26b4232c0c2ca19ccbc0083a68344180bba2a600605538ce51a40a6", size = 4643852, upload-time = "2026-04-24T18:17:14.305Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c3/6e/e78ffb61d4686f3d96ba3df2c801161843746dcbcbb17a1e927d4829312b/ruff-0.15.12-py3-none-linux_armv6l.whl", hash = "sha256:f86f176e188e94d6bdbc09f09bfd9dc729059ad93d0e7390b5a73efe19f8861c", size = 10640713, upload-time = "2026-04-24T18:17:22.841Z" }, + { url = "https://files.pythonhosted.org/packages/ae/08/a317bc231fb9e7b93e4ef3089501e51922ff88d6936ce5cf870c4fe55419/ruff-0.15.12-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:e3bcd123364c3770b8e1b7baaf343cc99a35f197c5c6e8af79015c666c423a6c", size = 11069267, upload-time = "2026-04-24T18:17:30.105Z" }, + { url = "https://files.pythonhosted.org/packages/aa/a4/f828e9718d3dce1f5f11c39c4f65afd32783c8b2aebb2e3d259e492c47bd/ruff-0.15.12-py3-none-macosx_11_0_arm64.whl", hash = "sha256:fe87510d000220aa1ed530d4448a7c696a0cae1213e5ec30e5874287b66557b5", size = 10397182, upload-time = "2026-04-24T18:17:07.177Z" }, + { url = "https://files.pythonhosted.org/packages/71/e0/3310fc6d1b5e1fdea22bf3b1b807c7e187b581021b0d7d4514cccdb5fb71/ruff-0.15.12-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:84a1630093121375a3e2a95b4a6dc7b59e2b4ee76216e32d81aae550a832d002", size = 10758012, upload-time = "2026-04-24T18:16:55.759Z" }, + { url = "https://files.pythonhosted.org/packages/11/c1/a606911aee04c324ddaa883ae418f3569792fd3c4a10c50e0dd0a2311e1e/ruff-0.15.12-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fb129f40f114f089ebe0ca56c0d251cf2061b17651d464bb6478dc01e69f11f5", size = 10447479, upload-time = "2026-04-24T18:16:51.677Z" }, + { url = "https://files.pythonhosted.org/packages/9d/68/4201e8444f0894f21ab4aeeaee68aa4f10b51613514a20d80bd628d57e88/ruff-0.15.12-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b0c862b172d695db7598426b8af465e7e9ac00a3ea2a3630ee67eb82e366aaa6", size = 11234040, upload-time = "2026-04-24T18:17:16.529Z" }, + { url = "https://files.pythonhosted.org/packages/34/ff/8a6d6cf4ccc23fd67060874e832c18919d1557a0611ebef03fdb01fff11e/ruff-0.15.12-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2849ea9f3484c3aca43a82f484210370319e7170df4dfe4843395ddf6c57bc33", size = 12087377, upload-time = "2026-04-24T18:17:04.944Z" }, + { url = "https://files.pythonhosted.org/packages/85/f6/c669cf73f5152f623d34e69866a46d5e6185816b19fcd5b6dd8a2d299922/ruff-0.15.12-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9e77c7e51c07fe396826d5969a5b846d9cd4c402535835fb6e21ce8b28fef847", size = 11367784, upload-time = "2026-04-24T18:17:25.409Z" }, + { url = "https://files.pythonhosted.org/packages/e8/39/c61d193b8a1daaa8977f7dea9e8d8ba866e02ea7b65d32f6861693aa4c12/ruff-0.15.12-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:83b2f4f2f3b1026b5fb449b467d9264bf22067b600f7b6f41fc5958909f449d0", size = 11344088, upload-time = "2026-04-24T18:17:12.258Z" }, + { url = "https://files.pythonhosted.org/packages/c2/8d/49afab3645e31e12c590acb6d3b5b69d7aab5b81926dbaf7461f9441f37a/ruff-0.15.12-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:9ba3b8f1afd7e2e43d8943e55f249e13f9682fde09711644a6e7290eb4f3e339", size = 11271770, upload-time = "2026-04-24T18:17:02.457Z" }, + { url = "https://files.pythonhosted.org/packages/46/06/33f41fe94403e2b755481cdfb9b7ef3e4e0ed031c4581124658d935d52b4/ruff-0.15.12-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:e852ba9fdc890655e1d78f2df1499efbe0e54126bd405362154a75e2bde159c5", size = 10719355, upload-time = "2026-04-24T18:17:27.648Z" }, + { url = "https://files.pythonhosted.org/packages/0d/59/18aa4e014debbf559670e4048e39260a85c7fcee84acfd761ac01e7b8d35/ruff-0.15.12-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:dd8aed930da53780d22fc70bdf84452c843cf64f8cb4eb38984319c24c5cd5fd", size = 10462758, upload-time = "2026-04-24T18:17:32.347Z" }, + { url = "https://files.pythonhosted.org/packages/25/e7/cc9f16fd0f3b5fddcbd7ec3d6ae30c8f3fde1047f32a4093a98d633c6570/ruff-0.15.12-py3-none-musllinux_1_2_i686.whl", hash = "sha256:01da3988d225628b709493d7dc67c3b9b12c0210016b08690ef9bd27970b262b", size = 10953498, upload-time = "2026-04-24T18:17:20.674Z" }, + { url = "https://files.pythonhosted.org/packages/72/7a/a9ba7f98c7a575978698f4230c5e8cc54bbc761af34f560818f933dafa0c/ruff-0.15.12-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:9cae0f92bd5700d1213188b31cd3bdd2b315361296d10b96b8e2337d3d11f53e", size = 11447765, upload-time = "2026-04-24T18:17:09.755Z" }, + { url = "https://files.pythonhosted.org/packages/ea/f9/0ae446942c846b8266059ad8a30702a35afae55f5cdc54c5adf8d7afdc27/ruff-0.15.12-py3-none-win32.whl", hash = "sha256:d0185894e038d7043ba8fd6aee7499ece6462dc0ea9f1e260c7451807c714c20", size = 10657277, upload-time = "2026-04-24T18:17:18.591Z" }, + { url = "https://files.pythonhosted.org/packages/33/f1/9614e03e1cdcbf9437570b5400ced8a720b5db22b28d8e0f1bda429f660d/ruff-0.15.12-py3-none-win_amd64.whl", hash = "sha256:c87a162d61ab3adca47c03f7f717c68672edec7d1b5499e652331780fe74950d", size = 11837758, upload-time = "2026-04-24T18:17:00.113Z" }, + { url = "https://files.pythonhosted.org/packages/c0/98/6beb4b351e472e5f4c4613f7c35a5290b8be2497e183825310c4c3a3984b/ruff-0.15.12-py3-none-win_arm64.whl", hash = "sha256:a538f7a82d061cee7be55542aca1d86d1393d55d81d4fcc314370f4340930d4f", size = 11120821, upload-time = "2026-04-24T18:16:57.979Z" }, +] + +[[package]] +name = "safetensors" +version = "0.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/29/9c/6e74567782559a63bd040a236edca26fd71bc7ba88de2ef35d75df3bca5e/safetensors-0.7.0.tar.gz", hash = "sha256:07663963b67e8bd9f0b8ad15bb9163606cd27cc5a1b96235a50d8369803b96b0", size = 200878, upload-time = "2025-11-19T15:18:43.199Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fa/47/aef6c06649039accf914afef490268e1067ed82be62bcfa5b7e886ad15e8/safetensors-0.7.0-cp38-abi3-macosx_10_12_x86_64.whl", hash = "sha256:c82f4d474cf725255d9e6acf17252991c3c8aac038d6ef363a4bf8be2f6db517", size = 467781, upload-time = "2025-11-19T15:18:35.84Z" }, + { url = "https://files.pythonhosted.org/packages/e8/00/374c0c068e30cd31f1e1b46b4b5738168ec79e7689ca82ee93ddfea05109/safetensors-0.7.0-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:94fd4858284736bb67a897a41608b5b0c2496c9bdb3bf2af1fa3409127f20d57", size = 447058, upload-time = "2025-11-19T15:18:34.416Z" }, + { url = "https://files.pythonhosted.org/packages/f1/06/578ffed52c2296f93d7fd2d844cabfa92be51a587c38c8afbb8ae449ca89/safetensors-0.7.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:e07d91d0c92a31200f25351f4acb2bc6aff7f48094e13ebb1d0fb995b54b6542", size = 491748, upload-time = "2025-11-19T15:18:09.79Z" }, + { url = "https://files.pythonhosted.org/packages/ae/33/1debbbb70e4791dde185edb9413d1fe01619255abb64b300157d7f15dddd/safetensors-0.7.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8469155f4cb518bafb4acf4865e8bb9d6804110d2d9bdcaa78564b9fd841e104", size = 503881, upload-time = "2025-11-19T15:18:16.145Z" }, + { url = "https://files.pythonhosted.org/packages/8e/1c/40c2ca924d60792c3be509833df711b553c60effbd91da6f5284a83f7122/safetensors-0.7.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:54bef08bf00a2bff599982f6b08e8770e09cc012d7bba00783fc7ea38f1fb37d", size = 623463, upload-time = "2025-11-19T15:18:21.11Z" }, + { url = "https://files.pythonhosted.org/packages/9b/3a/13784a9364bd43b0d61eef4bea2845039bc2030458b16594a1bd787ae26e/safetensors-0.7.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:42cb091236206bb2016d245c377ed383aa7f78691748f3bb6ee1bfa51ae2ce6a", size = 532855, upload-time = "2025-11-19T15:18:25.719Z" }, + { url = "https://files.pythonhosted.org/packages/a0/60/429e9b1cb3fc651937727befe258ea24122d9663e4d5709a48c9cbfceecb/safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dac7252938f0696ddea46f5e855dd3138444e82236e3be475f54929f0c510d48", size = 507152, upload-time = "2025-11-19T15:18:33.023Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a8/4b45e4e059270d17af60359713ffd83f97900d45a6afa73aaa0d737d48b6/safetensors-0.7.0-cp38-abi3-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1d060c70284127fa805085d8f10fbd0962792aed71879d00864acda69dbab981", size = 541856, upload-time = "2025-11-19T15:18:31.075Z" }, + { url = "https://files.pythonhosted.org/packages/06/87/d26d8407c44175d8ae164a95b5a62707fcc445f3c0c56108e37d98070a3d/safetensors-0.7.0-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:cdab83a366799fa730f90a4ebb563e494f28e9e92c4819e556152ad55e43591b", size = 674060, upload-time = "2025-11-19T15:18:37.211Z" }, + { url = "https://files.pythonhosted.org/packages/11/f5/57644a2ff08dc6325816ba7217e5095f17269dada2554b658442c66aed51/safetensors-0.7.0-cp38-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:672132907fcad9f2aedcb705b2d7b3b93354a2aec1b2f706c4db852abe338f85", size = 771715, upload-time = "2025-11-19T15:18:38.689Z" }, + { url = "https://files.pythonhosted.org/packages/86/31/17883e13a814bd278ae6e266b13282a01049b0c81341da7fd0e3e71a80a3/safetensors-0.7.0-cp38-abi3-musllinux_1_2_i686.whl", hash = "sha256:5d72abdb8a4d56d4020713724ba81dac065fedb7f3667151c4a637f1d3fb26c0", size = 714377, upload-time = "2025-11-19T15:18:40.162Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d8/0c8a7dc9b41dcac53c4cbf9df2b9c83e0e0097203de8b37a712b345c0be5/safetensors-0.7.0-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:b0f6d66c1c538d5a94a73aa9ddca8ccc4227e6c9ff555322ea40bdd142391dd4", size = 677368, upload-time = "2025-11-19T15:18:41.627Z" }, + { url = "https://files.pythonhosted.org/packages/05/e5/cb4b713c8a93469e3c5be7c3f8d77d307e65fe89673e731f5c2bfd0a9237/safetensors-0.7.0-cp38-abi3-win32.whl", hash = "sha256:c74af94bf3ac15ac4d0f2a7c7b4663a15f8c2ab15ed0fc7531ca61d0835eccba", size = 326423, upload-time = "2025-11-19T15:18:45.74Z" }, + { url = "https://files.pythonhosted.org/packages/5d/e6/ec8471c8072382cb91233ba7267fd931219753bb43814cbc71757bfd4dab/safetensors-0.7.0-cp38-abi3-win_amd64.whl", hash = "sha256:d1239932053f56f3456f32eb9625590cc7582e905021f94636202a864d470755", size = 341380, upload-time = "2025-11-19T15:18:44.427Z" }, +] + +[[package]] +name = "scikit-learn" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "joblib" }, + { name = "numpy" }, + { name = "scipy" }, + { name = "threadpoolctl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/d4/40988bf3b8e34feec1d0e6a051446b1f66225f8529b9309becaeef62b6c4/scikit_learn-1.8.0.tar.gz", hash = "sha256:9bccbb3b40e3de10351f8f5068e105d0f4083b1a65fa07b6634fbc401a6287fd", size = 7335585, upload-time = "2025-12-10T07:08:53.618Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/90/74/e6a7cc4b820e95cc38cf36cd74d5aa2b42e8ffc2d21fe5a9a9c45c1c7630/scikit_learn-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5fb63362b5a7ddab88e52b6dbb47dac3fd7dafeee740dc6c8d8a446ddedade8e", size = 8548242, upload-time = "2025-12-10T07:07:51.568Z" }, + { url = "https://files.pythonhosted.org/packages/49/d8/9be608c6024d021041c7f0b3928d4749a706f4e2c3832bbede4fb4f58c95/scikit_learn-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:5025ce924beccb28298246e589c691fe1b8c1c96507e6d27d12c5fadd85bfd76", size = 8079075, upload-time = "2025-12-10T07:07:53.697Z" }, + { url = "https://files.pythonhosted.org/packages/dd/47/f187b4636ff80cc63f21cd40b7b2d177134acaa10f6bb73746130ee8c2e5/scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4496bb2cf7a43ce1a2d7524a79e40bc5da45cf598dbf9545b7e8316ccba47bb4", size = 8660492, upload-time = "2025-12-10T07:07:55.574Z" }, + { url = "https://files.pythonhosted.org/packages/97/74/b7a304feb2b49df9fafa9382d4d09061a96ee9a9449a7cbea7988dda0828/scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a0bcfe4d0d14aec44921545fd2af2338c7471de9cb701f1da4c9d85906ab847a", size = 8931904, upload-time = "2025-12-10T07:07:57.666Z" }, + { url = "https://files.pythonhosted.org/packages/9f/c4/0ab22726a04ede56f689476b760f98f8f46607caecff993017ac1b64aa5d/scikit_learn-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:35c007dedb2ffe38fe3ee7d201ebac4a2deccd2408e8621d53067733e3c74809", size = 8019359, upload-time = "2025-12-10T07:07:59.838Z" }, + { url = "https://files.pythonhosted.org/packages/24/90/344a67811cfd561d7335c1b96ca21455e7e472d281c3c279c4d3f2300236/scikit_learn-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:8c497fff237d7b4e07e9ef1a640887fa4fb765647f86fbe00f969ff6280ce2bb", size = 7641898, upload-time = "2025-12-10T07:08:01.36Z" }, + { url = "https://files.pythonhosted.org/packages/03/aa/e22e0768512ce9255eba34775be2e85c2048da73da1193e841707f8f039c/scikit_learn-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0d6ae97234d5d7079dc0040990a6f7aeb97cb7fa7e8945f1999a429b23569e0a", size = 8513770, upload-time = "2025-12-10T07:08:03.251Z" }, + { url = "https://files.pythonhosted.org/packages/58/37/31b83b2594105f61a381fc74ca19e8780ee923be2d496fcd8d2e1147bd99/scikit_learn-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:edec98c5e7c128328124a029bceb09eda2d526997780fef8d65e9a69eead963e", size = 8044458, upload-time = "2025-12-10T07:08:05.336Z" }, + { url = "https://files.pythonhosted.org/packages/2d/5a/3f1caed8765f33eabb723596666da4ebbf43d11e96550fb18bdec42b467b/scikit_learn-1.8.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:74b66d8689d52ed04c271e1329f0c61635bcaf5b926db9b12d58914cdc01fe57", size = 8610341, upload-time = "2025-12-10T07:08:07.732Z" }, + { url = "https://files.pythonhosted.org/packages/38/cf/06896db3f71c75902a8e9943b444a56e727418f6b4b4a90c98c934f51ed4/scikit_learn-1.8.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8fdf95767f989b0cfedb85f7ed8ca215d4be728031f56ff5a519ee1e3276dc2e", size = 8900022, upload-time = "2025-12-10T07:08:09.862Z" }, + { url = "https://files.pythonhosted.org/packages/1c/f9/9b7563caf3ec8873e17a31401858efab6b39a882daf6c1bfa88879c0aa11/scikit_learn-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:2de443b9373b3b615aec1bb57f9baa6bb3a9bd093f1269ba95c17d870422b271", size = 7989409, upload-time = "2025-12-10T07:08:12.028Z" }, + { url = "https://files.pythonhosted.org/packages/49/bd/1f4001503650e72c4f6009ac0c4413cb17d2d601cef6f71c0453da2732fc/scikit_learn-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:eddde82a035681427cbedded4e6eff5e57fa59216c2e3e90b10b19ab1d0a65c3", size = 7619760, upload-time = "2025-12-10T07:08:13.688Z" }, + { url = "https://files.pythonhosted.org/packages/d2/7d/a630359fc9dcc95496588c8d8e3245cc8fd81980251079bc09c70d41d951/scikit_learn-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:7cc267b6108f0a1499a734167282c00c4ebf61328566b55ef262d48e9849c735", size = 8826045, upload-time = "2025-12-10T07:08:15.215Z" }, + { url = "https://files.pythonhosted.org/packages/cc/56/a0c86f6930cfcd1c7054a2bc417e26960bb88d32444fe7f71d5c2cfae891/scikit_learn-1.8.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:fe1c011a640a9f0791146011dfd3c7d9669785f9fed2b2a5f9e207536cf5c2fd", size = 8420324, upload-time = "2025-12-10T07:08:17.561Z" }, + { url = "https://files.pythonhosted.org/packages/46/1e/05962ea1cebc1cf3876667ecb14c283ef755bf409993c5946ade3b77e303/scikit_learn-1.8.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72358cce49465d140cc4e7792015bb1f0296a9742d5622c67e31399b75468b9e", size = 8680651, upload-time = "2025-12-10T07:08:19.952Z" }, + { url = "https://files.pythonhosted.org/packages/fe/56/a85473cd75f200c9759e3a5f0bcab2d116c92a8a02ee08ccd73b870f8bb4/scikit_learn-1.8.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80832434a6cc114f5219211eec13dcbc16c2bac0e31ef64c6d346cde3cf054cb", size = 8925045, upload-time = "2025-12-10T07:08:22.11Z" }, + { url = "https://files.pythonhosted.org/packages/cc/b7/64d8cfa896c64435ae57f4917a548d7ac7a44762ff9802f75a79b77cb633/scikit_learn-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ee787491dbfe082d9c3013f01f5991658b0f38aa8177e4cd4bf434c58f551702", size = 8507994, upload-time = "2025-12-10T07:08:23.943Z" }, + { url = "https://files.pythonhosted.org/packages/5e/37/e192ea709551799379958b4c4771ec507347027bb7c942662c7fbeba31cb/scikit_learn-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf97c10a3f5a7543f9b88cbf488d33d175e9146115a451ae34568597ba33dcde", size = 7869518, upload-time = "2025-12-10T07:08:25.71Z" }, + { url = "https://files.pythonhosted.org/packages/24/05/1af2c186174cc92dcab2233f327336058c077d38f6fe2aceb08e6ab4d509/scikit_learn-1.8.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c22a2da7a198c28dd1a6e1136f19c830beab7fdca5b3e5c8bba8394f8a5c45b3", size = 8528667, upload-time = "2025-12-10T07:08:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a8/25/01c0af38fe969473fb292bba9dc2b8f9b451f3112ff242c647fee3d0dfe7/scikit_learn-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:6b595b07a03069a2b1740dc08c2299993850ea81cce4fe19b2421e0c970de6b7", size = 8066524, upload-time = "2025-12-10T07:08:29.822Z" }, + { url = "https://files.pythonhosted.org/packages/be/ce/a0623350aa0b68647333940ee46fe45086c6060ec604874e38e9ab7d8e6c/scikit_learn-1.8.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:29ffc74089f3d5e87dfca4c2c8450f88bdc61b0fc6ed5d267f3988f19a1309f6", size = 8657133, upload-time = "2025-12-10T07:08:31.865Z" }, + { url = "https://files.pythonhosted.org/packages/b8/cb/861b41341d6f1245e6ca80b1c1a8c4dfce43255b03df034429089ca2a2c5/scikit_learn-1.8.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fb65db5d7531bccf3a4f6bec3462223bea71384e2cda41da0f10b7c292b9e7c4", size = 8923223, upload-time = "2025-12-10T07:08:34.166Z" }, + { url = "https://files.pythonhosted.org/packages/76/18/a8def8f91b18cd1ba6e05dbe02540168cb24d47e8dcf69e8d00b7da42a08/scikit_learn-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:56079a99c20d230e873ea40753102102734c5953366972a71d5cb39a32bc40c6", size = 8096518, upload-time = "2025-12-10T07:08:36.339Z" }, + { url = "https://files.pythonhosted.org/packages/d1/77/482076a678458307f0deb44e29891d6022617b2a64c840c725495bee343f/scikit_learn-1.8.0-cp314-cp314-win_arm64.whl", hash = "sha256:3bad7565bc9cf37ce19a7c0d107742b320c1285df7aab1a6e2d28780df167242", size = 7754546, upload-time = "2025-12-10T07:08:38.128Z" }, + { url = "https://files.pythonhosted.org/packages/2d/d1/ef294ca754826daa043b2a104e59960abfab4cf653891037d19dd5b6f3cf/scikit_learn-1.8.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4511be56637e46c25721e83d1a9cea9614e7badc7040c4d573d75fbe257d6fd7", size = 8848305, upload-time = "2025-12-10T07:08:41.013Z" }, + { url = "https://files.pythonhosted.org/packages/5b/e2/b1f8b05138ee813b8e1a4149f2f0d289547e60851fd1bb268886915adbda/scikit_learn-1.8.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:a69525355a641bf8ef136a7fa447672fb54fe8d60cab5538d9eb7c6438543fb9", size = 8432257, upload-time = "2025-12-10T07:08:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/26/11/c32b2138a85dcb0c99f6afd13a70a951bfdff8a6ab42d8160522542fb647/scikit_learn-1.8.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c2656924ec73e5939c76ac4c8b026fc203b83d8900362eb2599d8aee80e4880f", size = 8678673, upload-time = "2025-12-10T07:08:45.362Z" }, + { url = "https://files.pythonhosted.org/packages/c7/57/51f2384575bdec454f4fe4e7a919d696c9ebce914590abf3e52d47607ab8/scikit_learn-1.8.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15fc3b5d19cc2be65404786857f2e13c70c83dd4782676dd6814e3b89dc8f5b9", size = 8922467, upload-time = "2025-12-10T07:08:47.408Z" }, + { url = "https://files.pythonhosted.org/packages/35/4d/748c9e2872637a57981a04adc038dacaa16ba8ca887b23e34953f0b3f742/scikit_learn-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:00d6f1d66fbcf4eba6e356e1420d33cc06c70a45bb1363cd6f6a8e4ebbbdece2", size = 8774395, upload-time = "2025-12-10T07:08:49.337Z" }, + { url = "https://files.pythonhosted.org/packages/60/22/d7b2ebe4704a5e50790ba089d5c2ae308ab6bb852719e6c3bd4f04c3a363/scikit_learn-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f28dd15c6bb0b66ba09728cf09fd8736c304be29409bd8445a080c1280619e8c", size = 8002647, upload-time = "2025-12-10T07:08:51.601Z" }, +] + +[[package]] +name = "scipy" +version = "1.17.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7a/97/5a3609c4f8d58b039179648e62dd220f89864f56f7357f5d4f45c29eb2cc/scipy-1.17.1.tar.gz", hash = "sha256:95d8e012d8cb8816c226aef832200b1d45109ed4464303e997c5b13122b297c0", size = 30573822, upload-time = "2026-02-23T00:26:24.851Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/35/48/b992b488d6f299dbe3f11a20b24d3dda3d46f1a635ede1c46b5b17a7b163/scipy-1.17.1-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:35c3a56d2ef83efc372eaec584314bd0ef2e2f0d2adb21c55e6ad5b344c0dcb8", size = 31610954, upload-time = "2026-02-23T00:17:49.855Z" }, + { url = "https://files.pythonhosted.org/packages/b2/02/cf107b01494c19dc100f1d0b7ac3cc08666e96ba2d64db7626066cee895e/scipy-1.17.1-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:fcb310ddb270a06114bb64bbe53c94926b943f5b7f0842194d585c65eb4edd76", size = 28172662, upload-time = "2026-02-23T00:18:01.64Z" }, + { url = "https://files.pythonhosted.org/packages/cf/a9/599c28631bad314d219cf9ffd40e985b24d603fc8a2f4ccc5ae8419a535b/scipy-1.17.1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:cc90d2e9c7e5c7f1a482c9875007c095c3194b1cfedca3c2f3291cdc2bc7c086", size = 20344366, upload-time = "2026-02-23T00:18:12.015Z" }, + { url = "https://files.pythonhosted.org/packages/35/f5/906eda513271c8deb5af284e5ef0206d17a96239af79f9fa0aebfe0e36b4/scipy-1.17.1-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:c80be5ede8f3f8eded4eff73cc99a25c388ce98e555b17d31da05287015ffa5b", size = 22704017, upload-time = "2026-02-23T00:18:21.502Z" }, + { url = "https://files.pythonhosted.org/packages/da/34/16f10e3042d2f1d6b66e0428308ab52224b6a23049cb2f5c1756f713815f/scipy-1.17.1-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e19ebea31758fac5893a2ac360fedd00116cbb7628e650842a6691ba7ca28a21", size = 32927842, upload-time = "2026-02-23T00:18:35.367Z" }, + { url = "https://files.pythonhosted.org/packages/01/8e/1e35281b8ab6d5d72ebe9911edcdffa3f36b04ed9d51dec6dd140396e220/scipy-1.17.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:02ae3b274fde71c5e92ac4d54bc06c42d80e399fec704383dcd99b301df37458", size = 35235890, upload-time = "2026-02-23T00:18:49.188Z" }, + { url = "https://files.pythonhosted.org/packages/c5/5c/9d7f4c88bea6e0d5a4f1bc0506a53a00e9fcb198de372bfe4d3652cef482/scipy-1.17.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:8a604bae87c6195d8b1045eddece0514d041604b14f2727bbc2b3020172045eb", size = 35003557, upload-time = "2026-02-23T00:18:54.74Z" }, + { url = "https://files.pythonhosted.org/packages/65/94/7698add8f276dbab7a9de9fb6b0e02fc13ee61d51c7c3f85ac28b65e1239/scipy-1.17.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f590cd684941912d10becc07325a3eeb77886fe981415660d9265c4c418d0bea", size = 37625856, upload-time = "2026-02-23T00:19:00.307Z" }, + { url = "https://files.pythonhosted.org/packages/a2/84/dc08d77fbf3d87d3ee27f6a0c6dcce1de5829a64f2eae85a0ecc1f0daa73/scipy-1.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:41b71f4a3a4cab9d366cd9065b288efc4d4f3c0b37a91a8e0947fb5bd7f31d87", size = 36549682, upload-time = "2026-02-23T00:19:07.67Z" }, + { url = "https://files.pythonhosted.org/packages/bc/98/fe9ae9ffb3b54b62559f52dedaebe204b408db8109a8c66fdd04869e6424/scipy-1.17.1-cp312-cp312-win_arm64.whl", hash = "sha256:f4115102802df98b2b0db3cce5cb9b92572633a1197c77b7553e5203f284a5b3", size = 24547340, upload-time = "2026-02-23T00:19:12.024Z" }, + { url = "https://files.pythonhosted.org/packages/76/27/07ee1b57b65e92645f219b37148a7e7928b82e2b5dbeccecb4dff7c64f0b/scipy-1.17.1-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:5e3c5c011904115f88a39308379c17f91546f77c1667cea98739fe0fccea804c", size = 31590199, upload-time = "2026-02-23T00:19:17.192Z" }, + { url = "https://files.pythonhosted.org/packages/ec/ae/db19f8ab842e9b724bf5dbb7db29302a91f1e55bc4d04b1025d6d605a2c5/scipy-1.17.1-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:6fac755ca3d2c3edcb22f479fceaa241704111414831ddd3bc6056e18516892f", size = 28154001, upload-time = "2026-02-23T00:19:22.241Z" }, + { url = "https://files.pythonhosted.org/packages/5b/58/3ce96251560107b381cbd6e8413c483bbb1228a6b919fa8652b0d4090e7f/scipy-1.17.1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:7ff200bf9d24f2e4d5dc6ee8c3ac64d739d3a89e2326ba68aaf6c4a2b838fd7d", size = 20325719, upload-time = "2026-02-23T00:19:26.329Z" }, + { url = "https://files.pythonhosted.org/packages/b2/83/15087d945e0e4d48ce2377498abf5ad171ae013232ae31d06f336e64c999/scipy-1.17.1-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:4b400bdc6f79fa02a4d86640310dde87a21fba0c979efff5248908c6f15fad1b", size = 22683595, upload-time = "2026-02-23T00:19:30.304Z" }, + { url = "https://files.pythonhosted.org/packages/b4/e0/e58fbde4a1a594c8be8114eb4aac1a55bcd6587047efc18a61eb1f5c0d30/scipy-1.17.1-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2b64ca7d4aee0102a97f3ba22124052b4bd2152522355073580bf4845e2550b6", size = 32896429, upload-time = "2026-02-23T00:19:35.536Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5f/f17563f28ff03c7b6799c50d01d5d856a1d55f2676f537ca8d28c7f627cd/scipy-1.17.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:581b2264fc0aa555f3f435a5944da7504ea3a065d7029ad60e7c3d1ae09c5464", size = 35203952, upload-time = "2026-02-23T00:19:42.259Z" }, + { url = "https://files.pythonhosted.org/packages/8d/a5/9afd17de24f657fdfe4df9a3f1ea049b39aef7c06000c13db1530d81ccca/scipy-1.17.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:beeda3d4ae615106d7094f7e7cef6218392e4465cc95d25f900bebabfded0950", size = 34979063, upload-time = "2026-02-23T00:19:47.547Z" }, + { url = "https://files.pythonhosted.org/packages/8b/13/88b1d2384b424bf7c924f2038c1c409f8d88bb2a8d49d097861dd64a57b2/scipy-1.17.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6609bc224e9568f65064cfa72edc0f24ee6655b47575954ec6339534b2798369", size = 37598449, upload-time = "2026-02-23T00:19:53.238Z" }, + { url = "https://files.pythonhosted.org/packages/35/e5/d6d0e51fc888f692a35134336866341c08655d92614f492c6860dc45bb2c/scipy-1.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:37425bc9175607b0268f493d79a292c39f9d001a357bebb6b88fdfaff13f6448", size = 36510943, upload-time = "2026-02-23T00:20:50.89Z" }, + { url = "https://files.pythonhosted.org/packages/2a/fd/3be73c564e2a01e690e19cc618811540ba5354c67c8680dce3281123fb79/scipy-1.17.1-cp313-cp313-win_arm64.whl", hash = "sha256:5cf36e801231b6a2059bf354720274b7558746f3b1a4efb43fcf557ccd484a87", size = 24545621, upload-time = "2026-02-23T00:20:55.871Z" }, + { url = "https://files.pythonhosted.org/packages/6f/6b/17787db8b8114933a66f9dcc479a8272e4b4da75fe03b0c282f7b0ade8cd/scipy-1.17.1-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:d59c30000a16d8edc7e64152e30220bfbd724c9bbb08368c054e24c651314f0a", size = 31936708, upload-time = "2026-02-23T00:19:58.694Z" }, + { url = "https://files.pythonhosted.org/packages/38/2e/524405c2b6392765ab1e2b722a41d5da33dc5c7b7278184a8ad29b6cb206/scipy-1.17.1-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:010f4333c96c9bb1a4516269e33cb5917b08ef2166d5556ca2fd9f082a9e6ea0", size = 28570135, upload-time = "2026-02-23T00:20:03.934Z" }, + { url = "https://files.pythonhosted.org/packages/fd/c3/5bd7199f4ea8556c0c8e39f04ccb014ac37d1468e6cfa6a95c6b3562b76e/scipy-1.17.1-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:2ceb2d3e01c5f1d83c4189737a42d9cb2fc38a6eeed225e7515eef71ad301dce", size = 20741977, upload-time = "2026-02-23T00:20:07.935Z" }, + { url = "https://files.pythonhosted.org/packages/d9/b8/8ccd9b766ad14c78386599708eb745f6b44f08400a5fd0ade7cf89b6fc93/scipy-1.17.1-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:844e165636711ef41f80b4103ed234181646b98a53c8f05da12ca5ca289134f6", size = 23029601, upload-time = "2026-02-23T00:20:12.161Z" }, + { url = "https://files.pythonhosted.org/packages/6d/a0/3cb6f4d2fb3e17428ad2880333cac878909ad1a89f678527b5328b93c1d4/scipy-1.17.1-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:158dd96d2207e21c966063e1635b1063cd7787b627b6f07305315dd73d9c679e", size = 33019667, upload-time = "2026-02-23T00:20:17.208Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c3/2d834a5ac7bf3a0c806ad1508efc02dda3c8c61472a56132d7894c312dea/scipy-1.17.1-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74cbb80d93260fe2ffa334efa24cb8f2f0f622a9b9febf8b483c0b865bfb3475", size = 35264159, upload-time = "2026-02-23T00:20:23.087Z" }, + { url = "https://files.pythonhosted.org/packages/4d/77/d3ed4becfdbd217c52062fafe35a72388d1bd82c2d0ba5ca19d6fcc93e11/scipy-1.17.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:dbc12c9f3d185f5c737d801da555fb74b3dcfa1a50b66a1a93e09190f41fab50", size = 35102771, upload-time = "2026-02-23T00:20:28.636Z" }, + { url = "https://files.pythonhosted.org/packages/bd/12/d19da97efde68ca1ee5538bb261d5d2c062f0c055575128f11a2730e3ac1/scipy-1.17.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:94055a11dfebe37c656e70317e1996dc197e1a15bbcc351bcdd4610e128fe1ca", size = 37665910, upload-time = "2026-02-23T00:20:34.743Z" }, + { url = "https://files.pythonhosted.org/packages/06/1c/1172a88d507a4baaf72c5a09bb6c018fe2ae0ab622e5830b703a46cc9e44/scipy-1.17.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e30bdeaa5deed6bc27b4cc490823cd0347d7dae09119b8803ae576ea0ce52e4c", size = 36562980, upload-time = "2026-02-23T00:20:40.575Z" }, + { url = "https://files.pythonhosted.org/packages/70/b0/eb757336e5a76dfa7911f63252e3b7d1de00935d7705cf772db5b45ec238/scipy-1.17.1-cp313-cp313t-win_arm64.whl", hash = "sha256:a720477885a9d2411f94a93d16f9d89bad0f28ca23c3f8daa521e2dcc3f44d49", size = 24856543, upload-time = "2026-02-23T00:20:45.313Z" }, + { url = "https://files.pythonhosted.org/packages/cf/83/333afb452af6f0fd70414dc04f898647ee1423979ce02efa75c3b0f2c28e/scipy-1.17.1-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:a48a72c77a310327f6a3a920092fa2b8fd03d7deaa60f093038f22d98e096717", size = 31584510, upload-time = "2026-02-23T00:21:01.015Z" }, + { url = "https://files.pythonhosted.org/packages/ed/a6/d05a85fd51daeb2e4ea71d102f15b34fedca8e931af02594193ae4fd25f7/scipy-1.17.1-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:45abad819184f07240d8a696117a7aacd39787af9e0b719d00285549ed19a1e9", size = 28170131, upload-time = "2026-02-23T00:21:05.888Z" }, + { url = "https://files.pythonhosted.org/packages/db/7b/8624a203326675d7746a254083a187398090a179335b2e4a20e2ddc46e83/scipy-1.17.1-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:3fd1fcdab3ea951b610dc4cef356d416d5802991e7e32b5254828d342f7b7e0b", size = 20342032, upload-time = "2026-02-23T00:21:09.904Z" }, + { url = "https://files.pythonhosted.org/packages/c9/35/2c342897c00775d688d8ff3987aced3426858fd89d5a0e26e020b660b301/scipy-1.17.1-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:7bdf2da170b67fdf10bca777614b1c7d96ae3ca5794fd9587dce41eb2966e866", size = 22678766, upload-time = "2026-02-23T00:21:14.313Z" }, + { url = "https://files.pythonhosted.org/packages/ef/f2/7cdb8eb308a1a6ae1e19f945913c82c23c0c442a462a46480ce487fdc0ac/scipy-1.17.1-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:adb2642e060a6549c343603a3851ba76ef0b74cc8c079a9a58121c7ec9fe2350", size = 32957007, upload-time = "2026-02-23T00:21:19.663Z" }, + { url = "https://files.pythonhosted.org/packages/0b/2e/7eea398450457ecb54e18e9d10110993fa65561c4f3add5e8eccd2b9cd41/scipy-1.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:eee2cfda04c00a857206a4330f0c5e3e56535494e30ca445eb19ec624ae75118", size = 35221333, upload-time = "2026-02-23T00:21:25.278Z" }, + { url = "https://files.pythonhosted.org/packages/d9/77/5b8509d03b77f093a0d52e606d3c4f79e8b06d1d38c441dacb1e26cacf46/scipy-1.17.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d2650c1fb97e184d12d8ba010493ee7b322864f7d3d00d3f9bb97d9c21de4068", size = 35042066, upload-time = "2026-02-23T00:21:31.358Z" }, + { url = "https://files.pythonhosted.org/packages/f9/df/18f80fb99df40b4070328d5ae5c596f2f00fffb50167e31439e932f29e7d/scipy-1.17.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:08b900519463543aa604a06bec02461558a6e1cef8fdbb8098f77a48a83c8118", size = 37612763, upload-time = "2026-02-23T00:21:37.247Z" }, + { url = "https://files.pythonhosted.org/packages/4b/39/f0e8ea762a764a9dc52aa7dabcfad51a354819de1f0d4652b6a1122424d6/scipy-1.17.1-cp314-cp314-win_amd64.whl", hash = "sha256:3877ac408e14da24a6196de0ddcace62092bfc12a83823e92e49e40747e52c19", size = 37290984, upload-time = "2026-02-23T00:22:35.023Z" }, + { url = "https://files.pythonhosted.org/packages/7c/56/fe201e3b0f93d1a8bcf75d3379affd228a63d7e2d80ab45467a74b494947/scipy-1.17.1-cp314-cp314-win_arm64.whl", hash = "sha256:f8885db0bc2bffa59d5c1b72fad7a6a92d3e80e7257f967dd81abb553a90d293", size = 25192877, upload-time = "2026-02-23T00:22:39.798Z" }, + { url = "https://files.pythonhosted.org/packages/96/ad/f8c414e121f82e02d76f310f16db9899c4fcde36710329502a6b2a3c0392/scipy-1.17.1-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:1cc682cea2ae55524432f3cdff9e9a3be743d52a7443d0cba9017c23c87ae2f6", size = 31949750, upload-time = "2026-02-23T00:21:42.289Z" }, + { url = "https://files.pythonhosted.org/packages/7c/b0/c741e8865d61b67c81e255f4f0a832846c064e426636cd7de84e74d209be/scipy-1.17.1-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:2040ad4d1795a0ae89bfc7e8429677f365d45aa9fd5e4587cf1ea737f927b4a1", size = 28585858, upload-time = "2026-02-23T00:21:47.706Z" }, + { url = "https://files.pythonhosted.org/packages/ed/1b/3985219c6177866628fa7c2595bfd23f193ceebbe472c98a08824b9466ff/scipy-1.17.1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:131f5aaea57602008f9822e2115029b55d4b5f7c070287699fe45c661d051e39", size = 20757723, upload-time = "2026-02-23T00:21:52.039Z" }, + { url = "https://files.pythonhosted.org/packages/c0/19/2a04aa25050d656d6f7b9e7b685cc83d6957fb101665bfd9369ca6534563/scipy-1.17.1-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:9cdc1a2fcfd5c52cfb3045feb399f7b3ce822abdde3a193a6b9a60b3cb5854ca", size = 23043098, upload-time = "2026-02-23T00:21:56.185Z" }, + { url = "https://files.pythonhosted.org/packages/86/f1/3383beb9b5d0dbddd030335bf8a8b32d4317185efe495374f134d8be6cce/scipy-1.17.1-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e3dcd57ab780c741fde8dc68619de988b966db759a3c3152e8e9142c26295ad", size = 33030397, upload-time = "2026-02-23T00:22:01.404Z" }, + { url = "https://files.pythonhosted.org/packages/41/68/8f21e8a65a5a03f25a79165ec9d2b28c00e66dc80546cf5eb803aeeff35b/scipy-1.17.1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a9956e4d4f4a301ebf6cde39850333a6b6110799d470dbbb1e25326ac447f52a", size = 35281163, upload-time = "2026-02-23T00:22:07.024Z" }, + { url = "https://files.pythonhosted.org/packages/84/8d/c8a5e19479554007a5632ed7529e665c315ae7492b4f946b0deb39870e39/scipy-1.17.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:a4328d245944d09fd639771de275701ccadf5f781ba0ff092ad141e017eccda4", size = 35116291, upload-time = "2026-02-23T00:22:12.585Z" }, + { url = "https://files.pythonhosted.org/packages/52/52/e57eceff0e342a1f50e274264ed47497b59e6a4e3118808ee58ddda7b74a/scipy-1.17.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a77cbd07b940d326d39a1d1b37817e2ee4d79cb30e7338f3d0cddffae70fcaa2", size = 37682317, upload-time = "2026-02-23T00:22:18.513Z" }, + { url = "https://files.pythonhosted.org/packages/11/2f/b29eafe4a3fbc3d6de9662b36e028d5f039e72d345e05c250e121a230dd4/scipy-1.17.1-cp314-cp314t-win_amd64.whl", hash = "sha256:eb092099205ef62cd1782b006658db09e2fed75bffcae7cc0d44052d8aa0f484", size = 37345327, upload-time = "2026-02-23T00:22:24.442Z" }, + { url = "https://files.pythonhosted.org/packages/07/39/338d9219c4e87f3e708f18857ecd24d22a0c3094752393319553096b98af/scipy-1.17.1-cp314-cp314t-win_arm64.whl", hash = "sha256:200e1050faffacc162be6a486a984a0497866ec54149a01270adc8a59b7c7d21", size = 25489165, upload-time = "2026-02-23T00:22:29.563Z" }, +] + +[[package]] +name = "seaborn" +version = "0.13.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "matplotlib" }, + { name = "numpy" }, + { name = "pandas" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/86/59/a451d7420a77ab0b98f7affa3a1d78a313d2f7281a57afb1a34bae8ab412/seaborn-0.13.2.tar.gz", hash = "sha256:93e60a40988f4d65e9f4885df477e2fdaff6b73a9ded434c1ab356dd57eefff7", size = 1457696, upload-time = "2024-01-25T13:21:52.551Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/83/11/00d3c3dfc25ad54e731d91449895a79e4bf2384dc3ac01809010ba88f6d5/seaborn-0.13.2-py3-none-any.whl", hash = "sha256:636f8336facf092165e27924f223d3c62ca560b1f2bb5dff7ab7fad265361987", size = 294914, upload-time = "2024-01-25T13:21:49.598Z" }, +] + +[[package]] +name = "sentence-transformers" +version = "5.5.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "scikit-learn" }, + { name = "scipy" }, + { name = "torch", version = "2.11.0", source = { registry = "https://pypi.org/simple" }, marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torch", version = "2.12.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torch", version = "2.12.0", source = { registry = "https://pypi.org/simple" }, marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torch", version = "2.12.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "torch", version = "2.12.0+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" }, marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "tqdm" }, + { name = "transformers" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2c/27/16d127a61303e05847d878b23687f3371868c76e738557fa80b4373a8c2b/sentence_transformers-5.5.0.tar.gz", hash = "sha256:9cec675e68bfe09d07466d1f13ab06d1d79d60a0f45b154baf433bde6ae159cb", size = 444908, upload-time = "2026-05-12T14:05:42.383Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/55/20/18416624bcbae866ec0b111979766cebabe8e5ff7563ab953ecbaf3ff9e7/sentence_transformers-5.5.0-py3-none-any.whl", hash = "sha256:75313fdcc2397ec4b58297c25d6187fcca5a6b2aeb09570a72eff5a3223d8d58", size = 588665, upload-time = "2026-05-12T14:05:40.899Z" }, +] + +[[package]] +name = "setuptools" +version = "81.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/0d/1c/73e719955c59b8e424d015ab450f51c0af856ae46ea2da83eba51cc88de1/setuptools-81.0.0.tar.gz", hash = "sha256:487b53915f52501f0a79ccfd0c02c165ffe06631443a886740b91af4b7a5845a", size = 1198299, upload-time = "2026-02-06T21:10:39.601Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e1/e3/c164c88b2e5ce7b24d667b9bd83589cf4f3520d97cad01534cd3c4f55fdb/setuptools-81.0.0-py3-none-any.whl", hash = "sha256:fdd925d5c5d9f62e4b74b30d6dd7828ce236fd6ed998a08d81de62ce5a6310d6", size = 1062021, upload-time = "2026-02-06T21:10:37.175Z" }, +] + +[[package]] +name = "shellingham" +version = "1.5.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/58/15/8b3609fd3830ef7b27b655beb4b4e9c62313a4e8da8c676e142cc210d58e/shellingham-1.5.4.tar.gz", hash = "sha256:8dbca0739d487e5bd35ab3ca4b36e11c4078f3a234bfce294b0a0291363404de", size = 10310, upload-time = "2023-10-24T04:13:40.426Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/e0/f9/0595336914c5619e5f28a1fb793285925a8cd4b432c9da0a987836c7f822/shellingham-1.5.4-py2.py3-none-any.whl", hash = "sha256:7ecfff8f2fd72616f7481040475a65b2bf8af90a56c89140852d1120324e8686", size = 9755, upload-time = "2023-10-24T04:13:38.866Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "smmap" +version = "5.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" }, +] + +[[package]] +name = "sparqlwrapper" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "rdflib" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/4e/cc/453752fffa759ef41a3ceadb3f167e13dae1a74c1db057d9f6a7affa9240/SPARQLWrapper-2.0.0.tar.gz", hash = "sha256:3fed3ebcc77617a4a74d2644b86fd88e0f32e7f7003ac7b2b334c026201731f1", size = 98429, upload-time = "2022-03-13T23:14:00.671Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/31/89/176e3db96e31e795d7dfd91dd67749d3d1f0316bb30c6931a6140e1a0477/SPARQLWrapper-2.0.0-py3-none-any.whl", hash = "sha256:c99a7204fff676ee28e6acef327dc1ff8451c6f7217dcd8d49e8872f324a8a20", size = 28620, upload-time = "2022-03-13T23:13:58.969Z" }, +] + +[[package]] +name = "starlette" +version = "1.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/81/69/17425771797c36cded50b7fe44e850315d039f28b15901ab44839e70b593/starlette-1.0.0.tar.gz", hash = "sha256:6a4beaf1f81bb472fd19ea9b918b50dc3a77a6f2e190a12954b25e6ed5eea149", size = 2655289, upload-time = "2026-03-22T18:29:46.779Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/c9/584bc9651441b4ba60cc4d557d8a547b5aff901af35bda3a4ee30c819b82/starlette-1.0.0-py3-none-any.whl", hash = "sha256:d3ec55e0bb321692d275455ddfd3df75fff145d009685eb40dc91fc66b03d38b", size = 72651, upload-time = "2026-03-22T18:29:45.111Z" }, +] + +[[package]] +name = "streamlit" +version = "1.57.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "altair" }, + { name = "anyio" }, + { name = "blinker" }, + { name = "cachetools" }, + { name = "click" }, + { name = "gitpython" }, + { name = "httptools" }, + { name = "itsdangerous" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pandas" }, + { name = "pillow" }, + { name = "protobuf" }, + { name = "pyarrow" }, + { name = "pydeck" }, + { name = "python-multipart" }, + { name = "requests" }, + { name = "starlette" }, + { name = "tenacity" }, + { name = "toml" }, + { name = "typing-extensions" }, + { name = "uvicorn" }, + { name = "watchdog", marker = "sys_platform != 'darwin' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "websockets" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5d/f8/b2daf7a5f8ae15527daf94406e771bb6075e958a01c3dde9eba79dc3c9a3/streamlit-1.57.0.tar.gz", hash = "sha256:0b028d305c1a1a757071b2c9504966787602842fc8af6e873795ca58d2b4d12f", size = 8678859, upload-time = "2026-04-28T22:13:32.238Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d8/1a/3ca2293d8552bacea3e67e9600d2d1df7df4a325059769ad83d91c279595/streamlit-1.57.0-py3-none-any.whl", hash = "sha256:0d1d41972aeade5637dbb0e7f0eefa5312272f85304923d240a1b1f0475249c8", size = 9194216, upload-time = "2026-04-28T22:13:29.624Z" }, +] + +[[package]] +name = "streamlit-elements" +version = "0.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "streamlit" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8d/53/6ecfba409b61cdf246d4138bd9c1bef4b79a03fbeeee513ab0c7d43f18cb/streamlit-elements-0.1.0.tar.gz", hash = "sha256:5f9f116f22df3ce4a8636b1dee7c2fd3dc3cb0c66267fd28c3e0314aa1d303a7", size = 6649428, upload-time = "2022-04-25T18:32:53.523Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/0d/eecce69faeb4aec152f4036afe8a0f4f67b0af4fe0ff709243591ad2533d/streamlit_elements-0.1.0-py3-none-any.whl", hash = "sha256:593c4b88c399c55879aa76f7f42970f30106f66acaa4baada6338ae5571790df", size = 7833353, upload-time = "2022-04-25T18:32:49.447Z" }, +] + +[[package]] +name = "sympy" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mpmath" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, +] + +[[package]] +name = "tenacity" +version = "9.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/47/c6/ee486fd809e357697ee8a44d3d69222b344920433d3b6666ccd9b374630c/tenacity-9.1.4.tar.gz", hash = "sha256:adb31d4c263f2bd041081ab33b498309a57c77f9acf2db65aadf0898179cf93a", size = 49413, upload-time = "2026-02-07T10:45:33.841Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" }, +] + +[[package]] +name = "threadpoolctl" +version = "3.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" }, +] + +[[package]] +name = "tiktoken" +version = "0.12.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "regex" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/7d/ab/4d017d0f76ec3171d469d80fc03dfbb4e48a4bcaddaa831b31d526f05edc/tiktoken-0.12.0.tar.gz", hash = "sha256:b18ba7ee2b093863978fcb14f74b3707cdc8d4d4d3836853ce7ec60772139931", size = 37806, upload-time = "2025-10-06T20:22:45.419Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a4/85/be65d39d6b647c79800fd9d29241d081d4eeb06271f383bb87200d74cf76/tiktoken-0.12.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:b97f74aca0d78a1ff21b8cd9e9925714c15a9236d6ceacf5c7327c117e6e21e8", size = 1050728, upload-time = "2025-10-06T20:21:52.756Z" }, + { url = "https://files.pythonhosted.org/packages/4a/42/6573e9129bc55c9bf7300b3a35bef2c6b9117018acca0dc760ac2d93dffe/tiktoken-0.12.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:2b90f5ad190a4bb7c3eb30c5fa32e1e182ca1ca79f05e49b448438c3e225a49b", size = 994049, upload-time = "2025-10-06T20:21:53.782Z" }, + { url = "https://files.pythonhosted.org/packages/66/c5/ed88504d2f4a5fd6856990b230b56d85a777feab84e6129af0822f5d0f70/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:65b26c7a780e2139e73acc193e5c63ac754021f160df919add909c1492c0fb37", size = 1129008, upload-time = "2025-10-06T20:21:54.832Z" }, + { url = "https://files.pythonhosted.org/packages/f4/90/3dae6cc5436137ebd38944d396b5849e167896fc2073da643a49f372dc4f/tiktoken-0.12.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:edde1ec917dfd21c1f2f8046b86348b0f54a2c0547f68149d8600859598769ad", size = 1152665, upload-time = "2025-10-06T20:21:56.129Z" }, + { url = "https://files.pythonhosted.org/packages/a3/fe/26df24ce53ffde419a42f5f53d755b995c9318908288c17ec3f3448313a3/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:35a2f8ddd3824608b3d650a000c1ef71f730d0c56486845705a8248da00f9fe5", size = 1194230, upload-time = "2025-10-06T20:21:57.546Z" }, + { url = "https://files.pythonhosted.org/packages/20/cc/b064cae1a0e9fac84b0d2c46b89f4e57051a5f41324e385d10225a984c24/tiktoken-0.12.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:83d16643edb7fa2c99eff2ab7733508aae1eebb03d5dfc46f5565862810f24e3", size = 1254688, upload-time = "2025-10-06T20:21:58.619Z" }, + { url = "https://files.pythonhosted.org/packages/81/10/b8523105c590c5b8349f2587e2fdfe51a69544bd5a76295fc20f2374f470/tiktoken-0.12.0-cp312-cp312-win_amd64.whl", hash = "sha256:ffc5288f34a8bc02e1ea7047b8d041104791d2ddbf42d1e5fa07822cbffe16bd", size = 878694, upload-time = "2025-10-06T20:21:59.876Z" }, + { url = "https://files.pythonhosted.org/packages/00/61/441588ee21e6b5cdf59d6870f86beb9789e532ee9718c251b391b70c68d6/tiktoken-0.12.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:775c2c55de2310cc1bc9a3ad8826761cbdc87770e586fd7b6da7d4589e13dab3", size = 1050802, upload-time = "2025-10-06T20:22:00.96Z" }, + { url = "https://files.pythonhosted.org/packages/1f/05/dcf94486d5c5c8d34496abe271ac76c5b785507c8eae71b3708f1ad9b45a/tiktoken-0.12.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a01b12f69052fbe4b080a2cfb867c4de12c704b56178edf1d1d7b273561db160", size = 993995, upload-time = "2025-10-06T20:22:02.788Z" }, + { url = "https://files.pythonhosted.org/packages/a0/70/5163fe5359b943f8db9946b62f19be2305de8c3d78a16f629d4165e2f40e/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:01d99484dc93b129cd0964f9d34eee953f2737301f18b3c7257bf368d7615baa", size = 1128948, upload-time = "2025-10-06T20:22:03.814Z" }, + { url = "https://files.pythonhosted.org/packages/0c/da/c028aa0babf77315e1cef357d4d768800c5f8a6de04d0eac0f377cb619fa/tiktoken-0.12.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:4a1a4fcd021f022bfc81904a911d3df0f6543b9e7627b51411da75ff2fe7a1be", size = 1151986, upload-time = "2025-10-06T20:22:05.173Z" }, + { url = "https://files.pythonhosted.org/packages/a0/5a/886b108b766aa53e295f7216b509be95eb7d60b166049ce2c58416b25f2a/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:981a81e39812d57031efdc9ec59fa32b2a5a5524d20d4776574c4b4bd2e9014a", size = 1194222, upload-time = "2025-10-06T20:22:06.265Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f8/4db272048397636ac7a078d22773dd2795b1becee7bc4922fe6207288d57/tiktoken-0.12.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:9baf52f84a3f42eef3ff4e754a0db79a13a27921b457ca9832cf944c6be4f8f3", size = 1255097, upload-time = "2025-10-06T20:22:07.403Z" }, + { url = "https://files.pythonhosted.org/packages/8e/32/45d02e2e0ea2be3a9ed22afc47d93741247e75018aac967b713b2941f8ea/tiktoken-0.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:b8a0cd0c789a61f31bf44851defbd609e8dd1e2c8589c614cc1060940ef1f697", size = 879117, upload-time = "2025-10-06T20:22:08.418Z" }, + { url = "https://files.pythonhosted.org/packages/ce/76/994fc868f88e016e6d05b0da5ac24582a14c47893f4474c3e9744283f1d5/tiktoken-0.12.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:d5f89ea5680066b68bcb797ae85219c72916c922ef0fcdd3480c7d2315ffff16", size = 1050309, upload-time = "2025-10-06T20:22:10.939Z" }, + { url = "https://files.pythonhosted.org/packages/f6/b8/57ef1456504c43a849821920d582a738a461b76a047f352f18c0b26c6516/tiktoken-0.12.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b4e7ed1c6a7a8a60a3230965bdedba8cc58f68926b835e519341413370e0399a", size = 993712, upload-time = "2025-10-06T20:22:12.115Z" }, + { url = "https://files.pythonhosted.org/packages/72/90/13da56f664286ffbae9dbcfadcc625439142675845baa62715e49b87b68b/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:fc530a28591a2d74bce821d10b418b26a094bf33839e69042a6e86ddb7a7fb27", size = 1128725, upload-time = "2025-10-06T20:22:13.541Z" }, + { url = "https://files.pythonhosted.org/packages/05/df/4f80030d44682235bdaecd7346c90f67ae87ec8f3df4a3442cb53834f7e4/tiktoken-0.12.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:06a9f4f49884139013b138920a4c393aa6556b2f8f536345f11819389c703ebb", size = 1151875, upload-time = "2025-10-06T20:22:14.559Z" }, + { url = "https://files.pythonhosted.org/packages/22/1f/ae535223a8c4ef4c0c1192e3f9b82da660be9eb66b9279e95c99288e9dab/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:04f0e6a985d95913cabc96a741c5ffec525a2c72e9df086ff17ebe35985c800e", size = 1194451, upload-time = "2025-10-06T20:22:15.545Z" }, + { url = "https://files.pythonhosted.org/packages/78/a7/f8ead382fce0243cb625c4f266e66c27f65ae65ee9e77f59ea1653b6d730/tiktoken-0.12.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:0ee8f9ae00c41770b5f9b0bb1235474768884ae157de3beb5439ca0fd70f3e25", size = 1253794, upload-time = "2025-10-06T20:22:16.624Z" }, + { url = "https://files.pythonhosted.org/packages/93/e0/6cc82a562bc6365785a3ff0af27a2a092d57c47d7a81d9e2295d8c36f011/tiktoken-0.12.0-cp313-cp313t-win_amd64.whl", hash = "sha256:dc2dd125a62cb2b3d858484d6c614d136b5b848976794edfb63688d539b8b93f", size = 878777, upload-time = "2025-10-06T20:22:18.036Z" }, + { url = "https://files.pythonhosted.org/packages/72/05/3abc1db5d2c9aadc4d2c76fa5640134e475e58d9fbb82b5c535dc0de9b01/tiktoken-0.12.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:a90388128df3b3abeb2bfd1895b0681412a8d7dc644142519e6f0a97c2111646", size = 1050188, upload-time = "2025-10-06T20:22:19.563Z" }, + { url = "https://files.pythonhosted.org/packages/e3/7b/50c2f060412202d6c95f32b20755c7a6273543b125c0985d6fa9465105af/tiktoken-0.12.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:da900aa0ad52247d8794e307d6446bd3cdea8e192769b56276695d34d2c9aa88", size = 993978, upload-time = "2025-10-06T20:22:20.702Z" }, + { url = "https://files.pythonhosted.org/packages/14/27/bf795595a2b897e271771cd31cb847d479073497344c637966bdf2853da1/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:285ba9d73ea0d6171e7f9407039a290ca77efcdb026be7769dccc01d2c8d7fff", size = 1129271, upload-time = "2025-10-06T20:22:22.06Z" }, + { url = "https://files.pythonhosted.org/packages/f5/de/9341a6d7a8f1b448573bbf3425fa57669ac58258a667eb48a25dfe916d70/tiktoken-0.12.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:d186a5c60c6a0213f04a7a802264083dea1bbde92a2d4c7069e1a56630aef830", size = 1151216, upload-time = "2025-10-06T20:22:23.085Z" }, + { url = "https://files.pythonhosted.org/packages/75/0d/881866647b8d1be4d67cb24e50d0c26f9f807f994aa1510cb9ba2fe5f612/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:604831189bd05480f2b885ecd2d1986dc7686f609de48208ebbbddeea071fc0b", size = 1194860, upload-time = "2025-10-06T20:22:24.602Z" }, + { url = "https://files.pythonhosted.org/packages/b3/1e/b651ec3059474dab649b8d5b69f5c65cd8fcd8918568c1935bd4136c9392/tiktoken-0.12.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:8f317e8530bb3a222547b85a58583238c8f74fd7a7408305f9f63246d1a0958b", size = 1254567, upload-time = "2025-10-06T20:22:25.671Z" }, + { url = "https://files.pythonhosted.org/packages/80/57/ce64fd16ac390fafde001268c364d559447ba09b509181b2808622420eec/tiktoken-0.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:399c3dd672a6406719d84442299a490420b458c44d3ae65516302a99675888f3", size = 921067, upload-time = "2025-10-06T20:22:26.753Z" }, + { url = "https://files.pythonhosted.org/packages/ac/a4/72eed53e8976a099539cdd5eb36f241987212c29629d0a52c305173e0a68/tiktoken-0.12.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:c2c714c72bc00a38ca969dae79e8266ddec999c7ceccd603cc4f0d04ccd76365", size = 1050473, upload-time = "2025-10-06T20:22:27.775Z" }, + { url = "https://files.pythonhosted.org/packages/e6/d7/0110b8f54c008466b19672c615f2168896b83706a6611ba6e47313dbc6e9/tiktoken-0.12.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:cbb9a3ba275165a2cb0f9a83f5d7025afe6b9d0ab01a22b50f0e74fee2ad253e", size = 993855, upload-time = "2025-10-06T20:22:28.799Z" }, + { url = "https://files.pythonhosted.org/packages/5f/77/4f268c41a3957c418b084dd576ea2fad2e95da0d8e1ab705372892c2ca22/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:dfdfaa5ffff8993a3af94d1125870b1d27aed7cb97aa7eb8c1cefdbc87dbee63", size = 1129022, upload-time = "2025-10-06T20:22:29.981Z" }, + { url = "https://files.pythonhosted.org/packages/4e/2b/fc46c90fe5028bd094cd6ee25a7db321cb91d45dc87531e2bdbb26b4867a/tiktoken-0.12.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:584c3ad3d0c74f5269906eb8a659c8bfc6144a52895d9261cdaf90a0ae5f4de0", size = 1150736, upload-time = "2025-10-06T20:22:30.996Z" }, + { url = "https://files.pythonhosted.org/packages/28/c0/3c7a39ff68022ddfd7d93f3337ad90389a342f761c4d71de99a3ccc57857/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:54c891b416a0e36b8e2045b12b33dd66fb34a4fe7965565f1b482da50da3e86a", size = 1194908, upload-time = "2025-10-06T20:22:32.073Z" }, + { url = "https://files.pythonhosted.org/packages/ab/0d/c1ad6f4016a3968c048545f5d9b8ffebf577774b2ede3e2e352553b685fe/tiktoken-0.12.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5edb8743b88d5be814b1a8a8854494719080c28faaa1ccbef02e87354fe71ef0", size = 1253706, upload-time = "2025-10-06T20:22:33.385Z" }, + { url = "https://files.pythonhosted.org/packages/af/df/c7891ef9d2712ad774777271d39fdef63941ffba0a9d59b7ad1fd2765e57/tiktoken-0.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:f61c0aea5565ac82e2ec50a05e02a6c44734e91b51c10510b084ea1b8e633a71", size = 920667, upload-time = "2025-10-06T20:22:34.444Z" }, +] + +[[package]] +name = "tokenizers" +version = "0.22.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/73/6f/f80cfef4a312e1fb34baf7d85c72d4411afde10978d4657f8cdd811d3ccc/tokenizers-0.22.2.tar.gz", hash = "sha256:473b83b915e547aa366d1eee11806deaf419e17be16310ac0a14077f1e28f917", size = 372115, upload-time = "2026-01-05T10:45:15.988Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/92/97/5dbfabf04c7e348e655e907ed27913e03db0923abb5dfdd120d7b25630e1/tokenizers-0.22.2-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:544dd704ae7238755d790de45ba8da072e9af3eea688f698b137915ae959281c", size = 3100275, upload-time = "2026-01-05T10:41:02.158Z" }, + { url = "https://files.pythonhosted.org/packages/2e/47/174dca0502ef88b28f1c9e06b73ce33500eedfac7a7692108aec220464e7/tokenizers-0.22.2-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:1e418a55456beedca4621dbab65a318981467a2b188e982a23e117f115ce5001", size = 2981472, upload-time = "2026-01-05T10:41:00.276Z" }, + { url = "https://files.pythonhosted.org/packages/d6/84/7990e799f1309a8b87af6b948f31edaa12a3ed22d11b352eaf4f4b2e5753/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2249487018adec45d6e3554c71d46eb39fa8ea67156c640f7513eb26f318cec7", size = 3290736, upload-time = "2026-01-05T10:40:32.165Z" }, + { url = "https://files.pythonhosted.org/packages/78/59/09d0d9ba94dcd5f4f1368d4858d24546b4bdc0231c2354aa31d6199f0399/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:25b85325d0815e86e0bac263506dd114578953b7b53d7de09a6485e4a160a7dd", size = 3168835, upload-time = "2026-01-05T10:40:38.847Z" }, + { url = "https://files.pythonhosted.org/packages/47/50/b3ebb4243e7160bda8d34b731e54dd8ab8b133e50775872e7a434e524c28/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bfb88f22a209ff7b40a576d5324bf8286b519d7358663db21d6246fb17eea2d5", size = 3521673, upload-time = "2026-01-05T10:40:56.614Z" }, + { url = "https://files.pythonhosted.org/packages/e0/fa/89f4cb9e08df770b57adb96f8cbb7e22695a4cb6c2bd5f0c4f0ebcf33b66/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1c774b1276f71e1ef716e5486f21e76333464f47bece56bbd554485982a9e03e", size = 3724818, upload-time = "2026-01-05T10:40:44.507Z" }, + { url = "https://files.pythonhosted.org/packages/64/04/ca2363f0bfbe3b3d36e95bf67e56a4c88c8e3362b658e616d1ac185d47f2/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:df6c4265b289083bf710dff49bc51ef252f9d5be33a45ee2bed151114a56207b", size = 3379195, upload-time = "2026-01-05T10:40:51.139Z" }, + { url = "https://files.pythonhosted.org/packages/2e/76/932be4b50ef6ccedf9d3c6639b056a967a86258c6d9200643f01269211ca/tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:369cc9fc8cc10cb24143873a0d95438bb8ee257bb80c71989e3ee290e8d72c67", size = 3274982, upload-time = "2026-01-05T10:40:58.331Z" }, + { url = "https://files.pythonhosted.org/packages/1d/28/5f9f5a4cc211b69e89420980e483831bcc29dade307955cc9dc858a40f01/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:29c30b83d8dcd061078b05ae0cb94d3c710555fbb44861139f9f83dcca3dc3e4", size = 9478245, upload-time = "2026-01-05T10:41:04.053Z" }, + { url = "https://files.pythonhosted.org/packages/6c/fb/66e2da4704d6aadebf8cb39f1d6d1957df667ab24cff2326b77cda0dcb85/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:37ae80a28c1d3265bb1f22464c856bd23c02a05bb211e56d0c5301a435be6c1a", size = 9560069, upload-time = "2026-01-05T10:45:10.673Z" }, + { url = "https://files.pythonhosted.org/packages/16/04/fed398b05caa87ce9b1a1bb5166645e38196081b225059a6edaff6440fac/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:791135ee325f2336f498590eb2f11dc5c295232f288e75c99a36c5dbce63088a", size = 9899263, upload-time = "2026-01-05T10:45:12.559Z" }, + { url = "https://files.pythonhosted.org/packages/05/a1/d62dfe7376beaaf1394917e0f8e93ee5f67fea8fcf4107501db35996586b/tokenizers-0.22.2-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:38337540fbbddff8e999d59970f3c6f35a82de10053206a7562f1ea02d046fa5", size = 10033429, upload-time = "2026-01-05T10:45:14.333Z" }, + { url = "https://files.pythonhosted.org/packages/fd/18/a545c4ea42af3df6effd7d13d250ba77a0a86fb20393143bbb9a92e434d4/tokenizers-0.22.2-cp39-abi3-win32.whl", hash = "sha256:a6bf3f88c554a2b653af81f3204491c818ae2ac6fbc09e76ef4773351292bc92", size = 2502363, upload-time = "2026-01-05T10:45:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/65/71/0670843133a43d43070abeb1949abfdef12a86d490bea9cd9e18e37c5ff7/tokenizers-0.22.2-cp39-abi3-win_amd64.whl", hash = "sha256:c9ea31edff2968b44a88f97d784c2f16dc0729b8b143ed004699ebca91f05c48", size = 2747786, upload-time = "2026-01-05T10:45:18.411Z" }, + { url = "https://files.pythonhosted.org/packages/72/f4/0de46cfa12cdcbcd464cc59fde36912af405696f687e53a091fb432f694c/tokenizers-0.22.2-cp39-abi3-win_arm64.whl", hash = "sha256:9ce725d22864a1e965217204946f830c37876eee3b2ba6fc6255e8e903d5fcbc", size = 2612133, upload-time = "2026-01-05T10:45:17.232Z" }, +] + +[[package]] +name = "toml" +version = "0.10.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/be/ba/1f744cdc819428fc6b5084ec34d9b30660f6f9daaf70eead706e3203ec3c/toml-0.10.2.tar.gz", hash = "sha256:b3bda1d108d5dd99f4a20d24d9c348e91c4db7ab1b749200bded2f839ccbe68f", size = 22253, upload-time = "2020-11-01T01:40:22.204Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", hash = "sha256:806143ae5bfb6a3c6e736a764057db0e6a0e05e338b5630894a5f779cabb4f9b", size = 16588, upload-time = "2020-11-01T01:40:20.672Z" }, +] + +[[package]] +name = "torch" +version = "2.11.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", +] +dependencies = [ + { name = "filelock", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "fsspec", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "jinja2", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "networkx", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "setuptools", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "sympy", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "typing-extensions", marker = "(sys_platform == 'emscripten' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (sys_platform == 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/6f/8b/69e3008d78e5cee2b30183340cc425081b78afc5eff3d080daab0adda9aa/torch-2.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4b5866312ee6e52ea625cd211dcb97d6a2cdc1131a5f15cc0d87eec948f6dd34", size = 80606338, upload-time = "2026-03-23T18:11:34.781Z" }, + { url = "https://files.pythonhosted.org/packages/13/16/42e5915ebe4868caa6bac83a8ed59db57f12e9a61b7d749d584776ed53d5/torch-2.11.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:f99924682ef0aa6a4ab3b1b76f40dc6e273fca09f367d15a524266db100a723f", size = 419731115, upload-time = "2026-03-23T18:11:06.944Z" }, + { url = "https://files.pythonhosted.org/packages/1a/c9/82638ef24d7877510f83baf821f5619a61b45568ce21c0a87a91576510aa/torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:0f68f4ac6d95d12e896c3b7a912b5871619542ec54d3649cf48cc1edd4dd2756", size = 530712279, upload-time = "2026-03-23T18:10:31.481Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ff/6756f1c7ee302f6d202120e0f4f05b432b839908f9071157302cedfc5232/torch-2.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:fbf39280699d1b869f55eac536deceaa1b60bd6788ba74f399cc67e60a5fab10", size = 114556047, upload-time = "2026-03-23T18:10:55.931Z" }, + { url = "https://files.pythonhosted.org/packages/87/89/5ea6722763acee56b045435fb84258db7375c48165ec8be7880ab2b281c5/torch-2.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1e6debd97ccd3205bbb37eb806a9d8219e1139d15419982c09e23ef7d4369d18", size = 80606801, upload-time = "2026-03-23T18:10:18.649Z" }, + { url = "https://files.pythonhosted.org/packages/32/d1/8ed2173589cbfe744ed54e5a73efc107c0085ba5777ee93a5f4c1ab90553/torch-2.11.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:63a68fa59de8f87acc7e85a5478bb2dddbb3392b7593ec3e78827c793c4b73fd", size = 419732382, upload-time = "2026-03-23T18:08:30.835Z" }, + { url = "https://files.pythonhosted.org/packages/3d/e1/b73f7c575a4b8f87a5928f50a1e35416b5e27295d8be9397d5293e7e8d4c/torch-2.11.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:cc89b9b173d9adfab59fd227f0ab5e5516d9a52b658ae41d64e59d2e55a418db", size = 530711509, upload-time = "2026-03-23T18:08:47.213Z" }, + { url = "https://files.pythonhosted.org/packages/66/82/3e3fcdd388fbe54e29fd3f991f36846ff4ac90b0d0181e9c8f7236565f82/torch-2.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:4dda3b3f52d121063a731ddb835f010dc137b920d7fec2778e52f60d8e4bf0cd", size = 114555842, upload-time = "2026-03-23T18:09:52.111Z" }, + { url = "https://files.pythonhosted.org/packages/db/38/8ac78069621b8c2b4979c2f96dc8409ef5e9c4189f6aac629189a78677ca/torch-2.11.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8b394322f49af4362d4f80e424bcaca7efcd049619af03a4cf4501520bdf0fb4", size = 80959574, upload-time = "2026-03-23T18:10:14.214Z" }, + { url = "https://files.pythonhosted.org/packages/6d/6c/56bfb37073e7136e6dd86bfc6af7339946dd684e0ecf2155ac0eee687ae1/torch-2.11.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:2658f34ce7e2dabf4ec73b45e2ca68aedad7a5be87ea756ad656eaf32bf1e1ea", size = 419732324, upload-time = "2026-03-23T18:09:36.604Z" }, + { url = "https://files.pythonhosted.org/packages/07/f4/1b666b6d61d3394cca306ea543ed03a64aad0a201b6cd159f1d41010aeb1/torch-2.11.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:98bb213c3084cfe176302949bdc360074b18a9da7ab59ef2edc9d9f742504778", size = 530596026, upload-time = "2026-03-23T18:09:20.842Z" }, + { url = "https://files.pythonhosted.org/packages/48/6b/30d1459fa7e4b67e9e3fe1685ca1d8bb4ce7c62ef436c3a615963c6c866c/torch-2.11.0-cp313-cp313t-win_amd64.whl", hash = "sha256:a97b94bbf62992949b4730c6cd2cc9aee7b335921ee8dc207d930f2ed09ae2db", size = 114793702, upload-time = "2026-03-23T18:09:47.304Z" }, + { url = "https://files.pythonhosted.org/packages/26/0d/8603382f61abd0db35841148ddc1ffd607bf3100b11c6e1dab6d2fc44e72/torch-2.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:01018087326984a33b64e04c8cb5c2795f9120e0d775ada1f6638840227b04d7", size = 80573442, upload-time = "2026-03-23T18:09:10.117Z" }, + { url = "https://files.pythonhosted.org/packages/c7/86/7cd7c66cb9cec6be330fff36db5bd0eef386d80c031b581ec81be1d4b26c/torch-2.11.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:2bb3cc54bd0dea126b0060bb1ec9de0f9c7f7342d93d436646516b0330cd5be7", size = 419749385, upload-time = "2026-03-23T18:07:33.77Z" }, + { url = "https://files.pythonhosted.org/packages/47/e8/b98ca2d39b2e0e4730c0ee52537e488e7008025bc77ca89552ff91021f7c/torch-2.11.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:4dc8b3809469b6c30b411bb8c4cad3828efd26236153d9beb6a3ec500f211a60", size = 530716756, upload-time = "2026-03-23T18:07:50.02Z" }, + { url = "https://files.pythonhosted.org/packages/78/88/d4a4cda8362f8a30d1ed428564878c3cafb0d87971fbd3947d4c84552095/torch-2.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:2b4e811728bd0cc58fb2b0948fe939a1ee2bf1422f6025be2fca4c7bd9d79718", size = 114552300, upload-time = "2026-03-23T18:09:05.617Z" }, + { url = "https://files.pythonhosted.org/packages/bf/46/4419098ed6d801750f26567b478fc185c3432e11e2cad712bc6b4c2ab0d0/torch-2.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8245477871c3700d4370352ffec94b103cfcb737229445cf9946cddb7b2ca7cd", size = 80959460, upload-time = "2026-03-23T18:09:00.818Z" }, + { url = "https://files.pythonhosted.org/packages/fd/66/54a56a4a6ceaffb567231994a9745821d3af922a854ed33b0b3a278e0a99/torch-2.11.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:ab9a8482f475f9ba20e12db84b0e55e2f58784bdca43a854a6ccd3fd4b9f75e6", size = 419735835, upload-time = "2026-03-23T18:07:18.974Z" }, + { url = "https://files.pythonhosted.org/packages/b1/e7/0b6665f533aa9e337662dc190425abc0af1fe3234088f4454c52393ded61/torch-2.11.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:563ed3d25542d7e7bbc5b235ccfacfeb97fb470c7fee257eae599adb8005c8a2", size = 530613405, upload-time = "2026-03-23T18:08:07.014Z" }, + { url = "https://files.pythonhosted.org/packages/cf/bf/c8d12a2c86dbfd7f40fb2f56fbf5a505ccf2d9ce131eb559dfc7c51e1a04/torch-2.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b2a43985ff5ef6ddd923bbcf99943e5f58059805787c5c9a2622bf05ca2965b0", size = 114792991, upload-time = "2026-03-23T18:08:19.216Z" }, +] + +[[package]] +name = "torch" +version = "2.12.0" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'darwin'", + "python_full_version < '3.14' and sys_platform == 'darwin'", +] +dependencies = [ + { name = "filelock", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "fsspec", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "jinja2", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "networkx", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "setuptools", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "sympy", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "typing-extensions", marker = "(sys_platform == 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:b41339df93d491435e790ff8bcbae1c0ce777175889bfd1281d119862793e6a2", upload-time = "2026-05-12T16:20:12Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:90dd587a5f61bfe1307148b581e2084fc5bc4a06e2b90a20e9a36b81087ff16b", upload-time = "2026-05-12T16:20:17Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:10ee1448a9f304d3b987eb4656f664ba6e4d7b410ca7a5a7c642199777a2cf88", upload-time = "2026-05-12T16:20:21Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f7dfae4a519197dfa050e98d8e36378a0fb5899625a875c2b54445005a2e404e", upload-time = "2026-05-12T16:20:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:b4556715c8572758625d62b6e0ae3b1f76c440221913a6fb5e100f321fb4fb02", upload-time = "2026-05-12T16:20:31Z" }, +] + +[[package]] +name = "torch" +version = "2.12.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +dependencies = [ + { name = "cuda-bindings", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "cuda-toolkit", extra = ["cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "filelock", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "fsspec", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "jinja2", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "networkx", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cublas", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cudnn-cu13", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cusparselt-cu13", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-nccl-cu13", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-nvshmem-cu13", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "setuptools", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "sympy", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "triton", marker = "(sys_platform == 'linux' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "typing-extensions", marker = "(sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-6-kgpipe-cpu' and extra != 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/ef/bb/285d643f254731294c9b595a007eac39db4600a98682d7bca688f42ca164/torch-2.12.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:b41339df93d491435e790ff8bcbae1c0ce777175889bfd1281d119862793e6a2", size = 88010197, upload-time = "2026-05-13T14:55:35.414Z" }, + { url = "https://files.pythonhosted.org/packages/79/81/76debf1db1343bd929bbb5d74c89fb437c2ed88eb144712557e7bd3eea45/torch-2.12.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:8fbef9f108a863e7722a73740998967e3b074742a834fc5be3a535a2befa7057", size = 426376751, upload-time = "2026-05-13T14:55:03.353Z" }, + { url = "https://files.pythonhosted.org/packages/de/f0/80026028b603c4650ff270fc3785bdef4bd6738765a9cc5a0f5a637d65a2/torch-2.12.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:4b4f64c2c2b11f7510d93dd6412b87025ff6eddd6bb61c3b5a3d892ea20c4756", size = 532261691, upload-time = "2026-05-13T14:52:54.453Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c2/64b06cbb7830fb3cd9be13e1158b31a3f36b68e6a209105ee3c9d9480be0/torch-2.12.0-cp312-cp312-win_amd64.whl", hash = "sha256:8b958caff4a14d3a3b0b2dfc6a378f64dda9728a9dad28c08a0db9ce4dafb549", size = 122988114, upload-time = "2026-05-13T14:54:42.153Z" }, + { url = "https://files.pythonhosted.org/packages/86/ca/01896c80ba921676aa45886b2c5b8d774912de2a1f719de48169c6f755cd/torch-2.12.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:90dd587a5f61bfe1307148b581e2084fc5bc4a06e2b90a20e9a36b81087ff16b", size = 88009511, upload-time = "2026-05-13T14:54:47.411Z" }, + { url = "https://files.pythonhosted.org/packages/a5/04/52bdaf4787eab6ac7d7f5851dff934e4def0bc8ead9c8fd2b69b3e529699/torch-2.12.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:864392c73b7654f4d2b3ae712f607937d0dbb1101c4555fbb41848106b297f39", size = 426383231, upload-time = "2026-05-13T14:53:32.129Z" }, + { url = "https://files.pythonhosted.org/packages/49/8a/94bdecd13f5aaa90d45920b89789d9fe7c6f4af8c3cdd7ce01fcb59908fc/torch-2.12.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:5d6b560dfa7d56291c07d615c3bb73e8d9943d9b6d87f76cd0d9d570c4797fa6", size = 532269288, upload-time = "2026-05-13T14:53:49.423Z" }, + { url = "https://files.pythonhosted.org/packages/3e/2f/bdbaaa267de519ef1b73054bf590d8c93c37a266c9a4e24a01bd38b6918f/torch-2.12.0-cp313-cp313-win_amd64.whl", hash = "sha256:3fee918902090ade827643e758e98363278815de583c75d111fdd665ebffde9f", size = 122987706, upload-time = "2026-05-13T14:54:00.335Z" }, + { url = "https://files.pythonhosted.org/packages/9b/ad/e95e822f3538171e22640a7fbe839a1fdb666600bf6487025de2ff03b11a/torch-2.12.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:10ee1448a9f304d3b987eb4656f664ba6e4d7b410ca7a5a7c642199777a2cf88", size = 88319556, upload-time = "2026-05-13T14:54:05.574Z" }, + { url = "https://files.pythonhosted.org/packages/b7/07/055d06d985b445d67422d25b033c11cf55bbb81785d4c4e68e28bca5820e/torch-2.12.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:af68dbf403439cae9ceaeaaf92f8352b460787dcd27b92aa05c40dd4a19c0f1e", size = 426397656, upload-time = "2026-05-13T14:52:38.84Z" }, + { url = "https://files.pythonhosted.org/packages/43/94/b0b4fdc3014122e0a7302fb90086d352aa48f2576f0b252561ebb38c01a8/torch-2.12.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:a6a2eebb237d3b1d9ad3b378e86d9b9e0782afdea8b1e0eba6a13646b9b49c07", size = 532183124, upload-time = "2026-05-13T14:53:16.178Z" }, + { url = "https://files.pythonhosted.org/packages/d8/c8/052405e6ad05d3237bfe5a4df78f917773956f8e17813a2d44c059068b74/torch-2.12.0-cp313-cp313t-win_amd64.whl", hash = "sha256:2140e373e9a51a3e22ef62e8d14366d0b470d18f0adf19fdc757368077133a34", size = 123232462, upload-time = "2026-05-13T14:52:27.26Z" }, + { url = "https://files.pythonhosted.org/packages/67/dc/ac069f8d6e8be701535921141055293b0d4819d3d7f224a4612cf157c7f9/torch-2.12.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f7dfae4a519197dfa050e98d8e36378a0fb5899625a875c2b54445005a2e404e", size = 88027282, upload-time = "2026-05-13T14:53:05.258Z" }, + { url = "https://files.pythonhosted.org/packages/33/c3/1c1eb00e34555b536dddf792676026a988d710ed36981aa00499b36b0620/torch-2.12.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:891c769072637c74e9a5a77a3bc782894696d8ffec83b938df8536dee7f0ba78", size = 426386961, upload-time = "2026-05-13T14:51:28.406Z" }, + { url = "https://files.pythonhosted.org/packages/cd/d4/7e730dba0c7032a4154dc9056b76cf9625515e030e269cfbf8098fcfee7d/torch-2.12.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:e2ad3eb85d39c3cab62dfa93ed5a73516e6a53c6713cb97d004004fe089f0f1f", size = 532272265, upload-time = "2026-05-13T14:51:59.308Z" }, + { url = "https://files.pythonhosted.org/packages/f1/b4/92c80d1bbfee1c0036c06d1d2155a3065bd2423134c83bf8a47e65cd6b9b/torch-2.12.0-cp314-cp314-win_amd64.whl", hash = "sha256:c66696857e987efb8bc1777a37357ec4f60ab5e8af6250b83d6034437fa2d8f3", size = 122987138, upload-time = "2026-05-13T14:51:45.942Z" }, + { url = "https://files.pythonhosted.org/packages/7b/78/2e12b37ce50a19a037d7bc62d652a5a8f27385a7b05859d6bc9204f20cfe/torch-2.12.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:b4556715c8572758625d62b6e0ae3b1f76c440221913a6fb5e100f321fb4fb02", size = 88320100, upload-time = "2026-05-13T14:51:39.955Z" }, + { url = "https://files.pythonhosted.org/packages/56/5e/83c450ec7b0bb40a7b74611c1b5440f9260e33c54c90d556fd4a1f0fd955/torch-2.12.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:a43ac605a5e13116c72b64c359644cce0229f213dde48d2ae0ae5eb5becf7feb", size = 426391871, upload-time = "2026-05-13T14:52:14.989Z" }, + { url = "https://files.pythonhosted.org/packages/c9/e9/1a0b575d98d0afedd8f157d23fa3d2759421483660448e60d0a4b10b6daa/torch-2.12.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:6a7512adfdd7f6732e40de1c620831e3c75b39b98cef60b11d0c5f0a76473ec5", size = 532192241, upload-time = "2026-05-13T14:51:07.795Z" }, + { url = "https://files.pythonhosted.org/packages/88/21/afadd25ecd81b3cea1e11c73cf1ab41a983a50271548c3ec7ec3b9efc3e9/torch-2.12.0-cp314-cp314t-win_amd64.whl", hash = "sha256:5f96b63f8287f66a005dd1b5a6abba2920f11156c5e5c4d815f3e2050fd1aa16", size = 123231092, upload-time = "2026-05-13T14:51:18.854Z" }, +] + +[[package]] +name = "torch" +version = "2.12.0+cpu" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +dependencies = [ + { name = "filelock", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "fsspec", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "jinja2", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "networkx", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "setuptools", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "sympy", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "typing-extensions", marker = "(sys_platform != 'darwin' and extra == 'extra-6-kgpipe-cpu') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp312-cp312-linux_s390x.whl", hash = "sha256:b9d0e8eed0af9321ffb12b75f4aca371b071254f12cf75875d5a8e7cc8f52b51", upload-time = "2026-05-12T23:16:33Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:ce2ddb880b0813fcc91a737f08fdd973a8115a74c64ccb34e9c09a7964b4d448", upload-time = "2026-05-12T23:16:40Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:5e3dc83725581fa38b7b2e45c58692e30b2a3cde19191af54b675ffcac3840a6", upload-time = "2026-05-12T23:16:48Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp312-cp312-win_amd64.whl", hash = "sha256:70ead47f538417323a230c0e743e5cbb6d91f11bd8339abf8c05c9d02f8409bc", upload-time = "2026-05-12T23:16:56Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp312-cp312-win_arm64.whl", hash = "sha256:5a3b24f429d126a08acafd5cfe8b719409618ad57c49bf4f20df4f8cd32cd682", upload-time = "2026-05-12T23:17:00Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313-linux_s390x.whl", hash = "sha256:5e0da19e1c3bfdc9b92638c552579eac678354485d61fc8921b0461fd6c40449", upload-time = "2026-05-12T23:17:05Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:68b7ddd4db4603a03e106e74c7098c8d8c8943d33c1e5ada009ca4cd885759c3", upload-time = "2026-05-12T23:17:12Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:ada78018bdfa30d1c766596cd32d910dbf5b03424cd859231b6d2a00533de922", upload-time = "2026-05-12T23:17:20Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313-win_amd64.whl", hash = "sha256:59bc266826e683899d49ee0af9829f3eafd0a16e15b5db9dc591c8d955003b66", upload-time = "2026-05-12T23:17:27Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313-win_arm64.whl", hash = "sha256:97a5160abf3ca9d59a2cd7b4b4de89d9dfe290d36a1ac720262a55fbcee10b6c", upload-time = "2026-05-12T23:17:31Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313t-linux_s390x.whl", hash = "sha256:32b9b7a0974cd6149cb98def0a28a49d92d7c14a384273d5539da9624239e950", upload-time = "2026-05-12T23:17:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:93ed8dc52c113580daf6124982b3232629045dccc5cd83a8f5ed478f7bac7340", upload-time = "2026-05-12T23:17:43Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:6a1c86abd4ed15a0736cf2663ad69642ae5d1288c99e30346070e6241018a0a9", upload-time = "2026-05-12T23:17:54Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp313-cp313t-win_amd64.whl", hash = "sha256:768dce4b7b3353795f667d1cb0dd7dfba06f570cd39539576097335e05bb71fe", upload-time = "2026-05-12T23:18:02Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314-linux_s390x.whl", hash = "sha256:ee1f329acfd0c2a1ccaa3393bcaf9857ea58759549bb2d67e271a6eab42382b3", upload-time = "2026-05-12T23:18:08Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:797c066367792c92eb97cafba7fd0caa8d7455e6078a4ee880630077378dc372", upload-time = "2026-05-12T23:18:15Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:a8f419ce3f25388d36e67153ec63b3a1b17059c49f5a7759a7e91ac4843660d3", upload-time = "2026-05-12T23:18:22Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314-win_amd64.whl", hash = "sha256:1dd196c43e74e7b3b526ff434e7efbdef3f3792a2efbecfc983d7dce501840d2", upload-time = "2026-05-12T23:18:30Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314t-linux_s390x.whl", hash = "sha256:d0d2080cb13c94ebc0c884d237e404490743d0f40192c8a180abf3b6b6f334cf", upload-time = "2026-05-12T23:18:35Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:f7bc15972acad257723775237cdd120024cca844b8bc64701822fa596bcb7e14", upload-time = "2026-05-12T23:18:42Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:4d79f961250d1763487ecbc90af019a80009f9e87cadc5366b3ec4ba5671fea6", upload-time = "2026-05-12T23:18:50Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torch-2.12.0%2Bcpu-cp314-cp314t-win_amd64.whl", hash = "sha256:46b8f4c41ac36bb5d5b47f5437b3de5541b313275e59c1d2aefd3bef32b0f531", upload-time = "2026-05-12T23:18:58Z" }, +] + +[[package]] +name = "torch" +version = "2.12.0+cu130" +source = { registry = "https://download.pytorch.org/whl/cu130" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +dependencies = [ + { name = "cuda-bindings", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "cuda-toolkit", extra = ["cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "filelock", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "fsspec", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "jinja2", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "networkx", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "nvidia-cublas", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cudnn-cu13", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-cusparselt-cu13", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-nccl-cu13", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "nvidia-nvshmem-cu13", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "setuptools", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "sympy", marker = "extra == 'extra-6-kgpipe-cuda'" }, + { name = "triton", marker = "(sys_platform == 'linux' and extra == 'extra-6-kgpipe-cuda') or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, + { name = "typing-extensions", marker = "extra == 'extra-6-kgpipe-cuda'" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:cb95bd4626150e41aeea2b60e4635a878ebe01e63f3344409f4b7353fdb7998c", upload-time = "2026-05-12T23:49:12Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:9f512ea51c170a7cc1a0487c08f0154b78defba4eb8619cad0130c8615ed8526", upload-time = "2026-05-12T23:49:40Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp312-cp312-win_amd64.whl", hash = "sha256:24e75a0c3ea4243067d7560955f2eef6466e9365de7dd4a3a4b8693c9ac4bccf", upload-time = "2026-05-12T23:50:41Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:bf5f067d3a4d713b75ccd6a0141f8133c7495a016b917ce6dcec1492e3da98b0", upload-time = "2026-05-12T23:51:35Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:fe5fefb784a370d1ba4959de6e87bcd3b35441040a99bffe32f5cd03bbc834c0", upload-time = "2026-05-12T23:52:00Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313-win_amd64.whl", hash = "sha256:6e728c5fdeffa19b3fa6a759ff585147851772789f3dc84dec5f8cbde0f7a5b0", upload-time = "2026-05-12T23:53:01Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:fce821712a2881eafcfe9ddf646d953683ae39f2e4c9f9066c6ebe4adcc76495", upload-time = "2026-05-12T23:53:55Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:180389b4cebb5d8988e453ca35df8fbbf709734c35e882b6e9f4abaca979454a", upload-time = "2026-05-12T23:54:20Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp313-cp313t-win_amd64.whl", hash = "sha256:eb22ad632b19f6ab9e0852aa2229e9b1c7f5bab5220e39012b3056c31391ea02", upload-time = "2026-05-12T23:55:24Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:05be17ab4c1bd335cff4b6fb4c78eba6ff26ef7d8c997887e5cb59b0c29427b2", upload-time = "2026-05-12T23:56:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:3ff7366f6919232f099ef702c3ebd3509c91ab37c367e408cb3799c6bed214a4", upload-time = "2026-05-12T23:56:52Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314-win_amd64.whl", hash = "sha256:b30d09337048750c1bf10c2abc8cb3d3bf9bb5163d6a34df7c2eb6e9ee32c603", upload-time = "2026-05-12T23:57:58Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:4526d2f200d9e7c7f0a04bfdeff982c4e86f7a0bac3c190ce48cd8caa3d5c888", upload-time = "2026-05-12T23:58:58Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:a255426cf47827e73975378fd03f3a581fc1d21241f294d6ab43b7f610ccd49c", upload-time = "2026-05-12T23:59:20Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torch-2.12.0%2Bcu130-cp314-cp314t-win_amd64.whl", hash = "sha256:0fc041b5bed1e50ea54216f4e86dec696e773aea9b82c612d91fe024ad3af3f1", upload-time = "2026-05-13T00:00:20Z" }, +] + +[[package]] +name = "torchaudio" +version = "2.11.0" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'darwin'", + "python_full_version < '3.14' and sys_platform == 'darwin'", +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a1cf1acc883bee9cb906a933572fed6a8a933f86ef34e9ea7d803f72317e8c1b", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:e3f9696a9ef1d49acc452159b052370c636406d072e9d8f10895fda87b591ea9", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:bda09ea630ae7207384fb0f28c35e4f8c0d82dd6eba020b6b335ad0caa9fed49", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:cc09cd1f6015b8549e7fe255fb1be5346b57e7fee06541d3f3dbb012d8c4715f", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:da2725e250866da42a12934c9a6552f65a18b7187fd7a6221387f0e605fb3b96", upload-time = "2026-03-23T15:50:10Z" }, +] + +[[package]] +name = "torchaudio" +version = "2.11.0+cpu" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:b9dd2c6ac144001dc6dac38b564c1de73ac26ef0c195d5037c4a94990b0e2b5a", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:2354248848d06a9ae1e7a12165f800f0dda7df60ecac9fca892322b722b922c0", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp312-cp312-win_amd64.whl", hash = "sha256:95d517bd1a0a28dacd1c37550ced95cab64f3a7a4ef9b8219b41049388a71163", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:2faa8d8f251d1fa44813765b00791048b617e9dc06e6cd9222aba81023929119", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:3c0175d0ed054bf0dc3b154a744b1a127c94291b3f3b7bdd0639b4b238c89445", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313-win_amd64.whl", hash = "sha256:fa0c22f58f60c3537b7d28ed2501e0995acb2a65d9af2708f21edaad67186cd8", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:5d665234def325e22c15518c581b0107a651c9f843176e3192360b092ebcb656", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:a3df9c41706a22de0f43fe2734f54db31cb5a7314cc92ecc84657a8492b3ff8d", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp313-cp313t-win_amd64.whl", hash = "sha256:18818a326b779abc7bfd5cfb9fe88501916dde144cb1944fc9e7b4fb6208dfc7", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:591ed8279f0170ef28933873f65e9f5f8c439287088ef6285cda65988b0ba614", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:34c5dcd704e17a2b01c097b4fe3b5f83c5cdbc42b9f2abd095e026c588f873d4", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314-win_amd64.whl", hash = "sha256:be19f467eb7a173264653369426e7ecc4745e28d15f9c52eea2ad5316ec685eb", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:ad3e6141c1078c5c9c8ae7c3cdf6c80eb35612c99c8cf3f76fe295845f45bb9d", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:69db7c3d86af1bc8224f7053395ace3e2bd8c56b0c3922bc7b798114440c88e5", upload-time = "2026-03-23T15:50:10Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchaudio-2.11.0%2Bcpu-cp314-cp314t-win_amd64.whl", hash = "sha256:b6bfca873bd67d02fbbaa254312283be48dc1ca5532013152527210db432ef8c", upload-time = "2026-03-23T15:50:10Z" }, +] + +[[package]] +name = "torchaudio" +version = "2.11.0+cu130" +source = { registry = "https://download.pytorch.org/whl/cu130" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:7171f810887e7cd1a4763974d5a1f2e1466692404315bb70705e0f49fb3a28e0", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:3fba988f4301fe13547fe5e99c76d9ae36a27e19ded82eeffed9d2456e12edef", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp312-cp312-win_amd64.whl", hash = "sha256:f74949f9ace1e4a6cf9468bdb3211b9cfa0af6ea348125471ac71c8621d6c77d", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:23498b01097648e304e78d6495a9f5bdce8441a802afc3025e2561973d74c025", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:e9c07cfdab691454092ff12d21dd1407a4bb8ad081d38f222cf6fcf6abcc18c8", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313-win_amd64.whl", hash = "sha256:ce09a7b144b7982b46c8fe399cf5f91d43dda571e9d6ddba67e928567551f614", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:f9b277a0d3b2ab4385778146b7e879716f36b6f2080f7190ec744e3383511791", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:d07c4cbe4bec3e15bb18ba163058038f5f5fc1775c3061685c194439af4d2e9f", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp313-cp313t-win_amd64.whl", hash = "sha256:b9dd151f06842ca77dc341aed94ea2f5d13a89e5027aa032a47198d073bcf3db", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:0a36531ffb90f0b1870e994bc57643bf6466fd8c290b5b9b2b36dadc3445d0c0", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:378b49671b581114a2d25d40928f12a150872feadf11669a63f573e81c78019a", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314-win_amd64.whl", hash = "sha256:b345401d76a371031b2fe965bd9579b9621c9d87902ee3e586be665df04c179e", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:94e90c94c93a626e21686c3dc76d4597c3e4ad178a911611fed4f14c8de07293", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:1cef8561494c44b12eb6a390d9f2ff52f6fa540b9a7aba6c8951c074056a2f11", upload-time = "2026-03-23T15:50:26Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchaudio-2.11.0%2Bcu130-cp314-cp314t-win_amd64.whl", hash = "sha256:ac4988a74921500f9f2dc6502867876bf1bf15d845a518f4711d21ec81a1efef", upload-time = "2026-03-23T15:50:26Z" }, +] + +[[package]] +name = "torchvision" +version = "0.27.0" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'darwin'", + "python_full_version < '3.14' and sys_platform == 'darwin'", +] +dependencies = [ + { name = "numpy", marker = "sys_platform == 'darwin'" }, + { name = "pillow", marker = "sys_platform == 'darwin'" }, + { name = "torch", version = "2.12.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:1a6dd742a150645126df9e0b2e449874c1d635897c773b322c2e067e98382dfe", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:41d6dae73e1af09fa82ded597ae57f2a2314285acde54b25890a8f8e51b999d7", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:c5121f1b9ab09a7f73e837871deb8321551f7eaeb19d87aa00de9191968eae44", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:c1fac0fc2a7adf29481fc1938a0e7845c57ba1147a986784109c4d98f434ea8c", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:2c4099a15150143b9b034730b404a56d572efe0b79489b4c765d929cb4eac7f3", upload-time = "2026-05-12T16:20:37Z" }, +] + +[[package]] +name = "torchvision" +version = "0.27.0+cpu" +source = { registry = "https://download.pytorch.org/whl/cpu" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +dependencies = [ + { name = "numpy", marker = "sys_platform != 'darwin'" }, + { name = "pillow", marker = "sys_platform != 'darwin'" }, + { name = "torch", version = "2.12.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform != 'darwin'" }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:1b06f42d48b62098114923d8a3fe9fa864182715db06584a515155db0aa8eb30", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:912a8b008d95fc9f8f8507e0663238aa01abfeb29f38e218116217560a1b6401", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp312-cp312-win_amd64.whl", hash = "sha256:4c87d4bb691765173f186f5a9882fb85a75301547c4186dd75688bba907076e4", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:69093b64b2762c43df17be2db2be163029963d90bc3f1801500fdeb723e54833", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:ba77816bbde883c0c2075a1e284cf2e6f324472d4523442f5e3ae0812a98ae1e", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313-win_amd64.whl", hash = "sha256:159c7d32de5256ed946b4188335b00d11e0c3f7838998f94eaa98384b0249600", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:1777906f08c75be6daf9a11946894b63ef368951e7aa52da83f8bfc824856123", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:4388677d0903db4892587d5c87ab6f6cb31d19bac4838e083db022682036e152", upload-time = "2026-05-12T16:20:36Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp313-cp313t-win_amd64.whl", hash = "sha256:1d38651bd624dbbaf5ad77ccac93a60b16fc81f1ccacf2f5268bbc0dd2e7e1d7", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:c7e3c5662278ab5d64d150cc17694060e9ce5875b8739094dadf07a4ba45b90e", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:1417b91354698b045a4127cf7108b1dc8af1351a3002ecc5661c3ca69df577d4", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314-win_amd64.whl", hash = "sha256:58759994610e69773b1d7b038dfb2c72e8bf237fb3ba765f2ab1d6af7835dd88", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:4d2effc9740e711ea9b2db99e8874977eb6a3baa8b830b8a7416b2fde416de64", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:ae89bf1a81f3c7fdd222fbbddd8135a054d24ddd13f3b410e4f96d75f72952df", upload-time = "2026-05-12T16:20:37Z" }, + { url = "https://download-r2.pytorch.org/whl/cpu/torchvision-0.27.0%2Bcpu-cp314-cp314t-win_amd64.whl", hash = "sha256:c8b3c995d4294551b5ab33cbcf60d700819dc23d53b21a9c74936e521c88de33", upload-time = "2026-05-12T16:20:37Z" }, +] + +[[package]] +name = "torchvision" +version = "0.27.0+cu130" +source = { registry = "https://download.pytorch.org/whl/cu130" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] +dependencies = [ + { name = "numpy" }, + { name = "pillow" }, + { name = "torch", version = "2.12.0+cu130", source = { registry = "https://download.pytorch.org/whl/cu130" } }, +] +wheels = [ + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:0a839a2921410b1135add4c3d90f784c9d1e9e9f3c7b401b216d356ddca23ab2", upload-time = "2026-05-12T16:20:44Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:664dff46fac97a730c90a976a370ae2cad52780df6ae40fad74be77eee8b4528", upload-time = "2026-05-12T16:20:44Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp312-cp312-win_amd64.whl", hash = "sha256:a79f78d23557b5299c1a1eceeef846d6799ea0a3afe30c600c80ebd26a80bbf8", upload-time = "2026-05-13T02:00:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:da81245777c47f6dfd60e02f510d9778fb7f6e23119e2fc1ea1bb06777aae338", upload-time = "2026-05-12T16:20:44Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:afa4128f37066b83af9d426841a53147dd3c208efea893c93dc3eb6fa2af2287", upload-time = "2026-05-12T16:20:44Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313-win_amd64.whl", hash = "sha256:31533c28f23bf642989a9ae12caa40a2f8cc9b443d556ba2ffb7a51f759e6a11", upload-time = "2026-05-13T02:00:46Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:bb511f033cd3d6f304dc25753d2a28a1d77aa4dd54a219242d9df7fa57d8dd0a", upload-time = "2026-05-12T16:20:44Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:0c375ac4e9a1c09308f81b73d111d50b76eec335dc91a1811ae370467db2cf47", upload-time = "2026-05-12T16:20:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp313-cp313t-win_amd64.whl", hash = "sha256:34d108e1ce8255e017bf1f732a51ab2e9ddffb443d118db499a0fbbeb0164650", upload-time = "2026-05-13T02:00:47Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:5226fd6b558bb06a594959948e76e19ac73eec3d7ee0acc7c7b1ae3e061b5698", upload-time = "2026-05-12T16:20:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:74c2e33effcdad257800c178f87f22cff311efe7037568b3feae7b4e191ce209", upload-time = "2026-05-12T16:20:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314-win_amd64.whl", hash = "sha256:a87da1d0019ad7481e8d1ec0071e8c2f51145898248e8c590a1d886feee82129", upload-time = "2026-05-13T02:00:47Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:00fc9eaf3231a11ee566162483325b18f32635cab48babd7e0d775ab1fb047e4", upload-time = "2026-05-12T16:20:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:541e9ae408b5a43d623313eee52ef39732e04d1284807b926724e5e391a45d1b", upload-time = "2026-05-12T16:20:45Z" }, + { url = "https://download-r2.pytorch.org/whl/cu130/torchvision-0.27.0%2Bcu130-cp314-cp314t-win_amd64.whl", hash = "sha256:3641b3bb5ad0150e694c9d7042b8a2fb5e0683d5bcf701fa99a2200f98b7c91b", upload-time = "2026-05-13T02:00:48Z" }, +] + +[[package]] +name = "tqdm" +version = "4.67.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-6-kgpipe-cpu' and extra == 'extra-6-kgpipe-cuda')" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" }, +] + +[[package]] +name = "transformers" +version = "5.8.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy" }, + { name = "packaging" }, + { name = "pyyaml" }, + { name = "regex" }, + { name = "safetensors" }, + { name = "tokenizers" }, + { name = "tqdm" }, + { name = "typer" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e7/e6/4134ea2fbea322cddc7ffc94a0d8ee47fe32ce8e876b320cd37d88edfc4d/transformers-5.8.1.tar.gz", hash = "sha256:4dd5b6de4105725104d84fd6abd74b305f4debfc251b38c648ee5dd087cf543b", size = 8532019, upload-time = "2026-05-13T03:21:57.234Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fc/b1/8be7e7ef0b5200491312201918b6125ef9c9df9dd0f0240ccef9ac824e6b/transformers-5.8.1-py3-none-any.whl", hash = "sha256:5340fb95962162cdfdae5cc91d7f8fedd92ed75216c1154c5e1f590fcf56dd0e", size = 10632882, upload-time = "2026-05-13T03:21:52.876Z" }, +] + +[[package]] +name = "triton" +version = "3.7.0" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f7/13/ec05adfcd87311d532ba61e3af143e8be59fcd26675884c4682841406a20/triton-3.7.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a4bf49b00a7a377a68a6da603a876e797614e6455a80e9021669c476a953ad9a", size = 188505104, upload-time = "2026-05-07T19:05:09.843Z" }, + { url = "https://files.pythonhosted.org/packages/62/7b/468a576e35beef1426e0828e28e9ba9e65f5474d496f16ee126c15646324/triton-3.7.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8f111161d49bf903c0eaedde3962353a3d841c08a836839b7cc1025b8426efcf", size = 201457567, upload-time = "2026-05-07T18:46:13.505Z" }, + { url = "https://files.pythonhosted.org/packages/01/e1/a59a583de59b8f62c495d67c80ee3ea97d09e91ac80c4c6e76456ed8d8ac/triton-3.7.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abdf6beaa89b1bcfb9a43cd990536ce66091a997841a4814b260b7bee4c88c3c", size = 188503209, upload-time = "2026-05-07T19:05:17.935Z" }, + { url = "https://files.pythonhosted.org/packages/30/b1/b7507bb9815d403927c8dd51d4158ed2e11751a92dbc118a044f247b6848/triton-3.7.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a35d7afe3f3f058e7ec49fcce09794049e0ffc5c59019ac25ec3413741b8c4e7", size = 201453566, upload-time = "2026-05-07T18:46:20.427Z" }, + { url = "https://files.pythonhosted.org/packages/a6/8f/0bea7a6a0c989315c9135a1d7fb37e41905cfb3a17cbc1f10044ebd4cc3a/triton-3.7.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cc1d61c172d257db80ddf42595131fb196ad2e9bdd751e90fe2ef13531734e8b", size = 188612899, upload-time = "2026-05-07T19:05:24.955Z" }, + { url = "https://files.pythonhosted.org/packages/e1/02/d96f57828d0912aec733b9bc7e0e7dbfd2c6f079a8fa433ac25cb93d1a30/triton-3.7.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:70fb9bbdc9f400afc54bbf6eb2670af28829a6ae3996863317964783141daf56", size = 201553816, upload-time = "2026-05-07T18:46:27.49Z" }, + { url = "https://files.pythonhosted.org/packages/40/fb/82a802dac4689f2a2fb2e69302e6a138eecc3e175bbe976ba3cfc717683a/triton-3.7.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c4a44a8476d0d3571eac4e4d1048e1ff75aad81a09ff4602ccfc56c6dea1672e", size = 188507879, upload-time = "2026-05-07T19:05:32.209Z" }, + { url = "https://files.pythonhosted.org/packages/8f/af/9904ec6d3c93d9b24e5ec360445bbdf758b7f00bfbeedb89cb0eb64eb8bb/triton-3.7.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b9b85e72968a9d8bba5ddb24e9b64aaabaf48affb042f2755cb7cfa92b7531ce", size = 201460637, upload-time = "2026-05-07T18:46:34.749Z" }, + { url = "https://files.pythonhosted.org/packages/a1/f9/4835a8ea746b88727d8899f4e3ccce4f9cacb38abfc3bb0a638266c53111/triton-3.7.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:18a160de426fd99f92b0baf509045360afbd3bfaa0b4a5171dde800ec9f09684", size = 188608706, upload-time = "2026-05-07T19:05:39.218Z" }, + { url = "https://files.pythonhosted.org/packages/c1/68/fa86e5a39608000f645535b2c124920126327ab731f8c4fafd5b07ff8d4b/triton-3.7.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ce061073102714b725f3660ec6939d94a1da7984b3aa99c921417cae273672f5", size = 201546766, upload-time = "2026-05-07T18:46:42.088Z" }, +] + +[[package]] +name = "typer" +version = "0.25.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "annotated-doc" }, + { name = "click" }, + { name = "rich" }, + { name = "shellingham" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/51/9aed62104cea109b820bbd6c14245af756112017d309da813ef107d42e7e/typer-0.25.1.tar.gz", hash = "sha256:9616eb8853a09ffeabab1698952f33c6f29ffdbceb4eaeecf571880e8d7664cc", size = 122276, upload-time = "2026-04-30T19:32:16.964Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3f/f9/2b3ff4e56e5fa7debfaf9eb135d0da96f3e9a1d5b27222223c7296336e5f/typer-0.25.1-py3-none-any.whl", hash = "sha256:75caa44ed46a03fb2dab8808753ffacdbfea88495e74c85a28c5eefcf5f39c89", size = 58409, upload-time = "2026-04-30T19:32:18.271Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] + +[[package]] +name = "typing-inspection" +version = "0.4.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/55/e3/70399cb7dd41c10ac53367ae42139cf4b1ca5f36bb3dc6c9d33acdb43655/typing_inspection-0.4.2.tar.gz", hash = "sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464", size = 75949, upload-time = "2025-10-01T02:14:41.687Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/9b/47798a6c91d8bdb567fe2698fe81e0c6b7cb7ef4d13da4114b41d239f65d/typing_inspection-0.4.2-py3-none-any.whl", hash = "sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7", size = 14611, upload-time = "2025-10-01T02:14:40.154Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/19/1b9b0e29f30c6d35cb345486df41110984ea67ae69dddbc0e8a100999493/tzdata-2026.2.tar.gz", hash = "sha256:9173fde7d80d9018e02a662e168e5a2d04f87c41ea174b139fbef642eda62d10", size = 198254, upload-time = "2026-04-24T15:22:08.651Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/e4/dccd7f47c4b64213ac01ef921a1337ee6e30e8c6466046018326977efd95/tzdata-2026.2-py2.py3-none-any.whl", hash = "sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7", size = 349321, upload-time = "2026-04-24T15:22:05.876Z" }, +] + +[[package]] +name = "urllib3" +version = "2.7.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/53/0c/06f8b233b8fd13b9e5ee11424ef85419ba0d8ba0b3138bf360be2ff56953/urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c", size = 433602, upload-time = "2026-05-07T16:13:18.596Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897", size = 131087, upload-time = "2026-05-07T16:13:17.151Z" }, +] + +[[package]] +name = "uvicorn" +version = "0.46.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "h11" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1f/93/041fca8274050e40e6791f267d82e0e2e27dd165627bd640d3e0e378d877/uvicorn-0.46.0.tar.gz", hash = "sha256:fb9da0926999cc6cb22dc7cd71a94a632f078e6ae47ff683c5c420750fb7413d", size = 88758, upload-time = "2026-04-23T07:16:00.151Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/31/a3/5b1562db76a5a488274b2332a97199b32d0442aca0ed193697fd47786316/uvicorn-0.46.0-py3-none-any.whl", hash = "sha256:bbebbcbed972d162afca128605223022bedd345b7bc7855ce66deb31487a9048", size = 70926, upload-time = "2026-04-23T07:15:58.355Z" }, +] + +[[package]] +name = "watchdog" +version = "6.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/db/7d/7f3d619e951c88ed75c6037b246ddcf2d322812ee8ea189be89511721d54/watchdog-6.0.0.tar.gz", hash = "sha256:9ddf7c82fda3ae8e24decda1338ede66e1c99883db93711d8fb941eaa2d8c282", size = 131220, upload-time = "2024-11-01T14:07:13.037Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/39/ea/3930d07dafc9e286ed356a679aa02d777c06e9bfd1164fa7c19c288a5483/watchdog-6.0.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:bdd4e6f14b8b18c334febb9c4425a878a2ac20efd1e0b231978e7b150f92a948", size = 96471, upload-time = "2024-11-01T14:06:37.745Z" }, + { url = "https://files.pythonhosted.org/packages/12/87/48361531f70b1f87928b045df868a9fd4e253d9ae087fa4cf3f7113be363/watchdog-6.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c7c15dda13c4eb00d6fb6fc508b3c0ed88b9d5d374056b239c4ad1611125c860", size = 88449, upload-time = "2024-11-01T14:06:39.748Z" }, + { url = "https://files.pythonhosted.org/packages/5b/7e/8f322f5e600812e6f9a31b75d242631068ca8f4ef0582dd3ae6e72daecc8/watchdog-6.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6f10cb2d5902447c7d0da897e2c6768bca89174d0c6e1e30abec5421af97a5b0", size = 89054, upload-time = "2024-11-01T14:06:41.009Z" }, + { url = "https://files.pythonhosted.org/packages/68/98/b0345cabdce2041a01293ba483333582891a3bd5769b08eceb0d406056ef/watchdog-6.0.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:490ab2ef84f11129844c23fb14ecf30ef3d8a6abafd3754a6f75ca1e6654136c", size = 96480, upload-time = "2024-11-01T14:06:42.952Z" }, + { url = "https://files.pythonhosted.org/packages/85/83/cdf13902c626b28eedef7ec4f10745c52aad8a8fe7eb04ed7b1f111ca20e/watchdog-6.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:76aae96b00ae814b181bb25b1b98076d5fc84e8a53cd8885a318b42b6d3a5134", size = 88451, upload-time = "2024-11-01T14:06:45.084Z" }, + { url = "https://files.pythonhosted.org/packages/fe/c4/225c87bae08c8b9ec99030cd48ae9c4eca050a59bf5c2255853e18c87b50/watchdog-6.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a175f755fc2279e0b7312c0035d52e27211a5bc39719dd529625b1930917345b", size = 89057, upload-time = "2024-11-01T14:06:47.324Z" }, + { url = "https://files.pythonhosted.org/packages/a9/c7/ca4bf3e518cb57a686b2feb4f55a1892fd9a3dd13f470fca14e00f80ea36/watchdog-6.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:7607498efa04a3542ae3e05e64da8202e58159aa1fa4acddf7678d34a35d4f13", size = 79079, upload-time = "2024-11-01T14:06:59.472Z" }, + { url = "https://files.pythonhosted.org/packages/5c/51/d46dc9332f9a647593c947b4b88e2381c8dfc0942d15b8edc0310fa4abb1/watchdog-6.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:9041567ee8953024c83343288ccc458fd0a2d811d6a0fd68c4c22609e3490379", size = 79078, upload-time = "2024-11-01T14:07:01.431Z" }, + { url = "https://files.pythonhosted.org/packages/d4/57/04edbf5e169cd318d5f07b4766fee38e825d64b6913ca157ca32d1a42267/watchdog-6.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:82dc3e3143c7e38ec49d61af98d6558288c415eac98486a5c581726e0737c00e", size = 79076, upload-time = "2024-11-01T14:07:02.568Z" }, + { url = "https://files.pythonhosted.org/packages/ab/cc/da8422b300e13cb187d2203f20b9253e91058aaf7db65b74142013478e66/watchdog-6.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:212ac9b8bf1161dc91bd09c048048a95ca3a4c4f5e5d4a7d1b1a7d5752a7f96f", size = 79077, upload-time = "2024-11-01T14:07:03.893Z" }, + { url = "https://files.pythonhosted.org/packages/2c/3b/b8964e04ae1a025c44ba8e4291f86e97fac443bca31de8bd98d3263d2fcf/watchdog-6.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:e3df4cbb9a450c6d49318f6d14f4bbc80d763fa587ba46ec86f99f9e6876bb26", size = 79078, upload-time = "2024-11-01T14:07:05.189Z" }, + { url = "https://files.pythonhosted.org/packages/62/ae/a696eb424bedff7407801c257d4b1afda455fe40821a2be430e173660e81/watchdog-6.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:2cce7cfc2008eb51feb6aab51251fd79b85d9894e98ba847408f662b3395ca3c", size = 79077, upload-time = "2024-11-01T14:07:06.376Z" }, + { url = "https://files.pythonhosted.org/packages/b5/e8/dbf020b4d98251a9860752a094d09a65e1b436ad181faf929983f697048f/watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2", size = 79078, upload-time = "2024-11-01T14:07:07.547Z" }, + { url = "https://files.pythonhosted.org/packages/07/f6/d0e5b343768e8bcb4cda79f0f2f55051bf26177ecd5651f84c07567461cf/watchdog-6.0.0-py3-none-win32.whl", hash = "sha256:07df1fdd701c5d4c8e55ef6cf55b8f0120fe1aef7ef39a1c6fc6bc2e606d517a", size = 79065, upload-time = "2024-11-01T14:07:09.525Z" }, + { url = "https://files.pythonhosted.org/packages/db/d9/c495884c6e548fce18a8f40568ff120bc3a4b7b99813081c8ac0c936fa64/watchdog-6.0.0-py3-none-win_amd64.whl", hash = "sha256:cbafb470cf848d93b5d013e2ecb245d4aa1c8fd0504e863ccefa32445359d680", size = 79070, upload-time = "2024-11-01T14:07:10.686Z" }, + { url = "https://files.pythonhosted.org/packages/33/e8/e40370e6d74ddba47f002a32919d91310d6074130fe4e17dabcafc15cbf1/watchdog-6.0.0-py3-none-win_ia64.whl", hash = "sha256:a1914259fa9e1454315171103c6a30961236f508b9b623eae470268bbcc6a22f", size = 79067, upload-time = "2024-11-01T14:07:11.845Z" }, +] + +[[package]] +name = "websockets" +version = "16.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/04/24/4b2031d72e840ce4c1ccb255f693b15c334757fc50023e4db9537080b8c4/websockets-16.0.tar.gz", hash = "sha256:5f6261a5e56e8d5c42a4497b364ea24d94d9563e8fbd44e78ac40879c60179b5", size = 179346, upload-time = "2026-01-10T09:23:47.181Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/84/7b/bac442e6b96c9d25092695578dda82403c77936104b5682307bd4deb1ad4/websockets-16.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:71c989cbf3254fbd5e84d3bff31e4da39c43f884e64f2551d14bb3c186230f00", size = 177365, upload-time = "2026-01-10T09:22:46.787Z" }, + { url = "https://files.pythonhosted.org/packages/b0/fe/136ccece61bd690d9c1f715baaeefd953bb2360134de73519d5df19d29ca/websockets-16.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8b6e209ffee39ff1b6d0fa7bfef6de950c60dfb91b8fcead17da4ee539121a79", size = 175038, upload-time = "2026-01-10T09:22:47.999Z" }, + { url = "https://files.pythonhosted.org/packages/40/1e/9771421ac2286eaab95b8575b0cb701ae3663abf8b5e1f64f1fd90d0a673/websockets-16.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:86890e837d61574c92a97496d590968b23c2ef0aeb8a9bc9421d174cd378ae39", size = 175328, upload-time = "2026-01-10T09:22:49.809Z" }, + { url = "https://files.pythonhosted.org/packages/18/29/71729b4671f21e1eaa5d6573031ab810ad2936c8175f03f97f3ff164c802/websockets-16.0-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9b5aca38b67492ef518a8ab76851862488a478602229112c4b0d58d63a7a4d5c", size = 184915, upload-time = "2026-01-10T09:22:51.071Z" }, + { url = "https://files.pythonhosted.org/packages/97/bb/21c36b7dbbafc85d2d480cd65df02a1dc93bf76d97147605a8e27ff9409d/websockets-16.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e0334872c0a37b606418ac52f6ab9cfd17317ac26365f7f65e203e2d0d0d359f", size = 186152, upload-time = "2026-01-10T09:22:52.224Z" }, + { url = "https://files.pythonhosted.org/packages/4a/34/9bf8df0c0cf88fa7bfe36678dc7b02970c9a7d5e065a3099292db87b1be2/websockets-16.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a0b31e0b424cc6b5a04b8838bbaec1688834b2383256688cf47eb97412531da1", size = 185583, upload-time = "2026-01-10T09:22:53.443Z" }, + { url = "https://files.pythonhosted.org/packages/47/88/4dd516068e1a3d6ab3c7c183288404cd424a9a02d585efbac226cb61ff2d/websockets-16.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:485c49116d0af10ac698623c513c1cc01c9446c058a4e61e3bf6c19dff7335a2", size = 184880, upload-time = "2026-01-10T09:22:55.033Z" }, + { url = "https://files.pythonhosted.org/packages/91/d6/7d4553ad4bf1c0421e1ebd4b18de5d9098383b5caa1d937b63df8d04b565/websockets-16.0-cp312-cp312-win32.whl", hash = "sha256:eaded469f5e5b7294e2bdca0ab06becb6756ea86894a47806456089298813c89", size = 178261, upload-time = "2026-01-10T09:22:56.251Z" }, + { url = "https://files.pythonhosted.org/packages/c3/f0/f3a17365441ed1c27f850a80b2bc680a0fa9505d733fe152fdf5e98c1c0b/websockets-16.0-cp312-cp312-win_amd64.whl", hash = "sha256:5569417dc80977fc8c2d43a86f78e0a5a22fee17565d78621b6bb264a115d4ea", size = 178693, upload-time = "2026-01-10T09:22:57.478Z" }, + { url = "https://files.pythonhosted.org/packages/cc/9c/baa8456050d1c1b08dd0ec7346026668cbc6f145ab4e314d707bb845bf0d/websockets-16.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:878b336ac47938b474c8f982ac2f7266a540adc3fa4ad74ae96fea9823a02cc9", size = 177364, upload-time = "2026-01-10T09:22:59.333Z" }, + { url = "https://files.pythonhosted.org/packages/7e/0c/8811fc53e9bcff68fe7de2bcbe75116a8d959ac699a3200f4847a8925210/websockets-16.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:52a0fec0e6c8d9a784c2c78276a48a2bdf099e4ccc2a4cad53b27718dbfd0230", size = 175039, upload-time = "2026-01-10T09:23:01.171Z" }, + { url = "https://files.pythonhosted.org/packages/aa/82/39a5f910cb99ec0b59e482971238c845af9220d3ab9fa76dd9162cda9d62/websockets-16.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e6578ed5b6981005df1860a56e3617f14a6c307e6a71b4fff8c48fdc50f3ed2c", size = 175323, upload-time = "2026-01-10T09:23:02.341Z" }, + { url = "https://files.pythonhosted.org/packages/bd/28/0a25ee5342eb5d5f297d992a77e56892ecb65e7854c7898fb7d35e9b33bd/websockets-16.0-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95724e638f0f9c350bb1c2b0a7ad0e83d9cc0c9259f3ea94e40d7b02a2179ae5", size = 184975, upload-time = "2026-01-10T09:23:03.756Z" }, + { url = "https://files.pythonhosted.org/packages/f9/66/27ea52741752f5107c2e41fda05e8395a682a1e11c4e592a809a90c6a506/websockets-16.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c0204dc62a89dc9d50d682412c10b3542d748260d743500a85c13cd1ee4bde82", size = 186203, upload-time = "2026-01-10T09:23:05.01Z" }, + { url = "https://files.pythonhosted.org/packages/37/e5/8e32857371406a757816a2b471939d51c463509be73fa538216ea52b792a/websockets-16.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:52ac480f44d32970d66763115edea932f1c5b1312de36df06d6b219f6741eed8", size = 185653, upload-time = "2026-01-10T09:23:06.301Z" }, + { url = "https://files.pythonhosted.org/packages/9b/67/f926bac29882894669368dc73f4da900fcdf47955d0a0185d60103df5737/websockets-16.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6e5a82b677f8f6f59e8dfc34ec06ca6b5b48bc4fcda346acd093694cc2c24d8f", size = 184920, upload-time = "2026-01-10T09:23:07.492Z" }, + { url = "https://files.pythonhosted.org/packages/3c/a1/3d6ccdcd125b0a42a311bcd15a7f705d688f73b2a22d8cf1c0875d35d34a/websockets-16.0-cp313-cp313-win32.whl", hash = "sha256:abf050a199613f64c886ea10f38b47770a65154dc37181bfaff70c160f45315a", size = 178255, upload-time = "2026-01-10T09:23:09.245Z" }, + { url = "https://files.pythonhosted.org/packages/6b/ae/90366304d7c2ce80f9b826096a9e9048b4bb760e44d3b873bb272cba696b/websockets-16.0-cp313-cp313-win_amd64.whl", hash = "sha256:3425ac5cf448801335d6fdc7ae1eb22072055417a96cc6b31b3861f455fbc156", size = 178689, upload-time = "2026-01-10T09:23:10.483Z" }, + { url = "https://files.pythonhosted.org/packages/f3/1d/e88022630271f5bd349ed82417136281931e558d628dd52c4d8621b4a0b2/websockets-16.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:8cc451a50f2aee53042ac52d2d053d08bf89bcb31ae799cb4487587661c038a0", size = 177406, upload-time = "2026-01-10T09:23:12.178Z" }, + { url = "https://files.pythonhosted.org/packages/f2/78/e63be1bf0724eeb4616efb1ae1c9044f7c3953b7957799abb5915bffd38e/websockets-16.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:daa3b6ff70a9241cf6c7fc9e949d41232d9d7d26fd3522b1ad2b4d62487e9904", size = 175085, upload-time = "2026-01-10T09:23:13.511Z" }, + { url = "https://files.pythonhosted.org/packages/bb/f4/d3c9220d818ee955ae390cf319a7c7a467beceb24f05ee7aaaa2414345ba/websockets-16.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:fd3cb4adb94a2a6e2b7c0d8d05cb94e6f1c81a0cf9dc2694fb65c7e8d94c42e4", size = 175328, upload-time = "2026-01-10T09:23:14.727Z" }, + { url = "https://files.pythonhosted.org/packages/63/bc/d3e208028de777087e6fb2b122051a6ff7bbcca0d6df9d9c2bf1dd869ae9/websockets-16.0-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:781caf5e8eee67f663126490c2f96f40906594cb86b408a703630f95550a8c3e", size = 185044, upload-time = "2026-01-10T09:23:15.939Z" }, + { url = "https://files.pythonhosted.org/packages/ad/6e/9a0927ac24bd33a0a9af834d89e0abc7cfd8e13bed17a86407a66773cc0e/websockets-16.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:caab51a72c51973ca21fa8a18bd8165e1a0183f1ac7066a182ff27107b71e1a4", size = 186279, upload-time = "2026-01-10T09:23:17.148Z" }, + { url = "https://files.pythonhosted.org/packages/b9/ca/bf1c68440d7a868180e11be653c85959502efd3a709323230314fda6e0b3/websockets-16.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19c4dc84098e523fd63711e563077d39e90ec6702aff4b5d9e344a60cb3c0cb1", size = 185711, upload-time = "2026-01-10T09:23:18.372Z" }, + { url = "https://files.pythonhosted.org/packages/c4/f8/fdc34643a989561f217bb477cbc47a3a07212cbda91c0e4389c43c296ebf/websockets-16.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a5e18a238a2b2249c9a9235466b90e96ae4795672598a58772dd806edc7ac6d3", size = 184982, upload-time = "2026-01-10T09:23:19.652Z" }, + { url = "https://files.pythonhosted.org/packages/dd/d1/574fa27e233764dbac9c52730d63fcf2823b16f0856b3329fc6268d6ae4f/websockets-16.0-cp314-cp314-win32.whl", hash = "sha256:a069d734c4a043182729edd3e9f247c3b2a4035415a9172fd0f1b71658a320a8", size = 177915, upload-time = "2026-01-10T09:23:21.458Z" }, + { url = "https://files.pythonhosted.org/packages/8a/f1/ae6b937bf3126b5134ce1f482365fde31a357c784ac51852978768b5eff4/websockets-16.0-cp314-cp314-win_amd64.whl", hash = "sha256:c0ee0e63f23914732c6d7e0cce24915c48f3f1512ec1d079ed01fc629dab269d", size = 178381, upload-time = "2026-01-10T09:23:22.715Z" }, + { url = "https://files.pythonhosted.org/packages/06/9b/f791d1db48403e1f0a27577a6beb37afae94254a8c6f08be4a23e4930bc0/websockets-16.0-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:a35539cacc3febb22b8f4d4a99cc79b104226a756aa7400adc722e83b0d03244", size = 177737, upload-time = "2026-01-10T09:23:24.523Z" }, + { url = "https://files.pythonhosted.org/packages/bd/40/53ad02341fa33b3ce489023f635367a4ac98b73570102ad2cdd770dacc9a/websockets-16.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:b784ca5de850f4ce93ec85d3269d24d4c82f22b7212023c974c401d4980ebc5e", size = 175268, upload-time = "2026-01-10T09:23:25.781Z" }, + { url = "https://files.pythonhosted.org/packages/74/9b/6158d4e459b984f949dcbbb0c5d270154c7618e11c01029b9bbd1bb4c4f9/websockets-16.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:569d01a4e7fba956c5ae4fc988f0d4e187900f5497ce46339c996dbf24f17641", size = 175486, upload-time = "2026-01-10T09:23:27.033Z" }, + { url = "https://files.pythonhosted.org/packages/e5/2d/7583b30208b639c8090206f95073646c2c9ffd66f44df967981a64f849ad/websockets-16.0-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:50f23cdd8343b984957e4077839841146f67a3d31ab0d00e6b824e74c5b2f6e8", size = 185331, upload-time = "2026-01-10T09:23:28.259Z" }, + { url = "https://files.pythonhosted.org/packages/45/b0/cce3784eb519b7b5ad680d14b9673a31ab8dcb7aad8b64d81709d2430aa8/websockets-16.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:152284a83a00c59b759697b7f9e9cddf4e3c7861dd0d964b472b70f78f89e80e", size = 186501, upload-time = "2026-01-10T09:23:29.449Z" }, + { url = "https://files.pythonhosted.org/packages/19/60/b8ebe4c7e89fb5f6cdf080623c9d92789a53636950f7abacfc33fe2b3135/websockets-16.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bc59589ab64b0022385f429b94697348a6a234e8ce22544e3681b2e9331b5944", size = 186062, upload-time = "2026-01-10T09:23:31.368Z" }, + { url = "https://files.pythonhosted.org/packages/88/a8/a080593f89b0138b6cba1b28f8df5673b5506f72879322288b031337c0b8/websockets-16.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32da954ffa2814258030e5a57bc73a3635463238e797c7375dc8091327434206", size = 185356, upload-time = "2026-01-10T09:23:32.627Z" }, + { url = "https://files.pythonhosted.org/packages/c2/b6/b9afed2afadddaf5ebb2afa801abf4b0868f42f8539bfe4b071b5266c9fe/websockets-16.0-cp314-cp314t-win32.whl", hash = "sha256:5a4b4cc550cb665dd8a47f868c8d04c8230f857363ad3c9caf7a0c3bf8c61ca6", size = 178085, upload-time = "2026-01-10T09:23:33.816Z" }, + { url = "https://files.pythonhosted.org/packages/9f/3e/28135a24e384493fa804216b79a6a6759a38cc4ff59118787b9fb693df93/websockets-16.0-cp314-cp314t-win_amd64.whl", hash = "sha256:b14dc141ed6d2dde437cddb216004bcac6a1df0935d79656387bd41632ba0bbd", size = 178531, upload-time = "2026-01-10T09:23:35.016Z" }, + { url = "https://files.pythonhosted.org/packages/6f/28/258ebab549c2bf3e64d2b0217b973467394a9cea8c42f70418ca2c5d0d2e/websockets-16.0-py3-none-any.whl", hash = "sha256:1637db62fad1dc833276dded54215f2c7fa46912301a24bd94d45d46a011ceec", size = 171598, upload-time = "2026-01-10T09:23:45.395Z" }, +]