diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml index 1a92156..60e4340 100644 --- a/.github/FUNDING.yml +++ b/.github/FUNDING.yml @@ -1 +1,3 @@ github: harumiWeb +custom: + - https://buymeacoffee.com/harumi \ No newline at end of file diff --git a/README.ja.md b/README.ja.md index a46b0f3..87ea512 100644 --- a/README.ja.md +++ b/README.ja.md @@ -8,9 +8,21 @@ Excel Structured Extraction Engine

+
+ + Buy Me A Coffee + +
+ +
+
-[![PyPI version](https://badge.fury.io/py/exstruct.svg)](https://pypi.org/project/exstruct/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/exstruct?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/exstruct) ![Licence: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue?style=flat-square) [![pytest](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml/badge.svg)](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/e081cb4f634e4175b259eb7c34f54f60)](https://app.codacy.com/gh/harumiWeb/exstruct/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![codecov](https://codecov.io/gh/harumiWeb/exstruct/graph/badge.svg?token=2XI1O8TTA9)](https://codecov.io/gh/harumiWeb/exstruct) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/harumiWeb/exstruct) ![GitHub Repo stars](https://img.shields.io/github/stars/harumiWeb/exstruct) +[![PyPI version](https://badge.fury.io/py/exstruct.svg)](https://pypi.org/project/exstruct/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/exstruct?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/exstruct) ![Licence: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue?style=flat-square) [![pytest](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml/badge.svg)](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/e081cb4f634e4175b259eb7c34f54f60)](https://app.codacy.com/gh/harumiWeb/exstruct/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![codecov](https://codecov.io/gh/harumiWeb/exstruct/graph/badge.svg?token=2XI1O8TTA9)](https://codecov.io/gh/harumiWeb/exstruct) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/harumiWeb/exstruct) ![GitHub Repo stars](https://img.shields.io/github/stars/harumiWeb/exstruct) [![GitHub Sponsors](https://img.shields.io/badge/GitHub%20Sponsors-Sponsor-EA4AAA?logo=githubsponsors&logoColor=white)](https://github.com/sponsors/harumiWeb)
diff --git a/README.md b/README.md index cf3e91b..6b388a8 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,21 @@ Excel Structured Extraction Engine

+
+ + Buy Me A Coffee + +
+ +
+
-[![PyPI version](https://badge.fury.io/py/exstruct.svg)](https://pypi.org/project/exstruct/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/exstruct?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/exstruct) ![Licence: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue?style=flat-square) [![pytest](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml/badge.svg)](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/e081cb4f634e4175b259eb7c34f54f60)](https://app.codacy.com/gh/harumiWeb/exstruct/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![codecov](https://codecov.io/gh/harumiWeb/exstruct/graph/badge.svg?token=2XI1O8TTA9)](https://codecov.io/gh/harumiWeb/exstruct) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/harumiWeb/exstruct) ![GitHub Repo stars](https://img.shields.io/github/stars/harumiWeb/exstruct) +[![PyPI version](https://badge.fury.io/py/exstruct.svg)](https://pypi.org/project/exstruct/) [![PyPI Downloads](https://static.pepy.tech/personalized-badge/exstruct?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/exstruct) ![Licence: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue?style=flat-square) [![pytest](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml/badge.svg)](https://github.com/harumiWeb/exstruct/actions/workflows/pytest.yml) [![Codacy Badge](https://app.codacy.com/project/badge/Grade/e081cb4f634e4175b259eb7c34f54f60)](https://app.codacy.com/gh/harumiWeb/exstruct/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade) [![codecov](https://codecov.io/gh/harumiWeb/exstruct/graph/badge.svg?token=2XI1O8TTA9)](https://codecov.io/gh/harumiWeb/exstruct) [![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/harumiWeb/exstruct) ![GitHub Repo stars](https://img.shields.io/github/stars/harumiWeb/exstruct) [![GitHub Sponsors](https://img.shields.io/badge/GitHub%20Sponsors-Sponsor-EA4AAA?logo=githubsponsors&logoColor=white)](https://github.com/sponsors/harumiWeb)
@@ -24,36 +36,36 @@

-# ExStruct — Excel Structured Extraction Engine - -ExStruct reads Excel workbooks into structured data and applies patch-based -editing workflows through a shared core. It provides extraction APIs, a -JSON-first editing CLI, and an MCP server for host-managed integrations, with -options tuned for LLM/RAG preprocessing, reviewable edit flows, and local -automation. - -- In COM/Excel environments (Windows), it performs rich extraction. -- In non-COM environments (Linux/macOS): - - direct OOXML parsing extracts cells, shapes, charts, table candidates, and print areas on a best-effort basis - - if the LibreOffice runtime is available, cells, table candidates, shapes, and charts are also extracted on a best-effort basis - -Detection heuristics, editing workflows, and output modes are adjustable for -LLM/RAG pipelines and local automation. - -## Main Features - -- **Excel -> structured JSON**: outputs cells, shapes, charts, SmartArt, table candidates, merged-cell ranges, print areas, and auto page-break areas by sheet or by area. -- **Output modes**: - - `light`: cells + table candidates + print areas + best-effort shapes/charts via direct OOXML for `.xlsx` / `.xlsm` - - `libreoffice`: best-effort non-COM mode for `.xlsx/.xlsm`. When the LibreOffice runtime is available, it adds merged cells, shapes, connectors, and charts - - `standard`: Excel COM mode with texted shapes + arrows, charts, SmartArt, and merged-cell ranges - - `verbose`: outputs all shapes with width/height and also emits cell hyperlinks -- **Formula extraction**: emits `formulas_map` (formula string -> cell coordinates) via OOXML or the selected openpyxl/COM path. It is enabled by default in `verbose` and can be controlled with `include_formulas_map`. -- **Formats**: JSON (compact by default, `--pretty` for formatting), YAML, and TOON (optional dependencies). -- **Workbook editing interfaces**: use the editing CLI for primary ExStruct edit flows, keep MCP for host-owned safety controls, and use `exstruct.edit` only when you need the same patch contract from Python. -- **Table detection tuning**: heuristics can be adjusted dynamically through the API. -- **Hyperlink extraction**: in `verbose` mode, or with `include_cell_links=True`, cell links are emitted in `links`. -- **Safe fallback**: `light` uses direct OOXML for `.xlsx` / `.xlsm` without Excel COM or LibreOffice. Unsupported OOXML or `colors_map` requests restart the complete openpyxl compatibility pipeline with an `ooxml_compatibility` warning; partial results are discarded. A drawing failure isolated to one sheet remains best-effort. +# ExStruct — Excel Structured Extraction Engine + +ExStruct reads Excel workbooks into structured data and applies patch-based +editing workflows through a shared core. It provides extraction APIs, a +JSON-first editing CLI, and an MCP server for host-managed integrations, with +options tuned for LLM/RAG preprocessing, reviewable edit flows, and local +automation. + +- In COM/Excel environments (Windows), it performs rich extraction. +- In non-COM environments (Linux/macOS): + - direct OOXML parsing extracts cells, shapes, charts, table candidates, and print areas on a best-effort basis + - if the LibreOffice runtime is available, cells, table candidates, shapes, and charts are also extracted on a best-effort basis + +Detection heuristics, editing workflows, and output modes are adjustable for +LLM/RAG pipelines and local automation. + +## Main Features + +- **Excel -> structured JSON**: outputs cells, shapes, charts, SmartArt, table candidates, merged-cell ranges, print areas, and auto page-break areas by sheet or by area. +- **Output modes**: + - `light`: cells + table candidates + print areas + best-effort shapes/charts via direct OOXML for `.xlsx` / `.xlsm` + - `libreoffice`: best-effort non-COM mode for `.xlsx/.xlsm`. When the LibreOffice runtime is available, it adds merged cells, shapes, connectors, and charts + - `standard`: Excel COM mode with texted shapes + arrows, charts, SmartArt, and merged-cell ranges + - `verbose`: outputs all shapes with width/height and also emits cell hyperlinks +- **Formula extraction**: emits `formulas_map` (formula string -> cell coordinates) via OOXML or the selected openpyxl/COM path. It is enabled by default in `verbose` and can be controlled with `include_formulas_map`. +- **Formats**: JSON (compact by default, `--pretty` for formatting), YAML, and TOON (optional dependencies). +- **Workbook editing interfaces**: use the editing CLI for primary ExStruct edit flows, keep MCP for host-owned safety controls, and use `exstruct.edit` only when you need the same patch contract from Python. +- **Table detection tuning**: heuristics can be adjusted dynamically through the API. +- **Hyperlink extraction**: in `verbose` mode, or with `include_cell_links=True`, cell links are emitted in `links`. +- **Safe fallback**: `light` uses direct OOXML for `.xlsx` / `.xlsm` without Excel COM or LibreOffice. Unsupported OOXML or `colors_map` requests restart the complete openpyxl compatibility pipeline with an `ooxml_compatibility` warning; partial results are discarded. A drawing failure isolated to one sheet remains best-effort. ## Installation @@ -61,21 +73,21 @@ LLM/RAG pipelines and local automation. pip install exstruct ``` -Optional extras: - -- Border-clustering acceleration: `pip install exstruct[fast]` (SciPy). Normal direct light extraction uses Python clustering. SciPy acceleration and `EXSTRUCT_BORDER_CLUSTER_BACKEND` selection apply to other modes and compatibility extraction; NumPy remains a core dependency. +Optional extras: + +- Border-clustering acceleration: `pip install exstruct[fast]` (SciPy). Normal direct light extraction uses Python clustering. SciPy acceleration and `EXSTRUCT_BORDER_CLUSTER_BACKEND` selection apply to other modes and compatibility extraction; NumPy remains a core dependency. - YAML: `pip install pyyaml` - TOON: `pip install python-toon` - Rendering (PDF/PNG): Excel + `pip install pypdfium2 pillow` (`mode=libreoffice` is not supported) -- Install these extras at once: `pip install exstruct[yaml,toon,render,fast]` +- Install these extras at once: `pip install exstruct[yaml,toon,render,fast]` + +Platform note: -Platform note: - -- On Debian/Ubuntu/WSL, install LibreOffice together with `python3-uno`. ExStruct probes a compatible system Python automatically for `mode=libreoffice`; if your environment needs an explicit interpreter, set `EXSTRUCT_LIBREOFFICE_PYTHON_PATH=/usr/bin/python3`. -- LibreOffice Python detection now runs the bundled bridge in `--probe` mode before selection. An incompatible `EXSTRUCT_LIBREOFFICE_PYTHON_PATH` fails fast instead of surfacing a delayed bridge `SyntaxError` during extraction. -- If the isolated temporary LibreOffice profile fails before the UNO socket becomes ready, ExStruct retries once with the shared/default LibreOffice profile as a compatibility fallback and reports per-attempt startup detail if both launches fail. - -## Quick Start CLI +- On Debian/Ubuntu/WSL, install LibreOffice together with `python3-uno`. ExStruct probes a compatible system Python automatically for `mode=libreoffice`; if your environment needs an explicit interpreter, set `EXSTRUCT_LIBREOFFICE_PYTHON_PATH=/usr/bin/python3`. +- LibreOffice Python detection now runs the bundled bridge in `--probe` mode before selection. An incompatible `EXSTRUCT_LIBREOFFICE_PYTHON_PATH` fails fast instead of surfacing a delayed bridge `SyntaxError` during extraction. +- If the isolated temporary LibreOffice profile fails before the UNO socket becomes ready, ExStruct retries once with the shared/default LibreOffice profile as a compatibility fallback and reports per-attempt startup detail if both launches fail. + +## Quick Start CLI ```bash exstruct input.xlsx > output.json # compact JSON to stdout by default @@ -83,75 +95,75 @@ exstruct input.xlsx -o out.json --pretty # write pretty JSON to a file exstruct input.xlsx --format yaml # YAML (requires pyyaml) exstruct input.xlsx --format toon # TOON (requires python-toon) exstruct input.xlsx --sheets-dir sheets/ # write one file per sheet -exstruct input.xlsx --auto-page-breaks-dir auto_areas/ # always shown; execution requires standard/verbose + Excel COM -exstruct input.xlsx --alpha-col # output column keys as A, B, ..., AA -exstruct input.xlsx --include-backend-metadata # include shape/chart backend metadata -exstruct input.xlsx --mode light # cells + table candidates + best-effort OOXML shapes/charts -exstruct input.xlsx --mode libreoffice # best-effort extraction of shapes/connectors/charts without COM +exstruct input.xlsx --auto-page-breaks-dir auto_areas/ # always shown; execution requires standard/verbose + Excel COM +exstruct input.xlsx --alpha-col # output column keys as A, B, ..., AA +exstruct input.xlsx --include-backend-metadata # include shape/chart backend metadata +exstruct input.xlsx --mode light # cells + table candidates + best-effort OOXML shapes/charts +exstruct input.xlsx --mode libreoffice # best-effort extraction of shapes/connectors/charts without COM exstruct input.xlsx --pdf --image # PDF and PNGs (Excel COM required) ``` -Auto page-break export is available from both the API and the CLI when Excel/COM is available. The CLI always exposes `--auto-page-breaks-dir`, but validates it at execution time. -`mode=libreoffice` rejects `--pdf`, `--image`, and `--auto-page-breaks-dir` early, and `mode=light` also rejects `--auto-page-breaks-dir`. Use `standard` or `verbose` with Excel COM for those features. -By default, the CLI keeps legacy 0-based numeric string column keys (`"0"`, `"1"`, ...). Use `--alpha-col` when you need Excel-style keys (`"A"`, `"B"`, ...). -By default, serialized shape/chart output omits backend metadata (`provenance`, `approximation_level`, `confidence`) to reduce token usage. Use `--include-backend-metadata` or the corresponding Python/MCP option when you need it. - -## Quick Start Editing CLI - -```bash -exstruct patch --input book.xlsx --ops ops.json --backend openpyxl -exstruct patch --input book.xlsx --ops - --dry-run --pretty < ops.json -exstruct make --output new.xlsx --ops ops.json --backend openpyxl -exstruct ops list -exstruct ops describe create_chart --pretty -exstruct validate --input book.xlsx --pretty -``` - -- `patch` and `make` print JSON `PatchResult` to stdout. -- `ops list` / `ops describe` expose the public patch-op schema. -- `validate` reports workbook readability (`is_readable`, `warnings`, `errors`). - -Recommended edit flow: - -1. Build patch ops. -2. Run `exstruct patch --dry-run` and inspect `PatchResult`, warnings, and diff. -3. Pin `--backend openpyxl` when you want the dry run and the real apply to use the same engine. -4. If you keep `--backend auto`, inspect `PatchResult.engine`; on Windows/Excel hosts the real apply may switch to COM. -5. Re-run without `--dry-run` only after the result is acceptable. - -## ExStruct CLI Skill - -ExStruct also ships one repo-owned Skill for agents that should follow the -editing CLI safely instead of rediscovering the workflow each time. - -Canonical repo source: - -- `.agents/skills/exstruct-cli/` - -You can install it with the following single command: - -```bash -npx skills add harumiWeb/exstruct/.agents/skills --skill exstruct-cli -``` - -If your runtime cannot use `npx skills add`, place the same folder manually -into a local skill directory that discovers `SKILL.md`-based skills. - -Use this Skill when the agent needs help choosing between `patch`, `make`, -`validate`, `ops list`, and `ops describe`, or when it should follow the safe -`validate -> dry-run -> inspect -> apply -> verify` workflow. - -Example prompt for agents: - -> Use `$exstruct-cli` to choose the right ExStruct editing CLI command, follow a safe validate/dry-run/inspect workflow, and explain any backend constraints for this workbook task. - -## MCP Server (stdio) - -MCP is the integration / compatibility layer around the same editing core. Use -it when you need host-managed path restrictions, transport mapping, artifact -mirroring, or approval-aware agent execution. For ordinary Python workbook -editing, `openpyxl` / `xlwings` are usually a better fit. For local shell or -agent workflows, prefer the editing CLI. +Auto page-break export is available from both the API and the CLI when Excel/COM is available. The CLI always exposes `--auto-page-breaks-dir`, but validates it at execution time. +`mode=libreoffice` rejects `--pdf`, `--image`, and `--auto-page-breaks-dir` early, and `mode=light` also rejects `--auto-page-breaks-dir`. Use `standard` or `verbose` with Excel COM for those features. +By default, the CLI keeps legacy 0-based numeric string column keys (`"0"`, `"1"`, ...). Use `--alpha-col` when you need Excel-style keys (`"A"`, `"B"`, ...). +By default, serialized shape/chart output omits backend metadata (`provenance`, `approximation_level`, `confidence`) to reduce token usage. Use `--include-backend-metadata` or the corresponding Python/MCP option when you need it. + +## Quick Start Editing CLI + +```bash +exstruct patch --input book.xlsx --ops ops.json --backend openpyxl +exstruct patch --input book.xlsx --ops - --dry-run --pretty < ops.json +exstruct make --output new.xlsx --ops ops.json --backend openpyxl +exstruct ops list +exstruct ops describe create_chart --pretty +exstruct validate --input book.xlsx --pretty +``` + +- `patch` and `make` print JSON `PatchResult` to stdout. +- `ops list` / `ops describe` expose the public patch-op schema. +- `validate` reports workbook readability (`is_readable`, `warnings`, `errors`). + +Recommended edit flow: + +1. Build patch ops. +2. Run `exstruct patch --dry-run` and inspect `PatchResult`, warnings, and diff. +3. Pin `--backend openpyxl` when you want the dry run and the real apply to use the same engine. +4. If you keep `--backend auto`, inspect `PatchResult.engine`; on Windows/Excel hosts the real apply may switch to COM. +5. Re-run without `--dry-run` only after the result is acceptable. + +## ExStruct CLI Skill + +ExStruct also ships one repo-owned Skill for agents that should follow the +editing CLI safely instead of rediscovering the workflow each time. + +Canonical repo source: + +- `.agents/skills/exstruct-cli/` + +You can install it with the following single command: + +```bash +npx skills add harumiWeb/exstruct/.agents/skills --skill exstruct-cli +``` + +If your runtime cannot use `npx skills add`, place the same folder manually +into a local skill directory that discovers `SKILL.md`-based skills. + +Use this Skill when the agent needs help choosing between `patch`, `make`, +`validate`, `ops list`, and `ops describe`, or when it should follow the safe +`validate -> dry-run -> inspect -> apply -> verify` workflow. + +Example prompt for agents: + +> Use `$exstruct-cli` to choose the right ExStruct editing CLI command, follow a safe validate/dry-run/inspect workflow, and explain any backend constraints for this workbook task. + +## MCP Server (stdio) + +MCP is the integration / compatibility layer around the same editing core. Use +it when you need host-managed path restrictions, transport mapping, artifact +mirroring, or approval-aware agent execution. For ordinary Python workbook +editing, `openpyxl` / `xlwings` are usually a better fit. For local shell or +agent workflows, prefer the editing CLI. ### Quick Start with `uvx` (recommended) @@ -177,24 +189,24 @@ pip install exstruct[mcp] exstruct-mcp --root C:\data --log-file C:\logs\exstruct-mcp.log --on-conflict rename ``` -Available tools: - -| Tool name | Description | -| ------------------------------- | -------------------------------------- | -| `exstruct_extract` | Extracts data from a workbook. | -| `exstruct_capture_sheet_images` | Captures sheet images. | -| `exstruct_make` | Creates a new workbook. | -| `exstruct_patch` | Applies editing patches to a workbook. | -| `exstruct_read_json_chunk` | Reads extracted JSON chunks. | -| `exstruct_read_range` | Reads cells from a specified range. | -| `exstruct_read_cells` | Reads data cell by cell. | -| `exstruct_read_formulas` | Reads cell formulas. | -| `exstruct_validate_input` | Validates input data. | - -For more details and API usage, see the documentation site: -[MCP Server](https://harumiweb.github.io/exstruct/mcp/) - -## Quick Start Python Extraction +Available tools: + +| Tool name | Description | +| ------------------------------- | -------------------------------------- | +| `exstruct_extract` | Extracts data from a workbook. | +| `exstruct_capture_sheet_images` | Captures sheet images. | +| `exstruct_make` | Creates a new workbook. | +| `exstruct_patch` | Applies editing patches to a workbook. | +| `exstruct_read_json_chunk` | Reads extracted JSON chunks. | +| `exstruct_read_range` | Reads cells from a specified range. | +| `exstruct_read_cells` | Reads data cell by cell. | +| `exstruct_read_formulas` | Reads cell formulas. | +| `exstruct_validate_input` | Validates input data. | + +For more details and API usage, see the documentation site: +[MCP Server](https://harumiweb.github.io/exstruct/mcp/) + +## Quick Start Python Extraction ```python from pathlib import Path @@ -261,7 +273,7 @@ engine_auto.export(wb_auto, Path("out_with_auto.json")) export_auto_page_breaks(wb_auto, "auto_areas", fmt="json", pretty=True) ``` -**Note (non-COM environments):** even when Excel COM is unavailable, cells + `table_candidates` are still returned, and `.xlsx` / `.xlsm` keep best-effort OOXML `shapes` / `charts` when available. +**Note (non-COM environments):** even when Excel COM is unavailable, cells + `table_candidates` are still returned, and `.xlsx` / `.xlsm` keep best-effort OOXML `shapes` / `charts` when available. ## Table Detection Parameters @@ -276,18 +288,18 @@ set_table_detection_params( ) ``` -Higher values reduce false positives. Lower values reduce missed detections. - -## Output Modes - -- **light**: direct OOXML cells + table candidates + print areas and best-effort shapes/connectors/charts for `.xlsx` / `.xlsm`. The normal path does not import openpyxl, pandas, SciPy, or xlwings. -- **standard**: texted shapes + arrows, charts (when COM is available), and table candidates. Cell hyperlinks are emitted only when `include_cell_links=True`. -- **verbose**: all shapes, charts, `table_candidates`, hyperlinks, and `colors_map`. - -## Error Handling / Fallback - -- If Excel COM is unavailable, extraction falls back to cells + table candidates automatically; `.xlsx` / `.xlsm` still preserve best-effort OOXML shapes/charts when available. -- A per-sheet drawing failure omits that sheet's affected drawing artifacts while preserving healthy sheets. An unsupported OOXML construct or failure in any other direct pipeline stage discards all partial OOXML data, restarts the complete openpyxl compatibility pipeline, and logs `ooxml_compatibility`. +Higher values reduce false positives. Lower values reduce missed detections. + +## Output Modes + +- **light**: direct OOXML cells + table candidates + print areas and best-effort shapes/connectors/charts for `.xlsx` / `.xlsm`. The normal path does not import openpyxl, pandas, SciPy, or xlwings. +- **standard**: texted shapes + arrows, charts (when COM is available), and table candidates. Cell hyperlinks are emitted only when `include_cell_links=True`. +- **verbose**: all shapes, charts, `table_candidates`, hyperlinks, and `colors_map`. + +## Error Handling / Fallback + +- If Excel COM is unavailable, extraction falls back to cells + table candidates automatically; `.xlsx` / `.xlsm` still preserve best-effort OOXML shapes/charts when available. +- A per-sheet drawing failure omits that sheet's affected drawing artifacts while preserving healthy sheets. An unsupported OOXML construct or failure in any other direct pipeline stage discards all partial OOXML data, restarts the complete openpyxl compatibility pipeline, and logs `ooxml_compatibility`. - The CLI writes errors to stdout/stderr and exits with a non-zero status on failure. ## Optional Rendering @@ -305,13 +317,13 @@ This writes `.pdf` and PNG files under `_images/`. To show how far exstruct can structure Excel, we parse an Excel workbook that combines the following three elements on a single sheet and show an LLM reasoning example based on the JSON output. - a table (sales data) -- a line chart -- a flowchart built only with shapes - -The image below is the actual sample Excel sheet. -![Sample Excel](docs/assets/demo_sheet.png) - -Sample Excel: `sample/sample.xlsx` +- a line chart +- a flowchart built only with shapes + +The image below is the actual sample Excel sheet. +![Sample Excel](docs/assets/demo_sheet.png) + +Sample Excel: `sample/sample.xlsx` ### 1. Input: Excel Sheet Overview @@ -424,10 +436,10 @@ Below is a shortened JSON output example from parsing the workbook above. } ``` -### 3. How AI (Copilot / LLM) interprets the JSON - -````md -Below is a Markdown reconstruction of the Excel workbook. It expresses the table, chart, and flowchart as separate structures. +### 3. How AI (Copilot / LLM) interprets the JSON + +````md +Below is a Markdown reconstruction of the Excel workbook. It expresses the table, chart, and flowchart as separate structures. --- @@ -485,11 +497,11 @@ flowchart TD ``` ```` -## Example 2: A General Application Form - -### Excel data - -![General application form Excel](docs/assets/demo_form.ja.png) +## Example 2: A General Application Form + +### Excel data + +![General application form Excel](docs/assets/demo_form.ja.png) ### ExStruct JSON @@ -524,11 +536,11 @@ flowchart TD } ``` -### ExStruct JSON -> Markdown via LLM reasoning - -```md -# Long-Term Care Insurance Burden Limit Certification Application - +### ExStruct JSON -> Markdown via LLM reasoning + +```md +# Long-Term Care Insurance Burden Limit Certification Application + (Submitted to) Year Month Day Mayor of City X @@ -562,80 +574,80 @@ If the applicant has not entered a care insurance facility, or uses short stay o --- -## Presence of a Spouse - -| Item | Value | -| ---- | ----- | -| Spouse | Yes / No | - -If "No", the following spouse section is not required. - ---- - -## Spouse Information - -| Item | Value | -| ---- | ----- | -| Furigana | | -| Name | | -| Date of Birth | Meiji / Taisho / Showa Year Month Day | -| Personal Number | | -| Address | Postal code | -| Contact | | -| Address as of January 1 of this year (if different) | Postal code | -| Tax status | Municipal resident tax: taxable / non-taxable | - ---- - -## Declaration of Income and Other Status - -Check the applicable item below. - -- □ 1. Livelihood protection recipient -- □ 2. Old-age welfare pension recipient in a household exempt from municipal resident tax -- □ 3. Person exempt from municipal resident tax whose taxable pension income + survivor/disability pension + other income totals **800,000 JPY or less per year** -- □ 4. Same as above, but **over 800,000 JPY up to 1,200,000 JPY** -- □ 5. Same as above, but **over 1,200,000 JPY** - -Survivor pension includes widow's pension, widower's pension, mother's pension, quasi-mother's pension, and orphan's pension. - ---- - -## Declaration of Deposits and Other Assets - -- □ The total amount of deposits, securities, and other assets is below the following threshold: - - Category 2: 10 million JPY (20 million JPY for couples) - - Category 3: 6.5 million JPY (16.5 million JPY for couples) - - Category 4: 5.5 million JPY (15.5 million JPY for couples) - - Category 5: 5 million JPY (15 million JPY for couples) - - Second insured persons (ages 40-64): Categories 3-5 must be 10 million JPY or less (20 million JPY for couples) - -### Asset breakdown - -| Item | Amount | -| ---- | ------ | -| Deposits | JPY | -| Securities (estimated value) | JPY | -| Other (including cash / debt) | JPY (describe) | - ---- - -## Applicant Information (not required when the applicant is the insured person) - -| Item | Value | -| ---- | ----- | -| Applicant name | | -| Contact (home / office) | | -| Applicant address | | -| Relationship to insured person | | - ---- - -## Notes - -1. In this application, "spouse" includes a spouse living separately and a common-law partner. -2. If you own multiple assets of the same kind, list all of them and attach copies of bankbooks or equivalent documents. -3. If there is not enough space, write on the margin or on a separate sheet and attach it. +## Presence of a Spouse + +| Item | Value | +| ---- | ----- | +| Spouse | Yes / No | + +If "No", the following spouse section is not required. + +--- + +## Spouse Information + +| Item | Value | +| ---- | ----- | +| Furigana | | +| Name | | +| Date of Birth | Meiji / Taisho / Showa Year Month Day | +| Personal Number | | +| Address | Postal code | +| Contact | | +| Address as of January 1 of this year (if different) | Postal code | +| Tax status | Municipal resident tax: taxable / non-taxable | + +--- + +## Declaration of Income and Other Status + +Check the applicable item below. + +- □ 1. Livelihood protection recipient +- □ 2. Old-age welfare pension recipient in a household exempt from municipal resident tax +- □ 3. Person exempt from municipal resident tax whose taxable pension income + survivor/disability pension + other income totals **800,000 JPY or less per year** +- □ 4. Same as above, but **over 800,000 JPY up to 1,200,000 JPY** +- □ 5. Same as above, but **over 1,200,000 JPY** + +Survivor pension includes widow's pension, widower's pension, mother's pension, quasi-mother's pension, and orphan's pension. + +--- + +## Declaration of Deposits and Other Assets + +- □ The total amount of deposits, securities, and other assets is below the following threshold: + - Category 2: 10 million JPY (20 million JPY for couples) + - Category 3: 6.5 million JPY (16.5 million JPY for couples) + - Category 4: 5.5 million JPY (15.5 million JPY for couples) + - Category 5: 5 million JPY (15 million JPY for couples) + - Second insured persons (ages 40-64): Categories 3-5 must be 10 million JPY or less (20 million JPY for couples) + +### Asset breakdown + +| Item | Amount | +| ---- | ------ | +| Deposits | JPY | +| Securities (estimated value) | JPY | +| Other (including cash / debt) | JPY (describe) | + +--- + +## Applicant Information (not required when the applicant is the insured person) + +| Item | Value | +| ---- | ----- | +| Applicant name | | +| Contact (home / office) | | +| Applicant address | | +| Relationship to insured person | | + +--- + +## Notes + +1. In this application, "spouse" includes a spouse living separately and a common-law partner. +2. If you own multiple assets of the same kind, list all of them and attach copies of bankbooks or equivalent documents. +3. If there is not enough space, write on the margin or on a separate sheet and attach it. 4. If benefits are obtained through a false declaration, the paid amount and up to twice that amount as an additional charge may need to be repaid under Article 22, Paragraph 1 of the Long-Term Care Insurance Act. ``` @@ -662,9 +674,9 @@ This benchmark demonstrates that the library can: In short, **exstruct = "an engine that converts Excel into a format AI can understand."** -## Benchmark - -![Benchmark Chart](benchmark/public/plots/markdown_quality.png) +## Benchmark + +![Benchmark Chart](benchmark/public/plots/markdown_quality.png) This repository includes benchmark reports focused on RAG/LLM preprocessing of Excel documents. We track two perspectives: (1) core extraction accuracy and (2) reconstruction utility for downstream structure queries (RUB). @@ -701,13 +713,13 @@ It is suitable for teams that: ExStruct adopts a pipeline-oriented architecture that separates extraction strategy (Backend), orchestration (Pipeline), and semantic modeling. -See: [dev-docs/architecture/pipeline.md](dev-docs/architecture/pipeline.md) +See: [dev-docs/architecture/pipeline.md](dev-docs/architecture/pipeline.md) ## Contributing If you plan to extend ExStruct internals, read the contributor architecture guide first. -See: [dev-docs/architecture/contributor-guide.md](dev-docs/architecture/contributor-guide.md) +See: [dev-docs/architecture/contributor-guide.md](dev-docs/architecture/contributor-guide.md) ## Coverage Note