Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions .github/EXAMPLE_README_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,10 @@ Why this exists and when an enterprise integrator would use it.
## Prerequisites

- Tooling/runtime versions
- The example resolves the XSD itself (env `FUNDSXML_SCHEMA_DIR` →
`.schema-cache/` → official-release download); or
`python -m fundsxml_schema <version>` to pre-cache for a bare xmllint
- Network/proxy note when the official schema URL must be reached
(`$FUNDSXML_SCHEMA_DIR` is the offline escape hatch)
- XSD validation takes `<schema> <xml-file>`: pass the official release URL
or a local `FundsXML.xsd` path (no version arg, no cache, no env var)
- Network note: only needed when a schema **URL** is passed; a local
`FundsXML.xsd` path validates fully offline

## Run

Expand Down
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,5 @@ OS; and the relevant runtime version: Python / Java (`java -version`) /
Node (`node -v`) / .NET (`dotnet --version`) / `xmllint --version`.

**Validation already done** (helps a lot)
- [ ] `xmllint --noout --schema .schema-cache/<ver>/FundsXML.xsd <file>` result: …
- [ ] `XSD_Validation/cli/validate.sh <schema-url-or-path> <file>` result: …
- [ ] round-trip checked with `Database_Integration/tools/xml_equiv.py` (if applicable): …
4 changes: 2 additions & 2 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@
- [ ] English only; example is **self-contained** and **heavily commented**
(file-header: purpose / run / deps / FundsXML assumptions + what & why).
- [ ] FundsXML conventions: no XML namespace; XSD-validated against the
**official released schema** (each example resolves it itself; or
`python -m fundsxml_schema <ver>`); 4.0.0 has no `ControlData/Version`;
**official released schema** (validators take `<schema> <xml-file>` —
release URL or local path); 4.0.0 has no `ControlData/Version`;
secure XML parsing (DTD/external entities off).
- [ ] New/changed samples are **XSD-valid**; negative fixtures still fail.
- [ ] Round-trip examples: proven with `Database_Integration/tools/xml_equiv.py`
Expand Down
146 changes: 85 additions & 61 deletions .github/workflows/ci.yml

Large diffs are not rendered by default.

5 changes: 0 additions & 5 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,6 @@ CLAUDE.md
.DS_Store
._*

# Locally cached XSD releases — not committed, the source of truth is the
# official GitHub release. Each example downloads + caches it here itself
# (honouring $FUNDSXML_SCHEMA_DIR); see XSD_Validation/java/XsdValidate.java.
.schema-cache/

# Maven build output. Dependencies come from Maven Central via ./mvnw; the
# committed Maven Wrapper bootstraps Maven itself — nothing is vendored.
target/
Expand Down
26 changes: 13 additions & 13 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,11 @@ Please read this before opening a pull request.
the schema itself requires them). `xsi:noNamespaceSchemaLocation` must be in
the XMLSchema-instance namespace or validators reject it.
- **Validate against the official released schema**, never a hand-made
catalog. Every example resolves it itself (`$FUNDSXML_SCHEMA_DIR` →
`.schema-cache/` → official-release download, 302-aware, pulls the
`xmldsig-core-schema.xsd` sibling for 4.2.9+). For an xmllint check,
materialise the cache with `python -m fundsxml_schema <version>`
(after `pip install -e .`). Set sample `xsi:noNamespaceSchemaLocation`
to that release URL.
catalog. The validators take `<schema> <xml-file>` — pass the official
release URL (the URL stacks fetch it + the `xmldsig-core-schema.xsd`
sibling for 4.2.9+) or a local `FundsXML.xsd` path. No version arg, no
cache, no env var. Set sample `xsi:noNamespaceSchemaLocation` to that
release URL.
- **4.0.0 `ControlData` has no `<Version>` element** (added in 4.1.0) — never
add one to a 4.0.0 sample.
- Positions ↔ Assets link by a shared `UniqueID`; `AssetMasterData` is
Expand Down Expand Up @@ -60,18 +59,19 @@ Run what you changed and confirm it actually works — no "should pass" claims.

Java examples build standalone via the committed Maven Wrapper (`./mvnw`, or
`mvnw.cmd` on Windows). Python examples install once into a venv from
`pyproject.toml` and resolve the XSD themselves. No `fetch-tools.sh` and no
`fetch-schema.sh` — both are gone; every stack is standalone & cross-platform.
`pyproject.toml`. No `fetch-tools.sh`, no `fetch-schema.sh`, no
`fundsxml_schema` resolver — all gone; every stack is standalone &
cross-platform and takes the schema as an argument.

```bash
REL=https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd

# Python stack (cross-platform; Windows: .venv\Scripts\activate)
python -m venv .venv && . .venv/bin/activate && pip install -e .
python XSD_Validation/python/validate.py 4.2.9 <your-sample>.xml # self-resolves the XSD
python XSD_Validation/python/validate.py "$REL" <your-sample>.xml

# xmllint check: materialise the cache cross-platform, then validate
python -m fundsxml_schema 4.2.9
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd <your-sample>.xml
# (or just: XSD_Validation/cli/validate.sh 4.2.9 <your-sample>.xml — self-resolving)
# or the CLI (same args); a local FundsXML.xsd path works in place of $REL:
XSD_Validation/cli/validate.sh "$REL" <your-sample>.xml

# Schematron via the Maven Wrapper (positive sample -> exit 0)
./mvnw -q -pl Schematron_DataQuality_Checks/Basic_Checks/invocation \
Expand Down
28 changes: 19 additions & 9 deletions Data_Binding_JSON/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ python3 Data_Binding_JSON/python/fundsxml_json.py to-json \
FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml fund.json
python3 Data_Binding_JSON/python/fundsxml_json.py roundtrip \
FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml regenerated.xml
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd regenerated.xml
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
regenerated.xml # or a local FundsXML.xsd path
```

Verified: NAV, position count and percentage-sum preserved through
Expand All @@ -35,15 +37,23 @@ so a full generated model is heavy and brittle to maintain.

### Generated-binding references (when you do want codegen)

| Stack | Tool | Command (against the fetched schema) |
|-------|------|--------------------------------------|
| Java | JAXB `xjc` | `xjc -d src -p org.fundsxml.model .schema-cache/4.2.9/FundsXML.xsd` |
| Python | `xsdata` | `xsdata --package fundsxml.model .schema-cache/4.2.9/FundsXML.xsd` |
| .NET | `xsd.exe` / `XmlSerializer` | `xsd.exe /classes /namespace:FundsXml.Model .schema-cache\4.2.9\FundsXML.xsd` |
| Stack | Tool | Command (against a local FundsXML.xsd) |
|-------|------|----------------------------------------|
| Java | JAXB `xjc` | `xjc -d src -p org.fundsxml.model schema/FundsXML.xsd` |
| Python | `xsdata` | `xsdata --package fundsxml.model schema/FundsXML.xsd` |
| .NET | `xsd.exe` / `XmlSerializer` | `xsd.exe /classes /namespace:FundsXml.Model schema\FundsXML.xsd` |

All three consume the **official released schema**; materialise it with
`python -m fundsxml_schema 4.2.9` (after `pip install -e .` — cross-platform;
also pulls the imported `xmldsig-core-schema.xsd` for 4.2.9). Trade-off: generated models are type-safe but regenerate on every
All three consume the **official released schema** on disk. Codegen needs the
local file (and, for 4.2.9, its `xmldsig-core-schema.xsd` sibling beside it),
so fetch it once:

```bash
mkdir -p schema && B=https://github.com/fundsxml/schema/releases/download/4.2.9
curl -sSL -o schema/FundsXML.xsd "$B/FundsXML.xsd"
curl -sSL -o schema/xmldsig-core-schema.xsd "$B/xmldsig-core-schema.xsd"
```

Trade-off: generated models are type-safe but regenerate on every
schema bump and produce thousands of classes; the native binding stays small
and version-tolerant. Pick per use case.

Expand Down
5 changes: 3 additions & 2 deletions Database_Integration/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,6 @@ Each example is run as **import, then export** (the round-trip = both, then
compare). `DOC` is the document id the import prints.

```bash
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
FX=FundsXML_Files/4.2.9/positions/Multi-Fund_Positions.xml
DOC=FUNDSXML_MULTI_1

Expand All @@ -84,7 +83,9 @@ dotnet run --project Database_Integration/csharp/export -- fx.db "$DOC" out.xml

# prove it: exported file == input file, and schema-valid
python3 Database_Integration/tools/xml_equiv.py "$FX" out.xml
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd out.xml
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
out.xml # or a local FundsXML.xsd path
```

(`--enable-native-access=ALL-UNNAMED` only silences a JDK 24+ warning when
Expand Down
2 changes: 1 addition & 1 deletion Database_Integration/python/export_fundsxml.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
#
# Prove the round-trip (import file vs exported file):
# python3 ../tools/xml_equiv.py some.xml out.xml
# xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd out.xml
# XSD_Validation/cli/validate.sh <FundsXML.xsd path or release URL> out.xml
#
# DEPENDENCIES Python stdlib `sqlite3` + `lxml`.
#
Expand Down
6 changes: 4 additions & 2 deletions FundsXML_Files/4.0.0/positions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,9 @@ Content-identical to the 4.1.0 example (3 equity positions), but adapted to the
## Validation

```bash
python -m fundsxml_schema 4.0.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.0.0/FundsXML.xsd \
# Give the validator the schema (the official 4.0.0 release URL — or a local
# FundsXML.xsd path) + the XML file:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.0.0/FundsXML.xsd \
FundsXML_Files/4.0.0/positions/Equity-Fund_Positions.xml
```
6 changes: 4 additions & 2 deletions FundsXML_Files/4.1.0/positions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ EUR 40m NAV. Deliberately small to make the version comparison easy.
## Validation

```bash
python -m fundsxml_schema 4.1.0 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.1.0/FundsXML.xsd \
# Give the validator the schema (the official 4.1.0 release URL — or a local
# FundsXML.xsd path) + the XML file:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.1.0/FundsXML.xsd \
FundsXML_Files/4.1.0/positions/Equity-Fund_Positions.xml
```
6 changes: 4 additions & 2 deletions FundsXML_Files/4.2.9/documents/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ via `Document/Fund/Identifiers/LEI`.
## Validation

```bash
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
# Give the validator the schema (the official 4.2.9 release URL — or a local
# FundsXML.xsd path) + the XML file; the xmldsig sibling is handled for you:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/documents/Fund_Documents.xml
```
6 changes: 4 additions & 2 deletions FundsXML_Files/4.2.9/regulatory/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ Mandatory blocks included:
## Validation

```bash
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
# Give the validator the schema (the official 4.2.9 release URL — or a local
# FundsXML.xsd path) + the XML file; the xmldsig sibling is handled for you:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/regulatory/EFT_Regulatory.xml
```
11 changes: 7 additions & 4 deletions FundsXML_Files/4.2.9/signed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,9 @@

`ds:Signature` (namespace `http://www.w3.org/2000/09/xmldsig#`) is the **last
optional child** of `<FundsXML4>`. From release 4.2.9 on, `FundsXML.xsd` imports
`xmldsig-core-schema.xsd` for this — the schema resolvers fetch that sibling
automatically when it is imported (e.g. `python -m fundsxml_schema 4.2.9`).
`xmldsig-core-schema.xsd` for this — the validators fetch that sibling
alongside `FundsXML.xsd` when you pass the release URL (and it sits next to a
local `FundsXML.xsd` in any complete copy of the release).

> ⚠️ **Placeholder:** `DigestValue` and `SignatureValue` are schema-valid base64
> strings but **not cryptographically verifiable**. Real signing and
Expand All @@ -26,7 +27,9 @@ Algorithms used (enveloped signature): C14N 2001-03-15, RSA-SHA256, SHA-256.
## Validation

```bash
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
# Give the validator the schema (the official 4.2.9 release URL — or a local
# FundsXML.xsd path) + the XML file; the xmldsig sibling is handled for you:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/signed/Signed_Fund_Skeleton.xml
```
6 changes: 4 additions & 2 deletions FundsXML_Files/4.2.9/transactions/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ records:
## Validation

```bash
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
# Give the validator the schema (the official 4.2.9 release URL — or a local
# FundsXML.xsd path) + the XML file; the xmldsig sibling is handled for you:
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/transactions/Fund_Transactions.xml
```

Expand Down
38 changes: 19 additions & 19 deletions FundsXML_Files/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,36 +114,36 @@ FundsXML uses standardized codes for asset classification:

FundsXML documents should be validated against the official XSD schema:

### Download Schema
### The schema

Validation always targets the **official release** of the schema:
The canonical schema is the **official release**:

```
https://github.com/fundsxml/schema/releases/download/<version>/FundsXML.xsd
```

Two enterprise-relevant caveats are handled by every example's in-language
schema resolver (and by the `fundsxml_schema` module shown below):
You hand that URL (or a local `FundsXML.xsd` path) straight to a validator —
nothing is resolved by version. Two enterprise-relevant caveats the validators
handle for you:

1. That URL returns an HTTP 302 redirect; simple HTTP clients (libxml2 /
xmllint) do not follow it, so the schema must be materialised first.
2. From release 4.2.9 on, `FundsXML.xsd` imports `xmldsig-core-schema.xsd` via a
relative path — both files must sit in the same directory.
1. That URL returns an HTTP 302 redirect; the validators that rely on a simple
HTTP client (Python, the xmllint CLI, Java) fetch the schema into a temp
dir first, then validate locally.
2. From release 4.2.9 on, `FundsXML.xsd` imports `xmldsig-core-schema.xsd` via
a relative path — the URL stacks fetch that sibling alongside it; for a
local schema path it must sit in the same directory (it does in any
complete copy of a release).

The resolution order is the same everywhere: `$FUNDSXML_SCHEMA_DIR` (offline /
corporate-network escape hatch) → `.schema-cache/` → download from the
official release. No committed catalog.
### Validate (any stack — schema + xml)

```bash
# Fetches FundsXML.xsd (+ xmldsig-core-schema.xsd when needed) into .schema-cache/<version>/
python -m fundsxml_schema 4.2.9 # caches the XSD into .schema-cache/ (run `pip install -e .` once; cross-platform)
```

### Validate with xmllint (macOS/Linux)

```bash
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd \
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
FundsXML_Files/4.2.9/positions/Mixed-Fund_Positions.xml
# offline: pass a local FundsXML.xsd path instead of the URL. Bare xmllint
# needs the schema on disk (it can't follow the GitHub 302 itself):
# curl -sSL -o /tmp/FundsXML.xsd "<release-url>" # + xmldsig sibling for 4.2.9+
# xmllint --noout --schema /tmp/FundsXML.xsd <file>
```

### Validate with Saxon
Expand Down
4 changes: 3 additions & 1 deletion Large_File_Processing/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,9 @@ MAVEN_OPTS=-Xmx64m ./mvnw -q -pl Large_File_Processing/java compile exec:java \

# 3. split into XSD-valid chunks of 10k positions
python3 Large_File_Processing/python/split.py big.xml chunks/ 10000
xmllint --noout --schema .schema-cache/4.2.9/FundsXML.xsd chunks/chunk-0001.xml
XSD_Validation/cli/validate.sh \
https://github.com/fundsxml/schema/releases/download/4.2.9/FundsXML.xsd \
chunks/chunk-0001.xml # or a local FundsXML.xsd path

# 4. day-over-day position delta (exit 1 if anything changed)
python3 Large_File_Processing/python/delta_diff.py yesterday.xml today.xml
Expand Down
Loading
Loading