Translate the new-format network tables to buses and links by nick-gorman · Pull Request #126 · Open-ISP/ISPyPSA

nick-gorman · 2026-06-24T02:34:45Z

This PR does two things: it adds the translator that turns the new-format network tables into PyPSA buses and links, and it uses those tables to pilot a "wildcard" convention for sparse inputs — implemented end-to-end so there is a concrete example to react to before deciding whether to adopt it more broadly.

src/ispypsa/
├── translator/
│   ├── network.py            → network tables → buses + links (+ link_timeslice_limits)
│   └── helpers.py            → _resolve_wildcards, the generic wildcard resolver
└── validation/schemas/
    ├── network_transmission_path_limits.yaml           → wildcard keys, nan_fill, coverage rules
    ├── network_expansion_options.yaml                  → wildcard keys, pairing rule
    └── network_transmission_path_expansion_costs.yaml  → wildcard keys, nan_fill, coverage rules
tests/test_translator/
├── test_network.py            → buses/links coverage incl. wildcard and guard cases
└── test_translator_helpers.py → _resolve_wildcards unit coverage

The translation

network.py consumes network_geography, network_transmission_paths, network_transmission_path_limits, network_expansion_options and network_transmission_path_expansion_costs, and produces:

one bus per modelled geography (REZ buses only when REZs are modelled as discrete nodes),
one existing link per transmission path,
expansion links per investment period, from the unified options/costs tables.

Flow paths and REZ connections run through one pipeline — both are just paths.

A link's p_nom is the larger of its forward and reverse capacities; every directional, per-demand-condition limit is expressed per unit of that in a link_timeslice_limits table (forward limits as p_max_pu, reverse as p_min_pu), which pypsa_build later expands into per-snapshot series:

network_transmission_path_limits:
    path_id  direction  timeslice             capacity
    CQ-NQ    forward    qld_peak_demand       1200
    CQ-NQ    forward    qld_winter_reference  1400
    CQ-NQ    reverse    qld_peak_demand       1440
    CQ-NQ    reverse    qld_winter_reference  1910

links (abridged):
    name            p_nom  p_min_pu  p_max_pu
    CQ-NQ_existing  1910   0.0       1.0       # p_nom = max over both directions

link_timeslice_limits:
    name            attribute  timeslice             value
    CQ-NQ_existing  p_max_pu   qld_peak_demand        0.628   # 1200/1910
    CQ-NQ_existing  p_max_pu   qld_winter_reference   0.733
    CQ-NQ_existing  p_min_pu   qld_peak_demand       -0.754   # -1440/1910
    CQ-NQ_existing  p_min_pu   qld_winter_reference  -1.0

Design choices worth knowing before reading the diff:

p_nom as the max of both directions keeps every per-unit limit in [-1, 1], so no timeslice can credit a link above its physical rating.
The static p_min_pu/p_max_pu on each link are inert defaults (0.0/1.0); the real limits live entirely in link_timeslice_limits under a coverage contract (Translator timeslice implementation notes (decisions from staging PR #121) #123) — every snapshot is covered by a named-timeslice row or a blank-timeslice fallback.
Expansion links start at p_nom 0 and are individually unbounded; the expansion-limit custom constraints (ispypsa.translator.constraints) cap the total built across a path's links. Asymmetric options set p_max_pu/p_min_pu in the option's forward/reverse proportion.

The wildcard proposal (feedback wanted)

The three sparse tables (limits, expansion options, expansion costs) accept blank key cells as wildcards, resolved most-specific-wins, with blank value cells taking a schema-declared nan_fill:

network_transmission_path_limits (blank cells are wildcards):
    path_id  direction  timeslice  capacity
    CQ-NQ    forward               1500
    CQ-NQ    reverse               1000
                                   800       # table-wide default

resolved against modelled paths {CQ-NQ, Q1-NQ}:
    path_id  direction  timeslice  capacity
    CQ-NQ    forward               1500      # its own rows win (fewest wildcards)
    CQ-NQ    reverse               1000
    Q1-NQ    forward               800       # inherits the table-wide default
    Q1-NQ    reverse               800

_resolve_wildcards in translator/helpers.py is the generic resolver; the three schema files document the convention (wildcard keys, *_resolve_unambiguously and coverage rules, nan_fill). The validation rules are declarative-only for now — two runtime guards in network.py stand in for the option-pairing rule until schema validation lands.

This is deliberately a reference implementation. The same convention could apply to most sparse ISPyPSA tables (build costs, fuel prices, generator properties); the question for review is whether it earns its keep as a uniform way to express defaults and sparse overrides, or whether it's over-engineering that will be hard to apply consistently. Either way, concrete code seemed a better basis for that call than an abstract proposal.

Scope note

The translation code is dark — nothing calls it until the orchestrator PR, so there's no change to production behaviour. This PR is stacked on #129 (the transmission_default_limit rename) and targets that branch as its base; once #129 merges it retargets to main.

🤖 Generated with Claude Code

network_geography becomes buses (REZ buses dropped when rezs is attached_to_parent_node), and the path/limit/expansion tables become links. The winter_reference limit is the link's static p_nom — winter is the calendar's default season — with the other timeslices' forward and reverse limits emitted as per-unit values in a link_timeslice_limits table for pypsa_build to expand into p_max_pu/p_min_pu series (per-unit values can exceed 1.0 when a season's limit tops winter's). Asymmetric reverse limits land in p_min_pu rather than separate links. Expansion options become one extendable link per (path, investment period) with annuitised capital costs, gated by transmission_expansion / rez_transmission_expansion according to whether the path connects a REZ. Paths with no capacity data take rez_to_sub_region_transmission_default_limit, replicating the existing REZ behaviour. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

codecov · 2026-06-24T02:38:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines	Coverage Δ
src/ispypsa/translator/helpers.py	`100.00% <100.00%> (ø)`
src/ispypsa/translator/network.py	`100.00% <100.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Pilots the blank-key-as-wildcard convention on the three sparse network tables (limits, expansion options, expansion costs) as a reference implementation for wider adoption: _resolve_wildcards in translator/helpers.py does the resolution until schema validation lands, with runtime guards in network.py standing in for the option-pairing rule. Drops that are designed selection (constraint_relaxation routing, non-investment-period cost years) are no longer logged as data loss. Links now take p_nom as the larger of their two directional capacities so every per-unit limit sits in [-1, 1], and all real limits (including blank-timeslice fallbacks) move into link_timeslice_limits under the coverage contract (#123); the static link attributes are inert defaults. rez_to_sub_region_transmission_default_limit becomes transmission_default_limit -- the default now applies to any path without limit data, not just REZ connections. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The options/costs pair now has all its empty/populated combinations pinned -- including options-without-costs, which the schema declares invalid but which silently builds no expansion links until schema validation lands -- plus the no-expansion-study config (both flags off), which resolves wildcards against an empty allowed set. The two cost-wildcard behaviours documented in _prepare_expansion_costs but previously untested are exercised: a concrete-year row overriding a blank-year static row (also the one spot where wildcard-expanded int years concat with CSV-parsed float years before the int cast), and a blank expansion_id acting as a table-wide default cost. The limits drop log gets its firing case (the negative already existed), and the calendar-year NotImplementedError closes the last uncovered line in network.py. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…rk-to-buses-links

The name was a leftover from the winter-reference design, where the frame really did hold the link-level static limit; it now holds the per-direction maximum capacities that _extract_max_capacities produces, so the old name misdescribed it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ndings _resolve_wildcards previously dropped out-of-set values itself, logging or staying silent per an expected_drops escape hatch, which conflated designed config-driven selection with bad input data. Callers now filter their designed selections out first (logged at INFO) and the resolver raises on anything left outside the allowed set — for limits a typo'd path_id halts the run, while an options/costs typo is indistinguishable from a constraint group or disabled element, so the log line is its only trace. Also acts on the PR #126 review: the forward/reverse pairing guard counts a blank expansion_option as its own label (nunique dropna=False) so a blank/named mismatch raises; the expansion_option schema description no longer claims the column is informational-only; p_nom collapses to a single groupby max; annuitisation is vectorised; and name-list test assertions are replaced with full-frame comparisons. Expansion cost years are opaque labels matched against the config's investment periods — year_type only governs how time is chunked — so the calendar-year NotImplementedError guard is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

EllieKallmier

Solid! Have the usual few questions mostly to confirm that I'm understanding flows and the wildcarding process right - I don't think anything blocking from me :)

EllieKallmier · 2026-07-08T05:50:40Z

+    every value of that column. ``allowed_values`` lists, for each wildcardable
+    column, the concrete values it may take — the schema's allowed_values /
+    allowed_values_from, resolved to actual values. Each blank cell in those
+    columns is fanned out to every allowed value; a filled cell must itself be
+    an allowed value — anything else raises. Callers filter their designed
+    selections (e.g. costs for disabled elements or non-investment-period
+    years) out before calling, so an out-of-set value reaching this point is
+    bad input data, not selection. Key columns absent from ``allowed_values``


Honestly I get what this function is doing but this docstring is confusing to me! Definitely a personal preference thing I think but tbh I'd find a more basic set of input descriptions clearer in this case.

(The module docstring in network.py does make this clearer - but as it's a helper I kinda think shouldn't need that?)

Yep fair comment.

EllieKallmier · 2026-07-08T06:31:44Z

+    """Resolves the sparse limits table to one row per modelled path, direction
+    and timeslice, then fills empty capacities with the system default.
+
+    Blank path_id, direction or capacity cells are wildcards (see the


Are 'capacity' cells wildcards (terminology)? Just clarifying for myself - I've been thinking of the 'value' columns as kinda different entities from the wildcardable columns. I think prob another case where I'm getting a touch tangled in the docstring just in how some of the terminology gets used - no issue, just my comprehension in the afternoon I reckon

Your right, calling the blank capacity value a wilde card is really wrong. I'll change the wording.

EllieKallmier · 2026-07-08T07:03:40Z

+    wildcard resolution rather than inside it. Rows for disabled or non-modelled
+    elements drop out here, as do rows for constraint groups (their expansion_ids
+    are constraint_ids, routed to ispypsa.translator.constraints instead). The
+    filtered ids are logged at INFO — a typo'd expansion_id is indistinguishable


I think if we assume that inputs have been through validation (including any cross-table validation to cover allowed values/names that are expected to be consistent across inputs?) this is fine and (I think) typos would have been caught and flagged at that point?

Yeah, agreed, typos should be gone, here so maybe the logging isn't needed, but I think fine to leave it just in case for now.

EllieKallmier · 2026-07-08T07:11:32Z

+    could pair a forward from one option with a reverse from another; this guard
+    catches that. A blank expansion_option counts as its own label (nunique with


this guard catches that after the fact, with the idea being that validation should catch this before the fact right?

Yep, maybe not needed but just threw it in for now.

EllieKallmier · 2026-07-09T00:51:04Z

    required: false
    units: MW
    gte: 0.0
+    nan_fill: {config: network.transmission_default_limit}


sweet I like this

EllieKallmier · 2026-07-09T01:35:30Z

+    ispypsa_tables["network_transmission_path_limits"] = csv_str_to_df("""
+        path_id,  direction,  timeslice,             capacity
+        CQ-NQ,    forward,    qld_winter_reference,  0
+        CQ-NQ,    reverse,    qld_winter_reference,  1000


Piecing things together for my understanding: this input wouldn't normally be allowed (by validation) because not all timeslices are defined (and no empty/wildcard fallback either)... right? So then the link_timeslice_limits would be 'complete' (all timeslices accounted for) because the input is complete (or filled by wildcard). Is that a correct interpretation?

Yes, thats right in validation would fail on this input, its just kept shorter for testing. But I should add a note to the test, and if there are others like.

EllieKallmier · 2026-07-09T01:41:09Z

+            name            attribute  timeslice             value
+            CQ-NQ_existing  p_max_pu   qld_peak_demand       0.857   # 1200/1400
+            CQ-NQ_existing  p_max_pu   qld_winter_reference  1.0     # 1400/1400
+            CQ-NQ_existing  p_min_pu   ,                     -0.714  # fallback, -1000/1400


I'm interpretting this empty/nan timeslice in the returned table to imply wildcarding can continue as translator output/model input - is that what you're thinking too? (Or maybe I'm misunderstanding?)

EllieKallmier · 2026-07-09T01:45:15Z

+    """)
+    ispypsa_tables["network_transmission_path_limits"] = csv_str_to_df("""
+        path_id,  direction,  timeslice,  capacity
+        N1-CNSW,  ,           ,


(same just piecing together for myself here) - this 'capacity' value would be set to the default (from config) before being input to translator as a result of validation, so once validation is in place we wouldn't see any NaN/empty 'capacity' values in this table. But at current state makes sense as-is

EllieKallmier · 2026-07-09T02:50:39Z

+        path_id,  direction,  timeslice,  capacity
+        CQ-NQ,    forward,    ,           1500
+        CQ-NQ,    reverse,    ,           1000
+        ,         ,           ,           800


Vague question sitting in the back of my mind: is this blank-row-with-value kinda just a way to get two different default values for this field? (Applied slightly differently ofc). Just wondering about the utility or potential for confusion/double up if so. I guess the distinction in this case is that the config default is the value used for rows with some keys/no values, and this no-row 'capacity' is the value used for no keys/no values (path definitely does exist though). Is that right?

I don't have any problem with this btw - just I think something to be clear about with this wildcarding and the interactions between any various 'defaults'.

Co-authored-by: EllieKallmier <61219730+EllieKallmier@users.noreply.github.com>

for more information, see https://pre-commit.ci

nick-gorman mentioned this pull request Jun 24, 2026

Translator timeslice implementation notes (decisions from staging PR #121) #123

Open

nick-gorman and others added 2 commits July 2, 2026 09:57

nick-gorman mentioned this pull request Jul 2, 2026

Rename the transmission default limit config field #129

Merged

nick-gorman changed the base branch from main to rename-transmission-default-limit July 2, 2026 03:12

nick-gorman and others added 3 commits July 2, 2026 13:13

Merge branch 'rename-transmission-default-limit' into translate-netwo…

1f2db94

…rk-to-buses-links

nick-gorman requested a review from EllieKallmier July 3, 2026 05:21

EllieKallmier approved these changes Jul 9, 2026

View reviewed changes

nick-gorman changed the base branch from rename-transmission-default-limit to main July 14, 2026 01:35

nick-gorman and others added 2 commits July 14, 2026 11:36

Update src/ispypsa/translator/network.py

bd76163

Co-authored-by: EllieKallmier <61219730+EllieKallmier@users.noreply.github.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3e7bf2e

for more information, see https://pre-commit.ci

EllieKallmier mentioned this pull request Jul 23, 2026

Template new entrant regional granularity #134

Open

		could pair a forward from one option with a reverse from another; this guard
		catches that. A blank expansion_option counts as its own label (nunique with

Uh oh!

Conversation

nick-gorman commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The translation

The wildcard proposal (feedback wanted)

Scope note

Uh oh!

codecov Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

EllieKallmier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nick-gorman commented Jun 24, 2026 •

edited

Loading

codecov Bot commented Jun 24, 2026 •

edited

Loading