Implement 221100 split by jvendries · Pull Request #462 · cornerstone-data/bedrock

jvendries · 2026-06-09T02:42:30Z

What changed? Why?

Implements monetary disaggregation of the aggregate 221100 electricity sector into three sub-sectors — 221110 (generation), 221121 (transmission), and 221122 (distribution) — using gross-output shares derived from BEA UGO305-A data. This is the third major step in the electricity sector disaggregation pipeline, following the existing co-production reallocation work.

The disaggregation runs in four sequential steps:

Split the 221100 Make diagonal into a 3×3 diagonal block using GO weights
Split the 221100 Use self-use intersection diagonally
Split the 221100 industry column across sub-industries with fuel-commodity routing (generation-only fuels like coal, natural gas, petroleum go entirely to 221110) and VA balancing to preserve row totals
Split the 221100 commodity row across all consuming industries using equal-price GO weights

A new implement_electricity_disaggregation flag is added to USAConfig with a validator enforcing that both implement_waste_disaggregation and implement_electricity_reallocation must be enabled as prerequisites. When active, the schema expands from 405 to 407 sectors, and all Cornerstone matrix validations (V, U, A, B, X, Q) are routed through a new flag-aware validate_cornerstone() helper that replaces the previous static @pa.check_output decorators, allowing the same validation logic to work against either the 405- or 407-sector schema depending on config.

Emissions attribution (derive_E_usa) loads the eGRID-based FBS (GHG_national_Cornerstone_2023_egrid) when electricity disaggregation is enabled, with a fallback to the cached national FBS. SF6 emissions are routed to 221121 (transmission); all other gases go to 221110 (generation). Inflation helpers, the BEA summary-to-Cornerstone correspondence, and gross-output x distribution are all updated to propagate the parent 221100 values to the three sub-sectors.

Two new config files are added: a test config (test_usa_config_waste_disagg_electricity_disaggregation.yaml) and a full production config (2025_usa_cornerstone_full_model_electricity_disaggregation.yaml).

Testing

New integration tests in test_electricity_disaggregation.py cover:

GO weights summing to 1.0 across the three sub-sectors
A worked numerical example verifying column balancing, VA row preservation, and use-row totals after step 3
Full pipeline integration tests (marked eeio_integration) confirming the 407-sector schema shape, absence of 221100 from all output matrices, Aq column sums ≤ 1, and non-zero emissions attribution to the disaggregated electricity columns

jvendries · 2026-06-09T02:42:43Z

Implement unit conversion #492
Implement changes from resolutions from discussion #85 #490
implement analysis for decisions 3, 5, and 7 in dicussion #85 #481
Implement print to excel for PR3 #463
Implement 221100 split #462 👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

bl-young

Sharing some initial comments but given the complexity I will want to review more closely.

bl-young · 2026-06-09T20:03:19Z


    # Anything still NaN (truly no parent, e.g. S00402) gets neutral 1.0
    ratio = ratio.fillna(1.0)
+    if cfg.implement_electricity_disaggregation:


I'm curious how this is handled for waste right now. Because waste sectors face the same issue right? I would think that the same approach would suffice here potentially? Does the map right above with _cornerstone_to_ceda_v7_parent() attempt to handle that?

I'm not sure I fully understand the question.

If you are asking whether we can use the same mapping as for waste, I don't think so. The waste disagg is built into the cornerstone schema, whereas the electricity disagg is not. So we need additional mapping to expand the schema.

Does this answer the question?

bl-young · 2026-06-09T20:05:11Z

    A: pd.DataFrame, original_year: int, target_year: int
 ) -> pd.DataFrame:
    price_ratio = get_cornerstone_industry_price_ratio(original_year, target_year)
+    price_ratio = price_ratio.reindex(A.index, fill_value=1.0)


this feels like it should already be done within get_cornerstone_industry_price_ratio so I'm not sure it needs to be done again

Perhaps but this is the way it is structured in main. Should this be done in its own PR rather than here?

bl-young · 2026-06-09T20:06:53Z

    implement_waste_disaggregation: bool = False  # DRI: jorge.vendries
    eeio_waste_disaggregation: ta.Optional[EEIOWasteDisaggConfig] = None
    implement_electricity_reallocation: bool = False  # DRI: jorge.vendries
+    implement_electricity_disaggregation: bool = False  # DRI: jorge.vendries


This comment is relevant in a few places. Are we planning to evaluate the reallocation with and without disaggregation? I'm not sure what purpose that serves really. I would propose we just have a single flag and not over burden ourselves. (to be more clear, my suggestion would be to use implement_electricity_disaggregation and drop implement_electricity_reallocation

Yes I agree the ultimate goal is to drop reallocation flag, but I do think it is worthwhile to evaluate reallocation in isolation. In other words, my intent was to keep them separate for now so we can better evaluate any source of variabity in disaggregated EFs. I.e., if the EFs change a lot, is that caused by the disaggregation itself, or the reallocation?

Does this make sense, or do you think this separation is unnecesary even for diagnostics?

I would suggest keeping only implement_electricity_disaggregation

bl-young · 2026-06-09T20:08:10Z

    assert config.implement_electricity_reallocation is True


+def test_electricity_disaggregation_config_parsing() -> None:


I think this is duplicative of the validation in usa_config.py (though per my comment about the new flag this test might not be necessary at all)

bl-young · 2026-06-09T20:08:55Z

 ELECTRICITY_AGGREGATE = "221100"
 BALANCE_TOLERANCE = 1e6
+DISAGG_BALANCE_ATOL = 1.0
+IO_ACCOUNT_YEAR = 2017


this should come from the model parameter inside the model config

bl-young · 2026-06-09T20:12:36Z

    VA = derive_cornerstone_VA_after_waste()
    if electricity_reallocation_enabled():
        V, Udom, Uimp, VA = reallocate_electricity_coproduction(V, Udom, Uimp, VA)
+    if electricity_disaggregation_enabled():


I have this comment elsewhere as well but i think this double gating is unnecessary

bl-young · 2026-06-09T20:17:55Z

-        # `transform/output_data/` for 2019–2023. Load the cached parquet
-        # directly so the year-Y diagnostics get year-Y GHG data.
-        fbs = _load_cornerstone_ghg_fbs_from_gcs(usa.usa_ghg_data_year)
+        if usa.implement_electricity_disaggregation:


this implementation doesn't seem quite right. I think we should just be able to target the new method in _select_flowsa_ghg_method like other approaches. I think this particular if/else branch was intended for off years

…move unneeded files

… diagnostics

WesIngwersen · 2026-07-01T14:37:17Z

-        fbs = _load_cornerstone_ghg_fbs_from_gcs(year)
+        if usa.new_ghg_method and usa.implement_electricity_disaggregation:
+            try:
+                from flowsa import getFlowBySector as get_flowsa_fbs  # noqa: PLC0415


flowsa is no longer a separate required package.

This method must depend on eGRID. Coding in a fallback to the GHGI is overcomplicated.

There are correct references for getting an FBS elsewhere in this module.
getFlowBySector is already imported from
bedrock.transform.flowbysector

WesIngwersen · 2026-07-01T14:45:51Z

 )

+
+def _cornerstone_aq_matrix_set(


Why is this new internal function in here that is not specific to electricity disaggregation? Can it be avoided?

WesIngwersen · 2026-07-01T14:46:25Z


    if apply_inflation:
        V = inflate_cornerstone_V_with_industry_pi(V, target_year=target_year)
+    validate_cornerstone(V, "V")


Why add this validation check to this PR?

WesIngwersen · 2026-07-01T14:46:45Z

-    return _distribute_waste_parent_x_using_v_row_shares(x_cs)
+    x_cs.index.name = "sector"
+    x_out = _distribute_waste_parent_x_using_v_row_shares(x_cs)
+    validate_cornerstone(x_out, "X")


again this seems beyond the scope of this PR

WesIngwersen · 2026-07-01T15:51:46Z

@@ -3,6 +3,12 @@

 import pandas as pd


We should not be using this file any more for a main pipeline. I recommend moving all this code into usa_taxonomy_correspondence_helpers.py

WesIngwersen · 2026-07-01T15:54:40Z

+DISAGG_BALANCE_ATOL = 1.0
+IO_ACCOUNT_YEAR = 2017
+
+GENERATION_GO_SECTOR_NAMES: tuple[str, ...] = (


These tuples defining names should be in taxonomy

WesIngwersen · 2026-07-01T20:38:51Z

+    )
+
+
+def _diagonal_intersection_weights(w: pd.Series[float]) -> pd.DataFrame:


Unclear on what the w parameter is for..how do you prepare that. No documentation of that.

WesIngwersen · 2026-07-01T20:39:07Z

+    )
+
+
+def build_electricity_disagg_weights(w: pd.Series[float]) -> DisaggWeights:


Unclear on what the w parameter is for..how do you prepare that. No documentation of that.

WesIngwersen · 2026-07-01T21:08:09Z

Code runs to generate model. Model is generated and test_electricity_disaggregation test pass.

This was referenced Jun 9, 2026

Create branches and implement code disaggregation plan (sub-issues to be added as necessary) cornerstone-data/methods#79

Open

Modeling decisions for 221100 sector disaggregation cornerstone-data/methods#84

Closed

Implement print to excel for PR3 #463

Open

jvendries self-assigned this Jun 9, 2026

jvendries marked this pull request as ready for review June 9, 2026 16:56

jvendries requested review from MoLi7, WesIngwersen and bl-young and removed request for bl-young June 9, 2026 16:56

bl-young reviewed Jun 9, 2026

View reviewed changes

jvendries force-pushed the jv_electricity_disagg_PR3 branch 5 times, most recently from 93416b2 to f31e667 Compare June 19, 2026 23:46

jvendries mentioned this pull request Jun 19, 2026

implement analysis for decisions 3, 5, and 7 in dicussion #85 #481

Open

jvendries force-pushed the jv_electricity_disagg_PR3 branch 2 times, most recently from 309998f to 669acb4 Compare June 24, 2026 19:52

jvendries added 7 commits June 24, 2026 21:16

fix merge conflicts from #457

9717667

fix test failures

cd7b44c

fix mypy errors

6f76a9c

fix failing CI tests

18ab7f4

fix stale cache for electricity disagg tests

0085acc

add snapshot pointer to v0.2 in electricity disagg config file and re…

21810b7

…move unneeded files

fix formatting and mapping of disagg electricity sectors when running…

120547f

… diagnostics

jvendries force-pushed the jv_electricity_disagg_PR3 branch from 669acb4 to 120547f Compare June 25, 2026 01:20

jvendries mentioned this pull request Jun 25, 2026

Implement changes from resolutions from discussion #85 #490

Open

WesIngwersen added this to the v0.4 milestone Jun 29, 2026

jvendries mentioned this pull request Jul 1, 2026

Implement unit conversion #492

Draft

WesIngwersen reviewed Jul 1, 2026

View reviewed changes

Add config for full electricity model to the build_model_manual_check

0d11192

		assert config.implement_electricity_reallocation is True


		def test_electricity_disaggregation_config_parsing() -> None:

		)


		def _diagonal_intersection_weights(w: pd.Series[float]) -> pd.DataFrame:

		)


		def build_electricity_disagg_weights(w: pd.Series[float]) -> DisaggWeights:

Uh oh!

Conversation

jvendries commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changed? Why?

Testing

Uh oh!

jvendries commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bl-young left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jvendries Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WesIngwersen Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WesIngwersen commented Jul 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jvendries commented Jun 9, 2026 •

edited

Loading

jvendries commented Jun 9, 2026 •

edited

Loading

jvendries Jun 9, 2026 •

edited

Loading

WesIngwersen Jul 1, 2026 •

edited

Loading