Add 2024 ATB Update Pipeline by Yunzhi-Chen · Pull Request #14 · ReEDS-Model/ReEDS_Input_Processing

Yunzhi-Chen · 2026-08-19T03:57:32Z

Summary

Adds a workflow that converts corrected ATB 2024 data into ReEDS-compatible inputs. It downloads, formats, plots, and validates ATB data, fills historical years from an explicitly configured source, optionally smooths future years, and produces before/after plots for review.

Structure

atb/
├── config.yaml            # all user choices; validated by config.schema.json
├── scripts/               # pipeline (13 modules)
├── historical/            # prepared history, committed — atb.csv, real.csv, manual.csv
├── manual_input/          # projection-year config (CSP ratios, offshore multipliers, coal overrides)
├── scraped_input/         # raw download cache (git-ignored)
├── output/                # generated ReEDS inputs (git-ignored)
└── comparison/            # validation and before/after plots (git-ignored)

How to run

cd atb
python scripts/future_atb_scraper.py     # cache the ATB flat file and workbook
python scripts/run_pipeline.py           # format, plot, compare

run_pipeline.py runs offline against the committed history and never rewrites it. Use --only format to generate CSVs without plots.

To rebuild history from original sources:

python scripts/historical_data_scraper.py    # --no-download to rebuild from cache

Verification

With smoothing disabled and all metrics set to manual, the workflow reproduces the inputs ReEDS uses today. Of 32 files: 21 exact, 8 differing only by floating-point noise (≤1e-9 relative), 3 offshore files differing by up to 0.018% in the CF multiplier, consistent with a different ATB 2024 release than the pinned v3.0.0.

Sanity-check plots — smoothing off, manual history, vs current ReEDS:

sanity_check_2024.zip

New ATB 2024 plots for future years:

New ATB 2024 plots and the components:

New ATB 2024 vs current ReEDS 2024:

sanity_check_2024.zip

Scope and open questions

Geothermal is not generated;

LLM usage

LLM tools assisted with implementation, documentation, debugging, and validation. Outputs were reviewed and tested against the current ReEDS inputs.

Update Links

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…l/ReEDS_Input_Processing into yc/natural_gas_update

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…config.json" This reverts commit 92f0e8b.

…ession/README.md Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

Update AEO preprocessing to pull from AEO2026

…irectly

This reverts commit c43c67f.

…om AEO directly" This reverts commit 0e86670.

Add natural gas price regression pipeline

Remove highprice and lowprice since AEO 2026 does not have the two scenarios

Update natural gas regression parameters with AEO 2026 data

Fix formatting issues in README.md

…s' folder

… Vogtle

bsergi

Thanks Yunzhi for all the hard work on this.

Based on the outputs it seems to work, but my biggest concern is that it is very complex and would be difficult for anyone else to use. A few suggestions:

Put non-user facing helper scripts in a sub-folder (or if they are few enough, move the user-facing ones up to the root director)
Simplify some of the input files with lots of columns or text
Organize the root directory folders (e.g., have one inputs folder with subfolders for historical data or other categories).
Add more comments to the code to explain what is going on, including function docstrings
Try to reduce the length of the code
Consider break up config file into more manageable chunks, with instructions about what settings need to be re-evaluated with a new run.

bsergi · 2026-09-16T14:37:35Z

@@ -0,0 +1,34 @@
+{


Why is this file indented like this? It seems like all the lines could be moved 2 tabs to the left.

Same comment for the other json files in this folder.

bsergi · 2026-09-16T14:41:38Z

+  index; used for wind-ons and upv FOM, whose surveys report a partial-scope
+  level but a usable trend.


The phrase "whose surveys report a partial-scope level but a usable trend" doesn't make sense to me.

bsergi · 2026-09-16T14:44:50Z

+  investment before its own `firstyear`, so the value only needs to be far
+  above any real cost (ReEDS's own files use 9999, which is below Vogtle).
+- `atb`, `manual`, `broadcast`: archived ATB estimates, the ReEDS ATB 2024
+  baseline, or the first projection value held flat.


By baseline do you mean moderate? And I think it would be better call the broadcast option "backfill".

bsergi · 2026-09-16T14:58:42Z

+`indexed` for O&M), hold the first ATB projection value flat otherwise
+(`broadcast`), and write the ReEDS placeholder before a technology's first
+buildable year (`unavailable`). Archived ATB estimates (`atb`) and the ReEDS
+ATB 2024 baseline (`manual`) remain selectable per metric in `config.yaml`.


baseline or moderate? and should it be stuck to 2024 or alter with the ATB being processed? I'm not sure about calling this 'manual' either as that doesn't help understand what is being done.

bsergi · 2026-09-16T15:00:11Z

+
+The default projection start follows `atb.year - 2`; individual series retain
+their actual start year. Financial cases can be changed in config without
+editing `settings.yaml`. To match `yc/25ATB`, use:


should the README be hardcoded with a reference to a branch?

bsergi · 2026-09-16T15:39:28Z

@@ -0,0 +1,468 @@
+# yaml-language-server: $schema=./config.schema.json


I think it would be good to have the config be focused on fields the user is expected to update, with other files provided entries that aren't expected to change. I can't quite tell what in here falls in that category but given how long it is I worry that users won't know what needs to be updated.

bsergi · 2026-09-16T15:39:56Z

+          unit: USD/kW
+          capacity_basis: AC
+          statistic: project
+          geography: United States


why is this needed?

bsergi · 2026-09-16T15:41:49Z

+        avail_params = atb_data.loc[(atb_data.Technology == tech) 
+                                    & (atb_data.Scenario.isin(scenarios)), "Parameter"
+                                    ].unique()
+        if args.debug:


Is this still relevant or a holdover from earlier code?

bsergi · 2026-09-16T15:42:57Z

+
+    return output.sort_index()
+
+def add_beccs_techs(tech, settings, df, techcol='i'):


We'll probably need to debug this one when testing ATB 2025 for BECCS

bsergi · 2026-09-16T15:44:13Z

+        keys = keys if isinstance(keys, tuple) else (keys,)
+        labels = dict(zip(['Scenario', *ids], keys))
+        boundary = settings['atb_series_start'][tech].get(keys, smoothing['projection_start_year'])
+        series = labels.get(identity, '*') if tech != 'wind-ons' else '*'


What's going on here? Why is this specific to wind-ons ?

kennedy-mindermann and others added 30 commits April 30, 2026 17:13

Merge pull request #4 from ReEDS-Model/km/update-links

2fe99c1

Update Links

Update aeo_updates/natural_gas_price_regression/aeo_pipeline_config.json

550e815

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'yc/natural_gas_update' of https://github.com/ReEDS-Mode…

e61a0d5

…l/ReEDS_Input_Processing into yc/natural_gas_update

Fix stale filename references

8f15643

Update aeo_updates/natural_gas_price_regression/visualization.py

d1eb5f4

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update aeo_updates/natural_gas_price_regression/.gitignore

ba50fd1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Revert "Update aeo_updates/natural_gas_price_regression/aeo_pipeline_…

bb3aa74

…config.json" This reverts commit 92f0e8b.

Sort region in alphabetic sequence

b154384

Update aeo_updates/natural_gas_price_regression/inputs for alpha regr…

ae18ddb

…ession/README.md Co-authored-by: Wesley Cole <49044852+wesleyjcole@users.noreply.github.com>

Update README.md

03fc4de

Merge pull request #5 from ReEDS-Model/AEO_2026

aa41d33

Update AEO preprocessing to pull from AEO2026

Pull historical natural gas price, demand and total demand from AEO d…

0a34df7

…irectly

Update README.md

05cfc3c

Revert "Update README.md"

1c891a8

This reverts commit c43c67f.

Revert "Pull historical natural gas price, demand and total demand fr…

0d962fe

…om AEO directly" This reverts commit 0e86670.

Fix historical year dollar year issue

b55edb2

Merge pull request #2 from ReEDS-Model/yc/natural_gas_update

d043cf7

Add natural gas price regression pipeline

Merge branch 'main' into yc/natural_gas_update_2026

b7d1037

Delete national_beta.csv

f08e401

Update readme and historical info

a50bc40

Update aeo_pipeline_config.json

d173649

Remove highprice and lowprice since AEO 2026 does not have the two scenarios

Merge pull request #3 from ReEDS-Model/yc/natural_gas_update_2026

5940f9a

Update natural gas regression parameters with AEO 2026 data

Fix formatting issues in README.md

6b7e84c

Merge pull request #7 from ReEDS-Model/wesleyjcole-patch-1

f0da5ff

Fix formatting issues in README.md

Add temperature - gas price regression workflow

73cffd8

Remove unused imports, add clarifying comments, and export to 'output…

4a572b3

…s' folder

Remove intermediate input file

731d24c

Update README.md

eb6584e

Add .gitignore

b5b8638

Clarify README

4abf7f1

Yunzhi-Chen added 4 commits September 1, 2026 20:36

update

4f1f4ba

clean up

41b94db

Merge remote-tracking branch 'origin/main' into yc/24ATB

d195c93

Fix gas H frame stuffs

f5a0631

Yunzhi-Chen requested a review from bsergi September 2, 2026 22:04

update

5a6b7ed

bsergi mentioned this pull request Sep 5, 2026

Separate historical and projected technology costs ReEDS-Model/ReEDS#215

Open

Yunzhi-Chen and others added 11 commits September 4, 2026 23:31

Update atb_plotting.py

6cec19d

Recovers atb_plotting using An's code

1454d62

add historical cf and csp costs

4bcdd21

add historical battery costs

feefd46

Add atb as estimated historical

d0eb828

refine historical data

dc60af5

Update config.yaml

e7032c5

Merge branch 'main' into yc/24ATB

42210e2

Add real historical costs by scraping nuclear plants of Watts Bar and…

96d0b75

… Vogtle

Apply CAPEX to OCC factor to real historical costs

336b423

Enhance historical data processing

324522f

bsergi reviewed Sep 16, 2026

View reviewed changes

Yunzhi-Chen added 11 commits September 17, 2026 11:50

Remove coal override artifact

3e245ba

Remove historical_atb option and relative code

e518623

clean up

596a110

Update config.yaml

0e9764c

Update generate_atb_files.py

9d2d433

Update config.yaml

b42ce71

Update the plots style

aadf69d

update plots

a82b088

update

b7db5d7

Add future data source to config file

407c8cd

Refine future data options for config

d2718ce

Navigation Menu

Add 2024 ATB Update Pipeline - #14

Add 2024 ATB Update Pipeline#14
Yunzhi-Chen wants to merge 86 commits into
mainfrom
yc/24ATB

Yunzhi-Chen commented Aug 19, 2026 •

edited

Loading

Uh oh!

bsergi left a comment

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

bsergi Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		index; used for wind-ons and upv FOM, whose surveys report a partial-scope
		level but a usable trend.

		@@ -0,0 +1,468 @@
		# yaml-language-server: $schema=./config.schema.json


		return output.sort_index()

		def add_beccs_techs(tech, settings, df, techcol='i'):

Conversation

Yunzhi-Chen commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Structure

How to run

Verification

Scope and open questions

LLM usage

Uh oh!

bsergi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Yunzhi-Chen commented Aug 19, 2026 •

edited

Loading