Repository navigation
Discuss: Store price data #182
Description
Verification
- I have searched the existing issues to ensure this feature has not already been requested.
Provide a detailed description of the proposed feature.
Store price data in the data-platform
What is the motivation for the feature?
We currently collect DA price data from Entsoe everytime the NL Consumer and Forecast runs. This seems slightly over kill, and we should only collect this one.
Its possible to believe that we might need to store more of this data in the future
Outline a potential approach or solution (optional).
- New schema called prices
- one table thats stores the prices, columns could be, location_uuid, timestamp_utc, value, market (note market could be in a separate table to save space rather than reparting 'NL_DA')
- write function
- read function
Note it is quite similar to an observation (just not MW, its £/MW), so it might be possible to squeeze it into there
Activity
[-]Store price data[/-][+]Discuss: Store price data[/+]peterdudfield commentedon Sep 4, 2026
we current get prices here
We could even add a route to the data-platform
This means we wouldnt have to set up a cron job to get them, they would be available on request
This would be a new behaviour for the data-platform
Another ideas could be to built a separate consumer (maybe in Go) that goes and collects the data from Enstoe and saves it to the data-platform. In the future im sure we'll save more than just prices like wind generation, demand e.e.tc from Enstoe
peterdudfield commentedon Sep 7, 2026
A shortcut solution, wiht the current Data-platform, would be to store these prices as a observer (in solar-consumer) and load them in the (site-forecast-app)
braddf commentedon Sep 9, 2026
I would say in the immediate term that storing these in the DP sounds fine. I feel like @devsjc designed observers to be source-type-agnostic, so maybe we'd just add a new
source_typeand then use the existingentsoe_deobserver, with price values saved against that masquerading asobserved_generation_valuesas usual. These would be easy to shift elsewhere should we change our minds.As long as the price value stored in cents is always below 32k. This is okay during normal peace-time energy markets I think(?) with normal peaks around 200-240€, so 20,000-24,000 cents, but the historic max from 2018 was way over at 699.44€. I think @devsjc may be intending to change this
small inttype due to other constraints, but I'm not 100%. Another option is to arbitrarily offset by a set amount (e.g. 10k or 20k) on write/read from the DP, to give ourselves more headroom, as to my knowledge the negative prices never go too far below zero? This would obv add complexity and overheads on I/O, but I guess this doesn't plan to get read/written super often so maybe that's acceptable, if a little janky in the meantime!peterdudfield commentedon Sep 9, 2026
Thanks @braddf
yea, they are kinda of generation but do have a capacity_watts associated with them, which we would have to use in a slightly different way
Negative prices can get close to -100 euros per mwh and getting near 1000 euros per mwh is not un heard of.
So i would probably be tempted to
braddf commentedon Sep 9, 2026
Not sure I understand what you mean. What has capacity watts associated with them, the price data? Do you have an example of what they look like?
Why would we add to the price, and are you meaning euros or eurocents?
Storing negative values in the DB is fine, down to -32k as well, it shouldn't mind, so we should save together for simplicity.
peterdudfield commentedon Sep 9, 2026
Yea i agree, it shouldnt have a capacity watts associated with it. However
Ill have to check, im not sure generation values can be negative, i think the Data-platform had a check in for that
but we could put this as 1000 (dummy) and that should be ok.
peterdudfield commentedon Sep 9, 2026
Here I think means you can only add postive observation values. Hence a solution of saving both negative and positive prices
braddf commentedon Sep 9, 2026
Ah I see, might need to disable that check then, this could be similar for BESS etc. in future too, and yeah I guess we can dummy the capacity, although I'm just thinking... until these become past actuals... a DA price is still a forecast 🤔 so maybe that's its more sensible home?
braddf commentedon Sep 9, 2026
There's also potentially value in storing the predicted load alongside the price forecast in this way too?
braddf commentedon Sep 9, 2026
But we can do that afterwards/incremental improvements.
braddf commentedon Sep 9, 2026
Also just thought I assume we're only using the SDAC price atm, and not the EXAA price as well, not sure if that will also affect curtailment or impact forecasts.
peterdudfield commentedon Sep 9, 2026
I think DA prices are not a forecast, they are prices that things are traded on. They dont change. There are forecasts for DA prices, that are available before those prices come out, but DA prices are static. I think thats why I wanted it in the observer, not a forecast.
yep, storing negative values, and forecast for BESS, we might need something like that.
Note, im trying to get the balance of long term solution with data-platform and fix for NL ENSTOE problem (similar to Slack conversation)
peterdudfield commentedon Sep 9, 2026
Unfortunately, observers can only be in the past, whereas we want to save (static) prices for the future. So we might have to save them as forecasts, but we need to think about how best to save these
peterdudfield commentedon Sep 23, 2026
The short term solution of trying to save prices as observables didnt work (see comments above), I think the properly solution with changes to the data-platform is needed