feat: cache and back off Electricity Maps carbon intensity - #1358
feat: cache and back off Electricity Maps carbon intensity#1358davidberenstein1957 wants to merge 5 commits into
Conversation
Carbon intensity was fetched from the Electricity Maps API on every emissions computation, so a long run with a short measure_power_secs issued thousands of requests for a value the grid publishes hourly. A failing token produced one doomed request per measurement tick for the whole run. Extract get_carbon_intensity() from get_emissions(), cache its result for 5 minutes per location, and put the API in an exponential cooldown (30s to 1h) after a failure. get_emissions() is unchanged for callers. Refs #1354 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The test varies carbon intensity per measurement to prove emissions are accumulated as deltas rather than recomputed from the latest intensity. The new 5-minute TTL cache served the first value for all three ticks. Disable the TTL for this test so it still exercises the cumulation contract, and reset the module-level cache to avoid cross-test leakage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
CI fix pushed. The failing test ( What changed is the test only: it now patches The caching behaviour itself remains covered by Verified locally: |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1358 +/- ##
==========================================
+ Coverage 91.39% 91.60% +0.20%
==========================================
Files 49 49
Lines 5056 5109 +53
==========================================
+ Hits 4621 4680 +59
+ Misses 435 429 -6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- key the cache by token so trackers with different tokens do not share a value - raise a dedicated cooldown error and log it at debug, so a bad token no longer produces one error line per measurement tick - document the 5 minute cache TTL as a behaviour change Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The raw API token was part of the in-memory cache key, so it was stored in the module-level cache and rendered by the debug log that reports a cache hit. Key on a sha256 prefix instead: tokens still get distinct cache entries, but the secret is never held nor logged. The cache and cooldown state are read-modify-written from the background measurement thread. Guard them with one module-level lock, never held across the HTTP request. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL flags the SHA256 digest as py/weak-sensitive-data-hashing: the parameter name marks it as a credential, and any hashlib digest of a credential reads as an insecure password hash. The key only has to tell two tokens apart inside one process, so use the builtin randomly-seeded hash() instead. No caching, backoff or public API change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slice 1 of 5 of the pluggable carbon-intensity provider work described in #1354. It stands on its own: it fixes the two live defects in the current Electricity Maps path without introducing any new abstraction, config key, or output field.
What is in this slice
codecarbon/core/electricitymaps_api.pyfetched the grid carbon intensity on every emissions computation. On a long run with a shortmeasure_power_secsthat is thousands of HTTP requests for a value the grid publishes hourly at best. And when the token is wrong or the network is down, it is thousands of doomed requests plus onelogger.errorline per measurement tick for the entire run.get_carbon_intensity(geo, token) -> float(gCO2e/kWh) out ofget_emissions(). Asking for the intensity rather than for emissions-given-energy is the shape the provider layer needs, and it is useful on its own.lat/lonorcountryCode), so a 1000-tick run makes one request instead of 1000.ElectricityMapsAPICooldownError, whichemissions.pylogs at debug level, so a bad token produces one error line instead of one per tick.reset_cache()for tests.get_emissions()keeps its exact signature and return value and now delegates, so the three existing Electricity Maps test files are untouched.codecarbon/core/emissions.pygains oneexceptclause for the cooldown error.Behaviour change, not a pure optimisation: the 5 minute TTL means measurements inside that window convert energy with the same intensity value rather than a freshly fetched one. A run shorter than the TTL now uses a single reading.
tests/test_emissions_tracker.py::test_cumulative_emissions_with_varying_intensitypatches the TTL to-1because it deliberately feeds a different intensity per tick. The cache is keyed by location and token, so two trackers in one process with different tokens do not share a value. Documented indocs/how-to/configuration.md.What is deferred
CarbonIntensity/IntensityProviderincodecarbon/core/intensity/, with today's bundled-data branches lifted out ofemissions.pyunchanged asStaticProvider. No number moves.resolve_intensity()with the fallback chain, wired intoget_private_infra_emissions(), plus thecarbon_intensity_providersconfig key and its backward-compatibility default.carbon_intensity_g_co2e_kwhandcarbon_intensity_sourceonEmissionsData, so silent fallback to yearly averages becomes visible in the CSV and in Prometheus.Also deliberately not in this slice: stale-serve (returning an expired cached value when the API errors). It trades a silent inaccuracy for continuity and should land together with the
is_live/carbon_intensity_sourcereporting in slice 5, not before there is any way to see it happened.Tests
tests/test_electricitymaps_cache.py, all network mocked withresponsesfollowingtests/test_electricitymaps_api.py: cache hit within TTL, a 1000-tick run bounded to one request, refetch after expiry, per-location keying, no request while in cooldown, cooldown doubling to the ceiling, and cooldown reset after success.Draft pending review of the overall direction in #1354.
Refs #1354
🤖 Generated with Claude Code