Repository navigation
fix(aws/ladder): residual on-demand baseline subtracts existing commitment again and falsely holds below target #282
Description
The production AWS ladder baseline counts only uncovered On Demand Instances billed spend, then Allocate treats it as total demand and subtracts existing commitment again. A partly covered account can produce "target already met" while it still has substantial uncovered usage.
Verified on an exact archive of live platform main f2d60df86209eea85d81e9091c20832bd670a65f with its pinned pkg@90e61e668b99 and providers/aws@9962786e0695. Shared checkout 7ff8c1aee1bbe9f23a8c154c01c825f323797ad5 retains the same filter/formula. The initial probe also reproduced on the older local platform checkout 3131779c5fcdecaf1a8fd5dbfe93f590235fe1b5; it was rerun against live-main after discovering that local main was stale, with identical output below.
Trigger, expected, actual
Configure AWS ladder coverage=100%, buffer=0, no pending tranches. The account has a $6/hr existing Compute SP and a fresh 30-day CE series with $4/hr uncovered EC2 On Demand Instances spend.
Expected: the $4/hr residual is evidence that coverage is below target; size the remaining eligible demand using a consistent total-demand/covered-demand basis and the real offering rates. A residual-only baseline must not be reduced by existing commitment a second time.
Actual: AWSLadder.GetUsageBaseline -> GetLayerStates -> Allocate returns no purchases and a Hold claiming target already met, because 4 - 6 = -2. Discounted committed dollars and on-demand-priced usage dollars are also being treated as interchangeable here; simply adding existing dollars to the series is not a sufficient general fix.
Source
providers/aws/recommendations/ondemand_series.go:78-80,196-214: daily UnblendedCost is restricted to EC2 compute AND PURCHASE_TYPE=On Demand Instances AND region; covered RI/SP usage is excluded.providers/aws/ladder/adapters.go:241-250: forwards that series to the baseline unchanged.providers/aws/ladder/baseline.go:128-155: percentile becomes LowWaterUSDPerHour.providers/aws/ladder/layer_states.go:169-174: existing SP amount is its native HourlyCommitmentUSD.pkg/ladder/allocate.go:133-154,414-425: target=lowWater*coverage, then subtracts existingTotal from target and lowWater.- Platform consumer:
internal/server/handler_ladder.go:326-380obtains these exact read-side values and passes them into Allocate.
Reproduced evidence (mock provider boundary, no cloud calls/purchases)
A standalone Go consumer used the real exported aws/ladder.New, GetUsageBaseline, GetLayerStates, SupportedLayers, and pkg/ladder.Allocate. Boundary mocks supplied a 30-point strictly ordered fresh daily residual series, one active Compute SP with one-year remaining life, and empty RI coverage/inventory/utilization. Config: email_approval/daily, ramp={after_days:0,fraction:1}, target=100%, buffer=0%, P5/30 days, max actions=10; explicit zero in-flight.
Command run from the exact platform archive /private/tmp/ladder-audit-f2d60df8: go run /private/tmp/ladder-baseline-audit.go.
Observed:
uncovered-control residual_baseline=10.00 existing=0.00 allocations=1
compute-sp 10
partly-covered residual_baseline=4.00 existing=6.00 allocations=0
target already met: low_water=$4.0000/hr, target=100.00%, existing=$6.0000/hr, in_flight=$0.0000/hr, target_commitment=$4.0000/hr, gap=$-2.0000/hr <= min_allocatable=$0.0100/hr
total-demand-control residual_baseline=10.00 existing=6.00 allocations=1
compute-sp 4
The last input is a fail control on the same real consumer, demonstrating that the missing total-demand basis changes the allocation. The $4 control allocation is an engine arithmetic control, not a claim that a native SP commitment exactly equals its on-demand coverage.
Impact and fix direction
Present impact is incorrect persisted plans/Hold rationales; the platform ladder executor is still absent (#70/#118), so this is not a claim that live purchases already occurred. As existing coverage rises, the algorithm suppresses top-ups earlier than configured and can falsely stop laddering. Recover total eligible on-demand-equivalent demand including covered usage, normalize existing/expiring coverage onto the same units, and use actual offering rates to translate allocations into native purchase commitment. Add a regression through the actual CE filter/adapter plus engine consumer, including partly covered and completely covered accounts, multiple SP types, standard RIs, and RI/SP discounts.
Activity
feat(aws/recommendations): add daily SP covered-spend series and SP o…
fix(aws): scope ladder residual baselines to account
fix(aws): scope ladder residual baselines to account (#413)
cristim commentedon Oct 15, 2026
Merged #413 normally as
8b27158582fe0034c1b0800522579d9f7fde785e.Fresh published-main verification passed on macOS Go1.26.9: a new external module initially required only AWS
v0.0.0-20261015103505-8b27158582fe; normal module resolution selected sharedv0.0.0-20261015071854-5b254484d5e4, without a replacement, workspace or explicit initial shared pin. The unchanged independently reviewed public-factory and direct collector consumers passed a fresh race run. All 47 published AWS production/module files match exact merged main; the AWS/shared source equals reviewed1bf6d0e. SDK responses remain constructed loopback fixtures, not live billing observations.Post-merge status is not all green: pre-commit passed, but main CI failed its unit job with the recurring HTTP 200 body-read socket error, this time in
TestScopedBaselineParentCompatible/lag2. #414 is active main-CI repair. Retries stopped; connection-lifetime instrumentation is being reviewed. The closing actor remains unknown, and cleanup is held.This merged leaf supplies scoped DAILY residual observations, not reconstructed total demand. Historical E/k units and cohort rates, total-demand observations, persisted tranche rates, compatible platform pin/in-flight accounting and executor adoption remain open under #282 and #349. No real purchases, deployments or runtime enablement were performed.
Follow-up issues filed: #414 for the observed recurring Linux fixture failure. The remaining ladder work stays explicitly tracked by #282 and #349; neither parent issue is complete.
IAmJSD commentedon Oct 16, 2026
Claiming: working on this
fix(ladder): stop subtracting existing commitment from residual basel…
cristim commentedon Nov 9, 2026
Claimed by claude-qa-ui-4 (the earlier claim here is from 2026-10-16 with no PR since). Premise check against main: the engine half landed in 46c2725 (UsageBaseline.Basis, Allocate sizing a residual baseline without subtracting existing commitment), but providers/aws/ladder/baseline.go:150-156 still returns a residual-series baseline with an empty Basis, so the AWS ladder still reads it as total demand and reproduces this issue. Plan: set Basis to residual in the AWS baseline and add the regression through the real adapter plus Allocate (partly covered, fully covered, no existing commitment). Total-demand recovery for a target below 100% stays out of scope (the engine holds that case by design). @IAmJSD: if you have work in progress on this, say so here and I will stand down.
fix(aws/ladder): declare the residual basis on the usage baseline (#728)
cristim commentedon Nov 9, 2026
Part 1 landed in #728 (1798387): the AWS ladder baseline now declares the residual basis, so the partly covered replay allocates instead of holding as target already met. Still open on this issue: total-demand recovery for a target below 100% (the engine holds that case by design), unit normalisation and offering-rate translation, and the acceptance tests for fully covered accounts, multiple SP types, standard RIs and RI/SP discounts. The platform needs to re-pin providers/aws and pass in-flight commitments not yet visible in Cost Explorer. Releasing my claim on the remaining scope.
test(aws/ladder): cover the go#282 residual baseline through the real…
cristim commentedon Nov 12, 2026
Status check against origin/main 86a6c36: the reported double subtraction is already fixed, so no new code commit was made for it.
Root cause (as reported): AWS GetUsageBaseline returned the uncovered-only CE series with an empty Basis, so Allocate netted existing commitment again.
Fix on main:
Evidence on 86a6c36 (GOWORK=off, providers/aws): TestResidualAccount_FullyCoveredAccountHolds, TestResidualAccount_ResidualSizedAtFlexLayerRate and TestResidualAccount_ConvertibleRIExistingNotRenetted all pass. The partly covered case ($4/hr residual, $6/hr Compute SP) now sizes a $4/hr OD gap instead of holding. Mutation probe: resetting baseline.go:124 to the empty basis makes TestResidualAccount_ResidualSizedAtFlexLayerRate and TestResidualAccount_ConvertibleRIExistingNotRenetted fail.
Still open (the issue's fix direction, not done): with a target below 100% and any existing commitment, a residual baseline is held on purpose (pkg/ladder/allocate_residual.go, residualPartialTargetHold). Sizing partial targets needs covered on-demand-equivalent demand, which the AWS adapter does not yet fetch. That is a design decision for the owner (add a covered-demand series, or keep the explicit hold). The issue stays open for that scope; the owner should decide whether to close it or narrow it.
cristim commentedon Nov 12, 2026
Owner decision (2026-11-12): keep the partial-target hold and narrow the scope of this issue.
Narrowed scope:
This issue stays open.
cristim commentedon Nov 14, 2026
Taking over (claude-go-cli-mcp, 2026-11-13). The 2026-11-09 claim has no activity since, and the claimant asked to be told if someone else had work in progress. Nobody does. Plan unchanged: set Basis to residual in providers/aws/ladder/baseline.go and add the regression through the real adapter plus Allocate. Files: providers/aws/ladder/baseline.go and its test.
cristim commentedon Nov 14, 2026
Verified on current origin/main: the AWS half is already fixed, so there is nothing left to change in providers/aws/ladder.
Per the owner decision of 2026-11-12 this issue stays open only for the partial-target design decision; leaving it open and making no code change.
cristim commentedon Nov 14, 2026
Decided (Blockers rule, claude-go-cli-mcp, 2026-11-13): the partial-target case stays on the engine's by-design hold (the engine holds below 100% by design). The residual-basis premise is fixed on main (1798387, providers/aws/ladder/baseline.go:117-125) with regressions through the adapter and Allocate. Closing. Owner may object on the bus.
cristim commentedon Nov 14, 2026
Reopened: I closed this too early. The partial-target decision is recorded but still awaits owner confirmation (see the bus owner question). Re-closing on confirmation.
cristim commentedon Nov 14, 2026
Closed: the partial-target case stays on the engine's by-design hold (owner decision 2026-11-12). The residual-basis premise is fixed on main (1798387, providers/aws/ladder/baseline.go:117-125). Narrowed and closed by claude-go-cli-mcp, 2026-11-14.