Make df.list_instances() page-size cap a loud, tunable error (#146)#280
Open
crprashant wants to merge 1 commit into
Open
Make df.list_instances() page-size cap a loud, tunable error (#146)#280crprashant wants to merge 1 commit into
crprashant wants to merge 1 commit into
Conversation
…ft#146) Replace the silent limit_count.min(10000) clamp in both df.list_instances() overloads with a loud error when limit_count exceeds the new pg_durable.list_instances_max_limit GUC (SUSET, default 1000, range 1..1000000). An over-cap request now fails fast and points the caller at keyset pagination (after_cursor/next_cursor) instead of returning a silently short page that is indistinguishable from 'no more rows'. The cap is superuser-settable at runtime without a restart; by default an ordinary caller cannot raise it, so the guardrail holds. df.instance_executions keeps its own fixed min(10000) clamp (a different per-instance-history surface). No SQL/schema change: the GUC is registered in _PG_init at the C level, so there is no upgrade script and binary backward-compatibility holds trivially against all prior df schemas.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the silent page-size clamp in
df.list_instances()with a loud, tunable error.Both overloads of
df.list_instances()previously truncatedlimit_countsilently to a fixed ceiling of10000. A caller asking for more rows got a short page back with no signal that truncation happened — indistinguishable from "no more rows". This PR replaces that clamp with a new GUC and a fail-fast guard:pg_durable.list_instances_max_limit—int, default1000, range1–1000000,SUSETcontext.limit_countabove the cap now raises an error that names the GUC and points the caller at keyset pagination, instead of silently returning fewer rows.Both the basic 2-arg overload and the paginated 4-arg overload (PR #278) share one guard,
enforce_list_instances_limit().df.instance_executions()intentionally keeps its own fixedmin(10000)clamp — it reports a single instance's execution history, a different surface with a naturally small result set.Files:
src/lib.rs(GUC),src/monitoring.rs(shared guard + both call sites), docs, e2e.Why
This is PR 5 of the agreed, incremental monitoring-functions plan on #167, and it lands pinodeca's P0 from #146: a silently short page breaks external clients that page by
limit_count, because a truncated page looks identical to the last page. PR #278 already shipped keyset pagination (after_cursor/next_cursor) as the correct path for large result sets; this PR makes an over-cap request fail fast and steer the caller there, rather than returning misleading data.Design note —
SUSETcontextThe cap is read on the
df.list_instances()query path, so a superuser must be able to tune it per session at runtime — henceSUSETrather thanPostmaster.SUSETis also the guardrail: by default an ordinary caller cannot raise the cap to bypass it (a superuser can delegate that withGRANT SET ON PARAMETER pg_durable.list_instances_max_limit). The parameter is notSUPERUSER_ONLY, so any caller can stillSHOWit to see why a large request errored.Upgrade & compatibility
_PG_initat the C level — there is nosql/upgrade script, no.control/Cargoversion bump, and no Scenario A snapshot impact..soworks against all priordfschemas. The guard runs before any SQL is issued and reads no catalog or table state — it only compares the caller'slimit_countagainst the in-memory GUC — so it is correct against a pre-0.2.4 schema that has not runALTER EXTENSION UPDATE. The only visible difference against an old schema is the intended one: a largelimit_countnow errors instead of being silently capped at 10000.pg_durable.enable_superuser_instances.test-upgradeis not required (no schema change); it is documented indocs/upgrade-testing.mdalongside that precedent.Behavior change (flagged): the default ceiling drops from
10000to1000, and over-cap requests now error instead of truncating. Recorded inCHANGELOG.mdunder the unreleased0.2.4changeset.0.2.4is still unreleased, so there is no version bump.Testing
Full CONTRIBUTING pre-PR workflow, all green:
cargo fmt -p pg_durable -- --check— cleancargo build --features pg17/cargo clippy --features pg17— clean, no warningsscripts/test-unit.sh— 194 passed, 0 failedscripts/test-e2e-local.sh— 38/38.tests/e2e/sql/05_monitoring_and_explain.sqladds a three-part block (run as superuser afterRESET SESSION AUTHORIZATION): (1) default cap raises on both overloads and the message names the GUC, with the at-cap page (1000) provably allowed; (2) lowering the cap to3allows the at-limit page and raises on4echoing(3)for both overloads; (3) a non-superuserset_config(...)on the GUC raisesinsufficient_privilege(asserted on SQLSTATE42501).This branch is based on the latest
main(includes #277/#278).Scope
Truncation-policy GUC only.
df.instance_executions()keeps its own fixed clamp (out of this PR's minimal scope; give it its own bound as a follow-up if per-instance history ever needs one).cc @pinodeca