Skip to content

Add pagination iterator helpers (iter_* or paginate()) to avoid manual while-loop boilerplate #40

@makegov-mark

Description

@makegov-mark

Summary

Every paginated endpoint in the SDK requires callers to write a manual while loop to collect all pages. As soon as a second paginated endpoint was added (get_entity_budget_flows, which now returns PaginatedResponse[dict[str, Any]]), the same boilerplate appeared again. We should ship a first-class pagination helper before more callers bake the loop into their code.

Boilerplate this would replace:

all_flows: list[dict] = []
page = 1
while True:
    resp = client.get_entity_budget_flows(uei, page=page, limit=100, fiscal_year=2024)
    all_flows.extend(resp.results)
    if resp.next is None:
        break
    page += 1

Options for discussion

Three shapes are on the table. The issue should land a decision before implementation begins.

Option 1 — Per-endpoint generator methods

Add an iter_* method alongside each list_* / sub-resource method:

for flow in client.iter_entity_budget_flows(uei, fiscal_year=2024):
    process(flow)
  • Discoverable via IDE autocomplete.
  • Fully typed — return type is Iterator[T] where T matches the endpoint's item type.
  • Downside: doubles the public surface area (one get_* + one iter_* per paginated endpoint).

Option 2 — Generic paginate() helper

A single helper (as a method or free function) that accepts any paginated callable:

for flow in client.paginate(client.get_entity_budget_flows, uei, fiscal_year=2024):
    process(flow)
# or as a free function:
from tango import paginate
for flow in paginate(client.get_entity_budget_flows, uei, fiscal_year=2024):
    process(flow)
  • Minimal surface area — one new symbol covers every current and future paginated endpoint.
  • Call-site is slightly awkward (passing the method as a callable).
  • Generic typing is harder to express precisely (ParamSpec / TypeVar gymnastics).
  • Works for both page-based and cursor-based conventions if implemented carefully.

Option 3 — Method on PaginatedResponse

Attach iteration behavior to the response object itself:

resp = client.get_entity_budget_flows(uei, fiscal_year=2024)
for flow in resp.iter_remaining(client):
    process(flow)
  • No new top-level surface; discovery happens on the object you already hold.
  • Mixes data and behavior: PaginatedResponse becomes aware of the client and of how to fetch more pages, which doesn't match the current SDK design.
  • Partial-page scenarios (user already consumed some results from resp) are awkward.

Decision criteria

The decision should cover:

  1. Which shape (1, 2, 3, or a blend — e.g. Option 2 as the implementation primitive + Option 1 as thin wrappers built on top of it).
  2. Scope of the first pass: page-based pagination only, or cover cursor-based endpoints in the same PR?

Out of scope

  • Auto-pagination by default: changing list_* / get_* methods to return an iterator would be a breaking change to existing callers.
  • Async iteration: no async client today; that's a separate concern.
  • Bulk/concurrent page fetching utilities.

Acceptance signal

This issue is resolved when the team has agreed on a shape and scope. Implementation is a follow-on PR.

Metadata

Metadata

Assignees

Labels

DXDeveloper experienceenhancementNew feature or request

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions