Skip to content

Pace the requests Part-DB sends to the info providers - #1536

Open
killecaptron wants to merge 1 commit into
Part-DB:masterfrom
killecaptron:info-provider-rate-limit
Open

Pace the requests Part-DB sends to the info providers#1536
killecaptron wants to merge 1 commit into
Part-DB:masterfrom
killecaptron:info-provider-rate-limit

Conversation

@killecaptron

Copy link
Copy Markdown
Contributor

Info providers limit requests per account, usually in short windows - TrustedParts, for example,
allows 50 requests per 10 seconds and 150 per minute - and many of them bill per request.
Part-DB can easily produce bursts above that: a bulk import, or an update of many parts, walks
through them as fast as the network allows with nothing in between.

Worse, the limit belongs to the account, not to Part-DB. A script maintaining parts through the
API uses the same credentials, so its requests and Part-DB's add up while neither side can see
the other's count. Coordinating the two by hand is the only remedy today, and it fails the moment
somebody clicks "update from info provider" while a job is running.

Part-DB now paces itself. Two sliding windows per provider are enforced in PartInfoRetriever,
inside the cache callbacks - the single point every provider request passes, and the only place
where a cache hit can be told apart from a request that really goes out. The counters live in the
shared cache pool, so web requests, console commands and background jobs all count against the
same budget.

Requests are delayed, not rejected: a bulk operation simply takes longer, which is what it wants.
Only if a single request would wait longer than the configured maximum does it fail, with HTTP 429
and a Retry-After header naming the seconds until the next free slot, so an automated caller can
tell "too fast, come back shortly" apart from a real error.

The defaults (35 per 10 seconds, 120 per minute, 30 seconds maximum wait) stay deliberately below
what providers permit: the remaining headroom is what keeps Part-DB and an external script sharing
the account from pushing each other over the real limit. Each window can be switched off by
setting it to 0.

Two limitations worth naming. The counter updates are not synchronized - symfony/lock is not a
dependency - so parallel processes can slightly under-count, which is why the configured limit
should stay below the provider's own. And a provider with an internal cache of its own consumes a
slot even when it answers from that cache, which errs on the safe side.

This is the follow-up I offered in #1532.

Info providers limit requests per account, usually in short windows -
TrustedParts, for example, allows 50 requests per 10 seconds and 150 per
minute - and many of them bill per request. Part-DB happily exceeds
that: a bulk import or an update of many parts walks through them as
fast as the network allows, with nothing in between.

Worse, the limit belongs to the account, not to Part-DB. An external
script maintaining parts through the API uses the same credentials, so
its requests and Part-DB's add up while neither side can see the other's
count. Coordinating the two by hand is the only remedy today, and it
fails the moment somebody clicks "update from info provider" while a job
is running.

Part-DB now paces itself. Two sliding windows per provider are enforced
in PartInfoRetriever, inside the cache callbacks, which is the single
point every provider request passes and the only place where cache hits
can be told apart from requests that really go out. The counters live in
the shared cache pool, so web requests, console commands and background
jobs all count against the same budget.

Requests are delayed, not rejected - a bulk operation simply takes
longer, which is what it wants. Only when a single request would wait
longer than the configured maximum does it fail, so that a request
started from the user interface cannot hang indefinitely.

The defaults (35 per 10 seconds, 120 per minute, 30 seconds maximum
wait) stay deliberately below what providers permit: the remaining
headroom is what keeps Part-DB and an external script sharing the
account from pushing each other over the real limit. Each window can be
switched off individually by setting it to 0.

Two limitations are worth naming. The counter updates are not
synchronized - symfony/lock is not a dependency - so parallel processes
can slightly under-count, which is why the configured limit should stay
below the provider's own. And providers with an internal cache of their
own consume a slot even when they answer from it, which errs on the safe
side.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.42%. Comparing base (0f0ee60) to head (eb78f4f).

Files with missing lines Patch % Lines
...ces/InfoProviderSystem/InfoProviderRateLimiter.php 92.85% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1536      +/-   ##
============================================
+ Coverage     62.39%   62.42%   +0.03%     
- Complexity     9879     9895      +16     
============================================
  Files           736      738       +2     
  Lines         31779    31829      +50     
============================================
+ Hits          19829    19870      +41     
- Misses        11950    11959       +9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant