Pace the requests Part-DB sends to the info providers - #1536
Open
killecaptron wants to merge 1 commit into
Open
Conversation
Info providers limit requests per account, usually in short windows - TrustedParts, for example, allows 50 requests per 10 seconds and 150 per minute - and many of them bill per request. Part-DB happily exceeds that: a bulk import or an update of many parts walks through them as fast as the network allows, with nothing in between. Worse, the limit belongs to the account, not to Part-DB. An external script maintaining parts through the API uses the same credentials, so its requests and Part-DB's add up while neither side can see the other's count. Coordinating the two by hand is the only remedy today, and it fails the moment somebody clicks "update from info provider" while a job is running. Part-DB now paces itself. Two sliding windows per provider are enforced in PartInfoRetriever, inside the cache callbacks, which is the single point every provider request passes and the only place where cache hits can be told apart from requests that really go out. The counters live in the shared cache pool, so web requests, console commands and background jobs all count against the same budget. Requests are delayed, not rejected - a bulk operation simply takes longer, which is what it wants. Only when a single request would wait longer than the configured maximum does it fail, so that a request started from the user interface cannot hang indefinitely. The defaults (35 per 10 seconds, 120 per minute, 30 seconds maximum wait) stay deliberately below what providers permit: the remaining headroom is what keeps Part-DB and an external script sharing the account from pushing each other over the real limit. Each window can be switched off individually by setting it to 0. Two limitations are worth naming. The counter updates are not synchronized - symfony/lock is not a dependency - so parallel processes can slightly under-count, which is why the configured limit should stay below the provider's own. And providers with an internal cache of their own consume a slot even when they answer from it, which errs on the safe side. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1536 +/- ##
============================================
+ Coverage 62.39% 62.42% +0.03%
- Complexity 9879 9895 +16
============================================
Files 736 738 +2
Lines 31779 31829 +50
============================================
+ Hits 19829 19870 +41
- Misses 11950 11959 +9 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Info providers limit requests per account, usually in short windows - TrustedParts, for example,
allows 50 requests per 10 seconds and 150 per minute - and many of them bill per request.
Part-DB can easily produce bursts above that: a bulk import, or an update of many parts, walks
through them as fast as the network allows with nothing in between.
Worse, the limit belongs to the account, not to Part-DB. A script maintaining parts through the
API uses the same credentials, so its requests and Part-DB's add up while neither side can see
the other's count. Coordinating the two by hand is the only remedy today, and it fails the moment
somebody clicks "update from info provider" while a job is running.
Part-DB now paces itself. Two sliding windows per provider are enforced in PartInfoRetriever,
inside the cache callbacks - the single point every provider request passes, and the only place
where a cache hit can be told apart from a request that really goes out. The counters live in the
shared cache pool, so web requests, console commands and background jobs all count against the
same budget.
Requests are delayed, not rejected: a bulk operation simply takes longer, which is what it wants.
Only if a single request would wait longer than the configured maximum does it fail, with HTTP 429
and a Retry-After header naming the seconds until the next free slot, so an automated caller can
tell "too fast, come back shortly" apart from a real error.
The defaults (35 per 10 seconds, 120 per minute, 30 seconds maximum wait) stay deliberately below
what providers permit: the remaining headroom is what keeps Part-DB and an external script sharing
the account from pushing each other over the real limit. Each window can be switched off by
setting it to 0.
Two limitations worth naming. The counter updates are not synchronized - symfony/lock is not a
dependency - so parallel processes can slightly under-count, which is why the configured limit
should stay below the provider's own. And a provider with an internal cache of its own consumes a
slot even when it answers from that cache, which errs on the safe side.
This is the follow-up I offered in #1532.