Repository navigation
Performance Optimization for eth_getLogs/eth_getFilterLogs with Large-scale Query Parameters #6343
Description
Background
eth_getLogs and eth_getFilterLogs are interfaces for querying smart contract event logs. To quickly find blocks containing target events, these interfaces use Bloom filters to index data. The partialMatch method searches the Section-Bloom database using these filters. However, when query conditions contain a large number of topics or addresses (e.g. querying TRC20 transfers for multiple addresses or querying multiple contracts for a single address), it causes many duplicate database queries and slows down the response. While version 4.8.0 added limits on topic counts and block ranges to prevent long request times, these restrictions force users to split their queries into multiple smaller requests.
We have optimized the partialMatch method to avoid duplicate database queries and improve interface performance. This optimization allows nodes to support larger queries, reducing the number of requests users need to make.
Rationale
Why should this feature exist?
- Large queries are slow due to duplicate database operations. Users have to split their queries into smaller ones to work around current limits, which is not user-friendly.
What are the use-cases?
- NFT exchanges & DEX: Monitor Transfer events for multiple tokens
- Data analysis platforms: Batch query historical events for analysis and visualization
Specification
Let's take eth_getLogs as an example to see how event log queries locate target blocks:
- Converts query conditions (address, topics) into Bloom filter positions
- For each section (2048 blocks), call partialMatch to find possible blocks
- The partialMatch method queries Section-Bloom database for matching blocks
Due to Bloom filter properties, duplicate queries occur: with 2048 total bits and 3 bits per condition, when condition count reaches 683 (683 * 3 > 2048), bit collisions are inevitable, leading to duplicate database queries.
Test results using random from-address from USDT transfer records:
| Test Set | BitIndex Duplication Rate |
|---|---|
| 10 topics | 0% |
| 100 topics | 7% |
| 500 topics | 28.8% |
| 1000 topics | 48.4% |
Optimization of partialMatch method:
- Preprocess bitIndexes[][] to extract unique bitIndexes
- Query database only once for each unique bitIndex
- Reassemble final BitSet using query results
This reduces IO operations and improves query speed.
Test Specification
- Verifies result consistency before and after optimization
- Compares performance with same inputs
Scope Of Impact
Affected Modules:
- JSON-RPC interface layer (EthApi)
Affected interfaces:
- eth_getLogs
- eth_getFilterLogs
Implementation
Do you have ideas regarding the implementation of this feature?
Y
Are you willing to implement this feature?
Y
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
jwrct commentedon Jun 12, 2025
@0xbigapple After this optimization, what is the max query that the node can support? And how is its performance?
0xbigapple commentedon Jun 12, 2025
Hi, @jwrct The max query depends on the node machine's configuration, and I haven't specifically tested this yet. Based on my current data, comparing the execution times before and after optimization for the
partialMatchmethod, the response time shows an almost direct positive correlation with the bit index repetition rate. I will update here when I have more detailed test results.waynercheung commentedon Jun 12, 2025
@0xbigapple It's a good idea to extract unique bitIndexes before fetching it from DB, but I also have questions as belows:
partialMatch, it usesFutureto fetch data from DB in parallel, in your optimization, is there any change about it?BitIndex Duplication Rate, but for normal request, does it has some negative impact? After all, the count of subtopics for most requests are less than 20, so it is also important to ensure the performance of normal requests.Besides, looking forward to your test results about the
execution timesyou mentioned above.waynercheung commentedon Jun 12, 2025
@0xbigapple Hi, about this
Duplication Rate, can you explain more detail about how do you calculating the rate? or what does the rate mean?0xbigapple commentedon Jun 12, 2025
Hi, @waynercheung Thanks for your discussion
Let me address your concerns:
partialMatchmethod still maintains its original parallel execution using Futures with a default thread pool size of 5, which remains unchanged in this optimization.partialMatch execution times: 10 topics test was performed locally and the results might be slightly affected by other running programs on the machine.
0xbigapple commentedon Jun 12, 2025
The Duplication rate shows the percentage of repeated bit indexes. For example: if 1000 topics generate 3000 bitIndexes in total but only 1000 are unique, the duplication rate is 1 -1000/3000 = 66.67%.
waynercheung commentedon Jun 12, 2025
Got it, thanks.
0xbigapple commentedon Jun 13, 2025
@waynercheung @jwrct The following results(partialMatch execution times) were obtained from local testing on my Mac. The average value with 5 rounds(100 times per round).
jwrct commentedon Jun 13, 2025
@0xbigapple Great!
What are the limits on the number of topics and block ranges in version 4.8.0? Are there plans to relax these limits with this optimization?
0xbigapple commentedon Jun 13, 2025
@jwrct The default value is followed, but node can customize it in the configuration file, we have no plans to modify the default value at this time.
2 remaining items
0xbigapple commentedon Jun 20, 2025
@waynercheung I conducted a test of the eth_getLogs interface in the Nile environment (configured with maxBlockRange = 100000000 and maxSubTopics = 5000). The test range was fromBlock = 57,803,200 to toBlock = 57,853,200, covering a total of 50,000 blocks. The address used was JST (TF17BgPaZYbz8oxbjhriubPDsA7ArKoLX3). For transfer transactions, I randomly selected 10, 100, 500, 1,000, and 2,000 toAddresses as query topics. The average response times are as follows:
As we discussed above, when the number of topics is small, there is little difference in response times before and after the optimization. However, when the number of topics is large, this optimization brings significant performance improvements.
Benson0224 commentedon Jul 25, 2025
Status Update: This issue has been scheduled for implementation in the next release, tentatively V4.8.1. The resolution plan is as follows
However, all talks about this issue will be welcome. Please continue the conversation here if you have relevant topics to contribute. Thanks!
At last, please follow V4.8.1 status at Issue #6342 and watch this issue (via the Subscribe button) to track progress if needed. We value your involvement in TRON's development!