Skip to content

Speed up parse() ~7x with literal pre-filters (output unchanged) - #33

Open
Malkiz223 wants to merge 1 commit into
my8100:masterfrom
Malkiz223:speedup-parse
Open

Speed up parse() ~7x with literal pre-filters (output unchanged)#33
Malkiz223 wants to merge 1 commit into
my8100:masterfrom
Malkiz223:speedup-parse

Conversation

@Malkiz223

Copy link
Copy Markdown

Fixes #32.

parse() spent almost all of its time running heavy regexes over the whole log - even where there is nothing to find. The idea is simple: before each regex, do a quick plain-substring check (in / str.find) and only run the regex where a match is actually possible. The regex still decides everything - the check just skips the parts that can't match, so the output stays the same.

log before after speedup
142 KB 13.7 ms 1.8 ms 7.6×
1.1 MB 108 ms 15 ms 7.3×
10 MB 896 ms 126 ms 7.1×
1.1 MB (many error/retry/redirect lines) 98 ms 14 ms 7.2×

What changed (common.py gets a few dicts of marker substrings, the rest is in scrapylogparser.py):

  • re_search_final_match - can take a substring now. Lines that don't contain it are skipped before the regex runs.
  • extract_latest_matches - same for the 9 patterns, and it takes the last Scraped from straight from the end (rfind) instead of walking every item in the log.
  • extract_log_categories - if a category (ERROR, WARNING, etc.) isn't in the log, skip it. If it is, find its lines by substring and cut them out, instead of scanning the whole log with a regex per category.
  • extract_datas - the page/item-count regex now runs only on lines that contain " pages (at".
  • extract_shutdown_reason - the SIGTERM regex runs only if the line contains "Received SIG".
  • extract_stats_dumped - the stats block (always at the end) is found with a plain search, and the regex starts from there instead of byte 0.

Correctness: the existing tests pass unchanged, and everything parse() returns matches the original byte-for-byte (checked on real logs and on a synthetic \n/\r\n log that has every category, SIGTERM, offsite/duplicate, and "gave up retrying"). Passes the project's linter (flake8). These same changes are already running in our production, and everything works correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

parse() runs several whole-log regex scans - ~7× slower than needed

1 participant