Speed up parse() ~7x with literal pre-filters (output unchanged) - #33
Open
Malkiz223 wants to merge 1 commit into
Open
Speed up parse() ~7x with literal pre-filters (output unchanged)#33Malkiz223 wants to merge 1 commit into
Malkiz223 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #32.
parse()spent almost all of its time running heavy regexes over the whole log - even where there is nothing to find. The idea is simple: before each regex, do a quick plain-substring check (in/str.find) and only run the regex where a match is actually possible. The regex still decides everything - the check just skips the parts that can't match, so the output stays the same.What changed (
common.pygets a few dicts of marker substrings, the rest is inscrapylogparser.py):re_search_final_match- can take a substring now. Lines that don't contain it are skipped before the regex runs.extract_latest_matches- same for the 9 patterns, and it takes the lastScraped fromstraight from the end (rfind) instead of walking every item in the log.extract_log_categories- if a category (ERROR, WARNING, etc.) isn't in the log, skip it. If it is, find its lines by substring and cut them out, instead of scanning the whole log with a regex per category.extract_datas- the page/item-count regex now runs only on lines that contain" pages (at".extract_shutdown_reason- the SIGTERM regex runs only if the line contains"Received SIG".extract_stats_dumped- the stats block (always at the end) is found with a plain search, and the regex starts from there instead of byte 0.Correctness: the existing tests pass unchanged, and everything
parse()returns matches the original byte-for-byte (checked on real logs and on a synthetic\n/\r\nlog that has every category, SIGTERM, offsite/duplicate, and "gave up retrying"). Passes the project's linter (flake8). These same changes are already running in our production, and everything works correctly.