fix: improve _is_excluded_line

[CarstenLeue]

Added a 'pragma' not in line fast-path guard at the top of _is_excluded_line.
On any line that doesn't contain the word pragma (virtually every line in a normal repo), all 6 ALLOWLIST_REGEXES searches are skipped entirely and exclude_lines_regex is checked only if set.
The original slow path is reached only when pragma is actually present, preserving 100% of existing detection behaviour.

detect-secrets Performance Analysis

Profile date: Sun Jul 5 21:01:57 2026
Repository scanned: bob4z-shell-proxy
Total runtime: 228.995 seconds (~3 min 49 s)
Total function calls: 118,548,394
Files scanned: 574

---

Executive Summary

Over 69% of all CPU time is consumed by a single operation: running 6 allowlist
regex patterns against every line of every file for every plugin. This is not secret
detection work — it is overhead checking whether a line carries a # pragma: allowlist secret comment. On a real repository where essentially no lines carry that
comment, almost all of that cost is wasted.

---

Call-Tree Overview

main()  229s
└── _perform_scan()  218s
    └── baseline.initialize()  218s
        └── scan_file()  ×574 files  218s
            └── _extract_secrets_from_file()  214s
                └── plugin.analyze()  ×22 plugins  211s      ← 12,628 calls
                    └── _is_excluded_line()  159s            ← 4,737,282 calls
                        └── any(allowlist_regex.search())    ← 29,231,324 calls

---

Time Budget by Function

Rank	Function	File	Self time	Cum. time	Calls	% of total
1	re.Pattern.search	(built-in)	149.3 s	149.3 s	29,231,324	65.2 %
2	re.Pattern.findall	(built-in)	33.5 s	33.5 s	8,615,691	14.6 %
3	getaddrinfo (version check)	(built-in)	8.4 s	8.4 s	1	3.7 %
4	<genexpr> @ base.py:109	base.py	6.2 s	153.3 s	33,160,864	2.7 %
5	_io.open (file opens)	(built-in)	4.2 s	4.2 s	583	1.8 %
6	builtins.any	(built-in)	3.8 s	157.2 s	4,737,771	1.7 %
7	analyze_line	base.py:175	2.4 s	47.2 s	4,737,260	1.1 %
8	analyze	base.py:120	2.3 s	211.1 s	12,628	1.0 %
9	secret_generator (regex)	base.py:425	2.0 s	34.9 s	4,091,271	0.9 %
10	_is_excluded_line	base.py:107	2.0 s	159.1 s	4,737,282	0.9 %
11	analyze_string_content	base.py:410	1.5 s	36.6 s	4,091,270	0.7 %
12	calculate_shannon_entropy	high_entropy_strings.py:73	1.2 s	2.0 s	55,406	0.5 %
—	everything else	—	~8 s	—	—	~3.5 %

---

Bottleneck 1 — Allowlist Regex Overhead (149 s, 65%)

Root cause

_is_excluded_line in detect_secrets/plugins/base.py:107 runs 6 compiled regex
patterns against every line of every file to detect an inline
# pragma: allowlist secret comment:

# detect_secrets/plugins/common/constants.py
ALLOWLIST_REGEXES = [
    re.compile(r)
    for r in [
        r'[ \t]+{} *pragma: ?(allow|white)list[ -]secret.*?{}[ \t]*$'.format(start, end)
        for start, end in (
            ('#', ''),        # python / yaml
            ('//', ''),       # golang
            (r'/\*', ...),    # C
            ("'", ''),        # VB.NET
            ('--', ''),       # SQL
            (r'<!--...', ...) # XML
        )
    ]
]

# detect_secrets/plugins/base.py:107
def _is_excluded_line(self, line):
    return (
        any(
            allowlist_regex.search(line)     # ← up to 6 regex searches per line
            for allowlist_regex in ALLOWLIST_REGEXES
        )
        or (self.exclude_lines_regex and self.exclude_lines_regex.search(line))
    )

Why it is so expensive

• Called once per line per plugin. With 22 plugins and 574 files the call count
  reaches 4.7 million.
• On lines that do not contain a pragma comment (effectively every line in a normal
  repository) all 6 regexes run to completion and return no match.
• This generates 29.2 million re.Pattern.search calls, virtually all of them
  fruitless.
• The any(genexpr) + builtins.any machinery adds a further 33.2 million + 4.7
  million call-overhead frames.

Proportion of total work attributed to this path

Sub-call	Calls	Time
re.Pattern.search (allowlist)	29,231,324	149.3 s
<genexpr> driving the loop	33,160,864	6.2 s
builtins.any	4,737,771	3.8 s
_is_excluded_line self	4,737,282	2.0 s
Total		~161 s (70%)

Fix

A cheap str.__contains__ pre-filter eliminates the regex entirely on every ordinary
line. The word pragma is present in every allowlist comment and absent from
essentially all other lines:

_ALLOWLIST_HINT = 'pragma'

def _is_excluded_line(self, line):
    # Fast path: skip all regex work for the common case
    if _ALLOWLIST_HINT not in line:
        if not self.exclude_lines_regex:
            return False
        return bool(self.exclude_lines_regex.search(line))
    # Slow path: only reached for lines that actually contain 'pragma'
    return (
        any(allowlist_regex.search(line) for allowlist_regex in ALLOWLIST_REGEXES)
        or (self.exclude_lines_regex and self.exclude_lines_regex.search(line))
    )

Expected impact: eliminates ~149 s (65% of total runtime). Estimated post-fix
runtime: ~50–70 s.

---

Bottleneck 2 — Plugin findall for Secret Detection (33 s, 15%)

re.Pattern.findall is called 8.6 million times across all plugins for actual secret
pattern matching (RegexBasedDetector.secret_generator, KeywordDetector,
HighEntropyStringsPlugin). This is legitimate work and cannot be removed, but it can
be reduced:

• Disable plugins not relevant to the repository. Each disabled plugin removes one
  full pass of findall per line per file. With 22 plugins, disabling 5 saves ~23%
  of this cost.
• Raise entropy thresholds (--base64-limit, --hex-limit) to reduce the number
  of candidate strings that proceed to entropy calculation.

---

Bottleneck 3 — Version Check Network Call (8 s, 4%)

A single getaddrinfo call at startup consumes 8.4 s — 100% of which is a DNS/TCP
round-trip for the version check to PyPI. This is unrelated to scanning but adds fixed
overhead to every run.

Fix: pass --no-version-check on every invocation, or set it in the pre-commit
config. Saves ~8–9 s unconditionally.

---

Bottleneck 4 — calculate_shannon_entropy (1.2 s, 0.5%)

Only 55,406 calls were made in this profile, making this negligible for this
repository. It would become relevant on repositories with many YAML/INI files
containing large quoted strings.

---

Findings Not Confirmed by This Profile

The following issues identified in static analysis did not appear as significant
costs in this specific repository:

Issue	Reason not dominant here
4× YAML/INI parse per file per entropy plugin	YAML/INI files are rare in this repo; the fallback path short-circuits quickly
O(charset × len) Shannon entropy	Low call count (55 K) on this repo
No file-level parallelism	Real cost, but masked by the allowlist overhead dominating

---

Summary of Recommended Actions

Priority	Action	Estimated saving	Effort
P0	Add pragma pre-filter to _is_excluded_line	~149 s (65%)	10 lines of code
P1	Pass --no-version-check	~8 s (4%)	CLI flag / config
P2	Disable unused plugins via --no-<plugin>-scan	~5–8 s per plugin	CLI flag
P3	Parallelize file scanning with concurrent.futures	proportional to core count	medium refactor

Applying P0 alone is expected to reduce the observed 229 s runtime to approximately
60–70 s with no change to detection behaviour.

[CarstenLeue]

scan.zip