Suggestions

[pZetan]

Hello,

I was wondering if you've considered adding additional source lists to your ingestion pipeline. I noticed some domains that seem to be missing but are covered by:
https://github.com/ph00lt0/blocklist/
https://github.com/ShadowWhisperer/BlockLists/

I also noticed a number of dead domains in the current list. Have you considered using PyFunceble or a similar tool to help identify and remove inactive domains?

Just a couple of suggestions, thanks for this project. Made my day when i found it.

[fdezdaniel]

Thanks for the kind words and for the suggestions. Feedback like this is genuinely helpful.

Yes, I’ve considered PyFunceble or similar tools, and I agree they could be useful. The main issue is cost: running active checks daily across the full dataset would likely be too heavy for our current GitHub Actions budget. I'm open to feedback here.

For now, dead domain cleanup is more conservative. I remove clearly dead domains (ENOTFOUND) on smaller lists, and avoid more aggressive removals elsewhere to reduce false positives from temporary DNS issues. Cross-checking between sources also helps stale domains disappear over time as upstream lists refresh.

As for additional source lists, I’m open to adding them, after check overlap and quality first. If you can share a small sample of missing or repeatedly dead domains, I can validate them quickly and use that as direct input.

If these 2 sources were added today, this is the estimated impact based on a local simulation using the current rules:

Source	Total domains in source	Would become candidates	Candidate rate	Net new candidates vs current baseline	Net new rate
ph00lt0/blocklist	18,577	763	4.107%	763	4.107%
ShadowWhisperer/BlockLists	507,266	10,368	2.044%	10,166	2.004%

For ShadowWhisperer specifically:

• New candidate rate: 2.04%
• Filtered before final inclusion: 97.96%

Main filtering reasons:

• 78.57% below minimum source cross-check. This mostly means those domains are unique to that list, or not supported by enough other current sources.
• 6.22% unclassified
• 4.17% below category specific thresholds
• 7.84% already in the current snapshot
• 1.15% regex only, so they do not become final candidates

Overall, the list could still be a useful additional signal, but most entries are already filtered by the current quality gates.

[pZetan]

Thank you for the detailed information.

For dead-domain cleanup, I think we should be able to find a workable approach, as we can tackle the problem from several different angles. I’m also happy to mock up or code something if that would help. Based on my research so far, the following methods and logic are likely to make the largest impact on the datasets:

• Validate the TLD and domain syntax.

• Compare domains against ICANN CZDS and Cloudflare’s Domain Ranking datasets.
  Optional: if a domain is registered or appears in a ranking dataset, it is more likely to be active, although this does not guarantee that it is live.
  https://developers.cloudflare.com/radar/investigate/domain-ranking-datasets/
  https://czds.icann.org/home
  https://github.com/icann/czds-api-client-python

• Compare domains against the AdTidy checkDomains API.
  Optional: if a domain has been active recently, it is likely to still be alive.
  https://github.com/AdguardTeam/AGLint/issues/ 194
  https://github.com/AdguardTeam/DeadDomainsLinter

• Perform DNS lookups and check for NXDOMAIN responses.
  We could also consider using a local resolver such as ZDNS or massdns.
  https://github.com/zmap/zdns
  https://github.com/blechschmidt/massdns

• Parked-domain detection is very optional, as it is more compute-heavy. However, caching the current IP or DNS result would likely be the simplest way to keep ongoing processing low by only reprocessing a domain when the result changes.

Edited: FYI, another option i currently see used is parkedDomain2.
https://github.com/List-KR/parkedDomain2
https://github.com/adguardteam/adguardfilters/issues/208394

• Security categorisation using the Google Safe Browsing subset blocklist.
  https://github.com/elliotwutingfeng/Inversion-DNSBL-Blocklists

• Security categorisation using DNS security lookups.
  Based on my testing, DNS4EU has the best coverage, followed by CIRA. If implemented correctly, a single DNS query could give us both NXDOMAIN filtering and security categorisation.

  If the RCODE is NXDOMAIN, the AUTHORITY section contains the parent-zone SOA, and there is no EDE, then the domain can likely be treated as genuinely nonexistent.

  • 1.1.1.2, Cloudflare Security: blocked as 0.0.0.0, EDE 16 “Censored”
  • 86.54.11.1, DNS4EU Protected: blocked to sinkhole/blocking IPs with TTL 1
  • 9.9.9.9, Quad9 Security: blocked as NXDOMAIN, EDE 17 “Filtered”
  • 149.112.121.20, Canadian Internet Registration Authority Shield: blocked to sinkhole/blocking IPs, EDE 17 with “Malware Blocked”

• Compute optimisation:

  • Option 1: Use a GitHub self-hosted runner with a small, low-cost VPS.
  • Option 2: Use an additional public GitHub repository for dead-domain processing only. This repository would create a dead-domain list that is then consumed by the main repository. Public GitHub repositories include free runner compute for standard GitHub-hosted runners.
  • Option 3: Use a local cache database that processes all new records and a subset of older records each day, spreading the cleanup of old records over a full month. This would ensure that no dead domains remain in the list for longer than one month.

[pZetan]

Regarding the additional source lists:

78.57% below minimum source cross-check. This mostly means those domains are unique to that list, or not supported by enough other current sources.

I am a bit concerned about the initial results you provided. In many cases, the most valuable additions are the unique sources. If the new source lists primarily contain records that already exist in other lists, their value is significantly reduced. I can understand excluding these from the Standard option, but my expectation was that the Extended option would accommodate broader use cases and include more unique sources.

Could you please clarify the intended purpose of the Extended option and explain the reasoning behind the current logic?

6.22% unclassified

I suspect that most of these records would fall under the Security category, and I believe the DNS lookup logic I previously provided would address the majority of them.

4.17% below category-specific thresholds

What is the current handling process for records that fall below these thresholds?

My initial thought would be to apply a layered approach:
Security validation checks.
Automated classification using ML/AI (e.g., ChatGPT).
Manual review for any remaining unresolved records.

I would be interested to understand your current process and whether a similar workflow is already being considered.

[fdezdaniel]

Thanks for the detailed follow up, lots of good ideas here.

First, some context that explains the rest: ProtoConsent lists are purpose-based (ads, analytics, personalization, third parties, advanced tracking, security), not general "block everything" lists. A domain is added when it can be tied to a clear purpose, so the goal isn't to re-ship whole upstream lists. If you want the full raw coverage of ph00lt0 or ShadowWhisperer, the best setup is to subscribe to those directly alongside ProtoConsent, they're built for that and this repo isn't trying to copy it.

On Standard vs Extended: the difference is practical, not "filtered vs everything". These lists also feed the browser extension, where domain count has a real size/memory cost (Security alone is 250K+). Standard stays small and safe for extension users; Extended is the broader option for people using the lists directly in their own tools (uBlock, AdGuard, Pi-hole). So unique domains aren't dropped for being unique, they just need a clear purpose to enter a purpose-based list.

Lists: I added both. ShadowWhisperer turned out more useful than my first simulation suggested, because it's categorized per file (Ads, Tracking, Malware, etc.), so it gives a real category vote, not just a presence signal. I mapped those to the matching purposes and skipped the ones that don't fit a consent model (adult, gambling, URL shorteners, etc.). ph00lt0 is in too. On unique domains and Extended: I'd frame it differently. A domain only one source lists, with no corroboration anywhere, is also where most false positives come from. Extended is broader than Standard, but it's not just every list merged together. People still use it to block automatically, so it has to stay reliable, that's why the cross-check (a domain needs more than one source) stays in place.

On the DNS idea: I like it, and the realistic version here is your Option 3. Rather than resolving the whole list every run (too much for free runners), keep a small cache of each domain's last DNS result and date, and each day check only the new domains plus a slice of the old ones (say 1/30), covering the full list over a month. Removals stay conservative (NXDOMAIN/ENOTFOUND), transient failures are kept and retried. I'm not committing to a timeline, I want to look into it properly first, but that's the direction. If you do spot specific domains that are dead or missing, that's the most useful thing you can send, it goes straight into validation. Thanks again for the input.