Skip to content

fedcode-next: Code pipeline and models to continuously collect various URLs and references to articles, other advisories #1724

New issue
New issue
Closed
Closed
fedcode-next: Code pipeline and models to continuously collect various URLs and references to articles, other advisories#1724
Assignees
ziadhany

Description

@pombredanne
pombredanne
opened on Jan 5, 2025

The references are often vanishing from the web. We should collect them and ideally have a mechanism to archive a snapshot. This could be using archive.org and dow not mean we must store the content ourselves

Activity

  1. pombredanne
    added this to 00-AboutCodePlanneron Jul 22, 2025
  2. TG1999
    assigned
    ziadhany
    on Mar 13, 2026
  3. ziadhany
    moved this to Needs prep in 00-AboutCodePlanneron Mar 16, 2026
  4. ziadhany
    moved this from Needs prep to In progress in 00-AboutCodePlanneron Mar 20, 2026

  5. ziadhany commented on Mar 23, 2026

    @ziadhany
    ziadhany
    on Mar 23, 2026
    Member

    I think we should use web.archive.org to store archive URLs via their "Save Page Now" endpoint https://web.archive.org/save/<url>

    https://github.com/akamhy/waybackpy/blob/master/waybackpy/save_api.py.
    Since our URLs are stored in AdvisoryReference, we could add a new field archive_url to that model.

    One major constraint is the rate limit of 15 requests per minute. With 250,000 URLs to process, the initial run would take about 11.5 days:

    250,000 / (15 * 60 * 24) = 11.57 days

    This operational problem is tracked separately in:

  6. TG1999
    assigned
    keshav-space
    on Apr 9, 2026
  7. pombredanne
    unassigned
    keshav-space
    on May 14, 2026

  8. ziadhany commented on May 15, 2026

    @ziadhany
    ziadhany
    on May 15, 2026
    Member

    This is done now. We have a pipeline for archiving snapshots of reference URLs.

    See:

    Improving data using archive_improver_pipeline
INFO 2026-05-15 01:16:28.106691 UTC Pipeline [ArchiveImproverPipeline] starting
INFO 2026-05-15 01:16:28.106839 UTC Step [archive_urls] starting
INFO 2026-05-15 01:16:39.902555 UTC Searching for archive URL for this Reference URL: http://0day.2600.ir/exploits/3903
INFO 2026-05-15 01:17:12.952817 UTC Wayback Machine permission denied for 'http://0day.2600.ir/exploits/3903'.
INFO 2026-05-15 01:17:12.967435 UTC URL unreachable or returned no archive url: http://0day.2600.ir/exploits/3903
INFO 2026-05-15 01:17:12.967818 UTC Searching for archive URL for this Reference URL: http://0day5.com/archives/1346/
INFO 2026-05-15 01:17:45.315843 UTC Found Archived Reference URL: https://web.archive.org/web/20250315143614/http://0day5.com/archives/1346/
INFO 2026-05-15 01:17:45.326061 UTC Searching for archive URL for this Reference URL: http://0day5.com/archives/1542/
INFO 2026-05-15 01:18:17.674561 UTC Found Archived Reference URL: https://web.archive.org/web/20221129111515/http://0day5.com/archives/1542/
INFO 2026-05-15 01:18:17.682321 UTC Searching for archive URL for this Reference URL: http://0day.today/exploits/28864
INFO 2026-05-15 01:18:53.631962 UTC Wayback Machine permission denied for 'http://0day.today/exploits/28864'.
INFO 2026-05-15 01:18:53.639927 UTC URL unreachable or returned no archive url: http://0day.today/exploits/28864
INFO 2026-05-15 01:18:53.640075 UTC Searching for archive URL for this Reference URL: http://0dd.zone/2018/04/23/UMLet-XXE/
INFO 2026-05-15 01:19:25.536742 UTC Found Archived Reference URL: https://web.archive.org/web/20250425155342/https://0dd.zone/2018/04/23/UMLet-XXE/
INFO 2026-05-15 01:19:25.548668 UTC Searching for archive URL for this Reference URL: http://0nto.wordpress.com/2012/02/13/sths-v2-web-portal-2-2-sql-injection-vulnerabilty/
INFO 2026-05-15 01:19:57.536570 UTC URL unreachable or returned no archive url: http://0nto.wordpress.com/2012/02/13/sths-v2-web-portal-2-2-sql-injection-vulnerabilty/
INFO 2026-05-15 01:19:57.536942 UTC Searching for archive URL for this Reference URL: http://0pointer.de/cgi-bin/viewcvs.cgi/%2Acheckout%2A/trunk/docs/NEWS?root=avahi
INFO 2026-05-15 01:20:28.991002 UTC URL unreachable or returned no archive url: http://0pointer.de/cgi-bin/viewcvs.cgi/%2Acheckout%2A/trunk/docs/NEWS?root=avahi
INFO 2026-05-15 01:20:28.991468 UTC Searching for archive URL for this Reference URL: http://0x000000.com/index.php?i=409
INFO 2026-05-15 01:21:00.100585 UTC Wayback Machine permission denied for 'http://0x000000.com/index.php?i=409'.
INFO 2026-05-15 01:21:00.105413 UTC URL unreachable or returned no archive url: http://0x000000.com/index.php?i=409
INFO 2026-05-15 01:21:00.105655 UTC Searching for archive URL for this Reference URL: http://0x1byte.blogspot.com/2011/04/php-phar-extension-heap-overflow.html
INFO 2026-05-15 01:21:32.931998 UTC URL unreachable or returned no archive url: http://0x1byte.blogspot.com/2011/04/php-phar-extension-heap-overflow.html
INFO 2026-05-15 01:21:32.932231 UTC Searching for archive URL for this Reference URL: http://0x6a616d6573.blogspot.com/2011/07/phpmyadmin-fud.html
INFO 2026-05-15 01:22:04.564075 UTC URL unreachable or returned no archive url: http://0x6a616d6573.blogspot.com/2011/07/phpmyadmin-fud.html
INFO 2026-05-15 01:22:04.564360 UTC Searching for archive URL for this Reference URL: http://0x90.com.ar/Advisory/20071031.txt
INFO 2026-05-15 01:22:36.097418 UTC URL unreachable or returned no archive url: http://0x90.com.ar/Advisory/20071031.txt
INFO 2026-05-15 01:22:36.097552 UTC Searching for archive URL for this Reference URL: http://0x90.com.ar/Advisory/20071106.txt
INFO 2026-05-15 01:23:07.544668 UTC Found Archived Reference URL: https://web.archive.org/web/20071117152012/http://0x90.com.ar/Advisory/20071106.txt
INFO 2026-05-15 01:23:07.548540 UTC Searching for archive URL for this Reference URL: http://0x90.com.ar/Advisory/20080321.txt
INFO 2026-05-15 01:23:39.346769 UTC URL unreachable or returned no archive url: http://0x90.com.ar/Advisory/20080321.txt
INFO 2026-05-15 01:23:39.347023 UTC Searching for archive URL for this Reference URL: http://0x90.com.ar/Advisory/20080920-2.txt
INFO 2026-05-15 01:24:10.865208 UTC URL unreachable or returned no archive url: http://0x90.com.ar/Advisory/20080920-2.txt
INFO 2026-05-15 01:24:10.865473 UTC Searching for archive URL for this Reference URL: http://0x90.eu/ff_tls_poc.html
INFO 2026-05-15 01:24:43.712773 UTC Found Archived Reference URL: https://web.archive.org/web/20260323221404/https://0x90.eu/ff_tls_poc.html
INFO 2026-05-15 01:24:43.721987 UTC Searching for archive URL for this Reference URL: http://0xahmed.ninja/
INFO 2026-05-15 01:25:15.849024 UTC Found Archived Reference URL: https://web.archive.org/web/20180419201136/http://0xahmed.ninja/
INFO 2026-05-15 01:25:15.850990 UTC Searching for archive URL for this Reference URL: http://0xhuesca.com/cve-2019-18655.html
INFO 2026-05-15 01:25:47.612900 UTC URL unreachable or returned no archive url: http://0xhuesca.com/cve-2019-18655.html
INFO 2026-05-15 01:25:47.613212 UTC Searching for archive URL for this Reference URL: http://10degres.net/cve-2016-11018-image-gallery-sql-injection/
INFO 2026-05-15 01:26:22.645435 UTC Found Archived Reference URL: https://web.archive.org/web/20250422000337/https://10degres.net/cves/cve-2016-11018/
INFO 2026-05-15 01:26:22.654457 UTC Searching for archive URL for this Reference URL: http://118.89.230.52/about/details.docx
INFO 2026-05-15 01:27:03.882979 UTC URL unreachable or returned no archive url: http://118.89.230.52/about/details.docx
INFO 2026-05-15 01:27:03.883293 UTC Searching for archive URL for this Reference URL: http://124.71.147.32:8082
INFO 2026-05-15 01:27:37.222723 UTC Found Archived Reference URL: https://web.archive.org/web/20240229051203/http://124.71.147.32:8082/
INFO 2026-05-15 01:27:37.233313 UTC Searching for archive URL for this Reference URL: http://124.71.147.32:8082/IceCMS2.html```
  9. ziadhany
    closed this as completedon May 15, 2026
  10. github-project-automation
    moved this from In progress to Done in 00-AboutCodePlanneron May 15, 2026
  11. hesa
    moved this from Done to Reviewed in 00-AboutCodePlanneron Jun 22, 2026
  12. pombredanne
    moved this from Reviewed to Validated in 00-AboutCodePlanneron Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions