Sync web-scraping-with-scrapy-and-mongodb with the updated tutorial

[realpython-bot]

Syncs web-scraping-with-scrapy-and-mongodb/ with the maintenance update of the tutorial.

• Draft admin: https://realpython.com/pbjt/blog/post/2309/change/
• Maintenance card: https://trello.com/c/1a6qfpEm

What changed

books/books/spiders/book.py — def start_requests(self): → async def start(self):

Scrapy deprecated Spider.start_requests() in 2.13.0 and removed it in 2.16.0. On current Scrapy the method is never called, so the spider fell back to the default start() and the errback=self.log_error the tutorial wires onto the initial request was silently dropped. Confirmed on 2.18.0 with a scratch spider: an old-style start_requests() request comes back with errback: None, while the new async def start() yields https://books.toscrape.com/ with errback: log_error, callback: parse.

requirements.txt — re-pinned (== on every line) to match the tutorial's updated dependencies field: Scrapy==2.18.0, pymongo==4.17.0, resolved on Python 3.14. Transitive pins were regenerated from a clean resolve, so pyasn1, pyasn1_modules, setuptools, and six drop out of the tree and brotli comes in. No constraints.txt exists in this folder, so there was nothing to fold in.

No other file in the folder needed a change: items.py, pipelines.py, settings.py, tests/test_book.py, tests/sample.html, scrapy.cfg, and README.md all already match the article's current code blocks. The USER_AGENT / DOWNLOADER_MIDDLEWARES / RETRY_* snippets the article refreshed this cycle are illustrative examples in the prose and are not part of the committed settings.py.

How this was verified

Everything below was actually run, in a fresh Python 3.14.6 venv installed from the new requirements.txt:

• pip install -r requirements.txt resolves and installs cleanly on Python 3.14.
• scrapy version → Scrapy 2.18.0; the spider module imports cleanly.
• scrapy list → book.
• scrapy check → Ran 3 contracts ... OK (the @url / @returns items 20 20 / @returns request 1 50 / @scrapes url title price contracts, against the live Books to Scrape).
• python -m unittest discover → Ran 3 tests ... OK.
• Full end-to-end crawl with a real database. I downloaded and started a local MongoDB 8.0.4 (mongod --dbpath /tmp/mongodata, 127.0.0.1:27017) in the CI VM, since the distro has no mongodb-server package. scrapy crawl book completed against the live site and books_db.books held 1000 documents, each with the SHA-256 _id, url, title, and price — e.g. A Light in the Attic / £51.77. A second scrapy crawl book logged exactly 1000 Dropped: Duplicate item found warnings to book_scraper.log, so the dedup path in MongoPipeline.process_item() behaves as the tutorial describes.
• Repo gates on the pinned ruff==0.14.1 from the root requirements.txt: uvx ruff@0.14.1 format --check web-scraping-with-scrapy-and-mongodb → 9 files already formatted; uvx ruff@0.14.1 check web-scraping-with-scrapy-and-mongodb → All checks passed!.

What was NOT verified / noted for a human

• MongoDB version parity with the article. The tutorial's mongod --version and mongosh sample output still show MongoDB 7.0.12 (a known LOW finding left for a human on the card). I ran 8.0.4 here, and I did not install mongosh, so the article's sample output blocks are still unrefreshed. Nothing in this PR depends on that.
• New deprecation warnings on Scrapy 2.18.0, left alone on purpose. The crawl now emits three ScrapyDeprecationWarnings: MongoPipeline.open_spider(), .close_spider(), and .process_item() "requires a spider argument, this is deprecated and the argument will not be passed in future Scrapy versions." The tutorial's own code blocks still use the (self, item, spider) signatures, so I did not change the pipeline here — the repo must match the published article. This is worth filing as a deprecated-api finding on the next research pass so the article and this folder can move together.
• No proxy/user-agent/anti-scraping behavior was exercised; those remain prose-only examples.

Needs a human merge.

🤖 Generated with Claude Code