Repository navigation
Scraper only - #2
Merged
Merged
Conversation
Remove the UI/UX testing product (Studio backend and frontend, sandbox runner, test generator, flow recorder, a11y/visual/console checks, record/test/start commands and their tests). It is preserved on the archive/ui-testing-studio branch for its own repository. Keep the element engine (selector ladder, fingerprints, self-healing) and move the element picker to scrapewizard/engine/picker.js. Drop sqlmodel and pillow, remove the frontend CI and release steps, rewrite the README for the scraper, and add SCRAPER_PLAN.md with competitor research and the improvement plan.
Second research pass covering AutoScraper, Crawl4AI schemas, Scrapling, Spidermon, changedetection.io, mitmproxy2swagger, autopager, recipe-scrapers and others, mapped onto current gaps. Records two confirmed defects: CSV export fails on uneven records and pagination detection emits invalid selectors.
Restructure the plan into background, Part A (user flow) and Part B (how the code works). Part A defines four commands, one mode, a single preview-and-confirm question, files saved in the current folder, and the messages for login, site-change repair and failure cases, then maps each piece to a build phase.
…ss waits
write_csv now uses the union of all record keys instead of the first record's, so a later record with an extra field no longer raises ValueError.
PaginationDetector no longer emits invalid selectors (':contains', unescaped class names). It now recognises rel=next and icon-only links, numbered pages with a marked current page, and load-more buttons, and exposes find_next_url().
Build steps are printed as they happen instead of animating on a timer (about 18 seconds per build).
A recipe names a repeating container and typed fields, each with a ladder of selectors. The runtime fetches over plain HTTP by default (browser as an explicit mode), reads and cleans records, follows pagination, removes duplicates and evaluates checks (minimum records, required-field coverage). HTTP responses without a charset header are decoded using the page's own meta charset.
The builder finds the repeating block on a page, discovers the values each item holds, types and names them, detects pagination and returns a recipe. A recipe is only returned if running it against the same page yields records. It prefers a card over the grid cell or row that wraps it, keeps an item that merely contains a sub-list (tags), drops constant labels, cut-off copies of a title and text that only adds fixed words to another field, avoids generated class names, and merges two routes to the same value into a selector ladder. Example values (likes) choose between several lists on a page.
'scrapewizard <url>' fetches a page over plain HTTP (falling back to a browser only when the page shows no list without JavaScript), builds a recipe with no LLM, previews the data, asks one question (this page or all pages) and saves the data and the recipe in the current folder. --yes, --pages, --all-pages, --like, --format and --out answer everything up front. 'scrapewizard run <recipe>' runs a saved recipe, names the data file after the recipe file, saves nothing when no records are found, and exits non-zero when a check fails. README and SCRAPER_PLAN.md updated with the new flow, a status table and known limits.
A value repeated on every item is kept when it sits in a non-control element with a meaningful class (availability, year). Buttons, links and labels ending in a colon are still dropped. Multi-class field selectors get single-class fallbacks in their ladder, so a state class such as 'instock' changing on one item no longer loses the value.
…te changes Each successful run is remembered in .scrapewizard/<name>.state.json beside the recipe: row count, per-field fill rate, an index of every record and a sample of full records. 'scrapewizard run' now reports "N new, N changed, N removed since last run" when the page limit matches the remembered run. When a run finds no records, or a required or previously well-filled field stops matching, the page is searched again for the data. The recipe's fields are matched to what is found by comparing values with the remembered records. A repair is verified when known items are found again. Without that, it is accepted only if every field matches by name and type. A required field that cannot be found, or a page with no list, is refused and nothing is saved. Repaired recipes keep their old selectors as fallbacks and record the change in a history section. --no-repair turns this off.
Recipes that need a browser now keep one browser open for the whole run instead of starting one per page. Two pagination types grow the page in place: load_more clicks a button, scroll scrolls to the bottom. Both wait until the page has gained real content and stopped changing, rather than trusting "network idle" (reached once per page load) or a height change (a spinner). The builder records a detected "load more" button. When more pages are asked for and the plain page shows no way to continue, the page is checked in a browser for controls or scrolling that only appear once scripts have run.
With --follow, a few item pages are opened while building to learn what they add to the list: labelled rows in tables and definition lists, embedded JSON-LD, the heading, and the main description block. Values the list already holds, and values that are the same on every page, are left out. The fields are stored in the recipe's 'detail' section, so every later run collects them. Detail selectors apply to the whole page and may be CSS, 'jsonld:path.to.key' or 'meta:name'. An item page that cannot be read leaves its fields empty instead of failing the run. Plain HTTP requests now share one client, so connections are reused, and item pages are read four at a time. Reading 40 item pages took under 9 seconds where 20 took 41 before.
--ask "what I want" sends a pruned copy of the page to the configured model, which replies with a recipe (container and fields), not code. --ai lets the model try only when local analysis finds no list, and suggests better column names. Running a recipe never uses AI. The page is untrusted input: the prompt says so, the reply is never executed, a proposed recipe is run against the page and accepted only if it yields at least three rows, invented fields that match nothing are dropped, and names are accepted only if they are plain unique identifiers. Pruning removes scripts, styles, comments, hidden elements and most attributes, shortens long text and caps the page at 14,000 characters. Without a key, --ai carries on with local results and --ask stops with how to set one up. Tested against a stand-in model; no real AI service is called by the tests.
--login opens a visible browser. The user signs in and goes to the page with the data, then presses Enter. The recipe is built from the page they ended on and marked 'login: true'. The session (cookies and local storage) is saved in .scrapewizard/<name>.session.json beside the recipe and reused by 'run'. The recipe itself holds no credentials. The .scrapewizard folder now writes its own .gitignore, so run memory and sign-ins are not committed by accident. A missing session, or a run that finds nothing with a saved session, says that the sign-in is missing or has probably expired and gives the command to sign in again, instead of attempting a repair against a sign-in page. Suggested commands in error messages no longer wrap in narrow terminals.
…features The default install goes from 25 packages to 13. openai and anthropic move to an 'ai' extra, pandas and openpyxl to 'excel', and the test tools with fastapi and uvicorn (used only by the demo app fixture) to 'dev'. aiohttp, python-dotenv and prompt_toolkit were not imported anywhere and are removed. requirements.txt remains the full development set. Verified in a fresh virtual environment with only the core installed: the installed command builds, runs and re-runs a recipe. A missing AI package now says which package to install. The output format, including whether Excel export is available, is checked before anything is fetched. README and SCRAPER_PLAN.md describe pagination in place, item pages, run memory, self-repair, sign-in and optional AI, with the limits that remain.
Tried on 14 real pages the builder had not seen: correct list on 8 (was 4). Hacker News now works without --like, and Wikipedia tables are read in full with column names from the table headings. Choose between nested blocks by which fields each holds. Read records split over two neighbouring elements (dt + dd, title row + details row). Group all same-tag siblings and reach them through the nearest nameable ancestor. Name links after the text they belong to, drop spare links and images, avoid generated and layout class names. Count link-only blocks as one column, weigh small groups and position-only fields less, and let the main content landmark outrank a stray menu class. Also fix the hardware-detection test on Windows and remove two unused imports.
Many tests drive a real browser (JavaScript pages, "load more", infinite scroll, sign-in). The workflow installed the Python package but not the browser itself, so those tests could not pass on GitHub.
…ming Tables - A table whose heading row sits among its rows (no thead) lost every plain text column: the heading row has th cells, so the td cells looked like an optional extra and were dropped. Rows that agree with each other now keep their cells. - A value inside a cell (a link in the first column) takes that column's heading as its name instead of "text". Naming - A class that says "title" or "headline" now outranks plain link text. On arXiv the title column held the paper number; it now holds the title. Speed - Looking for lists asked the selector engine to scan the whole page hundreds of times. One index of the page, built once, answers the same questions by lookup. A 1.7 MB Wikipedia table: 11s -> 1s. GitHub trending: 2.8s -> 0.7s. Results on 14 saved real pages are identical. Tests - 11 new offline tests for page shapes met on live sites: records split over dt/dd, title row + details row, heading rows, links named after their text, card vs its stats line, a list marked "menu" inside the main content, neighbouring cards, few large sections vs a long list, rows with mixed classes.
A list could only be reached from an ancestor with a unique id or class. An old hand-written page names nothing, so it gave "no list found". The body is now the anchor of last resort.
README - New features: table headings as column names, records in two parts, links named after their text. - "How well it works": results on 14 public pages the tool was not built against (10 right, 1 partly, 3 not), and the weak spot (column names). - Known limits rewritten to match: bot protection, frames, generated class names, what has not been verified live. SCRAPER_PLAN.md - Status table brought up to date, with the same results, what they do not show, and the next steps in order of value. - Section 1 marked as the starting point, not the present. learn.md - Rewritten. It described only the older AI builder. It now walks through the recipe path module by module, with the older builder as a short last section. Also: package description no longer says "agentic AI"; a CI comment pointed at a document that is not on this branch.
On sites built from components or utility CSS, the class names say
nothing ("d-inline-block") or are made up by a build tool ("jeApUG").
Columns came out as text, text_2, number, and selectors rested on names
that change at the site's next release.
Selectors
- Use the markers a site adds for its own tests and for search engines:
data-testid, data-test, data-qa, data-cy, itemprop.
- Do not use short mixed-case made-up class names.
- Markers that differ only in a first word ("dundee-card", "london-card")
are one list. BBC News: 47 cards through div[data-testid$="-card"],
where one layout alone gave 17.
- When a larger block holds the chosen items and reads the same fields
the same way, it is the list.
Names, tried in order when an element has no class of its own
- the marker's value (itemprop="programmingLanguage")
- the constant end of its link (/owner/repo/stargazers)
- an icon's label, or the element's accessible label
- the look of the value ("3 hours ago" -> age)
- the words after a number ("9 comments", "12 stars today")
- a label before the value ("Language: Go")
- the class of the element around it
- one long text under a title is the description
- microformat classes lose their prefix (u-author -> author)
GitHub trending, before: title, text, text_2, number, number_2, text_3
after: title, description, programming_language, stargazers, forks,
stars_today
… list
Embedded data
- A page drawn by JavaScript often ships its list as JSON in a script:
__NEXT_DATA__, JSON-LD, or "var data = [...]". When the HTML shows no
list (or only a menu), that data is read instead. No browser, no
selectors. A recipe says so with a container like
"data:props.pageProps.products"; its fields are paths ("price.amount").
- quotes.toscrape.com/js needed a browser before. It now gives all 100
quotes over 10 pages with plain HTTP.
- Columns keep the data's own names. Constants, internal keys, tokens and
whole article bodies are left out. Small settings lists (menus,
language pickers) are not taken for records.
- --like works on it. Self-repair follows the list when its key is renamed.
Frames
- When a page is a shell around a frame from the same site, the list is
read from the frame and the recipe points at the frame's address.
Menus
- A list found only in navigation is no longer returned as the answer
straight away: embedded data and then the browser are tried first, and
if it is still all there is, the message says it looks like the menu.
Commands
- build, list, resume and clean (the older AI builder) still work but are
hidden from --help, which now shows six commands instead of ten.
Widening a list to "all the layouts" also took in picture-only banners that sat among the cards (Real Python: 22 rows, 4 of them empty). The items a larger block adds must now have the record's main value too. Real Python is back to 18 clean rows; BBC News keeps all 47 cards. Docs - README: embedded data, names from the page, lasting selectors, frames; results now 11 of 14 right; command table matches --help; limits redone. - SCRAPER_PLAN.md: status, a table of each gap against other tools and where it stands now, next steps. - learn.md: markers and grouped layouts, the naming order, embedded data, frames, repair of data recipes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.