Fetch/Extract/Batch first-class, plus current logo and brand fix - #24
Open
AnderRV wants to merge 6 commits into
Open
Fetch/Extract/Batch first-class, plus current logo and brand fix#24AnderRV wants to merge 6 commits into
AnderRV wants to merge 6 commits into
Conversation
- fetch()/extract() as primary methods; get() kept as a deprecated alias - New ZenRowsBatchClient (client.batch or standalone) covering the Conveyor open/closed job lifecycle: submit, addTasks, closeJob, list/get/delete job & run, results, task content. Accepts a baseURL override for parity with the Go/Python batch clients. - Fixed 2 real bugs found while testing against the documented API contract: deleteJob/deleteRun threw on the documented 202-empty-body response (only 204 was handled); getTaskContent threw parsing raw scraped HTML as JSON. - rerun()/addTasks() corrected to match the real request shape (status as query param + Idempotency-Key header; lastBatch option). - Coverage: batch.ts + index.ts both 100% (44→46 tests). - Replaced outdated logo SVGs with ZR-FE's current purple wordmark. Known gap, not addressed here: Python's batch client (58 methods — scheduling, webhook CRUD, HMAC rotation, CSV upload, exports, pagination iterators) is far more complete than this 13-method reference. Naming/constructor shape is now reconciled; method-count parity is not, and needs its own scoped follow-up. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
…, exports
Ander's feedback: Python's batch client wasn't "more mature" by
accident — it's real, fully-specified design (immutable resource
handles, jittered-backoff retries, RFC 7807 errors, a client-side
cost estimator with an exact rate card). Ported it faithfully rather
than leaving Node at 13 core methods.
Restructured src/batch.ts into src/batch/{client,resources,schedule,
waiters,estimate,errors,transport,download,types}.ts:
- Resource handles matching Python's exact design: JobRef/JobHandle,
RunRef/RunHandle, ExportRef/ExportHandle, plus CurrentRun and
ScheduleControls sub-facets. Immutable-snapshot semantics preserved
— mutating methods return a fresh handle, never mutate in place.
- Typed schedule builders (At/Rate/Calendar/Daily/Weekly/Monthly)
with the same validation rules as Python (tz-naive datetime
rejection, full-hour times, valid IANA timezones, day/range checks).
- Retry transport matching Python's exact constants: 250ms * 2^attempt
backoff, +/-20% jitter, capped at 10s, retries 429/502/503/504 plus
network errors on idempotent methods (or POST with an Idempotency-
Key), honors Retry-After.
- RFC 7807 error parsing with code/extras, matching BatchAPIError.
- Cost estimation: exact rate card ported (base=1, js=5, premium=10,
js+premium=25, auto=[1,25]), same tier-priority and per-task merge
rules.
- Webhook CRUD, HMAC key rotation, CSV upload (presigned S3 PUT),
results export + streaming download, task history, cursor-based
async-generator pagination (iterJobs/iterRuns/iterResults), bulk/
single downloads with concurrency + the same safety caps
(maxFiles=100k, maxBytesPerFile=50MiB, etc.).
Coverage: 79.48% -> 92.59% (was thin right after the restructure —
closed gaps in download.ts and resources.ts specifically). 116 tests,
all real (error paths, boundary validation, retry-after-honoring,
immutable-handle-returns-fresh-instance behavior) — not padding.
Fixed one brand-prose miss ("ZenRows Batch API" -> "Zenrows Batch
API" in client.ts's docstring) — everything else in this port was
already class/type identifiers, correctly left as ZenRows-cased
per the existing published-export convention.
Known gap, not addressed here: no tqdm-equivalent progress bars
(UX sugar, not API parity). At's Date handling reads local wall-clock
fields since JS has no naive-datetime concept, unlike Python's
tz-naive datetime type.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
extract() (mode defaults to "auto") is a domain-gated open beta - when the target domain isn't enabled yet, the API returns a 402 with code AUTH010. Catch that and retry once with autoparse: true instead of returning the error response, matching the CLI's own extract adapter. Opt out with fallbackToAutoparse: false. Also: Extract/Batch are beta, not invite-gated - confirmed live, no enablement step needed. Fixes a wording overclaim in the README. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
JobRef/RunRef/ExportRef/CurrentRun/ScheduleControls stored their client as a TS protected/private field - real at runtime, but still a plain enumerable property. JSON.stringify(job), console.log(job), and Object.keys(job) all walked into it and dumped the full client (including apiKey) in plaintext. Switched to true JS #private fields, which are invisible to all three. Verified live: none of them expose anything beyond jobId/submitResponse now. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
…e-merge-check # Conflicts: # pnpm-lock.yaml
4 tasks
Targets that need js_render/premium_proxy used to fail extract() with a 400 REQS002 unless the caller passed those manually - Zoopla listing pages are a real example. mode: "auto" makes Zenrows escalate automatically, on both the initial extract attempt and the Autoparse fallback. Opt out with adaptiveStealth: false. Verified live: extract(zoopla) now succeeds with zero extra params, and mode=auto doesn't interfere with AUTH010 detection on domains that are still gated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DTXXBERPGEnYiYP4Mt3FjF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fetch()/get()unchanged in behavior;extract(url, config)added — thin typed wrapper overfetch()with theextractparam set (defaults to"auto").extract()falls back to Autoparse once on a 402AUTH010(domain not yet enabled for the Extract beta), matching the CLI's own extract adapter. Opt out withfallbackToAutoparse: false.client.batch— new module covering the full Conveyor job/run/task lifecycle: submit (open/regular/scheduled), resource handles (JobRef/RunRef/ExportRef+ theirHandlesubclasses), pagination, CSV upload, webhooks, HMAC key rotation, results export, jittered-backoff retries, cost estimation — full parity with the Python SDK's Batch client.JSON.stringify(job)/console.log(job)leaked the raw API key. Switched to true#privatefields; verified live that nothing beyondjobId/submitResponseis exposed now.Test plan
pnpm run build— cleanpnpm exec vitest run— 120/120 passpnpm run check(biome) — cleanbatch.submitOpen()— all verified end to end, plus the key-leak fix re-verified against a realJobRef