diff --git a/.agents/skills/scrapingbee-cli/SKILL.md b/.agents/skills/scrapingbee-cli/SKILL.md index 0e21779..336214f 100644 --- a/.agents/skills/scrapingbee-cli/SKILL.md +++ b/.agents/skills/scrapingbee-cli/SKILL.md @@ -39,7 +39,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -81,7 +83,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/.agents/skills/scrapingbee-cli/rules/install.md b/.agents/skills/scrapingbee-cli/rules/install.md index 2030d4d..9853935 100644 --- a/.agents/skills/scrapingbee-cli/rules/install.md +++ b/.agents/skills/scrapingbee-cli/rules/install.md @@ -72,6 +72,16 @@ scrapingbee logout If `scrapingbee` is not found after install: 1. Activate the environment where `pip install scrapingbee-cli` was run (e.g. `source .venv/bin/activate`). Pip puts the `scrapingbee` script in that env’s bin (e.g. `.venv/bin`), so it’s on PATH only when that env is active. -2. Reinstall: `pip install --force-reinstall scrapingbee-cli`. +2. If pip warned that the script went to `~/.local/bin` (a user install, common in sandboxed shells), that directory is not on PATH in minimal shells: run `export PATH="$HOME/.local/bin:$PATH"`, or call `~/.local/bin/scrapingbee` by full path. +3. Reinstall: `pip install --force-reinstall scrapingbee-cli`. + +## Network allowlist (Claude Cowork and other locked-down sandboxes) + +If every command fails with `Cannot connect to host app.scrapingbee.com:443 ... Temporary failure in name resolution`, the shell can reach PyPI but not the ScrapingBee API: the host restricts outbound traffic to an allowlist. Two ways through: + +1. An org Owner adds `app.scrapingbee.com` to the allowed hosts (in Claude Cowork: Admin settings → Capabilities). The CLI then works unchanged. +2. Use the ScrapingBee MCP connector (`https://mcp.scrapingbee.com/mcp`, `Authorization: Bearer `) for single-page and search work — connectors are not subject to the sandbox allowlist. Crawls, batches, file output and scheduling still need the CLI, so prefer option 1 when those matter. + +Do not fall back to the host's built-in fetch tool as a substitute: it is subject to the same allowlist and cannot render JavaScript or pass anti-bot protection. **See also:** [rules/security.md](rules/security.md) (credits, output safety, shell safety). diff --git a/.github/skills/scrapingbee-cli/SKILL.md b/.github/skills/scrapingbee-cli/SKILL.md index 0e21779..336214f 100644 --- a/.github/skills/scrapingbee-cli/SKILL.md +++ b/.github/skills/scrapingbee-cli/SKILL.md @@ -39,7 +39,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -81,7 +83,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/.github/skills/scrapingbee-cli/rules/install.md b/.github/skills/scrapingbee-cli/rules/install.md index 2030d4d..9853935 100644 --- a/.github/skills/scrapingbee-cli/rules/install.md +++ b/.github/skills/scrapingbee-cli/rules/install.md @@ -72,6 +72,16 @@ scrapingbee logout If `scrapingbee` is not found after install: 1. Activate the environment where `pip install scrapingbee-cli` was run (e.g. `source .venv/bin/activate`). Pip puts the `scrapingbee` script in that env’s bin (e.g. `.venv/bin`), so it’s on PATH only when that env is active. -2. Reinstall: `pip install --force-reinstall scrapingbee-cli`. +2. If pip warned that the script went to `~/.local/bin` (a user install, common in sandboxed shells), that directory is not on PATH in minimal shells: run `export PATH="$HOME/.local/bin:$PATH"`, or call `~/.local/bin/scrapingbee` by full path. +3. Reinstall: `pip install --force-reinstall scrapingbee-cli`. + +## Network allowlist (Claude Cowork and other locked-down sandboxes) + +If every command fails with `Cannot connect to host app.scrapingbee.com:443 ... Temporary failure in name resolution`, the shell can reach PyPI but not the ScrapingBee API: the host restricts outbound traffic to an allowlist. Two ways through: + +1. An org Owner adds `app.scrapingbee.com` to the allowed hosts (in Claude Cowork: Admin settings → Capabilities). The CLI then works unchanged. +2. Use the ScrapingBee MCP connector (`https://mcp.scrapingbee.com/mcp`, `Authorization: Bearer `) for single-page and search work — connectors are not subject to the sandbox allowlist. Crawls, batches, file output and scheduling still need the CLI, so prefer option 1 when those matter. + +Do not fall back to the host's built-in fetch tool as a substitute: it is subject to the same allowlist and cannot render JavaScript or pass anti-bot protection. **See also:** [rules/security.md](rules/security.md) (credits, output safety, shell safety). diff --git a/.kiro/skills/scrapingbee-cli/SKILL.md b/.kiro/skills/scrapingbee-cli/SKILL.md index 0e21779..336214f 100644 --- a/.kiro/skills/scrapingbee-cli/SKILL.md +++ b/.kiro/skills/scrapingbee-cli/SKILL.md @@ -39,7 +39,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -81,7 +83,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/.kiro/skills/scrapingbee-cli/rules/install.md b/.kiro/skills/scrapingbee-cli/rules/install.md index 2030d4d..9853935 100644 --- a/.kiro/skills/scrapingbee-cli/rules/install.md +++ b/.kiro/skills/scrapingbee-cli/rules/install.md @@ -72,6 +72,16 @@ scrapingbee logout If `scrapingbee` is not found after install: 1. Activate the environment where `pip install scrapingbee-cli` was run (e.g. `source .venv/bin/activate`). Pip puts the `scrapingbee` script in that env’s bin (e.g. `.venv/bin`), so it’s on PATH only when that env is active. -2. Reinstall: `pip install --force-reinstall scrapingbee-cli`. +2. If pip warned that the script went to `~/.local/bin` (a user install, common in sandboxed shells), that directory is not on PATH in minimal shells: run `export PATH="$HOME/.local/bin:$PATH"`, or call `~/.local/bin/scrapingbee` by full path. +3. Reinstall: `pip install --force-reinstall scrapingbee-cli`. + +## Network allowlist (Claude Cowork and other locked-down sandboxes) + +If every command fails with `Cannot connect to host app.scrapingbee.com:443 ... Temporary failure in name resolution`, the shell can reach PyPI but not the ScrapingBee API: the host restricts outbound traffic to an allowlist. Two ways through: + +1. An org Owner adds `app.scrapingbee.com` to the allowed hosts (in Claude Cowork: Admin settings → Capabilities). The CLI then works unchanged. +2. Use the ScrapingBee MCP connector (`https://mcp.scrapingbee.com/mcp`, `Authorization: Bearer `) for single-page and search work — connectors are not subject to the sandbox allowlist. Crawls, batches, file output and scheduling still need the CLI, so prefer option 1 when those matter. + +Do not fall back to the host's built-in fetch tool as a substitute: it is subject to the same allowlist and cannot render JavaScript or pass anti-bot protection. **See also:** [rules/security.md](rules/security.md) (credits, output safety, shell safety). diff --git a/.opencode/skills/scrapingbee-cli/SKILL.md b/.opencode/skills/scrapingbee-cli/SKILL.md index 0e21779..336214f 100644 --- a/.opencode/skills/scrapingbee-cli/SKILL.md +++ b/.opencode/skills/scrapingbee-cli/SKILL.md @@ -39,7 +39,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -81,7 +83,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/.opencode/skills/scrapingbee-cli/rules/install.md b/.opencode/skills/scrapingbee-cli/rules/install.md index 2030d4d..9853935 100644 --- a/.opencode/skills/scrapingbee-cli/rules/install.md +++ b/.opencode/skills/scrapingbee-cli/rules/install.md @@ -72,6 +72,16 @@ scrapingbee logout If `scrapingbee` is not found after install: 1. Activate the environment where `pip install scrapingbee-cli` was run (e.g. `source .venv/bin/activate`). Pip puts the `scrapingbee` script in that env’s bin (e.g. `.venv/bin`), so it’s on PATH only when that env is active. -2. Reinstall: `pip install --force-reinstall scrapingbee-cli`. +2. If pip warned that the script went to `~/.local/bin` (a user install, common in sandboxed shells), that directory is not on PATH in minimal shells: run `export PATH="$HOME/.local/bin:$PATH"`, or call `~/.local/bin/scrapingbee` by full path. +3. Reinstall: `pip install --force-reinstall scrapingbee-cli`. + +## Network allowlist (Claude Cowork and other locked-down sandboxes) + +If every command fails with `Cannot connect to host app.scrapingbee.com:443 ... Temporary failure in name resolution`, the shell can reach PyPI but not the ScrapingBee API: the host restricts outbound traffic to an allowlist. Two ways through: + +1. An org Owner adds `app.scrapingbee.com` to the allowed hosts (in Claude Cowork: Admin settings → Capabilities). The CLI then works unchanged. +2. Use the ScrapingBee MCP connector (`https://mcp.scrapingbee.com/mcp`, `Authorization: Bearer `) for single-page and search work — connectors are not subject to the sandbox allowlist. Crawls, batches, file output and scheduling still need the CLI, so prefer option 1 when those matter. + +Do not fall back to the host's built-in fetch tool as a substitute: it is subject to the same allowlist and cannot render JavaScript or pass anti-bot protection. **See also:** [rules/security.md](rules/security.md) (credits, output safety, shell safety). diff --git a/AGENTS.md b/AGENTS.md index 2ab55dc..fcfd56a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -30,7 +30,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -72,7 +74,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/CHANGELOG.md b/CHANGELOG.md index 99ad2d4..db2e552 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,18 @@ All notable changes to this project are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **Plugin metadata:** `plugins/scrapingbee-cli/` now ships its own `README.md` and `LICENSE`, and the plugin manifest declares `license`, `homepage` and `repository`, as plugin directories require. +- **Cursor and ChatGPT / Codex manifests:** the same plugin folder now carries `.cursor-plugin/plugin.json` (with a `SCRAPINGBEE_API_KEY` variable and `mcp.json` that connects the ScrapingBee MCP server) and `.codex-plugin/plugin.json`, plus logo assets, so one folder can be submitted to each marketplace. + +### Fixed + +- **Skill install path:** the skill now tells agents that `uv tool install` places `scrapingbee` in `~/.local/bin` and how to run it when that directory is not on `PATH`, instead of giving up on the CLI. +- **Sandbox network allowlists:** documented what a "Temporary failure in name resolution" from the CLI means in a locked-down sandbox and that the MCP connector is the way around it. + ## [1.6.0] - 2026-09-11 ### Added diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 44ab6db..2b5971e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -28,8 +28,12 @@ Keep these in sync whenever you bump the version: - `pyproject.toml` → `[project] version` - `src/scrapingbee_cli/__init__.py` → `__version__` - `.claude-plugin/marketplace.json` → `plugins[0].version` -- `.claude-plugin/plugin.json` → `version` -- `skills/scrapingbee-cli/SKILL.md` → frontmatter `version` +- `plugins/scrapingbee-cli/.claude-plugin/plugin.json` → `version` (Claude Code) +- `plugins/scrapingbee-cli/.cursor-plugin/plugin.json` → `version` (Cursor) +- `plugins/scrapingbee-cli/.codex-plugin/plugin.json` → `version` (ChatGPT / Codex) +- `.agents/skills/scrapingbee-cli/SKILL.md` and `scrapingbee-cli-guard/SKILL.md` → frontmatter `version`, then `scripts/sync-skills.sh` + +Plugin directories re-review a plugin when its version changes, so bump all three manifests in the same release PR. The CI `check-version` job enforces that `pyproject.toml` and `__init__.py` stay in sync. diff --git a/plugins/scrapingbee-cli/.claude-plugin/plugin.json b/plugins/scrapingbee-cli/.claude-plugin/plugin.json index b29af8c..cff0fc1 100644 --- a/plugins/scrapingbee-cli/.claude-plugin/plugin.json +++ b/plugins/scrapingbee-cli/.claude-plugin/plugin.json @@ -2,6 +2,9 @@ "name": "scrapingbee", "description": "Fetch and read any web page, search the web, crawl a site, or pull structured data out of pages \u2014 whenever a task needs content from a website. Handles JavaScript, CAPTCHAs and anti-bot blocking that curl, requests and headless browsers fail on. Plain-English field extraction with --ai-extract-rules, --smart-extract to return only the part you need, Google/Amazon/Walmart/YouTube/ChatGPT JSON endpoints, batch, crawl and cron scheduling.", "version": "1.6.0", + "license": "MIT", + "homepage": "https://www.scrapingbee.com/documentation/cli/", + "repository": "https://github.com/ScrapingBee/scrapingbee-cli", "author": { "name": "ScrapingBee" }, diff --git a/plugins/scrapingbee-cli/.codex-plugin/plugin.json b/plugins/scrapingbee-cli/.codex-plugin/plugin.json new file mode 100644 index 0000000..ba0c430 --- /dev/null +++ b/plugins/scrapingbee-cli/.codex-plugin/plugin.json @@ -0,0 +1,31 @@ +{ + "name": "scrapingbee", + "version": "1.6.0", + "description": "Fetch and read any web page, search the web, crawl a site, or pull structured data out of pages — whenever a task needs content from a website. Handles JavaScript, CAPTCHAs and anti-bot blocking that curl, requests and headless browsers fail on. Plain-English field extraction with --ai-extract-rules, --smart-extract to return only the part you need, Google/Amazon/Walmart/YouTube/ChatGPT JSON endpoints, batch, crawl and cron scheduling.", + "author": { + "name": "ScrapingBee" + }, + "homepage": "https://www.scrapingbee.com/documentation/cli/", + "repository": "https://github.com/ScrapingBee/scrapingbee-cli", + "license": "MIT", + "skills": "./skills", + "interface": { + "displayName": "ScrapingBee", + "shortDescription": "Read, search and crawl the web", + "longDescription": "ScrapingBee fetches any web page as clean markdown, text or JSON — including JavaScript-rendered pages and sites behind CAPTCHAs or anti-bot blocking that plain HTTP tools fail on. This plugin teaches the agent when and how to use the ScrapingBee CLI: scrape a single page or a CSV of URLs, crawl a whole site into files for a RAG index, search Google, Amazon, Walmart or YouTube and get structured JSON back, extract fields described in plain English, and schedule recurring checks — while keeping credit spend to the minimum that works. Requires a ScrapingBee account and API key.", + "category": "Developer Tools", + "capabilities": ["Read"], + "composerIcon": "./assets/icon.png", + "logo": "./assets/icon.png", + "developerName": "ScrapingBee", + "websiteURL": "https://www.scrapingbee.com/documentation/cli/", + "privacyPolicyURL": "https://www.scrapingbee.com/privacy-policy/", + "termsOfServiceURL": "https://www.scrapingbee.com/terms-and-conditions/", + "defaultPrompt": [ + "Fetch this pricing page as markdown and save it to a file", + "Crawl this docs site and save every guide page for my RAG index", + "Get the current price and rating for this Amazon ASIN", + "I have a CSV of product URLs — refresh the price column in place" + ] + } +} diff --git a/plugins/scrapingbee-cli/.cursor-plugin/plugin.json b/plugins/scrapingbee-cli/.cursor-plugin/plugin.json new file mode 100644 index 0000000..d1d14df --- /dev/null +++ b/plugins/scrapingbee-cli/.cursor-plugin/plugin.json @@ -0,0 +1,24 @@ +{ + "name": "scrapingbee", + "description": "Fetch and read any web page, search the web, crawl a site, or pull structured data out of pages — whenever a task needs content from a website. Handles JavaScript, CAPTCHAs and anti-bot blocking that curl, requests and headless browsers fail on. Plain-English field extraction with --ai-extract-rules, --smart-extract to return only the part you need, Google/Amazon/Walmart/YouTube/ChatGPT JSON endpoints, batch, crawl and cron scheduling.", + "version": "1.6.0", + "author": { + "name": "ScrapingBee" + }, + "homepage": "https://www.scrapingbee.com/documentation/cli/", + "repository": "https://github.com/ScrapingBee/scrapingbee-cli", + "license": "MIT", + "keywords": ["web-scraping", "scraping", "crawl", "search", "serp", "amazon", "youtube", "data-extraction"], + "logo": "./assets/logo.svg", + "variables": { + "type": "object", + "properties": { + "SCRAPINGBEE_API_KEY": { + "type": "string", + "title": "ScrapingBee API key", + "description": "Your API key from https://app.scrapingbee.com/account/manage/api_key. Used to authenticate the ScrapingBee MCP server; the CLI reads it from the same environment variable." + } + }, + "required": ["SCRAPINGBEE_API_KEY"] + } +} diff --git a/plugins/scrapingbee-cli/LICENSE b/plugins/scrapingbee-cli/LICENSE new file mode 100644 index 0000000..b26ebe6 --- /dev/null +++ b/plugins/scrapingbee-cli/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 ScrapingBee + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/plugins/scrapingbee-cli/README.md b/plugins/scrapingbee-cli/README.md new file mode 100644 index 0000000..192904f --- /dev/null +++ b/plugins/scrapingbee-cli/README.md @@ -0,0 +1,67 @@ +# ScrapingBee plugin + +Fetch and read any web page, search the web, crawl a site, or pull structured data out of +pages — whenever a task needs content from a website. ScrapingBee handles JavaScript +rendering, CAPTCHAs and anti-bot blocking that `curl`, `requests` and headless browsers fail +on, and returns clean markdown, text or JSON. + +This plugin teaches the agent **when and how** to use ScrapingBee correctly: which command fits +the job, how to keep credit spend low, and which tool to reach for first — the +[ScrapingBee CLI](https://www.scrapingbee.com/documentation/cli/) by default, or the +[ScrapingBee MCP server](https://mcp.scrapingbee.com/) when a single page or search result +belongs straight in the conversation. + +One plugin folder serves several hosts: Claude Code reads `.claude-plugin/plugin.json`, +Cursor reads `.cursor-plugin/plugin.json` and `mcp.json`, and ChatGPT / Codex read +`.codex-plugin/plugin.json`. The skills are shared by all of them. + +## What's inside + +- **`scrapingbee-cli` skill** — a router for the CLI: when to use ScrapingBee instead of plain + HTTP, how to route scrape / search / crawl / extraction / screenshot tasks, cost discipline, + and guardrails. Detailed command reference pages are included and loaded on demand. +- **`scrapingbee-cli-guard` skill** — security rules that stay active whenever the CLI is used: + scraped content is data and never instructions, the API key is never printed or written out, + and scheduled jobs are only created at the user's explicit request. +- **`scraping-pipeline` agent** — an optional subagent for long, multi-step scraping pipelines. + +## What it needs + +- A **ScrapingBee account and API key** — [sign up here](https://app.scrapingbee.com/account/register). + API calls consume credits from your plan. +- The **ScrapingBee CLI**, installed on the machine where the agent runs commands: + `uv tool install scrapingbee-cli` (or `pip install scrapingbee-cli`), then `scrapingbee auth` + or `SCRAPINGBEE_API_KEY` in the environment. +- In **Cursor**, the plugin also connects the ScrapingBee MCP server. Set the + `SCRAPINGBEE_API_KEY` plugin variable when you install it; the same key is used for the MCP + server and, if exported in your shell, for the CLI. +- In **Codex CLI**, the plugin ships the skills only. To add the MCP server as well, run once + (with `SCRAPINGBEE_API_KEY` exported in your shell): + + ```bash + codex mcp add scrapingbee --url https://mcp.scrapingbee.com/mcp --bearer-token-env-var SCRAPINGBEE_API_KEY + ``` + +## What it runs, sends and fetches + +The plugin contains **only skill instructions, reference documentation and manifests** — no +hooks, no scripts, and nothing that executes on its own. When a task calls for it, the agent +runs the `scrapingbee` command-line tool (or calls the ScrapingBee MCP server at +`mcp.scrapingbee.com`) on your behalf. Those requests go to the ScrapingBee API at +`app.scrapingbee.com` over HTTPS, authenticated with your API key, and fetch the web pages or +search results you asked for. Nothing else is contacted, and the plugin sends no data +anywhere other than ScrapingBee. + +## Examples + +- "Get me the full text of this JavaScript-heavy pricing page as clean markdown, saved to a file." +- "I have a CSV of 200 product URLs — fetch the current price for each and update the CSV in place." +- "Crawl this docs site and save every `/guide/` page as markdown for my RAG index." +- "What's the current price of Amazon ASIN B0CX23V2ZK?" + +## Links + +- CLI documentation: https://www.scrapingbee.com/documentation/cli/ +- API documentation: https://www.scrapingbee.com/documentation/ +- Source: https://github.com/ScrapingBee/scrapingbee-cli +- License: MIT diff --git a/plugins/scrapingbee-cli/assets/icon.png b/plugins/scrapingbee-cli/assets/icon.png new file mode 100644 index 0000000..6c63d1f Binary files /dev/null and b/plugins/scrapingbee-cli/assets/icon.png differ diff --git a/plugins/scrapingbee-cli/assets/logo.svg b/plugins/scrapingbee-cli/assets/logo.svg new file mode 100644 index 0000000..f2ebc05 --- /dev/null +++ b/plugins/scrapingbee-cli/assets/logo.svg @@ -0,0 +1,72 @@ + + + + + + + + + + + + diff --git a/plugins/scrapingbee-cli/mcp.json b/plugins/scrapingbee-cli/mcp.json new file mode 100644 index 0000000..ab6be0f --- /dev/null +++ b/plugins/scrapingbee-cli/mcp.json @@ -0,0 +1,10 @@ +{ + "mcpServers": { + "scrapingbee": { + "url": "https://mcp.scrapingbee.com/mcp", + "headers": { + "Authorization": "Bearer ${SCRAPINGBEE_API_KEY}" + } + } + } +} diff --git a/plugins/scrapingbee-cli/skills/scrapingbee-cli/SKILL.md b/plugins/scrapingbee-cli/skills/scrapingbee-cli/SKILL.md index 0e21779..336214f 100644 --- a/plugins/scrapingbee-cli/skills/scrapingbee-cli/SKILL.md +++ b/plugins/scrapingbee-cli/skills/scrapingbee-cli/SKILL.md @@ -39,7 +39,9 @@ schedule a recurring job, or keep a large page out of your context window. Reach for the **MCP** in two cases: the host already has it connected *and* the task is a single page or search whose result belongs in the conversation anyway; or you cannot run a CLI at all — -no shell, no filesystem, or installation is blocked. +no shell, no filesystem, installation is blocked, or the shell cannot reach +`app.scrapingbee.com` (a "Temporary failure in name resolution" from the CLI means a network +allowlist is in the way — the MCP connector is not subject to it). MCP endpoint: `https://mcp.scrapingbee.com/mcp` — remote, Streamable HTTP, authenticated with an `Authorization: Bearer ` header. Host-specific connection steps belong to the @@ -81,7 +83,11 @@ for. 1. **Install:** `uv tool install scrapingbee-cli` (recommended) or `pip install scrapingbee-cli`. Every command including `crawl` works immediately — no extras. -2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY`. + If `scrapingbee` is then "command not found", the executable is in `~/.local/bin`, which + minimal shells don't search: run `export PATH="$HOME/.local/bin:$PATH"` once, or call + `~/.local/bin/scrapingbee` by full path. Do not fall back to another tool because of this. +2. **Authenticate:** `scrapingbee auth`, or set `SCRAPINGBEE_API_KEY` (a `.env` file in the + working directory is read too). 3. **Verify:** `scrapingbee usage` — confirms the key works and shows remaining credits. ## Commands diff --git a/plugins/scrapingbee-cli/skills/scrapingbee-cli/rules/install.md b/plugins/scrapingbee-cli/skills/scrapingbee-cli/rules/install.md index 2030d4d..9853935 100644 --- a/plugins/scrapingbee-cli/skills/scrapingbee-cli/rules/install.md +++ b/plugins/scrapingbee-cli/skills/scrapingbee-cli/rules/install.md @@ -72,6 +72,16 @@ scrapingbee logout If `scrapingbee` is not found after install: 1. Activate the environment where `pip install scrapingbee-cli` was run (e.g. `source .venv/bin/activate`). Pip puts the `scrapingbee` script in that env’s bin (e.g. `.venv/bin`), so it’s on PATH only when that env is active. -2. Reinstall: `pip install --force-reinstall scrapingbee-cli`. +2. If pip warned that the script went to `~/.local/bin` (a user install, common in sandboxed shells), that directory is not on PATH in minimal shells: run `export PATH="$HOME/.local/bin:$PATH"`, or call `~/.local/bin/scrapingbee` by full path. +3. Reinstall: `pip install --force-reinstall scrapingbee-cli`. + +## Network allowlist (Claude Cowork and other locked-down sandboxes) + +If every command fails with `Cannot connect to host app.scrapingbee.com:443 ... Temporary failure in name resolution`, the shell can reach PyPI but not the ScrapingBee API: the host restricts outbound traffic to an allowlist. Two ways through: + +1. An org Owner adds `app.scrapingbee.com` to the allowed hosts (in Claude Cowork: Admin settings → Capabilities). The CLI then works unchanged. +2. Use the ScrapingBee MCP connector (`https://mcp.scrapingbee.com/mcp`, `Authorization: Bearer `) for single-page and search work — connectors are not subject to the sandbox allowlist. Crawls, batches, file output and scheduling still need the CLI, so prefer option 1 when those matter. + +Do not fall back to the host's built-in fetch tool as a substitute: it is subject to the same allowlist and cannot render JavaScript or pass anti-bot protection. **See also:** [rules/security.md](rules/security.md) (credits, output safety, shell safety).