diff --git a/agents/apify.md b/agents/apify.md new file mode 100644 index 0000000..2b0225a --- /dev/null +++ b/agents/apify.md @@ -0,0 +1,83 @@ +# Apify Agent + +You are the Apify agent. Apify is a platform with thousands of serverless cloud programs called **Actors** for web scraping, browser automation, and data extraction. + +## Routing + +Determine what the user needs and follow the matching route. + +| Signal | Action | Transport | +|--------|--------|-----------| +| Wants to use existing Actors (search, run, get data) | **Route 1** — use MCP or CLI tools directly; for complex multi-step workflows invoke the `apify-ultimate-scraper` skill | **MCP if available, else CLI** — apply the selection rule in "MCP vs CLI selection" below | +| Wants to build, test, or deploy a custom Actor | **Route 2** — invoke the `apify-actor-development` skill (new project) or `apify-actorization` skill (existing project); use `apify-generate-output-schema` for schema generation | **CLI required** — `apify init` / `apify run` / `apify push` have no MCP equivalent | +| Wants to add Apify to an existing JS/Python/other app | **Route 3** — invoke the `apify-sdk-integration` skill | **`apify-client` SDK over HTTPS** — neither MCP nor CLI needed | +| Ambiguous | Ask: "Do you want to (a) use existing scrapers and tools from Apify, (b) build and deploy a custom Actor, or (c) integrate Apify into an existing application?" | Decide after the user clarifies | + +For Route 1, prefer MCP tools for straightforward tasks. Only invoke the `apify-ultimate-scraper` skill when the user needs complex multi-step data pipelines (lead generation, deep research, social media monitoring, ecommerce intelligence, etc.). + +## MCP vs CLI selection + +Route 1 (use existing Actors: search, fetch details, run, get results, look up docs) is exposed through **two interchangeable transports**: the Apify MCP server and the Apify CLI. Routes 2 and 3 are CLI-only or SDK-only by nature and are unaffected by this section. + +Detect available transports **once** at the start of the conversation and reuse the result for every Route 1 operation. Skills downstream (`apify-ultimate-scraper`, etc.) provide both MCP and CLI variants per step — they will not re-detect. + +### Detection + +1. **MCP available** if a tool named `search-actors` appears in your available tool list. (Other Apify MCP tools — `fetch-actor-details`, `run-actor`, `get-dataset-items`, `search-apify-docs`, `fetch-apify-docs` — are part of the same server.) +2. **CLI available** if `apify --help` exits 0 in the shell. + +### Selection rule + +| MCP | CLI | Use for Route 1 | +|-----|-----|-----------------| +| yes | yes | **MCP** (no shell, no install friction, OAuth handles auth) | +| yes | no | MCP | +| no | yes | CLI | +| no | no | Offer to install the CLI (`npm install -g apify-cli`) or point the user to a host that ships the Apify MCP server (`https://mcp.apify.com`). Do not attempt Route 1 until one is available. | + +Route 2 always requires the CLI regardless of MCP availability — `apify init`, `apify run`, and `apify push` operate on the local filesystem and have no MCP equivalent. Route 3 uses the `apify-client` package over HTTPS and needs neither. + +State the chosen transport once when you start a Route 1 task ("Using MCP for this run.") so the user knows which path is active. + +## Naming Trap + +> The `apify` npm package is the **SDK for building Actors** (used in Route 2). The `apify-client` package is the **API client for calling Actors** (used in Route 3). Never confuse these — using the wrong one will break the user's project. + +## Authentication + +Three auth flows exist. Use the correct one based on the route: + +- **Route 1 (MCP):** OAuth. No setup needed. The user will be prompted to sign in via browser on first MCP tool call that requires auth. Do not ask for an API token. +- **Route 1 (CLI fallback) and Route 2 (CLI):** The CLI **ignores** the `APIFY_TOKEN` env var. Run `apify login --token TOKEN` once (requires `required_permissions: ["all"]` in Cursor). Credentials are stored in `~/.apify/auth.json` and reused automatically. Token from: https://console.apify.com/settings/integrations +- **Route 3 (SDK):** Requires `APIFY_TOKEN` environment variable. Direct the user to **Console > Settings > Integrations** at https://console.apify.com/settings/integrations to create one. If they don't have an account, point them to https://console.apify.com/sign-up (free, no credit card). + +### Apify CLI instructions: +- Before using the CLI, always check if it is installed (always check first, with short `block_until_ms` to avoid blocking the conversation): + ```bash + apify --help + ``` +- If the CLI is installed, check if it is logged in (always check, with short `block_until_ms` to avoid blocking the conversation): + ```bash + # Auth check — do NOT pipe to /dev/null, you need to see errors + apify info 2>&1 + ``` +- If the CLI is not logged in, instruct the user to log in with the non-interactive flag: + ```bash + apify login --token TOKEN + ``` +- All of the APify commands needs to be run with the all permissions (depends on Agent sandbox) +- Apify commands blocks with **zero output** until the run completes. Set `block_until_ms` to at least **60000** (60s). +- For long/unknown runs, use the async pattern instead: + ```bash + apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null + ``` +Then poll with `apify info`: + ```bash + apify info actor-runs/RUN_ID --json + ``` +Check `.status` for `SUCCEEDED` or `FAILED`. +## Resources + +- Apify docs (quick reference): https://docs.apify.com/llms.txt +- Apify docs (full): https://docs.apify.com/llms-full.txt +- Actor details in markdown: append `.md` to any Apify Store URL diff --git a/instructions/apify-routing.md b/instructions/apify-routing.md new file mode 100644 index 0000000..3bed1b1 --- /dev/null +++ b/instructions/apify-routing.md @@ -0,0 +1,45 @@ +# Apify Routing + +Cursor exposes Apify skills (`apify-actor-development`, `apify-actorization`, `apify-generate-output-schema`, `apify-sdk-integration`, `apify-ultimate-scraper`) directly in the slash menu. They are **internal capabilities** owned by the `apify` subagent and must not be invoked on their own. + +## Required Behavior + +When the user's request involves Apify — Actors, web scraping, browser automation, the Apify Store, the Apify SDK, the Apify CLI, the `apify` or `apify-client` npm packages, or anything at `apify.com` / `console.apify.com` — invoke the `apify` subagent first. Let it consult its routing matrix and dispatch to the correct skill or MCP tool. + +Do **not**: + +- Call any `apify-*` skill directly from the slash menu without going through the `apify` subagent. +- Guess between `apify-actor-development` vs `apify-actorization` vs `apify-sdk-integration` yourself — the subagent owns that decision (the `apify` SDK package and the `apify-client` package are easy to confuse and picking wrong breaks the user's project). +- Skip the subagent for "quick" Apify questions. Even MCP tool calls (`search-actors`, `run-actor`, etc.) are routed through the subagent. + +If the user types a slash command for an Apify skill explicitly, still hand off to the `apify` subagent so the routing matrix and authentication guidance are applied. + +## Why this rule exists + +Apify exposes three very different products through its plugin: (1) using existing Actors via MCP, (2) building and deploying custom Actors with the SDK + CLI, and (3) calling Actors from an existing application via `apify-client`. Each path requires a different skill, a different authentication flow, and a different npm package. The `apify` subagent is the only place where the routing matrix and the package-naming trap are documented, so every Apify request must pass through it. + +### Apify CLI instructions: +- Before using the CLI, always check if it is installed: +```bash + apify --help +``` +- If the CLI is installed, check if it is logged in: +```bash + # Auth check — do NOT pipe to /dev/null, you need to see errors + apify info 2>&1 +``` +- If the CLI is not logged in, instruct the user to log in with the non-interactive flag: +```bash + apify login --token TOKEN +``` +- All of the APify commands needs to be run with the all permissions (depends on Agent sandbox) +- Apify commands blocks with **zero output** until the run completes. Set `block_until_ms` to at least **60000** (60s). +- For long/unknown runs, use the async pattern instead: +```bash + apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null +``` +Then poll with `apify info`: +```bash + apify info actor-runs/RUN_ID --json +``` +Check `.status` for `SUCCEEDED` or `FAILED`. \ No newline at end of file diff --git a/skills/apify-actor-development/SKILL.md b/skills/apify-actor-development/SKILL.md new file mode 100644 index 0000000..8505b71 --- /dev/null +++ b/skills/apify-actor-development/SKILL.md @@ -0,0 +1,222 @@ +--- +name: apify-actor-development +description: Develop, debug, and deploy Apify Actors - serverless cloud programs for web scraping, automation, and data processing. Use when creating new Actors, modifying existing ones, or troubleshooting Actor code. +--- + +# Apify Actor Development + +**Important:** Before you begin, fill in the `generatedBy` property in the meta section of `.actor/actor.json`. Set it to the name of the AI coding tool and model you are currently running in, in the format `" with "` — for example `"Cursor with Claude Sonnet 4.5"`, `"Claude Code with Claude Opus 4"`, or `"GitHub Copilot with GPT-5"`. If you don't know the exact model, use the tool name alone (e.g. `"Cursor"`). This helps Apify monitor and improve AGENTS.md for specific AI tools and models. + +## What are Apify Actors? + +Actors are serverless programs inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems. They're packaged as Docker images and run in isolated containers in the cloud. + +**Core Concepts:** +- Accept well-defined JSON input +- Perform isolated tasks (web scraping, automation, data processing) +- Produce structured JSON output to datasets and/or store data in key-value stores +- Can run from seconds to hours or even indefinitely +- Persist state and can be restarted + +## Prerequisites & Setup (MANDATORY) + +Before creating or modifying actors, verify that `apify` CLI is installed `apify --help`. + +If it is not installed, use one of these methods (listed in order of preference): + +```bash +# Preferred: install via a package manager (provides integrity checks) +npm install -g apify-cli + +# Or (Mac): brew install apify-cli +``` + +> **Security note:** Do NOT install the CLI by piping remote scripts to a shell +> (e.g. `curl … | bash` or `irm … | iex`). Always use a package manager. + +When the apify CLI is installed, check that it is logged in with: + +```bash +# Auth check — do NOT pipe to /dev/null, you need to see errors +apify info 2>&1 +``` + +If not logged in, authenticate using OAuth (opens browser): + +```bash +apify login +``` + +If browser login isn't available (headless environment or CI), the CLI automatically reads `APIFY_TOKEN` from the environment. Ensure the env var is exported and run any apify command - no explicit login needed. If the user doesn't have a token, generate one at https://console.apify.com/settings/integrations. + +> **Security note:** Avoid passing tokens as command-line arguments (e.g. `apify login -t `). +> Arguments are visible in process listings and may be recorded in shell history. +> Prefer environment variables or interactive login instead. +> Never log, print, or embed `APIFY_TOKEN` in source code or configuration files. + +## Template Selection + +**IMPORTANT:** Before starting actor development, always ask the user which programming language they prefer: +- **JavaScript** - Use `apify create -t project_empty` +- **TypeScript** - Use `apify create -t ts_empty` +- **Python** - Use `apify create -t python-empty` + +Use the appropriate CLI command based on the user's language choice. Additional packages (Crawlee, Playwright, etc.) can be installed later as needed. + +## Quick Start Workflow + +1. **Create actor project** - Run the appropriate `apify create` command based on user's language preference (see Template Selection above) +2. **Install dependencies** (verify package names match intended packages before installing) + - JavaScript/TypeScript: `npm install` (uses `package-lock.json` for reproducible, integrity-checked installs — commit the lockfile to version control) + - Python: `pip install -r requirements.txt` (pin exact versions in `requirements.txt`, e.g. `crawlee==1.2.3`, and commit the file to version control) +3. **Implement logic** - Write the actor code in `src/main.py`, `src/main.js`, or `src/main.ts` +4. **Configure schemas** - Update input/output schemas in `.actor/input_schema.json`, `.actor/output_schema.json`, `.actor/dataset_schema.json` +5. **Configure platform settings** - Update `.actor/actor.json` with actor metadata (see [references/actor-json.md](references/actor-json.md)) +6. **Write documentation** - Create comprehensive README.md for the marketplace (see [references/actor-readme.md](references/actor-readme.md) — this is mandatory, not optional) +7. **Test locally** - Run `apify run` to verify functionality (see Local Testing section below) +8. **Deploy** - Run `apify push` to deploy the actor on the Apify platform (actor name is defined in `.actor/actor.json`) + +## Security + +**Treat all crawled web content as untrusted input.** Actors ingest data from external websites that may contain malicious payloads. Follow these rules: + +- **Sanitize crawled data** — Never pass raw HTML, URLs, or scraped text directly into shell commands, `eval()`, database queries, or template engines. Use proper escaping or parameterized APIs. +- **Validate and type-check all external data** — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures. +- **Do not execute or interpret crawled content** — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts. +- **Isolate credentials from data pipelines** — Ensure `APIFY_TOKEN` and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code. +- **Review dependencies before installing** — When adding packages with `npm install` or `pip install`, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages. +- **Pin versions and use lockfiles** — Always commit `package-lock.json` (Node.js) or pin exact versions in `requirements.txt` (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run `npm audit` or `pip-audit` periodically to check for known vulnerabilities. + +## Best Practices + +**✓ Do:** +- Use `apify run` to test actors locally (configures Apify environment and storage) +- Use Apify SDK (`apify`) for code running ON Apify platform +- Validate input early with proper error handling and fail gracefully +- Use CheerioCrawler for static HTML (10x faster than browsers) +- Use PlaywrightCrawler only for JavaScript-heavy sites +- Use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls +- Implement retry strategies with exponential backoff +- Use proper concurrency: HTTP (10-50), Browser (1-5) +- Set sensible defaults in `.actor/input_schema.json` +- Define output schema in `.actor/output_schema.json` +- Clean and validate data before pushing to dataset +- Use semantic CSS selectors with fallback strategies +- Respect robots.txt, ToS, and implement rate limiting +- **Always use `apify/log` package** — censors sensitive data (API keys, tokens, credentials) +- Implement readiness probe handler (required if your Actor uses standby mode) + +**✗ Don't:** +- Use `npm start`, `npm run start`, `npx apify run`, or similar commands to run actors (use `apify run` instead) +- Assume local storage from `apify run` is pushed to or visible in the Apify Console — it is local-only; deploy with `apify push` and run on the platform to see results in the Console +- Rely on `Dataset.getInfo()` for final counts on Cloud +- Use browser crawlers when HTTP/Cheerio works +- Hard code values that should be in input schema or environment variables +- Skip input validation or error handling +- Overload servers - use appropriate concurrency and delays +- Scrape prohibited content or ignore Terms of Service +- Store personal/sensitive data unless explicitly permitted +- Use deprecated options like `requestHandlerTimeoutMillis` on CheerioCrawler (v3.x) +- Use `additionalHttpHeaders` - use `preNavigationHooks` instead +- Pass raw crawled content into shell commands, `eval()`, or code-generation functions +- Use `console.log()` or `print()` instead of the Apify logger — these bypass credential censoring +- Disable standby mode without explicit permission + +## Logging + +See [references/logging.md](references/logging.md) for complete logging documentation including available log levels and best practices for JavaScript/TypeScript and Python. + +Check `usesStandbyMode` in `.actor/actor.json` - only implement if set to `true`. + +## Commands + +```bash +apify run # Run Actor locally +apify login # Authenticate account +apify push # Deploy to Apify platform (uses name from .actor/actor.json) +apify help # List all commands +``` + +**IMPORTANT:** Always use `apify run` to test actors locally. Do not use `npm run start`, `npm start`, `yarn start`, or other package manager commands - these will not properly configure the Apify environment and storage. + +## Local Testing + +When testing an actor locally with `apify run`, provide input data by creating a JSON file at: + +``` +storage/key_value_stores/default/INPUT.json +``` + +This file should contain the input parameters defined in your `.actor/input_schema.json`. The actor will read this input when running locally, mirroring how it receives input on the Apify platform. + +**IMPORTANT - Local storage is NOT synced to the Apify Console:** +- Running `apify run` stores all data (datasets, key-value stores, request queues) **only on your local filesystem** in the `storage/` directory. +- This data is **never** automatically uploaded or pushed to the Apify platform. It exists only on your machine. +- To verify results on the Apify Console, you must deploy the Actor with `apify push` and then run it on the platform. +- Do **not** rely on checking the Apify Console to verify results from local runs — instead, inspect the local `storage/` directory or check the Actor's log output. + +## Standby Mode + +See [references/standby-mode.md](references/standby-mode.md) for complete standby mode documentation including readiness probe implementation for JavaScript/TypeScript and Python. + +## Project Structure + +``` +.actor/ +├── actor.json # Actor config: name, version, env vars, runtime +├── input_schema.json # Input validation & Console form definition +└── output_schema.json # Output storage and display templates +src/ +└── main.js/ts/py # Actor entry point +storage/ # Local-only storage (NOT synced to Apify Console) +├── datasets/ # Output items (JSON objects) +├── key_value_stores/ # Files, config, INPUT +└── request_queues/ # Pending crawl requests +Dockerfile # Container image definition +``` + +## Actor Configuration + +See [references/actor-json.md](references/actor-json.md) for complete actor.json structure and configuration options. + +## Input Schema + +See [references/input-schema.md](references/input-schema.md) for input schema structure and examples. + +## Output Schema + +See [references/output-schema.md](references/output-schema.md) for output schema structure, examples, and template variables. + +## Dataset Schema + +See [references/dataset-schema.md](references/dataset-schema.md) for dataset schema structure, configuration, and display properties. + +## Key-Value Store Schema + +See [references/key-value-store-schema.md](references/key-value-store-schema.md) for key-value store schema structure, collections, and configuration. + +## Actor README + +**IMPORTANT:** Always generate a README.md as part of Actor development. The README is the Actor's landing page on Apify Store and is critical for discoverability (SEO), user onboarding, and support. Do not consider an Actor complete without a proper README. + +See [references/actor-readme.md](references/actor-readme.md) for the required structure, SEO best practices, and content guidelines. Also review these top Actors for best practices: + +- [Instagram Scraper](https://apify.com/apify/instagram-scraper) +- [Google Maps Scraper](https://apify.com/compass/crawler-google-places) + +## Apify MCP Tools + +If MCP server is configured, use these tools for documentation: + +- `search-apify-docs` - Search documentation +- `fetch-apify-docs` - Get full doc pages + +Otherwise, the MCP Server url: `https://mcp.apify.com/?tools=docs`. + +## Resources + +- [docs.apify.com/llms.txt](https://docs.apify.com/llms.txt) - Apify quick reference documentation +- [docs.apify.com/llms-full.txt](https://docs.apify.com/llms-full.txt) - Apify complete documentation +- [https://crawlee.dev/llms.txt](https://crawlee.dev/llms.txt) - Crawlee quick reference documentation +- [https://crawlee.dev/llms-full.txt](https://crawlee.dev/llms-full.txt) - Crawlee complete documentation +- [whitepaper.actor](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete Actor specification \ No newline at end of file diff --git a/skills/apify-actor-development/references/actor-json.md b/skills/apify-actor-development/references/actor-json.md new file mode 100644 index 0000000..f698139 --- /dev/null +++ b/skills/apify-actor-development/references/actor-json.md @@ -0,0 +1,66 @@ +# Actor Configuration (actor.json) + +The `.actor/actor.json` file contains the Actor's configuration including metadata, schema references, and platform settings. + +## Structure + +```json +{ + "actorSpecification": 1, + "name": "project-name", + "title": "Project Title", + "description": "Actor description", + "version": "0.0", + "meta": { + "templateId": "template-id", + "generatedBy": "" + }, + "input": "./input_schema.json", + "output": "./output_schema.json", + "storages": { + "dataset": "./dataset_schema.json" + }, + "dockerfile": "../Dockerfile" +} +``` + +## Example + +```json +{ + "actorSpecification": 1, + "name": "project-cheerio-crawler-javascript", + "title": "Project Cheerio Crawler Javascript", + "description": "Crawlee and Cheerio project in javascript.", + "version": "0.0", + "meta": { + "templateId": "js-crawlee-cheerio", + "generatedBy": "Claude Code with Claude Sonnet 4.5" + }, + "input": "./input_schema.json", + "output": "./output_schema.json", + "storages": { + "dataset": "./dataset_schema.json" + }, + "dockerfile": "../Dockerfile" +} +``` + +## Properties + +- `actorSpecification` (integer, required) - Version of actor specification (currently 1) +- `name` (string, required) - Actor identifier (lowercase, hyphens allowed) +- `title` (string, required) - Human-readable title displayed in UI +- `description` (string, optional) - Actor description for marketplace +- `version` (string, required) - Semantic version number +- `meta` (object, optional) - Metadata about actor generation + - `templateId` (string) - ID of template used to create the actor + - `generatedBy` (string) - Tool and model name that generated/modified the actor (e.g., "Claude Code with Claude Sonnet 4.5") +- `input` (string, optional) - Path to input schema file +- `output` (string, optional) - Path to output schema file +- `storages` (object, optional) - Storage schema references + - `dataset` (string) - Path to dataset schema file + - `keyValueStore` (string) - Path to key-value store schema file +- `dockerfile` (string, optional) - Path to Dockerfile + +**Important:** Always fill in the `generatedBy` property with the tool and model you're currently using (e.g., "Claude Code with Claude Sonnet 4.5") to help Apify improve documentation. diff --git a/skills/apify-actor-development/references/actor-readme.md b/skills/apify-actor-development/references/actor-readme.md new file mode 100644 index 0000000..40407cc --- /dev/null +++ b/skills/apify-actor-development/references/actor-readme.md @@ -0,0 +1,93 @@ +# Actor README Guidelines + +The README is the Actor's landing page on Apify Store. It serves as SEO content, first impression, usage guide, and support resource. **Always generate a README.md when creating or deploying an Actor.** + +## Required Structure + +Write in Markdown. Use H2 (`##`) for main sections (these form the table of contents) and H3 (`###`) for subsections. Do not use H1 — the Actor name is automatically used as H1. + +### 1. What does [Actor name] do? + +- 1-2 sentences explaining what the Actor does and doesn't do +- Include a link to the target website +- Mention keywords like "API" (e.g., "Instagram API alternative") +- Bold the most important terms + +### 2. Why use [Actor name]? / Why scrape [target site]? + +- Business use cases and benefits +- List main features and capabilities +- Highlight Apify platform advantages: scheduling, API access, integrations, proxy rotation, monitoring + +### 3. What data can [Actor name] extract? + +- Table showing main data fields the Actor outputs (field name, type, description) +- Don't list every field — focus on the most useful and understandable ones + +### 4. How to scrape [target site] + +- Numbered step-by-step tutorial (Google may pick these up as rich snippets) +- Include a link to blog tutorials if they exist + +### 5. How much will it cost to scrape [target site]? + +- Set pricing expectations based on the Actor's pricing model +- For pay-per-result: mention free tier limits and what larger plans offer +- For compute units: explain average data volume per dollar +- Cost-related questions rank well in Google search + +### 6. Input + +- Reference the input tab: "See the input tab for full configuration options" +- Explain any complex input fields or special formatting requirements +- Screenshot of the input schema is optional but helpful + +### 7. Output + +- Include: "You can download the dataset in various formats such as JSON, HTML, CSV, or Excel" +- Show a simplified JSON output example (2-3 items) +- If output is complex, show separate examples for different data types + +### 8. Tips / Advanced options (if applicable) + +- How to limit compute unit usage +- How to get more accurate results or improve speed + +### 9. FAQ, Disclaimers, and Support + +- Legal/scraping disclaimer (use this template and customize with the target site name): + > Our Actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our Actors, when used for ethical purposes by Apify users, are safe. However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. +- Common troubleshooting tips +- Mention the Issues tab for feedback +- Link to API tab for programmatic access +- Use cases for the extracted data + +## SEO Best Practices + +- Include keywords naturally in H2/H3 headings (e.g., "How to scrape Instagram" not just "How to use") +- Target "People Also Ask" style questions as H3 headings +- Aim for at least 300 words total +- Embed a YouTube video URL if available (renders automatically as a player) +- Make images clickable with links + +## Tone + +- Match the README tone to the target audience skill level +- For no-code users: use plain language, avoid code blocks early on +- For developers: include technical details, code examples, and API references +- Be clear about what technical knowledge is needed to use the Actor + +## Reference Actors + +Before writing a README, review these top Actors on the Apify Store for best practices on structure, tone, and content: + +- [Instagram Scraper](https://apify.com/apify/instagram-scraper) +- [Google Maps Scraper](https://apify.com/compass/crawler-google-places) + +## Key Rules + +- Always write the README as part of Actor development — do not skip this step +- The first 25% of the README is what most visitors read — put the most important info there +- Use emojis sparingly as bullet points to break up text +- Keep images compressed but good quality +- Use [Carbon](https://github.com/carbon-app/carbon) for code snippet screenshots if needed diff --git a/skills/apify-actor-development/references/dataset-schema.md b/skills/apify-actor-development/references/dataset-schema.md new file mode 100644 index 0000000..c61a8ce --- /dev/null +++ b/skills/apify-actor-development/references/dataset-schema.md @@ -0,0 +1,209 @@ +# Dataset Schema Reference + +The dataset schema defines how your Actor's output data is structured, transformed, and displayed in the Output tab in the Apify Console. + +## Examples + +### JavaScript and TypeScript + +Consider an example Actor that calls `Actor.pushData()` to store data into dataset: + +```javascript +import { Actor } from 'apify'; +// Initialize the JavaScript SDK +await Actor.init(); + +/** + * Actor code + */ +await Actor.pushData({ + numericField: 10, + pictureUrl: 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png', + linkUrl: 'https://google.com', + textField: 'Google', + booleanField: true, + dateField: new Date(), + arrayField: ['#hello', '#world'], + objectField: {}, +}); + +// Exit successfully +await Actor.exit(); +``` + +### Python + +Consider an example Actor that calls `Actor.push_data()` to store data into dataset: + +```python +# Dataset push example (Python) +import asyncio +from datetime import datetime +from apify import Actor + +async def main(): + await Actor.init() + + # Actor code + await Actor.push_data({ + 'numericField': 10, + 'pictureUrl': 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_92x30dp.png', + 'linkUrl': 'https://google.com', + 'textField': 'Google', + 'booleanField': True, + 'dateField': datetime.now().isoformat(), + 'arrayField': ['#hello', '#world'], + 'objectField': {}, + }) + + # Exit successfully + await Actor.exit() + +if __name__ == '__main__': + asyncio.run(main()) +``` + +## Configuration + +To set up the Actor's output tab UI, reference a dataset schema file in `.actor/actor.json`: + +```json +{ + "actorSpecification": 1, + "name": "book-library-scraper", + "title": "Book Library Scraper", + "version": "1.0.0", + "storages": { + "dataset": "./dataset_schema.json" + } +} +``` + +Then create the dataset schema in `.actor/dataset_schema.json`: + +```json +{ + "actorSpecification": 1, + "fields": {}, + "views": { + "overview": { + "title": "Overview", + "transformation": { + "fields": [ + "pictureUrl", + "linkUrl", + "textField", + "booleanField", + "arrayField", + "objectField", + "dateField", + "numericField" + ] + }, + "display": { + "component": "table", + "properties": { + "pictureUrl": { + "label": "Image", + "format": "image" + }, + "linkUrl": { + "label": "Link", + "format": "link" + }, + "textField": { + "label": "Text", + "format": "text" + }, + "booleanField": { + "label": "Boolean", + "format": "boolean" + }, + "arrayField": { + "label": "Array", + "format": "array" + }, + "objectField": { + "label": "Object", + "format": "object" + }, + "dateField": { + "label": "Date", + "format": "date" + }, + "numericField": { + "label": "Number", + "format": "number" + } + } + } + } + } +} +``` + +## Structure + +```json +{ + "actorSpecification": 1, + "fields": {}, + "views": { + "": { + "title": "string (required)", + "description": "string (optional)", + "transformation": { + "fields": ["string (required)"], + "unwind": ["string (optional)"], + "flatten": ["string (optional)"], + "omit": ["string (optional)"], + "limit": "integer (optional)", + "desc": "boolean (optional)" + }, + "display": { + "component": "table (required)", + "properties": { + "": { + "label": "string (optional)", + "format": "text|number|date|link|boolean|image|array|object (optional)" + } + } + } + } + } +} +``` + +## Properties + +### Dataset Schema Properties + +- `actorSpecification` (integer, required) - Specifies the version of dataset schema structure document (currently only version 1) +- `fields` (JSONSchema object, required) - Schema of one dataset object (use JsonSchema Draft 2020-12 or compatible) +- `views` (DatasetView object, required) - Object with API and UI views description + +### DatasetView Properties + +- `title` (string, required) - Visible in UI Output tab and API +- `description` (string, optional) - Only available in API response +- `transformation` (ViewTransformation object, required) - Data transformation applied when loading from Dataset API +- `display` (ViewDisplay object, required) - Output tab UI visualization definition + +### ViewTransformation Properties + +- `fields` (string[], required) - Fields to present in output (order matches column order) +- `unwind` (string[], optional) - Deconstructs nested children into parent object +- `flatten` (string[], optional) - Transforms nested object into flat structure +- `omit` (string[], optional) - Removes specified fields from output +- `limit` (integer, optional) - Maximum number of results (default: all) +- `desc` (boolean, optional) - Sort order (true = newest first) + +### ViewDisplay Properties + +- `component` (string, required) - Only `table` is available +- `properties` (Object, optional) - Keys matching `transformation.fields` with ViewDisplayProperty values + +### ViewDisplayProperty Properties + +- `label` (string, optional) - Table column header +- `format` (string, optional) - One of: `text`, `number`, `date`, `link`, `boolean`, `image`, `array`, `object` diff --git a/skills/apify-actor-development/references/input-schema.md b/skills/apify-actor-development/references/input-schema.md new file mode 100644 index 0000000..0acfeb0 --- /dev/null +++ b/skills/apify-actor-development/references/input-schema.md @@ -0,0 +1,66 @@ +# Input Schema Reference + +The input schema defines the input parameters for an Actor. It's a JSON object comprising various field types supported by the Apify platform. + +## Structure + +```json +{ + "title": "", + "type": "object", + "schemaVersion": 1, + "properties": { + /* define input fields here */ + }, + "required": [] +} +``` + +## Example + +```json +{ + "title": "E-commerce Product Scraper Input", + "type": "object", + "schemaVersion": 1, + "properties": { + "startUrls": { + "title": "Start URLs", + "type": "array", + "description": "URLs to start scraping from (category pages or product pages)", + "editor": "requestListSources", + "default": [{ "url": "https://example.com/category" }], + "prefill": [{ "url": "https://example.com/category" }] + }, + "followVariants": { + "title": "Follow Product Variants", + "type": "boolean", + "description": "Whether to scrape product variants (different colors, sizes)", + "default": true + }, + "maxRequestsPerCrawl": { + "title": "Max Requests per Crawl", + "type": "integer", + "description": "Maximum number of pages to scrape (0 = unlimited)", + "default": 1000, + "minimum": 0 + }, + "proxyConfiguration": { + "title": "Proxy Configuration", + "type": "object", + "description": "Proxy settings for anti-bot protection", + "editor": "proxy", + "default": { "useApifyProxy": false } + }, + "locale": { + "title": "Locale", + "type": "string", + "description": "Language/country code for localized content", + "default": "cs", + "enum": ["cs", "en", "de", "sk"], + "enumTitles": ["Czech", "English", "German", "Slovak"] + } + }, + "required": ["startUrls"] +} +``` diff --git a/skills/apify-actor-development/references/key-value-store-schema.md b/skills/apify-actor-development/references/key-value-store-schema.md new file mode 100644 index 0000000..81b588f --- /dev/null +++ b/skills/apify-actor-development/references/key-value-store-schema.md @@ -0,0 +1,129 @@ +# Key-Value Store Schema Reference + +The key-value store schema organizes keys into logical groups called collections for easier data management. + +## Examples + +### JavaScript and TypeScript + +Consider an example Actor that calls `Actor.setValue()` to save records into the key-value store: + +```javascript +import { Actor } from 'apify'; +// Initialize the JavaScript SDK +await Actor.init(); + +/** + * Actor code + */ +await Actor.setValue('document-1', 'my text data', { contentType: 'text/plain' }); + +await Actor.setValue(`image-${imageID}`, imageBuffer, { contentType: 'image/jpeg' }); + +// Exit successfully +await Actor.exit(); +``` + +### Python + +Consider an example Actor that calls `Actor.set_value()` to save records into the key-value store: + +```python +# Key-Value Store set example (Python) +import asyncio +from apify import Actor + +async def main(): + await Actor.init() + + # Actor code + await Actor.set_value('document-1', 'my text data', content_type='text/plain') + + image_id = '123' # example placeholder + image_buffer = b'...' # bytes buffer with image data + await Actor.set_value(f'image-{image_id}', image_buffer, content_type='image/jpeg') + + # Exit successfully + await Actor.exit() + +if __name__ == '__main__': + asyncio.run(main()) +``` + +## Configuration + +To configure the key-value store schema, reference a schema file in `.actor/actor.json`: + +```json +{ + "actorSpecification": 1, + "name": "data-collector", + "title": "Data Collector", + "version": "1.0.0", + "storages": { + "keyValueStore": "./key_value_store_schema.json" + } +} +``` + +Then create the key-value store schema in `.actor/key_value_store_schema.json`: + +```json +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "Key-Value Store Schema", + "collections": { + "documents": { + "title": "Documents", + "description": "Text documents stored by the Actor", + "keyPrefix": "document-" + }, + "images": { + "title": "Images", + "description": "Images stored by the Actor", + "keyPrefix": "image-", + "contentTypes": ["image/jpeg"] + } + } +} +``` + +## Structure + +```json +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "string (required)", + "description": "string (optional)", + "collections": { + "": { + "title": "string (required)", + "description": "string (optional)", + "key": "string (conditional - use key OR keyPrefix)", + "keyPrefix": "string (conditional - use key OR keyPrefix)", + "contentTypes": ["string (optional)"], + "jsonSchema": "object (optional)" + } + } +} +``` + +## Properties + +### Key-Value Store Schema Properties + +- `actorKeyValueStoreSchemaVersion` (integer, required) - Version of key-value store schema structure document (currently only version 1) +- `title` (string, required) - Title of the schema +- `description` (string, optional) - Description of the schema +- `collections` (Object, required) - Object where each key is a collection ID and value is a Collection object + +### Collection Properties + +- `title` (string, required) - Collection title shown in UI tabs +- `description` (string, optional) - Description appearing in UI tooltips +- `key` (string, conditional) - Single specific key for this collection +- `keyPrefix` (string, conditional) - Prefix for keys included in this collection +- `contentTypes` (string[], optional) - Allowed content types for validation +- `jsonSchema` (object, optional) - JSON Schema Draft 07 format for `application/json` content type validation + +Either `key` or `keyPrefix` must be specified for each collection, but not both. diff --git a/skills/apify-actor-development/references/logging.md b/skills/apify-actor-development/references/logging.md new file mode 100644 index 0000000..cc39bf3 --- /dev/null +++ b/skills/apify-actor-development/references/logging.md @@ -0,0 +1,50 @@ +# Actor Logging Reference + +## JavaScript and TypeScript + +**ALWAYS use the `apify/log` package for logging** - This package contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs. + +### Available Log Levels in `apify/log` + +The Apify log package provides the following methods for logging: + +- `log.debug()` - Debug level logs (detailed diagnostic information) +- `log.info()` - Info level logs (general informational messages) +- `log.warning()` - Warning level logs (warning messages for potentially problematic situations) +- `log.warningOnce()` - Warning level logs (same warning message logged only once) +- `log.error()` - Error level logs (error messages for failures) +- `log.exception()` - Exception level logs (for exceptions with stack traces) +- `log.perf()` - Performance level logs (performance metrics and timing information) +- `log.deprecated()` - Deprecation level logs (warnings about deprecated code) +- `log.softFail()` - Soft failure logs (non-critical failures that don't stop execution, e.g., input validation errors, skipped items) +- `log.internal()` - Internal level logs (internal/system messages) + +### Best Practices + +- Use `log.debug()` for detailed operation-level diagnostics (inside functions) +- Use `log.info()` for general informational messages (API requests, successful operations) +- Use `log.warning()` for potentially problematic situations (validation failures, unexpected states) +- Use `log.error()` for actual errors and failures +- Use `log.exception()` for caught exceptions with stack traces + +## Python + +**ALWAYS use `Actor.log` for logging** - This logger contains critical security logic including censoring sensitive data (Apify tokens, API keys, credentials) to prevent accidental exposure in logs. + +### Available Log Levels + +The Apify Actor logger provides the following methods for logging: + +- `Actor.log.debug()` - Debug level logs (detailed diagnostic information) +- `Actor.log.info()` - Info level logs (general informational messages) +- `Actor.log.warning()` - Warning level logs (warning messages for potentially problematic situations) +- `Actor.log.error()` - Error level logs (error messages for failures) +- `Actor.log.exception()` - Exception level logs (for exceptions with stack traces) + +### Best Practices + +- Use `Actor.log.debug()` for detailed operation-level diagnostics (inside functions) +- Use `Actor.log.info()` for general informational messages (API requests, successful operations) +- Use `Actor.log.warning()` for potentially problematic situations (validation failures, unexpected states) +- Use `Actor.log.error()` for actual errors and failures +- Use `Actor.log.exception()` for caught exceptions with stack traces diff --git a/skills/apify-actor-development/references/output-schema.md b/skills/apify-actor-development/references/output-schema.md new file mode 100644 index 0000000..89e439c --- /dev/null +++ b/skills/apify-actor-development/references/output-schema.md @@ -0,0 +1,49 @@ +# Output Schema Reference + +The Actor output schema builds upon the schemas for the dataset and key-value store. It specifies where an Actor stores its output and defines templates for accessing that output. Apify Console uses these output definitions to display run results. + +## Structure + +```json +{ + "actorOutputSchemaVersion": 1, + "title": "", + "properties": { + /* define your outputs here */ + } +} +``` + +## Example + +```json +{ + "actorOutputSchemaVersion": 1, + "title": "Output schema of the files scraper", + "properties": { + "files": { + "type": "string", + "title": "Files", + "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys" + }, + "dataset": { + "type": "string", + "title": "Dataset", + "template": "{{links.apiDefaultDatasetUrl}}/items" + } + } +} +``` + +## Output Schema Template Variables + +- `links` (object) - Contains quick links to most commonly used URLs +- `links.publicRunUrl` (string) - Public run url in format `https://console.apify.com/view/runs/:runId` +- `links.consoleRunUrl` (string) - Console run url in format `https://console.apify.com/actors/runs/:runId` +- `links.apiRunUrl` (string) - API run url in format `https://api.apify.com/v2/actor-runs/:runId` +- `links.apiDefaultDatasetUrl` (string) - API url of default dataset in format `https://api.apify.com/v2/datasets/:defaultDatasetId` +- `links.apiDefaultKeyValueStoreUrl` (string) - API url of default key-value store in format `https://api.apify.com/v2/key-value-stores/:defaultKeyValueStoreId` +- `links.containerRunUrl` (string) - URL of a webserver running inside the run in format `https://.runs.apify.net/` +- `run` (object) - Contains information about the run same as it is returned from the `GET Run` API endpoint +- `run.defaultDatasetId` (string) - ID of the default dataset +- `run.defaultKeyValueStoreId` (string) - ID of the default key-value store diff --git a/skills/apify-actor-development/references/standby-mode.md b/skills/apify-actor-development/references/standby-mode.md new file mode 100644 index 0000000..73d6025 --- /dev/null +++ b/skills/apify-actor-development/references/standby-mode.md @@ -0,0 +1,61 @@ +# Actor Standby Mode Reference + +## JavaScript and TypeScript + +- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it +- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management + +You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`. + +### Readiness Probe Implementation Example + +```javascript +// Apify standby readiness probe at root path +app.get('/', (req, res) => { + res.writeHead(200, { 'Content-Type': 'text/plain' }); + if (req.headers['x-apify-container-server-readiness-probe']) { + res.end('Readiness probe OK\n'); + } else { + res.end('Actor is ready\n'); + } +}); +``` + +Key points: + +- Detect the `x-apify-container-server-readiness-probe` header in incoming requests +- Respond with HTTP 200 status code for both readiness probe and normal requests +- This enables proper Actor lifecycle management in standby mode + +## Python + +- **NEVER disable standby mode (`usesStandbyMode: false`) in `.actor/actor.json` without explicit permission** - Actor Standby mode solves this problem by letting you have the Actor ready in the background, waiting for the incoming HTTP requests. In a sense, the Actor behaves like a real-time web server or standard API server instead of running the logic once to process everything in batch. Always keep `usesStandbyMode: true` unless there is a specific documented reason to disable it +- **ALWAYS implement readiness probe handler for standby Actors** - Handle the `x-apify-container-server-readiness-probe` header at GET / endpoint to ensure proper Actor lifecycle management + +You can recognize a standby Actor by checking the `usesStandbyMode` property in `.actor/actor.json`. Only implement the readiness probe if this property is set to `true`. + +### Readiness Probe Implementation Example + +```python +# Apify standby readiness probe +from http.server import SimpleHTTPRequestHandler + +class GetHandler(SimpleHTTPRequestHandler): + def do_GET(self): + # Handle Apify standby readiness probe + if 'x-apify-container-server-readiness-probe' in self.headers: + self.send_response(200) + self.end_headers() + self.wfile.write(b'Readiness probe OK') + return + + self.send_response(200) + self.end_headers() + self.wfile.write(b'Actor is ready') +``` + +Key points: + +- Detect the `x-apify-container-server-readiness-probe` header in incoming requests +- Respond with HTTP 200 status code for both readiness probe and normal requests +- This enables proper Actor lifecycle management in standby mode diff --git a/skills/apify-actorization/SKILL.md b/skills/apify-actorization/SKILL.md new file mode 100644 index 0000000..c84700e --- /dev/null +++ b/skills/apify-actorization/SKILL.md @@ -0,0 +1,216 @@ +--- +name: apify-actorization +description: Convert existing projects into Apify Actors - serverless cloud programs. Actorize JavaScript/TypeScript (SDK with Actor.init/exit), Python (async context manager), or any language (CLI wrapper). Use when migrating code to Apify, wrapping CLI tools as Actors, or adding Actor SDK to existing projects. +--- + +# Apify Actorization + +Actorization converts existing software into reusable serverless applications compatible with the Apify platform. Actors are programs packaged as Docker images that accept well-defined JSON input, perform an action, and optionally produce structured JSON output. + +## Quick Start + +1. Run `apify init` in project root +2. Wrap code with SDK lifecycle (see language-specific section below) +3. Configure `.actor/input_schema.json` +4. Test with `apify run --input '{"key": "value"}'` +5. Deploy with `apify push` + +## When to Use This Skill + +- Converting an existing project to run on Apify platform +- Adding Apify SDK integration to a project +- Wrapping a CLI tool or script as an Actor +- Migrating a Crawlee project to Apify + +## Prerequisites + +Verify `apify` CLI is installed: + +```bash +apify --help +``` + +If not installed, use one of these methods (listed in order of preference): + +```bash +# Preferred: install via a package manager (provides integrity checks) +npm install -g apify-cli + +# Or (Mac): brew install apify-cli +``` + +> **Security note:** Do NOT install the CLI by piping remote scripts to a shell +> (e.g. `curl ... | bash` or `irm ... | iex`). Always use a package manager. + +Verify CLI is logged in: + +```bash +# Auth check — do NOT pipe to /dev/null, you need to see errors +apify info 2>&1 +``` + +If not logged in, authenticate using OAuth (opens browser): + +```bash +apify login +``` + +If browser login isn't available (headless environment or CI), ensure the `APIFY_TOKEN` environment variable is exported. The CLI reads it automatically - no explicit login needed. If the user doesn't have a token, generate one at https://console.apify.com/settings/integrations. + +> **Security note:** Avoid passing tokens as command-line arguments (e.g. `apify login -t `). +> Arguments are visible in process listings and may be recorded in shell history. +> Prefer OAuth login or environment variables instead. +> Never log, print, or embed `APIFY_TOKEN` in source code or configuration files. + +## Actorization Checklist + +Copy this checklist to track progress: + +- [ ] Step 1: Analyze project (language, entry point, inputs, outputs) +- [ ] Step 2: Run `apify init` to create Actor structure +- [ ] Step 3: Apply language-specific SDK integration +- [ ] Step 4: Configure `.actor/input_schema.json` +- [ ] Step 5: Configure `.actor/output_schema.json` (if applicable) +- [ ] Step 6: Update `.actor/actor.json` metadata +- [ ] Step 7: Write README.md for the Apify Store listing +- [ ] Step 8: Test locally with `apify run` +- [ ] Step 9: Deploy with `apify push` + +## Step 1: Analyze the Project + +Before making changes, understand the project: + +1. **Identify the language** - JavaScript/TypeScript, Python, or other +2. **Find the entry point** - The main file that starts execution +3. **Identify inputs** - Command-line arguments, environment variables, config files +4. **Identify outputs** - Files, console output, API responses +5. **Check for state** - Does it need to persist data between runs? + +## Step 2: Initialize Actor Structure + +Run in the project root: + +```bash +apify init +``` + +This creates: +- `.actor/actor.json` - Actor configuration and metadata +- `.actor/input_schema.json` - Input definition for the Apify Console +- `Dockerfile` (if not present) - Container image definition + +## Step 3: Apply Language-Specific Changes + +Choose based on your project's language: + +- **JavaScript/TypeScript**: See [js-ts-actorization.md](references/js-ts-actorization.md) +- **Python**: See [python-actorization.md](references/python-actorization.md) +- **Other Languages (CLI-based)**: See [cli-actorization.md](references/cli-actorization.md) + +### Quick Reference + +| Language | Install | Wrap Code | +|----------|---------|-----------| +| JS/TS | `npm install apify` | `await Actor.init()` ... `await Actor.exit()` | +| Python | `pip install apify` | `async with Actor:` | +| Other | Use CLI in wrapper script | `apify actor:get-input` / `apify actor:push-data` | + +## Steps 4-6: Configure Schemas + +See [schemas-and-output.md](references/schemas-and-output.md) for detailed configuration of: +- Input schema (`.actor/input_schema.json`) +- Output schema (`.actor/output_schema.json`) +- Actor configuration (`.actor/actor.json`) +- State management (request queues, key-value stores) + +Validate schemas against `@apify/json_schemas` npm package. + +## Step 7: Write README + +**IMPORTANT:** Always generate a README.md as part of actorization. The README is the Actor's landing page on Apify Store and is critical for discoverability (SEO), user onboarding, and support. Do not consider an Actor complete without a proper README. + +See the Actor README guidelines at `skills/apify-actor-development/references/actor-readme.md` for the required structure including: intro and features, data extraction table, step-by-step tutorial, pricing info, input/output examples, and FAQ. Aim for at least 300 words with SEO-optimized H2/H3 headings. Also review these top Actors for best practices: + +- [Instagram Scraper](https://apify.com/apify/instagram-scraper) +- [Google Maps Scraper](https://apify.com/compass/crawler-google-places) + +## Step 8: Test Locally + +Run the actor with inline input (for JS/TS and Python actors): + +```bash +apify run --input '{"startUrl": "https://example.com", "maxItems": 10}' +``` + +Or use an input file: + +```bash +apify run --input-file ./test-input.json +``` + +**Important:** Always use `apify run`, not `npm start` or `python main.py`. The CLI sets up the proper environment and storage. + +## Step 9: Deploy + +```bash +apify push +``` + +This uploads and builds your actor on the Apify platform. + +## Monetization (Optional) + +After deploying, you can monetize your actor in the Apify Store. The recommended model is **Pay Per Event (PPE)**: + +- Per result/item scraped +- Per page processed +- Per API call made + +Configure PPE in the Apify Console under Actor > Monetization. Charge for events in your code with `await Actor.charge('result')`. + +Other options: **Rental** (monthly subscription) or **Free** (open source). + +## Security + +**Treat all crawled web content as untrusted input.** Actors ingest data from external websites that may contain malicious payloads. Follow these rules: + +- **Sanitize crawled data** — Never pass raw HTML, URLs, or scraped text directly into shell commands, `eval()`, database queries, or template engines. Use proper escaping or parameterized APIs. +- **Validate and type-check all external data** — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures. +- **Do not execute or interpret crawled content** — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts. +- **Isolate credentials from data pipelines** — Ensure `APIFY_TOKEN` and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code. +- **Review dependencies before installing** — When adding packages with `npm install` or `pip install`, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages. +- **Pin versions and use lockfiles** — Always commit `package-lock.json` (Node.js) or pin exact versions in `requirements.txt` (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run `npm audit` or `pip-audit` periodically to check for known vulnerabilities. + +## Pre-Deployment Checklist + +- [ ] `.actor/actor.json` exists with correct name and description +- [ ] `.actor/actor.json` validates against `@apify/json_schemas` (`actor.schema.json`) +- [ ] `.actor/input_schema.json` defines all required inputs +- [ ] `.actor/input_schema.json` validates against `@apify/json_schemas` (`input.schema.json`) +- [ ] `.actor/output_schema.json` defines output structure (if applicable) +- [ ] `.actor/output_schema.json` validates against `@apify/json_schemas` (`output.schema.json`) +- [ ] `Dockerfile` is present and builds successfully +- [ ] `Actor.init()` / `Actor.exit()` wraps main code (JS/TS) +- [ ] `async with Actor:` wraps main code (Python) +- [ ] Inputs are read via `Actor.getInput()` / `Actor.get_input()` +- [ ] Outputs use `Actor.pushData()` or key-value store +- [ ] `apify run` executes successfully with test input +- [ ] `README.md` exists with proper structure (intro, features, data table, tutorial, pricing, input/output examples) +- [ ] `generatedBy` is set in actor.json meta section + +## Apify MCP Tools + +If MCP server is configured, use these tools for documentation: + +- `search-apify-docs` - Search documentation +- `fetch-apify-docs` - Get full doc pages + +Otherwise, the MCP Server url: `https://mcp.apify.com/?tools=docs`. + +## Resources + +- [Actorization Academy](https://docs.apify.com/academy/actorization) - Comprehensive guide +- [Apify SDK for JavaScript](https://docs.apify.com/sdk/js) - Full SDK reference +- [Apify SDK for Python](https://docs.apify.com/sdk/python) - Full SDK reference +- [Apify CLI Reference](https://docs.apify.com/cli) - CLI commands +- [Actor Specification](https://raw.githubusercontent.com/apify/actor-whitepaper/refs/heads/master/README.md) - Complete specification diff --git a/skills/apify-actorization/references/cli-actorization.md b/skills/apify-actorization/references/cli-actorization.md new file mode 100644 index 0000000..73b4ca6 --- /dev/null +++ b/skills/apify-actorization/references/cli-actorization.md @@ -0,0 +1,81 @@ +# CLI-Based Actorization + +For languages without an SDK (Go, Rust, Java, etc.), create a wrapper script that uses the Apify CLI. + +## Create Wrapper Script + +Create `start.sh` in project root: + +```bash +#!/bin/bash +set -e + +# Get input from Apify key-value store +INPUT=$(apify actor:get-input) + +# Parse input values (adjust based on your input schema) +MY_PARAM=$(echo "$INPUT" | jq -r '.myParam // "default"') + +# Run your application with the input +./your-application --param "$MY_PARAM" + +# If your app writes to a file, push it to key-value store +# apify actor:set-value OUTPUT --contentType application/json < output.json + +# Or push structured data to dataset +# apify actor:push-data '{"result": "value"}' +``` + +## Update Dockerfile + +Reference the [cli-start template Dockerfile](https://github.com/apify/actor-templates/blob/master/templates/cli-start/Dockerfile) which includes the `ubi` utility for installing binaries from GitHub releases. + +```dockerfile +FROM apify/actor-node:20 + +# Install ubi for easy GitHub release installation +RUN curl --silent --location \ + https://raw.githubusercontent.com/houseabsolute/ubi/master/bootstrap/bootstrap-ubi.sh | sh + +# Install your CLI tool from GitHub releases (example) +# RUN ubi --project your-org/your-tool --in /usr/local/bin + +# Or install apify-cli and jq manually +RUN npm install -g apify-cli +RUN apt-get update && apt-get install -y jq + +# Copy your application +COPY . . + +# Build your application if needed +# RUN ./build.sh + +# Make start script executable +RUN chmod +x start.sh + +# Run the wrapper script +CMD ["./start.sh"] +``` + +## Testing CLI-Based Actors + +For CLI-based actors (shell wrapper scripts), you may need to test the underlying application directly with mock input, as `apify run` requires a Node.js or Python entry point. + +Test your wrapper script locally: + +```bash +# Set up mock input +export INPUT='{"myParam": "test-value"}' + +# Run wrapper script +./start.sh +``` + +## CLI Commands Reference + +| Command | Description | +|---------|-------------| +| `apify actor:get-input` | Get input JSON from key-value store | +| `apify actor:set-value KEY` | Store value in key-value store | +| `apify actor:push-data JSON` | Push data to dataset | +| `apify actor:get-value KEY` | Retrieve value from key-value store | diff --git a/skills/apify-actorization/references/js-ts-actorization.md b/skills/apify-actorization/references/js-ts-actorization.md new file mode 100644 index 0000000..2b2c894 --- /dev/null +++ b/skills/apify-actorization/references/js-ts-actorization.md @@ -0,0 +1,111 @@ +# JavaScript/TypeScript Actorization + +## Install the Apify SDK + +```bash +npm install apify +``` + +## Wrap Main Code with Actor Lifecycle + +```javascript +import { Actor } from 'apify'; + +// Initialize connection to Apify platform +await Actor.init(); + +// ============================================ +// Your existing code goes here +// ============================================ + +// Example: Get input from Apify Console or API +const input = await Actor.getInput(); +console.log('Input:', input); + +// Example: Your crawler or processing logic +// const crawler = new PlaywrightCrawler({ ... }); +// await crawler.run([input.startUrl]); + +// Example: Push results to dataset +// await Actor.pushData({ result: 'data' }); + +// ============================================ +// End of your code +// ============================================ + +// Graceful shutdown +await Actor.exit(); +``` + +## Key Points + +- `Actor.init()` configures storage to use Apify API when running on platform +- `Actor.exit()` handles graceful shutdown and cleanup +- Both calls must be awaited +- Local execution remains unchanged - the SDK automatically detects the environment + +## Crawlee Projects + +Crawlee projects require minimal changes - just wrap with Actor lifecycle: + +```javascript +import { Actor } from 'apify'; +import { PlaywrightCrawler } from 'crawlee'; + +await Actor.init(); + +// Get and validate input +const input = await Actor.getInput(); +const { + startUrl = 'https://example.com', + maxItems = 100, +} = input ?? {}; + +let itemCount = 0; + +const crawler = new PlaywrightCrawler({ + requestHandler: async ({ page, request, pushData }) => { + if (itemCount >= maxItems) return; + + const title = await page.title(); + await pushData({ url: request.url, title }); + itemCount++; + }, +}); + +await crawler.run([startUrl]); + +await Actor.exit(); +``` + +## Express/HTTP Servers + +For web servers, use standby mode in actor.json: + +```json +{ + "actorSpecification": 1, + "name": "my-api", + "usesStandbyMode": true +} +``` + +Then implement readiness probe. See [standby-mode.md](../../apify-actor-development/references/standby-mode.md). + +## Batch Processing Scripts + +```javascript +import { Actor } from 'apify'; + +await Actor.init(); + +const input = await Actor.getInput(); +const items = input.items || []; + +for (const item of items) { + const result = processItem(item); + await Actor.pushData(result); +} + +await Actor.exit(); +``` diff --git a/skills/apify-actorization/references/python-actorization.md b/skills/apify-actorization/references/python-actorization.md new file mode 100644 index 0000000..b536206 --- /dev/null +++ b/skills/apify-actorization/references/python-actorization.md @@ -0,0 +1,95 @@ +# Python Actorization + +## Install the Apify SDK + +```bash +pip install apify +``` + +## Wrap Main Function with Actor Context Manager + +```python +import asyncio +from apify import Actor + +async def main() -> None: + async with Actor: + # ============================================ + # Your existing code goes here + # ============================================ + + # Example: Get input from Apify Console or API + actor_input = await Actor.get_input() + print(f'Input: {actor_input}') + + # Example: Your crawler or processing logic + # crawler = PlaywrightCrawler(...) + # await crawler.run([actor_input.get('startUrl')]) + + # Example: Push results to dataset + # await Actor.push_data({'result': 'data'}) + + # ============================================ + # End of your code + # ============================================ + +if __name__ == '__main__': + asyncio.run(main()) +``` + +## Key Points + +- `async with Actor:` handles both initialization and cleanup +- Automatically manages platform event listeners and graceful shutdown +- Local execution remains unchanged - the SDK automatically detects the environment + +## Crawlee Python Projects + +```python +import asyncio +from apify import Actor +from crawlee.playwright_crawler import PlaywrightCrawler + +async def main() -> None: + async with Actor: + # Get and validate input + actor_input = await Actor.get_input() or {} + start_url = actor_input.get('startUrl', 'https://example.com') + max_items = actor_input.get('maxItems', 100) + + item_count = 0 + + async def request_handler(context): + nonlocal item_count + if item_count >= max_items: + return + + title = await context.page.title() + await context.push_data({'url': context.request.url, 'title': title}) + item_count += 1 + + crawler = PlaywrightCrawler(request_handler=request_handler) + await crawler.run([start_url]) + +if __name__ == '__main__': + asyncio.run(main()) +``` + +## Batch Processing Scripts + +```python +import asyncio +from apify import Actor + +async def main() -> None: + async with Actor: + actor_input = await Actor.get_input() or {} + items = actor_input.get('items', []) + + for item in items: + result = process_item(item) + await Actor.push_data(result) + +if __name__ == '__main__': + asyncio.run(main()) +``` diff --git a/skills/apify-actorization/references/schemas-and-output.md b/skills/apify-actorization/references/schemas-and-output.md new file mode 100644 index 0000000..a838768 --- /dev/null +++ b/skills/apify-actorization/references/schemas-and-output.md @@ -0,0 +1,140 @@ +# Schemas and Output Configuration + +## Input Schema + +Map your application's inputs to `.actor/input_schema.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`input.schema.json`). + +```json +{ + "title": "My Actor Input", + "type": "object", + "schemaVersion": 1, + "properties": { + "startUrl": { + "title": "Start URL", + "type": "string", + "description": "The URL to start processing from", + "editor": "textfield", + "prefill": "https://example.com" + }, + "maxItems": { + "title": "Max Items", + "type": "integer", + "description": "Maximum number of items to process", + "default": 100, + "minimum": 1 + } + }, + "required": ["startUrl"] +} +``` + +### Mapping Guidelines + +- Command-line arguments → input schema properties +- Environment variables → input schema or Actor env vars in actor.json +- Config files → input schema with object/array types +- Flatten deeply nested structures for better UX + +## Output Schema + +Define output structure in `.actor/output_schema.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`output.schema.json`). + +### For Table-Like Data (Multiple Items) + +- Use `Actor.pushData()` (JS) or `Actor.push_data()` (Python) +- Each item becomes a row in the dataset + +### For Single Files or Blobs + +- Use key-value store: `Actor.setValue()` / `Actor.set_value()` +- Get the public URL and include it in the dataset: + +```javascript +// Store file with public access +await Actor.setValue('report.pdf', pdfBuffer, { contentType: 'application/pdf' }); + +// Get the public URL +const storeInfo = await Actor.openKeyValueStore(); +const publicUrl = `https://api.apify.com/v2/key-value-stores/${storeInfo.id}/records/report.pdf`; + +// Include URL in dataset output +await Actor.pushData({ reportUrl: publicUrl }); +``` + +### For Multiple Files with a Common Prefix (Collections) + +```javascript +// Store multiple files with a prefix +for (const [name, data] of files) { + await Actor.setValue(`screenshots/${name}`, data, { contentType: 'image/png' }); +} +// Files are accessible at: .../records/screenshots%2F{name} +``` + +## Actor Configuration (actor.json) + +Configure `.actor/actor.json`. Validate against the JSON Schema from the `@apify/json_schemas` npm package (`actor.schema.json`). + +```json +{ + "actorSpecification": 1, + "name": "my-actor", + "title": "My Actor", + "description": "Brief description of what the actor does", + "version": "1.0.0", + "meta": { + "templateId": "ts_empty", + "generatedBy": "Claude Code with Claude Opus 4.5" + }, + "input": "./input_schema.json", + "dockerfile": "../Dockerfile" +} +``` + +**Important:** Fill in the `generatedBy` property with the tool/model used. + +## State Management + +### Request Queue - For Pausable Task Processing + +The request queue works for any task processing, not just web scraping. Use a dummy URL with custom `uniqueKey` and `userData` for non-URL tasks: + +```javascript +const requestQueue = await Actor.openRequestQueue(); + +// Add tasks to the queue (works for any processing, not just URLs) +await requestQueue.addRequest({ + url: 'https://placeholder.local', // Dummy URL for non-scraping tasks + uniqueKey: `task-${taskId}`, // Unique identifier for deduplication + userData: { itemId: 123, action: 'process' }, // Your custom task data +}); + +// Process tasks from the queue (with Crawlee) +const crawler = new BasicCrawler({ + requestQueue, + requestHandler: async ({ request }) => { + const { itemId, action } = request.userData; + // Process your task using userData + await processTask(itemId, action); + }, +}); +await crawler.run(); + +// Or manually consume without Crawlee: +let request; +while ((request = await requestQueue.fetchNextRequest())) { + await processTask(request.userData); + await requestQueue.markRequestHandled(request); +} +``` + +### Key-Value Store - For Checkpoint State + +```javascript +// Save state +await Actor.setValue('STATE', { processedCount: 100 }); + +// Restore state on restart +const state = await Actor.getValue('STATE') || { processedCount: 0 }; +``` diff --git a/skills/apify-generate-output-schema/SKILL.md b/skills/apify-generate-output-schema/SKILL.md new file mode 100644 index 0000000..710194b --- /dev/null +++ b/skills/apify-generate-output-schema/SKILL.md @@ -0,0 +1,415 @@ +--- +name: apify-generate-output-schema +description: Generate output schemas (dataset_schema.json, output_schema.json, key_value_store_schema.json) for an Apify Actor by analyzing its source code. Use when creating or updating Actor output schemas. +--- + +# Generate Actor Output Schema + +You are generating output schema files for an Apify Actor. The output schema tells Apify Console how to display run results. You will analyze the Actor's source code, create `dataset_schema.json`, `output_schema.json`, and `key_value_store_schema.json` (if the Actor uses key-value store), and update `actor.json`. + +## Core Principles + +- **Analyze code first**: Read the Actor's source to understand what data it actually pushes to the dataset — never guess +- **Every field is nullable**: APIs and websites are unpredictable — always set `"nullable": true` +- **Anonymize examples**: Never use real user IDs, usernames, or personal data in examples +- **Verify against code**: If TypeScript types exist, cross-check the schema against both the type definition AND the code that produces the values +- **Reuse existing patterns**: Before generating schemas, check if other Actors in the same repository already have output schemas — match their structure, naming conventions, description style, and formatting +- **Don't reinvent the wheel**: Reuse existing type definitions, interfaces, and utilities from the codebase instead of creating duplicate definitions + +--- + +## Phase 1: Discover Actor Structure + +**Goal**: Locate the Actor and understand its output + +Use the user's most recent request as the scope for this skill (which Actor to target, which subdirectory, any specific fields to focus on). If the scope is unclear, ask one clarifying question before continuing. + +**Actions**: +1. Create todo list with all phases +2. Find the `.actor/` directory containing `actor.json` +3. Read `actor.json` to understand the Actor's configuration +4. Check if `dataset_schema.json`, `output_schema.json`, and `key_value_store_schema.json` already exist +5. **Search for existing schemas in the repository**: Look for other `.actor/` directories or schema files (e.g., `**/dataset_schema.json`, `**/output_schema.json`, `**/key_value_store_schema.json`) to learn the repo's conventions — match their description style, field naming, example formatting, and overall structure +6. Find all places where data is pushed to the dataset: + - **JavaScript/TypeScript**: Search for `Actor.pushData(`, `dataset.pushData(`, `Dataset.pushData(` + - **Python**: Search for `Actor.push_data(`, `dataset.push_data(`, `Dataset.push_data(` +7. Find all places where data is stored in the key-value store: + - **JavaScript/TypeScript**: Search for `Actor.setValue(`, `keyValueStore.setValue(`, `KeyValueStore.setValue(` + - **Python**: Search for `Actor.set_value(`, `key_value_store.set_value(`, `KeyValueStore.set_value(` +8. Find output type definitions — **reuse them directly** instead of recreating from scratch: + - **TypeScript**: Look for output type interfaces/types (e.g., in `src/types/`, `src/types/output.ts`). If an interface or type already defines the output shape, derive the schema fields from it — do not create a parallel definition + - **Python**: Look for TypedDict, dataclass, or Pydantic model definitions. Use the existing field names, types, and docstrings as the source of truth +9. Check for existing shared schema utilities or helper functions in the codebase that handle schema generation or validation — reuse them rather than creating new logic +10. If inline `storages.dataset` or `storages.keyValueStore` config exists in `actor.json`, note it for migration + +Present findings to user: list all discovered dataset output fields, key-value store keys, their types, and where they come from. + +--- + +## Phase 2: Generate `dataset_schema.json` + +**Goal**: Create a complete dataset schema with field definitions and display views + +### File structure + +```json +{ + "actorSpecification": 1, + "fields": { + "$schema": "http://json-schema.org/draft-07/schema#", + "type": "object", + "properties": { + // ALL output fields here — every field the Actor can produce, + // not just the ones shown in the overview view + }, + "required": [], + "additionalProperties": true + }, + "views": { + "overview": { + "title": "Overview", + "description": "Most important fields at a glance", + "transformation": { + "fields": [ + // 8-12 most important field names + ] + }, + "display": { + "component": "table", + "properties": { + // Display config for each overview field + } + } + } + } +} +``` + +### Consistency with existing schemas + +If existing output schemas were found in the repository during Phase 1 (step 5), follow their conventions: +- Match the **description writing style** (sentence case vs. lowercase, period vs. no period, etc.) +- Match the **field naming convention** (camelCase vs. snake_case) — this must also match the actual keys produced by the Actor code +- Match the **example value style** (e.g., date formats, URL patterns, placeholder names) +- Match the **view structure** (number of fields in overview, display format choices) +- Match the **JSON formatting** (indentation, property ordering, spacing) — all schemas in the same repository must use identical formatting, including standalone Actors + +When the Actor code already has well-defined TypeScript interfaces or Python type classes, derive fields directly from those types rather than re-analyzing pushData/push_data calls from scratch. The type definition is the canonical source. + +### Hard rules (no exceptions) + +| Rule | Detail | +|------|--------| +| **All fields in `properties`** | The `fields.properties` object must contain **every** field the Actor can output, not just the fields shown in the overview view. The views section selects a subset for display — the `properties` section must be the complete superset | +| `"nullable": true` | On **every** field — APIs are unpredictable | +| `"additionalProperties": true` | On the **top-level `fields` object** AND on **every nested object** within `properties`. This is the most commonly missed rule — it must appear at both levels | +| `"required": []` | Always empty array — on the **top-level `fields` object** AND on **every nested object** within `properties` | +| Anonymized examples | No real user IDs, usernames, or content | +| `"type"` required with `"nullable"` | AJV rejects `nullable` without a `type` on the same field | + +> **Warning — most common mistakes**: +> 1. Only including fields that appear in the overview view. The `fields.properties` must list ALL output fields, even if they are not in the `views` section. +> 2. Only adding `"required": []` and `"additionalProperties": true` on nested object-type properties but forgetting them on the top-level `fields` object. Both levels need them. + +> **Note**: `nullable` is an Apify-specific extension to JSON Schema draft-07. It is intentional and correct. + +### Field type patterns + +**String field:** +```json +"title": { + "type": "string", + "description": "Title of the scraped item", + "nullable": true, + "example": "Example Item Title" +} +``` + +**Number field:** +```json +"viewCount": { + "type": "number", + "description": "Number of views", + "nullable": true, + "example": 15000 +} +``` + +**Boolean field:** +```json +"isVerified": { + "type": "boolean", + "description": "Whether the account is verified", + "nullable": true, + "example": true +} +``` + +**Array field:** +```json +"hashtags": { + "type": "array", + "description": "Hashtags associated with the item", + "items": { "type": "string" }, + "nullable": true, + "example": ["#example", "#demo"] +} +``` + +**Nested object field:** +```json +"authorInfo": { + "type": "object", + "description": "Information about the author", + "properties": { + "name": { "type": "string", "nullable": true }, + "url": { "type": "string", "nullable": true } + }, + "required": [], + "additionalProperties": true, + "nullable": true, + "example": { "name": "Example Author", "url": "https://example.com/author" } +} +``` + +**Enum field:** +```json +"contentType": { + "type": "string", + "description": "Type of content", + "enum": ["article", "video", "image"], + "nullable": true, + "example": "article" +} +``` + +**Union type (e.g., TypeScript `ObjectType | string`):** +```json +"metadata": { + "type": ["object", "string"], + "description": "Structured metadata object, or error string if unavailable", + "nullable": true, + "example": { "key": "value" } +} +``` + +### Anonymized example values + +Use realistic but generic values. Follow platform ID format conventions: + +| Field type | Example approach | +|---|---| +| IDs | Match platform format and length (e.g., 11 chars for YouTube video IDs) | +| Usernames | `"exampleuser"`, `"sampleuser123"` | +| Display names | `"Example Channel"`, `"Sample Author"` | +| URLs | Use platform's standard URL format with fake IDs | +| Dates | `"2025-01-15T12:00:00.000Z"` (ISO 8601) | +| Text content | Generic descriptive text, e.g., `"This is an example description."` | + +### Views section + +- `transformation.fields`: List 8–12 most important field names (order = column order in UI) +- `display.properties`: One entry per overview field with `label` and `format` +- Available formats: `"text"`, `"number"`, `"date"`, `"link"`, `"boolean"`, `"image"`, `"array"`, `"object"` + +Pick fields that give users the most useful at-a-glance summary of the data. + +--- + +## Phase 3: Generate `key_value_store_schema.json` (if applicable) + +**Goal**: Define key-value store collections if the Actor stores data in the key-value store + +> **Skip this phase** if no `Actor.setValue()` / `Actor.set_value()` calls were found in Phase 1 (beyond the default `INPUT` key). + +### File structure + +```json +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "", + "description": "", + "collections": { + "": { + "title": "", + "description": "", + "keyPrefix": "" + } + } +} +``` + +### How to identify collections + +Group the discovered `setValue` / `set_value` calls by key pattern: + +1. **Fixed keys** (e.g., `"RESULTS"`, `"summary"`) — use `"key"` (exact match) +2. **Dynamic keys with a prefix** (e.g., `"screenshot-${id}"`, `f"image-{name}"`) — use `"keyPrefix"` + +Each group becomes a collection. + +### Collection properties + +| Property | Required | Description | +|----------|----------|-------------| +| `title` | Yes | Shown in UI tabs | +| `description` | No | Shown in UI tooltips | +| `key` | Conditional | Exact key for single-key collections (use `key` OR `keyPrefix`, not both) | +| `keyPrefix` | Conditional | Prefix for multi-key collections (use `key` OR `keyPrefix`, not both) | +| `contentTypes` | No | Restrict allowed MIME types (e.g., `["image/jpeg"]`, `["application/json"]`) | +| `jsonSchema` | No | JSON Schema draft-07 for validating `application/json` content | + +### Examples + +**Single file output (e.g., a report):** +```json +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "Analysis Results", + "description": "Key-value store containing analysis output", + "collections": { + "report": { + "title": "Report", + "description": "Final analysis report", + "key": "REPORT", + "contentTypes": ["application/json"] + } + } +} +``` + +**Multiple files with prefix (e.g., screenshots):** +```json +{ + "actorKeyValueStoreSchemaVersion": 1, + "title": "Scraped Files", + "description": "Key-value store containing downloaded files and screenshots", + "collections": { + "screenshots": { + "title": "Screenshots", + "description": "Page screenshots captured during scraping", + "keyPrefix": "screenshot-", + "contentTypes": ["image/png", "image/jpeg"] + }, + "documents": { + "title": "Documents", + "description": "Downloaded document files", + "keyPrefix": "doc-", + "contentTypes": ["application/pdf", "text/html"] + } + } +} +``` + +--- + +## Phase 4: Generate `output_schema.json` + +**Goal**: Create the output schema that tells Apify Console where to find results + +For most Actors that push data to a dataset, this is a minimal file: + +```json +{ + "actorOutputSchemaVersion": 1, + "title": "", + "description": "", + "properties": { + "dataset": { + "type": "string", + "title": "Results", + "description": "Dataset containing all scraped data", + "template": "{{links.apiDefaultDatasetUrl}}/items" + } + } +} +``` + +> **Critical**: Each property entry **must** include `"type": "string"` — this is an Apify-specific convention. The Apify meta-validator rejects properties without it (and rejects `"type": "object"` — only `"string"` is valid here). + +If `key_value_store_schema.json` was generated in Phase 3, add a second property: +```json +"files": { + "type": "string", + "title": "Files", + "description": "Key-value store containing downloaded files", + "template": "{{links.apiDefaultKeyValueStoreUrl}}/keys" +} +``` + +### Available template variables + +- `{{links.apiDefaultDatasetUrl}}` — API URL of default dataset +- `{{links.apiDefaultKeyValueStoreUrl}}` — API URL of default key-value store +- `{{links.publicRunUrl}}` — Public run URL +- `{{links.consoleRunUrl}}` — Console run URL +- `{{links.apiRunUrl}}` — API run URL +- `{{links.containerRunUrl}}` — URL of webserver running inside the run +- `{{run.defaultDatasetId}}` — ID of the default dataset +- `{{run.defaultKeyValueStoreId}}` — ID of the default key-value store + +--- + +## Phase 5: Update `actor.json` + +**Goal**: Wire the schema files into the Actor configuration + +**Actions**: +1. Read the current `actor.json` +2. Add or update the `storages.dataset` reference: + ```json + "storages": { + "dataset": "./dataset_schema.json" + } + ``` +3. If `key_value_store_schema.json` was generated, add the reference: + ```json + "storages": { + "dataset": "./dataset_schema.json", + "keyValueStore": "./key_value_store_schema.json" + } + ``` +4. Add or update the `output` reference: + ```json + "output": "./output_schema.json" + ``` +5. If `actor.json` had inline `storages.dataset` or `storages.keyValueStore` objects (not string paths), migrate their content into the respective schema files and replace the inline objects with file path strings + +--- + +## Phase 6: Review and Validate + +**Goal**: Ensure correctness and completeness + +**Checklist**: +- [ ] **Every** output field from the source code is in `dataset_schema.json` `fields.properties` — not just the overview view fields but ALL fields the Actor can produce +- [ ] Every field has `"nullable": true` +- [ ] The **top-level `fields` object** has both `"additionalProperties": true` and `"required": []` +- [ ] Every **nested object** within `properties` also has `"additionalProperties": true` and `"required": []` +- [ ] Every field has a `"description"` and an `"example"` +- [ ] All example values are anonymized +- [ ] `"type"` is present on every field that has `"nullable"` +- [ ] Views list 8–12 most useful fields with correct display formats +- [ ] `output_schema.json` has `"type": "string"` on every property +- [ ] If key-value store is used: `key_value_store_schema.json` has collections matching all `setValue`/`set_value` calls +- [ ] If key-value store is used: each collection uses either `key` or `keyPrefix` (not both) +- [ ] `actor.json` references all generated schema files +- [ ] Schema field names match the actual keys in the code (camelCase/snake_case consistency) +- [ ] If existing schemas were found in the repo, the new schema follows their conventions (description style, example format, view structure) +- [ ] Schema fields are derived from existing type definitions (interfaces, TypedDicts, dataclasses) where available — no duplicated or divergent field definitions + +Present the generated schemas to the user for review before writing them. + +--- + +## Phase 7: Summary + +**Goal**: Document what was created + +Report: +- Files created or updated +- Number of fields in the dataset schema +- Number of collections in the key-value store schema (if generated) +- Fields selected for the overview view +- Any fields that need user clarification (ambiguous types, unclear nullability) +- Suggested next steps (test locally with `apify run`, verify output tab in Console) \ No newline at end of file diff --git a/skills/apify-sdk-integration/SKILL.md b/skills/apify-sdk-integration/SKILL.md new file mode 100644 index 0000000..2c49751 --- /dev/null +++ b/skills/apify-sdk-integration/SKILL.md @@ -0,0 +1,210 @@ +--- +name: apify-sdk-integration +description: Integrate Apify into an existing JavaScript/TypeScript or Python application using the apify-client package. Use when adding web scraping, automation, or data extraction capabilities to an existing app via the Apify API. +--- + +# Apify SDK Integration + +Add Apify Actor execution to an existing application. This skill covers the `apify-client` package for JS/TS and Python, plus the REST API for other languages. + +## When to Use This Skill + +- Adding web scraping or automation to an existing app +- Calling Apify Actors programmatically from application code +- Building a product that uses Apify as a backend service +- Integrating Actor results into a data pipeline + +## Critical: Package Naming + +> **`apify-client`** is the API client for **calling** Actors from your app. +> **`apify`** is the SDK for **building** Actors (wrong package for this use case). +> +> Always install `apify-client`. Never install `apify` for integration work. + +## Prerequisites + +The user needs an `APIFY_TOKEN`. Direct them to **Console > Settings > Integrations** at https://console.apify.com/settings/integrations to create one. If they don't have an account: https://console.apify.com/sign-up (free, no credit card). + +Store the token securely — environment variable or secrets manager, never hardcoded. + +## Finding the Right Actor + +Before writing integration code, find the Actor that fits the user's needs. Use the MCP tools if available: +- `search-actors` — search the Apify Store by keyword +- `fetch-actor-details` — get the Actor's input schema, output format, and pricing + +Alternatively, browse https://apify.com/store. Append `.md` to any Actor's Store URL to get its docs in markdown. + +## JavaScript / TypeScript + +### Install + +```bash +npm install apify-client +``` + +### Synchronous Execution (wait for results) + +```typescript +import { ApifyClient } from 'apify-client'; + +const client = new ApifyClient({ token: process.env.APIFY_TOKEN }); + +const run = await client.actor('apify/web-scraper').call({ + startUrls: [{ url: 'https://example.com' }], + maxPagesPerCrawl: 10, +}); + +const { items } = await client.dataset(run.defaultDatasetId).listItems(); +``` + +`.call()` blocks until the Actor finishes. Use for short-running Actors (under a few minutes). + +### Asynchronous Execution (start and poll/retrieve later) + +```typescript +const run = await client.actor('apify/web-scraper').start({ + startUrls: [{ url: 'https://example.com' }], +}); + +// Poll for completion +const finishedRun = await client.run(run.id).waitForFinish(); + +// Retrieve results +const { items } = await client.dataset(finishedRun.defaultDatasetId).listItems(); +``` + +Use `.start()` + `.waitForFinish()` for long-running Actors or when you need the run ID immediately. + +### Retrieving Results + +```typescript +// Dataset items (structured data from pushData) +const { items } = await client.dataset(run.defaultDatasetId).listItems({ + limit: 100, + offset: 0, +}); + +// Key-value store (files, screenshots, etc.) +const record = await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT'); +``` + +### Error Handling + +```typescript +try { + const run = await client.actor('apify/web-scraper').call(input); + + if (run.status !== 'SUCCEEDED') { + const log = await client.log(run.id).get(); + throw new Error(`Actor failed with status ${run.status}: ${log}`); + } + + const { items } = await client.dataset(run.defaultDatasetId).listItems(); +} catch (error) { + if (error.message?.includes('not found')) { + // Actor ID is wrong or Actor was deleted + } else if (error.statusCode === 401) { + // Invalid or missing APIFY_TOKEN + } + throw error; +} +``` + +## Python + +### Install + +```bash +pip install apify-client +``` + +### Synchronous Execution + +```python +from apify_client import ApifyClient +import os + +client = ApifyClient(token=os.environ['APIFY_TOKEN']) + +run = client.actor('apify/web-scraper').call(run_input={ + 'startUrls': [{'url': 'https://example.com'}], + 'maxPagesPerCrawl': 10, +}) + +items = client.dataset(run['defaultDatasetId']).list_items().items +``` + +### Asynchronous Execution + +```python +run = client.actor('apify/web-scraper').start(run_input={ + 'startUrls': [{'url': 'https://example.com'}], +}) + +# Poll for completion +finished_run = client.run(run['id']).wait_for_finish() + +items = client.dataset(finished_run['defaultDatasetId']).list_items().items +``` + +### Async Client (asyncio) + +```python +from apify_client import ApifyClientAsync + +client = ApifyClientAsync(token=os.environ['APIFY_TOKEN']) + +run = await client.actor('apify/web-scraper').call(run_input={ + 'startUrls': [{'url': 'https://example.com'}], +}) + +items = (await client.dataset(run['defaultDatasetId']).list_items()).items +``` + +## REST API (Any Language) + +For languages without an official client, use the REST API directly. + +### Start a Run + +``` +POST https://api.apify.com/v2/acts/{actorId}/runs +Authorization: Bearer +Content-Type: application/json + +{ "startUrls": [{ "url": "https://example.com" }] } +``` + +### Get Run Status + +``` +GET https://api.apify.com/v2/acts/{actorId}/runs/{runId} +Authorization: Bearer +``` + +### Get Dataset Items + +``` +GET https://api.apify.com/v2/datasets/{datasetId}/items?format=json +Authorization: Bearer +``` + +Full API reference: https://docs.apify.com/api/v2 + +## Best Practices + +- **Set timeouts:** Pass `timeoutSecs` in the Actor input or use `waitSecs` on `.call()` to avoid indefinite waits. +- **Paginate large datasets:** Use `limit` and `offset` when retrieving dataset items. Default limit is 250K items. +- **Reuse clients:** Create one `ApifyClient` instance and reuse it across calls. +- **Handle Actor-specific input:** Every Actor has its own input schema. Use `fetch-actor-details` MCP tool or append `.md` to the Actor's Store URL to get the schema before constructing input. + +## Documentation + +- Apify API client for JS: https://docs.apify.com/api/client/js +- Apify API client for Python: https://docs.apify.com/api/client/python +- REST API reference: https://docs.apify.com/api/v2 +- Apify docs (LLM-friendly): https://docs.apify.com/llms.txt +- Apify docs (full): https://docs.apify.com/llms-full.txt + +If the Apify MCP server is available, use `search-apify-docs` and `fetch-apify-docs` tools for contextual documentation lookups during development. diff --git a/skills/apify-ultimate-scraper/SKILL.md b/skills/apify-ultimate-scraper/SKILL.md new file mode 100644 index 0000000..9754737 --- /dev/null +++ b/skills/apify-ultimate-scraper/SKILL.md @@ -0,0 +1,176 @@ +--- +name: apify-ultimate-scraper +description: Universal AI-powered web scraper for any platform. Scrape data from Instagram, Facebook, TikTok, YouTube, LinkedIn, X/Twitter, Google Maps, Google Search, Google Trends, Reddit, Airbnb, Yelp, and 15+ more platforms. Use for lead generation, brand monitoring, competitor analysis, influencer discovery, trend research, content analytics, audience analysis, review analysis, SEO intelligence, recruitment, or any data extraction task. +--- + +# Universal web scraper + +AI-driven data extraction from ~100 Actors across 15+ platforms via the Apify CLI. + +**Rule: Pass `--json` and redirect stderr with `2>/dev/null` on data-returning commands** (`actors call`, `actors start`, `actors info`, `actors search`, `datasets get-items`, `runs info`). JSON output is stable across CLI versions. stderr contains progress messages and version warnings that break JSON parsers if not redirected. + +This rule does **not** apply to status/auth commands (`apify info`, `apify --version`, `apify login`). For those, use `2>&1` so authentication and version errors are visible. + +**Exception:** if `--input` returns no data, re-run with `2>&1` to confirm whether the cause is a missing schema vs. a network/auth error. + +## Prerequisites + +- Apify CLI v1.4.0+ (`npm install -g apify-cli`) +- Authenticated session (see below) + +## Authentication + +If a CLI command fails with an auth error, authenticate using one of these methods: + +1. **OAuth (interactive):** `apify login` (opens browser) +2. **Environment variable:** `export APIFY_TOKEN=your_token_here` +3. **From .env file:** `source .env` (if the file contains `APIFY_TOKEN=...`) + +Generate token: https://console.apify.com/settings/integrations + +## Workflow + +### Step 0: Verify CLI readiness before doing anything else + +Before using the Apify CLI, always verify the local environment: + +1. Check that the CLI is installed: +```bash + apify --help +``` +If this fails, install the CLI first: +```bash + npm install -g apify-cli +``` +2. Check that the CLI is authenticated: + +```bash + # Auth check — do NOT pipe to /dev/null, you need to see errors + apify info 2>&1 +``` + If this shows the user is not logged in, instruct them to authenticate with a token: + +```bash + apify login --token TOKEN +``` + +3. Run Apify CLI commands with `all` permissions when needed by the agent sandbox. + +4. Assume many Apify commands block with zero output until completion. For blocking runs, set `block_until_ms` to at least `60000`. + +5. For long or unknown-duration runs, prefer the async pattern: + +```bash + apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null +``` + + Then poll the run status: + +```bash + apify info actor-runs/RUN_ID --json +``` + + Check `.status` for `SUCCEEDED` or `FAILED`. + +### Step 1: Understand goal and select Actor + +Identify the target platform and use case. Read `references/actor-index.md` to find the right Actor. +Prefer `apify`-tier actors; use `community`-tier only when no `apify` actor covers the task. +For input schemas, fetch dynamically: `apify actors info "ACTOR_ID" --input --json 2>/dev/null` +If the output is empty, re-run without the redirect (`2>&1`) to surface auth or network errors before proceeding. + +If the task involves a multi-step pipeline, also read the matching workflow guide: + +| Task involves... | Read | +|-----------------|------| +| leads, contacts, emails, B2B | `references/workflows/lead-generation.md` | +| competitor, ads, pricing | `references/workflows/competitive-intel.md` | +| influencer, creator | `references/workflows/influencer-vetting.md` | +| brand, mentions, sentiment | `references/workflows/brand-monitoring.md` | +| reviews, ratings, reputation | `references/workflows/review-analysis.md` | +| SEO, SERP, crawl, content, RAG | `references/workflows/content-and-seo.md` | +| analytics, engagement, performance | `references/workflows/social-media-analytics.md` | +| trends, keywords, hashtags | `references/workflows/trend-research.md` | +| jobs, recruiting, candidates | `references/workflows/job-market-and-recruitment.md` | +| real estate, listings, hotels | `references/workflows/real-estate-and-hospitality.md` | +| price monitoring, e-commerce, products | `references/workflows/ecommerce-price-monitoring.md` | +| contact enrichment, email extraction | `references/workflows/contact-enrichment.md` | +| knowledge base, RAG, LLM data feed | `references/workflows/knowledge-base-and-rag.md` | +| company research, due diligence | `references/workflows/company-research.md` | + +If no Actor matches in the index, search dynamically: + + apify actors search "KEYWORDS" --json --limit 10 2>/dev/null + +From results: `items[].username`/`items[].name` (Actor ID), `items[].title`, `items[].stats.totalUsers30Days`, `items[].currentPricingInfo.pricingModel`. + +### Step 2: Fetch Actor schema and check gotchas + +Some Actors don't register an input schema with the platform (their schema lives in code). Try schema sources in this order — fall through on empty/error: + +1. **Input schema (human-readable):** +```bash + apify actors info "ACTOR_ID" --input 2>/dev/null +``` + If output is `Error: No input schema found for this Actor`, skip to source 2. + +2. **Input schema (JSON keys only):** +```bash + apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties // empty | keys' +``` + Empty result means no registered schema — fall through to source 3. To drill into a specific field: +```bash + apify actors info "ACTOR_ID" --input --json 2>/dev/null | jq '.input.schema.properties.FIELD_NAME' +``` + +3. **README fallback** (always works, contains usage examples): +```bash + apify actors info "ACTOR_ID" --readme 2>/dev/null +``` + Grep the README for an "Input" / "Example input" section to copy the JSON shape. + +4. **Last resort — call with minimal known input** (e.g. `{"startUrls":[{"url":"..."}]}` for crawlers) and let the Actor surface validation errors that reveal required fields. See `references/gotchas.md` for known-good minimal inputs for common Actors. + +Also read `references/gotchas.md` to check for common pitfalls and cost guardrails for the selected Actor. + +### Step 3: Configure and run + +**Skip user preferences** for simple lookups (e.g., "Nike's follower count"). Go straight to running with quick answer mode. + +For larger tasks, confirm output format (quick answer / CSV / JSON) and result count. + +Before starting the run, double-check whether the task is short enough for a blocking call or should use the async pattern from Step 0. + +**Standard run (blocking):** +```bash + apify actors call "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null +``` +From output: `.id` (run ID), `.status`, `.defaultDatasetId`, `.stats.durationMillis` + +**Fetch results:** +```bash + apify datasets get-items DATASET_ID --format json +``` +For CSV: `apify datasets get-items DATASET_ID --format csv` + +**Quick answer mode:** Fetch results as JSON, pick top 5, present formatted in chat. + +**Save to file:** Fetch results, use Write tool to save as `YYYY-MM-DD_descriptive-name.csv` or `.json`. + +**Large/long-running scrapes:** +```bash + apify actors start "ACTOR_ID" -i 'JSON_INPUT' --json 2>/dev/null +``` +Poll: `apify info actor-runs/RUN_ID --json` (check `.status` for `SUCCEEDED` or `FAILED`). + +### Step 4: Deliver results + +Report: result count, file location (if saved), key data fields, and links: +- Dataset: `https://console.apify.com/storage/datasets/DATASET_ID` +- Run: `https://console.apify.com/actors/runs/RUN_ID` + +For multi-step workflows: suggest the next pipeline step from the workflow guide. + +## Troubleshooting + +Common errors and pitfalls are documented in `references/gotchas.md`. Read it before running PPE (pay-per-event) Actors. diff --git a/skills/apify-ultimate-scraper/references/actor-index.md b/skills/apify-ultimate-scraper/references/actor-index.md new file mode 100644 index 0000000..76d97ab --- /dev/null +++ b/skills/apify-ultimate-scraper/references/actor-index.md @@ -0,0 +1,199 @@ +## Instagram + +| Actor | Tier | Best for | +|-------|------|----------| +| apify/instagram-scraper | apify | all Instagram data | +| apify/instagram-profile-scraper | apify | profiles, followers, bio | +| apify/instagram-post-scraper | apify | posts, engagement metrics | +| apify/instagram-comment-scraper | apify | post and reel comments | +| apify/instagram-hashtag-scraper | apify | posts by hashtag | +| apify/instagram-hashtag-analytics-scraper | apify | hashtag metrics, trends | +| apify/instagram-reel-scraper | apify | reels, transcripts, engagement | +| apify/instagram-api-scraper | apify | API-based, no login | +| apify/instagram-search-scraper | apify | search users, places | +| apify/instagram-tagged-scraper | apify | tagged/mentioned posts | +| apify/instagram-topic-scraper | apify | posts by topic | +| apify/instagram-followers-count-scraper | apify | follower count tracking | +| apify/export-instagram-comments-posts | apify | bulk posts + comments | + +## Facebook + +| Actor | Tier | Best for | +|-------|------|----------| +| apify/facebook-posts-scraper | apify | posts, videos, engagement | +| apify/facebook-comments-scraper | apify | comment extraction | +| apify/facebook-likes-scraper | apify | reactions, liker info | +| apify/facebook-groups-scraper | apify | public group content | +| apify/facebook-events-scraper | apify | events, attendees | +| apify/facebook-reels-scraper | apify | reels, engagement | +| apify/facebook-photos-scraper | apify | photos with OCR | +| apify/facebook-search-scraper | apify | page search | +| apify/facebook-marketplace-scraper | apify | marketplace listings | +| apify/facebook-followers-following-scraper | apify | follower lists | +| apify/facebook-video-search-scraper | apify | video search | +| apify/facebook-ads-scraper | apify | ad library, creatives | +| apify/facebook-page-contact-information | apify | page contact info | +| apify/facebook-reviews-scraper | apify | page reviews | +| apify/facebook-hashtag-scraper | apify | hashtag posts | +| apify/threads-profile-api-scraper | apify | Threads profiles | + +## TikTok + +| Actor | Tier | Best for | +|-------|------|----------| +| clockworks/tiktok-scraper | apify | all TikTok data | +| clockworks/tiktok-profile-scraper | apify | profiles, videos | +| clockworks/tiktok-video-scraper | apify | video details, metrics | +| clockworks/tiktok-comments-scraper | apify | video comments | +| clockworks/tiktok-hashtag-scraper | apify | videos by hashtag | +| clockworks/tiktok-followers-scraper | apify | follower profiles | +| clockworks/tiktok-user-search-scraper | apify | user search | +| clockworks/tiktok-sound-scraper | apify | videos by sound | +| clockworks/free-tiktok-scraper | apify | free tier extraction | +| clockworks/tiktok-ads-scraper | apify | hashtag analytics | +| clockworks/tiktok-trends-scraper | apify | trending content | +| clockworks/tiktok-explore-scraper | apify | explore categories | +| clockworks/tiktok-discover-scraper | apify | discover by hashtag | + +## YouTube + +| Actor | Tier | Best for | +|-------|------|----------| +| streamers/youtube-scraper | apify | videos, metrics | +| streamers/youtube-channel-scraper | apify | channel info | +| streamers/youtube-comments-scraper | apify | video comments | +| streamers/youtube-shorts-scraper | apify | shorts data | +| streamers/youtube-video-scraper-by-hashtag | apify | videos by hashtag | +| streamers/youtube-video-downloader | apify | video download | +| curious_coder/youtube-transcript-scraper | community | transcripts, captions | + +## X/Twitter + +| Actor | Tier | Best for | +|-------|------|----------| +| apidojo/tweet-scraper | community | tweet search | +| apidojo/twitter-scraper-lite | community | comprehensive, no limits | +| apidojo/twitter-user-scraper | community | user profiles | +| apidojo/twitter-profile-scraper | community | profiles + recent tweets | +| apidojo/twitter-list-scraper | community | tweets from lists | + +## LinkedIn + +| Actor | Tier | Best for | +|-------|------|----------| +| harvestapi/linkedin-profile-search | community | find profiles | +| harvestapi/linkedin-profile-scraper | community | profile with email | +| harvestapi/linkedin-company | community | company details | +| harvestapi/linkedin-company-employees | community | employee lists | +| harvestapi/linkedin-company-posts | community | company page posts | +| harvestapi/linkedin-profile-posts | community | profile posts | +| harvestapi/linkedin-job-search | community | job listings | +| harvestapi/linkedin-post-search | community | post search | +| harvestapi/linkedin-post-comments | community | post comments | +| harvestapi/linkedin-profile-search-by-name | community | find by name | +| harvestapi/linkedin-profile-search-by-services | community | find by service | +| apimaestro/linkedin-companies-search-scraper | community | company search | +| apimaestro/linkedin-company-detail | community | company deep data | +| apimaestro/linkedin-jobs-scraper-api | community | job search | +| apimaestro/linkedin-job-detail | community | job details | +| apimaestro/linkedin-batch-profile-posts-scraper | community | batch profile posts | +| apimaestro/linkedin-post-reshares | community | post reshares | +| apimaestro/linkedin-post-detail | community | post details | +| apimaestro/linkedin-profile-full-sections-scraper | community | full profile data | +| dev_fusion/linkedin-profile-scraper | community | mass scraping + email | + +## Google Maps + +| Actor | Tier | Best for | +|-------|------|----------| +| compass/crawler-google-places | apify | business listings | +| compass/google-maps-extractor | apify | detailed business data | +| compass/Google-Maps-Reviews-Scraper | apify | reviews, ratings | +| compass/enrich-google-maps-dataset-with-contacts | apify | email enrichment | +| compass/contact-details-scraper-standby | apify | quick contact extract | +| lukaskrivka/google-maps-with-contact-details | community | listings + contacts | +| curious_coder/google-maps-reviews-scraper | community | cheap review scraping | + +## Google Search and Trends + +| Actor | Tier | Best for | +|-------|------|----------| +| apify/google-search-scraper | apify | SERP, ads, AI overviews | +| apify/google-trends-scraper | apify | trend data | +| tri_angle/bing-search-scraper | apify | Bing SERP data | + +## Reviews (cross-platform) + +| Actor | Tier | Best for | +|-------|------|----------| +| tri_angle/hotel-review-aggregator | apify | 7-platform hotel reviews | +| tri_angle/restaurant-review-aggregator | apify | 6-platform restaurant reviews | +| tri_angle/yelp-scraper | apify | Yelp business data | +| tri_angle/yelp-review-scraper | apify | Yelp reviews | +| tri_angle/get-tripadvisor-urls | apify | find TripAdvisor URLs | +| tri_angle/get-yelp-urls | apify | find Yelp URLs | +| tri_angle/airbnb-reviews-scraper | apify | Airbnb reviews | +| tri_angle/social-media-sentiment-analysis-tool | apify | sentiment analysis | + +## Real estate and hospitality + +| Actor | Tier | Best for | +|-------|------|----------| +| tri_angle/airbnb-scraper | apify | Airbnb listings | +| tri_angle/new-fast-airbnb-scraper | apify | fast Airbnb search | +| tri_angle/airbnb-rooms-urls-scraper | apify | detailed room data | +| tri_angle/redfin-search | apify | Redfin property search | +| tri_angle/redfin-detail | apify | Redfin property details | +| tri_angle/real-estate-aggregator | apify | multi-source listings | +| tri_angle/fast-zoopla-properties-scraper | apify | UK properties | +| tri_angle/doordash-store-details-scraper | apify | DoorDash stores | +| tri_angle/cargurus-zipcode-search-scraper | apify | CarGurus listings | +| tri_angle/carmax-zipcode-search-scraper | apify | Carmax listings | + +## SEO tools + +| Actor | Tier | Best for | +|-------|------|----------| +| radeance/similarweb-scraper | community | traffic, rankings | +| radeance/ahrefs-scraper | community | backlinks, keywords | +| radeance/semrush-scraper | community | domain authority | +| radeance/moz-scraper | community | DA, spam score | +| radeance/ubersuggest-scraper | community | keyword suggestions | +| radeance/se-ranking-scraper | community | keyword CPC | + +## Content and web crawling + +| Actor | Tier | Best for | +|-------|------|----------| +| apify/website-content-crawler | apify | clean text for AI | +| apify/rag-web-browser | apify | RAG pipelines | +| apify/web-scraper | apify | general web scraping | +| apify/cheerio-scraper | apify | fast HTML parsing | +| apify/playwright-scraper | apify | JS-heavy sites | +| apify/camoufox-scraper | apify | anti-bot sites | +| apify/sitemap-extractor | apify | sitemap URLs | +| lukaskrivka/article-extractor-smart | community | article extraction | + +## Other platforms + +| Actor | Tier | Best for | +|-------|------|----------| +| tri_angle/telegram-scraper | apify | Telegram messages | +| tri_angle/snapchat-scraper | apify | Snapchat profiles | +| tri_angle/snapchat-spotlight-scraper | apify | Snapchat Spotlight | +| tri_angle/truth-scraper | apify | Truth Social | +| tri_angle/social-media-finder | apify | cross-platform search | +| tri_angle/website-changes-detector | apify | website monitoring | +| tri_angle/e-commerce-product-matching-tool | apify | product matching | +| trudax/reddit-scraper-lite | community | Reddit posts | +| janbuchar/github-contributors-scraper | community | GitHub contributors | + +## Enrichment and contacts + +| Actor | Tier | Best for | +|-------|------|----------| +| apify/social-media-leads-analyzer | apify | emails from websites | +| apify/social-media-hashtag-research | apify | cross-platform hashtags | +| apify/e-commerce-scraping-tool | apify | product data enrichment | +| vdrmota/contact-info-scraper | community | contact extraction | +| code_crafter/leads-finder | community | B2B leads | diff --git a/skills/apify-ultimate-scraper/references/gotchas.md b/skills/apify-ultimate-scraper/references/gotchas.md new file mode 100644 index 0000000..1ffba78 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/gotchas.md @@ -0,0 +1,119 @@ +# Gotchas and cost guardrails + +## Pricing models + +| Model | How it works | Action before running | +|-------|-------------|----------------------| +| FREE | No per-result cost, only platform compute | None needed | +| PAY_PER_EVENT (PPE) | Charged per result item | MUST estimate cost first | +| FLAT_PRICE_PER_MONTH | Monthly subscription | Verify user has active subscription | + +To check an Actor's pricing: + + apify actors info "ACTOR_ID" --json + +Read `.currentPricingInfo.pricingModel` and `.currentPricingInfo.pricePerEvent`. + +## Cost estimation protocol + +Before running any PPE Actor: + +1. Get the per-event price from Actor info (`.currentPricingInfo.pricePerEvent`) +2. Multiply by the requested result count +3. Present the estimate to the user with this disclaimer: + +> **Estimated cost: ~$X for Y results.** This is a rough estimate only - actual costs can vary significantly depending on the Actor, data complexity, retries, and platform changes. Always check your Apify billing dashboard for actual charges. + +4. If estimate > $5: warn explicitly +5. If estimate > $20: require explicit user confirmation before proceeding + +**Important:** Cost estimates in the workflow guides are approximate and may be inaccurate. Always present them as rough guidance with the disclaimer above, never as exact amounts. + +## Common pitfalls + +**CLI version warning on stderr** +Outdated Apify CLI installations print a "You are using an old version of Apify CLI" banner to stderr on every invocation. This breaks naive JSON parsing when stderr is not redirected. Always redirect stderr explicitly on data-returning commands (`2>/dev/null`). For status/auth commands (`apify info`, `apify login`), use `2>&1` so errors are visible rather than swallowed. + +**jq exit-code 5 on null paths** +`jq` exits with code 5 when a filter selects `null` on a strict path (e.g. `.input.schema.properties` when `.input` is missing). This causes the shell to treat the command as failed, masking the real issue (no registered schema). Use the `// empty` alternative operator in jq filters — e.g. `jq '.input.schema.properties // empty | keys'` — so the output is simply empty on a missing path and the agent can branch on empty output instead of a non-zero exit code. + +**Cookie-dependent Actors** +Some social media scrapers require cookies or login sessions. If an Actor returns auth errors or empty results unexpectedly, check its README: + + apify actors info "ACTOR_ID" --readme + +Look for mentions of "cookies", "login", "session", or "proxy". + +**Rate limiting on large scrapes** +Platforms throttle or block large-volume scraping. Mitigations: +- Use proxy configuration when available: `"proxyConfiguration": {"useApifyProxy": true}` +- Set reasonable concurrency limits (check the Actor's `maxConcurrency` input) +- For 1,000+ results, suggest splitting into smaller batches + +**Empty results** +Common causes: +- Too-narrow search query or geo-restriction (try broader terms) +- Platform blocking without proxy (enable Apify Proxy) +- Actor requires cookies/login but none provided +- Wrong input field name (always verify with `--input --json`) + +**maxResults vs maxCrawledPages** +Different Actors use different limit field names. Common variants: +- `maxResults`, `resultsLimit`, `maxItems` - limit output items +- `maxCrawledPages`, `maxRequestsPerCrawl` - limit pages visited +Always fetch the input schema to find the correct field for the specific Actor. + +**Deprecated Actors** +Check `.isDeprecated` in `apify actors info --json`. If `true`: +1. Search for alternatives: `apify actors search "SIMILAR_KEYWORDS" --json` +2. Prefer `apify` tier replacements over `community` alternatives + +**LinkedIn pricing** +LinkedIn Actors are all PPE and vary significantly: +- `harvestapi/` Actors: generally cheaper ($0.001-0.01/result) +- `apimaestro/` Actors: generally more expensive ($0.005-0.02/result) +- `dev_fusion/` Actors: mid-range, useful for mass scraping with email enrichment +Always compare pricing before selecting a LinkedIn Actor. + +**SEO tool pricing** +`radeance/` SEO scrapers (SimilarWeb, Ahrefs, SEMrush, Moz) have the highest per-result costs ($0.005-0.0275/result). For large-scale SEO analysis, estimate costs carefully and suggest batching. + +## Error recovery + +| Symptom | Likely cause | Fix | +|---------|-------------|-----| +| `status: FAILED` in run output | Actor crashed or input invalid | Read `.statusMessage` in JSON; check run log at `https://console.apify.com/actors/runs/RUN_ID/log` | +| `isDeprecated: true` in Actor info | Actor is end-of-life | Search for replacement: `apify actors search "KEYWORDS" --json` | +| Empty dataset (0 items) | Query too narrow, geo-restriction, or anti-bot block | Broaden search terms; enable Apify Proxy; check Actor README with `apify actors info ACTOR_ID --readme` | +| Run takes >10 minutes | Large scrape or slow target site | Switch to fire-and-forget: `apify actors start --json`, poll with `apify runs info RUN_ID --json` | + +## Why Apify Actors vs raw HTTP scraping + +Many n8n and automation workflows use raw HTTP Request nodes or self-hosted Puppeteer for web scraping. These hit common walls that Apify Actors handle transparently: + +**Cloudflare and WAF bypass** +Raw HTTP requests fail on sites with Cloudflare Turnstile, DataDome, or other WAFs. Apify Actors use residential proxies and browser fingerprint rotation automatically. For the toughest sites, use `apify/camoufox-scraper`. + +**JavaScript-rendered pages (SPAs)** +React, Vue, and Angular sites return empty HTML to plain HTTP requests. Apify's `apify/playwright-scraper` and `apify/camoufox-scraper` fully render JavaScript before extracting data. + +**Anti-bot fingerprinting** +Even headless browsers get detected via TLS fingerprints (JA3 hashes). Apify's browser pool rotates fingerprints across requests automatically. + +**Session and cookie management** +Social media platforms (LinkedIn, Instagram) require persistent sessions. Social media Actors handle cookie management and session rotation internally. + +**Scaling without infrastructure** +Self-hosted Puppeteer at scale requires 4-8 GB RAM per browser instance. Apify Actors run on serverless infrastructure - no browser pool management, no RAM provisioning, no Docker orchestration. + +## Platform-specific rate limits + +**Instagram:** Aggressive rate limiting. Keep `maxResults` under 200 per run for profile/post scrapers. Use delays between runs. Instagram API scrapers (`apify/instagram-api-scraper`) have higher limits than browser-based ones. + +**LinkedIn:** All LinkedIn Actors are community-maintained and PPE. LinkedIn actively blocks scraping at scale. Keep batch sizes under 100 profiles. Space runs at least 5 minutes apart. Expect occasional empty results. + +**TikTok:** Anti-bot measures increasing. `clockworks/tiktok-scraper` handles most cases. For blocked regions, enable Apify Proxy with residential IPs. + +**Google Maps:** Generally stable. Set `language: "en"` explicitly for consistent results. Large-area searches may return different results depending on zoom level - use specific location queries over broad city names. + +**Amazon/E-commerce:** Heavy anti-bot. The `apify/e-commerce-scraping-tool` handles this via built-in proxy rotation. Raw HTTP requests will fail. diff --git a/skills/apify-ultimate-scraper/references/workflows/brand-monitoring.md b/skills/apify-ultimate-scraper/references/workflows/brand-monitoring.md new file mode 100644 index 0000000..89e82d0 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/brand-monitoring.md @@ -0,0 +1,84 @@ +# Brand monitoring workflows + +## Cross-platform brand mention tracking +**When:** User wants to monitor brand mentions, hashtags, or sentiment across social platforms. + +### Pipeline (run each independently, combine results) +1. **Instagram mentions** -> `apify/instagram-tagged-scraper` + - Key input: `username` (brand handle) +2. **Instagram hashtags** -> `apify/instagram-hashtag-scraper` + - Key input: `hashtags` (branded hashtags) +3. **X/Twitter mentions** -> `apidojo/tweet-scraper` + - Key input: `searchTerms` (brand name, handle, hashtags) +4. **Reddit mentions** -> `trudax/reddit-scraper-lite` + - Key input: `searchQuery` (brand name) + +### Output fields +Instagram: `caption`, `likesCount`, `commentsCount`, `timestamp`, `ownerUsername` +X/Twitter: `text`, `retweetCount`, `likeCount`, `replyCount`, `createdAt`, `author` +Reddit: `title`, `body`, `score`, `numComments`, `subreddit`, `createdAt` + +### Gotcha +This is a parallel workflow, not sequential. Run each Actor independently. Combine results by date for a timeline view. + +## Twitter/X real-time mention routing +**When:** User wants to route brand mentions on X to the right team channel - negative to support, positive to wins - with sentiment scoring. + +### Pipeline +1. **Collect tweets** -> `apidojo/tweet-scraper` + - Key input: `searchTerms` (brand name + variants), `maxItems`, `since` (ISO date for incremental runs) +2. **Score sentiment** -> `tri_angle/social-media-sentiment-analysis-tool` + - Pipe: `results[].url` -> `urls` + - Key input: `urls`, `platforms` + +### Output fields +Step 1: `text`, `author.userName`, `createdAt`, `likeCount`, `retweetCount`, `url` +Step 2: `sentiment` (positive/negative/neutral), `score`, `text`, `platform` + +### Gotcha +Use `since` on each run (store last tweet `createdAt` in Sheets) to avoid reprocessing the same mentions. Without dedup, alerts fire on the same tweet repeatedly. + +## Reddit brand and topic monitoring +**When:** User wants weekly surfacing of brand mentions, product feedback, and competitor comparisons from Reddit. + +### Pipeline +1. **Scrape Reddit** -> `trudax/reddit-scraper-lite` + - Key input: `subreddits` (target subreddit array), `searchTerms` (brand + competitor names), `maxItems`, `sort` (hot/new/top) + +### Output fields +Step 1: `title`, `body`, `subreddit`, `url`, `score`, `numberOfComments`, `createdAt` + +### Gotcha +Set `sort: "new"` for monitoring runs; use `sort: "top"` for periodic digest reports. Mixing both in one run returns inconsistent result sets. + +## Multi-platform social listening with sentiment +**When:** User wants a unified brand health view across Instagram, Facebook, TikTok, and Twitter simultaneously. + +### Pipeline (run in parallel) +1. **Instagram** -> `apify/instagram-search-scraper` + - Key input: `searchTerms` (brand variants), `maxItems` +2. **Facebook** -> `apify/facebook-search-scraper` + - Key input: `searchTerms`, `maxItems` +3. **TikTok** -> `clockworks/tiktok-user-search-scraper` + - Key input: `searchTerms`, `maxItems` +4. **Twitter** -> `apidojo/tweet-scraper` + - Key input: `searchTerms`, `maxItems` +5. **Sentiment scoring** -> `tri_angle/social-media-sentiment-analysis-tool` + - Pipe: merged post URLs from steps 1-4 -> `urls` + - Key input: `urls`, `platforms` + +### Output fields +Steps 1-4 (normalized): `text`, `platform`, `author`, `timestamp`, `engagementCount` +Step 5: `sentiment`, `score`, `text`, `platform` + +## Sentiment analysis +**When:** User wants sentiment scoring on collected mentions. + +### Pipeline +1. **Collect mentions** (use any step from above) +2. **Analyze sentiment** -> `tri_angle/social-media-sentiment-analysis-tool` + - Pipe: collected post URLs -> `urls` + - Key input: `urls`, `platforms` + +### Output fields +Step 2: `sentiment` (positive/negative/neutral), `score`, `text`, `platform` diff --git a/skills/apify-ultimate-scraper/references/workflows/company-research.md b/skills/apify-ultimate-scraper/references/workflows/company-research.md new file mode 100644 index 0000000..6de344e --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/company-research.md @@ -0,0 +1,76 @@ +# Company research workflows + +## Company intelligence profiling for sales or ABM +**When:** User has a list of target accounts and wants structured firmographic data, ICP signals, and key personnel for outreach or account-based marketing. + +### Pipeline +1. **Crawl company website** -> `apify/website-content-crawler` + - Key input: `startUrls` (company domains), `maxCrawlDepth` (2), `includeUrlGlobs` (about, pricing, team, careers, blog) +2. **Enrich with LinkedIn company data** -> `harvestapi/linkedin-company` + - Pipe: company name or LinkedIn URL extracted from WCC text -> Actor input + - Key input: company identifier, `includeEmployees: false` +3. **AI extract structured signals** (n8n: OpenAI node outputs JSON schema with `companySize`, `industry`, `techStack`, `keyPersonnel`, `painSignals`) +4. **Store** in Supabase, Airtable, or HubSpot with AI-extracted fields as custom properties + +### Output fields +WCC: `text` (per page), `url` +LinkedIn: `employeeCount`, `industry`, `headquarters`, `description`, `specialties` +AI-extracted: `companySize`, `industry`, `techStack`, `keyPersonnel`, `painSignals` + +### Gotcha +WCC crawl at depth 2 can return 20-50 pages per company. For large batches, set `maxCrawlPages: 5` focused on the About and Pricing pages via `includeUrlGlobs`. This keeps cost and latency manageable without sacrificing signal quality. + +--- + +## Startup scouting from Product Hunt +**When:** User wants weekly discovery of recently launched or funded startups in a target category for investor outreach, partnership, or competitive tracking. + +### Pipeline +1. **Scrape Product Hunt launches** -> `apify/web-scraper` + - Key input: `startUrls` (Product Hunt today/weekly/topic pages), `maxCrawlPages` (3-5) + - Note: no dedicated Actor exists - search `apify actors search "product hunt"` for community options +2. **Filter by category + upvote threshold** (n8n: Filter node on extracted `upvotes`, `category`) +3. **Crawl company sites** -> `apify/website-content-crawler` + - Pipe: `results[].website` -> `startUrls` + - Key input: `maxCrawlPages` (3), `includeUrlGlobs` (about, team) +4. **LinkedIn founder lookup** -> `harvestapi/linkedin-profile-search` (by name + company) + - Pipe: extracted founder names from step 3 -> search input +5. **AI ICP scoring** (n8n: OpenAI node scores each startup against defined criteria) +6. **Output** to Airtable pipeline + Slack alert for top matches + +### Output fields +Step 1: `title`, `tagline`, `upvotes`, `website`, `makers[].name`, `makers[].profileUrl` +WCC: `text`, `url` +LinkedIn: `fullName`, `headline`, `profileUrl`, `currentCompany` + +### Gotcha +Product Hunt ranking changes throughout the day. Schedule the scrape for end-of-day (11 pm UTC) to capture final vote counts. For AngelList/Wellfound, no maintained public Actor exists - use `apify/website-content-crawler` on search result pages as a fallback. + +--- + +## Sales meeting prep from LinkedIn and news +**When:** A calendar event is detected and the user needs a briefing on meeting attendees - their recent activity, company context, and conversation talking points - delivered before the meeting. + +### Pipeline +1. **Trigger on calendar event** (n8n: Google Calendar Trigger, filter for events starting in 30 min) +2. **Extract attendee LinkedIn activity** -> `harvestapi/linkedin-profile-scraper` + `harvestapi/linkedin-profile-posts` + - Key input: `profiles` (attendee LinkedIn URLs), `maxPosts` (5 recent posts) +3. **Extract company context** -> `harvestapi/linkedin-company` + - Pipe: `results[].currentCompany` -> company name +4. **Search recent news** -> `apify/google-search-scraper` + - Pipe: company name + "news" -> `queries` + - Key input: `queries`, `maxResultsPerPage` (5) +5. **AI synthesize brief** (n8n: OpenAI node produces: key topics, recent signals, suggested talking points) +6. **Deliver** via Gmail or WhatsApp node 30 minutes before meeting start + +### Output fields +Profiles: `fullName`, `headline`, `recentPosts[]`, `experience[0]` +Posts: `postText`, `publishedAt`, `likes`, `comments` +Company: `description`, `employeeCount`, `recentNews` +Search: `organicResults[].title`, `organicResults[].snippet`, `organicResults[].url` + +### Cost estimate +All HarvestAPI Actors are PPE. Per meeting with 2 attendees: profile scraper ~$0.02, posts ~$0.01, company ~$0.005, Google search ~$0.01. Total: ~$0.05 per meeting prep. + +### Gotcha +LinkedIn profile URLs must be in the calendar event description or a linked CRM record - they won't auto-resolve from email addresses. Set up a step in your CRM or calendar template to include LinkedIn URLs for attendees. Without a valid `profileUrl`, the HarvestAPI Actors return empty results. diff --git a/skills/apify-ultimate-scraper/references/workflows/competitive-intel.md b/skills/apify-ultimate-scraper/references/workflows/competitive-intel.md new file mode 100644 index 0000000..12e7481 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/competitive-intel.md @@ -0,0 +1,85 @@ +# Competitive intelligence workflows + +## Competitor ad monitoring +**When:** User wants to see competitor advertising creatives, targeting, or ad spend signals. + +### Pipeline +1. **Scrape ad library** -> `apify/facebook-ads-scraper` + - Key input: `searchQuery` (competitor name), `country`, `adType`, `maxItems` + +### Output fields +Step 1: `adTitle`, `adBody`, `adCreativeUrl`, `startDate`, `pageInfo.name`, `platform` + +### Gotcha +Facebook Ad Library is public data, no auth needed. But results are limited to currently active or recently inactive ads. + +--- + +## Competitor web presence analysis +**When:** User wants traffic, rankings, and SEO data for competitor domains. + +### Pipeline +1. **Get traffic data** -> `radeance/similarweb-scraper` + - Key input: `urls` (competitor domains) +2. **Get backlink profile** -> `radeance/ahrefs-scraper` + - Key input: `urls` (same domains) + +### Output fields +Step 1: `globalRank`, `monthlyVisits`, `bounceRate`, `avgVisitDuration`, `trafficSources` +Step 2: `domainRating`, `backlinks`, `referringDomains`, `organicKeywords` + +### Cost estimate +radeance/ Actors cost $0.005-0.0275/result. A single domain audit across both steps costs ~$0.04-0.06. + +--- + +## Competitor website change detection +**When:** User wants to monitor competitor pricing pages, feature announcements, or product pages and get alerted when meaningful changes occur. + +### Pipeline +1. **Detect changes** -> `tri_angle/website-changes-detector` + - Key input: `startUrls` (competitor page URLs), `notificationEmail`, `checkIntervalHours` + +### Output fields +Step 1: `url`, `changedAt`, `diff` (text diff), `screenshotUrl` + +### Gotcha +`tri_angle/website-changes-detector` handles baseline storage internally - do not attempt to manage baselines externally or you will lose the diff history between runs. + +--- + +## Competitor SERP position monitoring +**When:** User wants to track where competitor domains rank for target keywords and get alerted on significant position shifts. + +### Pipeline +1. **Scrape SERP rankings** -> `apify/google-search-scraper` + - Key input: `queries` (target keywords array), `countryCode`, `maxResultsPerPage` +2. **Track traffic estimates** -> `radeance/similarweb-scraper` + - Key input: `urls` (competitor domains) + - Pipe: run separately per competitor domain after extracting domains from step 1 results + +### Output fields +Step 1: `organicResults[].url`, `organicResults[].position`, `organicResults[].title` +Step 2: `globalRank`, `monthlyVisits`, `trafficSources` + +### Cost estimate +`radeance/similarweb-scraper` costs ~$0.02-0.03 per domain. For 5 competitors, budget ~$0.10-0.15 per weekly run. + +--- + +## Competitor feature and pricing benchmarking +**When:** User wants a structured comparison of competitor pricing tiers, feature lists, and positioning across 5-10 competitor sites. + +### Pipeline +1. **Crawl pricing and feature pages** -> `apify/website-content-crawler` + - Key input: `startUrls` (competitor pricing page URLs), `maxCrawlDepth` (set to 1), `includeUrlGlobs` +2. **Extract structured data** -> AI node (GPT-4o or Claude) + - Pipe: `results[].text` -> extraction prompt per competitor + - Key input: extraction schema (tiers, prices, key features, positioning statement) + +### Output fields +Step 1: `text` (clean markdown with pricing tables), `url`, `metadata.title` +Step 2: AI-extracted structured JSON with tiers, prices, feature flags per competitor + +### Gotcha +Set `maxCrawlDepth: 1` and use `includeUrlGlobs` to restrict crawl to pricing and features paths only. Without this, WCC will crawl the full site and inflate cost significantly. diff --git a/skills/apify-ultimate-scraper/references/workflows/contact-enrichment.md b/skills/apify-ultimate-scraper/references/workflows/contact-enrichment.md new file mode 100644 index 0000000..8962a3b --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/contact-enrichment.md @@ -0,0 +1,65 @@ +# Contact enrichment workflows + +## Website contact extraction from a URL list +**When:** User has a list of company websites and wants emails, phone numbers, and social links for outreach. + +### Pipeline +1. **Extract contact info** -> `vdrmota/contact-info-scraper` or `compass/contact-details-scraper-standby` + - Key input: `startUrls` (company website URLs), `maxDepth` (crawl depth, 1-2 is usually enough) +2. **Supplement with social signals** -> `apify/social-media-leads-analyzer` + - Pipe: `results[].domain` -> input domain list + - Key input: `domains` (company domains), `extractLinkedIn`, `extractTwitter` +3. **Dedup and verify** (n8n: dedup by email domain, optional ZeroBounce/Hunter node for verification) + +### Output fields +Step 1: `emails[]`, `phones[]`, `linkedinUrl`, `twitterUrl`, `domain` +Step 2: `socialProfiles`, `linkedinCompanyUrl`, `facebookUrl` + +### Gotcha +`contact-details-scraper-standby` is a Standby Actor - it stays warm and responds in < 1s, making it ideal for real-time enrichment in webhook flows. Use it when latency matters. For batch jobs, `vdrmota/contact-info-scraper` is more cost-effective. + +--- + +## LinkedIn warm lead identification from post comments +**When:** User wants to find engaged prospects who commented on relevant LinkedIn posts (competitor content, thought leader posts, industry discussions). + +### Pipeline +1. **Scrape post comments** -> `harvestapi/linkedin-post-comments` + - Key input: `postUrl` (LinkedIn post URL), `maxComments` +2. **Enrich commenter profiles** -> `harvestapi/linkedin-profile-scraper` + - Pipe: `results[].commenter.profileUrl` -> `urls` + - Key input: `urls`, `includeEmail: true` +3. **Filter by ICP criteria** (n8n: Filter node on `headline` or `companyName`) +4. **AI draft outreach** (n8n: OpenAI node generates personalized message using `commentText` + `headline`) + +### Output fields +Step 1: `commenter.name`, `commenter.headline`, `commenter.profileUrl`, `commentText`, `timestamp` +Step 2: `experience[]`, `education[]`, `email`, `phone`, `skills[]` + +### Cost estimate +Both Actors are PPE. Step 1 ~ $0.005/comment. Step 2 with `includeEmail: true` ~ $0.01/profile. For 100 commenters enriched: ~$1.50 total. + +### Gotcha +Not all LinkedIn posts are publicly accessible. Test `postUrl` manually before building a workflow around it. Private or restricted posts return empty results - no error, just zero items. + +--- + +## Real-time lead enrichment on form submission +**When:** A prospect fills a form and their company needs to be enriched automatically for ICP scoring and sales routing. + +### Pipeline +1. **Receive form webhook** (n8n: Webhook trigger from HubSpot/Typeform/custom form) +2. **Extract company domain** (n8n: Function node parses email domain) +3. **Crawl company site** -> `apify/website-content-crawler` + - Key input: `startUrls` (company domain), `maxCrawlPages` (3-5), `includeUrlGlobs` (about, pricing, careers, team) +4. **Enrich with LinkedIn firmographics** -> `harvestapi/linkedin-company` (optional, for headcount + industry) + - Pipe: company name or LinkedIn URL derived from WCC output +5. **AI extract signals** (n8n: OpenAI node extracts companySize, industry, techStack, ICPFit score from crawl text) +6. **Route** (n8n: Switch node sends high-ICP leads to Slack sales channel, others to nurture sequence) + +### Output fields +WCC: `text`, `title`, `url`, `metadata.description` +LinkedIn: `employeeCount`, `industry`, `headquarters`, `description`, `website` + +### Gotcha +WCC crawl on a small startup site can take 30-60 seconds. For synchronous form flows, set `maxCrawlPages: 3` and use a timeout. If latency is critical, use `apify/cheerio-scraper` for the About page only and skip LinkedIn enrichment for first response. diff --git a/skills/apify-ultimate-scraper/references/workflows/content-and-seo.md b/skills/apify-ultimate-scraper/references/workflows/content-and-seo.md new file mode 100644 index 0000000..b0cc0d5 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/content-and-seo.md @@ -0,0 +1,113 @@ +# Content and SEO workflows + +## Website content extraction for RAG +**When:** User wants to crawl a website and extract clean text for AI/LLM pipelines or knowledge bases. + +### Pipeline +1. **Crawl website** -> `apify/website-content-crawler` + - Key input: `startUrls`, `maxCrawlPages`, `crawlerType` ("cheerio" for speed, "playwright" for JS sites) + +### Output fields +Step 1: `url`, `title`, `text`, `markdown`, `metadata`, `links[]` + +### Gotcha +For JS-heavy sites (SPAs), set `crawlerType: "playwright"`. For static sites, use `"cheerio"` (10x faster). For anti-bot sites, use `apify/camoufox-scraper` instead. + +## SERP-based SEO content brief generation +**When:** User wants to generate a content brief for a target keyword by analyzing competitor SERP results and heading structures. + +### Pipeline +1. **SERP scrape** -> `apify/google-search-scraper` + - Key input: `queries`, `countryCode`, `maxResultsPerPage` +2. **Extract heading structure** -> `apify/cheerio-scraper` + - Pipe: `results[].organicResults[].url` -> `startUrls` (filter non-article URLs first) + - Key input: `startUrls`, `pageFunction` (extract h1/h2/h3 nodes) + +### Output fields +Step 1: `organicResults[].url`, `organicResults[].title`, `organicResults[].snippet` +Step 2: heading structure (h1/h2/h3 text), word count per page + +### Gotcha +Use `apify/cheerio-scraper` for heading extraction - it's 10x faster than `website-content-crawler` for HTML-only pages. Only escalate to `website-content-crawler` when you need full body text for AI synthesis. + +## Sitemap content audit +**When:** User wants to crawl all URLs on a competitor's site from the sitemap and build a keyword and topic inventory. + +### Pipeline +1. **Extract sitemap URLs** -> `apify/sitemap-extractor` + - Key input: `startUrls` (sitemap.xml URL) +2. **Crawl each URL** -> `apify/website-content-crawler` + - Pipe: `results[].urls[]` -> `startUrls` + - Key input: `startUrls`, `maxCrawlPages`, `htmlTransformer` (readableText) + +### Output fields +Step 1: `urls[]` (all discovered page URLs) +Step 2: `text`, `metadata`, `url` + +### Gotcha +Large sitemaps (1,000+ URLs) can be expensive. Filter step 1 results to a specific path prefix (e.g., `/blog/`) before passing to step 2 to avoid crawling low-value pages like tag archives and pagination. + +## Keyword rank tracking with alerts +**When:** User wants weekly tracking of keyword positions for own domain and competitors, with Slack alerts on drops over 5 positions. + +### Pipeline +1. **SERP scrape** -> `apify/google-search-scraper` + - Key input: `queries` (tracked keywords array), `countryCode`, `device` (desktop/mobile), `maxResultsPerPage` (set to 100) + +### Output fields +Step 1: `organicResults[].position`, `organicResults[].url`, `organicResults[].domain` + +### Cost estimate +`apify/google-search-scraper` is a fixed-cost Actor. 100 keywords weekly ≈ $1-3/month depending on query volume. + +### Gotcha +Set `maxResultsPerPage: 100` to capture positions 11-100. Default is 10 results, which misses any keyword ranking outside page 1 - making it impossible to detect rank drops from position 12 to 18. + +## SERP analysis +**When:** User wants to analyze search engine results for specific keywords. + +### Pipeline +1. **Google SERP** -> `apify/google-search-scraper` + - Key input: `queries`, `maxPagesPerQuery`, `countryCode`, `languageCode` + +### Output fields +Step 1: `organicResults[]` (title, url, description, position), `paidResults[]`, `peopleAlsoAsk[]`, `relatedSearches[]` + +## Deep research agent +**When:** User wants an AI agent that takes a research question, generates search queries, crawls top results, and synthesizes findings into a structured report. + +### Pipeline +1. **Generate search queries** (LLM node - 3 queries per research question) +2. **SERP scrape** -> `apify/google-search-scraper` + - Pipe: generated queries -> `queries` + - Key input: `queries`, `maxResultsPerPage` +3. **Extract content** -> `apify/rag-web-browser` + - Pipe: `results[].organicResults[].url` (AI-selected relevant URLs) -> `query` + - Key input: `query`, `maxResults`, `requestTimeoutSecs` +4. **Synthesize** (LLM node - final report generation) + +### Output fields +Step 2: `organicResults[].url`, `organicResults[].snippet` +Step 3: `text`, `url`, `metadata.title` + +### Gotcha +Use `apify/rag-web-browser` at step 3 rather than `website-content-crawler` - it's optimized for agent-based single-URL retrieval and returns clean markdown with lower latency. Reserve `website-content-crawler` for bulk batch crawls. + +## Domain authority and backlink analysis +**When:** User wants SEO metrics for specific domains. + +### Pipeline +1. **Traffic overview** -> `radeance/similarweb-scraper` + - Key input: `urls` +2. **Backlink profile** -> `radeance/ahrefs-scraper` + - Key input: `urls` +3. **Domain authority** -> `radeance/semrush-scraper` + - Key input: `urls` + +### Output fields +Step 1: `globalRank`, `monthlyVisits`, `bounceRate`, `trafficSources` +Step 2: `domainRating`, `backlinks`, `referringDomains`, `organicKeywords` +Step 3: `authorityScore`, `organicSearchTraffic`, `paidSearchTraffic` + +### Cost estimate +All radeance/ SEO Actors are PPE at $0.005-0.0275/result. Running all 3 for one domain costs ~$0.05-0.08. For 50 domains, estimate $2.50-$4.00. diff --git a/skills/apify-ultimate-scraper/references/workflows/ecommerce-price-monitoring.md b/skills/apify-ultimate-scraper/references/workflows/ecommerce-price-monitoring.md new file mode 100644 index 0000000..77c5610 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/ecommerce-price-monitoring.md @@ -0,0 +1,88 @@ +# E-commerce price monitoring workflows + +## Competitor product price monitoring with alerts +**When:** User wants to track competitor prices across product pages and get notified when prices change. + +### Pipeline +1. **Scrape product pages** -> `apify/e-commerce-scraping-tool` + - Key input: `startUrls` (competitor product page URLs), `proxyConfiguration` +2. **Match products across sites** -> `tri_angle/e-commerce-product-matching-tool` + - Pipe: `results[].url` + `results[].name` -> matching input + - Key input: source dataset ID from step 1, target product list +3. **Compare vs. baseline** (n8n logic: read Google Sheets last_price, compute % change, filter if changed) +4. **Alert** via Telegram/Slack node with price delta + +### Output fields +Step 1: `price`, `currency`, `name`, `sku`, `availability`, `url` +Step 2: matched pairs with `similarityScore`, `sourceProduct`, `targetProduct` + +### Cost estimate +`apify/e-commerce-scraping-tool` is pay-per-result. For 200 product URLs daily, expect ~$0.50-$1/run depending on site complexity. + +### Gotcha +Many e-commerce sites use bot protection. If `e-commerce-scraping-tool` returns empty prices, fall back to `apify/camoufox-scraper` with residential proxy. Set `sessionPoolName` to reuse sessions and reduce blocks. + +--- + +## Amazon product and review tracking +**When:** User wants to monitor own or competitor Amazon listings for price drops or review score changes. + +### Pipeline +1. **Extract Amazon data** -> `apify/e-commerce-scraping-tool` + - Key input: `startUrls` (Amazon product URLs), `extractReviews` (bool) +2. **Compare vs. stored baseline** (n8n: read last values from Sheets or DB) +3. **Alert on new low price or rating drop** (n8n: If node + Telegram/Slack send) + +### Output fields +`price`, `currency`, `rating`, `reviewsCount`, `title`, `asin`, `availability` + +### Cost estimate +Flat per-result pricing. 50 ASINs daily ~ $0.10-$0.25/run. + +### Gotcha +Amazon aggressively rotates prices and sometimes shows regional prices. Always store `currency` alongside `price`. For review text (not just counts), search `apify actors search "amazon reviews"` for a dedicated Actor. + +--- + +## Supplier catalog extraction to draft products +**When:** User wants to pull new products from a supplier portal and create draft listings with AI-enriched descriptions. + +### Pipeline +1. **Crawl supplier catalog** -> `apify/playwright-scraper` (JS-heavy portals) or `apify/cheerio-scraper` (static HTML) + - Key input: `startUrls` (supplier category pages), `pseudoUrls` (product URL patterns), `maxCrawlPages` +2. **Extract product content** -> `apify/website-content-crawler` (optional second pass for detail pages) + - Pipe: `results[].url` -> `startUrls` + - Key input: `maxCrawlPages` (1 per product), `htmlTransformer: "readableText"` +3. **AI rewrite** (n8n: OpenAI node generates SEO title + bullets from raw specs) +4. **Create draft product** (n8n: Shopify node `POST /products.json` with `status: "draft"`) + +### Output fields +Step 1/2: `text`, `url`, `metadata.title`, inline image URLs + +### Cost estimate +Depends on catalog size. `playwright-scraper` is PPE; 500 product pages ~ $1-3. + +### Gotcha +Supplier portals often require login. Use `apify/playwright-scraper` with `initialCookies` or a pre-login script in `preNavigationHooks`. Never hardcode credentials - pass via Actor input from n8n credentials store. + +--- + +## Multi-site deal and coupon monitoring +**When:** User wants to detect when competitors run promotions or publish coupon codes so marketing can respond. + +### Pipeline +1. **Scrape deals pages** -> `apify/e-commerce-scraping-tool` + - Key input: `startUrls` (competitor deal/sale page URLs), `proxyConfiguration` +2. **Dynamic JS deal pages** (fallback) -> `apify/camoufox-scraper` + - Pipe: failed URLs from step 1 -> `startUrls` +3. **AI extract promotion details** (n8n: OpenAI node extracts discount %, promo code, expiry from raw text) +4. **Dedup and alert** (n8n: compare against stored deals DB, Slack notify on new deals) + +### Output fields +Raw: `price`, `discountText`, `url`; AI-extracted: `promoCode`, `validUntil`, `discountPercent`, `category` + +### Cost estimate +Light scraping - deals pages are few. Expect < $0.20/run for 20 competitor pages. + +### Gotcha +Promo codes and flash deals may only be visible after login or in geofenced regions. Test each target URL manually first. AI extraction of expiry dates is unreliable - treat as best-effort signal, not exact data. diff --git a/skills/apify-ultimate-scraper/references/workflows/influencer-vetting.md b/skills/apify-ultimate-scraper/references/workflows/influencer-vetting.md new file mode 100644 index 0000000..b452fc3 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/influencer-vetting.md @@ -0,0 +1,90 @@ +# Influencer vetting workflows + +## Instagram creator vetting +**When:** User wants to evaluate an influencer's profile, audience, and engagement quality. + +### Pipeline +1. **Get profile data** -> `apify/instagram-profile-scraper` + - Key input: `usernames` (list of handles) +2. **Analyze engagement** -> `apify/instagram-comment-scraper` + - Pipe: `results[].latestPosts[].url` -> `directUrls` (pick 3-5 recent posts) + - Key input: `directUrls`, `resultsLimit` + +### Output fields +Step 1: `username`, `fullName`, `followersCount`, `followsCount`, `postsCount`, `biography`, `isVerified`, `latestPosts[]` +Step 2: `text`, `ownerUsername`, `timestamp` (scan for bot patterns: generic praise, emoji-only, irrelevant content) + +### Gotcha +High follower count with low comment quality suggests fake followers. Compare comment sentiment to post content. + +--- + +## Cross-platform influencer discovery +**When:** User wants to find an influencer's presence across multiple platforms. + +### Pipeline +1. **Search across platforms** -> `tri_angle/social-media-finder` + - Key input: `query` (influencer name or handle), `platforms` + +### Output fields +Step 1: `platform`, `profileUrl`, `username`, `followers`, `isVerified` + +--- + +## TikTok creator vetting +**When:** User wants to vet TikTok creators by niche or handle for partnership fit based on engagement rate and content quality. + +### Pipeline +1. **Get profile metrics** -> `clockworks/tiktok-profile-scraper` + - Key input: `profiles` (username array), `resultsPerPage` +2. **Pull recent videos** -> `clockworks/tiktok-video-scraper` + - Pipe: `results[].authorMeta.name` -> `profiles` + - Key input: `profiles`, `maxItems` + +### Output fields +Step 1: `authorMeta.name`, `authorMeta.fans`, `authorMeta.heart`, `authorMeta.video` +Step 2: `diggCount`, `playCount`, `commentCount`, `shareCount`, `createTimeISO`, `hashtags`, `text` + +### Gotcha +TikTok engagement rate must be calculated manually: `(diggCount + commentCount + shareCount) / playCount`. The Actor does not return a pre-calculated ER field. + +--- + +## YouTube channel audit +**When:** User wants to audit YouTube channels for subscriber growth, average views, and topic consistency before sponsorship. + +### Pipeline +1. **Get channel overview** -> `streamers/youtube-channel-scraper` + - Key input: `startUrls` (channel URLs), `maxResults` +2. **Pull video metrics** -> `streamers/youtube-scraper` + - Pipe: `results[].channelUrl` -> `startUrls` + - Key input: `startUrls`, `maxResults` +3. **Analyze content themes** -> `curious_coder/youtube-transcript-scraper` + - Pipe: `results[].url` -> video URLs (pick 5-10 recent videos) + - Key input: video URLs + +### Output fields +Step 1: `channelName`, `numberOfSubscribers`, `channelTotalViews`, `channelUrl` +Step 2: `videos[].viewCount`, `videos[].likeCount`, `videos[].title`, `videos[].publishedAt` +Step 3: `transcript` (raw text for AI topic classification) + +--- + +## Cross-platform hashtag discovery +**When:** User wants to discover new influencer candidates across Instagram, TikTok, and YouTube using niche hashtags for a unified shortlist. + +### Pipeline +1. **Instagram hashtag scrape** -> `apify/instagram-hashtag-scraper` + - Key input: `hashtags` (array), `resultsLimit` +2. **TikTok hashtag scrape** -> `clockworks/tiktok-hashtag-scraper` + - Key input: `hashtags` (same array), `maxItems` +3. **YouTube hashtag scrape** -> `streamers/youtube-video-scraper-by-hashtag` + - Key input: `hashtags` (same array), `resultsLimit` + +### Output fields +Step 1: `ownerUsername`, `followersCount`, `profileUrl`, `likesCount`, `commentsCount` +Step 2: `authorMeta.name`, `authorMeta.fans`, `playCount`, `diggCount`, `shareCount` +Step 3: `channelName`, `numberOfSubscribers`, `viewCount`, `channelUrl` + +### Gotcha +Each platform returns platform-specific field names. Normalize to a common schema (`username`, `platform`, `followersCount`, `avgEngagement`, `profileUrl`) in a downstream merge step before scoring. diff --git a/skills/apify-ultimate-scraper/references/workflows/job-market-and-recruitment.md b/skills/apify-ultimate-scraper/references/workflows/job-market-and-recruitment.md new file mode 100644 index 0000000..209c5b2 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/job-market-and-recruitment.md @@ -0,0 +1,74 @@ +# Job market and recruitment workflows + +## Job listing research +**When:** User wants to find and analyze job postings by role, company, or location. + +### Pipeline +1. **Search jobs** -> `harvestapi/linkedin-job-search` + - Key input: `keyword`, `location`, `datePosted`, `limit` +2. **Get job details** -> `apimaestro/linkedin-job-detail` + - Pipe: `results[].jobUrl` -> `urls` + - Key input: `urls` + +### Output fields +Step 1: `title`, `company`, `location`, `jobUrl`, `postedDate`, `applicantsCount` +Step 2: `description`, `requirements`, `seniority`, `employmentType`, `salary` + +### Gotcha +Both Actors are PPE. Step 1: ~$0.001/job. Step 2: ~$0.005/job. For 200 jobs, total ~$1.20. Estimate and confirm with user. + +## Candidate sourcing +**When:** User wants to find potential candidates matching specific criteria. + +### Pipeline +1. **Search profiles** -> `harvestapi/linkedin-profile-search` + - Key input: `keyword`, `title`, `location`, `industry`, `limit` +2. **Enrich with details** -> `apimaestro/linkedin-profile-full-sections-scraper` + - Pipe: `results[].profileUrl` -> `urls` + - Key input: `urls` + +### Output fields +Step 1: `fullName`, `headline`, `location`, `profileUrl`, `currentCompany` +Step 2: `experience[]`, `education[]`, `skills[]`, `certifications[]`, `languages[]` + +### Gotcha +Step 2 (`apimaestro/linkedin-profile-full-sections-scraper`) costs ~$0.01/profile - the most expensive LinkedIn scraper. Use sparingly for shortlisted candidates only. + +## Sales signal outreach - job posting as buying signal +**When:** User wants to monitor company job postings as a signal to identify sales opportunities - e.g., a "Head of Data Engineering" hire suggests budget for data tooling. + +### Pipeline +1. **Monitor target postings** -> `harvestapi/linkedin-job-search` + - Key input: `searchUrl` (LinkedIn Jobs URL with company or role filters), `keywords` +2. **Get company context** -> `harvestapi/linkedin-company` + - Pipe: `results[].companyUrl` -> `companyUrls` + +### Output fields +Step 1: `title`, `companyName`, `description`, `employmentType`, `seniorityLevel`, `jobUrl` +Step 2: `name`, `industry`, `employeeCount`, `description`, `specialties[]` + +### Gotcha +Job descriptions contain implicit buying signals - tech stack mentions, pain points, and headcount growth. Pass `description` to an LLM to extract inferred tech stack and budget tier before prioritizing outreach. Contact finding (Hunter.io) uses the native n8n node, not an Apify Actor. + +## Upwork job monitoring for freelancers +**When:** User wants to continuously monitor Upwork for new jobs matching their skills. + +### Pipeline +1. **Scrape Upwork search** -> `apify/playwright-scraper` + - Key input: `startUrls` (Upwork search URL with skill filters), `pseudoUrls`, `maxCrawledPages` + +### Output fields +Step 1: `title`, `description`, `budget`, `clientJobsPosted`, `clientHireRate`, `postedAt`, `url` + +### Gotcha +No dedicated Upwork Actor exists in Apify Store - verify with `apify actors search "upwork"` for community options before defaulting to `apify/playwright-scraper`. Upwork pages are JS-heavy so Playwright is required over basic HTTP scraping. For high-frequency monitoring (every 15 min), store seen job URLs to avoid re-processing duplicates. + +## GitHub contributor discovery +**When:** User wants to find developers who contribute to specific open-source projects. + +### Pipeline +1. **Get contributors** -> `janbuchar/github-contributors-scraper` + - Key input: `repoUrls` + +### Output fields +Step 1: `username`, `contributions`, `profileUrl`, `avatarUrl` diff --git a/skills/apify-ultimate-scraper/references/workflows/knowledge-base-and-rag.md b/skills/apify-ultimate-scraper/references/workflows/knowledge-base-and-rag.md new file mode 100644 index 0000000..d4b66b5 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/knowledge-base-and-rag.md @@ -0,0 +1,64 @@ +# Knowledge base and RAG pipeline workflows + +## Website to RAG knowledge base via sitemap crawl +**When:** User wants to ingest an entire website or documentation site into a vector database for AI retrieval (chatbots, search, AI agents). + +### Pipeline +1. **Extract sitemap** -> `apify/sitemap-extractor` + - Key input: `sitemapUrl` or `domain` +2. **Crawl and convert to markdown** -> `apify/website-content-crawler` + - Pipe: `results[].url` -> `startUrls` (or pass dataset ID) + - Key input: `startUrls`, `maxCrawlPages`, `htmlTransformer: "readableText"`, `outputFormats: ["markdown"]` +3. **Chunk and embed** (n8n: Recursive Character Text Splitter -> OpenAI Embeddings node) +4. **Upsert to vector DB** (n8n: Supabase / Qdrant node with document + metadata) + +### Output fields +`text` (clean markdown), `url`, `metadata.title`, `metadata.description`, `crawledAt` + +### Gotcha +`apify/rag-web-browser` is purpose-built for RAG use cases and returns pre-chunked, clean text without boilerplate - use it when you want simpler output and don't need full site coverage. For comprehensive crawls (full docs sites, 100+ pages), `website-content-crawler` gives more control over depth and URL filtering. + +--- + +## Deep research agent with web crawling +**When:** User or an AI agent submits a research question and wants a synthesized report drawn from live web sources. + +### Pipeline +1. **Generate search queries** (n8n: AI node expands research question into 3-5 distinct queries) +2. **Search** -> `apify/google-search-scraper` + - Pipe: generated queries -> `queries` (array) + - Key input: `queries`, `maxResultsPerPage` (5-10) +3. **Retrieve content** -> `apify/rag-web-browser` + - Pipe: `results[].organicResults[].url` -> `query` (RAG browser takes query + crawls most relevant result) + - Key input: `query`, `maxResults`, `requestTimeoutSecs` +4. **Synthesize** (n8n: OpenAI node assembles final report from per-source summaries) +5. **Output** to n8n Data Table, Notion, or Google Docs + +### Output fields +Search: `organicResults[].url`, `organicResults[].title`, `organicResults[].snippet` +RAG browser: `text`, `url`, `metadata.title` + +### Gotcha +`apify/rag-web-browser` fetches and summarizes a single URL per call. To process multiple search results in parallel, use n8n's Split In Batches node with a concurrency of 3-5 rather than running them sequentially. This cuts total runtime significantly for 10+ URLs. + +--- + +## Scheduled news monitoring to AI knowledge feed +**When:** User wants to track industry news sources daily, filter new articles, summarize them, and store in a searchable knowledge base (Notion, NocoDB, Supabase). + +### Pipeline +1. **Extract articles** -> `lukaskrivka/article-extractor-smart` + - Key input: `startUrls` (news site listing pages), `maxCrawlPages`, `articleSelector` (optional CSS hint) +2. **Filter new articles only** (n8n: compare `publishedAt` or URL against stored records in DB) +3. **Full article content** (optional) -> `apify/website-content-crawler` + - Pipe: new article `url` values -> `startUrls` + - Use when listing-page extract is too short for quality summarization +4. **AI summarize + tag** (n8n: OpenAI node generates 3-sentence summary + keyword tags) +5. **Upsert to knowledge base** (n8n: Notion / NocoDB / Supabase node) + +### Output fields +Step 1: `title`, `text`, `publishedAt`, `author`, `url`, `tags` +Step 3 (WCC): full `text`, `metadata.title`, `metadata.description` + +### Gotcha +`lukaskrivka/article-extractor-smart` handles most news formats well, but paywalled sites return truncated content. Check `text` length - if consistently under 200 characters for a given source, that site is paywalled and should be removed from the list. Deduplicate by URL before summarizing to avoid re-processing old articles on re-runs. diff --git a/skills/apify-ultimate-scraper/references/workflows/lead-generation.md b/skills/apify-ultimate-scraper/references/workflows/lead-generation.md new file mode 100644 index 0000000..96fe025 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/lead-generation.md @@ -0,0 +1,118 @@ +# Lead generation workflows + +## Local business leads with email enrichment +**When:** User wants business contacts, emails, or phone numbers for businesses in a specific location. + +### Pipeline +1. **Find businesses** -> `compass/crawler-google-places` + - Key input: `searchStringsArray`, `locationQuery`, `maxCrawledPlaces` +2. **Enrich with contacts** -> `compass/enrich-google-maps-dataset-with-contacts` + - Pipe: `results[].url` -> `startUrls` (or pass the dataset ID directly) + - Key input: `datasetId` (from step 1), `maxRequestsPerCrawl` + +### Output fields +Step 1: `title`, `address`, `phone`, `website`, `categoryName`, `totalScore`, `reviewsCount`, `url` +Step 2: `emails[]`, `phones[]`, `socialLinks`, `linkedInUrl`, `twitterUrl` + +### Gotcha +Google Maps results vary by language and location. Set `language: "en"` explicitly. Also set `locationQuery` to a specific city/region, not just a country. + +--- + +## B2B prospect discovery via LinkedIn +**When:** User wants to find professionals by role, company, or industry. + +### Pipeline +1. **Search profiles** -> `harvestapi/linkedin-profile-search` + - Key input: `keyword`, `location`, `title`, `limit` +2. **Enrich with details** -> `harvestapi/linkedin-profile-scraper` + - Pipe: `results[].profileUrl` -> `urls` + - Key input: `urls`, `includeEmail` (set to `true` for email discovery) + +### Output fields +Step 1: `fullName`, `headline`, `location`, `profileUrl`, `currentCompany` +Step 2: `experience[]`, `education[]`, `skills[]`, `email`, `phone` + +### Cost estimate +Step 2 with `includeEmail: true` costs ~$0.01/profile. For 500 profiles, budget ~$5. + +### Gotcha +LinkedIn Actors are all PPE. Estimate and confirm with user before running at scale. + +--- + +## Sales Navigator bulk lead extraction +**When:** User wants daily 100-1,000 lead extraction from a Sales Navigator search for outbound sequences. + +### Pipeline +1. **Extract leads** -> `harvestapi/linkedin-profile-search` + - Key input: `searchUrl` (Sales Navigator search URL), `maxResults`, `proxy` settings +2. **Verify emails** -> native n8n Hunter.io node or HTTP Request to ZeroBounce API + - Pipe: `results[].email` -> email verification input + +### Output fields +Step 1: `fullName`, `email`, `companyName`, `jobTitle`, `connectionDegree`, `profileUrl` +Step 2: `result` (valid/risky/invalid), `score` + +### Cost estimate +`harvestapi/linkedin-profile-search` is PPE. 1,000 leads at typical rates runs ~$5-10. Confirm before scheduling daily runs. + +### Gotcha +Sales Navigator URL must be a saved search URL, not a one-time results URL. The URL changes each session unless saved. + +--- + +## SERP-based B2B prospect discovery +**When:** User wants to find companies matching niche keywords via Google, AI-qualify them against ICP criteria, and push qualified leads to CRM. + +### Pipeline +1. **Find companies** -> `apify/google-search-scraper` + - Key input: `queries` (search terms array), `maxResultsPerPage`, `countryCode` +2. **Crawl company sites** -> `apify/website-content-crawler` + - Pipe: `results[].organicResults[].url` -> `startUrls` + - Key input: `startUrls`, `maxCrawlDepth` (set to 2), `maxCrawlPages` (set to 5) + +### Output fields +Step 1: `organicResults[].url`, `organicResults[].title`, `organicResults[].snippet` +Step 2: `text` (clean markdown), `url`, `metadata.title`, `metadata.description` + +### Gotcha +Pass only company root domains from SERP results into WCC - not individual blog post URLs. Filter `organicResults[].url` for root domains before piping. + +--- + +## Apollo leads + AI website icebreakers +**When:** User has an Apollo lead list with company websites and wants personalized cold email icebreakers generated from each company's web presence. + +### Pipeline +1. **Scrape company sites** -> `apify/website-content-crawler` + - Key input: `startUrls` (homepage URLs from Apollo export), `maxCrawlDepth` (set to 2), `maxCrawlPages` (set to 5) +2. **Generate icebreakers** -> AI node (GPT-4o or Claude) + - Pipe: `results[].text` -> prompt context per lead + - Key input: company summary + lead name + role + +### Output fields +Step 1: `text` (clean markdown), `metadata.title`, `metadata.description`, `url` +Step 2: AI-generated icebreaker string per lead + +### Gotcha +Some Apollo exports include LinkedIn URLs instead of company websites. Filter the list for `http` URLs before passing to WCC - LinkedIn blocks crawlers. + +--- + +## Reddit community lead mining +**When:** User wants to find prospects actively posting problems that their product or service solves in relevant subreddits. + +### Pipeline +1. **Mine subreddit posts** -> `trudax/reddit-scraper-lite` + - Key input: `startUrls` (subreddit URLs), `searchTerms` (problem keywords), `maxItems`, `sort` (hot/new/top) +2. **Qualify leads** -> AI node + - Pipe: `results[].title`, `results[].body` -> qualification prompt + - Key input: ICP criteria, pain point keywords + +### Output fields +Step 1: `title`, `body`, `subreddit`, `url`, `score`, `numberOfComments`, `createdAt`, `author` +Step 2: AI qualification score, extracted contact intent, suggested outreach angle + +### Gotcha +Reddit usernames are pseudonymous - there is no direct email enrichment path. The output is intent signals and post URLs for manual outreach via Reddit DM or to cross-reference against other platforms. diff --git a/skills/apify-ultimate-scraper/references/workflows/real-estate-and-hospitality.md b/skills/apify-ultimate-scraper/references/workflows/real-estate-and-hospitality.md new file mode 100644 index 0000000..007402d --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/real-estate-and-hospitality.md @@ -0,0 +1,74 @@ +# Real estate and hospitality workflows + +## Property search and analysis +**When:** User wants to find and compare property listings in a specific area. + +### Pipeline +1. **Search properties** -> `tri_angle/redfin-search` + - Key input: `location`, `propertyType`, `minPrice`, `maxPrice` +2. **Get details** -> `tri_angle/redfin-detail` + - Pipe: `results[].url` -> `startUrls` + - Key input: `startUrls` + +### Output fields +Step 1: `address`, `price`, `beds`, `baths`, `sqft`, `url`, `status` +Step 2: `description`, `yearBuilt`, `lotSize`, `priceHistory[]`, `taxHistory[]`, `schools[]` + +## Airbnb market analysis +**When:** User wants to analyze Airbnb listings, pricing, and reviews in a destination. + +### Pipeline +1. **Search listings** -> `tri_angle/new-fast-airbnb-scraper` + - Key input: `location`, `checkIn`, `checkOut`, `maxItems` +2. **Get reviews** -> `tri_angle/airbnb-reviews-scraper` + - Pipe: `results[].url` -> `startUrls` + - Key input: `startUrls`, `maxReviews` + +### Output fields +Step 1: `name`, `price`, `rating`, `reviews`, `type`, `amenities[]`, `url`, `images[]` +Step 2: `text`, `rating`, `date`, `reviewerName` + +### Gotcha +Airbnb pricing varies by date. Always set `checkIn` and `checkOut` for accurate pricing. For market analysis, run multiple date ranges to capture seasonal variation. + +## Real estate lead scoring and agent routing +**When:** User wants to qualify inbound leads from listing portals by budget signals and urgency, then route them to the right agent. + +### Pipeline +1. **Search matching properties** -> `tri_angle/redfin-search` + - Key input: `location`, `minPrice`, `maxPrice` (from lead payload) +2. **Enrich lead with LinkedIn signals** -> `harvestapi/linkedin-profile-scraper` + - Key input: `profileUrls` (optional - use only when lead email resolves to a LinkedIn profile) + +### Output fields +Step 1: `address`, `price`, `beds`, `baths`, `status`, `url` +Step 2: `headline`, `currentCompany`, `experience[]` (income/seniority signals) + +### Gotcha +The LinkedIn enrichment step is optional - only run it when the lead's identity is known and a LinkedIn profile URL is available. The core routing logic (hot/warm/cold tier + agent assignment) runs on the MLS webhook payload itself, with scraping used as enrichment. Lead scoring and routing output fields are AI-generated: `leadTier`, `assignedAgent`, `routingReason`. + +## Construction and pre-market property discovery +**When:** User wants to find new-construction projects or pre-market inventory before they appear on major listing portals. + +### Pipeline +1. **Scrape construction portals** -> `apify/playwright-scraper` + - Key input: `startUrls` (local MLS or construction project portal URLs), `proxyConfiguration` +2. **Extract clean text** -> `lukaskrivka/article-extractor-smart` + - Pipe: `results[].url` -> `urls` + +### Output fields +Step 1: raw HTML / structured page data +Step 2: `projectName`, `price`, `location`, `possessionDate`, `constructionStatus` + +### Gotcha +No market-specific Actor exists for most construction portals (e.g., 99acres). Run `apify actors search "real estate"` to check for community-built options before using `apify/playwright-scraper`. For JS-heavy portals, `playwright-scraper` is required. Step 2 cleans raw output into structured fields - pipe all Step 1 URLs through it. + +## Multi-source property comparison +**When:** User wants to compare listings across Zillow, Realtor, Zumper, and other US/UK sources. + +### Pipeline +1. **Aggregate listings** -> `tri_angle/real-estate-aggregator` + - Key input: `location`, `propertyType`, `sources` (Zillow, Realtor, Zumper, Apartments.com, Rightmove) + +### Output fields +Step 1: `address`, `price`, `beds`, `baths`, `sqft`, `source`, `url`, `listingDate` diff --git a/skills/apify-ultimate-scraper/references/workflows/review-analysis.md b/skills/apify-ultimate-scraper/references/workflows/review-analysis.md new file mode 100644 index 0000000..f4b3f14 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/review-analysis.md @@ -0,0 +1,90 @@ +# Review analysis workflows + +## Google Maps review extraction +**When:** User wants to collect and analyze business reviews from Google Maps. + +### Pipeline +1. **Find businesses** -> `compass/crawler-google-places` + - Key input: `searchStringsArray`, `locationQuery`, `maxCrawledPlaces` +2. **Extract reviews** -> `compass/Google-Maps-Reviews-Scraper` + - Pipe: `results[].url` -> `startUrls` + - Key input: `startUrls`, `maxReviews` + +### Output fields +Step 1: `title`, `totalScore`, `reviewsCount`, `url`, `categoryName` +Step 2: `text`, `stars`, `publishedAtDate`, `reviewerName`, `ownerResponse` + +## Competitor review intelligence +**When:** User wants to extract competitor reviews to surface customer pain points and compare against own product strengths for positioning. + +### Pipeline +1. **Scrape competitor reviews** -> `compass/Google-Maps-Reviews-Scraper` + - Key input: `startUrls` (competitor Google Maps URLs), `maxReviews`, `sort` (newest or most relevant) +2. **Yelp competitor reviews** -> `tri_angle/yelp-review-scraper` + - Key input: `startUrls` (competitor Yelp URLs), `maxReviews` + +### Output fields +Step 1: `stars`, `text`, `name`, `publishedAtDate`, `reviewId` +Step 2: `text`, `rating`, `date`, `userName` + +### Gotcha +Run steps 1 and 2 in parallel for the same competitor, then merge by date. AI analysis works best when you label each review with the competitor name before passing to LLM for theme extraction. + +## Google Play app review monitoring +**When:** User wants daily low-rating alerts from Google Play to route urgent negative feedback to the support team. + +### Pipeline +1. **Scrape app reviews** -> `apify/playwright-scraper` + - Key input: `startUrls` (Google Play app URL), `maxRequestsPerCrawl` +2. **Filter and alert** (n8n native - IF node) + - Pipe: `results[].stars` -> filter where `stars < 4` + +### Output fields +Step 1: `stars`, `text`, `date`, `appVersion`, `thumbsUpCount` + +### Gotcha +Google Play uses heavy client-side rendering. Use `apify/playwright-scraper` rather than cheerio. If results are thin, search Apify Store for a dedicated Google Play reviews Actor - the ecosystem updates frequently. + +## Cross-platform hotel/restaurant reviews +**When:** User wants reviews aggregated from multiple platforms for the same business. + +### Pipeline (hotels) +1. **Aggregate reviews** -> `tri_angle/hotel-review-aggregator` + - Key input: `urls` (hotel URLs from TripAdvisor, Yelp, Google Maps, Booking.com, etc.) + +### Pipeline (restaurants) +1. **Aggregate reviews** -> `tri_angle/restaurant-review-aggregator` + - Key input: `urls` (restaurant URLs from Yelp, Google Maps, DoorDash, UberEats, etc.) + +### Output fields +Both: `text`, `rating`, `date`, `platform`, `reviewerName`, `title` + +## Multi-platform review aggregation for hospitality +**When:** User wants a weekly sentiment digest across TripAdvisor, Booking.com, Google, and Yelp for a property - including theme extraction by category (service, rooms, location, price). + +### Pipeline +1. **Aggregate all platforms** -> `tri_angle/hotel-review-aggregator` + - Key input: `startUrls` (property page URLs per platform), `maxReviews`, `includeReviews` +2. **Airbnb reviews** (if applicable) -> `tri_angle/airbnb-reviews-scraper` + - Key input: `startUrls` (Airbnb listing URLs), `maxReviews` + +### Output fields +Step 1: `stars`, `text`, `title`, `reviewDate`, `source`, `userProfile.name` +Step 2: `stars`, `text`, `reviewDate`, `reviewerName` + +### Gotcha +Review aggregators pull from multiple platforms in one run - cheaper than running separate scrapers per platform. Use the aggregators when covering 3+ platforms. For Airbnb specifically, run the dedicated `tri_angle/airbnb-reviews-scraper` separately and merge by date. + +## Yelp review pipeline +**When:** User wants Yelp reviews for businesses in a specific area. + +### Pipeline +1. **Find businesses** -> `tri_angle/get-yelp-urls` + - Key input: `location`, `category` +2. **Extract reviews** -> `tri_angle/yelp-review-scraper` + - Pipe: `results[].url` -> `startUrls` + - Key input: `startUrls`, `maxReviews` + +### Output fields +Step 1: `name`, `url`, `rating`, `reviewCount`, `address` +Step 2: `text`, `rating`, `date`, `userName` diff --git a/skills/apify-ultimate-scraper/references/workflows/social-media-analytics.md b/skills/apify-ultimate-scraper/references/workflows/social-media-analytics.md new file mode 100644 index 0000000..5543831 --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/social-media-analytics.md @@ -0,0 +1,74 @@ +# Social media analytics workflows + +## Instagram account performance analysis +**When:** User wants engagement metrics and content performance for an Instagram account. + +### Pipeline +1. **Get profile** -> `apify/instagram-profile-scraper` + - Key input: `usernames` +2. **Get recent posts** -> `apify/instagram-post-scraper` + - Key input: `directUrls` (from profile's `latestPosts[].url`) or `usernames` + +### Output fields +Step 1: `followersCount`, `followsCount`, `postsCount`, `biography`, `isVerified` +Step 2: `caption`, `likesCount`, `commentsCount`, `timestamp`, `type` (photo/video/reel), `url` + +## TikTok creator analytics +**When:** User wants performance data for a TikTok creator. + +### Pipeline +1. **Get profile** -> `clockworks/tiktok-profile-scraper` + - Key input: `profiles` (handles or URLs) + +### Output fields +Step 1: `nickname`, `followers`, `following`, `likes`, `videos`, `verified`, `recentVideos[]` (with views, likes, shares per video) + +## Instagram competitor content analysis +**When:** User wants to identify top-performing content formats and engagement patterns from competitor Instagram accounts. + +### Pipeline +1. **Get competitor posts** -> `apify/instagram-post-scraper` + - Key input: `usernames` (competitor handles), `resultsLimit` (100), `scrapePostsUntilDate` +2. **Get reels separately** -> `apify/instagram-reel-scraper` + - Key input: `usernames` (same handles) + +### Output fields +Step 1: `likesCount`, `commentsCount`, `timestamp`, `type` (post/reel/story), `caption`, `displayUrl`, `url` +Step 2: `likesCount`, `commentsCount`, `playsCount`, `duration`, `caption`, `url` + +### Gotcha +Run both Actors to capture full content mix - the post scraper may under-count reels. Calculate engagement rate per post (likes + comments / follower count) and sort to surface top performers. + +## LinkedIn company page analytics +**When:** User wants to track LinkedIn post performance for a company page or benchmark against competitors. + +### Pipeline +1. **Get company posts** -> `harvestapi/linkedin-company-posts` + - Key input: `companyUrl`, `maxPosts`, `publishedAfter` +2. **Enrich post details** -> `apimaestro/linkedin-post-detail` + - Pipe: `results[].url` -> `urls` + +### Output fields +Step 1: `likesCount`, `commentsCount`, `repostsCount`, `text`, `publishedAt`, `url` +Step 2: `reactions{}` (breakdown by type), `topComments[]`, `impressions` + +### Gotcha +Both Actors are PPE. Step 1: ~$0.002/post, Step 2: ~$0.005/post. For 100 posts across 3 companies, estimate ~$2.10. Confirm with user before running. + +## Multi-platform engagement comparison +**When:** User wants to compare an account's performance across platforms. + +### Pipeline (run independently, combine) +1. **Instagram** -> `apify/instagram-profile-scraper` with `usernames` +2. **TikTok** -> `clockworks/tiktok-profile-scraper` with `profiles` +3. **YouTube** -> `streamers/youtube-channel-scraper` with `channelUrls` +4. **X/Twitter** -> `apidojo/twitter-user-scraper` with `handles` + +### Output fields +Instagram: `followersCount`, `postsCount`, `biography` +TikTok: `followers`, `likes`, `videos` +YouTube: `subscriberCount`, `videoCount`, `viewCount` +X/Twitter: `followers`, `tweets`, `likes` + +### Gotcha +Parallel workflow - run each Actor independently. Normalize metric names for comparison (followers/subscribers, posts/videos/tweets). diff --git a/skills/apify-ultimate-scraper/references/workflows/trend-research.md b/skills/apify-ultimate-scraper/references/workflows/trend-research.md new file mode 100644 index 0000000..1f675ac --- /dev/null +++ b/skills/apify-ultimate-scraper/references/workflows/trend-research.md @@ -0,0 +1,79 @@ +# Trend and keyword research workflows + +## Google Trends analysis +**When:** User wants to analyze search demand trends for keywords or topics. + +### Pipeline +1. **Get trend data** -> `apify/google-trends-scraper` + - Key input: `searchTerms`, `timeRange`, `geo` (country code) + +### Output fields +Step 1: `term`, `timelineData[]` (date, value), `relatedQueries[]`, `relatedTopics[]` + +## Cross-platform hashtag research +**When:** User wants to evaluate a hashtag's reach and usage across platforms. + +### Pipeline +1. **Cross-platform overview** -> `apify/social-media-hashtag-research` + - Key input: `hashtags`, `platforms` (instagram, youtube, tiktok, facebook) + +### Output fields +Step 1: `hashtag`, `platform`, `postsCount`, `topPosts[]`, `relatedHashtags[]` + +## TikTok trend discovery +**When:** User wants to find trending content, sounds, or hashtags on TikTok. + +### Pipeline +1. **Trending content** -> `clockworks/tiktok-trends-scraper` + - Key input: `channel` (trending category) +2. **Explore categories** -> `clockworks/tiktok-explore-scraper` + - Key input: `exploreCategories` + +### Output fields +Step 1: `videoUrl`, `description`, `likes`, `shares`, `views`, `author`, `music` +Step 2: `category`, `posts[]`, `authors[]`, `music[]` + +## Reddit trend and community insight mining +**When:** User wants to surface emerging trends, product feedback themes, or competitor mentions from Reddit communities. + +### Pipeline +1. **Scrape subreddits** -> `trudax/reddit-scraper-lite` + - Key input: `startUrls` (subreddit URLs), `maxItems`, `sort` (hot/rising/new) + +### Output fields +Step 1: `title`, `body`, `subreddit`, `score`, `numberOfComments`, `url`, `createdAt`, `comments[]` + +### Gotcha +Use `sort: rising` for early trend signals, `sort: hot` for confirmed trending topics. Filter by `score` threshold (e.g., >50) to reduce noise. Comments array provides qualitative context for AI sentiment analysis. + +## YouTube outlier video discovery +**When:** User wants to identify breakout videos in a niche with disproportionate views vs. channel subscriber count - a signal for content strategy pivots. + +### Pipeline +1. **Search niche videos** -> `streamers/youtube-scraper` + - Key input: `searchTerms`, `maxResults`, `sort` (viewCount), `uploadDate` (filter range) +2. **Get channel context** -> `streamers/youtube-channel-scraper` + - Pipe: `results[].channelUrl` -> `channelUrls` + +### Output fields +Step 1: `title`, `viewCount`, `likeCount`, `commentCount`, `channelName`, `publishedAt`, `url` +Step 2: `subscriberCount`, `videoCount`, `viewCount` (channel totals) + +### Gotcha +Outlier score = video `viewCount` / channel `subscriberCount`. Ratios > 10x indicate breakout potential. Run Step 2 on a filtered shortlist only - no need to fetch channel data for every result. + +## Content topic validation +**When:** User wants to validate whether a topic has demand before creating content. + +### Pipeline +1. **Search demand** -> `apify/google-trends-scraper` + - Key input: `searchTerms` (topic keywords) +2. **Social reach** -> `apify/social-media-hashtag-research` + - Key input: `hashtags` (topic hashtags) + +### Output fields +Step 1: `timelineData[]` (trending up/down), `relatedQueries[]` +Step 2: `postsCount` per platform, `topPosts[]` + +### Gotcha +Google Trends shows relative interest (0-100 scale), not absolute volume. Combine with hashtag post counts for a fuller picture.