Apify

Fresh current-pass runtime rerun passed for Apify scrape.extract through Rhumb Resolve. Managed and direct executions both crawled example.com and matched on URL, metadata.title, and markdown body.

Fresh runtime rerun passed for Apify scrape.extract through Rhumb Resolve. Managed and direct executions both crawled example.com and returned Example Domain.

Production Rhumb-managed scrape.extract via Apify website-content-crawler matched direct Apify control on Example Domain and returned 201 with a completed crawl. The integration is working, but the rolling 24h health window is still red because earlier malformed test inputs generated four 400 responses before the final pass.

REST API at api.apify.com/v2/. Actor execution: POST /acts/{actorId}/runs with input payload triggers an Actor run. Run status: GET /acts/{actorId}/runs/{runId} returns status (READY, RUNNING, SUCCEEDED, FAILED) with timing and resource metrics. Dataset access: GET /datasets/{datasetId}/items returns structured output data. Key-Value Store: provides arbitrary data storage for runs. Request Queue: manages URLs for crawler-based Actors. The Actor-Run-Dataset model is clean: trigger an Actor, poll for completion, retrieve results from the dataset. Webhook notifications on run completion eliminate polling. Scheduled runs provide cron-like recurring execution. The API supports Actor input schema validation. For agents, the most common pattern is: find an Actor, call it with input, get structured data back.

API token authentication via token query parameter or Authorization: Bearer header. Personal tokens provide full account access. Scoped tokens can be restricted to specific Actors, datasets, and operations — this is more granular than most scraping platforms. No OAuth for third-party integrations. Tokens don't expire by default. For agents, scoped tokens enable least-privilege access: a token that can only run specific Actors and read their datasets. Usage is tracked per-account with compute unit pricing. The proxy service uses the same authentication. For multi-agent environments, scoped tokens prevent cross-agent resource access.

Documentation at docs.apify.com is comprehensive, covering the platform (Actors, datasets, proxies), the API reference, Crawlee framework, and deployment. The API reference covers all endpoints with examples. The Actor Store provides per-Actor documentation with input schemas and output descriptions. Crawlee documentation is extensive — tutorials, API reference, and anti-blocking guides. SDKs for JavaScript/TypeScript, Python are official and maintained. The documentation's strength: it covers both using existing Actors (simple) and building custom Actors (advanced). For agents, the 'Running an Actor via API' guide and the Actor Store are the essential starting points. The documentation effectively serves both API consumers and scraper developers.

API errors return JSON with error object containing type, message, and statusCode. Actor run failures include detailed error logs accessible via the API. The platform handles common scraping failures automatically: proxy rotation on IP blocks, browser restart on crashes, request retries on network errors. The Crawlee framework provides anti-blocking features (fingerprint rotation, session management, adaptive crawling). Rate limits on the API are generous. The main reliability consideration: scraping is inherently fragile — target website changes can break scrapers. Pre-built Actors from the Store are community-maintained with varying update frequency. Custom Actors require maintenance when target sites change. For agents, using popular, well-maintained Actors from the Store reduces maintenance burden.

Apify is a full web scraping and automation platform built around the Actor model — self-contained scraping programs that run in Apify's cloud. For agents, the key value is the Actor Store: hundreds of pre-built scrapers for Google Search, Amazon products, social media profiles, job boards, and more. Agents can run these Actors via API, passing input parameters and receiving structured data output. Custom Actors can be built using Crawlee (Apify's open-source scraping framework built on Puppeteer/Playwright). The platform handles proxy rotation, browser management, request queuing, and data storage. For agents needing structured data from websites — competitive intelligence, market research, lead generation, content aggregation — Apify's Actor ecosystem provides ready-made solutions. The trade-off: dependence on Apify's cloud and per-compute-unit pricing.