Scrapingbee

Single endpoint: GET https://app.scrapingbee.com/api/v1/?api_key={key}&url={url}. Parameters control behavior: render_js (boolean — JavaScript rendering), premium_proxy (residential proxies), country_code (geo-targeting), wait (milliseconds to wait after page load), css_extractor (JSON mapping CSS selectors to output keys), extract_rules (structured data extraction), screenshots (return page screenshot). The Google Search endpoint returns structured SERP data. The AI extraction endpoint uses LLMs for content extraction. Response is the raw HTML by default, or structured JSON when using extraction features. The API design achieves maximum simplicity: one endpoint, URL as input, content as output. For agents, integration requires no libraries — a single HTTP GET with query parameters.

ScrapingBee simplifies web scraping to a single API call: send a URL, receive the rendered HTML content. For agents, this eliminates the need to manage proxies, headless browsers, CAPTCHA solving, and anti-bot detection. The API handles JavaScript rendering (optional), residential proxy rotation, and stealth browser headers automatically. The Google Search API provides structured Google SERP data. The AI extraction feature uses LLMs to extract structured data from HTML without CSS selectors. For agents needing content from web pages without building scraping infrastructure, ScrapingBee is the simplest option. The trade-off versus Apify: less customization and no Actor ecosystem, but dramatically simpler integration. For agents with straightforward scraping needs (fetch page content, extract data), ScrapingBee's simplicity is its strength.

API key authentication via api_key query parameter. Keys are generated in the dashboard. No fine-grained scoping. Credit-based pricing: each request consumes credits, with premium features (JavaScript rendering, residential proxies) consuming more credits. No OAuth. Keys don't expire. For agents, the credit model means monitoring credit consumption is important — JavaScript-rendered requests with residential proxies consume 10-100x more credits than simple requests. The dashboard shows credit usage. No IP restrictions. The auth model is the simplest possible. For agents with high-volume scraping needs, credit cost optimization (minimizing JavaScript rendering and premium proxy usage) is the main concern.

Documentation at scrapingbee.com/documentation covers the API reference, parameter descriptions, and use case guides. The single-endpoint design means the documentation is concise — all parameters are documented on one page. Code examples in Python, Node.js, PHP, Ruby, and Go show common patterns. The Google Search API and AI extraction features have dedicated documentation. The documentation includes best practices for credit optimization (when to enable JavaScript rendering, when to use premium proxies). No official SDKs — the API is simple enough for raw HTTP. For agents, the API reference page is the only documentation needed. The documentation's strength: its brevity matches the API's simplicity.

API responses include a status code reflecting the target page's response. ScrapingBee retries failed requests automatically (no credit charged for failed requests — only successful responses consume credits). Common failure modes: target site blocking (mitigated by proxy rotation), JavaScript rendering timeout (configurable wait time), and CAPTCHA challenges (handled by premium proxies). The API returns the target page's status code, enabling agents to distinguish between scraping failures and target page errors (404, 500). Rate limits are per-plan. The main reliability consideration: scraping success rates vary by target site — major sites with aggressive anti-bot measures may have lower success rates. Premium proxies improve success rates at higher credit cost.