Key Takeaways: Crawlee turns messy web scraping into a structured, reliable crawler stack for JavaScript and Python, with built‑in browsers, queues, storage, and proxy handling.
What Crawlee is and why it matters
Crawlee is Apify’s open‑source web scraping and browser automation library that abstracts away the hardest parts of crawling: anti‑bot protections, browser orchestration, proxies, request queues, and data storage. It ships with first‑class support for JavaScript/TypeScript and Python, letting you build crawlers on top of tools like Playwright, Puppeteer, Cheerio, and BeautifulSoup using a unified, high‑level API. Under the hood, Crawlee includes autoscaled crawling, anti‑fingerprinting patterns, and a storage layer for datasets, key‑value stores, and request queues, turning one‑off scripts into robust scraping workflows.
For AI and LLM use cases, Crawlee is particularly valuable because it can reliably extract HTML, PDFs, images, and structured JSON from sites you want to feed into RAG pipelines or fine‑tuning corpora.
- GitHub (JavaScript/TypeScript): https://github.com/apify/crawlee
- GitHub (Python): https://github.com/apify/crawlee-python
- Homepage & docs: https://crawlee.dev
Core features you get with Crawlee
Multi‑crawler abstraction
Crawlee exposes several crawler classes that share the same interface, so you can switch between plain HTTP and full browser automation without rewriting your entire scraper.
- CheerioCrawler – fast HTTP crawler using
got-scrapingand Cheerio for HTML parsing, ideal for static pages. - PuppeteerCrawler – headless Chrome/Chromium via Puppeteer for sites that rely on client‑side rendering.
- PlaywrightCrawler – browser‑agnostic crawling via Playwright, supporting Chromium, Chrome, Firefox, WebKit, and more.
All three share a common requestHandler signature, enqueueLinks() for recursion, and common options like maxRequestsPerCrawl, making it easy to upgrade from static to dynamic scraping when needed.
Built‑in autoscaling, queues, and storage
Crawlee manages concurrency and resource usage via autoscaled crawling and request queues that keep your scraper within hardware and site limits. It stores scraped results, metadata, and state under a ./storage directory by default, including datasets, key_value_stores, and request_queues, and lets you override the location via CRAWLEE_STORAGE_DIR for custom setups.
On the Python side, the generated projects integrate with Poetry to manage dependencies and provide commands like poetry run python -m crawlee-python-demo plus crawler.export_data('output.csv') to export data.
Proxy support and anti‑blocking patterns
Crawlee’s Playwright crawler and proxy configuration classes make it straightforward to rotate proxies, use tiered proxy lists, and fall back from cheap datacenter IPs to residential IPs only when needed.
Combined with autoscaling and request queue management, this helps you build scrapers that respect robots.txt, avoid hammering targets, and handle Cloudflare/Akamai‑style bot mitigations gracefully.
Installing Crawlee for JavaScript / TypeScript
Crawlee’s JavaScript/TypeScript distribution targets Node.js 16+ and works great with modern bundlers and TS tooling.
Option 1: Crawlee CLI (fastest start)
The quickest way to bootstrap a new crawler is the Crawlee CLI.
npx crawlee create my-crawler
cd my-crawler
npm startThe CLI scaffolds a Node.js project with Crawlee, sets up storage directories, and drops in a ready‑to‑run example targeting https://crawlee.dev.
Option 2: Manual installation in an existing Node project
If you already have a Node.js app, install Crawlee and your preferred browser library manually.
Plain HTTP / Cheerio only:
npm install crawleePlaywright‑powered crawling:
npm install crawlee playwrightPuppeteer‑powered crawling:
npm install crawlee puppeteerCrawlee intentionally doesn’t bundle Playwright or Puppeteer to keep install size small and let you pick the browser engine that fits your project.
Quick JavaScript example: CheerioCrawler
Here’s a minimal CheerioCrawler that recursively scrapes titles from https://crawlee.dev and stores them to a dataset.
import { CheerioCrawler, Dataset } from 'crawlee';
const crawler = new CheerioCrawler({
async requestHandler({ request, $, enqueueLinks, log }) {
const title = $('title').text();
log.info(`Title of ${request.loadedUrl} is '${title}'`);
await Dataset.pushData({ title, url: request.loadedUrl });
await enqueueLinks();
},
maxRequestsPerCrawl: 50,
});
await crawler.run(['https://crawlee.dev']);When you run this with node main.mjs (and "type": "module" in package.json), Crawlee will automatically store results as JSON under ./storage/datasets/default and log progress to the console.
To turn this into an illustrative image for your article or docs, you can grab a screenshot of the terminal logs (“Title of … is …”) or the storage/datasets folder structure.
Quick JavaScript example: PlaywrightCrawler (headless browser)
If the target site relies on JavaScript for rendering, swap to PlaywrightCrawler.
import { PlaywrightCrawler, Dataset } from 'crawlee';
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, log }) {
const title = await page.title();
log.info(`Title of ${request.loadedUrl} is '${title}'`);
await Dataset.pushData({ title, url: request.loadedUrl });
await enqueueLinks();
},
// Turn this off for visible browser windows during development.
// headless: false,
maxRequestsPerCrawl: 50,
});
await crawler.run(['[https://crawlee.dev]);Setting headless: false lets you watch the browser as Crawlee navigates and extracts data, which makes a great demo GIF or screenshot.
You can capture a browser window plus terminal logs and reference it in your blog as an illustrative image, for example:
- Crawlee homepage hero and example crawler screenshots: https://crawlee.dev
Installing Crawlee for Python
Crawlee’s Python package builds on BeautifulSoup and Playwright, offering similar abstractions for crawlers and storage, with CLI support via pipx and project scaffolding.
Python quickstart via CLI
The fastest way to start a Python crawler is the Crawlee CLI.
pipx run crawlee create my-crawlerThis command creates a my-crawler directory with preconfigured routes, storage, and boilerplate code targeting Crawlee’s own documentation site.
Then install dependencies with Poetry and run the demo:
pip install poetry # if you don’t have it yet
cd my-crawler
poetry install
poetry run python -m crawlee-python-demoYou’ll see Crawlee crawl the target site, push scraped items into datasets, and populate storage directories similar to the Node version.
Adding Playwright support in Python
To upgrade a BeautifulSoup‑based scraper to a Playwright crawler, install the extra and Playwright itself.
pip install 'crawlee[playwright]'
playwright installThen adjust your __main__.py to use PlaywrightCrawler instead of BeautifulSoupCrawler, for example targeting Mint Mobile’s phone listings:
import asyncio
from crawlee.playwright_crawler.playwright_crawler import PlaywrightCrawler
from .routes import router
async def main() -> None:
crawler = PlaywrightCrawler(
browser_type='firefox',
headless=True,
request_handler=router,
)
await crawler.run(['https://addrom.com'])
await crawler.export_data('output.csv')
if __name__ == '__main__':
asyncio.run(main())This produces a CSV with scraped data in your project directory—ideal for feeding downstream ETL, search indexes, or RAG stores.
Proxies, tiered proxy lists, and anti‑blocking
Crawlee’s Python Playwright integration includes a ProxyConfiguration class that helps you define proxy URLs or tiered lists where cheaper datacenter proxies are used first and more expensive residential proxies are reserved for fallback.
A simple proxy configuration example:
from crawlee.proxy_configuration import ProxyConfiguration
proxy_configuration = ProxyConfiguration(
proxy_urls=[
'http://proxy-example-1.com/',
'http://proxy-example-2.com/',
]
)Tiered configuration (cheap first, expensive as backup) looks like this:
proxy_configuration = ProxyConfiguration(
tiered_proxy_urls=[
['http://cheap-datacenter-proxy-1.com/', 'http://cheap-datacenter-proxy-2.com/'],
['http://expensive-residential-proxy-1.com/', 'http://expensive-residential-proxy-2.com/'],
]
)You then pass proxy_configuration into PlaywrightCrawler, and Crawlee handles the rotation and fallback logic around blocked or unreliable proxies.
Combined with max_requests_per_crawl and sensible request pacing, this gives you a robust foundation for production‑grade scrapers that remain respectful of target sites while still collecting the data you need for analytics or AI pipelines.
