MediaCrawler: A modern, multi‑platform social media crawler for AI and data workflows

Key Takeaways: MediaCrawler turns messy, multi-platform social media data into structured, analyzable datasets with a modern Python stack and a friendly WebUI.

What is MediaCrawler?

MediaCrawler is an open-source, multi-platform social media data collection toolkit built in Python, designed to grab publicly available posts, videos, comments, and interactions from major Chinese and global content platforms. Out of the box it supports Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, and Zhihu, focusing on open data such as notes, posts, videos, questions, answers, and comment threads.

Under the hood, MediaCrawler uses Playwright and browser automation to log in and preserve browser sessions, allowing it to operate with your existing cookies and login state instead of reverse‑engineering platform JavaScript. By reusing a logged‑in browser context and evaluating JavaScript expressions to obtain required signature parameters, it largely avoids complex JS/Javascript “anti‑scraping” reverse engineering.

For documentation and updates, the main resources are:

• GitHub repository: https://github.com/NanmiCoder/MediaCrawler
• Project homepage and docs: https://nanmicoder.github.io/MediaCrawler/

Key capabilities and use cases

MediaCrawler is built for multi-platform workflows such as content intelligence, social listening, and dataset creation for AI models. Core features are organized around searching, targeting specific posts, traversing comment trees, and working with creator accounts for each supported platform.

At a high level, MediaCrawler can:

• Search by keywords and crawl matching posts across supported platforms.
• Crawl specific post IDs, including primary content and comments, with optional secondary (nested) comments.
• Target creator homepages to collect their content and interactions.
• Cache login state to reduce friction when working with authenticated views.
• Use pluggable IP proxy pools to distribute traffic.
• Generate comment word‑cloud images, which can serve as illustrative visuals in reports or dashboards.

For visual inspiration, you can refer to the official docs gallery and examples, which include WebUI screenshots and diagrams: https://nanmicoder.github.io/MediaCrawler/

Legal and ethical reminders

The project is explicitly positioned as a learning and research tool for web data collection — not as a turnkey system for large‑scale scraping or circumventing platform policies. The author stresses that MediaCrawler’s contents are for educational use only, must not be used for illegal purposes, and should follow local laws such as network security regulations.

Using MediaCrawler responsibly means:

• Respecting platform terms of service and robots policies.
• Limiting scale, frequency, and sensitivity of crawls.
• Avoiding commercial or privacy‑invasive applications without explicit permission.

Prerequisites and environment setup

MediaCrawler favors a modern Python toolchain centered around uv, a fast Python package and environment manager. It also relies on Node.js for some platforms (e.g., Douyin and Zhihu) due to JS‑related tasks, and optionally Playwright’s browser drivers when not using Chrome CDP mode.

Recommended toolchain (uv + Node.js)

Before installing MediaCrawler, prepare the following:

• uv: Install following the official uv installation guide, then confirm with uv --version in your shell.
• Node.js ≥ 16.0.0: Install from https://nodejs.org/en/download/ and ensure node -v reports a suitable version.
• Python: Use a reasonably recent version (e.g., 3.11 as per the repo’s requirements examples); uv will align dependencies accordingly.

This stack makes dependency management faster and more deterministic than traditional pip‑only workflows.

Installing MediaCrawler with uv (recommended)

Once your prerequisites are ready, clone the repository and let uv handle everything.

Step 1: Clone the repository

git clone https://github.com/NanmiCoder/MediaCrawler.git
cd MediaCrawler

This gives you the full project layout including main.py, config files, API server, and documentation.

Step 2: Sync Python environment and dependencies

Use uv to create and sync the environment:

uv sync

This command ensures your Python version and dependency set matches what the project expects, using pyproject.toml and lock files in the repo.

Step 3: (Optional) Install Playwright browser drivers

If you plan to use standard Playwright mode rather than Chrome CDP mode, install Playwright drivers via uv:

uv run playwright install

In CDP mode (connecting to an already‑running Chrome), using Playwright’s own browsers is optional; you can skip this step if you stick to CDP.

Configuring Chrome CDP mode

One of MediaCrawler’s standout features is its ability to attach to your existing Chrome instance using the Chrome DevTools Protocol (CDP).

This lets you reuse cookies, login state, and even extensions, which can significantly reduce the chance of triggering platform anti‑bot mechanisms.

To set up CDP mode:

1. Install the latest Chrome (version 144 or above) from the official site.
2. Open chrome://inspect/#remote-debugging in the address bar.
3. Enable “Allow remote debugging for this browser instance.”
4. Confirm that you see a message like Server running at: 127.0.0.1:9222, indicating CDP is active.

When MediaCrawler starts in CDP mode, Chrome will display a confirmation dialog; you need to accept within 60 seconds for the crawler to proceed.

If you prefer standard Playwright mode, set ENABLE_CDP_MODE = False in config/base_config.py to disable CDP and fall back to Playwright’s managed browsers.

Running your first crawl from the CLI

MediaCrawler is controlled primarily through main.py and a set of command‑line options that select platform, login type, and crawl mode.

Configuration defaults are defined in config/base_config.py, with Chinese comments explaining each feature and flag.

Example: Keyword search on Xiaohongshu

To crawl Xiaohongshu posts via QR login and keyword search:

uv run main.py --platform xhs --lt qrcode --type search

This reads keywords and other parameters from the config file, then fetches matching notes and related comments.

Example: Specific post IDs on Xiaohongshu

To crawl a set of specified post IDs:

uv run main.py --platform xhs --lt qrcode --type detail

MediaCrawler will pull the target posts and associated comment threads based on ID lists defined in the config.

You can always inspect available platforms and options with:

uv run main.py --help

For users who prefer traditional virtual environments, the repo also documents a python -m venv venv flow, installing dependencies via pip install -r requirements.txt and using python main.py with equivalent arguments.

WebUI: Visual control panel and monitoring

Beyond the CLI, MediaCrawler provides a WebUI for configuring and monitoring crawls from a browser.

This is ideal if you want to hand off operations to less technical users or simply prefer a dashboard over command‑line scripts.

Starting the WebUI API server

From the project root, start the FastAPI/Uvicorn server:

uv run uvicorn api.main:app --port 8080 --reload
# or
uv run python -m api.main

Once the server is running, open http://localhost:8080 in your browser to access the WebUI.

The interface allows you to configure platform, login method, crawl type, and other parameters, while viewing logs and data previews in real time.

For illustrative screenshots of the WebUI and diagrams of the architecture, refer to the official docs site, which includes images embedded in the guide: https://nanmicoder.github.io/MediaCrawler/

Data storage and export options

MediaCrawler is designed for data engineering workflows and supports multiple storage formats out of the box.

You can choose between CSV, JSON, JSONL, Excel, SQLite, and MySQL, depending on whether you’re prototyping, building pipelines, or feeding data into analytical systems.

Recent updates added support for JSONL as a storage format and made it the default, which is particularly convenient for streaming large comment datasets and training data for AI models.

Configuration for storage backends lives in the database and store modules and can be tuned to match your infrastructure preferences.

For more details and diagrams (including example data layouts and screenshot images), see the “数据存储指南” linked from the README and docs homepage.

MediaCrawlerPro and AI agent integration

Alongside the open-source version, there is a MediaCrawlerPro variant aimed at more advanced users and production‑like architectures.

The Pro version introduces a self‑media content decomposition agent, resume‑able crawling, multi‑account plus proxy pool support, removal of direct Playwright dependencies, and full Linux environment support.

From an architecture standpoint, Pro focuses on refactored code, decoupled JS signature logic, and more modular design suitable for large crawler projects.

It also adds a desktop video downloader, multi‑platform home feed crawling, and built‑in AI Agent Skills for tools like OpenClaw, Claude Code, and Cursor, enabling agents to automatically trigger crawls.

An AI comment analysis agent is under development, which will eventually make it easier to plug crawled data into sentiment models and topic classifiers without writing your own pipeline from scratch.

Best practices for using MediaCrawler in your AI/ML stack

For AI/ML practitioners, MediaCrawler is most powerful when paired with a clear data strategy and robust governance.

Because it supports multiple platforms and flexible formats, you can treat it as a data ingestion layer in a larger MLOps or data engineering system.

Some practical patterns:

• Use JSONL output to feed comment streams into labeling tools or direct model fine‑tuning pipelines.
• Combine multi‑platform crawls to reduce bias in datasets, while keeping per‑platform quotas conservative and compliant.
• Deploy the WebUI behind authentication for internal teams to schedule crawls and export datasets without touching the CLI.
• Integrate IP proxy pools and multi‑account support carefully, keeping usage transparently documented and audited.

Always cross‑check platform terms of service and local regulations before deploying MediaCrawler as part of a production workflow.

Final thoughts

MediaCrawler stands out as a practical, real‑world codebase for learning modern crawler architecture and building social media data pipelines, especially in the Chinese ecosystem.

With uv‑based installation, Chrome CDP integration, WebUI control, versatile storage, and ongoing work around AI agents, it fits naturally into research, content intelligence, and AI/ML experimentation stacks.

If you want to go deeper, explore the GitHub repo for examples, issues, and community discussions: https://github.com/NanmiCoder/MediaCrawler , and browse the official docs with screenshots and diagrams at https://nanmicoder.github.io/MediaCrawler/