Crawl4AI：Python网络爬虫与数据提取工具，支持CLI和SDK，高效生成Markdown

crawl4ai by brettdavies/crawl4ai-skill

285 周安装量

14 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/brettdavies/crawl4ai-skill --skill crawl4ai

Python Web框架自动化数据处理

🇨🇳中文介绍

Crawl4AI

概述

Crawl4AI 提供全面的网络爬取和数据提取功能。该技能支持 CLI（推荐用于快速任务）和 Python SDK（用于程序化控制）。

选择您的接口：

CLI (crwl) - 快速、可脚本化的命令：CLI 指南
Python SDK - 完整的程序化控制：SDK 指南

快速开始

安装

pip install crawl4ai
crawl4ai-setup

# 验证安装
crawl4ai-doctor

CLI（推荐）

# 基础爬取 - 返回 markdown
crwl https://example.com

# 获取 markdown 输出
crwl https://example.com -o markdown

# JSON 输出并绕过缓存
crwl https://example.com -o json -v --bypass-cache

# 查看更多示例
crwl --example

Python SDK

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:500])

asyncio.run(main())

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

概念	CLI	SDK
浏览器设置	`-B browser.yml` 或 `-b "param=value"`	`BrowserConfig(...)`
爬取设置	`-C crawler.yml` 或 `-c "param=value"`	`CrawlerRunConfig(...)`
提取	`-e extract.yml -s schema.json`	`extraction_strategy=...`
内容过滤器	`-f filter.yml`	`markdown_generator=...`

Markdown 生成（主要用例）

Crawl4AI 擅长生成干净、格式良好的 markdown：

# 基础 markdown
crwl https://docs.example.com -o markdown

# 过滤后的 markdown（移除噪音）
crwl https://docs.example.com -o markdown-fit

# 使用内容过滤器
crwl https://docs.example.com -f filter_bm25.yml -o markdown-fit

过滤器配置：

# filter_bm25.yml (基于相关性)
type: "bm25"
query: "machine learning tutorials"
threshold: 1.0

from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

bm25_filter = BM25ContentFilter(user_query="machine learning", bm25_threshold=1.0)
md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)

config = CrawlerRunConfig(markdown_generator=md_generator)
result = await crawler.arun(url, config=config)

print(result.markdown.fit_markdown)  # 过滤后的
print(result.markdown.raw_markdown)  # 原始的

关于内容过滤器：内容处理 (第 2481-3101 行)

1. 基于模式的 CSS 提取（最高效）

无需 LLM - 快速、确定性、零成本。

# 生成模式（使用 LLM）
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# 使用模式进行提取（无需 LLM）
crwl https://shop.com -e extract_css.yml -s product_schema.json -o json

{
  "name": "products",
  "baseSelector": ".product-card",
  "fields": [
    {"name": "title", "selector": "h2", "type": "text"},
    {"name": "price", "selector": ".price", "type": "text"},
    {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
  ]
}

2. 基于 LLM 的提取

适用于复杂或不规则的内容：

# extract_llm.yml
type: "llm"
provider: "openai/gpt-4o-mini"
instruction: "Extract product names and prices"
api_token: "your-token"



crwl https://shop.com -e extract_llm.yml -o json

关于提取详情：提取策略 (第 4522-5429 行)

动态内容（JavaScript 密集型网站）

crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000"

爬取器配置：

# crawler.yml
wait_for: "css:.ajax-content"
scan_full_page: true
page_timeout: 60000
delay_before_return_html: 2.0

CLI（顺序）：

for url in url1 url2 url3; do crwl "$url" -o markdown; done

Python SDK（并发）：

urls = ["https://site1.com", "https://site2.com", "https://site3.com"]
results = await crawler.arun_many(urls, config=config)

关于批处理：arun_many() 参考 (第 1057-1224 行)

会话与身份验证

# login_crawler.yml
session_id: "user_session"
js_code: |
  document.querySelector('#username').value = 'user';
  document.querySelector('#password').value = 'pass';
  document.querySelector('#submit').click();
wait_for: "css:.dashboard"



# 登录
crwl https://site.com/login -C login_crawler.yml

# 访问受保护内容（会话复用）
crwl https://site.com/protected -c "session_id=user_session"

关于会话管理：高级功能 (第 5429-5940 行)

# browser.yml
headless: true
proxy_config:
  server: "http://proxy:8080"
  username: "user"
  password: "pass"
user_agent_mode: "random"



crwl https://example.com -B browser.yml

crwl https://docs.example.com -o markdown > docs.md

电子商务产品监控

# 生成模式
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# 监控（无 LLM 成本）
crwl https://shop.com -e extract_css.yml -s schema.json -o json

# 多源过滤
for url in news1.com news2.com news3.com; do
  crwl "https://$url" -f filter_bm25.yml -o markdown-fit
done

# 先查看内容
crwl https://example.com -o markdown

# 然后提问
crwl https://example.com -q "What are the main conclusions?"
crwl https://example.com -q "Summarize the key points"

scripts/extraction_pipeline.py - 模式生成和提取
scripts/basic_crawler.py - 简单的 markdown 提取
scripts/batch_crawler.py - 多 URL 处理

文档	用途
CLI 指南	命令行接口参考
SDK 指南	Python SDK 快速参考
完整 SDK 参考	完整 API 文档（5900+ 行）

从 CLI 开始 处理快速任务，SDK 用于自动化
使用基于模式的提取 - 比 LLM 高效 10-100 倍
在开发时启用缓存 - 仅在需要时使用 --bypass-cache
设置适当的超时 - 普通网站 30 秒，JS 密集型网站 60 秒以上
使用内容过滤器 以获得更干净、更聚焦的 markdown
遵守速率限制 - 在请求之间添加延迟

JavaScript 未加载

crwl https://example.com -c "wait_for=css:.dynamic-content,page_timeout=60000"

机器人检测问题

crwl https://example.com -B browser.yml



# browser.yml
headless: false
viewport_width: 1920
viewport_height: 1080
user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

# 调试：查看完整输出
crwl https://example.com -o all -v

# 尝试不同的等待策略
crwl https://example.com -c "wait_for=js:document.querySelector('.content')!==null"

# 验证会话
crwl https://site.com -c "session_id=test" -o all | grep -i session

完整的 API 文档，请参阅完整 SDK 参考。

🇺🇸English

Crawl4AI

Overview

Crawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both CLI (recommended for quick tasks) and Python SDK (for programmatic control).

Choose your interface:

CLI (crwl) - Quick, scriptable commands: CLI Guide
Python SDK - Full programmatic control: SDK Guide

Quick Start

Installation

pip install crawl4ai
crawl4ai-setup

# Verify installation
crawl4ai-doctor

CLI (Recommended)

# Basic crawling - returns markdown
crwl https://example.com

# Get markdown output
crwl https://example.com -o markdown

# JSON output with cache bypass
crwl https://example.com -o json -v --bypass-cache

# See more examples
crwl --example

Python SDK

import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun("https://example.com")
        print(result.markdown[:500])

asyncio.run(main())

For SDK configuration details: SDK Guide - Configuration (lines 61-150)

Core Concepts

Configuration Layers

Both CLI and SDK use the same underlying configuration:

Concept	CLI	SDK
Browser settings	`-B browser.yml` or `-b "param=value"`	`BrowserConfig(...)`
Crawl settings	`-C crawler.yml` or `-c "param=value"`	`CrawlerRunConfig(...)`
Extraction	`-e extract.yml -s schema.json`

Key Parameters

Browser Configuration:

headless: Run with/without GUI
viewport_width/height: Browser dimensions
user_agent: Custom user agent
proxy_config: Proxy settings

Crawler Configuration:

page_timeout: Max page load time (ms)
wait_for: CSS selector or JS condition to wait for
cache_mode: bypass, enabled, disabled
js_code: JavaScript to execute
css_selector: Focus on specific element

For complete parameters: CLI Config | SDK Config

Output Content

Every crawl returns:

markdown - Clean, formatted markdown
html - Raw HTML
links - Internal and external links discovered
media - Images, videos, audio found
extracted_content - Structured data (if extraction configured)

Markdown Generation (Primary Use Case)

Crawl4AI excels at generating clean, well-formatted markdown:

CLI

# Basic markdown
crwl https://docs.example.com -o markdown

# Filtered markdown (removes noise)
crwl https://docs.example.com -o markdown-fit

# With content filter
crwl https://docs.example.com -f filter_bm25.yml -o markdown-fit

Filter configuration:

# filter_bm25.yml (relevance-based)
type: "bm25"
query: "machine learning tutorials"
threshold: 1.0

Python SDK

from crawl4ai.content_filter_strategy import BM25ContentFilter
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

bm25_filter = BM25ContentFilter(user_query="machine learning", bm25_threshold=1.0)
md_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)

config = CrawlerRunConfig(markdown_generator=md_generator)
result = await crawler.arun(url, config=config)

print(result.markdown.fit_markdown)  # Filtered
print(result.markdown.raw_markdown)  # Original

For content filters: Content Processing (lines 2481-3101)

Data Extraction

1. Schema-Based CSS Extraction (Most Efficient)

No LLM required - fast, deterministic, cost-free.

CLI:

# Generate schema once (uses LLM)
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# Use schema for extraction (no LLM)
crwl https://shop.com -e extract_css.yml -s product_schema.json -o json

Schema format:

{
  "name": "products",
  "baseSelector": ".product-card",
  "fields": [
    {"name": "title", "selector": "h2", "type": "text"},
    {"name": "price", "selector": ".price", "type": "text"},
    {"name": "link", "selector": "a", "type": "attribute", "attribute": "href"}
  ]
}

2. LLM-Based Extraction

For complex or irregular content:

CLI:

# extract_llm.yml
type: "llm"
provider: "openai/gpt-4o-mini"
instruction: "Extract product names and prices"
api_token: "your-token"



crwl https://shop.com -e extract_llm.yml -o json

For extraction details: Extraction Strategies (lines 4522-5429)

Advanced Patterns

Dynamic Content (JavaScript-Heavy Sites)

CLI:

crwl https://example.com -c "wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000"

Crawler config:

# crawler.yml
wait_for: "css:.ajax-content"
scan_full_page: true
page_timeout: 60000
delay_before_return_html: 2.0

Multi-URL Processing

CLI (sequential):

for url in url1 url2 url3; do crwl "$url" -o markdown; done

Python SDK (concurrent):

urls = ["https://site1.com", "https://site2.com", "https://site3.com"]
results = await crawler.arun_many(urls, config=config)

For batch processing: arun_many() Reference (lines 1057-1224)

Session & Authentication

CLI:

# login_crawler.yml
session_id: "user_session"
js_code: |
  document.querySelector('#username').value = 'user';
  document.querySelector('#password').value = 'pass';
  document.querySelector('#submit').click();
wait_for: "css:.dashboard"



# Login
crwl https://site.com/login -C login_crawler.yml

# Access protected content (session reused)
crwl https://site.com/protected -c "session_id=user_session"

For session management: Advanced Features (lines 5429-5940)

Anti-Detection & Proxies

CLI:

# browser.yml
headless: true
proxy_config:
  server: "http://proxy:8080"
  username: "user"
  password: "pass"
user_agent_mode: "random"



crwl https://example.com -B browser.yml

Common Use Cases

Documentation to Markdown

crwl https://docs.example.com -o markdown > docs.md

E-commerce Product Monitoring

# Generate schema once
python scripts/extraction_pipeline.py --generate-schema https://shop.com "extract products"

# Monitor (no LLM costs)
crwl https://shop.com -e extract_css.yml -s schema.json -o json

News Aggregation

# Multiple sources with filtering
for url in news1.com news2.com news3.com; do
  crwl "https://$url" -f filter_bm25.yml -o markdown-fit
done

Interactive Q&A

# First view content
crwl https://example.com -o markdown

# Then ask questions
crwl https://example.com -q "What are the main conclusions?"
crwl https://example.com -q "Summarize the key points"

Resources

Provided Scripts

scripts/extraction_pipeline.py - Schema generation and extraction
scripts/basic_crawler.py - Simple markdown extraction
scripts/batch_crawler.py - Multi-URL processing

Reference Documentation

Document	Purpose
CLI Guide	Command-line interface reference
SDK Guide	Python SDK quick reference
Complete SDK Reference	Full API documentation (5900+ lines)

Best Practices

Start with CLI for quick tasks, SDK for automation
Use schema-based extraction - 10-100x more efficient than LLM
Enable caching during development - --bypass-cache only when needed
Set appropriate timeouts - 30s normal, 60s+ for JS-heavy sites
Use content filters for cleaner, focused markdown
Respect rate limits - Add delays between requests

Troubleshooting

JavaScript Not Loading

crwl https://example.com -c "wait_for=css:.dynamic-content,page_timeout=60000"

Bot Detection Issues

crwl https://example.com -B browser.yml



# browser.yml
headless: false
viewport_width: 1920
viewport_height: 1080
user_agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"

Content Not Extracted

# Debug: see full output
crwl https://example.com -o all -v

# Try different wait strategy
crwl https://example.com -c "wait_for=js:document.querySelector('.content')!==null"

Session Issues

# Verify session
crwl https://site.com -c "session_id=test" -o all | grep -i session

For comprehensive API documentation, see Complete SDK Reference.

Weekly Installs

285

Repository

brettdavies/cra…ai-skill

GitHub Stars

First Seen

Feb 6, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykFail

Installed on

codex262

opencode262

gemini-cli260

github-copilot255

amp247

kimi-cli246

Skills CLI 使用指南：AI Agent 技能包管理器安装与管理教程

27,400 周安装

Crawl4AI：Python网络爬虫与数据提取工具，支持CLI和SDK，高效生成Markdown

🇨🇳中文介绍

Crawl4AI

概述

快速开始

安装

CLI（推荐）

Python SDK

相关 Skills

核心概念

配置层级

关键参数

输出内容

Markdown 生成（主要用例）

CLI

Python SDK

数据提取

1. 基于模式的 CSS 提取（最高效）

2. 基于 LLM 的提取

高级模式

动态内容（JavaScript 密集型网站）

多 URL 处理

会话与身份验证

反检测与代理

常见用例

文档转 Markdown

电子商务产品监控

新闻聚合

交互式问答

资源

提供的脚本

参考文档

最佳实践

故障排除

JavaScript 未加载

机器人检测问题

内容未提取

会话问题

🇺🇸English

Crawl4AI

Overview

Quick Start

Installation

CLI (Recommended)

Python SDK

Core Concepts

Configuration Layers

Key Parameters

Output Content

Markdown Generation (Primary Use Case)

CLI

Python SDK

Data Extraction

1. Schema-Based CSS Extraction (Most Efficient)

2. LLM-Based Extraction

Advanced Patterns

Dynamic Content (JavaScript-Heavy Sites)

Multi-URL Processing

Session & Authentication

Anti-Detection & Proxies

Common Use Cases

Documentation to Markdown

E-commerce Product Monitoring

News Aggregation

Interactive Q&A

Resources

Provided Scripts

Reference Documentation

Best Practices

Troubleshooting

JavaScript Not Loading

Bot Detection Issues

Content Not Extracted

Session Issues

最新 Skills