Firecrawl API：AI数据提取与网站爬取工具，支持Markdown/HTML转换和结构化数据抓取

firecrawl by vm0-ai/vm0-skills

162 周安装量

47 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/vm0-ai/vm0-skills --skill firecrawl

AI/机器学习自动化数据处理

🇨🇳中文介绍

Firecrawl

通过直接的 curl 调用来使用 Firecrawl API，以抓取网站并为 AI 提取数据。

官方文档：https://docs.firecrawl.dev/

使用时机

当你需要以下功能时，请使用此技能：

抓取单个网页并转换为 Markdown/HTML
爬取整个网站并提取所有页面
发现网站上的所有 URL
搜索网络并获取完整的页面内容
使用 AI 提取结构化数据

前提条件

在 https://www.firecrawl.dev/ 注册
从仪表板获取你的 API 密钥

export FIRECRAWL_TOKEN="fc-your-api-key"

使用方法

以下所有示例均假设你已设置 FIRECRAWL_TOKEN 环境变量。

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

参数	类型	描述
`url`	string	要抓取的 URL（必需）
`formats`	array	`markdown`、`html`、`rawHtml`、`screenshot`、`links`
`onlyMainContent`	boolean	跳过页眉/页脚
`timeout`	number	超时时间（毫秒）

2. 爬取 - 整个网站

爬取网站的所有页面（异步操作）。

写入 /tmp/firecrawl_request.json：

{
  "url": "https://example.com",
  "limit": 50,
  "maxDepth": 2
}

curl -s -X POST "https://api.firecrawl.dev/v1/crawl" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json

{
  "success": true,
  "id": "crawl-job-id-here"
}

将 <job-id> 替换为爬取请求返回的实际作业 ID：

curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq '{status, completed, total}'

将 <job-id> 替换为实际的作业 ID：

curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq '.data[] | {url: .metadata.url, title: .metadata.title}'

带路径过滤器的爬取

写入 /tmp/firecrawl_request.json：

{
  "url": "https://blog.example.com",
  "limit": 20,
  "maxDepth": 3,
  "includePaths": ["/posts/*"],
  "excludePaths": ["/admin/*", "/login"]
}

curl -s -X POST "https://api.firecrawl.dev/v1/crawl" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json

参数	类型	描述
`url`	string	起始 URL（必需）
`limit`	number	最大爬取页面数（默认：100）
`maxDepth`	number	最大爬取深度（默认：3）
`includePaths`	array	要包含的路径（例如，`/blog/*`）
`excludePaths`	array	要排除的路径

3. 映射 - URL 发现

快速获取网站上的所有 URL。

写入 /tmp/firecrawl_request.json：

{
  "url": "https://example.com"
}

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.links[:10]'

带搜索过滤器的映射

写入 /tmp/firecrawl_request.json：

{
  "url": "https://shop.example.com",
  "search": "product",
  "limit": 500
}

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.links'

参数	类型	描述
`url`	string	网站 URL（必需）
`search`	string	过滤包含关键词的 URL
`limit`	number	返回的最大 URL 数（默认：1000）

4. 搜索 - 网络搜索

搜索网络并获取完整的页面内容。

写入 /tmp/firecrawl_request.json：

{
  "query": "AI news 2024",
  "limit": 5
}

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, url: .url}'

带完整内容的搜索

写入 /tmp/firecrawl_request.json：

{
  "query": "machine learning tutorials",
  "limit": 3,
  "scrapeOptions": {
    "formats": ["markdown"]
  }
}

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, content: .markdown[:500]}'

参数	类型	描述
`query`	string	搜索查询（必需）
`limit`	number	结果数量（默认：10）
`scrapeOptions`	object	抓取结果的选项

5. 提取 - AI 数据提取

使用 AI 从页面中提取结构化数据。

写入 /tmp/firecrawl_request.json：

{
  "urls": ["https://example.com/product/123"],
  "prompt": "Extract the product name, price, and description"
}

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

写入 /tmp/firecrawl_request.json：

{
  "urls": ["https://example.com/product/123"],
  "prompt": "Extract product information",
  "schema": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "price": {"type": "number"},
      "currency": {"type": "string"},
      "inStock": {"type": "boolean"}
    }
  }
}

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

从多个 URL 提取

写入 /tmp/firecrawl_request.json：

{
  "urls": [
    "https://example.com/product/1",
    "https://example.com/product/2"
  ],
  "prompt": "Extract product name and price"
}

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

参数	类型	描述
`urls`	array	要从中提取的 URL（必需）
`prompt`	string	要提取的数据描述（必需）
`schema`	object	用于结构化输出的 JSON 模式

写入 /tmp/firecrawl_request.json：

{
  "url": "https://docs.python.org/3/tutorial/",
  "formats": ["markdown"],
  "onlyMainContent": true
}

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq -r '.data.markdown' > python-tutorial.md

查找所有博客文章

写入 /tmp/firecrawl_request.json：

{
  "url": "https://blog.example.com",
  "search": "post"
}

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq -r '.links[]'

写入 /tmp/firecrawl_request.json：

{
  "query": "best practices REST API design 2024",
  "limit": 5,
  "scrapeOptions": {"formats": ["markdown"]}
}

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, url: .url}'

写入 /tmp/firecrawl_request.json：

{
  "urls": ["https://example.com/pricing"],
  "prompt": "Extract all pricing tiers with name, price, and features"
}

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

轮询爬取直到完成

将 <job-id> 替换为实际的作业 ID：

while true; do
  STATUS="$(curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq -r '.status')"
  echo "Status: $STATUS"
  [ "$STATUS" = "completed" ] && break
  sleep 5
done

{
  "success": true,
  "data": {
  "markdown": "# Page Title\n\nContent...",
  "metadata": {
  "title": "Page Title",
  "description": "...",
  "url": "https://..."
  }
  }
}

{
  "success": true,
  "status": "completed",
  "completed": 50,
  "total": 50,
  "data": [...]
}

速率限制：在请求之间添加延迟以避免 429 错误
爬取限制：设置合理的 limit 值以控制 API 使用
主要内容：使用 onlyMainContent: true 以获得更清晰的输出
异步爬取：大型爬取是异步的；轮询 /crawl/{id} 获取状态
提取提示：提示要具体以获得更好的 AI 提取结果
检查成功状态：始终检查响应中的 success 字段

2026 年 1 月 24 日

🇺🇸English

Firecrawl

Use the Firecrawl API via direct curl calls to scrape websites and extract data for AI.

Official docs: https://docs.firecrawl.dev/

When to Use

Use this skill when you need to:

Scrape a webpage and convert to markdown/HTML
Crawl an entire website and extract all pages
Discover all URLs on a website
Search the web and get full page content
Extract structured data using AI

Prerequisites

Sign up at https://www.firecrawl.dev/
Get your API key from the dashboard

export FIRECRAWL_TOKEN="fc-your-api-key"

How to Use

All examples below assume you have FIRECRAWL_TOKEN set.

Base URL: https://api.firecrawl.dev/v1

1. Scrape - Single Page

Extract content from a single webpage.

Basic Scrape

Write to /tmp/firecrawl_request.json:

{
  "url": "https://example.com",
  "formats": ["markdown"]
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json

Scrape with Options

Write to /tmp/firecrawl_request.json:

{
  "url": "https://docs.example.com/api",
  "formats": ["markdown"],
  "onlyMainContent": true,
  "timeout": 30000
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data.markdown'

Get HTML Instead

Write to /tmp/firecrawl_request.json:

{
  "url": "https://example.com",
  "formats": ["html"]
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data.html'

Get Screenshot

Write to /tmp/firecrawl_request.json:

{
  "url": "https://example.com",
  "formats": ["screenshot"]
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data.screenshot'

Scrape Parameters:

Parameter	Type	Description
`url`	string	URL to scrape (required)
`formats`	array	`markdown`, `html`, `rawHtml`, `screenshot`, `links`
`onlyMainContent`

2. Crawl - Entire Website

Crawl all pages of a website (async operation).

Start a Crawl

Write to /tmp/firecrawl_request.json:

{
  "url": "https://example.com",
  "limit": 50,
  "maxDepth": 2
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/crawl" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json

Response:

{
  "success": true,
  "id": "crawl-job-id-here"
}

Check Crawl Status

Replace <job-id> with the actual job ID returned from the crawl request:

curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq '{status, completed, total}'

Get Crawl Results

Replace <job-id> with the actual job ID:

curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq '.data[] | {url: .metadata.url, title: .metadata.title}'

Crawl with Path Filters

Write to /tmp/firecrawl_request.json:

{
  "url": "https://blog.example.com",
  "limit": 20,
  "maxDepth": 3,
  "includePaths": ["/posts/*"],
  "excludePaths": ["/admin/*", "/login"]
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/crawl" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json

Crawl Parameters:

Parameter	Type	Description
`url`	string	Starting URL (required)
`limit`	number	Max pages to crawl (default: 100)
`maxDepth`	number	Max crawl depth (default: 3)
`includePaths`	array	Paths to include (e.g., `/blog/*`)
`excludePaths`

3. Map - URL Discovery

Get all URLs from a website quickly.

Basic Map

Write to /tmp/firecrawl_request.json:

{
  "url": "https://example.com"
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.links[:10]'

Map with Search Filter

Write to /tmp/firecrawl_request.json:

{
  "url": "https://shop.example.com",
  "search": "product",
  "limit": 500
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.links'

Map Parameters:

Parameter	Type	Description
`url`	string	Website URL (required)
`search`	string	Filter URLs containing keyword
`limit`	number	Max URLs to return (default: 1000)

4. Search - Web Search

Search the web and get full page content.

Basic Search

Write to /tmp/firecrawl_request.json:

{
  "query": "AI news 2024",
  "limit": 5
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, url: .url}'

Search with Full Content

Write to /tmp/firecrawl_request.json:

{
  "query": "machine learning tutorials",
  "limit": 3,
  "scrapeOptions": {
    "formats": ["markdown"]
  }
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, content: .markdown[:500]}'

Search Parameters:

Parameter	Type	Description
`query`	string	Search query (required)
`limit`	number	Number of results (default: 10)
`scrapeOptions`	object	Options for scraping results

5. Extract - AI Data Extraction

Extract structured data from pages using AI.

Basic Extract

Write to /tmp/firecrawl_request.json:

{
  "urls": ["https://example.com/product/123"],
  "prompt": "Extract the product name, price, and description"
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

Extract with Schema

Write to /tmp/firecrawl_request.json:

{
  "urls": ["https://example.com/product/123"],
  "prompt": "Extract product information",
  "schema": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "price": {"type": "number"},
      "currency": {"type": "string"},
      "inStock": {"type": "boolean"}
    }
  }
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

Extract from Multiple URLs

Write to /tmp/firecrawl_request.json:

{
  "urls": [
    "https://example.com/product/1",
    "https://example.com/product/2"
  ],
  "prompt": "Extract product name and price"
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

Extract Parameters:

Parameter	Type	Description
`urls`	array	URLs to extract from (required)
`prompt`	string	Description of data to extract (required)
`schema`	object	JSON schema for structured output

Practical Examples

Scrape Documentation

Write to /tmp/firecrawl_request.json:

{
  "url": "https://docs.python.org/3/tutorial/",
  "formats": ["markdown"],
  "onlyMainContent": true
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/scrape" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq -r '.data.markdown' > python-tutorial.md

Find All Blog Posts

Write to /tmp/firecrawl_request.json:

{
  "url": "https://blog.example.com",
  "search": "post"
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/map" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq -r '.links[]'

Research a Topic

Write to /tmp/firecrawl_request.json:

{
  "query": "best practices REST API design 2024",
  "limit": 5,
  "scrapeOptions": {"formats": ["markdown"]}
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/search" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data[] | {title: .metadata.title, url: .url}'

Extract Pricing Data

Write to /tmp/firecrawl_request.json:

{
  "urls": ["https://example.com/pricing"],
  "prompt": "Extract all pricing tiers with name, price, and features"
}

Then run:

curl -s -X POST "https://api.firecrawl.dev/v1/extract" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" -H "Content-Type: application/json" -d @/tmp/firecrawl_request.json | jq '.data'

Poll Crawl Until Complete

Replace <job-id> with the actual job ID:

while true; do
  STATUS="$(curl -s "https://api.firecrawl.dev/v1/crawl/<job-id>" -H "Authorization: Bearer $(printenv FIRECRAWL_TOKEN)" | jq -r '.status')"
  echo "Status: $STATUS"
  [ "$STATUS" = "completed" ] && break
  sleep 5
done

Response Format

Scrape Response

{
  "success": true,
  "data": {
  "markdown": "# Page Title\n\nContent...",
  "metadata": {
  "title": "Page Title",
  "description": "...",
  "url": "https://..."
  }
  }
}

Crawl Status Response

{
  "success": true,
  "status": "completed",
  "completed": 50,
  "total": 50,
  "data": [...]
}

Guidelines

Rate limits : Add delays between requests to avoid 429 errors
Crawl limits : Set reasonable limit values to control API usage
Main content : Use onlyMainContent: true for cleaner output
Async crawls : Large crawls are async; poll /crawl/{id} for status
Extract prompts : Be specific for better AI extraction results
Check success : Always check success field in responses

Weekly Installs

162

Repository

vm0-ai/vm0-skills

GitHub Stars

First Seen

Jan 24, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

opencode145

codex143

gemini-cli142

github-copilot139

cursor139

amp134

AI Elements：基于shadcn/ui的AI原生应用组件库，快速构建对话界面

63,800 周安装

Firecrawl API：AI数据提取与网站爬取工具，支持Markdown/HTML转换和结构化数据抓取

🇨🇳中文介绍

Firecrawl

使用时机

前提条件

使用方法

相关 Skills

1. 抓取 - 单页

基础抓取

带选项的抓取

获取 HTML 格式

获取截图

2. 爬取 - 整个网站

开始爬取

检查爬取状态

获取爬取结果

带路径过滤器的爬取

3. 映射 - URL 发现

基础映射

带搜索过滤器的映射

4. 搜索 - 网络搜索

基础搜索

带完整内容的搜索

5. 提取 - AI 数据提取

基础提取

带模式的提取

从多个 URL 提取

实用示例

抓取文档

查找所有博客文章

研究一个主题

提取定价数据

轮询爬取直到完成

响应格式

抓取响应

爬取状态响应

指南

🇺🇸English

Firecrawl

When to Use

Prerequisites

How to Use

1. Scrape - Single Page

Basic Scrape

Scrape with Options

Get HTML Instead

Get Screenshot

2. Crawl - Entire Website

Start a Crawl

Check Crawl Status

Get Crawl Results

Crawl with Path Filters

3. Map - URL Discovery

Basic Map

Map with Search Filter

4. Search - Web Search

Basic Search

Search with Full Content

5. Extract - AI Data Extraction

Basic Extract

Extract with Schema

Extract from Multiple URLs

Practical Examples

Scrape Documentation

Find All Blog Posts

Research a Topic

Extract Pricing Data

Poll Crawl Until Complete

Response Format

Scrape Response

Crawl Status Response

Guidelines

最新 Skills