FireCrawl 自动化网络研究工具：从Markdown提取主题，自动抓取并生成研究文档

firecrawl-research by glebis/claude-skills

147 周安装量

87 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/glebis/claude-skills --skill firecrawl-research

研究方法信息聚合自动化

🇨🇳中文介绍

FireCrawl 研究

概述

通过使用 FireCrawl API 自动搜索和抓取网络资源来丰富研究文档。从 Markdown 文件中提取研究主题，并生成包含原始材料的全面研究文档。

何时使用此技能

当用户出现以下情况时使用此技能：

说“使用 FireCrawl 研究这个主题”
请求用网络资源丰富笔记或文档
想要收集 Markdown 文件中列出主题的信息
需要系统地搜索和抓取多个主题

工作原理

1. 主题提取

脚本使用两种方法自动从 Markdown 文件中提取研究主题：

方法 1：标题

## Spatial Reasoning in AI
### Computer Vision Applications

Spatial Reasoning in AI 和 Computer Vision Applications 都将成为研究主题。

方法 2：研究标签

- [research] Large Language Models for robotics
- [search] Theory of Mind in autonomous driving

两个带标签的条目都将成为研究主题。

2. 搜索与抓取

对于每个主题：

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

限制每个主题的结果数量

python scripts/firecrawl_research.py research.md ./output 3

每个主题最多检索 3 个结果。

将 .env.example 复制为 .env：
```
 cp .env.example .env
 
```

添加 FireCrawl API 密钥：

 FIRECRAWL_API_KEY=fc-your-actual-api-key

脚本会自动从技能的 .env 文件中加载 API 密钥。

脚本包含针对 FireCrawl 免费层的自动速率限制：

免费层限制： 5 次请求/分钟
内置延迟： 主题之间间隔 12 秒
防止 API 错误和积分耗尽

处理多个主题时，预计：

5 个主题：约 1 分钟
10 个主题：约 2 分钟
20 个主题：约 4 分钟

用户请求： “使用 FireCrawl 研究这些 AI 主题”

输入文件 (ai-research.md):

# AI Research Topics

## Spatial Reasoning in Vision-Language Models

- [research] Embodied AI for robotics
- [research] Computer Use Agents

python scripts/firecrawl_research.py ai-research.md ./research_output 5

research_output/
├── Spatial_Reasoning_in_Vision-Language_Models_20251122_140530.md
├── Embodied_AI_for_robotics_20251122_140542.md
└── Computer_Use_Agents_20251122_140554.md

每个文件包含：

主题标题
时间戳
来源数量
来自最多 5 个来源的完整抓取内容
来源 URL

模式 1：快速研究

从现有笔记中提取主题，进行研究，保存到当前文件夹：

python scripts/firecrawl_research.py my-notes.md

模式 2：有组织的研究

为研究结果创建专用的输出文件夹：

python scripts/firecrawl_research.py topics.md ./research_results

模式 3：深度研究

增加每个主题的结果数量以获得全面覆盖：

python scripts/firecrawl_research.py topics.md ./deep_research 10

模式 4：Obsidian 知识库集成

直接输出到知识库的研究文件夹：

python scripts/firecrawl_research.py topics.md ~/Brains/brain/Research

“未找到 API 密钥”

在技能文件夹中创建包含 FIRECRAWL_API_KEY=... 的 .env 文件。

“超出速率限制”

免费层：5 次请求/分钟
脚本内置 12 秒延迟
如果仍然超出限制，请减少主题数量或在运行之间等待

检查 FireCrawl 账户积分
升级计划或等待积分重置

“未找到主题”

使用以下格式向 Markdown 添加主题：

## Header format
- [research] Topic format
- [search] Topic format

位置： scripts/firecrawl_research.py

python-dotenv - 环境变量管理
requests - 向 FireCrawl API 发送 HTTP 请求

安装依赖项：

pip install python-dotenv requests

使用的 FireCrawl 功能：

/v1/search 端点 - 搜索并自动抓取
scrapeOptions.formats: ['markdown'] - Markdown 输出
scrapeOptions.onlyMainContent: true - 过滤噪音

此技能包含用于以 Markdown 格式撰写科学论文的模板。

1. Pandoc 学术论文 (assets/templates/pandoc-scholarly-paper.md)

标准学术论文格式
兼容 Pandoc 转换器
通过 BibTeX 支持引用
可导出为 PDF、DOCX、HTML

2. MyST 科学论文 (assets/templates/myst-scientific-paper.md)

MyST（Markedly Structured Text）格式
高级交叉引用
专业科学出版
多格式导出（PDF、LaTeX、DOCX）

将模板复制到您的项目：

cp assets/templates/pandoc-scholarly-paper.md my-paper.md
# 或
cp assets/templates/myst-scientific-paper.md my-paper.md

更新 YAML 前置元数据（标题、作者、所属机构）
在各部分中撰写您的内容
使用 [@AuthorYear]（Pandoc）或 {cite}\AuthorYear``（MyST）添加引用

转换为 PDF/DOCX：

python scripts/convert_academic.py my-paper.md pdf
python scripts/convert_academic.py my-paper.md docx
python scripts/convert_academic.py my-paper.md pdf --myst  # 用于 MyST

将 FireCrawl 研究结果转换为 BibTeX 参考文献条目：

python scripts/generate_bibliography.py research_output/*.md -o references.bib

从 FireCrawl Markdown 文件中提取 URL 和标题
生成 BibTeX @misc 条目
自动创建引用键
添加访问日期

示例工作流程：

# 1. 研究主题
python scripts/firecrawl_research.py topics.md ./research

# 2. 生成参考文献
python scripts/generate_bibliography.py research/*.md -o refs.bib

# 3. 复制模板
cp assets/templates/pandoc-scholarly-paper.md paper.md

# 4. 编辑 paper.md（添加内容，引用来源）

# 5. 转换为 PDF
python scripts/convert_academic.py paper.md pdf

Pandoc 语法：

Recent research [@Smith2024] shows...
Multiple studies [@Jones2023; @Brown2024] indicate...

Recent research {cite}`Smith2024` shows...
Multiple studies {cite}`Jones2023,Brown2024` indicate...

示例参考文献文件

assets/references.bib 中提供了一个包含常见条目类型的示例参考文献：

期刊文章 (@article)
会议论文 (@inproceedings)
书籍 (@book)
博士论文 (@phdthesis)
网络资源 (@misc)
预印本 (@article 带 arXiv)

分层组织主题 - 使用 ## 表示主要主题，### 表示子主题
使用描述性名称 - 主题文本将成为文件名，请使其清晰明了
批量处理 - 将相关主题分组在一个文件中以提高效率
输出组织 - 为不同的研究项目创建单独的文件夹
内容审查 - 为便于阅读，每个来源的结果截断为 3000 字符
学术工作流程 - 使用参考文献生成器在论文中引用研究来源
模板定制 - 根据您领域的引用风格修改模板

无摘要功能 - 返回原始抓取内容，而非摘要
无去重功能 - 重复来源可能出现在不同主题中
无质量排名 - 所有结果同等对待
仅创建新文件 - 不追加到现有文件
免费层限制 - 速率限制影响处理速度

🇺🇸English

FireCrawl Research

Overview

Enrich research documents by automatically searching and scraping web sources using the FireCrawl API. Extract research topics from markdown files and generate comprehensive research documents with source material.

When to Use This Skill

Use this skill when the user:

Says "Research this topic using FireCrawl"
Requests to enrich notes or documents with web sources
Wants to gather information about topics listed in a markdown file
Needs to search and scrape multiple topics systematically

How It Works

1. Topic Extraction

The script automatically extracts research topics from markdown files using two methods:

Method 1: Headers

## Spatial Reasoning in AI
### Computer Vision Applications

Both Spatial Reasoning in AI and Computer Vision Applications become research topics.

Method 2: Research Tags

- [research] Large Language Models for robotics
- [search] Theory of Mind in autonomous driving

Both tagged items become research topics.

2. Search and Scrape

For each topic:

Searches FireCrawl with the topic as query
Retrieves up to N results (default: 5)
Automatically scrapes full content from each result
Extracts markdown-formatted content (main content only)

3. Output Generation

Creates new markdown files in the specified output directory:

One file per topic
Filename: {topic}_{timestamp}.md
Contains: title, date, sources count, full scraped content
Each source includes: title, URL, markdown content

Usage

Basic Usage

python scripts/firecrawl_research.py research.md

Outputs to current directory.

Specify Output Directory

python scripts/firecrawl_research.py research.md ./output

Creates files in ./output/ folder.

Limit Results Per Topic

python scripts/firecrawl_research.py research.md ./output 3

Retrieves maximum 3 results per topic.

Configuration

API Key Setup

Copy .env.example to .env:
```
cp .env.example .env
```

Add FireCrawl API key:

FIRECRAWL_API_KEY=fc-your-actual-api-key

The script automatically loads the API key from the skill's .env file.

Rate Limiting

The script includes automatic rate limiting for FireCrawl's free tier:

Free tier limit: 5 requests/minute
Built-in delay: 12 seconds between topics
Prevents API errors and credit exhaustion

When processing multiple topics, expect:

5 topics: ~1 minute
10 topics: ~2 minutes
20 topics: ~4 minutes

Workflow Example

User request: "Research these AI topics using FireCrawl"

Input file (ai-research.md):

# AI Research Topics

## Spatial Reasoning in Vision-Language Models

- [research] Embodied AI for robotics
- [research] Computer Use Agents

Command:

python scripts/firecrawl_research.py ai-research.md ./research_output 5

Output:

research_output/
├── Spatial_Reasoning_in_Vision-Language_Models_20251122_140530.md
├── Embodied_AI_for_robotics_20251122_140542.md
└── Computer_Use_Agents_20251122_140554.md

Each file contains:

Topic title
Timestamp
Source count
Full scraped content from up to 5 sources
Source URLs

Common Patterns

Pattern 1: Quick Research

Extract topics from existing notes, research them, save to current folder:

python scripts/firecrawl_research.py my-notes.md

Pattern 2: Organized Research

Create dedicated output folder for research results:

python scripts/firecrawl_research.py topics.md ./research_results

Pattern 3: Deep Dive

Increase results per topic for comprehensive coverage:

python scripts/firecrawl_research.py topics.md ./deep_research 10

Pattern 4: Obsidian Vault Integration

Direct output to vault's research folder:

python scripts/firecrawl_research.py topics.md ~/Brains/brain/Research

Error Handling

"API key not found"

Create .env file in skill folder with FIRECRAWL_API_KEY=...

"Rate limit exceeded"

Free tier: 5 req/min
Script has 12s delay built-in
If still hitting limit, reduce topics or wait between runs

"Insufficient credits"

Check FireCrawl account credits
Upgrade plan or wait for credit reset

"No topics found"

Add topics to markdown using:

## Header format
- [research] Topic format
- [search] Topic format

Script Details

Location: scripts/firecrawl_research.py

Dependencies:

python-dotenv - Environment variable management
requests - HTTP requests to FireCrawl API

Install dependencies:

pip install python-dotenv requests

FireCrawl Features Used:

/v1/search endpoint - Search with automatic scraping
scrapeOptions.formats: ['markdown'] - Markdown output
scrapeOptions.onlyMainContent: true - Filter noise

Academic Writing Templates

This skill includes templates for writing scientific papers in markdown format.

Available Templates

1. Pandoc Scholarly Paper (assets/templates/pandoc-scholarly-paper.md)

Standard academic paper format
Compatible with Pandoc converter
Supports citations via BibTeX
Exports to PDF, DOCX, HTML

2. MyST Scientific Paper (assets/templates/myst-scientific-paper.md)

MyST (Markedly Structured Text) format
Advanced cross-referencing
Professional scientific publishing
Multi-format export (PDF, LaTeX, DOCX)

Using Templates

Copy template to your project:

cp assets/templates/pandoc-scholarly-paper.md my-paper.md
# or
cp assets/templates/myst-scientific-paper.md my-paper.md

Edit content:

Update YAML frontmatter (title, authors, affiliations)
Write your content in sections
Add citations using [@AuthorYear] (Pandoc) or {cite}\AuthorYear`` (MyST)

Convert to PDF/DOCX:

python scripts/convert_academic.py my-paper.md pdf
python scripts/convert_academic.py my-paper.md docx
python scripts/convert_academic.py my-paper.md pdf --myst  # For MyST

Bibliography Generation

Convert FireCrawl research results into BibTeX bibliography entries:

python scripts/generate_bibliography.py research_output/*.md -o references.bib

What it does:

Extracts URLs and titles from FireCrawl markdown files
Generates BibTeX @misc entries
Creates citation keys automatically
Adds access dates

Example workflow:

# 1. Research topics
python scripts/firecrawl_research.py topics.md ./research

# 2. Generate bibliography
python scripts/generate_bibliography.py research/*.md -o refs.bib

# 3. Copy template
cp assets/templates/pandoc-scholarly-paper.md paper.md

# 4. Edit paper.md (add content, cite sources)

# 5. Convert to PDF
python scripts/convert_academic.py paper.md pdf

Citation Examples

Pandoc syntax:

Recent research [@Smith2024] shows...
Multiple studies [@Jones2023; @Brown2024] indicate...

MyST syntax:

Recent research {cite}`Smith2024` shows...
Multiple studies {cite}`Jones2023,Brown2024` indicate...

Example Bibliography File

An example bibliography is provided in assets/references.bib with common entry types:

Journal articles (@article)
Conference papers (@inproceedings)
Books (@book)
PhD theses (@phdthesis)
Web resources (@misc)
Preprints (@article with arXiv)

Tips

Organize topics hierarchically - Use ## for main topics, ### for subtopics
Use descriptive names - Topic text becomes filename, make it clear
Batch processing - Group related topics in one file for efficiency
Output organization - Create separate folders for different research projects
Content review - Results are truncated at 3000 chars/source for readability
Academic workflow - Use bibliography generator to cite research sources in papers
Template customization - Modify templates for your field's citation style

Limitations

No summarization - Returns raw scraped content, not summaries
No deduplication - Duplicate sources may appear across topics
No quality ranking - All results treated equally
New files only - Does not append to existing files
Free tier constraints - Rate limiting affects processing speed

Weekly Installs

124

Repository

glebis/claude-skills

GitHub Stars

First Seen

Jan 24, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

cursor115

opencode114

gemini-cli113

codex112

github-copilot109

claude-code107

Skills CLI 使用指南：AI Agent 技能包管理器安装与管理教程

43,100 周安装