目录

1. 核心工作流
2. 速率限制策略
3. URL 验证
4. 数据提取
5. 错误处理
6. 处理结果
7. 支持文件
8. 预期结果

核心工作流

步骤 1：准备 URL 列表

创建一个包含要抓取的 URL 的文本文件（每行一个）：

https://example.com/products?page=1
https://example.com/products?page=2
https://example.com/products?page=3

验证 URL 并检查 robots.txt 合规性：

python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"

步骤 2：初始化抓取会话

导航到网站并拍摄快照以了解结构：

await browser_navigate({ url: firstUrl });
await browser_snapshot();

使用快照识别用于数据提取的 CSS 选择器。

步骤 3：实施带速率限制的爬取

在请求之间使用随机延迟（至少 1-3 秒）：

const results = [];

for (const url of urlList) {
  // 导航到页面
  await browser_navigate({ url });

  // 等待内容加载
  await browser_wait_for({ text: 'Expected content marker' });

  // 添加礼貌延迟（1-3 秒）
  const delay = Math.random() * 2 + 1;
  await browser_wait_for({ time: delay });

  // 提取数据
  const pageData = await browser_evaluate({
    function: `/* extraction code */`
  });

  results.push(...pageData);

  // 检查控制台是否有错误/警告
  const console = await browser_console_messages();
  // 监控速率限制警告
}

步骤 4：提取结构化数据

使用 browser_evaluate 通过 JavaScript 提取数据：

const data = await browser_evaluate({
  function: `
    try {
      return Array.from(document.querySelectorAll('.item')).map(el => ({
        title: el.querySelector('.title')?.textContent?.trim(),
        price: el.querySelector('.price')?.textContent?.trim(),
        rating: el.querySelector('.rating')?.textContent?.trim(),
        url: el.querySelector('a')?.getAttribute('href')
      })).filter(item => item.title && item.price); // 过滤不完整记录
    } catch (e) {
      console.error('Extraction failed:', e);
      return [];
    }
  `
});

有关全面的提取模式，请参阅 references/extraction-patterns.md。

步骤 5：处理错误和速率限制

监控速率限制指示器：

// 通过 browser_network_requests 检查 HTTP 响应
const requests = await browser_network_requests();
const rateLimited = requests.some(r => r.status === 429 || r.status === 503);

if (rateLimited) {
  // 指数退避
  await browser_wait_for({ time: 10 }); // 等待 10 秒
  // 重试或跳过
}

// 检查控制台是否有阻止消息
const console = await browser_console_messages({ pattern: 'rate limit|blocked|captcha' });
if (console.length > 0) {
  // 处理阻止
}

步骤 6：聚合和存储结果

将结果保存到 JSON 文件：

// 在您的抓取脚本中
fs.writeFileSync('scraped.json', JSON.stringify({ results }, null, 2));

处理并转换为所需格式：

# 查看统计信息
python scripts/process_results.py scraped.json --stats

# 转换为 CSV
python scripts/process_results.py scraped.json -o output.csv

# 转换为 Markdown 表格
python scripts/process_results.py scraped.json -o output.md

速率限制策略

最小延迟

始终在请求之间添加延迟：

• 标准网站：1-3 秒（随机）
• 高流量网站：3-5 秒
• 小型网站：5-10 秒
• 错误后：指数退避（5秒、10秒、20秒、40秒）

实现

// 1-3 秒之间的随机延迟
const randomDelay = () => Math.random() * 2 + 1;
await browser_wait_for({ time: randomDelay() });

// 速率限制后的指数退避
let backoffSeconds = 5;
for (let retry = 0; retry < 3; retry++) {
  try {
    await browser_navigate({ url });
    break; // 成功
  } catch (e) {
    await browser_wait_for({ time: backoffSeconds });
    backoffSeconds *= 2; // 每次重试延迟加倍
  }
}

自适应速率限制

根据响应调整延迟：

有关详细的速率限制策略，请参阅 references/ethical-scraping.md。

URL 验证

在抓取前使用 validate_urls.py 以确保合规性：

# 基本验证
python scripts/validate_urls.py urls.txt

# 使用特定用户代理检查 robots.txt
python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"

# 严格模式（任何无效/不允许的 URL 都退出）
python scripts/validate_urls.py urls.txt --strict

输出包括：

• URL 格式验证
• 域名分组
• robots.txt 合规性检查
• 摘要统计信息

数据提取

基本模式

// 单页提取
const data = await browser_evaluate({
  function: `
    Array.from(document.querySelectorAll('.item')).map(el => ({
      field1: el.querySelector('.selector1')?.textContent?.trim(),
      field2: el.querySelector('.selector2')?.getAttribute('href')
    }))
  `
});

分页模式

let hasMore = true;
let page = 1;

while (hasMore) {
  await browser_navigate({ url: `${baseUrl}?page=${page}` });
  await browser_wait_for({ time: randomDelay() });

  const pageData = await browser_evaluate({ function: extractionCode });
  results.push(...pageData);

  // 检查是否有下一页
  hasMore = await browser_evaluate({
    function: `document.querySelector('.next:not(.disabled)') !== null`
  });

  page++;
}

有关以下内容，请参阅 references/extraction-patterns.md：

• 高级选择器
• 数据清理模式
• 表格提取
• JSON-LD 提取
• Shadow DOM 访问

错误处理

网络错误

try {
  await browser_navigate({ url });
} catch (e) {
  console.error(`Failed to load ${url}:`, e);
  failedUrls.push(url);
  continue; // 跳到下一个 URL
}

内容验证

const data = await browser_evaluate({ function: extractionCode });

if (!data || data.length === 0) {
  console.warn(`No data extracted from ${url}`);
  // 记录以供手动审查
}

// 验证数据结构
const validData = data.filter(item =>
  item.title && item.price // 确保必填字段存在
);

监控指示器

检查阻止/错误：

// 监控控制台
const console = await browser_console_messages({
  pattern: 'error|rate|limit|captcha',
  onlyErrors: true
});

if (console.length > 0) {
  console.log('Warnings detected:', console);
}

// 监控网络
const requests = await browser_network_requests();
const errors = requests.filter(r => r.status >= 400);

处理结果

查看统计信息

python scripts/process_results.py scraped.json --stats

输出：

📊 Statistics:
  Total records: 150
  Fields (5): title, price, rating, url, image
  Sample record: {...}

转换格式

# 转换为 CSV
python scripts/process_results.py scraped.json -o products.csv

# 转换为 JSON（紧凑格式）
python scripts/process_results.py scraped.json -o products.json --compact

# 转换为 Markdown 表格
python scripts/process_results.py scraped.json -o products.md

结合统计信息与转换

python scripts/process_results.py scraped.json -o products.csv --stats

支持文件

脚本

• scripts/validate_urls.py - 验证 URL 列表，检查 robots.txt 合规性，按域名分组
• scripts/process_results.py - 将抓取的 JSON 转换为 CSV/JSON/Markdown，查看统计信息

参考资料

• references/ethical-scraping.md - 关于速率限制、robots.txt、错误处理和监控的全面指南
• references/extraction-patterns.md - 用于数据提取、选择器、分页、表格的 JavaScript 模式

预期结果

成功抓取

✅ 已验证 50 个 URL
✅ 在 5 分钟内抓取了 50 个页面（6 次请求/分钟）
✅ 提取了 1,250 个产品
✅ 零速率限制错误
✅ 已导出到 products.csv（1,250 行）

带错误处理

⚠️  已验证 50 个 URL（2 个被 robots.txt 禁止）
✅ 抓取了 48 个页面
⚠️  3 个页面未返回数据（已记录供审查）
✅ 提取了 1,100 个产品
⚠️  1 个速率限制警告（已成功退避）
✅ 已导出到 products.csv（1,100 行）

速率限制检测

❌ 抓取 20 个页面后受到速率限制（429 响应）
✅ 已指数退避（5秒 → 10秒 → 20秒）
✅ 已成功恢复抓取
✅ 从 25 个页面提取了 450 个产品

预期收益

成功指标

• 成功率 > 95%（成功抓取的页面）
• 速率限制错误 < 5% 的请求
• 有效数据率 > 90%（完整记录）
• 抓取速度 6-12 次请求/分钟（礼貌爬取）

要求

工具

• Playwright MCP 浏览器工具
• Python 3.8+（用于脚本）
• 仅标准库（脚本无外部依赖）

知识

• 基本 CSS 选择器
• 用于数据提取的 JavaScript
• 了解 HTTP 状态码
• 了解网络抓取道德规范

应避免的危险信号

• ❌ 不检查 robots.txt 就进行抓取
• ❌ 请求之间没有延迟（攻击服务器）
• ❌ 忽略 429/503 响应代码
• ❌ 抓取个人/私人信息
• ❌ 不监控控制台是否有阻止消息
• ❌ 抓取明确禁止抓取的网站（检查服务条款）
• ❌ 违反版权使用抓取的数据
• ❌ 未正确处理分页（数据缺失）
• ❌ 硬编码选择器而没有备用方案
• ❌ 未验证提取的数据结构

注意事项

• 默认使用礼貌爬取：至少 1-3 秒延迟，根据网站响应调整
• 始终先检查 robots.txt：在抓取前使用 validate_urls.py
• 监控控制台和网络：注意速率限制警告并调整延迟
• 从小规模开始：在扩展到数百个 URL 之前，先用 5-10 个 URL 测试
• 保存进度：在中断的情况下增量写入结果
• 尊重服务条款：一些网站在其服务条款中禁止抓取
• 使用描述性用户代理：明确标识您的机器人
• 优雅地处理错误：记录失败以供手动审查，不要崩溃

每周安装次数

52

仓库

dawiddutoit/cus…m-claude

首次出现

2026年1月23日

安全审计

Gen Agent Trust HubPassSocketPassSnykWarn

已安装于

gemini-cli51

codex51

opencode51

github-copilot50

cursor50

amp49

Table of Contents

1. Core Workflow
2. Rate Limiting Strategy
3. URL Validation
4. Data Extraction
5. Error Handling
6. Processing Results
7. Supporting Files
8. Expected Outcomes

Core Workflow

Step 1: Prepare URL List

Create a text file with URLs to scrape (one per line):

https://example.com/products?page=1
https://example.com/products?page=2
https://example.com/products?page=3

Validate URLs and check robots.txt compliance:

python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"

Step 2: Initialize Scraping Session

Navigate to the site and take a snapshot to understand structure:

await browser_navigate({ url: firstUrl });
await browser_snapshot();

Identify CSS selectors for data extraction using the snapshot.

Step 3: Implement Rate-Limited Crawling

Use random delays between requests (1-3 seconds minimum):

const results = [];

for (const url of urlList) {
  // Navigate to page
  await browser_navigate({ url });

  // Wait for content to load
  await browser_wait_for({ text: 'Expected content marker' });

  // Add respectful delay (1-3 seconds)
  const delay = Math.random() * 2 + 1;
  await browser_wait_for({ time: delay });

  // Extract data
  const pageData = await browser_evaluate({
    function: `/* extraction code */`
  });

  results.push(...pageData);

  // Check console for errors/warnings
  const console = await browser_console_messages();
  // Monitor for rate limit warnings
}

Step 4: Extract Structured Data

Use browser_evaluate to extract data with JavaScript:

const data = await browser_evaluate({
  function: `
    try {
      return Array.from(document.querySelectorAll('.item')).map(el => ({
        title: el.querySelector('.title')?.textContent?.trim(),
        price: el.querySelector('.price')?.textContent?.trim(),
        rating: el.querySelector('.rating')?.textContent?.trim(),
        url: el.querySelector('a')?.getAttribute('href')
      })).filter(item => item.title && item.price); // Filter incomplete records
    } catch (e) {
      console.error('Extraction failed:', e);
      return [];
    }
  `
});

See references/extraction-patterns.md for comprehensive extraction patterns.

Step 5: Handle Errors and Rate Limits

Monitor for rate limiting indicators:

// Check HTTP responses via browser_network_requests
const requests = await browser_network_requests();
const rateLimited = requests.some(r => r.status === 429 || r.status === 503);

if (rateLimited) {
  // Back off exponentially
  await browser_wait_for({ time: 10 }); // Wait 10 seconds
  // Retry or skip
}

// Check console for blocking messages
const console = await browser_console_messages({ pattern: 'rate limit|blocked|captcha' });
if (console.length > 0) {
  // Handle blocking
}

Step 6: Aggregate and Store Results

Save results to JSON file:

// In your scraping script
fs.writeFileSync('scraped.json', JSON.stringify({ results }, null, 2));

Process and convert to desired format:

# View statistics
python scripts/process_results.py scraped.json --stats

# Convert to CSV
python scripts/process_results.py scraped.json -o output.csv

# Convert to Markdown table
python scripts/process_results.py scraped.json -o output.md

Rate Limiting Strategy

Minimum Delays

Always add delays between requests:

• Standard sites : 1-3 seconds (random)
• High-traffic sites : 3-5 seconds
• Small sites : 5-10 seconds
• After errors : Exponential backoff (5s, 10s, 20s, 40s)

Implementation

// Random delay between 1-3 seconds
const randomDelay = () => Math.random() * 2 + 1;
await browser_wait_for({ time: randomDelay() });

// Exponential backoff after rate limit
let backoffSeconds = 5;
for (let retry = 0; retry < 3; retry++) {
  try {
    await browser_navigate({ url });
    break; // Success
  } catch (e) {
    await browser_wait_for({ time: backoffSeconds });
    backoffSeconds *= 2; // Double delay each retry
  }
}

Adaptive Rate Limiting

Adjust delays based on response:

See references/ethical-scraping.md for detailed rate limiting strategies.

URL Validation

Use validate_urls.py before scraping to ensure compliance:

# Basic validation
python scripts/validate_urls.py urls.txt

# Check robots.txt with specific user agent
python scripts/validate_urls.py urls.txt --user-agent "MyBot/1.0"

# Strict mode (exit on any invalid/disallowed URL)
python scripts/validate_urls.py urls.txt --strict

Output includes :

• URL format validation
• Domain grouping
• robots.txt compliance check
• Summary statistics

Data Extraction

Basic Pattern

// Single page extraction
const data = await browser_evaluate({
  function: `
    Array.from(document.querySelectorAll('.item')).map(el => ({
      field1: el.querySelector('.selector1')?.textContent?.trim(),
      field2: el.querySelector('.selector2')?.getAttribute('href')
    }))
  `
});

Pagination Pattern

let hasMore = true;
let page = 1;

while (hasMore) {
  await browser_navigate({ url: `${baseUrl}?page=${page}` });
  await browser_wait_for({ time: randomDelay() });

  const pageData = await browser_evaluate({ function: extractionCode });
  results.push(...pageData);

  // Check for next page
  hasMore = await browser_evaluate({
    function: `document.querySelector('.next:not(.disabled)') !== null`
  });

  page++;
}

See references/extraction-patterns.md for:

• Advanced selectors
• Data cleaning patterns
• Table extraction
• JSON-LD extraction
• Shadow DOM access

Error Handling

Network Errors

try {
  await browser_navigate({ url });
} catch (e) {
  console.error(`Failed to load ${url}:`, e);
  failedUrls.push(url);
  continue; // Skip to next URL
}

Content Validation

const data = await browser_evaluate({ function: extractionCode });

if (!data || data.length === 0) {
  console.warn(`No data extracted from ${url}`);
  // Log for manual review
}

// Validate data structure
const validData = data.filter(item =>
  item.title && item.price // Ensure required fields exist
);

Monitoring Indicators

Check for blocking/errors:

// Monitor console
const console = await browser_console_messages({
  pattern: 'error|rate|limit|captcha',
  onlyErrors: true
});

if (console.length > 0) {
  console.log('Warnings detected:', console);
}

// Monitor network
const requests = await browser_network_requests();
const errors = requests.filter(r => r.status >= 400);

Processing Results

View Statistics

python scripts/process_results.py scraped.json --stats

Output:

📊 Statistics:
  Total records: 150
  Fields (5): title, price, rating, url, image
  Sample record: {...}

Convert Formats

# To CSV
python scripts/process_results.py scraped.json -o products.csv

# To JSON (compact)
python scripts/process_results.py scraped.json -o products.json --compact

# To Markdown table
python scripts/process_results.py scraped.json -o products.md

Combine Statistics with Conversion

python scripts/process_results.py scraped.json -o products.csv --stats

Supporting Files

Scripts

• scripts/validate_urls.py - Validate URL lists, check robots.txt compliance, group by domain
• scripts/process_results.py - Convert scraped JSON to CSV/JSON/Markdown, view statistics

References

• references/ethical-scraping.md - Comprehensive guide to rate limiting, robots.txt, error handling, and monitoring
• references/extraction-patterns.md - JavaScript patterns for data extraction, selectors, pagination, tables

Expected Outcomes

Successful Scraping

✅ Validated 50 URLs
✅ Scraped 50 pages in 5 minutes (6 req/min)
✅ Extracted 1,250 products
✅ Zero rate limit errors
✅ Exported to products.csv (1,250 rows)

With Error Handling

⚠️  Validated 50 URLs (2 disallowed by robots.txt)
✅ Scraped 48 pages
⚠️  3 pages returned no data (logged for review)
✅ Extracted 1,100 products
⚠️  1 rate limit warning (backed off successfully)
✅ Exported to products.csv (1,100 rows)

Rate Limit Detection

❌ Rate limited after 20 pages (429 responses)
✅ Backed off exponentially (5s → 10s → 20s)
✅ Resumed scraping successfully
✅ Extracted 450 products from 25 pages

Expected Benefits

Success Metrics

• Success rate > 95% (pages successfully scraped)
• Rate limit errors < 5% of requests
• Valid data rate > 90% (complete records)
• Scraping speed 6-12 requests/minute (polite crawling)

Requirements

Tools

• Playwright MCP browser tools
• Python 3.8+ (for scripts)
• Standard library only (no external dependencies for scripts)

Knowledge

• Basic CSS selectors
• JavaScript for data extraction
• Understanding of HTTP status codes
• Awareness of web scraping ethics

Red Flags to Avoid

• ❌ Scraping without checking robots.txt
• ❌ No delays between requests (hammering servers)
• ❌ Ignoring 429/503 response codes
• ❌ Scraping personal/private information
• ❌ Not monitoring console for blocking messages
• ❌ Scraping sites that explicitly prohibit it (check ToS)
• ❌ Using scraped data in violation of copyright
• ❌ Not handling pagination correctly (missing data)
• ❌ Hardcoding selectors without fallbacks
• ❌ Not validating extracted data structure

Notes

• Default to polite crawling : 1-3 second delays minimum, adjust based on site response
• Always check robots.txt first : Use validate_urls.py before scraping
• Monitor console and network : Watch for rate limit warnings and adjust delays
• Start small : Test with 5-10 URLs before scaling to hundreds
• Save progress : Write results incrementally in case of interruption
• Respect ToS : Some sites prohibit scraping in their terms of service
• Use descriptive user agents : Identify your bot clearly
• Handle errors gracefully : Log failures for manual review, don't crash

Weekly Installs

52

Repository

dawiddutoit/cus…m-claude

First Seen

Jan 23, 2026

Security Audits

Gen Agent Trust HubPassSocketPassSnykWarn

Installed on

gemini-cli51

codex51

opencode51

github-copilot50

cursor50

amp49