批处理作业系统实现指南：大规模数据处理、计划任务与异步操作的最佳实践

batch-processing-jobs by aj-geddes/useful-ai-prompts

151 周安装量

141 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/aj-geddes/useful-ai-prompts --skill batch-processing-jobs

自动化系统架构数据处理

🇨🇳中文介绍

批处理作业

概述

实现可扩展的批处理系统，用于高效处理大规模数据处理、计划任务和异步操作。

适用场景

处理大型数据集
计划报告生成
电子邮件/通知活动
数据导入和导出
图像/视频处理
ETL 管道
清理和维护任务
长时间运行的计算
批量数据更新

快速开始

最小工作示例：

import Queue from "bull";
import { v4 as uuidv4 } from "uuid";

interface JobData {
  id: string;
  type: string;
  payload: any;
  userId?: string;
  metadata?: Record<string, any>;
}

interface JobResult {
  success: boolean;
  data?: any;
  error?: string;
  processedAt: number;
  duration: number;
}

class BatchProcessor {
  private queue: Queue.Queue<JobData>;
  private resultQueue: Queue.Queue<JobResult>;

  constructor(redisUrl: string) {
    // 主处理队列
// ... (完整实现请参阅参考指南)

参考指南

references/ 目录下的详细实现：

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

🇺🇸English

Batch Processing Jobs

Overview
When to Use
Quick Start
Reference Guides
Best Practices

Overview

Implement scalable batch processing systems for handling large-scale data processing, scheduled tasks, and async operations efficiently.

When to Use

Processing large datasets
Scheduled report generation
Email/notification campaigns
Data imports and exports
Image/video processing
ETL pipelines
Cleanup and maintenance tasks
Long-running computations
Bulk data updates

Quick Start

Minimal working example:

import Queue from "bull";
import { v4 as uuidv4 } from "uuid";

interface JobData {
  id: string;
  type: string;
  payload: any;
  userId?: string;
  metadata?: Record<string, any>;
}

interface JobResult {
  success: boolean;
  data?: any;
  error?: string;
  processedAt: number;
  duration: number;
}

class BatchProcessor {
  private queue: Queue.Queue<JobData>;
  private resultQueue: Queue.Queue<JobResult>;

  constructor(redisUrl: string) {
    // Main processing queue
// ... (see reference guides for full implementation)

Reference Guides

Detailed implementations in the references/ directory:

Guide	Contents
Bull Queue (Node.js)	Bull Queue (Node.js)
Celery-Style Worker (Python)	Celery-Style Worker (Python)
Cron Job Scheduler	Cron Job Scheduler

Best Practices

✅ DO

Implement idempotency for all jobs
Use job queues for distributed processing
Monitor job success/failure rates
Implement retry logic with exponential backoff
Set appropriate timeouts
Log job execution details
Use dead letter queues for failed jobs
Implement job priority levels
Batch similar operations together
Use connection pooling
Implement graceful shutdown
Monitor queue depth and processing time

❌ DON'T

Process jobs synchronously in request handlers
Ignore failed jobs
Set unlimited retries
Skip monitoring and alerting
Process jobs without timeouts
Store large payloads in queue
Forget to clean up completed jobs

Weekly Installs

132

Repository

aj-geddes/usefu…-prompts

GitHub Stars

126

First Seen

Jan 21, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykPass

Installed on

opencode112

gemini-cli109

codex106

cursor98

claude-code95

github-copilot89