Agent skill · software engineering · aj-geddes
batch-processing-jobs
Implement robust batch processing systems with job queues, schedulers, background tasks, and distributed workers. Use when processing large datasets, scheduled tasks, async operations, or resource-intensive computations.
Why this skill is useful
Provides specific implementation patterns and best practices for batch processing systems that the AI wouldn't reliably generate on its own.
What it needs
Requires bull, uuid installed locally. About 1k tokens when loaded. Last updated 2026-03-04. 307 stars on the source repository.
What this skill does
Batch Processing Jobs Table of Contents Overview When to Use Quick Start Reference Guides Best Practices Overview Implement scalable batch processing systems for handling large-scale data processing, scheduled tasks, and async operations efficiently. When to Use Processing large datasets Scheduled report generation Email/notification campaigns Data imports and exports Image/video processing ETL pipelines Cleanup and maintenance tasks Long-running computations Bulk data updates Quick Start Minimal working example: Reference Guides Detailed implementations in the references/ directory: Guide Contents --- --- Bull Queue (Node.js) Bull Queue (Node.js) Celery-Style Worker (Python) Celery-Style Worker (Python) Cron Job Scheduler Cron Job Scheduler Best Practices ✅ DO Implement idempotency for all jobs Use job queues for distributed processing Monitor job success/failure rates Implement retry logic with exponential backoff Set appropriate timeouts Log job execution details Use dead letter queues for failed jobs Implement job priority levels Batch similar operations together Use connection pooling Implement graceful shutdown Monitor queue depth and processing time ❌ DON'T Process jobs synchronously in request handlers Ignore failed jobs Set unlimited retries Skip monitoring and alerting Process jobs without timeouts Store large payloads in queue Forget to clean up completed jobs
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aj-geddes/batch-processing-jobs