Agent skill · software engineering · wshobson
python-background-jobs
Python background job patterns including task queues, workers, and event-driven architecture. Use when implementing async task processing, job queues, long-running operations, or decoupling work from request/response cycles.
Why this skill is useful
Provides specific patterns and examples for implementing asynchronous task processing with Celery, which the AI may not generate reliably on its own.
What it needs
Requires celery, redis installed locally. About 3k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Python Background Jobs & Task Queues Decouple long-running or unreliable work from request/response cycles. Return immediately to the user while background workers handle the heavy lifting asynchronously. When to Use This Skill Processing tasks that take longer than a few seconds Sending emails, notifications, or webhooks Generating reports or exporting data Processing uploads or media transformations Integrating with unreliable external services Building event-driven architectures Core Concepts 1. Task Queue Pattern API accepts request, enqueues a job, returns immediately with a job ID. Workers process jobs asynchronously. 2. Idempotency Tasks may be retried on failure. Design for safe re-execution. 3. Job State Machine Jobs transition through states: pending → running → succeeded/failed. 4. At-Least-Once Delivery Most queues guarantee at-least-once delivery. Your code must handle duplicates. Quick Start This skill uses Celery for examples, a widely adopted task queue. Alternatives like RQ, Dramatiq, and cloud-native solutions (AWS SQS, GCP Tasks) are equally valid choices. Fundamental Patterns Pattern 1: Return Job ID Immediately For operations exceeding a few seconds, return a job ID and process asynchronously. Pattern 2: Celery Task Configuration Configure Celery tasks with proper retry and timeout settings. Pattern 3: Make Tasks Idempotent Workers may retry on crash or timeout. Design for safe re-execution. Idempotency Strategies: 1. Check-before-write: Verify state before action 2. Idempotency keys: Use unique tokens with external services 3. Upsert patterns: INSERT ... ON CONFLICT UPDATE 4. Deduplication window: Track processed IDs for N hours Pattern 4: Job State Management Persist job state transitions for visibility and debugging. Detailed worked examples and patterns Detailed sections (starting with ## Advanced Patterns) live in references/details.md. Read that file when the navigation summary above is insufficient. Best Practices Summary 1. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/python-background-jobs