Agent skill · software engineering · jeremylongshore
clay-load-scale
Scale Clay enrichment pipelines for high-volume processing (10K-100K+
Why this skill is useful
Adds executable scripts for capacity planning and batch processing architecture specific to Clay, which are not commonly found in public documentation.
What it needs
Requires bullmq, redis installed locally. Requires clay account access. About 5k tokens when loaded. Last updated 2026-08-07. 2,604 stars on the source repository.
What this skill does
Clay Load & Scale Overview Strategies for processing 10K-100K+ leads through Clay monthly. Clay is a hosted platform -- you can't add servers. Scaling focuses on: table partitioning, webhook management, batch submission pacing, credit budgeting at scale, and multi-table architectures. Prerequisites Clay Growth or Enterprise plan Understanding of Clay's credit model (Data Credits + Actions) Queue infrastructure for batch processing (Redis, SQS, or BullMQ) Monitoring for credit consumption Instructions Step 1: Capacity Planning Step 2: Implement Batch Queue Architecture Step 3: Multi-Table Strategy For large volumes, split data across multiple Clay tables: Step 4: Webhook Rotation for High Volume Step 5: Auto-Delete for Stream-Through Processing For high-volume use cases where Clay enriches and pushes data onward, enable auto-delete to keep tables lean: In Clay UI: Table Settings > Auto-delete When enabled, Clay enriches incoming webhook data, sends results via HTTP API column to your destination, then deletes the rows. This keeps Clay functioning as a streaming enrichment service rather than a database. Error Handling Issue Cause Solution ------- ------- ---------- Processing stuck at 400/hr Explorer plan throttle Upgrade to Growth (no throttle) Webhook exhausted (50K) High volume Rotate to new webhook, implement rotator Queue backing up Webhook rate limiting Reduce concurrency, increase delay Table too large to manage 10K+ rows Split into multiple focused tables Credit overrun Uncontrolled batch size Add budget check before queueing Output Produce a scale-run record with source, table/webhook identifiers, batch size, queue and credit guardrails, observed throughput, rejection/dead-letter count, data-retention decision, and rollback owner. Do not rotate webhook URLs or enable auto-delete as an unreviewed workaround for capacity or data-governance problems. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills jeremylongshore/clay-load-scale