Agent skill · product strategy · wshobson
recsys-pipeline-architect
Design composable recommendation, ranking, and feed pipelines using the six-stage Source→Hydrator→Filter→Scorer→Selector→SideEffect framework popularized by xAI's open-sourced X For You algorithm. Use when building any system that picks "the top K items for a (user, context)" — content feeds, search ranking, RAG rerankers, task prioritizers, notification triage, ad selection.
Why this skill is useful
Provides a structured framework for building recommendation systems that the AI wouldn't reliably generate on its own.
What it needs
About 5k tokens when loaded. Last updated 2026-08-05. 38,570 stars on the source repository.
What this skill does
Recsys Pipeline Architect A spec-and-scaffold skill for building composable recommendation, ranking, and feed pipelines. Encodes the six-stage pattern popularized by xAI's open-sourced For You algorithm (Apache 2.0) and applies it to any "top K for (user, context)" problem. Overview Most "recommendation systems" in production aren't exotic ML — they're pipelines: fetch candidates from one or more sources, enrich them with metadata, drop the ineligible, score the rest, sort and pick the top K, then fire async side effects. The pattern is universal. The scoring function and the items change; the pipeline shape doesn't. This skill is an independent reimplementation of the pattern (MIT) — no code copied from the original. When to Use This Skill Building any system that returns "the top K items for a user/context" Designing or refactoring a personalized feed (content, search results, notifications) Wrapping an LLM/ML scorer in proper pipeline plumbing (sources, hydration, filters, side effects) Adding multi-action prediction with tunable weights (instead of a single relevance score) Building a RAG retrieval reranker (cheap retrieval → expensive rerank) Designing a task prioritizer or alert triage system The Six-Stage Framework # Stage Job Parallel? --- --- --- --- 1 Source Fetch candidates from one or more origins Yes — multiple sources run in parallel 2 Hydrator Enrich candidates with metadata needed for filtering and scoring Yes — independent hydrators run in parallel 3 Filter Drop ineligible candidates (blocked, expired, duplicate, ineligible) Sequential — each filter sees fewer items 4 Scorer Assign each surviving candidate one or more scores Sequential — later scorers see earlier scores 5 Selector Sort by final score, return top K Single op 6 SideEffect Cache, log, emit events, update served-history Async — must never block the response Why this exact order Sources before hydration: know what candidates exist before paying to enrich Hydration before filtering: many …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills wshobson/recsys-pipeline-architect