--- name: research-paper-writing title: Research Paper Writing Pipeline description: "Write ML papers for NeurIPS/ICML/ICLR: design→submit." version: 1.1.0 author: Orchestra Research license: MIT dependencies: [semanticscholar, arxiv, habanero, requests, scipy, numpy, matplotlib, SciencePlots] platforms: [linux, macos] metadata: hermes: tags: [Research, Paper Writing, Experiments, ML, AI, NeurIPS, ICML, ICLR, ACL, AAAI, COLM, LaTeX, Citations, Statistical Analysis] category: research related_skills: [arxiv, subagent-driven-development] requires_toolsets: [terminal, files] --- # Research Paper Writing Pipeline End-to-end pipeline for producing publication-ready ML/AI research papers targeting **NeurIPS, ICML, ICLR, ACL, AAAI, and COLM**. This skill covers the full research lifecycle: experiment design, execution, monitoring, analysis, paper writing, review, revision, and submission. This is **not a linear pipeline** — it is an iterative loop. Results trigger new experiments. Reviews trigger new analysis. The agent must handle these feedback loops. ``` ┌─────────────────────────────────────────────────────────────┐ │ RESEARCH PAPER PIPELINE │ │ │ │ Phase 0: Project Setup ──► Phase 1: Literature Review │ │ │ │ │ │ ▼ ▼ │ │ Phase 2: Experiment Phase 5: Paper Drafting ◄──┐ │ │ Design │ │ │ │ │ ▼ │ │ │ ▼ Phase 6: Self-Review │ │ │ Phase 3: Execution & & Revision ──────────┘ │ │ Monitoring │ │ │ │ ▼ │ │ ▼ Phase 7: Submission │ │ Phase 4: Analysis ─────► (feeds back to Phase 2 or 5) │ │ │ └─────────────────────────────────────────────────────────────┘ ``` --- ## When To Use This Skill Use this skill when: - **Starting a new research paper** from an existing codebase or idea - **Designing and running experiments** to support paper claims - **Writing or revising** any section of a research paper - **Preparing for submission** to a specific conference or workshop - **Responding to reviews** with additional experiments or revisions - **Converting** a paper between conference formats - **Writing non-empirical papers** — theory, survey, benchmark, or position papers (see [Paper Types Beyond Empirical ML](#paper-types-beyond-empirical-ml)) - **Designing human evaluations** for NLP, HCI, or alignment research - **Preparing post-acceptance deliverables** — posters, talks, code releases ## Core Philosophy 1. **Be proactive.** Deliver complete drafts, not questions. Scientists are busy — produce something concrete they can react to, then iterate. 2. **Never hallucinate citations.** AI-generated citations have ~40% error rate. Always fetch programmatically. Mark unverifiable citations as `[CITATION NEEDED]`. 3. **Paper is a story, not a collection of experiments.** Every paper needs one clear contribution stated in a single sentence. If you can't do that, the paper isn't ready. 4. **Experiments serve claims.** Every experiment must explicitly state which claim it supports. Never run experiments that don't connect to the paper's narrative. 5. **Commit early, commit often.** Every completed experiment batch, every paper draft update — commit with descriptive messages. Git log is the experiment history. ### Proactivity and Collaboration **Default: Be proactive. Draft first, ask with the draft.** | Confidence Level | Action | |-----------------|--------| | **High** (clear repo, obvious contribution) | Write full draft, deliver, iterate on feedback | | **Medium** (some ambiguity) | Write draft with flagged uncertainties, continue | | **Low** (major unknowns) | Ask 1-2 targeted questions via `clarify`, then draft | | Section | Draft Autonomously? | Flag With Draft | |---------|-------------------|-----------------| | Abstract | Yes | "Framed contribution as X — adjust if needed" | | Introduction | Yes | "Emphasized problem Y — correct if wrong" | | Methods | Yes | "Included details A, B, C — add missing pieces" | | Experiments | Yes | "Highlighted results 1, 2, 3 — reorder if needed" | | Related Work | Yes | "Cited papers X, Y, Z — add any I missed" | **Block for input only when**: target venue unclear, multiple contradictory framings, results seem incomplete, explicit request to review first. --- ## Phase 0: Project Setup **Goal**: Establish the workspace, understand existing work, identify the contribution. ### Step 0.1: Explore the Repository ```bash # Understand project structure ls -la find . -name "*.py" | head -30 find . -name "*.md" -o -name "*.txt" | xargs grep -l -i "result\|conclusion\|finding" ``` Look for: - `README.md` — project overview and claims - `results/`, `outputs/`, `experiments/` — existing findings - `configs/` — experimental settings - `.bib` files — existing citations - Draft documents or notes ### Step 0.2: Organize the Workspace Establish a consistent workspace structure: ``` workspace/ paper/ # LaTeX source, figures, compiled PDFs experiments/ # Experiment runner scripts code/ # Core method implementation results/ # Raw experiment results (auto-generated) tasks/ # Task/benchmark definitions human_eval/ # Human evaluation materials (if needed) ``` ### Step 0.3: Set Up Version Control ```bash git init # if not already git remote add origin git checkout -b paper-draft # or main ``` **Git discipline**: Every completed experiment batch gets committed with a descriptive message. Example: ``` Add Monte Carlo constrained results (5 runs, Sonnet 4.6, policy memo task) Add Haiku baseline comparison: autoreason vs refinement baselines at cheap model tier ``` ### Step 0.4: Identify the Contribution Before writing anything, articulate: - **The What**: What is the single thing this paper contributes? - **The Why**: What evidence supports it? - **The So What**: Why should readers care? > Propose to the scientist: "Based on my understanding, the main contribution is: [one sentence]. The key results show [Y]. Is this the framing you want?" ### Step 0.5: Create a TODO List Use the `todo` tool to create a structured project plan: ``` Research Paper TODO: - [ ] Define one-sentence contribution - [ ] Literature review (related work + baselines) - [ ] Design core experiments - [ ] Run experiments - [ ] Analyze results - [ ] Write first draft - [ ] Self-review (simulate reviewers) - [ ] Revise based on review - [ ] Submission prep ``` Update this throughout the project. It serves as the persistent state across sessions. ### Step 0.6: Estimate Compute Budget Before running experiments, estimate total cost and time: ``` Compute Budget Checklist: - [ ] API costs: (model price per token) × (estimated tokens per run) × (number of runs) - [ ] GPU hours: (time per experiment) × (number of experiments) × (number of seeds) - [ ] Human evaluation costs: (annotators) × (hours) × (hourly rate) - [ ] Total budget ceiling and contingency (add 30-50% for reruns) ``` Track actual spend as experiments run: ```python # Simple cost tracker pattern import json, os from datetime import datetime COST_LOG = "results/cost_log.jsonl" def log_cost(experiment: str, model: str, input_tokens: int, output_tokens: int, cost_usd: float): entry = { "timestamp": datetime.now().isoformat(), "experiment": experiment, "model": model, "input_tokens": input_tokens, "output_tokens": output_tokens, "cost_usd": cost_usd, } with open(COST_LOG, "a") as f: f.write(json.dumps(entry) + "\n") ``` **When budget is tight**: Run pilot experiments (1-2 seeds, subset of tasks) before committing to full sweeps. Use cheaper models for debugging pipelines, then switch to target models for final runs. ### Step 0.7: Multi-Author Coordination Most papers have 3-10 authors. Establish workflows early: | Workflow | Tool | When to Use | |----------|------|-------------| | **Overleaf** | Browser-based | Multiple authors editing simultaneously, no git experience | | **Git + LaTeX** | `git` with `.gitignore` for aux files | Technical teams, need branch-based review | | **Overleaf + Git sync** | Overleaf premium | Best of both — live collab with version history | **Section ownership**: Assign each section to one primary author. Others comment but don't edit directly. Prevents merge conflicts and style inconsistency. ``` Author Coordination Checklist: - [ ] Agree on section ownership (who writes what) - [ ] Set up shared workspace (Overleaf or git repo) - [ ] Establish notation conventions (before anyone writes) - [ ] Schedule internal review rounds (not just at the end) - [ ] Designate one person for final formatting pass - [ ] Agree on figure style (colors, fonts, sizes) before creating figures ``` **LaTeX conventions to agree on early**: - `\method{}` macro for consistent method naming - Citation style: `\citet{}` vs `\citep{}` usage - Math notation: lowercase bold for vectors, uppercase bold for matrices, etc. - British vs American spelling --- ## Phase 1: Literature Review **Goal**: Find related work, identify baselines, gather citations. ### Step 1.1: Identify Seed Papers Start from papers already referenced in the codebase: ```bash # Via terminal: grep -r "arxiv\|doi\|cite" --include="*.md" --include="*.bib" --include="*.py" find . -name "*.bib" ``` ### Step 1.2: Search for Related Work **Load the `arxiv` skill** for structured paper discovery: `skill_view("arxiv")`. It provides arXiv REST API search, Semantic Scholar citation graphs, author profiles, and BibTeX generation. Use `web_search` for broad discovery, `web_extract` for fetching specific papers: ``` # Via web_search: web_search("[main technique] + [application domain] site:arxiv.org") web_search("[baseline method] comparison ICML NeurIPS 2024") # Via web_extract (for specific papers): web_extract("https://arxiv.org/abs/2303.17651") ``` Additional search queries to try: ``` Search queries: - "[main technique] + [application domain]" - "[baseline method] comparison" - "[problem name] state-of-the-art" - Author names from existing citations ``` **Recommended**: Install **Exa MCP** for real-time academic search: ```bash claude mcp add exa -- npx -y mcp-remote "https://mcp.exa.ai/mcp" ``` ### Step 1.2b: Deepen the Search (Breadth-First, Then Depth) A flat search (one round of queries) typically misses important related work. Use an iterative **breadth-then-depth** pattern inspired by deep research pipelines: ``` Iterative Literature Search: Round 1 (Breadth): 4-6 parallel queries covering different angles - "[method] + [domain]" - "[problem name] state-of-the-art 2024 2025" - "[baseline method] comparison" - "[alternative approach] vs [your approach]" → Collect papers, extract key concepts and terminology Round 2 (Depth): Generate follow-up queries from Round 1 learnings - New terminology discovered in Round 1 papers - Papers cited by the most relevant Round 1 results - Contradictory findings that need investigation → Collect papers, identify remaining gaps Round 3 (Targeted): Fill specific gaps - Missing baselines identified in Rounds 1-2 - Concurrent work (last 6 months, same problem) - Key negative results or failed approaches → Stop when new queries return mostly papers you've already seen ``` **When to stop**: If a round returns >80% papers already in your collection, the search is saturated. Typically 2-3 rounds suffice. For survey papers, expect 4-5 rounds. **For agent-based workflows**: Delegate each round's queries in parallel via `delegate_task`. Collect results, deduplicate, then generate the next round's queries from the combined learnings. ### Step 1.3: Verify Every Citation **NEVER generate BibTeX from memory. ALWAYS fetch programmatically.** For each citation, follow the mandatory 5-step process: ``` Citation Verification (MANDATORY per citation): 1. SEARCH → Query Semantic Scholar or Exa MCP with specific keywords 2. VERIFY → Confirm paper exists in 2+ sources (Semantic Scholar + arXiv/CrossRef) 3. RETRIEVE → Get BibTeX via DOI content negotiation (programmatically, not from memory) 4. VALIDATE → Confirm the claim you're citing actually appears in the paper 5. ADD → Add verified BibTeX to bibliography If ANY step fails → mark as [CITATION NEEDED], inform scientist ``` ```python # Fetch BibTeX via DOI import requests def doi_to_bibtex(doi: str) -> str: response = requests.get( f"https://doi.org/{doi}", headers={"Accept": "application/x-bibtex"} ) response.raise_for_status() return response.text ``` If you cannot verify a citation: ```latex \cite{PLACEHOLDER_author2024_verify_this} % TODO: Verify this citation exists ``` **Always tell the scientist**: "I've marked [X] citations as placeholders that need verification." See [references/citation-workflow.md](references/citation-workflow.md) for complete API documentation and the full `CitationManager` class. ### Step 1.4: Organize Related Work Group papers by methodology, not paper-by-paper: **Good**: "One line of work uses X's assumption [refs] whereas we use Y's assumption because..." **Bad**: "Smith et al. introduced X. Jones et al. introduced Y. We combine both." --- ## Phase 2: Experiment Design **Goal**: Design experiments that directly support paper claims. Every experiment must answer a specific question. ### Step 2.1: Map Claims to Experiments Create an explicit mapping: | Claim | Experiment | Expected Evidence | |-------|-----------|-------------------| | "Our method outperforms baselines" | Main comparison (Table 1) | Win rate, statistical significance | | "Effect is larger for weaker models" | Model scaling study | Monotonic improvement curve | | "Convergence requires scope constraints" | Constrained vs unconstrained | Convergence rate comparison | **Rule**: If an experiment doesn't map to a claim, don't run it. ### Step 2.2: Design Baselines Strong baselines are what separates accepted papers from rejected ones. Reviewers will ask: "Did they compare against X?" Standard baseline categories: - **Naive baseline**: Simplest possible approach - **Strong baseline**: Best known existing method - **Ablation baselines**: Your method minus one component - **Compute-matched baselines**: Same compute budget, different allocation ### Step 2.3: Define Evaluation Protocol Before running anything, specify: - **Metrics**: What you're measuring, direction symbols (higher/lower better) - **Aggregation**: How results are combined across runs/tasks - **Statistical tests**: What tests will establish significance - **Sample sizes**: How many runs/problems/tasks ### Step 2.4: Write Experiment Scripts Follow these patterns from successful research pipelines: **Incremental saving** — save results after each step for crash recovery: ```python # Save after each problem/task result_path = f"results/{task}/{strategy}/result.json" if os.path.exists(result_path): continue # Skip already-completed work # ... run experiment ... with open(result_path, 'w') as f: json.dump(result, f, indent=2) ``` **Artifact preservation** — save all intermediate outputs: ``` results// / / final_output.md # Final result history.json # Full trajectory pass_01/ # Per-iteration artifacts version_a.md version_b.md critic.md ``` **Separation of concerns** — keep generation, evaluation, and visualization separate: ``` run_experiment.py # Core experiment runner run_baselines.py # Baseline comparison run_comparison_judge.py # Blind evaluation analyze_results.py # Statistical analysis make_charts.py # Visualization ``` See [references/experiment-patterns.md](references/experiment-patterns.md) for complete design patterns, cron monitoring, and error recovery. ### Step 2.5: Design Human Evaluation (If Applicable) Many NLP, HCI, and alignment papers require human evaluation as primary or complementary evidence. Design this before running automated experiments — human eval often has longer lead times (IRB approval, annotator recruitment). **When human evaluation is needed:** - Automated metrics don't capture what you care about (fluency, helpfulness, safety) - Your contribution is about human-facing qualities (readability, preference, trust) - Reviewers at NLP venues (ACL, EMNLP) expect it for generation tasks **Key design decisions:** | Decision | Options | Guidance | |----------|---------|----------| | **Annotator type** | Expert, crowdworker, end-user | Match to what your claims require | | **Scale** | Likert (1-5), pairwise comparison, ranking | Pairwise is more reliable than Likert for LLM outputs | | **Sample size** | Per annotator and total items | Power analysis or minimum 100 items, 3+ annotators | | **Agreement metric** | Cohen's kappa, Krippendorff's alpha, ICC | Krippendorff's alpha for >2 annotators; report raw agreement too | | **Platform** | Prolific, MTurk, internal team | Prolific for quality; MTurk for scale; internal for domain expertise | **Annotation guideline checklist:** ``` - [ ] Clear task description with examples (good AND bad) - [ ] Decision criteria for ambiguous cases - [ ] At least 2 worked examples per category - [ ] Attention checks / gold standard items (10-15% of total) - [ ] Qualification task or screening round - [ ] Estimated time per item and fair compensation (>= local minimum wage) - [ ] IRB/ethics review if required by your institution ``` **Reporting requirements** (reviewers check all of these): - Number of annotators and their qualifications - Inter-annotator agreement with specific metric and value - Compensation details (amount, estimated hourly rate) - Annotation interface description or screenshot (appendix) - Total annotation time See [references/human-evaluation.md](references/human-evaluation.md) for complete guide including statistical tests for human eval data, crowdsourcing quality control patterns, and IRB guidance. --- ## Phase 3: Experiment Execution & Monitoring **Goal**: Run experiments reliably, monitor progress, recover from failures. ### Step 3.1: Launch Experiments Use `nohup` for long-running experiments: ```bash nohup python run_experiment.py --config config.yaml > logs/experiment_01.log 2>&1 & echo $! # Record the PID ``` **Parallel execution**: Run independent experiments simultaneously, but be aware of API rate limits. 4+ concurrent experiments on the same API will slow each down. ### Step 3.2: Set Up Monitoring (Cron Pattern) For long-running experiments, set up periodic status checks. The cron prompt should follow this template: ``` Monitor Prompt Template: 1. Check if process is still running: ps aux | grep 2. Read last 30 lines of log: tail -30 3. Check for completed results: ls 4. If results exist, read and report: cat 5. If all done, commit: git add -A && git commit -m "" && git push 6. Report in structured format (tables with key metrics) 7. Answer the key analytical question for this experiment ``` **Silent mode**: If nothing has changed since the last check, respond with `[SILENT]` to suppress notification to the user. Only report when there's news. ### Step 3.3: Handle Failures Common failure modes and recovery: |