---
name: cicd-intelligent-recovery
description: Loop 3 of the Three-Loop Integrated Development System. CI/CD automation with intelligent failure recovery, root cause analysis, and comprehensive quality validation. Receives implementation from Loop 2, feeds failure patterns back to Loop 1. Achieves 100% test success through automated repair and theater validation. v2.0.0 with explicit agent SOPs.
---

# CI/CD Quality & Debugging Loop (Loop 3)

**Purpose**: Continuous integration with automated failure recovery and authentic quality validation.

**SOP Workflow**: Specification → Research → Planning → Execution → Knowledge

**Output**: 100% test success rate with authentic quality improvements and failure pattern analysis

**Integration**: This is Loop 3 of 3. Receives from `parallel-swarm-implementation` (Loop 2), feeds failure data back to `research-driven-planning` (Loop 1).

**Version**: 2.0.0
**Optimization**: Evidence-based prompting with explicit agent SOPs

---

## When to Use This Skill

Activate this skill when:
- Have complete implementation from Loop 2 (parallel-swarm-implementation)
- Need CI/CD pipeline automation with intelligent recovery
- Require root cause analysis for test failures
- Want automated repair with connascence-aware fixes
- Need validation of authentic quality (no theater)
- Generating failure patterns for Loop 1 feedback

**DO NOT** use this skill for:
- Initial development (use Loop 2 first)
- Manual debugging without CI/CD integration
- Quality checks during development (use Loop 2 theater detection)

---

## Input/Output Contracts

### Input Requirements

```yaml
input:
  loop2_delivery_package:
    location: .claude/.artifacts/loop2-delivery-package.json
    schema:
      implementation: object (complete codebase)
      tests: object (test suite)
      theater_baseline: object (theater metrics from Loop 2)
      integration_points: array[string]
    validation:
      - Must exist and be valid JSON
      - Must include theater_baseline for differential analysis

  ci_cd_failures:
    source: GitHub Actions workflow runs
    format: JSON array of failure objects
    required_fields: [file, line, column, testName, errorMessage, runId]

  github_credentials:
    required: gh CLI authenticated
    check: gh auth status
```

### Output Guarantees

```yaml
output:
  test_success_rate: 100% (guaranteed)

  quality_validation:
    theater_audit: PASSED (no false improvements)
    sandbox_validation: 100% test pass
    differential_analysis: improvement metrics

  failure_patterns:
    location: .claude/.artifacts/loop3-failure-patterns.json
    feeds_to: Loop 1 (next iteration)
    schema:
      patterns: array[failure_pattern]
      recommendations: object (planning/architecture/testing)

  delivery_package:
    location: .claude/.artifacts/loop3-delivery-package.json
    contains:
      - quality metrics (test success, failures fixed)
      - analysis data (root causes, connascence context)
      - validation results (theater, sandbox, differential)
      - feedback for Loop 1
```

---

## Prerequisites

Before starting Loop 3, ensure Loop 2 completion:

```bash
# Verify Loop 2 delivery package exists
test -f .claude/.artifacts/loop2-delivery-package.json && echo "✅ Ready" || echo "❌ Run parallel-swarm-implementation first"

# Load implementation data
npx claude-flow@alpha memory query "loop2_complete" --namespace "integration/loop2-to-loop3"

# Verify GitHub CLI authenticated
gh auth status || gh auth login
```

---

## 8-Step CI/CD Process Overview

```
Step 1: GitHub Hook Integration (Download CI/CD failure reports)
        ↓
Step 2: AI-Powered Analysis (Gemini + 7-agent synthesis with Byzantine consensus)
        ↓
Step 3: Root Cause Detection (Graph analysis + Raft consensus)
        ↓
Step 4: Intelligent Fixes (Program-of-thought: Plan → Execute → Validate → Approve)
        ↓
Step 5: Theater Detection Audit (6-agent Byzantine consensus validation)
        ↓
Step 6: Sandbox Validation (Isolated production-like testing)
        ↓
Step 7: Differential Analysis (Compare to baseline with metrics)
        ↓
Step 8: GitHub Feedback (Automated reporting and loop closure)
```

---

## Step 1: GitHub Hook Integration

**Objective**: Download and process CI/CD pipeline failure reports from GitHub Actions.

**Agent Coordination**: Single orchestrator agent manages data collection.

### Configure GitHub Hooks

```bash
# Install GitHub CLI if needed
which gh || brew install gh

# Authenticate
gh auth login

# Configure webhook listener
gh api repos/{owner}/{repo}/hooks \
  -X POST \
  -f name='web' \
  -f active=true \
  -f events='["check_run", "workflow_run"]' \
  -f config[url]='http://localhost:3000/hooks/github' \
  -f config[content_type]='application/json'
```

### Download Failure Reports

```bash
# Get recent workflow runs
gh run list --repo {owner}/{repo} --limit 10 --json conclusion,databaseId \
  | jq '.[] | select(.conclusion == "failure")' \
  > .claude/.artifacts/failed-runs.json

# Download logs for each failure
cat .claude/.artifacts/failed-runs.json | jq -r '.databaseId' | while read RUN_ID; do
  gh run view $RUN_ID --log \
    > .claude/.artifacts/failure-logs-$RUN_ID.txt
done
```

### Parse Failure Data

```bash
node <<'EOF'
const fs = require('fs');
const failures = [];

// Parse all failure logs
const logFiles = fs.readdirSync('.claude/.artifacts')
  .filter(f => f.startsWith('failure-logs-'));

logFiles.forEach(file => {
  const log = fs.readFileSync(`.claude/.artifacts/${file}`, 'utf8');

  // Extract structured failure data
  const failureMatches = log.matchAll(/FAIL (.+?):(\d+):(\d+)\n(.+?)\n(.+)/g);

  for (const match of failureMatches) {
    failures.push({
      file: match[1],
      line: parseInt(match[2]),
      column: parseInt(match[3]),
      testName: match[4],
      errorMessage: match[5],
      runId: file.match(/failure-logs-(\d+)/)[1]
    });
  }
});

fs.writeFileSync(
  '.claude/.artifacts/parsed-failures.json',
  JSON.stringify(failures, null, 2)
);

console.log(`✅ Parsed ${failures.length} failures`);
EOF
```

**Validation Checkpoint**:
- ✅ Failure data parsed and structured
- ✅ All required fields present (file, line, testName, errorMessage)

---

## Step 2: AI-Powered Analysis

**Objective**: Use Gemini large-context analysis + 7 research agents with Byzantine consensus to examine each failure deeply.

**Evidence-Based Techniques**: Self-consistency, Byzantine consensus, program-of-thought

### Phase 1: Gemini Large-Context Analysis

**Leverage Gemini's 2M token window for full codebase analysis**

```bash
# Analyze failures with full codebase context
/gemini:impact "Analyze CI/CD test failures:

FAILURE DATA:
$(cat .claude/.artifacts/parsed-failures.json)

CODEBASE CONTEXT:
Full repository (all files)

LOOP 2 IMPLEMENTATION:
$(cat .claude/.artifacts/loop2-delivery-package.json)

ANALYSIS OBJECTIVES:
1. Identify cross-file dependencies related to failures
2. Detect failure cascade patterns (root → secondary → tertiary)
3. Analyze what changed between working and failing states
4. Assess system-level architectural impact
5. Identify connascence patterns in failing code

OUTPUT FORMAT:
{
  dependency_graph: { nodes: [files], edges: [dependencies] },
  cascade_map: { root_failures: [], cascaded_failures: [] },
  change_analysis: { changed_files: [], change_impact: [] },
  architectural_impact: { affected_systems: [], coupling_issues: [] }
}"

# Store Gemini analysis
cat .claude/.artifacts/gemini-response.json \
  > .claude/.artifacts/gemini-analysis.json
```

### Phase 2: Parallel Multi-Agent Deep Dive (Self-Consistency)

**7 parallel agents for cross-validation and consensus**

```javascript
// PARALLEL ANALYSIS AGENTS - Evidence-Based Self-Consistency
[Single Message - Spawn All 7 Analysis Agents]:

  // Failure Pattern Research (Dual agents for cross-validation)
  Task("Failure Pattern Researcher 1",
    `Research similar failures in external sources:
    - GitHub issues for libraries we use
    - Stack Overflow questions with similar error messages
    - Documentation of known issues

    Failures to research: $(cat .claude/.artifacts/parsed-failures.json | jq -r '.[].errorMessage')

    For each failure:
    1. Find similar reported issues
    2. Document known solutions with evidence (links, code examples)
    3. Note confidence level (high/medium/low)

    Store findings: .claude/.artifacts/failure-patterns-researcher1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-1"`,
    "researcher")

  Task("Failure Pattern Researcher 2",
    `Cross-validate findings from Researcher 1:
    - Load: .claude/.artifacts/failure-patterns-researcher1.json
    - Verify each claimed solution independently
    - Check for conflicting solutions
    - Identify most reliable approaches

    For conflicts:
    1. Research both approaches
    2. Determine which is more current/reliable
    3. Flag disagreements for consensus

    Store findings: .claude/.artifacts/failure-patterns-researcher2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "research-patterns-2"`,
    "researcher")

  // Error Analysis Specialist
  Task("Error Message Analyzer",
    `Deep dive into error messages and stack traces:

    Failures: $(cat .claude/.artifacts/parsed-failures.json)

    For each error message:
    1. Parse error semantics (syntax error vs runtime vs logic)
    2. Extract root cause from stack trace (not just symptoms)
    3. Identify error propagation patterns
    4. Distinguish between:
       - Direct causes (code that threw error)
       - Indirect causes (code that set up failure conditions)

    Apply program-of-thought reasoning:
    "Error X occurred because Y, which was caused by Z"

    Store analysis: .claude/.artifacts/error-analysis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "error-analysis"`,
    "analyst")

  // Code Context Investigator
  Task("Code Context Investigator",
    `Analyze surrounding code context for failures:

    Load: .claude/.artifacts/parsed-failures.json
    Load: .claude/.artifacts/gemini-analysis.json (for dependency context)

    For each failure:
    1. Read file at failure line ±50 lines
    2. Identify why failure occurs in THIS specific codebase
    3. Find coupling issues (tight coupling → cascading failures)
    4. Analyze code smells that contributed to failure

    Context analysis:
    - Variable/function naming clarity
    - Error handling presence/absence
    - Input validation
    - Edge case handling

    Store findings: .claude/.artifacts/code-context.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "code-context"`,
    "code-analyzer")

  // Test Validity Auditors (Dual agents for critical validation)
  Task("Test Validity Auditor 1",
    `Determine if tests are correctly written:

    Load failures: .claude/.artifacts/parsed-failures.json

    For each failing test:
    1. Is test logic correct? (proper assertions, valid test data)
    2. Is failure indicating real bug or test issue?
    3. Check test quality:
       - Proper setup/teardown
       - Isolated (not depending on other tests)
       - Deterministic (not flaky)

    Categorize:
    - Real bugs: Code is wrong, test is correct
    - Test issues: Code is correct, test is wrong
    - Both wrong: Code and test both have issues

    Store analysis: .claude/.artifacts/test-validity-1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-1"`,
    "tester")

  Task("Test Validity Auditor 2",
    `Cross-validate test analysis from Auditor 1:

    Load: .claude/.artifacts/test-validity-1.json
    Load: .claude/.artifacts/loop2-delivery-package.json (theater baseline)

    Additional checks:
    1. Compare to Loop 2 theater baseline
    2. Check for test theater patterns:
       - Meaningless assertions (expect(1).toBe(1))
       - Over-mocking (mocking the thing being tested)
       - False positives (tests that don't actually test)

    For disagreements with Auditor 1:
    1. Re-examine test thoroughly
    2. Document reasoning for different conclusion
    3. Flag for consensus resolution

    Store analysis: .claude/.artifacts/test-validity-2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "test-validity-2"`,
    "tester")

  // Dependency Specialist
  Task("Dependency Conflict Detector",
    `Check for dependency-related failures:

    Load failures: .claude/.artifacts/parsed-failures.json

    Analysis steps:
    1. Check package.json/requirements.txt for version conflicts
    2. Identify breaking changes in dependencies:
       - Compare current versions to last working versions
       - Review CHANGELOG files for breaking changes
       - Check deprecation warnings

    3. Analyze transitive dependencies:
       - npm ls or pip list --tree
       - Find version conflicts in dep tree

    4. Check for missing dependencies:
       - ImportError / Cannot find module
       - Missing peer dependencies

    Store findings: .claude/.artifacts/dependency-analysis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "dependency-analysis"`,
    "analyst")

// Wait for all 7 agents to complete
npx claude-flow@alpha task wait --all --namespace "cicd/analysis"
```

### Phase 3: Synthesis with Byzantine Consensus

**Byzantine fault-tolerant synthesis requires 5/7 agent agreement**

```javascript
[Single Message - Synthesis Coordinator]:
  Task("Analysis Synthesis Coordinator",
    `Synthesize findings from Gemini + 7 agents using Byzantine consensus.

    INPUTS:
    - Gemini Analysis: .claude/.artifacts/gemini-analysis.json
    - Researcher 1: .claude/.artifacts/failure-patterns-researcher1.json
    - Researcher 2: .claude/.artifacts/failure-patterns-researcher2.json
    - Error Analyzer: .claude/.artifacts/error-analysis.json
    - Code Context: .claude/.artifacts/code-context.json
    - Test Auditor 1: .claude/.artifacts/test-validity-1.json
    - Test Auditor 2: .claude/.artifacts/test-validity-2.json
    - Dependency Detector: .claude/.artifacts/dependency-analysis.json

    SYNTHESIS PROCESS:

    1. Cross-Reference Analysis:
       For each failure, collect all agent findings
       Build confidence matrix: which agents agree on root cause

    2. Byzantine Consensus:
       For each root cause claim:
       - Count agent agreement (need 5/7 for consensus)
       - Weight by agent confidence scores
       - Flag conflicts (< 5/7 agreement) for manual review

    3. Consolidate Root Causes:
       - Primary causes: 7/7 agreement (highest confidence)
       - Secondary causes: 5-6/7 agreement (medium confidence)
       - Disputed causes: < 5/7 agreement (flag for review)

    4. Generate Synthesis Report:
       {
         rootCauses: [
           {
             failure: failure_object,
             cause: "root cause description",
             evidence: ["agent1 finding", "agent2 finding"],
             consensus: 7/7 or 6/7 or 5/7,
             confidence: "high" | "medium" | "low"
           }
         ],
         cascadingFailures: [
           { root: failure_id, cascaded: [failure_ids] }
         ],
         quickWins: [ /* easy fixes */ ],
         complexIssues: [ /* require architecture changes */ ]
       }

    VALIDATION:
    - All failures must be categorized
    - Root causes must have >= 5/7 consensus or be flagged
    - Cascading relationships must be validated by Gemini graph

    Store: .claude/.artifacts/analysis-synthesis.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "synthesis-consensus"`,
    "byzantine-coordinator")
```

**Validation Checkpoint**:
- ✅ Gemini analysis complete (dependency graph, cascade map)
- ✅ All 7 agents completed analysis
- ✅ Byzantine consensus achieved (5/7 agreement on root causes)
- ✅ Synthesis report generated with confidence scores

---

## Step 3: Root Cause Detection

**Objective**: Reverse engineer to find cascade issues and true root causes using graph analysis and Raft consensus.

**Evidence-Based Techniques**: Graph algorithms, connascence analysis, Raft consensus

### Phase 1: Parallel Cascade Graph Analysis

**Multiple graph analysts for validation**

```javascript
[Single Message - Parallel Graph Analysis]:

  Task("Failure Graph Analyst 1",
    `Build failure dependency graph using graph algorithms:

    Load: .claude/.artifacts/parsed-failures.json
    Load: .claude/.artifacts/gemini-analysis.json (dependency context)

    GRAPH CONSTRUCTION:
    1. Nodes: Each failure is a node
    2. Edges: Failure A → Failure B if:
       - B's error message references A's file
       - B's file imports A's file
       - B's line number > A's line number in same file
       - Gemini dependency graph shows A → B relationship

    3. Apply graph algorithms:
       - Topological sort to find root nodes (no incoming edges)
       - Calculate cascade depth (max distance from root)
       - Find strongly connected components (circular dependencies)

    4. Identify root causes:
       Root = node with 0 incoming edges OR
       Root = node in cycle with most outgoing edges

    OUTPUT:
    {
      graph: { nodes: [], edges: [] },
      roots: [ /* root failure nodes */ ],
      cascadeMap: { /* depth levels */ },
      circularDeps: [ /* cycles detected */ ]
    }

    Store: .claude/.artifacts/failure-graph-1.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-1"`,
    "analyst")

  Task("Failure Graph Analyst 2",
    `Validate graph structure from Analyst 1:

    Load: .claude/.artifacts/failure-graph-1.json
    Load: .claude/.artifacts/analysis-synthesis.json (consensus data)

    VALIDATION PROCESS:
    1. Cross-check edges:
       - Verify each edge using synthesis consensus
       - Remove edges with low confidence (< 5/7 agreement)
       - Add missing edges identified by consensus

    2. Identify hidden cascades:
       - Indirect cascades (A → B → C, but A → C not obvious)
       - Time-based cascades (A fails first, causes B later)
       - State-based cascades (A leaves bad state, B fails on it)

    3. Validate root cause claims:
       For each claimed root:
       - Verify no hidden dependencies
       - Check if truly primary or just first detected
       - Use 5-Whys: "Why did this fail?" → repeat 5 times

    OUTPUT:
    {
      validatedGraph: { /* corrected graph */ },
      validatedRoots: [ /* confirmed roots */ ],
      hiddenCascades: [ /* newly discovered */ ],
      conflicts: [ /* disagreements with Analyst 1 */ ]
    }

    Store: .claude/.artifacts/failure-graph-2.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "graph-2"`,
    "analyst")

// Wait for graph analysis
npx claude-flow@alpha task wait --namespace "cicd/graph-analysis"
```

### Phase 2: Connascence Analysis

**Identify code coupling that affects fix strategy**

```javascript
[Single Message - Parallel Connascence Detection]:

  Task("Connascence Detector (Name)",
    `Scan for connascence of name: shared variable/function names causing failures.

    Load root causes: .claude/.artifacts/failure-graph-2.json (validatedRoots)

    For each root cause file:
    1. Find all references to symbols (variables, functions, classes)
    2. Identify which files import/use these symbols
    3. Determine if failure requires changes across multiple files

    Connascence of Name = When changing a name requires changing it everywhere

    Impact on fixes:
    - High connascence = Must fix all references atomically
    - Low connascence = Can fix in isolation

    Store: .claude/.artifacts/connascence-name.json
    Use hooks: npx claude-flow@alpha hooks post-task --task-id "conn-name"`,
    "code-analyzer")

  Task("Connascence Detector (Type)",
    `Scan for connascence of type: type dependencies causing failures.

    Load root causes: .claude/.artifacts/failure-graph-2.json

    For each root cause:
    1. Identify type signature