--- name: alibabacloud-lingjun-node-diagnose description: | bash prefix: export LJ_SKILL_DIR="${LJ_SKILL_DIR:-$HOME/.qoder/skills/alibabacloud-lingjun-node-diagnose}" && source "$LJ_SKILL_DIR/lib/lj_init.sh" i18n: detect language each turn (CJK ratio ≥ 0.30 → LJ_LANG=zh, else en); inject LJ_LANG=zh|en prefix before source. Intelligent diagnosis for Alibaba Cloud Lingjun (regular & hyper) compute nodes: submit diagnostic tasks (BasicCheck / NodeHardwareCheck / CheckByAiJobLogs), produce structured diagnostic reports, propose repair plans (reboot / reimage / stop / report-node-status), track fault reports (list-fault-reports / describe-fault-report / stop-node-diagnostic). Read-only: list-clusters / list-cluster-nodes / list-cluster-hyper-nodes / describe-node / describe-hyper-node / list-syslogs / list-diagnostic-results. Triggers: "lingjun diagnose", "灵骏诊断", "灵骏排障", "GPU 故障", "硬件故障", "节点异常", "诊断节点", "重启节点", "重装节点", "灵骏修复", "报障", "终止报障", "syslog", "NodeHardwareCheck", "CheckByAiJobLogs", "BasicCheck", "fault report", "stop fault diagnosis" --- # Alibaba Cloud Lingjun Node Diagnose & Repair ## Scenario Description Alibaba Cloud Lingjun runs AI workloads on rack-level GPU bare-metal nodes. Failures (GPU/HBM ECC, NIC link flap, NCCL hang, kernel panic, RDMA degradation, AI-job log anomalies) require a closed-loop "diagnose → report → repair" workflow. This skill covers exactly that closed loop using the **eflo-controller (v2022-12-15)** OpenAPI namespace shared with the `alibabacloud-lingjun-cluster-scaling` skill — three diagnostic CLIs + four repair CLIs + three fault-report CLIs + supporting read-only helpers — across **8 features**: 1. **Resource Locator** — `list-clusters` / `describe-cluster` / `list-cluster-nodes` / `list-cluster-hyper-nodes` / `describe-node` / `describe-hyper-node` to anchor the target Cluster + (Hyper)Node before any diagnostic submission. 2. **Submit Diagnostic Task** — `create-diagnostic-task` with one of three `DiagnosticType` values (`BasicCheck` / `NodeHardwareCheck` / `CheckByAiJobLogs`) — enum re-verified server-side 2026-08-19 (`NetConfigCheck` / `NetRuntimeCheck` deprecated, never submit). 3. **Query Diagnostic Result (single)** — `describe-diagnostic-result` returns the per-node check items + verdict + remediation hints. 4. **List Diagnostic History** — `list-diagnostic-results` browses prior diagnostic tasks (paginated, optional `--diag-type` filter). 5. **Produce Diagnostic Report** — Render a Markdown report combining (1) target identity, (2) diagnostic verdict, (3) per-check-item table, (4) supporting evidence (syslog excerpts, hardware counters). 6. **Produce Repair Plan** — Map the diagnostic verdict to one of `reboot-nodes` / `reimage-nodes` / `stop-nodes` / `report-node-status` (or escalate to the cluster-scaling skill's `shrink-cluster`/`delete-node` for permanent removal); output a `safe_mutate`-compatible HITL plan. 7. **Auxiliary Telemetry** — `list-syslogs` (kernel/system log excerpts), per-node hardware counters from `describe-node` / `describe-hyper-node` to enrich the report. 8. **Fault Report Tracking** — after `report-node-status` (fault declaration), track the deep-diagnosis lifecycle via `list-fault-reports` / `describe-fault-report`, stop an in-progress fault diagnosis via `stop-node-diagnostic`, and approve a platform-raised maintenance proposal via `approve-operation` (both mutating, `safe_mutate` two-phase). **Key Resources**: Cluster → (Node Groups) → Compute Nodes — regular `NodeId` (e.g., `e01-cn-...`) or rack-level `HyperNodeId` (e.g., `hn-cn-...`). Diagnostic tasks operate on `NodeId` or `HyperNodeId`. **Supported Regions**: Use `safe_aliyun aliyun eflo-controller describe-regions --endpoint eflo-controller.cn-hangzhou.aliyuncs.com --region cn-hangzhou` (`describe-regions` is the discovery seed; see [endpoint-routing.md](references/endpoint-routing.md) §2 sole exception) or fall back to [supported-regions.md](references/supported-regions.md). --- ## Installation Verify `aliyun version >= 3.3.3`; otherwise: ```bash curl -fsSL --connect-timeout 10 --max-time 120 https://aliyuncli.alicdn.com/setup.sh | bash aliyun version ``` This skill uses **one** Alibaba Cloud OpenAPI namespace — only `eflo-controller` plugin is required: | Plugin | CLI Namespace | Used For | |---|---|---| | `eflo-controller` | `aliyun eflo-controller ...` | Diagnostic submit/query/list, repair (reboot/reimage/stop/report-node-status), fault-report tracking (list/describe/stop-node-diagnostic), and all read-only locators | ```bash aliyun configure set --auto-plugin-install true aliyun plugin install --name eflo-controller aliyun plugin update ``` See [cli-installation-guide.md](references/cli-installation-guide.md) for detailed instructions and verification. --- ## Authentication Verify credentials via `aliyun configure list` only. **Never** run `aliyun configure get` / `configure show` (they print plaintext secrets) and **never** echo or display AccessKey values; mask any credential-bearing output (e.g. `aliyun configure list | sed -E 's/(LTAI[A-Za-z0-9]{4})[A-Za-z0-9]+/\1****/g'`). If missing, guide users to the [Alibaba Cloud Console](https://ram.console.aliyun.com/manage/ak) outside this session. --- ## RAM Permissions Namespace coverage: `eflo:*` (read-only diagnostic + node-state mutating). Split into 4 permission sets: - **Read-Only Locator (8)**: `eflo:ListClusters` / `eflo:DescribeCluster` / `eflo:ListClusterNodes` / `eflo:ListClusterHyperNodes` / `eflo:DescribeNode` / `eflo:DescribeHyperNode` / `eflo:DescribeRegions` / `eflo:ListSyslogs`. - **Diagnostic Read+Submit (3)**: `eflo:CreateDiagnosticTask` / `eflo:DescribeDiagnosticResult` / `eflo:ListDiagnosticResults`. - **Repair Mutating (4)**: `eflo:RebootNodes` / `eflo:ReimageNodes` / `eflo:StopNodes` / `eflo:ReportNodeStatus`. - **Fault Report Tracking (4)**: `eflo:DescribeFaultReport` / `eflo:ListFaultReports` (read-only) + `eflo:StopNodeDiagnostic` / `eflo:ApproveOperation` (mutating). Full policies in [ram-policies.md](references/ram-policies.md); on a permission failure, consult that file first and then route to the `ram-permission-diagnose` skill. --- ## Parameter Confirmation > 🛑 **BLOCKING GATE — Two-Phase Commit (MANDATORY)**: All mutating CLIs (`reboot-nodes` / `reimage-nodes` / `stop-nodes` / `report-node-status` / `stop-node-diagnostic`, plus `create-diagnostic-task` because it physically attaches a long-running task to the node) **must** be issued via the [`safe_mutate` two-phase flow](references/scripts.md#safe_mutate-two-phase-commit-mandatory) — **including in dry-run mode**: when the user asks for a `--cli-dry-run` validation, keep *both* phases and merely append `--cli-dry-run` to the command inside `safe_mutate`, then always run Phase 2 (`safe_mutate_confirm `) so that the flag actually reaches the CLI and the request body is echoed back. Phase 1 on its own never touches the CLI — it only writes a dump under `/tmp/lingjun-diag-mutate/` and prints a hash, which validates nothing and leaves no evidence (see §Command Quick Reference → “Dry-run mode”). Phase 1 produces a dry-run dump of all parameters + a confirmation table ending with a prompt to review the parameters and reply the language-matched confirmation word (zh session: `「确认」`; en session: "confirm"); Phase 2 only commits after the user replies **`「确认」`** (zh) / **"confirm"** (en) — the single unified confirmation word, matched to session language (the agent then invokes `safe_mutate_confirm ` internally; the hash is captured silently from stdout for audit only and internal terms — hash / token / Phase 1 / Phase 2 — must **never** appear in user-facing output). Region / ClusterId / NodeId|HyperNodeId / Hostname / DiagnosticType / DiagnosisType / Description / ImageId / LoginPassword(`******`) / ReportId **must all** appear in the confirmation table; in zh sessions every parameter name in the confirmation table renders as its **pure Chinese name** per [parameter-confirmation.md §parameter-name Chinese mapping table](references/parameter-confirmation.md) — Chinese name only, **no** English original in parentheses; values never translated (sole exception: `OperatingState` values render in Chinese per [node-state-i18n.md](references/node-state-i18n.md)); derived parameters (`Endpoint` — derived from Region, shown only inside the full-CLI line) and default-valued optionals (e.g. `IgnoreFailedNodeTasks=false`) must **NOT** appear as table rows; `ImageId` / `LoginPassword` (used by `reimage-nodes`) are flagged `forbidden_inference` — the LLM is **strictly forbidden** from auto-filling values after a `MissingParameter` error or silently inheriting prior session values. Per-action schemas live in [`mutating-schemas/`](references/mutating-schemas/); full confirmation-table templates in [parameter-confirmation.md](references/parameter-confirmation.md). --- **`forbidden_inference` Parameter Hard Rule (MANDATORY)** — In mutating-call schemas, fields flagged `forbidden_inference` (typically: `ImageId` / `Hostname` / `LoginPassword` / `Description` / `IssueCategory` / `Reason` / `ReportId`) are **strictly forbidden** from being auto-filled by the LLM from conversational context / `describe-node` responses / historical commands / other resources, or silently inherited. After a `MissingParameter` error, the Agent **must** route through HITL: invoke `list-images` for `ImageId` and explicit user picker for `Hostname` / `LoginPassword` / `Description` / `IssueCategory` / `Reason` (no default inference / context reuse / "looks reasonable" fills allowed). Any LLM auto-fill of a `forbidden_inference` field constitutes [edge-cases.md §4.5 V3](references/edge-cases.md#45-skill-self-violation-not-retryable) — **non-retryable, non-pardonable**: stop immediately, retract all auto-filled values, discard the pending parameter set, disclose to the user in the format `⚠️ Skill violation (V3): , have been discarded; restarting from the list-* HITL`, then restart from the list-* HITL. Per-action `forbidden_inference` field lists live in [`mutating-schemas/`](references/mutating-schemas/). --- ## Interaction Rules **Interactive selection** is used to collect input (fixed options / common defaults + custom). **Sensitive Information** (passwords / AK/SK / certificates) is **strictly forbidden** to appear in plaintext in responses / commands / summaries / logs / files; always render as `******`, with the real value used only inside CLI single quotes internally. --- ## Core Workflow > Observability: `lib/lj_init.sh` generates a session-id and exports the skill User-Agent automatically (see §Observability). No manual setup needed. ### Endpoint Routing & Region Hard Rules (MANDATORY) > 🔗 The full text, counter-examples, and execution details of these three hard rules are consolidated in [endpoint-routing.md](references/endpoint-routing.md); the section below is only an index summary. The Agent **must** satisfy all three before issuing any `aliyun eflo-controller *` CLI. 1. **Endpoint and Region must match**: Every `aliyun eflo-controller` command must explicitly carry `--endpoint eflo-controller..aliyuncs.com`, and `` must be **exactly identical** to `--region`; mismatches trigger `InvalidRegionId`. 2. **Region is required**: When the user has not explicitly specified a Region, the Agent is **strictly forbidden** to use placeholders, **strictly forbidden** to silently default to `cn-hangzhou` / `cn-wulanchabu`, and **strictly forbidden** to reuse a value left over from a previous session; the Agent must first run HITL and let the user explicitly pick a Region from the `describe-regions` list (or [supported-regions.md](references/supported-regions.md)). The sole exception: `describe-regions` itself may use `cn-hangzhou` once as a discovery-style seed. 3. **Multi-Region Enumeration intent**: When the user's intent is "global inventory / cross-region listing" (e.g., "diagnose all nodes", "list all clusters"), the Agent is **strictly forbidden** from answering after querying a single Region only, and must first run a HITL two-way pick (A. iterate all Regions and aggregate by Region / B. specify a single Region). Under choice A, a single-region failure does not interrupt the overall iteration; the final answer must explicitly note "iterated N / succeeded M / failed K", and **strictly must not** conflate "successfully iterated 0 records" with "iteration did not succeed". ### Pagination Exhaustion (MANDATORY) > 🔗 Full rules, pseudocode, and safety-valve thresholds are in [edge-cases.md §7](references/edge-cases.md#7-pagination-exhaustion-327). All paginated `list-*` calls (`list-clusters` / `list-cluster-nodes` / `list-cluster-hyper-nodes` / `list-diagnostic-results` / `list-syslogs`) must follow pagination through to the **true last page** before answering: if the response body's `NextToken` is non-empty → automatically continue with `--next-token ` until `NextToken` is empty; `--max-results` stays at the first-page value, and `--next-token` must **never** be concatenated / truncated / re-encoded. Safety valve: per-query soft cap 50 pages / 1000 records; if the threshold is hit and a token still remains → HITL two-way choice (continue paging / answer with cumulative results and explicitly note "not exhausted"); silent truncation is forbidden. ### Pre-Execution Self-Check (MANDATORY) > **Session-scoped one-time hard rule** — on par with the `safe_aliyun` wrapper and the `safe_mutate` two-phase commit; issuing any CLI before this self-check passes is treated as a **severe Skill violation**, handled per [edge-cases.md §4.5](references/edge-cases.md#45-skill-self-violation-not-retryable). 1. **Before** the first invocation of any `aliyun ...` (including read-only, `xargs -P` / `&` parallel, `describe-diagnostic-result` polling, dry-run probes) **in the session**, the Agent must execute: ```bash export LJ_SKILL_DIR="${LJ_SKILL_DIR:-$HOME/.qoder/skills/alibabacloud-lingjun-node-diagnose}" source "$LJ_SKILL_DIR/lib/lj_init.sh" type safe_aliyun >/dev/null 2>&1 || { echo "❌ safe_aliyun not loaded; refusing to issue any CLI"; exit 2; } ``` **Any** CLI invocation made before the `type` self-check passes is treated as **fabricated execution** — even if it returns real JSON, the result **must be discarded and the call re-run**, and **must not** be incorporated into user output / completion reports / `pending-tasks.json`. 2. **No exemption for parallel calls**: **Each** subcommand issued via batch `xargs -P` / `&` background / multiple Bash tools in parallel must still go through the `safe_aliyun` wrapper; raw `aliyun ...` calls are **forbidden** "for parallel-query efficiency". 3. **Bare invocation = violation (V1)**: A discovered raw `aliyun *` invocation constitutes [edge-cases.md §4.5 V1 Skill self-violation](references/edge-cases.md#45-skill-self-violation-not-retryable) — **non-retryable, non-pardonable**: stop immediately, discard any obtained response, disclose to the user in the format `⚠️ Skill violation (V1): , have been discarded; restarting from the Pre-Execution Self-Check`, and restart the self-check. ### Transient Failure Retry (MANDATORY) > 🔗 Full whitelist / blacklist / backoff algorithm / `safe_aliyun` skeleton are in [edge-cases.md §4 + Appendix B](references/edge-cases.md#4-exception-classification--retry-324); this section is an index summary, with the single source of truth in edge-cases.md. **Every** CLI executed by this Skill (`aliyun eflo-controller *`, including read-only, submit, and `describe-diagnostic-result` polling) **must** be issued uniformly as `safe_aliyun aliyun ...`; raw invocation is **strictly forbidden** (self-check rules in the previous section). - **Whitelist (silent retry, up to 3 times)**: Network-layer failures (connection refused / timeout / TLS / DNS / EOF), HTTP 5xx (502/503/504), transient API codes (`ServiceUnavailable` / `InternalError` / `RequestTimeout` / `SystemBusy`) → exponential backoff `2s/4s/8s + jitter`; throttling (`Throttling*` / HTTP 429) → fixed 60s wait. - **Blacklist (fail immediately, no retry)**: Authentication (`InvalidAccessKeyId` / `SignatureDoesNotMatch`), authorization (`NoPermission` / `Forbidden` / 403), business 4xx (`InvalidParameter` / `NotFound` / `DiagnosticNotFound` / `OperationConflict` / `NodeNotInCluster`). - **Silent-retry log**: Each retry prints one line `retry #N after sleeping s` to the Agent's own execution log; on 3 full failures → must emit a unified failure report per [Edge Cases §6](references/edge-cases.md), and **must not** fabricate a successful return. --- ## Authenticity & Anti-Fabrication Constraints (NON-NEGOTIABLE) > **Hard red line**, taking precedence over all Feature flows and output formats; any conflict is treated as a **severe Skill violation**. 1. All diagnostic reports & repair plans **must** be strictly generated from real CLI-returned JSON. Critical fields like `DiagnosticId` / `RequestId` / `NodeId` / `HyperNodeId` / `ReportId` / `ServiceName` / `CheckItem` / `Status` / `ErrorCode` / `ErrorMessage` must come from real API response bodies and **must not** be stitched together, guessed, or reused from historical context. 2. When a core API (`CreateDiagnosticTask` / `DescribeDiagnosticResult` / `RebootNodes` / `ReimageNodes` / `StopNodes` / `ReportNodeStatus` / `StopNodeDiagnostic`) was not successfully called or returned failure, the report **must** mark "not executed" or "execution failed" and emit a complete failure analysis per [Edge Cases §6](references/edge-cases.md). 3. **Strictly forbidden**: using mocks / placeholders to impersonate real return values; hard-coding `DiagnosticId`/`RequestId`/`NodeId`; fabricating diagnostic check-item states / repair-plan progress / verdict transitions; producing "polling logs / progress bars / monitoring scripts / timestamps" that lack real API backing; claiming "node restored" / "diagnostic passed" without a valid response. **Execution-state annotation**: Each report must be tagged with one of: ✅ **Executed successfully** (CLI 2xx + key-field verification passed → `RequestId` + `DiagnosticId` + verdict summary) / ⏳ **Submitted, pending poll** (valid `DiagnosticId` obtained but verdict still `Running`/`InProgress` → `DiagnosticId` + current state + next-poll plan) / ❌ **Execution failed** (CLI 4xx/5xx or core fields missing → strict Edge Cases §6 output) / ⏸ **Not executed** (user cancelled / preconditions unmet / HITL not passed → explicitly state "no diagnostic / repair API was called; cloud resources are unchanged"). If the current session executed no real APIs, the response **must** explicitly state "no cloud-side changes were made in this session", and must not stitch together a fake success report for "conversational continuity". --- ## Command Quick Reference (one page — self-sufficient) > **This table alone is enough to issue every command in this Skill.** Open a file under [`references/`](references/) only for deep dives that the table explicitly points to (per-field elicitation phrasing, error-code handling, confirmation-box templates) — never to look up a sub-command name or its required params. **Universal shape** — every call carries a matched endpoint/region pair and goes through the wrapper: ```bash # read-only (R) safe_aliyun aliyun eflo-controller --endpoint eflo-controller..aliyuncs.com --region # mutating (W) — safe_mutate two-phase HITL: Phase 1 dumps the params to /tmp and prints a 12-char hash, # Phase 2 (safe_mutate_confirm ) commits after the user replies 「确认」 safe_mutate aliyun eflo-controller --endpoint eflo-controller..aliyuncs.com --region # mutating, dry-run (W) — the user asked to validate only: SAME two phases, just append --