Agent skill · operations · aradotso
pi-computer-use
Control macOS applications with Pi agents using semantic Accessibility API targets and optional screenshots
Why this skill is useful
Provides a set of specific commands and workflows for automating macOS GUI interactions using Accessibility API that the AI wouldn't reliably generate on its own.
What it needs
Requires npm, pi installed locally. About 5k tokens when loaded. Last updated 2026-07-09. 70 stars on the source repository.
What this skill does
pi-computer-use Skill by ara.so — Daily 2026 Skills collection. pi-computer-use gives Pi agents a semantic computer-use surface for visible macOS windows. It prefers Accessibility (AX) targets (like @e1) over raw coordinates, returns semantic state after every action, and attaches screenshots only when AX coverage is too weak. --- Installation Via Pi (recommended) Pin to a specific version: Via npm Remove --- First-Run Permissions On first session, macOS will prompt for permissions for: Grant both: Accessibility — required for AX ref targeting Screen Recording — required for screenshots --- How It Works Three components: 1. Pi extension (extensions/computer-use.ts) — registers public tools and /computer-use command 2. TypeScript bridge (src/bridge.ts) — manages window state, AX refs, fallback policy, batching, execution metadata 3. Native Swift helper (native/macos/bridge.swift) — talks to macOS Accessibility, ScreenCaptureKit, AppKit, CoreGraphics --- Available Tools Tool Purpose ------ --------- listapps List running apps listwindows List windows for an app screenshot Capture window + return AX state click Click element or coordinate doubleclick Double-click element or coordinate movemouse Move cursor drag Drag from point to point scroll Scroll element or coordinate keypress Press key combination typetext Type raw text settext Replace element value via AX wait Pause execution arrangewindow Position/resize window computeractions Batch multiple actions --- Core Workflow Always start a session with screenshot to select the controlled window and obtain AX refs: --- AX Ref Targeting (Preferred) AX refs like @e1, @e2 are returned by screenshot and carry capability metadata: canSetValue — supports settext canPress — supports click canFocus — can receive focus canScroll — supports scroll adjust — supports value adjustment --- Coordinate Fallback Use coordinates only when no suitable AX target exists. …
How to use it
Reference it in AdaL, Claude Code, Cursor or any coding agent — nothing to install:
@skills aradotso/pi-computer-use