# Glasswarp — Agent Setup Skill

Glasswarp gives an AI agent **eyes and hands on a real Windows machine**:
native-resolution vision and native input through one API. The customer supplies
the brain (model, prompts, and task logic). Built for automation developers and
small shops already doing no-API Windows work on a PC they or their client owns —
not a hosted outcome agent.

If the user only wants Claude / Cursor (or another MCP client) connected — with
**no project code** — stop and direct them to
https://www.glasswarp.com/mcp and https://docs.glasswarp.com/get-started/mcp
instead of installing the SDK.

## Ways to run agents (same rules as MCP `instructions`)

Glasswarp gives you eyes and hands on a real Windows PC that its owner paired to this
account. Glasswarp supplies vision and input; you make every decision. Attribute actions
to yourself, not to Glasswarp.

## Before anything: a usable rig

Call `rigs.list`. You need a rig with status USABLE (online, API access enabled by the
owner). If none exists, direct the user to install the host agent
(https://www.glasswarp.com/downloads) — no Glasswarp account on that PC — then pair
the tray code while logged into the automator's account (https://www.glasswarp.com/pair)
and enable API access. This is a physical owner-consent step: never attempt to work
around it, and never ask for machine passwords or credentials.

## Choose a path (capability + consent)

### Short tasks and human-in-the-loop — always direct MCP
Glances, one or a few clicks, typing a string, "what's on my screen?", open an app and
do one thing: use MCP tools in this chat.
`session.start` -> `screen.observe` -> act -> `session.end`.
Never scaffold code for a short task — "click OK" must never generate a file.

### Longer tasks — offer, don't impose
Multi-step work, games, workflows, "solve / clear / finish X":

1. Optionally `demos.list` — if a showcase matches, show its run contract from
   `demos.get` (install + command).
2. Then branch on what THIS client can do and what the USER wants:

**Code-capable client** (can write files and run a terminal):
- Offer to write a small SDK agent that loops screen.observe -> decide -> act outside chat
  (or the matching showcase command). Explain the tradeoff plainly: chat-driven play is
  one model turn per action — slow and minute-hungry; a scaffolded agent runs the loop
  at full speed.
- Do not impose it. If the user prefers chat or says "just do it here," proceed with
  direct MCP tools.
- If they accept: scaffolded agents must follow SKILL.md
  (https://www.glasswarp.com/SKILL.md) — `session.end` in a `finally` block, bounded
  loops, observe-verify between actions, and re-observe + re-resolve targets (never
  hardcode target ids). Use MCP only for preflight, Live View, and recovery
  ("it crashed — look at the screen").

**Chat-only client** (no reliable shell or file write):
- Proceed with direct MCP tools for the task.
- Briefly note once that a faster path exists via the Python SDK or `glasswarp-demo`
  (https://docs.glasswarp.com/guides/ways-to-run-agents) if they later use a
  code-capable environment — then keep helping here.

These path names (Assist / Showcase / Scaffold) are internal vocabulary — never say
them to the user; just describe what you will do.

## Showcase tools
- `demos.list` / `demos.get` return run contracts only; they do not execute solvers.
- Resources: `glasswarp://demos`, `glasswarp://demos/{id}`

## Working hygiene
- Prefer `input.send_actions` for any predictable multi-step sequence (click field → type →
  tab → shortcut). Solo `input.click_target` / `input.type_text` / `input.send_keys` are for one-off
  acts; chaining them costs a chat turn each.
- Plan ahead when the next steps are predictable. After an `screen.observe`, if you can
  confidently predict the next 3–6 actions (menu paths, dialog fields, keyboard
  shortcuts, typing into a field you just clicked), send them as one batch via
  `input.send_actions` (observe_after defaults true; verification is text+targets unless
  you set `observe_image=true`) — then continue from that result, not after every
  individual action. Single-step when the screen may change unpredictably: page
  loads, network waits, installers, anything that can pop a modal. If the verifying
  observe shows something unexpected, re-plan from there.
- Observe after every meaningful step (after a batch or after an unpredictable
  single action), not blindly after every click. `screen.observe` defaults to text +
  targets only — set `image=true` only when you must read or judge the screen
  visually. If `screen.observe` reports no change since your last frame (`changed: false`),
  do not re-analyze — the server omits any JPEG even if `image=true`. Re-check or
  wait instead. If dirty data is missing (`dirty` is null / unavailable — e.g. GDI
  fallback), treat that as **assume changed** and keep verifying; do not skip the
  model call.
- On a Chromium or Electron surface (Chrome, Edge, VS Code, Slack, Discord, any
  Electron app), the UIA tree builds lazily. If the first observe returns fewer than
  ~30 targets with no named content controls, wait 2-3 seconds and observe again
  before concluding the screen is unreadable.
- Verification-grade JPEGs: `max_width=960`, quality ~60 (MCP defaults when
  `image=true` / `observe_image=true`).
- Prefer `input.click_target` (numbered targets from the latest observe) over `input.click_xy`.
  `input.click_xy` uses NATIVE screen coordinates as reported by observe — not the downscaled
  JPEG size.
- Target ids are valid for the observe that produced them. Do not persist them across
  sessions, and re-observe before acting on a screen whose contents may have changed
  (lists, file browsers, tables).
- Always `session.end` when done or when abandoning a task. Billing is wall-clock from
  session start to end; idle sessions auto-end after ~15 minutes.
- If `session.start` fails on a concurrency limit, tell the user which session holds
  the slot and that it will auto-end when idle (or can be ended in the Console).
- `session.live_view` is for the rig owner (Console login). Offer it when starting
  long or sensitive tasks so they can watch and intervene.

Guide: https://docs.glasswarp.com/guides/ways-to-run-agents

```bash
pip install "glasswarp[demos]"
export GLASSWARP_API_KEY=...
glasswarp-demo minesweeper
```

Follow every step below when installing the SDK. Do not claim setup is complete
until the verification step passes.

## Step 1: Inspect the project and ask one question

Before installing anything, inspect:

- Language, framework, and runtime versions
- Package manager and dependency files
- Project structure, entry points, tests, and file conventions
- Existing agent/computer-use code
- Existing environment-variable and secrets conventions

Then ask:

> Should I add Glasswarp to this existing project, or create a minimal Python
> quickstart alongside it?

The supported SDK is Python 3.9+. If the existing project is not Python, explain
that clearly and offer either:

1. A small Python worker/sidecar using the supported SDK, or
2. Direct REST integration from the existing language using
   `https://docs.glasswarp.com`.

Do not silently introduce a second runtime.

## Step 2: Install dependencies using the project's package manager

Core SDK dependencies are managed by the package: `httpx>=0.24`.

Use the command matching the project:

```bash
# pip / venv
python -m pip install glasswarp

# uv
uv add glasswarp

# Poetry
poetry add glasswarp

# Pipenv
pipenv install glasswarp
```

If the project uses `requirements.txt` or `pyproject.toml`, update that file
rather than installing an untracked dependency.

For grounding helpers, ROI processing, or image inspection, install the optional
Pillow dependency:

```bash
python -m pip install "glasswarp[grounding]"
# or: uv add "glasswarp[grounding]"
# or: poetry add "glasswarp[grounding]"
```

Do not install the grounding extra for the basic session/input loop.

Install from PyPI: `pip install glasswarp` ([pypi.org/project/glasswarp](https://pypi.org/project/glasswarp/)).
When working inside the Glasswarp monorepo and iterating on the SDK itself, you
may use the editable install instead:

```bash
python -m pip install -e sdk/python
```

## Step 3: Collect credentials safely

Ask the user to create an API key at:

`https://www.glasswarp.com/console` → **API Keys**

Create an account at `https://www.glasswarp.com/signup`, then create a key in
Console → API Keys.
Store the key as `GLASSWARP_API_KEY` using the project's existing secrets
convention. Never print, commit, or hardcode the key.

- If the project already loads `.env`, add `GLASSWARP_API_KEY` to `.env` and
  add only a placeholder to `.env.example`.
- If it uses a secret manager, follow that system.
- Add `python-dotenv` only when the project needs local `.env` loading and does
  not already provide it.

## Step 4: Guide the human through Windows host setup

A real Windows host is a product requirement, not an error. The coding agent
cannot complete this physical-machine step for the user.

Tell the user:

1. On the Windows machine, download `GlasswarpSetup.exe` from
   `https://www.glasswarp.com/downloads` (no Glasswarp account on that PC) and
   complete the setup wizard.
2. The automator signs in at `https://www.glasswarp.com/pair` and enters the
   pairing code from the tray icon (near the clock; check the `^` overflow on
   Windows 11). The person at the PC should not create a Glasswarp login.

3. Open `https://www.glasswarp.com/console` → **Rigs**.
4. Confirm the rig is online.
5. Enable **API access** for that rig. It is off by default and requires owner
   consent.

Pause and ask the user to confirm that an online, API-enabled rig appears in the
console. Continue after confirmation. Do not ask for remote-machine passwords
or attempt to bypass owner consent.

## Step 5: Create or integrate

For a fresh quickstart, create a small file matching the project's conventions
(for example `glasswarp_quickstart.py`). For an existing agent, put the
Glasswarp adapter at the existing integration boundary; do not mix prompts or
task logic into the transport client.

Use this minimal, safe loop:

```python
import os

from glasswarp import GlasswarpClient

gw = GlasswarpClient(api_key=os.environ["GLASSWARP_API_KEY"])

rigs = gw.list_rigs()
rig = next(
    (r for r in rigs if r.online and r.api_access_enabled),
    None,
)
if rig is None:
    raise RuntimeError("No online rig with API access enabled")

session = gw.create_session(rig_id=rig.id, mode="desktop")
sid = session.session_id

try:
    shot = gw.screenshot(sid)                 # eyes: native-resolution JPEG
    with open("glasswarp_first_frame.jpg", "wb") as output:
        output.write(shot.jpeg)

    gw.click(sid, x=500, y=400)               # hands: native input
    gw.type_text(sid, "Hello from an agent")
finally:
    gw.end_session(sid)                       # always end metered sessions
```

Useful primitives:

- `screenshot(sid)` — native-resolution JPEG
- `observe(sid)` — text + targets by default; JPEG when `image=true` (+ dirty / grounding)
- `list_targets(sid)` / `input.click_target(...)` — UIA roles, names, and bounds
- `click`, `input.type_text`, `key_press`, `drag`, `mouse_down`, `mouse_up` — input
- `app.launch(sid, path)` — launch a Windows application
- Console **Sessions → Eye** — human Live View at 60fps

Rules:

- Always end sessions in `finally`; sessions meter usage.
- Treat an owner kill as final. A `410` with `code: aborted_by_owner` means the human
  stopped you mid-batch — `executed` events landed, `aborted` ones did not. Stop the
  loop and report; never restart a session the owner just killed.
- Treat a `503 capture_recovering` as "no eyes right now" — retry observe for a few
  seconds. Never fall back to acting on your last known frame.
- Re-observe and re-resolve target ids before clicking; do not hardcode ids across
  observes or sessions (lists, file browsers, and tables change).
- Reuse the project's logging/error conventions.
- Do not log API keys or full sensitive screenshots.
- Glasswarp supplies eyes, hands, and safe session controls. Prompts, CV,
  solvers, and agent decisions remain customer code.

## Step 6: Verify and fix

Run the integration and verify all of the following:

1. The SDK imports in the project's actual environment.
2. `rigs.list()` returns an online rig with API access enabled.
3. A desktop session starts.
4. `glasswarp_first_frame.jpg` is a non-empty valid JPEG.
5. The session appears at **Console → Sessions** and can be opened in Live View.
6. The session ends even if an action fails.

Run the project's formatter, type checker, and relevant tests after integration.

Common failures:

- `No matching distribution found` — upgrade pip and retry `pip install glasswarp`
  (https://pypi.org/project/glasswarp/). In the monorepo, `pip install -e sdk/python`.
- `401` — missing, expired, or invalid `GLASSWARP_API_KEY`.
- No eligible rig — host offline or API access disabled in **Console → Rigs**.
- Request timeout — verify the host remains online and retry once.
- Input/drag not recognized — update the Windows host agent.
- Screenshot succeeds but model misses targets — use `observe()` or
  `list_targets()` instead of guessing coordinates.

Report exactly what was installed, which files changed, what command was run,
and which verification checks passed.

## Reference

- API reference: `https://docs.glasswarp.com`
- Host download: `https://www.glasswarp.com/downloads`
- Console: `https://www.glasswarp.com/console`
- Example code: `sdk/python/examples/` in the approved SDK repository