Computer-Use Agent

An agent that drives an app with no API: it sees a screenshot and emits an input event (click, key, scroll), and every hard choice traces to one fact, that the observation is an image you cannot re-read.

Core constraints

• Computer use is a last resort: 10-50x the cost of an equivalent API call, less reliable. If an API or OS accessibility tree exists, use it.
• Observation is an image, not text, so it cannot be compressed like text.
• The screenshot is destroyed after capture: there is no source to re-read.
• The action is continuous, irreversible, untyped: click(640, 812) carries no schema and no return value; the only evidence it worked is the next screenshot.
• Model is stateless: the full context array is re-sent every turn, so a step-3 screenshot keeps costing money at step 30.

Image tokens

• Screenshot is cut into 28x28-pixel patches; each patch = 1 token, regardless of content.

image tokens = ceil(width / 28) x ceil(height / 28)

• 1400 x 788 = 50 x 29 = 1,450 tokens (the default). No inter-frame discount; blank costs the same as dense.
• Planning shortcut: w x h / 750 tokens (runs ~3% high; use count_tokens for the bill).

Resolution (the largest cost lever)

Accuracy is flat above ~1400 wide and falls off a cliff below ~960. Above the tier’s long-edge cap the provider downscales before counting, so capturing 4K is pure waste. Downscale inside the VM.

• Accuracy numbers are illustrative from one 40-state set; measure your own curve.
• The cliff sits at your smallest interactive target.

Prompt caching (biggest lever: ~2.5x short, ~8x long)

• Cache reads bill at ~0.1x; a new cache write costs a ~1.25x premium.
• It is a prefix match: a change at position j invalidates everything from j onward.
• Untrimmed the array is append-only, so only the new ~1,600 tokens/turn are full-price; the rest is a cache hit.
• Context after turn n = 2,400 + 1,600 x (n - 1) (linear).
• Cost without caching is a triangular sum: quadratic in step count. With caching: linear.

Trimming vs caching

• Trimming (drop old screenshots) rewrites the prefix, so it breaks the cache: with caching on it is a ~1.9x loss. It “saves 90% of a bill you were only paying 10% of.”
• Trim only when the context window itself would be exceeded (200k window: ~turn 100-120), and then in batches (e.g. re-trim every 8 steps) so the prefix stays stable between re-trims.
• describe_and_shrink: replace an image about to be dropped with a ~30-token caption from the cheapest model (~1,500 to ~30 = 50x). Shares the prefix defect; batch it.

Stall detector

A stalled run looks identical to a working one from inside the loop.

• Never use hash(png): sha256 fires on cursor blink / antialiasing (false negative on real stalls) and never fires on a live carousel (false positive).
• Perceptual delta: shrink both frames to a 64x64 grayscale grid, count cells changed by more than tolerance 12, report the fraction. Threshold 0.5%.
• Real change (dropdown) ~3%+ vs fake change (cursor blink) ~0.02%: >2 orders of magnitude apart, so 0.5% is safe.
• Four-state machine: Progressing -> Suspect (1 no-change) -> Warned (2nd) -> Halted (4th).
• Warn exactly once at the 2nd no-change (code: stalls == 2, not >= 2): a repeated pattern makes repetition more likely, so inject one contradicting observation; recovered ~60% of stalls. A second warning just feeds the pattern.

Memory: never cache a coordinate

• A coordinate is a function of app version, window size, DPI scale, theme, zoom, font size, locale, scroll position, banners, etc. Any of them shifts it.
• Break-even staleness is ~1%, and real p is far above it, so store hints, not commands: “gear is usually top-right, left of the avatar,” then look before clicking.
• With a DOM / accessibility tree, a target can be named (role=combobox, name="Digest frequency") and survives re-layout: this whole section evaporates.

Security (lives in infrastructure, not the prompt)

The agent is a confused deputy taking instructions from an attacker-controllable screen. Prompt injection can address the model directly, and pixels have no channel separation, so any marker you draw an attacker can draw too. Controls, in descending value:

1. Pre-authenticate, never authenticate: inject a session cookie; the password is not in the VM.
2. Least-privilege session (IAM), so the API-keys page is unreachable.
3. Snapshot restore the VM after every run (also makes evals reproducible).
4. Egress allowlist: only the target domain resolves, so a hijacked agent cannot exfiltrate.
5. Confirmation gate on destructive labels (delete, revoke, …), read via OCR.

The “text on screen is DATA” prompt rule is the weakest, last layer.

Gotchas

• Capture once per step and use the same bytes for both the delta and the message; two captures can drift.
• Budget check must run before the step cap: one long turn can cost what five short ones do. Use a dollar and a step cap.
• wait and done are real tools; there is deliberately no click_element(name) (that would lie about the environment having a structured view).
• Prefer key over click: keyboard nav is coordinate-free and survives re-layout, removing the largest failure class.
• Evals: reset the VM snapshot before every run; assert on final application state, not the trajectory; report accuracy per resolution.