A browser in the sandbox

The agent can use a real browser. It runs inside the session's sandbox, next to the code, so the agent can start your app there, open it, click through it and take a screenshot to check its work.

Give your sandbox a browserCopy Anchor Link

There is nothing to switch on. When a sandbox starts, Pistis looks for a browser in it, starts it once in the background to check that it works, and offers the browser tools in that session only if it does. A sandbox without a browser simply has no browser tools.

So add Chromium to the image your sandboxes use:

Pistis looks for chromium, chromium-browser, google-chrome and chrome-headless-shell on the PATH, then in Playwright's download folder. If yours is somewhere else, set PISTIS_BROWSER to its path in the image. Only Chromium-based browsers work. The browser runs without its own sandbox (--no-sandbox): the sandbox around it is what keeps it apart.

Pistis also ships an image with a browser, pistis/browser:dev (Debian with its Chromium, plus Node and Git), built with node scripts/image.ts browser; pnpm pistis dev --browser uses it. Each release publishes it as ghcr.io/akshay5995/pistis-browser.

Choose per agentCopy Anchor Link

An agent's Browser setting, and "browser" in pistis.json for sessions without an agent:

• "browser": "auto" (the default): browser tools wherever the sandbox has a browser.
• "browser": "required": a session starts only on a sandbox with a browser. Without one it is refused with "This sandbox image has no browser. Add Chromium: …".
• "browser": "off": never.

A session's sidebar row says which browser its sandbox has, for example "sandbox running · browser Chromium 131.0.6778.204".

{
  "browser": "required",
  "sandboxes": { "provider": "docker", "image": "pistis/browser:dev" }
}

What the agent can doCopy Anchor Link

The agent sees each screenshot as a picture. It also shows in the session's timeline as a small image; click it to see it full size.

These are tools like any other: name one in approvals (for example browser__click, or browser__* for all of them) and each call waits for you (Approvals). Plan mode lets the agent read pages and take screenshots, but not click or type.

Watch the screenCopy Anchor Link

Once the agent has used a browser, a small picture of it sits next to the session's status at the top. Click it to see what the agent's browser shows, live, in a window beside the timeline (drag its edge to make it wider; on a smaller screen it opens over the timeline, on a phone as a full-screen sheet). The picture changes as the agent navigates, clicks and types. Expand makes it fill the screen. Close it with the × or Esc.

At the top of the window, one control says what you are doing: Watch, Annotate (point at something for the agent) or Drive (take over).

• Who can watch: anyone who may view the session: its owner, people it is shared with in any role, and admins who may see it. Anyone else is refused.
• When pictures flow: only while someone watches. The sandbox sends a new picture only when the screen changes, and stops altogether when the last person closes the panel, so watching costs nothing when nobody looks. A slow connection gets the newest picture and skips the ones in between.
• What is kept: nothing. The live pictures go from the sandbox to your browser through your Pistis server and are not saved. Only the screenshots the agent takes are recorded.

Over HTTP, GET /sessions/:id/screen streams the screen as multipart/x-mixed-replace: one JPEG part per picture, or a JSON part saying why there is none yet (HTTP API).

Take overCopy Anchor Link

Sometimes a person has to step in: a sign-in only you know, a captcha, a page the agent is stuck on. In the Screen panel, choose Drive to take over. While you are in control:

• Your clicks, scrolling and typing go to the sandbox's browser. Click the picture first so your keys go there; Esc stops your keys going there.
• The agent waits: its next browser step holds until you hand back, even if Pistis restarts in between. One step it already started finishes first.
• Only one person is in control at a time; anyone else watching sees who it is.

When you're done, write a note for the agent if you like (what you did, where it should carry on) and choose Watch to Hand back (or press Enter in the note). The agent gets your note as its next message and continues from the page as you left it.

Closing the tab hands back for you: control returns to the agent once you have been gone for screen.returnAfter (two minutes by default) in pistis.json.

• Who can take over: the session's owner, and admins. People it is shared with can watch but not take over.
• What is recorded: that you took over, for how long, and your note: the timeline shows "took over the screen · 1m 12s · your note". What you typed and clicked, and what the screen showed, is never recorded: not in the session, the logs or the saved workspace. A password you type there stays between you and the site.

Over HTTP: POST /sessions/:id/control takes over (asking again keeps it), POST /sessions/:id/screen/input sends one click or key, and POST /sessions/:id/control/return with { "note": "…" } hands back; GET /sessions/:id/control says who is in control (HTTP API).

Point at thingsCopy Anchor Link

Sometimes it is quicker to show than to say: "this button", "the number in this corner". While you watch, choose Annotate (or press A): the picture holds still, and you drag a box around something or click to drop a pin. Mark up to nine things; they are numbered, so your note can say "make 1 look like 2". Write the note and press Enter.

• What the agent gets: an ordinary message from you: your note, the page's address, each mark's place on the page in its CSS pixels (from the page's top-left, counting how far it was scrolled), and a picture of the marked area with the marks drawn on it, which the model sees. If the agent is working, the message steers it, like any other you send.
• In the timeline: your message, with that picture.
• Who can annotate: anyone who may send the session messages. Someone who can only read it can watch but not annotate. Esc cancels.

A whole desktopCopy Anchor Link

For an app that isn't a web page, give the sandbox a screen of its own: add "desktop": true to pistis.json. Sandboxes then use the desktop image, pistis/desktop:dev (build it with node scripts/image.ts desktop, or use pnpm pistis dev --desktop): a virtual display of 1280×800 that every program the agent starts draws on. The agent works it by pixel, like a person at a screen:

Watching and taking over work the same way as for the browser: Screen shows the whole display, and while you are in control your clicks and keys go to it and the agent's next step on the screen waits. These are tools from an MCP server in the sandbox, so they work with any model that takes pictures; approvals, the record and egress apply to them as to any tool.

What it can reachCopy Anchor Link

The browser reaches what the sandbox reaches. With the default egress, none, that is only the sandbox itself: an app the agent starts there, such as http://127.0.0.1:3000/. Any other address fails to load. To let it reach a site, allow that site in the sandbox's egress, or allow the internet (Sandboxes).

What is keptCopy Anchor Link

Every call and its result are recorded in the session, screenshots included. A screenshot is kept small (at most about 200 KB). The browser keeps its cookies and history outside your workspace, so they are not saved with the session's files and are gone when the sandbox is replaced.