Read a window as an outline

Turns a window into a short outline of its headings, text and controls, with a ref on every control that later steps can click or fill.

Read the outline of window into into

Examples #

Read the window, then press its first button

Read the outline of the front window into win
Click an element

Open in Switchboard to try it

Details #

Reads the front window of app (or the one whose title contains window) and writes it down the way a person reads it:

Mail — "Inbox (412)"
  14 controls · 3 fields · 1 table (30 of 412 rows)
  toolbar: [@e1 Get Mail] [@e2 Compose] [@e3 Archive] [@e4 Search…]
  sidebar
    Favorites
      @e5 row "Inbox" (412) [selected]
  message list  (30 of 412 rows)
    @e8 row "Dana Lee | Q4 invoice | 2 Nov" [selected]
    … 382 more rows (app.rows reads them all)

Scroll bars, splitters, unlabelled pictures and the layers a toolkit nests for layout are dropped. The window's regions (a sidebar, a list, a sheet's sections) stay as indented levels, a toolbar is one line, and every control keeps its ref. A password field shows [secure] and never its value. Text longer than 600 characters is cut with a count of what was left out.

mode is reader (the default), interactive (a flat list of controls) or full (every container is a level, nothing folds). maxChars (12,000) is a ceiling: past it the deepest, longest text goes first, then rows, then whole regions from the bottom, each keeping its name line. fullText keeps long text whole.

The result is {text, refs, app, window, truncated}. Later steps in the same run take a ref as their locator: click {locator: "@e3"}, setValue {locator: "@e4", value: "invoice"}, readValue. A ref is a full locator under the hood, so it finds the element again by role, title and place, never by position. If the window changed and the element is gone, the step fails saying so; read the window again.

An app that describes nothing (a game, a canvas, an Electron app before accessibility is on) is read by text recognition instead, with a first line saying there are no refs. A browser tab Switchboard's extension is connected to is read with Snapshot the page for an agent instead, which knows more than the accessibility tree.

Parameters #

Parameter	Type	Values	What it does
`app`	text		The app, by bundle id. Defaults to the front app.
`fullText`	yes / no
`into`	text		The variable that receives the result.
`markdown`	yes / no		Also return `markdown`, the outline in a fenced block under a heading with the date, for pasting. The result also carries `name` and `lines`.
`maxChars`	number		, fullText.
`mode`	one of	`reader` `interactive` `full`	`reader`, `interactive` or `full`.
`sample`	any value
`window`	text		A piece of the window's title.

Every step also accepts label, onFailure, timeout and dryRunSafe.

Gotchas #

Only rows on screen are in the outline; a long list says how many more there are. Read the rows of an app's list with scroll reads them all.
Refs last until the next app.snapshot in the run. A dry run skips this step unless you mark it Safe in dry run.

Examples #

Details #

Parameters #

Gotchas #

See also #