read_screen
Returns the front window as a short outline of text and controls, each control with a ref ui_act can use, and the app's window list.
Details #
By default the window comes back as an outline: headings, text and every control, grouped by the window's regions, with a ref (@e3) on each control. Pass a ref as the locator of a ui_act step to click or fill it. Read this first; use find_elements when the window is huge. The result is {app, window, text, refs, truncated, windows}; password fields show [secure] and no value.
as: "ocr" gives text recognition instead: each line of text with its frame, which works for any app. An app that describes nothing to Accessibility answers that way on its own, and says so. It returns text, never a picture, and only for the session's own apps: the whole screen is not on offer. Text recognition needs the Screen Recording permission. Read-only.