An empty accessibility tree offscreen: registering control frames instead

TL;DR

• The tutorial recorder needs to know where the Scan button and the tab icons are, so a scripted cursor can move to them. The accessibility tree of the offscreen window had nothing useful in it, so buttons could not be found by label.
• The fix is a 19 line file, DemoTargets.swift: a dictionary of named frames and a .demoTarget("name") modifier that fills it, only while the sample data mode is on. 32 call sites in 18 files name the controls the clips touch.
• An in process check on 4 November 2026 found the same near empty tree for a window on screen as for one at x = -30000. Position was not the variable, and the cause behind it is stated as a hypothesis, not as a finding.

The problem: finding a button by name

The recorder in the NotchStar tests plays a script against the real interface: move the cursor to the Clipboard tab, click, type “meet” into the search field, move to the first row. For that it needs a point on screen for each of those controls.

Hard coded coordinates work until a label gets wider or the panel changes size, and then every point after it is wrong. Looking the control up by its name is what testing tools do, and the platform’s way to do that is the accessibility tree. NotchStar’s own end to end UI tests do exactly that for the real app. This helper from NotchStarUITests.swift finds any element in the notch whose label or value contains a string:

private func shown(_ text: String) -> XCUIElement {
    notch.descendants(matching: .any)
        .matching(NSPredicate(format: "label CONTAINS[c] %@ OR value CONTAINS[c] %@", text, text)).firstMatch
}

Those tests launch the actual app with --ui-testing, use the real pointer, and need the Accessibility permission for the test runner in System Settings. They work on a visible app that an accessibility client is attached to.

The recorder is a different animal. It renders the interface inside the test process, in a panel placed at x = -30000, y = -30000 so it never appears on a display, and it takes the frames itself. The first attempt was to ask that window for its buttons by label, and the notes from the time record the result in one line: the accessibility tree was empty for an offscreen window, so buttons could not be found by label.

What an in process check shows

The line in the notes says “offscreen”. A small experiment was run on 4 November 2026 to see whether the position was the cause. It is a command line program compiled with swiftc on macOS 27.0.1 with Swift 6.4. It hosts a SwiftUI Button("Scan") with an accessibility label, and a Text, in an NSHostingView inside a borderless panel, waits half a second, and walks accessibilityChildren() from the hosting view:

func dump(_ e: Any, depth: Int = 0, into out: inout [String]) {
    guard let o = e as? NSObject else { return }
    let role = (o.value(forKey: "accessibilityRole") as? String) ?? "?"
    let label = (o.value(forKey: "accessibilityLabel") as? String) ?? ""
    out.append(String(repeating: " ", count: depth) + role + " \"" + label + "\"")
    if let kids = (o as? NSAccessibilityProtocol)?.accessibilityChildren() {
        for k in kids { dump(k, depth: depth + 1, into: &out) }
    }
}

The output for the panel on screen at x = 100, and then at x = -30000:

origin x = 100.0 nodes under the hosting view: 1
    AXGroup ""
origin x = -30000.0 nodes under the hosting view: 1
    AXGroup ""
AX trusted for this process: false

Both windows give a single group and no button, no label, no text. So the emptiness was not caused by the window being offscreen. The Scan button was missing from the on screen window too, as seen from inside the process, with no assistive client attached.

A likely explanation, which this check does not prove, is that SwiftUI creates its accessibility elements lazily, when something that uses accessibility (VoiceOver, an XCUITest runner, Accessibility Inspector) is connected. That would also explain why the real app shows a full tree to the UI tests and the recorder’s hosting view shows none. It was not tested with a client attached, and the program did not trust itself for accessibility, so treat the mechanism as open.

Options that were not taken

Option	Why not
Hard coded points	Break when any label, font or panel size changes. A script of 106 awaited steps would need maintenance on each one
Use the real accessibility API against the recorder window	Needs the Accessibility permission, which the recorder avoids on purpose, and the tree for this window was empty
Find views by walking the NSView hierarchy	The hierarchy gives no stable names to look a control up by
Let each control report where it is	Needs a small API, but it uses only things the app already owns

The last option is the one that shipped. The recorder is also designed to need no Screen Recording permission, so an approach that asked for Accessibility would have moved the permission problem instead of removing it.

The registry

Here is the whole file, Sources/NotchStar/Demo/DemoTargets.swift:

@MainActor
enum DemoTargets {
    static var frames: [String: CGRect] = [:]
}

extension View {
    @ViewBuilder func demoTarget(_ id: String?) -> some View {
        if DemoMode.isOn, let id {
            onGeometryChange(for: CGRect.self) { $0.frame(in: .global) } action: { DemoTargets.frames[id] = $0 }
        } else {
            self
        }
    }
}

onGeometryChange calls the action when the control’s frame changes, with the frame in global coordinates, which for the hosting view in the recorder are its own top left coordinates. The recorder resets the dictionary when it starts (DemoTargets.frames = [:]) and then looks names up:

private func resolve(_ target: String) -> CGRect? {
    DemoTargets.frames[target] ?? DemoTargets.frames.filter { $0.key.hasPrefix(target) }.sorted { $0.key < $1.key }.first?.value
}

An exact name wins; otherwise the first key, in alphabetical order, that starts with the name. That is how a script clicks clip.row.Meeting or simply moves to tab. plus an id, and why names follow a pattern: tab.clipboard, clip.search, disk.scan, timers.toggle, reminder.Send invoice, shelf.chip.<file name>.

The names in use follow the tool they belong to, and several are built from data:

Name pattern	Where it comes from	Used for
tab.<tool id>	NotchRootView.swift	Switching tools with rec.tab("clipboard")
clip.search, clip.row.<first word>	ClipboardTool.swift	Typing a query, moving over a history row
disk.scan, disk.action	DiskOverviewView.swift	The Scan button and the action button
timers.toggle, timers.mode	TimersTool.swift	Start and stop, and the Timer or Stopwatch switch
shelf.zone, shelf.chip.<file name>	FileShelfTool.swift	The drop area and each file on the shelf

A row name is built from the first word of its text. clip.row.Meeting is the clipboard row that starts with “Meeting”, which is the sample text “Meeting moved to Thursday at 3 pm”. Two rows that start with the same word would share a name, which is the ambiguity listed under the problems below.

Two guards make scripts forgiving. A control may not be laid out right after a tab switch, so move(to:) polls for up to two seconds:

var waited = 0.0
while resolve(target) == nil, waited < 2 { await run(0.1); waited += 0.1 }

And if a name never appears, the script stops at once with a message that lists everything known, sorted: unknown demo target ...; known: [...]. A typo costs seconds and not a mystery clip.

From a name to a click

A frame is only useful if the script can act on it. point(_:) returns the midpoint of the registered frame, move(to:) glides the drawn cursor there, and click() sends a synthetic mouse event straight to the panel:

private func send(_ type: NSEvent.EventType) {
    guard let c = cursor else { return }
    let loc = host.convert(c, to: nil)
    guard let e = NSEvent.mouseEvent(with: type, location: loc, modifierFlags: [], timestamp: ProcessInfo.processInfo.systemUptime,
                                     windowNumber: panel.windowNumber, context: nil, eventNumber: 0, clickCount: 1,
                                     pressure: type == .leftMouseUp ? 0 : 1) else { return }
    panel.sendEvent(e)
}

The event goes to panel.sendEvent, not through the window server, so no pointer moves on the user’s Mac and no permission is involved. Because the frame came from the same coordinate space as the cursor, the same number is used to draw the cursor in the video and to hit the control.

One more habit follows from that. tab(_:) clicks a tab and then checks that model.selectedToolID really changed, retrying up to three times. A registered frame says where a control was a moment ago, and a click on a busy Mac can still be dropped. The script verifies state, not just position.

The shipping app pays nothing

The comment above the registry says it registers nothing unless the sample data mode is on, and the code backs that. When DemoMode.isOn is false the modifier returns self, so no geometry observer is installed. The mode is switched on by a launch argument, an environment variable or in process by the recorder and the render tests. Ordinary users do not have it on.

A search on 4 November 2026 counted 32 uses of .demoTarget( across 18 files in Sources. Some are generated from data, such as "tab.\(tool.id)" in NotchRootView.swift and "reminder.\(reminder.title)" in the calendar tool.

Accessibility is still its own thing

A registered target is not an accessibility element. The tab buttons carry both, side by side in the code:

.accessibilityLabel(tool.title)
.accessibilityAddTraits(selected ? .isSelected : [])
.demoTarget("tab.\(tool.id)")

The first two lines are what VoiceOver and the UI tests read. The third is for the recorder. The registry does not replace them, and nothing about it tests assistive technology. The real accessibility behaviour is checked where it should be: NotchPanel sets setAccessibilityIdentifier("notch") so the UI tests can find the window, and the UI tests read the tree of the running app.

Numbers and method

Measure	Value	Method
Lines in DemoTargets.swift	19	wc -l, 4 November 2026
.demoTarget( call sites	32 in 18 files	grep -rho "\.demoTarget(" Sources | wc -l, same day
Clip tests that use targets	13	grep -c "func test" Tests/NotchStarTests/Video/ClipScripts.swift
Awaited recorder steps in the scripts	106	grep -n "await rec\." ... | wc -l
Longest wait for a control to appear	2 s	the waited < 2 loop in ClipRecorder.move
Nodes found under the hosting view, on screen and offscreen	1 and 1	in process program, 4 November 2026
Tests that exercise DemoTargets directly	0	grep -rn "DemoTargets|demoTarget" Tests, only Video/ matches

What is still bad

• Entries are never removed. A control that disappears keeps its last frame in the dictionary until the next recorder starts, so a script could move to a place where nothing is anymore.
• Prefix matching picks the alphabetically first key. Two controls that share a prefix are ambiguous, and the script gets one of them silently.
• Nothing tests the registry itself. It is exercised by the clips, and a broken frame shows up as a cursor in the wrong place in a video.
• The cause of the empty tree is a hypothesis. The check above rules out position as the reason and does not identify the real one.
• Names are strings. Rename a control and a clip script fails at run time, not at compile time.

Try it yourself

1. In a throwaway SwiftUI project, host a view with a Button in an NSHostingView inside a panel.
2. Walk accessibilityChildren() from the hosting view, as in the dump above, with the panel on screen. Note what you find with no assistive client attached.
3. Run the same walk with VoiceOver on, or with Accessibility Inspector attached to the process, and compare.
4. Add a registry modifier that stores onGeometryChange frames by name behind a flag, and look a name up.
5. Point a synthetic mouse event at the stored frame’s midpoint and check the button reacts.

Cheat sheet

Question	Answer
Where do frames come from	onGeometryChange with frame(in: .global)
When is the registry active	Only when the sample data mode is on
How are controls named	A dotted string, such as tab.clipboard or disk.scan
How are late controls handled	Poll up to 2 s, then stop with the list of known names
What does it not do	Replace accessibility labels or test assistive technology
What to watch for	Stale entries, prefix ambiguity, string typos

Close

NotchStar is a Mac app that puts ten tools in the notch, and these target names are how its tutorial clips click through them. If you want to try the app, the details are at notchstar.com.