Repository navigation
fix(macos): hit-test click candidates in window points - #126
Merged
Merged
Conversation
hitTestElement converted its point as a screenshot pixel, but both callers (clickCandidates and the nearby hit-test fallback for element-index clicks) pass window points. When the screenshot is not 1:1 with the window the AX hit test probed the wrong spot: an auto click on a table row pressed the column header above it, and element-index clicks after include_screenshot:false threw the coordinate-input screenshot error. Use windowPointToGlobalPoint.
…review # Conflicts: # docs/releases/feature-release-notes.md
…review # Conflicts: # docs/releases/feature-release-notes.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and fix
AX hit testing receives window-relative points from both click callers. Treating them as screenshot pixels applied screenshot scaling a second time, so Retina/capped screenshots could target another control; AX-only indexed clicks could incorrectly require a screenshot.
Convert these points directly with the window's screen origin. Screenshot-coordinate input still converts once at its entry point. Merged current main (
a7fdec1) and retained the contributor's fix. Added opt-in hit-test diagnostics and a reproducible real AppKit/native MCP smoke.Validation
git diff --checkpass.Reproduce:
node scripts/tests/macos-hit-test-smoke.mjs <signed-ocu-executable>with existing Accessibility/Screen Recording permissions and runtime admission on an unlocked Retina desktop.Scope
One physical Retina display was tested. General background mouse delivery to actionless custom views is not established by this test. The independent descendant-selection bug in #127 remains separate; this change fixes coordinate conversion only. Windows and Linux do not use this Swift AX path.