Skip to content

fix(windows): capture target windows and route posted clicks to child controls - #147

Open
zhx8702 wants to merge 1 commit into
iFurySt:mainfrom
zhx8702:fix/windows-target-capture-child-clicks
Open

zhx8702 wants to merge 1 commit into
iFurySt:mainfrom
zhx8702:fix/windows-target-capture-child-clicks

Conversation

@zhx8702

@zhx8702 zhx8702 commented Nov 3, 2026

Copy link
Copy Markdown

Problem and fix

On Windows, covering a target window with another application can make get_app_state return the covering application's pixels: capture currently crops a desktop rectangle. Coordinate clicks are also posted to the top-level HWND, so a nested control may receive no mouse messages even though the tool reports success.

Capture the selected HWND with PrintWindow(PW_RENDERFULLCONTENT) and retain full-window physical-pixel coordinates. Validate geometry before and after capture and omit the image when capture fails. Resolve posted clicks within the target's native child hierarchy, convert to child-local coordinates, and check enabled state, process ownership, current window/element geometry and message delivery. Attempt mouse release when down delivery fails or is interrupted. UI Automation-first actions and the global input opt-in are preserved.

Add a disposable fixture, native MCP regression, production-function fault tests and Windows CI coverage. Correct the existing observation harness for PowerShell 5.1 array/UTF-8 behavior and localized UIA role names. Update documentation, history and release notes; no version bump is included.

Validation

  • Reproduced both defects with v1.1.3 at 192 DPI (200%): the blue target's screenshot contains the pink occluder, and its nested button receives no mouse-up after a successful tool call.
  • Patched native MCP regression passed: target pixels under occlusion; nested button mouse delivery; standard button Click via auto; right-click and panel counters; refusal of disabled controls, invalid points and stale window/element geometry; minimized-image omission.
  • The complete existing WPF/native MCP/CLI/Worker JS observation smoke passed: selective channels, stale-index rejection and AX recovery, real Unicode writes, and 1x1/minimized/cloaked image omission and recovery.
  • The initial apparent WPF image failure was a harness bug: one image was present, but a scalar PSCustomObject's Count was null under PowerShell 5.1. Preserving the array resolves it. Stable fixture-name matching and explicit UTF-8 JSON-RPC input resolve the subsequent locale/encoding assumptions. Regression tests cover image counts and localized roles.
  • Windows Go tests and vet, PowerShell observation/input/posted-click/tree-budget function tests, make check-docs and git diff --check passed locally. No GitHub CI result is claimed for this local candidate.

Reproduce from the repository root on an interactive Windows desktop: build the Windows executable, then run scripts/tests/windows-target-smoke.ps1 -NativePath <executable>. Use v1.1.3 with -ExpectOriginal to reproduce the original defects. Add -EvidencePath <output.png> to save the fixture-only crop. Full commands and coverage are documented in docs/windows-targeting.md.

Original v1.1.3 and patched native MCP captures of an occluded test window

Both panels are actual native MCP captures from the same fixture setup. Identical client-area crops exclude window chrome and desktop edge pixels; captured content is unaltered. The image demonstrates capture behavior; fixture counters separately verify click delivery and acceptance by a standard button.

Scope

These are capture-source and input-target defects, not defects specific to 200% scaling. Native validation here covers the tested WinForms and WPF fixtures at 200%; 100%, 150%, mixed-DPI monitor transitions and other toolkits were not independently tested.

PrintWindow depends on the renderer and may return blank or incomplete content for unsupported surfaces. Successful PostMessage delivery proves that messages were queued, not that every custom or owner-drawn control accepted the action. Foreground/cursor readings are diagnostic observations; they do not guarantee that arbitrary applications cannot change focus.

The fix does not silently activate the target or move the physical pointer as a workaround. macOS and Linux runtimes are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant