fix(set_value): verify actual values across desktop platforms by wicgee · Pull Request #111 · iFurySt/open-computer-use

wicgee · 2026-10-21T04:19:37Z

set_value could report success when a control accepted the API call but ignored or changed the value. This PR now verifies the actual value on macOS, Windows and Linux, including when full action read-back is disabled.

Text comparison is exact: whitespace, case, leading zeros and Unicode representation are preserved. Numeric/boolean equivalence is used only when the native attribute has that type. Empty strings clear fields; Linux/Windows no longer trim value. Verification polls for up to 0.6 seconds using a monotonic clock, never rewrites or falls back to another input method, and reports an error when the result cannot be verified. macOS checks the session before reads and before returning success. Error messages advise observing the current value before retrying.

Validation completed:

Swift: 428 tests, 9 skipped, zero failures; numeric-boundary and session-revocation tests.
Real signed AppKit/native MCP: normal, empty, no-op, ignored, normalized and delayed writes with full read-back on/off; JS errors propagate. Existing text-input regression passes. Old main incorrectly returns ok for the same ignored-write fixture.
Real GTK/AT-SPI: literal text, empty/no-op, ignored, length-limited and delayed writes, plus a native numeric slider; both read-back modes pass. Existing JS/selective-observation regression passes. Python 42 tests and Linux Go tests pass.
Windows Go and production PowerShell verification tests pass. Real WPF/UIA native MCP tests pass for literal/empty/no-op/delayed writes and ignored/normalized failures. Existing Windows JS/CLI observation, physical input and 2000-item tree regression all pass in Windows CI.
Node 54 tests, documentation and action-pinning checks pass.

Reproducible commands and behavioral limits: docs/set-value-verification.md. This builds on wicgee's original contribution and is synchronized with current main. No new configuration is required.

- 新增 SetValueReadBack 纯函数：trim / 数值 / 布尔比较，支持 CFString、CFNumber、CFBoolean 读回 - setValue 写入前记录当前值，写入后 0.6s 内轮询回读；未变化或不一致时抛出带元素索引与实际值的明确错误 - set_value 工具描述与 usage 文档说明校验行为 - 单测覆盖字符串/数字/布尔/空白/同值 no-op/不一致/不可读、AX 值转换、错误文案与 fixture Unicode 值回归

hyprcat

The core fix works. I built the head and ran set_value against an AppKit fixture with a text field that accepts the AX write and then ignores it. Main returns ok. This PR returns the new error after about 0.55 s. Writing the same value twice is still a silent success, as intended. Tests pass: 386 tests, 7 skipped, 0 failures (main: 382 / 7 / 0).

Two cases now fail even though the write landed, and I think they block the PR as it stands:

Password fields. AX never reports the value of a secure text field, so every set_value on a password field now returns an error. The fixture's own log shows the password was set. Login forms are one of the main reasons to use set_value.
Controls that normalize the value. A field with a NumberFormatter turns 1234.5 into 1,234.5, and this PR reports that as set_value failed. The same will happen with phone, date and other formatting fields. An agent that trusts the error will retry with type_text and append to a field that is already filled.

Suggestion: only fail on .unchanged, where the value still equals what it was before the write. That is the silent failure from #83. Treat .mismatched as success: the action result already re-renders the element, so the agent sees the normalized value (or add a one-line note). Skip the read-back for AXSecureTextField. Treat an element whose value can't be read as unverifiable, not as failed.

How I checked

Tests. swift test --package-path <worktree>:

main 43eb4b5: Executed 382 tests, with 7 tests skipped and 0 failures
head 91677dc: Executed 386 tests, with 7 tests skipped and 0 failures
No new compiler warnings.

Live run. AppKit fixture, in-process CLI (OPEN_COMPUTER_USE_DISABLE_APP_AGENT_PROXY=1, so the installed app agent isn't used):

element	value	main	#111	what the field holds
plain text field	`Draft` (twice)	ok	ok (1.76 s)	`Draft`
field that ignores `setAccessibilityValue`	`Draft`	ok (silent)	error "still reports "Old"" (2.30 s)	`Old`
field that uppercases	`Draft`	ok	error "reports "DRAFT""	`DRAFT`
`NSSecureTextField`	`hunter2`	ok	error "reports """	`hunter2` (from the fixture's own log)
`NumberFormatter` field	`1234.5`	ok	error "reports "1,234.5""	`1,234.5`
`NSSearchField`	`query`	ok	ok	`query`
`NSSlider`	`50`	`-25201` (unchanged)	`-25201`	—

The fixture's log for the #111 run (it polls stringValue every 200 ms):

secure= amount=1 ignoring=Old
secure=hunter2 amount=1 ignoring=Old
secure=hunter2 amount=1,234.5 ignoring=Old

hyprcat · 2026-10-22T19:58:16Z


        let cursorTarget = visualCursorTarget(for: record, snapshot: snapshot)
        moveVisualCursor(to: cursorTarget)
+        let valueBeforeWrite = setValueReadBackValue(of: element)


A secure text field never reports its value through AX, so this read-back can't succeed for one. In the fixture, set_value on an NSSecureTextField returned isError: true ("element 8 reports "" instead of "hunter2"") while the field held hunter2. On main the same call returns ok. Could we skip verification when the element's subrole is AXSecureTextField?

hyprcat · 2026-10-22T19:58:16Z

+        if setValueValuesMatch(requested, before) { return .verified }
+        return .unchanged(actual: after)
+    }
+    return .mismatched(actual: after)


.mismatched means the value did change, just not to the exact string requested. With a NumberFormatter field, 1234.5 reads back as 1,234.5, which is applied and correct, but the call errors. I'd only error on .unchanged and let the re-rendered state show the normalized value.

hyprcat · 2026-10-22T19:58:16Z

+/// parse as a number (so `15` matches `15.0`), and as booleans for the
+/// `true`/`false`/`1`/`0` literals that `CFBoolean` produces.
+func setValueReadBackResult(requested: String, before: String?, after: String?) -> SetValueReadBackResult {
+    guard let after else { return .unchanged(actual: nil) }


after == nil means the value can't be read. That is not the same as the write failing, but .unchanged(actual: nil) turns it into an error for any settable element with an unreadable AXValue, where main returned ok. Suggest returning .verified, or a separate "unverified" case that doesn't throw.

hyprcat · 2026-10-22T19:58:16Z

+returns an explicit error naming the element and the actual value instead of
+reporting a silent success. Requesting the value the element already reports is
+a no-op success.
+```


This stray ``` opens a code fence that never closes. The file's fence count goes from 34 to 35, so from here to the end of the file, prose renders as code and code renders as prose. make check-docs doesn't catch it.

hyprcat · 2026-10-22T19:58:17Z

 open-computer-use call set_value --args '{"app":"TextEdit","element_index":"1","value":"Draft"}'
 ```

+`set_value` reads the element back after the AX write and compares the result


This reference is shared across platforms, but only macOS reads the value back. apps/OpenComputerUseWindows/main.go and apps/OpenComputerUseLinux/main.go keep the old set_value. Suggest starting the paragraph with "On macOS".

hyprcat · 2026-10-22T19:58:17Z

+            setValueReadBackResult(requested: "15", before: "0", after: "15.0"),
+            .verified
+        )
+        // CFBoolean reads back as "1"; a requested "true" still matches.


The comment says CFBoolean reads back as "1", but setValueReadBackString returns "true"/"false", which lines 2097–2098 assert.

hyprcat · 2026-10-22T19:58:17Z

+    /// read-back by design. The comparison layer must still accept the exact
+    /// unicode values those smoke tests write, so the new verification cannot
+    /// regress the fixture harness.
+    func testFixtureSetValueValuesVerifyWithoutAXReadBack() {


This test never touches the fixture path. It calls the same pure function as the first test, on two Unicode strings. I'd rename it or drop it, and spend the slot on the secure-field or normalized-value case once the behaviour is decided.

hyprcat · 2026-10-22T19:58:17Z

+
+- `swift build` 成功。
+- `swift test`：386 个测试，7 个跳过，0 失败；新增 4 个用例全部通过（字符串/数字/布尔/空白/同值 no-op/不一致/不可读、AX 值转换、错误文案、fixture Unicode 值回归）。
+- 变更只在本地分支 `fix/set-value-read-back`，未 push、未开 PR。


Nit: "未 push、未开 PR" is out of date now that this is a PR.

iFurySt · 2026-11-02T15:30:40Z

LGTM. Thanks!

wicgee mentioned this pull request Oct 21, 2026

[Bug] macOS 26 real-world findings on 0.3.5: wrong-window targeting, silent set_value, docs/version mismatch, and more #83

Open

hyprcat suggested changes Oct 22, 2026

View reviewed changes

hyprcat mentioned this pull request Oct 22, 2026

feat(macos): select rows on click and add set_position #123

Open

iFurySt added 4 commits November 2, 2026 23:01

Merge remote-tracking branch 'origin/main' into codex/pr111-set-value

77b2cf6

fix(set-value): verify literal values across desktop platforms

cd4a6b3

Merge remote-tracking branch 'origin/main' into codex/pr111-set-value

fdb7489

test(set-value): cover numeric bounds and real receiver failures

f4efb46

iFurySt changed the title ~~fix(set_value): verify the write and fail loudly when a control ignores it~~ fix(set_value): verify actual values across desktop platforms Nov 2, 2026

docs: record cross-platform set-value validation

6a9601d

iFurySt merged commit 7657d49 into iFurySt:main Nov 3, 2026
2 checks passed

Navigation Menu

Repository navigation

fix(set_value): verify actual values across desktop platforms - #111

fix(set_value): verify actual values across desktop platforms#111
iFurySt merged 6 commits into
iFurySt:mainfrom
wicgee:fix/set-value-read-back

wicgee commented Oct 21, 2026 •

edited by iFurySt

Loading

Uh oh!

hyprcat left a comment

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

iFurySt commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wicgee commented Oct 21, 2026 • edited by iFurySt Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hyprcat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

iFurySt commented Nov 2, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wicgee commented Oct 21, 2026 •

edited by iFurySt

Loading