中文说明. The current public release is 0.4.2, a public-testing prerelease for macOS, protocol 8. China and international WorkBuddy use the same OpenGUI package and version. Custom MCP installation does not require marketplace publication; real-device acceptance remains a separate check.
See the root installation steps for edition-specific commands, or the GitHub installation flow (Chinese) to install from a WorkBuddy chat. OpenGUI has a separate SMS account that currently requires a mainland-China phone number.
OpenGUI opens the device display beside the current chat using WorkBuddy built-in present_files with the workbenchUrl and current working directory. A visible decoded H.264 frame (Android) or interval JPEG image (iOS simulator) must reach the backend before the first observation or action. An opening request or screenshot preview is insufficient. Display transport is read-only and local; model observations still use explicit screenshots.
New tasks default to 1000 phone operations and plugin inference calls, with initial ceilings configurable from 1 to 10000. Recovery preserves older saved limits. The 2/3-operation QA caps are test conditions, not product requirements. For explicit user ceilings, pass executionBudget: {operationLimit?, inferenceLimit?} (1–10000) to opengui_open_session before operations. Phone observations and actions share operationLimit; inferenceLimit counts only plugin-owned calls. Verify the returned hard limits before dispatch. Recognized Arabic-numeral total-operation clauses in the saved goal/criteria also lower the ceiling; other wording must use the structured field. Initial limits persist through recovery and cannot be raised by a model. The workbench and report show saved limits and actual use; additional allowance requires the existing human continuation form.
For a pre-submit endpoint, pass stopBeforeSubmit: true when opening control. Recognized native prompt wording, frozen goals/criteria and begun test endpoints also latch it. The persisted restriction rejects marked submit/send/publish/purchase/delete and Enter before dispatch, requires explicit externalSideEffect on every tap/swipe, and uses clipboard-only text input. Recovery cannot remove it. Record final submission as not checked. Use none only for preparation/navigation identified on the current image; this declaration is not independent visual verification. Ambiguous controls require human handling.
- Call
opengui_open_viewerwith the goal and stopping condition. A signed-in account's available last selected phone is reused for a new untargeted task. An unavailable preference requires explicit selection even if another phone is connected. Without a preference, the sole eligible phone is selected automatically; zero or multiple eligible phones require the user's graphical selection. - Open its workbenchUrl in the host's right browser, and call
opengui_viewer_statuswithwaitMs: 600000to await Start and the first frame. Renew expired waits silently, retaining the same task. Never read or screenshot the workbench webpage to monitor it. A first-display timeout is terminal for that logical task, including repeated opens; report the blocker. - For pure viewing, finish. For control, call
opengui_open_sessionwith the sameviewerIdand devices, then observe and act one screenshot at a time. - Close/cancel control when finished. Watching continues. Hiding pauses video; closing the last page releases its source within the cleanup budget. Reopening watching does not restart the task.
After first display authorization, video failure or page closure does not cancel healthy screenshot control. Physical disconnection or lost device authorization pauses execution, aborts in-flight inference/input and invalidates observations. The workbench preserves the original task, device, TODOs, saved drafts and evidence. A returned connection alone does not resume execution: the human must recheck the original device, followed by a fresh observation and reconciliation of the last result. Rechecking during manual takeover does not hand control back. The original lease and budgets still apply; ended ownership needs archived recovery rather than old authority. Never switch phones or replay uncertain actions. Each task owns control exclusively. Viewer credentials never authorize phone actions. Independent host processes cannot coordinate external controllers, so do not control the same phone through another host concurrently.
UI language (0.4.2): Chinese/English switching, following WorkBuddy’s display language, and an improved emulator picker are included. The default is Follow WorkBuddy, with manual 中文 / English overrides and a browser-language fallback when the host setting is unavailable. Only interface labels change; task descriptions, TODO content, phone content, review drafts and recorded results retain their original text.
An exhausted blocked run can receive a human-confirmed finite continuation budget in its original workbench: 1–100 additional phone operations and plugin inference calls. The grant starts no work, retains the task ID and earlier result, and rejects stale/duplicate revisions. Copy its continuation prompt into the current chat, recover the same resumeTaskId, and establish fresh original-phone display and observation. This does not extend comment quantity/deadline endpoints, change task scope/model, clear unknown sends, or resume completed/cancelled/stopped tasks. Follow-mode host model calls remain unmeasured.
The light workbench follows the supplied prototype: device controls, a proportionate screen, the existing TODO panel, tool Trace with filtering/details, and a report pane. A bare @opengui mention uses opengui_open_guide without preparing video. A requested task uses open_viewer with its original goal: an available account device preference is reused; otherwise multiple phones or an unavailable preference require graphical single-choice confirmation with manufacturer/model, OS version, serial suffix and connection type. Detection and confirmation never grant phone control; a new decoded frame is required. Waiting host tasks survive a normal final stop and a follow-up prompt, but the page cannot wake an ended host turn. It provides two copyable scenario templates and USB authorization/troubleshooting guidance. Normal MCP tasks bind one Android phone, Android emulator or supported macOS iOS simulator; lower-level legacy display primitives retain multi-device compatibility. Device switching requires a new task. Physical iOS devices and cloud phones are not implemented. Execution follows the current WorkBuddy conversation model. The official CoreMate service handles account sign-in; the plugin does not read an online model catalog.
Open workbenchUrl to enable takeover, 恢复控制 (resume) and comment decisions. Takeover is the only user pause: no screenshots or model calls until the person resumes control. On Android the person can operate the phone directly on the workbench picture: click/drag for touch, wheel to scroll, right click for Back, and the keyboard including input methods. Input travels over the same scrcpy connection, is accepted only during takeover for the bound device, and stops as soon as control resumes. The separate capability lives in the URL fragment and the current tab session storage; it is never included in HTTP status. The plain url remains read-only. Board controls cannot dispatch arbitrary phone inputs. Pause aborts in-flight work; an already issued mutation may remain unknown. Handback/recheck invalidates previous screenshots and requires a fresh observation before acting. Host Stop hooks yield for human review/takeover instead of forcing autonomous continuation; an explicit stop still cancels.
Comment drafts use opengui_review_comment and human approval/edit/skip in the workbench. Input must match the approved draft. opengui_comment_input copies the identified focused comment field without exposing unrelated clipboard text. Different originals require a separate human keep/replace decision; replacement and sending recheck the live input. Identical text is never appended. Native copy requires a fully selectable/copyable Android field (API 24+); unsupported/custom fields or nontext clipboards require handoff. Destination/focus/selection and real platform compatibility still require visual and device acceptance. Sending records intent once; only a latest-image verification via opengui_verify_comment counts as sent. Target/account interpretation and business verification remain the executor model's responsibility. The private local archive blocks previously sent and unresolved submissions for the same signed-in principal and stable platform account/target; unreadable history blocks sending.
Export Markdown, genuine DOCX, PDF, or a ZIP containing the report and archived screenshot evidence. Direct PDF embeds bundled Chinese/emoji fonts and local screenshots; browser Print/Save as PDF remains an additional option. Completion/cancellation automatically archives PDF and Word and returns actual paths, or an explicit export error. The current WorkBuddy-following mode has no host model timing; historical reports may retain timing from earlier model configurations. Tool inputs are omitted from Trace. Reports and evidence are automatically stored under the private local reports directory; the 32 MB memory cache does not evict archived screenshots. Standalone Markdown references the companion evidence directory.
Execution follows WorkBuddy and waits on saved state events without inspecting the workbench webpage. Keep one owned MCP status wait pending for Start, reviews or human handback; renew timeouts silently. Notifications cannot wake an ended host turn. If the connection recycles, recover the same frozen task with a fresh owned session and screenshot.
Download the installer and run it in macOS Terminal, restart WorkBuddy once, then authorize the Skill and connector.
Important
Restart WorkBuddy once after installation. Finish other tasks, fully quit with ⌘Q, and reopen WorkBuddy before authorizing OpenGUI. Closing its window is not enough.
The complete steps and download link are in the root README. See installation verification and troubleshooting for details; the archive command below is for maintenance and local verification.
Use the supplied installer and matching archive with adjacent SHA-256 sidecars:
bash scripts/install-macos.command --app "/Applications/WorkBuddy AI.app" --download-source official --archive /absolute/path/opengui-mcp-0.4.2.tgzThe installer prepares private Node, verifies the package and scrcpy resources, and only then changes this host's configuration. After installation, restart WorkBuddy once to load the new MCP configuration, review external Hook changes in /hooks, and confirm the installed Skill in /skills.
Complete old phone tasks and close old displays before upgrading. If an old broker remains, disable the old OpenGUI MCP and wait for its idle exit before retrying. No migration force-kills an old runtime. Repeated installation reuses verified caches; download failure reports its stage and leaves previous configuration available. Keep the old installer/archive and recovery record to reinstall the old version. The installer reports configuration, host loading and real-viewer acceptance separately.
WorkBuddy 5.5.3 domestic and overseas configuration discovery, version-aware running-host preflight, per-configuration receipts, and native Hooks are retained. Use --check or --app /absolute/WorkBuddy.app for explicit preflight. Legacy opengui_start and native mirror tools are compatibility-only, on explicit request. They never substitute for browser first-frame authorization.
Automatic discovery accepts com.workbuddy.workbuddy, com.workbuddy.workbuddy-ai, and legacy com.tencent.workbuddy.* identities. It prefers WorkBuddy (including legacy bundles) over WorkBuddy AI; multiple bundles in the preferred group require --app. An explicit --app overrides this preference. Configuration paths come from the selected application's product metadata, including .workbuddy and .workbuddy-ai.
The root README’s China command selects npmmirror with official fallback for Node/npm; its international command selects official sources. The public installer defaults to npmmirror when --download-source is omitted. This selection applies only to the current installation. Video archives remain separate GitHub downloads; --video-mirror https://host/archive-directory optionally selects an identical archive mirror with size/checksum verification and official fallback. See download sources; older packages cannot honor the new video option.
The start page's 高级选项 offers 发内容前需要审核 (off by default). When checked, posting any text (posts, comments, replies, messages) requires the person's approval, and the runtime publishes only when the field reads back exactly the approved text; likes, follows and favorites need no review. See docs/content-review.zh-CN.md.
Without a phone, the confirmation page and the device dialog offer an emulator entry. Existing Android Studio emulators can be started directly; otherwise, after the person accepts the Android SDK license, the plugin downloads the official emulator (~0.4 GB) and the Android 14 Google APIs system image (~1.6 GB; arm64-v8a on Apple silicon, x86_64 on Intel Macs and 64-bit Windows). Archives come from Google or the public Tencent Cloud mirror (a 1 MB speed probe picks the faster) and are verified against Google's published sizes and SHA-1 checksums pinned in the code. The system curl downloads with resume and proxy environment support; the system tar extracts. Everything lives in the plugin's private directory (~/.workbuddy/opengui/android); existing SDKs, ~/.android and environment variables are untouched. The emulator boots headless and appears in the device list when boot completes; a failed start reports the emulator's own reason from android/emulator.log. Windows needs the Windows Hypervisor Platform enabled. About 8 GB of free space is required.
On macOS with Xcode installed, booted iOS simulators appear in the same single-device picker. First selection prepares the checksum-pinned AXe 1.8.0 driver in a private cache; its packaged frameworks and notices are retained. An optional absolute OPENGUI_AXE_PATH must identify the same version. The adapter shares the existing freshness, frozen-device, execution-budget and no-progress guards. It supports screenshot-based taps, swipes, app launch, Home and single-line Unicode paste, with a timestamped one-second screenshot display. Back/AppSwitch/Enter, exact comment readback/replacement, APK installation and physical iOS devices are unavailable. Required iOS permissions and account/service checks remain unknown and block dependent tasks; foreground application identity is explicitly unknown rather than inferred from the last launch. Local Xcode 26.2/iOS 26.3 fixture verification does not establish compatibility with every runtime or real WorkBuddy behavior.
Each plugin independently packages scrcpy 4.1 transport and browser H.264 decoding: maximum dimension 960, 30 fps, 2 Mbps, no audio and no video control channel. Up to four per-device sources are shared within this host. Clients use bounded queues and recover on configuration/key frames. There is no shared background service or dependency on DSH.
The local HTTP/WebSocket server checks loopback Host, Origin and private viewer credentials. Do not share Viewer URLs. Phone screenshots sent to the model follow host data policies; video is not sent frame by frame. Session locks and observation authority are never restored after restart. See VIDEO-NOTICE.md and LICENSE for provenance.
npm ci
npm run pack:release
npm run smoke:packedOn macOS with WorkBuddy installed, npm run test:host -- /Applications/WorkBuddy.app checks the application's actual bundled headless engine against the current MCP tool directory using isolated configuration. It refuses all tool execution and sends no model prompt. This proves directory/schema loading, while desktop activation, image interpretation, Hooks and phone tasks still need separate acceptance.
The browser test additionally needs development-only agent-browser and ffmpeg. It validates actual H.264 decode and canvas changes, first-frame receipts, continued playback after completion, and subscriber cleanup. End users do not need those tools. See the candidate acceptance report for separate host, device, performance, installation and release evidence. Candidate packaging alone does not pass those gates.
The first workbench tab contains outcome-based TODO steps, structured checks and comment review cards. The panel reuses DeepSeek Harness source under MIT. It supports pending, in_progress, awaiting_user, completed, failed and skipped with separate counts and reasons. Review/failure/skip states belong to the runtime and cannot be forged through todo_write. Test verdicts differ from execution completion counts; ending or cancelling never completes unfinished steps.
Task node synchronization: initialize pending TODOs and use the runtime-assigned stable stepId in existing observe/act calls. Keep the ID within a step; select progress.nextStepId only after verifying the prior image. Act reuses observationId; advancing observe also takes evidenceObservationId. The board displays exactly the returned progress.message, even while collapsed. Paused control retains unfinished nodes with an explicit pause message. No extra screenshot or TODO call is needed solely to update progress.
Completed/active steps keep their content, ID, status and order; pending steps may be renamed/reordered with their existing IDs, added without an ID, or removed. Legacy taskNodeIndex is compatible only while positions remain unchanged; reordering invalidates index-only calls. Step IDs survive through the archive, and opengui_history exposes saved tasks. Local device IDs persist across processes using a private per-installation identity key. The resumeTaskId entry resolves the exact original ADB serial; legacy random IDs are reconciled against that serial without replacing the phone by model or suffix. Recovery preserves goals and operation budgets, requires a new decoded display and fresh session, and never restores old observations or unsubmitted approvals. Missing or unauthorized original phones block recovery. The model still judges semantic completion from the images.
The default workbench home shows an editable task description, model, device and advanced options instead of an empty device picker. A fixed @opengui label sits above the input; requests from the chat are prefilled. The user signs in if needed and clicks 开始执行 before the device is bound. Control still requires a verified first frame. While choosing, a small live view of the selected Android device is shown beside the list (display only; it falls back to periodic screenshots where video decoding is unavailable and closes when the task starts).
The report's 新建任务 button copies @opengui for a new WorkBuddy request and preserves the previous report. An empty home keeps one host event wait pending until the person starts; execution then uses WorkBuddy's model after first-display verification.
The plugin connects to the official CoreMate account service (https://cm2backend.dmyh.tech, defined in src/service-config.ts) for SMS login and automatic registration through /api/user-auth/*. Builds bundle the service URL; OPENGUI_ACCOUNT_SERVICE_URL supports another service at build time or runtime. The service receives the phone number, SMS code and account session. Execution only follows the current WorkBuddy conversation model: the plugin does not fetch an online model catalog or send task prompts/screenshots to the account service. Prompts and screenshots are handled by WorkBuddy's normal model/tool loop. Legacy model preferences are cleared while account-scoped device preferences remain. Provider keys are not stored by the plugin.
SMS sign-in requires the official account service to be reachable. Local fixture tests do not establish production availability or real WorkBuddy/device acceptance. Device preferences and task history are scoped to the service and signed-in identity. The device preference stores only an opaque local ID in the private account file, survives restart/logout, and never overrides explicit targets, frozen tasks or recovery. Anonymous device choices are not persisted. Unavailable/unauthorized/busy/incompatible preferences require a human choice; a failed preference save is shown while the successful task binding remains valid. Cross-machine synchronization is not implemented.
Use opengui_test_case to define/begin/record/read bounded checks on existing stepIds. A passed/failed verdict requires a known original expectation, executed steps and the latest valid current-task screenshot on that step. Failed checks require the actual observed page and derive a linked defect ID; unknown expectations remain unverified. Dependencies must pass before dependent execution; omitted checks preserve reasons without invented steps. Retest copies a same-account archived case into a new run, preserving original expectations, inputs, steps and stopping conditions, and reports changed conditions without modifying old results/reports or claiming a universal fix. Reports expose each check's actual result and evidence; the runtime records the model's comparison rather than independently interpreting images.
Sensitive password, OTP, login, payment or security prompts can request human handling through opengui_handoff. The workbench displays the redacted reason and enters manual control immediately, stopping new inference and phone dispatch. Only the user can hand control back; fresh observation is required before actions. Handoff history is archived, and an unresolved restored request needs a new human decision. Recognizing a sensitive screen still depends on the model; fresh observation does not prove that verification or payment succeeded.
Comment runs can declare a verified send target and/or wall-clock duration at session opening. The runtime stops new phone/model work at that limit, preserves unfinished steps, and reserves uncertain submissions without counting them as success. The workbench saves human edits before approval, shows generated/human/observed phone draft versions, rejects stale approval, and records the exact approved content version. Restoring a run retains its original budget; no silent clock reset or additional send budget is allowed.
Explicit Android input-injection permission denials pause control and open device security guidance. Xiaomi/Redmi/POCO metadata selects the additional USB debugging security-setting instructions; other devices get conditional vendor guidance. The user's workbench recheck invalidates old observations but does not claim input permission is granted. A later input receipt and result image can resolve the saved diagnosis; failed actions are never automatically replayed. Sanitized diagnostic state and handling times are archived and reported. Generic errors and unchanged images are not treated as proof of missing permission. The setting guidance follows the official scrcpy prerequisites.
Declare the real target package and task-required version, permission, account or service prerequisites through open_session.environment or opengui_environment before phone actions. Read-only checks project the current Android user's installation/version/grants and SDK; required failed or unknown checks block actions, test beginnings and completed outcomes. Account/service claims require the latest target-app screenshot and are labeled model observations, not independent assertions. Only launching the original installed compatible app or waiting is allowed to obtain those screenshots. Human handoff, reconnect and restoration revoke readiness; recheck before continuing. The workbench and archived reports retain explicit unknown conditions. Undeclared conditions are not claimed checked. USB MTP is a diagnostic hint for working ADB. These checks do not grant permissions or bypass security settings. See the Android ADB reference for the underlying commands.
Connection help provides Check current connection: a read-only inspection of fresh ADB USB rows and OS USB interface descriptors (macOS I/O Registry, Windows present PnP properties, Linux sysfs). It reports authorized, unauthorized and unavailable USB phones separately from USB debug/media interface counts. Network devices and simulators are excluded from USB phone counts. Failed reads stay unknown; interface presence does not identify the requested phone, prove authorization or diagnose a charge-only cable. Results contain counts and guidance, not hardware names or serials, and cannot bind a device or resume a task. Windows/Linux adapters require native-platform acceptance; the current Mac check does not substitute for real Android USB/cable validation.
For an explicitly requested local APK test, use opengui_prepare_apk to inspect the user's absolute file path before actions. A private staged copy freezes its SHA-256, package, version and minimum SDK to the original phone and task. Existing-app updates show the replacement/data-migration warning and require the user's workbench confirmation; changed installed versions revoke that confirmation. Installation uses the current Android user and retains application data. Test-only APKs require explicit user authorization. The installation intent is archived before dispatch; failed, interrupted or uncertain attempts are not automatically replayed. Successful installation still requires fresh environment checks and screenshots. Only standalone APKs up to 512 MiB are supported; remote downloads, split bundles, downgrades, uninstalling and clearing data are not implemented. Local fixture/browser checks do not establish real APK signature, ABI, vendor or device compatibility.
First use: send @opengui, sign in and select a model/device in the task home, then fill a task and click Start. Try it: open phone Settings, return to the home screen, verify it and finish. Filling an example does not start execution. App tests need an app/page, test data, expected result and stopping point; comment tasks also need the account/target, scope and count/time bound.
The home-page comment template enables explicit run limits. Fill in the task and a verified-send target or duration (at least one); the run stops at the first reached limit, including time spent awaiting review. Human-confirmed limits are retained by the runtime and cannot be widened or removed by a later host-model call. Unfilled template placeholders cannot start.