agent-device 移动端自动化测试工具 | 跨平台UI测试与调试解决方案

agent-device by callstackincubator/agent-device

1,700 周安装量

1,300 GitHub Stars

GitHub

安装命令

npx skills add https://github.com/callstackincubator/agent-device --skill agent-device

开发开发运维

🇨🇳中文介绍

使用 agent-device 进行移动端自动化

探索时使用快照引用。确定性回放时使用选择器。结构化探索性 QA 缺陷排查和报告时，请使用 ../dogfood/SKILL.md。

从这里开始（请先阅读）

将此技能用作路由器，而非完整手册。

选择一种模式：
- 正常交互流程
- 调试/崩溃流程
- 回放维护流程
运行下方的一个规范流程。
仅在遇到阻塞时查阅参考资料。

决策图

尚无目标上下文：devices -> 选择目标 -> open。
正常 UI 任务：open -> snapshot -i -> press/fill -> diff snapshot -i -> close

广告位招租

在这里展示您的产品或服务

触达数万 AI 开发者，精准高效

联系我们

1) 正常交互流程

agent-device open Settings --platform ios
agent-device snapshot -i
agent-device press @e3
agent-device diff snapshot -i
agent-device fill @e5 "test"
agent-device close

2) 调试/崩溃流程

agent-device open MyApp --platform ios
agent-device logs clear --restart
agent-device network dump 25
agent-device logs path

默认情况下日志记录是关闭的。仅在调试窗口期间启用。logs clear --restart 需要一个活跃的应用会话（先执行 open <app>）。

3) 回放维护流程

agent-device replay -u ./session.ad

4) 远程租户租约流程（HTTP JSON-RPC）

# 客户端直接指向远程守护进程 HTTP 基础 URL。
export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310
export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<token>

# 分配租约
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"runId":"run-123","tenantId":"acme","ttlMs":60000}}'

# 在租户隔离的命令执行中使用租约
agent-device \
  --tenant acme \
  --session-isolation tenant \
  --run-id run-123 \
  --lease-id <lease-id> \
  session list --json

# 心跳和释放
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"<lease-id>","ttlMs":60000}}'
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":"<lease-id>"}}'

AGENT_DEVICE_DAEMON_BASE_URL 使 CLI 跳过本地守护进程发现/启动，直接调用远程 HTTP 守护进程。
AGENT_DEVICE_DAEMON_AUTH_TOKEN 会同时发送在 JSON-RPC 请求令牌和 HTTP 认证头中。
在远程守护进程模式下，--debug 不会跟踪本地的 daemon.log；请检查远程主机上的日志。

命令框架（最小集）

agent-device devices
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
agent-device ensure-simulator --device "iPhone 16" --ios-simulator-device-set /tmp/tenant-a/simulators
agent-device ensure-simulator --device "iPhone 16" --runtime com.apple.CoreSimulator.SimRuntime.iOS-18-4 --ios-simulator-device-set /tmp/tenant-a/simulators --boot
agent-device open [app|url] [url]
agent-device open [app] --relaunch
agent-device close [app]
agent-device install <app> <path-to-binary>
agent-device reinstall <app> <path-to-binary>
agent-device session list

仅当 open 无法找到/连接到就绪的目标时，才将 boot 作为后备方案使用。对于按 AVD 名称指定的 Android 模拟器，使用 boot --platform android --device <avd-name>。对于无 GUI 的 Android 模拟器，添加 --headless。使用 --target mobile|tv 与 --platform（必需）来选择手机/平板与电视目标（AndroidTV/tvOS）。

隔离范围快速参考：

--ios-simulator-device-set <path> 将 iOS 模拟器发现 + 命令执行范围限定在一个模拟器集内。
--android-device-allowlist <serials> 将 Android 发现/选择范围限定在以逗号/空格分隔的序列号内。
范围在选择器（--device、--udid、--serial）之前应用；超出范围的选择器会失败并提示 DEVICE_NOT_FOUND。
启用 iOS 模拟器集范围后，不会枚举 iOS 物理设备。

模拟器配置快速参考：

使用 ensure-simulator 在启动会话前，在设备集内创建或重用指定名称的 iOS 模拟器。
--device <name> 是必需的（例如 "iPhone 16 Pro"）。--runtime <id> 固定运行时；省略则使用最新的兼容版本。
--boot 立即启动它。返回 udid、device、runtime、ios_simulator_device_set、created、booted。
幂等性：可安全重复调用；默认情况下重用现有的匹配模拟器。

电视快速参考：

AndroidTV：open/apps 自动使用电视启动器发现。
电视目标选择适用于模拟器/仿真器和已连接的物理设备（AndroidTV + AppleTV）。
tvOS：支持运行器驱动的交互和快照（snapshot、wait、press、fill、get、scroll、back、home、app-switcher、record 及相关选择器流程）。
tvOS 的 back/home/app-switcher 映射到运行器中的 Siri Remote 操作（menu、home、double-home）。
tvOS 遵循 iOS 模拟器专用命令语义，适用于 pinch、settings 和 push 等辅助命令。

快照和目标定位

agent-device snapshot -i
agent-device diff snapshot -i
agent-device find "Sign In" click
agent-device press @e1
agent-device fill @e2 "text"
agent-device is visible 'id="anchor"'

press 是规范的点击命令；click 是其别名。

agent-device appstate
agent-device clipboard read
agent-device clipboard write "token"
agent-device keyboard status
agent-device keyboard dismiss
agent-device perf --json
agent-device network dump [limit] [summary|headers|body|all]
agent-device push <bundle|package> <payload.json|inline-json>
agent-device trigger-app-event screenshot_taken '{"source":"qa"}'
agent-device get text @e1
agent-device screenshot out.png
agent-device settings permission grant notifications
agent-device settings permission reset camera
agent-device trace start
agent-device trace stop ./trace.log

批处理（当序列已知时）

agent-device batch --steps-file /tmp/batch-steps.json --json

在 open 之后使用 agent-device perf --json（或 metrics --json）。
有关详细指标语义、注意事项和解读指南，请参阅 references/perf-metrics.md。

防护措施（仅限高价值项）

UI 变更（导航/模态框/列表更改）后重新快照。
优先使用 snapshot -i；仅在需要时指定范围/深度。
使用引用进行发现，使用选择器进行回放/断言。
find "<query>" click --json 返回 { ref, locator, query, x, y } — 所有这些都源自匹配的快照节点。不要依赖原始 press/click 响应中的这些字段进行可观测性分析；请改用 find。
使用 fill 实现清除后输入的语义；使用 type 实现聚焦追加输入的语义。
使用 install 进行原地应用升级（在平台允许时保留应用数据），使用 reinstall 进行确定性的全新状态运行。
install/reinstall 支持的应用二进制格式：Android .apk/.aab，iOS .app/.ipa。
Android .aab 要求 PATH 中有 bundletool，或者设置 AGENT_DEVICE_BUNDLETOOL_JAR=<path-to-bundletool-all.jar> 且 PATH 中有 java。
Android .aab 可选：设置 AGENT_DEVICE_ANDROID_BUNDLETOOL_MODE=<mode> 以控制 bundletool 的 build-apks --mode（默认值：universal）。
iOS .ipa：从 Payload/*.app 中提取/安装；当存在多个应用包时，<app> 用作包 ID/名称提示。
iOS 的 appstate 是会话范围的；Android 的 appstate 是实时前台状态。iOS 响应包含 device_udid 和 ios_simulator_device_set 用于隔离验证。
iOS 的 open 响应包含 device_udid 和 ios_simulator_device_set 以确认哪个模拟器处理了该会话。
剪贴板助手：clipboard read / clipboard write <text> 在 Android 和 iOS 模拟器上受支持；iOS 物理设备尚不支持。
Android 键盘助手：keyboard status|get|dismiss 报告键盘可见性/类型，并在可见时通过 keyevent 关闭键盘。
network dump 是尽力而为的，并从会话应用日志文件中解析 HTTP(s) 条目。
生物识别设置：iOS 模拟器支持 settings faceid|touchid <match|nonmatch|enroll|unenroll>；Android 在运行时工具可用时支持 settings fingerprint <match|nonmatch>。
对于 AndroidTV/tvOS 选择，始终将 --target 与 --platform（ios、android 或 apple 别名）配对使用；仅使用目标选择是无效的。
push 模拟通知传递：
- iOS 模拟器使用 APNs 样式的负载 JSON。
- Android 使用广播操作 + 类型化附加信息（字符串/布尔值/数字）。
trigger-app-event 需要应用定义的深度链接钩子和 URL 模板配置（AGENT_DEVICE_APP_EVENT_URL_TEMPLATE 或平台特定的变体）。
trigger-app-event 需要一个活跃的会话或明确的选择器（--platform、--device、--udid、--serial）；在 iOS 物理设备上，自定义方案触发需要活跃的应用上下文。
规范的触发行为和注意事项记录在 website/docs/docs/commands.md 的 应用事件触发器 部分下。
权限设置是应用范围的，并且需要一个活跃的会话应用：settings permission <grant|deny|reset> <camera|microphone|photos|contacts|notifications> [full|limited]
iOS 模拟器权限弹窗：使用 alert wait 然后 alert accept/dismiss — accept/dismiss 内部会重试最多 2 秒，因此您不需要手动休眠。请参阅 references/permissions.md。
full|limited 模式仅适用于 iOS 的 photos；其他目标会拒绝此模式。
在 Android 上，非 ASCII 字符的 fill/type 在某些系统映像上可能需要 ADB 键盘 IME；仅从可信来源安装 IME APK 并验证校验和/签名。
如果使用 --save-script，请优先使用显式路径语法（--save-script=flow.ad 或 ./flow.ad）。
对于租户隔离的远程运行，始终同时传递 --tenant、--session-isolation tenant、--run-id 和 --lease-id。
使用较短的租约 TTL，并仅在活动工作时进行心跳；运行完成/失败后立即释放租约。
范围化运行的环境变量等效项：AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET（兼容 IOS_SIMULATOR_DEVICE_SET）和 AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST（兼容 ANDROID_DEVICE_ALLOWLIST）。
对于显式的远程客户端模式，优先使用 AGENT_DEVICE_DAEMON_BASE_URL / --daemon-base-url，而不是依赖本地守护进程元数据或仅限环回的端口。

安全与信任说明

优先使用预安装的 agent-device 二进制文件，而不是按需执行包。
如果需要安装，请固定确切的版本（例如：npx --yes agent-device@<exact-version> --help）。
签名/预配环境变量是可选的、敏感的，并且仅用于 iOS 物理设备设置。
日志/工件写入 ~/.agent-device 目录下；回放脚本写入您提供的显式路径。
对于远程守护进程模式，建议在主机上使用 AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual 加上客户端的 AGENT_DEVICE_DAEMON_BASE_URL，并根据需要配合 AGENT_DEVICE_HTTP_AUTH_HOOK 和租户范围的租约准入。
除非调试，否则保持日志关闭，并为自主运行使用最小权限/隔离环境。

将调试流程混入正常运行中（除非调试，否则保持日志关闭）。
在屏幕转换后继续使用过时的引用。
将 URL 打开与 Android 的 --activity 结合使用（不支持的组合）。
将 boot 视为默认的第一步，而不是后备方案。

🇺🇸English

Mobile Automation with agent-device

For exploration, use snapshot refs. For deterministic replay, use selectors. For structured exploratory QA bug hunts and reporting, use ../dogfood/SKILL.md.

Start Here (Read This First)

Use this skill as a router, not a full manual.

Pick one mode:
- Normal interaction flow
- Debug/crash flow
- Replay maintenance flow
Run one canonical flow below.
Open references only if blocked.

Decision Map

No target context yet: devices -> pick target -> open.
Normal UI task: open -> snapshot -i -> press/fill -> diff snapshot -i -> close
Debug/crash: open <app> -> logs clear --restart -> reproduce -> network dump -> logs path -> targeted grep
Replay drift: replay -u <path> -> verify updated selectors
Remote multi-tenant run: allocate lease -> point client at remote daemon base URL -> run commands with tenant isolation flags -> heartbeat/release lease
Device-scope isolation run: set iOS simulator set / Android allowlist -> run selectors within scope only

Canonical Flows

1) Normal Interaction Flow

agent-device open Settings --platform ios
agent-device snapshot -i
agent-device press @e3
agent-device diff snapshot -i
agent-device fill @e5 "test"
agent-device close

2) Debug/Crash Flow

agent-device open MyApp --platform ios
agent-device logs clear --restart
agent-device network dump 25
agent-device logs path

Logging is off by default. Enable only for debugging windows. logs clear --restart requires an active app session (open <app> first).

3) Replay Maintenance Flow

agent-device replay -u ./session.ad

4) Remote Tenant Lease Flow (HTTP JSON-RPC)

# Client points directly at the remote daemon HTTP base URL.
export AGENT_DEVICE_DAEMON_BASE_URL=http://mac-host.example:4310
export AGENT_DEVICE_DAEMON_AUTH_TOKEN=<token>

# Allocate lease
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"alloc-1","method":"agent_device.lease.allocate","params":{"runId":"run-123","tenantId":"acme","ttlMs":60000}}'

# Use lease in tenant-isolated command execution
agent-device \
  --tenant acme \
  --session-isolation tenant \
  --run-id run-123 \
  --lease-id <lease-id> \
  session list --json

# Heartbeat and release
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"hb-1","method":"agent_device.lease.heartbeat","params":{"leaseId":"<lease-id>","ttlMs":60000}}'
curl -sS "${AGENT_DEVICE_DAEMON_BASE_URL}/rpc" \
  -H "content-type: application/json" \
  -H "Authorization: Bearer <token>" \
  -d '{"jsonrpc":"2.0","id":"rel-1","method":"agent_device.lease.release","params":{"leaseId":"<lease-id>"}}'

Notes:

AGENT_DEVICE_DAEMON_BASE_URL makes the CLI skip local daemon discovery/startup and call the remote HTTP daemon directly.
AGENT_DEVICE_DAEMON_AUTH_TOKEN is sent in both the JSON-RPC request token and HTTP auth headers.
In remote daemon mode, --debug does not tail a local daemon.log; inspect logs on the remote host instead.

Command Skeleton (Minimal)

Session and navigation

agent-device devices
agent-device devices --platform ios --ios-simulator-device-set /tmp/tenant-a/simulators
agent-device devices --platform android --android-device-allowlist emulator-5554,device-1234
agent-device ensure-simulator --device "iPhone 16" --ios-simulator-device-set /tmp/tenant-a/simulators
agent-device ensure-simulator --device "iPhone 16" --runtime com.apple.CoreSimulator.SimRuntime.iOS-18-4 --ios-simulator-device-set /tmp/tenant-a/simulators --boot
agent-device open [app|url] [url]
agent-device open [app] --relaunch
agent-device close [app]
agent-device install <app> <path-to-binary>
agent-device reinstall <app> <path-to-binary>
agent-device session list

Use boot only as fallback when open cannot find/connect to a ready target. For Android emulators by AVD name, use boot --platform android --device <avd-name>. For Android emulators without GUI, add --headless. Use --target mobile|tv with --platform (required) to pick phone/tablet vs TV targets (AndroidTV/tvOS).

Isolation scoping quick reference:

--ios-simulator-device-set <path> scopes iOS simulator discovery + command execution to one simulator set.
--android-device-allowlist <serials> scopes Android discovery/selection to comma/space separated serials.
Scope is applied before selectors (--device, --udid, --serial); out-of-scope selectors fail with DEVICE_NOT_FOUND.
With iOS simulator-set scope enabled, iOS physical devices are not enumerated.

Simulator provisioning quick reference:

Use ensure-simulator to create or reuse a named iOS simulator inside a device set before starting a session.
--device <name> is required (e.g. "iPhone 16 Pro"). --runtime <id> pins the runtime; omit to use the newest compatible one.
--boot boots it immediately. Returns udid, device, runtime, ios_simulator_device_set, created, booted.

TV quick reference:

AndroidTV: open/apps use TV launcher discovery automatically.
TV target selection works on emulators/simulators and connected physical devices (AndroidTV + AppleTV).
tvOS: runner-driven interactions and snapshots are supported (snapshot, wait, press, fill, get, scroll, back, home, app-switcher, and related selector flows).

Snapshot and targeting

agent-device snapshot -i
agent-device diff snapshot -i
agent-device find "Sign In" click
agent-device press @e1
agent-device fill @e2 "text"
agent-device is visible 'id="anchor"'

press is canonical tap command; click is an alias.

Utilities

agent-device appstate
agent-device clipboard read
agent-device clipboard write "token"
agent-device keyboard status
agent-device keyboard dismiss
agent-device perf --json
agent-device network dump [limit] [summary|headers|body|all]
agent-device push <bundle|package> <payload.json|inline-json>
agent-device trigger-app-event screenshot_taken '{"source":"qa"}'
agent-device get text @e1
agent-device screenshot out.png
agent-device settings permission grant notifications
agent-device settings permission reset camera
agent-device trace start
agent-device trace stop ./trace.log

Batch (when sequence is already known)

agent-device batch --steps-file /tmp/batch-steps.json --json

Performance Check

Use agent-device perf --json (or metrics --json) after open.
For detailed metric semantics, caveats, and interpretation guidance, see references/perf-metrics.md.

Guardrails (High Value Only)

Re-snapshot after UI mutations (navigation/modal/list changes).
Prefer snapshot -i; scope/depth only when needed.
Use refs for discovery, selectors for replay/assertions.
find "<query>" click --json returns { ref, locator, query, x, y } — all derived from the matched snapshot node. Do not rely on these fields from raw press/click responses for observability; use find instead.
Use fill for clear-then-type semantics; use type for focused append typing.
Use install for in-place app upgrades (keep app data when platform permits), and for deterministic fresh-state runs.

Security and Trust Notes

Prefer a preinstalled agent-device binary over on-demand package execution.
If install is required, pin an exact version (for example: npx --yes agent-device@<exact-version> --help).
Signing/provisioning environment variables are optional, sensitive, and only for iOS physical-device setup.
Logs/artifacts are written under ~/.agent-device; replay scripts write to explicit paths you provide.
For remote daemon mode, prefer AGENT_DEVICE_DAEMON_SERVER_MODE=http|dual on the host plus client-side AGENT_DEVICE_DAEMON_BASE_URL, with AGENT_DEVICE_HTTP_AUTH_HOOK and tenant-scoped lease admission where needed.
Keep logging off unless debugging and use least-privilege/isolated environments for autonomous runs.

Common Mistakes

Mixing debug flow into normal runs (keep logs off unless debugging).
Continuing to use stale refs after screen transitions.
Using URL opens with Android --activity (unsupported combination).
Treating boot as default first step instead of fallback.

References

Weekly Installs

1.1K

Repository

callstackincuba…t-device

GitHub Stars

1.0K

First Seen

Feb 3, 2026

Security Audits

Gen Agent Trust HubPass SocketPass SnykWarn

Installed on

codex1.0K

opencode982

gemini-cli973

github-copilot941

amp866

kimi-cli860

React 组合模式指南：Vercel 组件架构最佳实践，提升代码可维护性

102,200 周安装

Idempotent: safe to call repeatedly; reuses an existing matching simulator by default.

tvOS back/home/app-switcher map to Siri Remote actions (menu, home, double-home) in the runner.

tvOS follows iOS simulator-only command semantics for helpers like pinch, settings, and push.

App binary format support for install/reinstall: Android .apk/.aab, iOS .app/.ipa.

Android .aab requires bundletool in PATH, or AGENT_DEVICE_BUNDLETOOL_JAR=<path-to-bundletool-all.jar> with java in PATH.

Android .aab optional: set AGENT_DEVICE_ANDROID_BUNDLETOOL_MODE=<mode> to control bundletool build-apks --mode (default: universal).

iOS .ipa: extract/install from Payload/*.app; when multiple app bundles are present, <app> is used as a bundle id/name hint.

iOS appstate is session-scoped; Android appstate is live foreground state. iOS responses include device_udid and ios_simulator_device_set for isolation verification.

iOS open responses include device_udid and ios_simulator_device_set to confirm which simulator handled the session.

Clipboard helpers: clipboard read / clipboard write <text> are supported on Android and iOS simulators; iOS physical devices are not supported yet.

Android keyboard helpers: keyboard status|get|dismiss report keyboard visibility/type and dismiss via keyevent when visible.

network dump is best-effort and parses HTTP(s) entries from the session app log file.

For AndroidTV/tvOS selection, always pair --target with --platform (ios, android, or apple alias); target-only selection is invalid.

push simulates notification delivery:

iOS simulator uses APNs-style payload JSON.
Android uses broadcast action + typed extras (string/boolean/number).

trigger-app-event requires app-defined deep-link hooks and URL template configuration (AGENT_DEVICE_APP_EVENT_URL_TEMPLATE or platform-specific variants).

trigger-app-event requires an active session or explicit selectors (--platform, --device, --udid, --serial); on iOS physical devices, custom-scheme triggers require active app context.

Canonical trigger behavior and caveats are documented in website/docs/docs/commands.md under App event triggers.

iOS simulator permission alerts: use alert wait then alert accept/dismiss — accept/dismiss retry internally for up to 2 s so you do not need manual sleeps. See references/permissions.md.

full|limited mode applies only to iOS photos; other targets reject mode.

On Android, non-ASCII fill/type may require an ADB keyboard IME on some system images; only install IME APKs from trusted sources and verify checksum/signature.

If using --save-script, prefer explicit path syntax (--save-script=flow.ad or ./flow.ad).

For tenant-isolated remote runs, always pass --tenant, --session-isolation tenant, --run-id, and --lease-id together.

Use short lease TTLs and heartbeat only while work is active; release leases immediately after run completion/failure.

Env equivalents for scoped runs: AGENT_DEVICE_IOS_SIMULATOR_DEVICE_SET (compat IOS_SIMULATOR_DEVICE_SET) and AGENT_DEVICE_ANDROID_DEVICE_ALLOWLIST (compat ANDROID_DEVICE_ALLOWLIST).

For explicit remote client mode, prefer AGENT_DEVICE_DAEMON_BASE_URL / --daemon-base-url instead of relying on local daemon metadata or loopback-only ports.