joseph/internal-joseph
1
0
Fork 0
Code Issues 21 Pull requests Projects Releases Packages Wiki Activity Actions

[OneDev #13] research: tui-use — let AI agents control interactive terminal programs #19

New issue
Closed
opened 2026-05-24 04:46:37 +00:00 by joseph · 5 comments
joseph commented 2026-05-24 04:46:37 +00:00
Owner
Copy link

Imported from OneDev issue #13 (id 35279)
Original project: internal-joseph
Original state: Closed
Original submitterId: 5
Original submitDate: 2026-04-10T17:07:19.174+00:00
Original lastActivity: {'date': '2026-04-24T21:24:32.242+00:00', 'description': "changed state to 'Closed'", 'userId': 5}

Original fields:

  • Type: New Feature
  • Priority: Normal
  • Assignees: None

Idea

Evaluate tui-use (https://github.com/onesuper/tui-use) as a foundation for AI-driven interactive terminal control. It lets agents operate TUI programs (ncurses, prompts, menus) that normally block automation — think expect, but designed for AI agents.

Why

Directly complements internal-joseph#11 (interactive terminal driver app). Could be the ready-made substrate for that idea rather than building from scratch. Also useful for automating installer wizards, interactive CLI tools, and legacy TUI apps on managed endpoints.

Open questions

  • What's the interface — MCP tool, Python API, or pty wrapper?
  • Does it handle raw ANSI/terminfo or just common TUI frameworks?
  • Licensing — MIT/Apache/other?
Imported from OneDev issue #13 (id 35279) Original project: internal-joseph Original state: Closed Original submitterId: 5 Original submitDate: 2026-04-10T17:07:19.174+00:00 Original lastActivity: {'date': '2026-04-24T21:24:32.242+00:00', 'description': "changed state to 'Closed'", 'userId': 5} Original fields: - Type: New Feature - Priority: Normal - Assignees: None ## Idea Evaluate tui-use (https://github.com/onesuper/tui-use) as a foundation for AI-driven interactive terminal control. It lets agents operate TUI programs (ncurses, prompts, menus) that normally block automation — think expect, but designed for AI agents. ## Why Directly complements internal-joseph#11 (interactive terminal driver app). Could be the ready-made substrate for that idea rather than building from scratch. Also useful for automating installer wizards, interactive CLI tools, and legacy TUI apps on managed endpoints. ## Open questions - What's the interface — MCP tool, Python API, or pty wrapper? - Does it handle raw ANSI/terminfo or just common TUI frameworks? - Licensing — MIT/Apache/other?
joseph commented 2026-05-24 04:46:47 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 242 on issue #13
Original userId: 5
Original date: 2026-04-10T17:34:13.965+00:00

Research findings — tui-use

What it is

tui-use is a Node.js/TypeScript CLI tool (npm install -g tui-use, v0.1.17) that wraps any command in a PTY session managed by a background daemon. An AI agent drives the program via shell subcommands (start, type, press, wait, snapshot), reading back a plain-text screen rendered by a headless xterm emulator. Self-described as "like BrowserUse, but for the terminal." The key insight: wait blocks until the screen stabilizes (debounce) rather than requiring sleep guesses, and highlights exposes inverse-video spans so agents can detect selected menu items without parsing ANSI codes.

Interface & API

Pure CLI — no MCP server, no Python API. Agent calls shell subcommands:

SID=$(tui-use start python3)
tui-use wait --text ">>>"
tui-use type "import os\n"
tui-use snapshot --format json   # returns lines[], cursor, highlights[], status
tui-use press arrow_down
tui-use press enter

Session model: background daemon holds PTY sessions across CLI invocations. Sessions identified by random adjective-noun ID (e.g. calm-otter). On Linux: Unix socket. On Windows: TCP port 7654.

Snapshot JSON: lines[] (plain text per row), cursor {x,y}, highlights[] (inverse-video spans = selection indicator), status, is_fullscreen, title.

Ships a Claude Code plugin (.claude-plugin/ dir, marketplace install via /plugin marketplace add onesuper/tui-use). Also an OpenAI Codex plugin.

ANSI / terminfo coverage

Uses @xterm/headless (xterm.js 6.x — same engine as VS Code terminal). Full VT100/VT220/xterm support. By default snapshot returns plain text (colors stripped). --color flag (just merged 2026-04-10) re-encodes cell buffer as ANSI for the agent. Inverse-video is extracted into highlights[] automatically.

Key gaps: No F11/F12, no alt+key combos (alt+f/b for readline word nav), no shift+arrows, no mouse support, cursor-only moves not detected by wait (open issue #4).

License

MIT.

Maturity signals

Signal Value
Created 2026-04-07 (3 days old)
Stars 159
npm versions 0.1.0 → 0.1.17 in 2 days
Real maintainers 1 (onesuper, 55 commits)
Test suite Mostly documentation stubs, not live integration tests
Windows support ConPTY PR landed 2026-04-10 (same day as this research)

Limitations

  1. Colors lost by default (--color just added, adds token overhead)
  2. Key mapping gaps (no alt+key, F11/F12, shift+arrows)
  3. No mouse support
  4. Cursor-only changes not detected by wait (issue #4 open)
  5. Windows IPC uses TCP 7654 (port conflict risk; verify bind address)
  6. node-pty native module — needs build tools on unusual arches
  7. Single maintainer, 3 days old, API in flux
  8. No timeout on type/press — only wait has timeout logic

Alternatives comparison

Tool Wait mechanism Windows Maturity AI-native
tui-use Debounce + text pattern Yes (ConPTY) 3 days Yes
pexpect expect(pattern, timeout) Partial Very mature No
expect (tcl) expect "pattern" No (WSL only) Very mature No
tmux send-keys + capture-pane Polling/sleep No Mature No

tui-use's rendered-screen snapshot model is fundamentally better for AI agents than pexpect's raw byte stream. The gap it fills in PCT: interactive programs (wizard installers, fzf pickers, pdb sessions) on Linux LXC containers where sshubble handles non-interactive commands fine.

Recommendation: BENCH

The design is correct. The snapshot model + debounce wait + inverse-video highlights is the right abstraction for AI agents. The Claude Code plugin packaging is a direct fit.

However: 3 days old, one real maintainer, test suite is mostly stubs, API visibly in flux, Windows ConPTY support just landed. Too early to put on a production path.

ADOPT conditions: revisit ~2026-04-24. Check: stars trend, npm version stabilization, issue #4 (cursor detection) closed, second real contributor merging a feature PR.

Suggested next steps (if watching)

  1. Re-evaluate around 2026-04-24
  2. Pilot: drive git rebase -i inside a Linux LXC container via Claude Code on localinf — tests debounce/highlight flow without blocking any production path
  3. Test Windows path on VMID 130 (win11-test): npm install -g tui-use, drive a PowerShell interactive installer, verify TCP 7654 isn't firewalled
  4. If ADOPT: write tui-use/SKILL.md and add to meta/skills for shared availability on localinf and joseph-desktop
Imported from OneDev comment id 242 on issue #13 Original userId: 5 Original date: 2026-04-10T17:34:13.965+00:00 ## Research findings — tui-use ### What it is tui-use is a Node.js/TypeScript CLI tool (`npm install -g tui-use`, v0.1.17) that wraps any command in a PTY session managed by a background daemon. An AI agent drives the program via shell subcommands (`start`, `type`, `press`, `wait`, `snapshot`), reading back a plain-text screen rendered by a headless xterm emulator. Self-described as "like BrowserUse, but for the terminal." The key insight: `wait` blocks until the screen stabilizes (debounce) rather than requiring sleep guesses, and `highlights` exposes inverse-video spans so agents can detect selected menu items without parsing ANSI codes. --- ### Interface & API Pure **CLI** — no MCP server, no Python API. Agent calls shell subcommands: ```bash SID=$(tui-use start python3) tui-use wait --text ">>>" tui-use type "import os\n" tui-use snapshot --format json # returns lines[], cursor, highlights[], status tui-use press arrow_down tui-use press enter ``` Session model: background daemon holds PTY sessions across CLI invocations. Sessions identified by random `adjective-noun` ID (e.g. `calm-otter`). On Linux: Unix socket. On Windows: TCP port 7654. Snapshot JSON: `lines[]` (plain text per row), `cursor {x,y}`, `highlights[]` (inverse-video spans = selection indicator), `status`, `is_fullscreen`, `title`. Ships a **Claude Code plugin** (`.claude-plugin/` dir, marketplace install via `/plugin marketplace add onesuper/tui-use`). Also an OpenAI Codex plugin. --- ### ANSI / terminfo coverage Uses `@xterm/headless` (xterm.js 6.x — same engine as VS Code terminal). Full VT100/VT220/xterm support. By default snapshot returns plain text (colors stripped). `--color` flag (just merged 2026-04-10) re-encodes cell buffer as ANSI for the agent. Inverse-video is extracted into `highlights[]` automatically. **Key gaps:** No F11/F12, no alt+key combos (alt+f/b for readline word nav), no shift+arrows, no mouse support, cursor-only moves not detected by `wait` (open issue #4). --- ### License MIT. --- ### Maturity signals | Signal | Value | |---|---| | Created | 2026-04-07 (3 days old) | | Stars | 159 | | npm versions | 0.1.0 → 0.1.17 in 2 days | | Real maintainers | 1 (onesuper, 55 commits) | | Test suite | Mostly documentation stubs, not live integration tests | | Windows support | ConPTY PR landed 2026-04-10 (same day as this research) | --- ### Limitations 1. Colors lost by default (--color just added, adds token overhead) 2. Key mapping gaps (no alt+key, F11/F12, shift+arrows) 3. No mouse support 4. Cursor-only changes not detected by `wait` (issue #4 open) 5. Windows IPC uses TCP 7654 (port conflict risk; verify bind address) 6. node-pty native module — needs build tools on unusual arches 7. Single maintainer, 3 days old, API in flux 8. No timeout on `type`/`press` — only `wait` has timeout logic --- ### Alternatives comparison | Tool | Wait mechanism | Windows | Maturity | AI-native | |---|---|---|---|---| | **tui-use** | Debounce + text pattern | Yes (ConPTY) | 3 days | Yes | | **pexpect** | expect(pattern, timeout) | Partial | Very mature | No | | **expect (tcl)** | expect "pattern" | No (WSL only) | Very mature | No | | **tmux send-keys + capture-pane** | Polling/sleep | No | Mature | No | tui-use's rendered-screen snapshot model is fundamentally better for AI agents than pexpect's raw byte stream. The gap it fills in PCT: interactive programs (wizard installers, fzf pickers, pdb sessions) on Linux LXC containers where sshubble handles non-interactive commands fine. --- ### Recommendation: **BENCH** The design is correct. The snapshot model + debounce wait + inverse-video highlights is the right abstraction for AI agents. The Claude Code plugin packaging is a direct fit. However: 3 days old, one real maintainer, test suite is mostly stubs, API visibly in flux, Windows ConPTY support just landed. Too early to put on a production path. **ADOPT conditions:** revisit ~2026-04-24. Check: stars trend, npm version stabilization, issue #4 (cursor detection) closed, second real contributor merging a feature PR. --- ### Suggested next steps (if watching) 1. Re-evaluate around 2026-04-24 2. Pilot: drive `git rebase -i` inside a Linux LXC container via Claude Code on localinf — tests debounce/highlight flow without blocking any production path 3. Test Windows path on VMID 130 (win11-test): `npm install -g tui-use`, drive a PowerShell interactive installer, verify TCP 7654 isn't firewalled 4. If ADOPT: write `tui-use/SKILL.md` and add to meta/skills for shared availability on localinf and joseph-desktop
joseph commented 2026-05-24 04:46:47 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 377 on issue #13
Original userId: 5
Original date: 2026-04-18T12:52:52.880+00:00

Expanding scope of this issue to cover the full PTY control evaluation, not just tui-use in isolation.

Expanded scope: ghst vs tui-use eval

PCT already has ghst — a self-built PTY session tool used daily for subagent dispatch. The original "self-implement" recommendation was pragmatic, not ideological. This issue now tracks a head-to-head eval to determine one of three outcomes:

  1. Keep ghst as-is — tui-use offers nothing materially better
  2. Borrow from tui-use into ghst — specific features worth porting (e.g. semantic wait --text, xterm-headless rendering)
  3. Abandon ghst, adopt tui-use — only if tui-use wins by a margin large enough to justify migration cost

Eval rubric dimensions (proposed)

  • Semantic waiting (wait --text "..." vs polling)
  • VT/ANSI rendering fidelity
  • Windows ConPTY support parity
  • Snapshot → action loop latency
  • Plugin/integration model (Claude Code, MCP)
  • Migration cost (all ghst tooling: briefs, skill, Monitor pattern, session naming)

"Significantly better" must clear the migration cost bar — a narrow win still means keep ghst.

tui-use current status (as of 2026-04-18)

193 stars, v0.1.18, MIT, TypeScript, last commit April 11 2026, CI passing, ConPTY/Windows tests present. CLI daemon via @xterm/headless v6.0.0. Integrates with Claude Code via plugin. Supports vim, lazygit, htop, fzf, Python/Node REPLs.

Also in scope: #11 and #14 closed and folded here.

Imported from OneDev comment id 377 on issue #13 Original userId: 5 Original date: 2026-04-18T12:52:52.880+00:00 Expanding scope of this issue to cover the full PTY control evaluation, not just tui-use in isolation. ## Expanded scope: ghst vs tui-use eval PCT already has **ghst** — a self-built PTY session tool used daily for subagent dispatch. The original "self-implement" recommendation was pragmatic, not ideological. This issue now tracks a head-to-head eval to determine one of three outcomes: 1. **Keep ghst as-is** — tui-use offers nothing materially better 2. **Borrow from tui-use into ghst** — specific features worth porting (e.g. semantic `wait --text`, xterm-headless rendering) 3. **Abandon ghst, adopt tui-use** — only if tui-use wins by a margin large enough to justify migration cost ## Eval rubric dimensions (proposed) - Semantic waiting (`wait --text "..."` vs polling) - VT/ANSI rendering fidelity - Windows ConPTY support parity - Snapshot → action loop latency - Plugin/integration model (Claude Code, MCP) - Migration cost (all ghst tooling: briefs, skill, Monitor pattern, session naming) "Significantly better" must clear the migration cost bar — a narrow win still means keep ghst. ## tui-use current status (as of 2026-04-18) 193 stars, v0.1.18, MIT, TypeScript, last commit April 11 2026, CI passing, ConPTY/Windows tests present. CLI daemon via `@xterm/headless` v6.0.0. Integrates with Claude Code via plugin. Supports vim, lazygit, htop, fzf, Python/Node REPLs. Also in scope: **#11 and #14 closed and folded here.**
joseph commented 2026-05-24 04:46:47 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 378 on issue #13
Original userId: 5
Original date: 2026-04-18T13:08:07.566+00:00

Eval Plan — ghst vs tui-use

Test case: interactive diagnostic menu

Drive a shell script that presents a numbered menu (select a site, confirm action, read result) — realistic shape of PCT onboarding/diagnostic runbooks. Both tools must complete the same task: launch the script, read the menu, send a selection, read the result.

Setup

  • Write a small eval-target.sh in C:/Users/Joseph/git/lab/tui-eval/ — a bash script with a select menu and a confirmation prompt
  • Run it through ghst (existing tool) and tui-use (npm install, Claude Code plugin)
  • Score each against the rubric below

Rubric (each dimension 1–5)

Dimension What to measure
Semantic waiting Does it know when the menu appeared vs. polling blindly?
VT/ANSI fidelity Does it correctly read menu text with escape codes stripped?
ConPTY/Windows Does it work natively on Windows without WSL shim?
Snapshot latency Time from "menu appeared" to "selection sent"
Integration model How much glue code to wire into a Claude Code session?
Migration cost (ghst only) — estimated effort to replace all existing ghst tooling

Verdict thresholds

  • tui-use wins by ≥8 pts → abandon ghst, adopt tui-use
  • tui-use wins by 4–7 pts → borrow specific features into ghst
  • tui-use wins by <4 pts or ghst wins → keep ghst as-is

Session handoff

  1. Install tui-use: npm install -g tui-use
  2. Write tui-eval/eval-target.sh (select menu + confirm prompt)
  3. Run eval-target.sh through ghst, record score
  4. Run eval-target.sh through tui-use, record score
  5. Post scored result table here and verdict
Imported from OneDev comment id 378 on issue #13 Original userId: 5 Original date: 2026-04-18T13:08:07.566+00:00 ## Eval Plan — ghst vs tui-use ### Test case: interactive diagnostic menu Drive a shell script that presents a numbered menu (select a site, confirm action, read result) — realistic shape of PCT onboarding/diagnostic runbooks. Both tools must complete the same task: launch the script, read the menu, send a selection, read the result. ### Setup - Write a small `eval-target.sh` in `C:/Users/Joseph/git/lab/tui-eval/` — a bash script with a `select` menu and a confirmation prompt - Run it through **ghst** (existing tool) and **tui-use** (npm install, Claude Code plugin) - Score each against the rubric below ### Rubric (each dimension 1–5) | Dimension | What to measure | |-----------|----------------| | Semantic waiting | Does it know when the menu appeared vs. polling blindly? | | VT/ANSI fidelity | Does it correctly read menu text with escape codes stripped? | | ConPTY/Windows | Does it work natively on Windows without WSL shim? | | Snapshot latency | Time from "menu appeared" to "selection sent" | | Integration model | How much glue code to wire into a Claude Code session? | | Migration cost | (ghst only) — estimated effort to replace all existing ghst tooling | ### Verdict thresholds - tui-use wins by ≥8 pts → abandon ghst, adopt tui-use - tui-use wins by 4–7 pts → borrow specific features into ghst - tui-use wins by <4 pts or ghst wins → keep ghst as-is ### Session handoff 1. Install tui-use: `npm install -g tui-use` 2. Write `tui-eval/eval-target.sh` (select menu + confirm prompt) 3. Run eval-target.sh through ghst, record score 4. Run eval-target.sh through tui-use, record score 5. Post scored result table here and verdict
joseph commented 2026-05-24 04:46:47 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 382 on issue #13
Original userId: 5
Original date: 2026-04-18T13:36:18.754+00:00

Eval Results — ghst vs tui-use

Test: eval-target.sh — bash select menu (4 options), site selection, confirmation prompt, result output. Both tools completed the full flow successfully.

Setup notes

  • tui-use install blocked on Windows (Node 20.11.0 — needs ≥20.17 or ≥22 for node-pty native build). Ran via WSL (Node 22.22.2) as workaround.
  • tui-use v0.1.20 API: --session flag removed; must use tui-use use <id> to set current session.
  • ghst: PowerShell session → wsl bash → ran script directly.

Rubric scores (1–5)

Dimension ghst tui-use Notes
Semantic waiting 2 5 wait --text genuinely blocks until pattern matches — zero polling. ghst relies on 400ms settle heuristic (works for fast programs, fragile for slow-starting ones).
VT/ANSI fidelity 4 3 Both strip ANSI correctly. tui-use snapshots are noisy: captures full xterm screen including shell motd, keybinding tables from Zellij profile. Program output is present but buried. ghst read returns only recent lines — cleaner.
ConPTY/Windows 5 2 ghst works natively on Windows. tui-use install fails on Node 20.11 (PCT's current system Node). Requires WSL or Node upgrade. Hard blocker for PCT's Windows-primary environment.
Snapshot latency 4 3 ghst send returns buffer in one call with 400ms settle. tui-use requires separate wait + snapshot subprocess calls per step. Total tui-use run: ~6.4s.
Integration model 5 3 ghst has existing skill, Monitor pattern, brief templates — proven in production. tui-use has a Claude Code plugin but requires new PCT tooling from scratch.
Total 20 16 ghst wins by 4

Verdict: Keep ghst as-is

tui-use wins by −4 (ghst wins). Rubric threshold for "borrow features" was +4–7 for tui-use. Does not meet bar.

One genuine tui-use advantage: wait --text semantic blocking is the right abstraction. ghst's settle heuristic is fragile for programs with variable startup time. Worth adding a wait_for_text helper to the ghst skill (poll mcp__ghst__read until pattern matches, with timeout) — that closes the gap without migration.

Hard blocker confirmed: tui-use cannot install on Node 20.11 without either upgrading system Node or using WSL. Until PCT migrates to Node 22+ as default, tui-use is not deployable on the primary Windows dev environment.

Closing recommendation

Close this issue as won't adopt. Open a separate small issue if we want to add a wait_for_text helper to the ghst skill.

Imported from OneDev comment id 382 on issue #13 Original userId: 5 Original date: 2026-04-18T13:36:18.754+00:00 ## Eval Results — ghst vs tui-use **Test:** `eval-target.sh` — bash `select` menu (4 options), site selection, confirmation prompt, result output. Both tools completed the full flow successfully. ### Setup notes - tui-use install **blocked on Windows** (Node 20.11.0 — needs ≥20.17 or ≥22 for node-pty native build). Ran via WSL (Node 22.22.2) as workaround. - tui-use v0.1.20 API: `--session` flag removed; must use `tui-use use <id>` to set current session. - ghst: PowerShell session → `wsl bash` → ran script directly. ### Rubric scores (1–5) | Dimension | ghst | tui-use | Notes | |-----------|------|---------|-------| | Semantic waiting | 2 | 5 | `wait --text` genuinely blocks until pattern matches — zero polling. ghst relies on 400ms settle heuristic (works for fast programs, fragile for slow-starting ones). | | VT/ANSI fidelity | 4 | 3 | Both strip ANSI correctly. tui-use snapshots are noisy: captures full xterm screen including shell motd, keybinding tables from Zellij profile. Program output is present but buried. ghst `read` returns only recent lines — cleaner. | | ConPTY/Windows | 5 | 2 | ghst works natively on Windows. tui-use install fails on Node 20.11 (PCT's current system Node). Requires WSL or Node upgrade. Hard blocker for PCT's Windows-primary environment. | | Snapshot latency | 4 | 3 | ghst `send` returns buffer in one call with 400ms settle. tui-use requires separate `wait` + `snapshot` subprocess calls per step. Total tui-use run: ~6.4s. | | Integration model | 5 | 3 | ghst has existing skill, Monitor pattern, brief templates — proven in production. tui-use has a Claude Code plugin but requires new PCT tooling from scratch. | | **Total** | **20** | **16** | ghst wins by 4 | ### Verdict: **Keep ghst as-is** tui-use wins by −4 (ghst wins). Rubric threshold for "borrow features" was +4–7 for tui-use. Does not meet bar. **One genuine tui-use advantage:** `wait --text` semantic blocking is the right abstraction. ghst's settle heuristic is fragile for programs with variable startup time. Worth adding a `wait_for_text` helper to the ghst skill (poll `mcp__ghst__read` until pattern matches, with timeout) — that closes the gap without migration. **Hard blocker confirmed:** tui-use cannot install on Node 20.11 without either upgrading system Node or using WSL. Until PCT migrates to Node 22+ as default, tui-use is not deployable on the primary Windows dev environment. ### Closing recommendation Close this issue as **won't adopt**. Open a separate small issue if we want to add a `wait_for_text` helper to the ghst skill.
joseph commented 2026-05-24 04:46:47 +00:00
Author
Owner
Copy link

Imported from OneDev comment id 452 on issue #13
Original userId: 5
Original date: 2026-04-24T21:24:31.242+00:00

Eval complete, verdict is keep-ghst-as-is (ghst 20 / tui-use 16, details in 2026-04-18 comment). Closing as won't-adopt. File a separate narrow issue if we want the wait_for_text helper ported into the ghst skill.

Imported from OneDev comment id 452 on issue #13 Original userId: 5 Original date: 2026-04-24T21:24:31.242+00:00 Eval complete, verdict is keep-ghst-as-is (ghst 20 / tui-use 16, details in 2026-04-18 comment). Closing as won't-adopt. File a separate narrow issue if we want the `wait_for_text` helper ported into the ghst skill.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
joseph/internal-joseph#19
No description provided.