Pyjab MCP README

pyjab-mcp

An MCP server that lets an AI agent drive Java desktop applications — Swing and AWT — through pyjab, which reads the JVM's own accessibility tree via Java Access Bridge. JavaFX is not supported: it publishes its accessibility through UI Automation rather than Java Access Bridge, so a JavaFX window is not a Java window as far as JAB is concerned (pyjab#227).

mcp-name: io.github.gaozhao1989/pyjab-mcp

It never modifies, restarts or injects into the target application. That constraint is the point: launch parameters, classpaths and dependency JARs stay exactly as they are, which is what makes this usable against a production Java client that you cannot change.

The alternative is to put an agent inside the target JVM. An agent is attached to a JVM that is already running, or named on the command line with -javaagent when the application starts, and either way the tool's code sits inside someone else's process — a JVM it did not launch and does not control. This server does neither: it loads nothing into the application and reads the accessibility tree through Java Access Bridge from outside it, which is how it can attach to a process it did not launch.

The dynamic half of that — attaching an agent to a JVM that is already running — is the half the JDK has a direction on, with no release date attached. JEP 451 shipped in JDK 21 as the preparation stage: loading an agent into an already-running JVM now warns. The JEP states its aim as "prepare users for a future release which disallows the dynamic loading of agents by default", a release that has not been named; its own non-goals leave agents named with -javaagent at startup, and any use of the Attach API that is not loading an agent, outside that direction.

Status

Released on PyPI — 16 tools: java_diagnostics, java_list_windows, java_attach_window, java_snapshot, java_find, java_get_element, java_get_text, java_get_table, java_click, java_set_text, java_select, java_fill_form, java_send_keys, java_focus_window, java_wait_for, and java_screenshot. It needs pyjab>=1.11.0: 1.9.0 brought the tree walk a snapshot is built on, 1.10.0 brought the public window listing and JABDriver.detach(), which releases a driver's binding without terminating the process — this server has no tool that releases a session, so a binding lasts for the life of the process — and 1.11.0 brought JABDriver.send_keys, so java_send_keys sends a real chord, and JABDriver.focus(), so java_focus_window can bring a window forward.

java_send_keys sends a chord to whichever window has the keyboard focus, and attaching does not give a window focus (pyjab#180). java_focus_window is the missing half: it brings the window forward and, because asked for focus and got focus are different facts, it reports whether the window actually came forward rather than assuming it did. It probes for the API rather than assuming it, so an install below the pin is told which release has it instead of failing with an AttributeError.

Two elements can share a name, which is why a handle carries where an element is as well as what it is called: searching by name returns every match, each gets its own handle, and the end-to-end suite checks that clicking one acts on that one and not the other.

Nothing is acted on until it has been checked. Every action re-finds the element and compares role, name, position among siblings and bounds against what the snapshot recorded, then acts on the element it just checked — and if anything differs, it does nothing and reports the difference.

java_get_table is where this substrate earns its place: it reads a JTable through the JVM's own AccessibleTable interface — row and column counts, then cell text, a page of rows at a time — rather than through a platform bridge that does not carry that information at all.

Live behaviour is verified on a hosted Windows runner by windows-gui.yml: the end-to-end suite attaches to a real Swing application, snapshots it, resolves a handle against the real element, and checks that a window opened after the bridge had been idle is still seen. That last one is the reason the layer exists — a message pump on the wrong thread passes every other test in this repository. What it still cannot cover is your application: the run drives pyjab's test app, and the one that matters is the one you care about.

It needs the same pyjab>=1.11.0 the tool list above pins, and for the releases named there; the one addition worth repeating here is JABDriver.focus(), which java_focus_window uses to bring a window forward and report whether it came forward. That return value is why the tool exists at all: what a caller needs to know before sending a chord is where the keyboard focus actually is, not that focus was requested.

A handle from java_snapshot is a position, not a reference. A later call re-finds the element and checks its role, name, position and bounds before doing anything, so a window that moved on produces a message rather than an action on the wrong control.

The one thing that was already settled is the question the whole direction rested on. Measured on a real Java window, through the same UIAutomationCore that generic Windows automation uses:

The control is what makes the six mean anything: the same measurement over the whole desktop finishes, so the measurement works and the Java window is the hard case.

Where to read first

Maintainer working notes are kept out of this repository deliberately. Everything a user or a contributor needs is above, and the rules that are not prose are enforced by the guards in tools/ — which run in CI on every push.

Requirements

• Windows
• Python 3.10 or newer
• A JDK with Java Access Bridge enabled — "%JAVA_HOME%\bin\jabswitch" -enable, then restart the Java application — for the application being driven
• pyjab, which brings the rest

Install

pip install pyjab-mcp

Working on the server itself is pip install -e ".[dev]" — see Working on it.

Or install the bundle: one file, no build step

v0.1.0–v0.1.2 predate the bundle and carry no assets; every release cut since carries one. It is pyjab-mcp-<version>.mcpb — the built wheel and its dependency closure, installed together into one tree per supported CPython minor, plus one entry script, in a single file. Each tree carries the package itself, so the file holds one copy of pyjab_mcp per minor rather than one wheel beside the trees. Download it from the releases page and hand it to the client: Claude Desktop installs a .mcpb by opening it, and docs/CLIENTS.md has the details.

The bundle changes none of the requirements above, and it narrows one of them to the interpreters it actually carries. It bundles no Python interpreter and no JDK: this server drives Java Access Bridge, so 64-bit Windows, 64-bit CPython 3.10-3.13, and a JDK with Java Access Bridge enabled are still what the machine has to provide — and the Java application it drives still has to be running before you attach to it. That range is narrower than pip install, which has no upper bound, and it is not a detail: the bundle is tens of megabytes because it carries one dependency tree per supported minor, since pydantic-core, cffi, rpds-py, cryptography and pywin32 ship compiled wheels tagged for one minor each. Started on any other interpreter it stops with a sentence naming the interpreter it found and the ones the file carries, rather than an ImportError from a native module.

The interpreter has to be the python on PATH. The manifest declares no user_config fallback, so the client runs whatever python resolves to there — and a machine with CPython installed but not on PATH gets a client that shows no tools, with nothing in it that names the cause: no interpreter ever starts, so the bundle's own refusal cannot run. where.exe python prints the interpreter that will be used; if it prints nothing, put the installation's directory on PATH.

MCPB enforces no permissions. There is no manifest-level sandbox — a bundled server runs with the full privileges of the user who installed it, exactly as it would if you had started it by hand. Read the source before installing a .mcpb from anyone, this one included.

Use

The console script the package installs:

{
  "mcpServers": {
    "pyjab-mcp": { "command": "python", "args": ["-m", "pyjab_mcp"] }
  }
}

Claude Desktop reads %APPDATA%\Claude\claude_desktop_config.json, and takes this (double the backslashes — JSON needs it):

{
  "mcpServers": {
    "pyjab-mcp": { "command": "C:\\Users\\you\\.venv\\Scripts\\pyjab-mcp.exe", "args": [] }
  }
}

Claude Code writes the same entry for you: claude mcp add pyjab-mcp -- pyjab-mcp.

docs/CLIENTS.md has Cursor and VS Code as well, the config file each one reads, the Windows path problem that makes most setups fail, and what to do when a tool answers with an error.

Start with java_diagnostics. It reports which part of the environment is missing — Java Access Bridge not enabled, no bridge DLL, a 32-bit interpreter (which makes it search for the 32-bit bridge DLL) — instead of leaving a tool call to fail with a locator error.

What it does, and what it cannot

Java Access Bridge exposes what an application declares about itself: a tree of roles, names, values, states and the actions a control supports. That is what this server drives, and it sets the boundary of what it can reach.

It works on Java applications that expose accessibility — Swing and AWT applications and applets running as desktop applications. JavaFX is not supported: it publishes its accessibility through UI Automation rather than Java Access Bridge. Buttons, menus, tables, trees, lists, text fields and their values are all readable and actionable, and java_get_table reads a table's cells through the table's own interface rather than the pixels.

It cannot reach, because the information is not there to read:

Also worth knowing:

• The application must be JAB-enabled before it starts. Enabling JAB afterwards does not add the tree to a process already running; restart it.
• This server only attaches. It never launches an application, and it never terminates one — JABDriver.detach() releases the binding and leaves the process running.
• A picture is not structure. java_screenshot is for looking; anything you want to act on has to be found in the tree, because that is where handles and names live.

Deliberately not in scope

The list above is what the substrate cannot reach. This is the other kind: things this server could do and does not, so that somebody evaluating it against a hosted alternative knows what they would be giving up — and what would have to change to add each one.

Each is a deliberate boundary rather than an unfinished feature, and the ones that are architectural (transport, multi-tenancy) would change the shape of ServerState rather than sit beside it.

Support

• Free support — bugs, questions and feature requests go to GitHub issues. Include the output of pyjab-mcp --version and of the java_diagnostics tool: between them they answer most environmental questions before they are asked.
• Commercial and enterprise support — deployment inside a restricted network, adaptation to a particular application, or an SLA: gaozhao89@qq.com (the address in this package's metadata). The open-source package is complete and unrestricted; there is no licence key, no quota and no feature held back for it.

Verifying it

The suite that runs on every push cannot drive an application: it needs Windows, a JDK, a desktop session and a running Swing app. That layer is tests/e2e/, skipped unless PYJAB_MCP_E2E=1, and windows-gui.yml runs it on a hosted windows-latest runner — by hand, or on a weekly schedule. It starts a second JVM after the bridge has been idling, because a message pump on the wrong thread passes everything else and fails exactly that.

gh workflow run windows-gui.yml -f java=17     # the end-to-end run and the JDK matrix
gh run watch

It is not a status check: nobody runs it unless somebody asks (or the week comes round), so a green default CI run still says nothing about a live window.

Releasing

Tag-driven: bump src/pyjab_mcp/__init__.py, move the changelog's Unreleased section under the new version, set server.json, then python tools/check_release_version.py vX.Y.Z and tag. docs/RELEASING.md has the whole procedure, the two one-time setups (a PyPI pending publisher and the registry namespace), and what the four release jobs do.

Working on it

python -m pytest tests/                    # the portable suite, any platform
python tools/check_pyjab_api_surface.py    # every pyjab call is public, released, recorded
python tools/check_documented_surface.py   # docs/ARCHITECTURE.md's tool table and the code agree
python tools/check_local_only_files.py     # maintainer notes stay unpublished
python tools/check_pyjab_readiness.py      # which pyjab APIs the pin expects, and whether it resolves them
python tools/check_pyjab_changelog.py      # every marked pyjab change has an answer here
python tools/check_published_references.py # shipped files do not cite unpublished ones
python tools/check_changelog_immutable.py  # a released changelog section is frozen
python tools/check_release_version.py vX.Y.Z  # the four places a version lives agree
python -m pytest tests/contract/           # the pyjab shapes this project depends on
python -m build && python tools/check_dist_contents.py --dist dist

Two more -- check_release_l3.py and check_release_unpublished.py -- only mean anything with a tag or a run to read, so they run inside the release pipeline rather than by hand.

The guards are the project's rules made mechanical, in the same shape pyjab uses for its own: a registry of what is allowed, checked in both directions, so adding a call site fails until somebody writes down why it is acceptable. docs/ARCHITECTURE.md lists them and what each one enforces.