Skip to content

mcp: an execution failure is a tool error (isError), not an answer (#2006) - #2007

Open
G6KlayWang wants to merge 1 commit into
AxiomCodeAI:0.2.3from
G6KlayWang:audit/0.2.0
Open

G6KlayWang wants to merge 1 commit into
AxiomCodeAI:0.2.3from
G6KlayWang:audit/0.2.0

Conversation

@G6KlayWang

Copy link
Copy Markdown

Fixes #2006

What changed

A call the CLI could not carry out came back as an ordinary result on both server paths; only schema-invalid arguments carried isError. run() in plugins/axiomcode/mcp/server.py now raises the server's own ToolError with the same words, which the SDK and the fallback both mark isError: true, so the client sees the failure and the model still reads the correction.

  • plugins/axiomcode/mcp/_fallback.py: a ToolError class. _handle returns it as {"content": [...], "isError": true} with the words alone; any other exception keeps the TypeName: prefix, as before.
  • plugins/axiomcode/mcp/server.py: ToolError is resolved from _fallback when the fallback serves and from the SDK's exceptions module otherwise. repository(args, cwd) names the repository a call asks about; failed(r, repo) decides what a failure is (table below); run() raises on a failure and returns the text otherwise; answer() passes both outcomes through plain(), so a flag the tool cannot take is named in neither. AXIOMCODE_MCP_TIMEOUT (seconds, default 900) exists so a test can see the timeout.

building below is ax_fresh.building(realpath(repo)), the build-lock probe run() already used on a timeout.

condition isError text
bash cannot start (OSError) yes "axiomcode could not start bash (…)"
timeout, build in progress no the building note
timeout, no build yes "axiomcode did not answer within N s …"
exit 0 (even with empty stdout) no stdout (+ graph refresh: lines) or "(no output, exit 0)"
exit 1 with words, no traceback no e.g. path's "no chain of resolved calls connects …"
exit 1 with no output yes "(no output, exit 1)"
exit 2, build in progress no the building note
exit 2, no build (a refusal: nothing named X / nearest names, no graph, no such directory) yes the refusal + "(exit 2)"
stderr holds "Traceback (most recent call last)" yes the output + "(exit N)"
exit < 0 (signal) or any code outside {0, 1, 2} yes the output + "(exit N)", or "(no output, exit N)"

Why a refusal is an error: a wrong value ("nothing named X") is the same class as a wrong parameter name, which the server already refuses with isError: true (#1567, tests/mcp.py check_arguments). The words, the nearest names and the fix, still reach the model as content; on the SDK path they carry the SDK's own "Error executing tool : " prefix.

The orphaned-install note (#1973), which run() returns before starting bash, is left as an ordinary result.

Why the goldens moved, or why they did not

No golden moved: no rule changed. The engine and the parser are untouched.

Evidence

Synthetic only. tests/mcp.py check_failures drives the server with a fake bash per case, in every launch mode, on the fallback and, with AXIOMCODE_SDK_PYTHON, on the SDK. The four cases that must NOT be errors are the controls.

case fake bash does isError words checked
missing no such file yes could not start
boom "boom" on stderr, exit 7 yes boom, exit 7
slow sleeps past the timeout yes did not answer within
crash a traceback on stderr, exit 1 yes Traceback
no chain "no chain of resolved calls connects …", exit 1 no no chain
empty ok exit 0 no (no output, exit 0)
building the building note, exit 2, build lock held by another process no being built
refusal "nothing named zz … close: a, b", exit 2 yes close:, exit 2
silent 1 exit 1 yes (no output, exit 1)

Runs on macOS arm64, Node 22.23.3, Python 3.13.10, mcp 2.3.0 in a venv:

  • tests/mcp.py: ok on v0.2.0 + this patch and ok on 0.2.3 + this patch, failures checked in all seven launch modes (bin/axiomcode mcp, server.py fallback, server.py SDK, .codex-plugin/mcp.json, .cursor-plugin/plugin.json, .mcp.json, .mcp.json with the SDK python); each mode prints which half served it. Without the venv the SDK modes are skipped out loud.
  • tests/latency.py --no-engine: the timeout check passes as a tool error; 17 of 18 held. The one failure, verbs:, splits a logged command on whitespace and fails under a checkout path containing a space; it fails the same on pristine v0.2.0 and passes in a path without one. Harness defect, separate issue.
  • tests/repo_arg.py ok; tests/front_door.py 21 of 21; tests/corrupt_graph.py 0 failures over 26 checks, including "mcp: the tool answer says the graph is corrupt, no traceback" and "mcp: once rebuilt the tool answers again".
  • Manual, AXIOMCODE_BASH=/nonexistent/bash, fallback and SDK: isError: true, words unchanged.

What to watch:

  • Corrupt-graph race. ax_contract._corrupt starts a detached rebuild and exits 2 "a rebuild is running" before that build may hold the lock; failed() can then mark that one call isError with the correct words. The next call answers normally. tests/corrupt_graph.py tolerates it.
  • Windows. check_failures is POSIX-only (fake /bin/sh scripts, fcntl); the AXIOMCODE_ARGV branch of run() is untouched. Worth a run on the Windows gate.
  • Clients that stop on isError. Any such client already stopped on the unknown-argument refusal; Claude Code, Cursor and Codex hand the content to the model.
  • tests/mcp.py is not run by .github/workflows/ci.yml; the runs above are local.

Checklist

  • The engine suites are untouched; the plugin suites named above pass locally
  • No golden moved
  • Validated on nine failure shapes with four controls, on both server paths and every launch mode
  • Labels bug, plugin, mcp (I cannot set them; please add)

…xiomCodeAI#2006)

A call the CLI could not carry out — bash missing, the verb crashed or was killed, an exit no verb
uses, silence, a timeout with no build running, a refusal (exit 2) — came back as an ordinary result
on both server paths; only schema-invalid arguments were flagged. run() now raises the server's own
ToolError with the same words, which the SDK and the fallback both mark isError, so the model still
reads the correction. An answer, path's "no chain" (exit 1 in words) and the note that a graph is
still being built (exit 2 while the build lock is held) stay ordinary results.

AXIOMCODE_MCP_TIMEOUT (seconds) exists so a test can see the timeout. tests/mcp.py checks nine
cases per launch mode, on the fallback and, with AXIOMCODE_SDK_PYTHON, on the SDK.
@swapnilpaliwal-sd

Copy link
Copy Markdown
Contributor

Running tests and validating #2006. Can you please check it on Windows and validate it? In the mean time I'm going to run e2e tests to ensure no regressions.

@G6KlayWang

Copy link
Copy Markdown
Author

Windows: works

Tested on Windows 11 with Git Bash 5.2.32, Python 3.11, mcp 2.3.0 and the released engine 0.2.2 (engine-win32-x64), comparing 14eaf4c8 with its parent.

Synthetic, 44 of 44 hold

This is check_failures ported to Windows: real Git Bash, a fake CLI fed through AXIOMCODE_ARGV, and the build lock held with msvcrt. It covers 11 cases in 4 modes: fallback, SDK, launch.js, and launch.js with the SDK python. The results are the same in every mode. The text column is from the fallback run.

case fake CLI does parent PR text the client gets
missing AXIOMCODE_BASH=C:\nope\... false true axiomcode could not start bash (C:\nope\Git\bin\bash.exe): [WinError 2] …
boom stderr boom, exit 7 false true boom (exit 7)
slow sleeps past a 3 s timeout false true axiomcode path did not answer within 3 s. Nothing was changed; …
crash traceback, exit 1 false true Traceback (most recent call last): … ValueError: bad (exit 1)
killed kill -9 $$ false true partial output (exit 2304)
refusal exit 2, no build false true nothing named zz is declared in this graph. Exact names that are close: a, b (exit 2)
silent 1 exit 1, no output false true (no output, exit 1)
answer exit 0; the task holds ' and " false false 1. checkout -> save_cart (cart.py:12) … (args arrived intact)
no chain words, exit 1 false false no chain of resolved calls connects a and b in either direction
empty ok exit 0 false false (no output, exit 0)
building exit 2, build lock held false false the graph for … is being built (starting; 0m01s so far). …

Real engine, 13 of 13 hold

I ran 0.2.2 with this PR's two files on a small Python repo through launch.js. The repo is checkout → save_cart → store.write, plus an unrelated audit_log.

tool call parent PR text the client gets (PR)
context "how does the user's cart get saved" false false where this task's words land: shop … task terms: cart …
path checkout write false false 1. shop/cart.py:24 [resolved · hop 1/2 checkout → save_cart] …
path audit_log checkout false false no chain of resolved calls connects audit_log and checkout in either direction …
impact save_cart false false 1. shop/cart.py:24 [resolved · checkout] …
impact nothing_named_this false true Error executing tool impact: nothing named 'nothing_named_this' is declared in this graph, and nothing close to it. …
path zz_missing checkout false true Error executing tool path: nothing named 'zz_missing' is declared in this graph, …
impact save_cart, bad AXIOMCODE_BASH false true Error executing tool impact: axiomcode could not start bash (C:\nope\Git\bin\bash.exe): [WinError 2] …

Both modes were served by the SDK and gave the same results. The "being built" control is covered only by the synthetic run, because the real build finished in about 12 s.

Separate, and already true before this PR

On Windows a timeout doesn't return on time. Killing bash.exe leaves its children holding the stdout pipe, so subprocess.run(capture_output=True, timeout=3) returned after about 20 s, once the child exited. The result is still correctly isError. The fix is to kill the whole process tree (taskkill /T or a Job Object). I can open an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants