Building an app end to end: the CHO-8 audit and scripts/eval_app.py

Part of the developer notes; the index is in CLAUDE.md.

• Building an app end to end (CHO-8: search the web, open a Linear issue, scaffold and write a Vite + React app, start it, check it in the browser, close the issue; split Qwen3.6-27B at 64k, a real chat driven through tmux). What it broke on, and the fixes:

  • run has no terminal and leaves nothing behind (Workspace.run): stdin is closed, so a command that asks a question ends instead of waiting on the user’s terminal unseen (npm create vite sat there for the whole 120 s); it runs in its own process group, which the timeout or an interrupt kills (only the shell was killed, and npm create stayed alive); a timeout returns the output until then, and says to pass answers as flags or use start.

  • start (Workspace.start): a command that keeps running. Its output goes to a temporary log; the call returns once it has printed something and gone quiet for 2 s (15 s at most), or with the exit code if it ended. Called again with the same command it returns only the new output. Everything started is stopped when the chat exits (stop_jobs, at exit; not on SIGHUP or SIGKILL). It is asked like run, and the question still reads “Run this command?”.

  • Lookups may repeat after the files changed: an applied edit, a run or a start clears the repeat guard. Before, a read after a half-applied edit came back as “You already did exactly this at step 1” three times and the turn ended stuck, the model convinced the reads were cached.

  • create replaces a whole file that was read (REWRITE_PROMPT; the old text goes to the diff and /undo). As SEARCH/REPLACE, replacing Vite’s starter App.jsx meant copying all 122 lines first; the model wrote one sentence and no block.

  • A cut-off action is asked for again (_action): the grammar allows any whitespace between fields, and one action filled its 400 tokens with newlines after "missing" (11 s) and was then taken as answer. Action calls stop at a blank line, and one without a tool is retried once at temperature 0.7.

  • An empty plan is asked for again (_think), first character constrained, as for edit text: “(no plan written)” showed whenever the plan prompt was answered with the action.

  • Connector calls that only read are named as such in the reply’s facts: asked to mark the issue Done, the model called linear.get_issue and reported “Status changed from Backlog to Done” (it was still Backlog). Asked “Did you actually…?” on the new build, it checked and made the change.

  • The browser line says it is Playwright: told to “run Playwright”, the model went to npm install -D @playwright/test instead of the browser action.

  • OPPX_DUMP_REQUESTS now works for the agent engine too (each call with its reply and usage; agent._dump).

  • Result: all five steps done over four sessions with these hints from the driver: scaffold without prompts, the right Algolia host, “the server is already running” (it wanted npm run dev -- --host 0.0.0.0 through run), and the question about the issue. The app builds and loads 30 stories; typing LLM and submitting filters them with no console errors. Left as the model wrote it: it searches on submit, not as you type; it shows Algolia’s relevance order, not the front page; it invented an Algolia application id and key before being corrected; lucide-react is installed and unused.

  • “Improve the app” (five changes, then lint, build, dev server and a browser check), same model. The first two tries ended with nothing changed and “Let me start by making the changes.” followed by a raw {"enough": false, …} line in the reply. The model wrote "enough": true meaning “enough to start”, and the only tool that goes with it is answer. Fixes: the system prompt says that for a change request enough stays false until the edits are made; a change request that chooses answer a second time with nothing changed gets one step with only edit, create, run and start on offer (with lookups and plan on offer too, it planned twice more and answered again); an action tacked onto the end of a reply is neither shown nor kept (_ACTION_TAIL). Third try, no hints: all in one turn (56 model calls), lint and build clean, typing “Rust” filtered and highlighted with no console errors. It reported “front page by default” as done though the request to Algolia is unchanged: a change the harness can’t check is still only as true as the model.

  • A resumed conversation is shown (app.show_history, after the welcome box on -c/-r): the earlier messages and replies (the latest 30, 40 lines each at most; slash commands and our appended notes left out), then a line saying the steps in between aren’t kept. Before, a resumed chat looked like a new one. Seen in a terminal on the CHO-8 session (10 messages).

  • As a repeatable eval (scripts/eval_app.py, CHO-12): both prompts in a fresh repository, in print mode, with nobody helping, and scored by the script instead of the reply: it installs and builds the app, serves the build, opens it in headless Chrome (eval_app_check.mjs: the requests made, the story links, the console), types the keyword, and reads the issue tracker’s state. What a user would have said “always” to is allowed beforehand (npm, npx, mkdir, cp, mv, the browser, the tracker); Linear is a small stdio connector the script writes (same tool names, simple schemas), so runs leave nothing in a real workspace; a run gets “Continue…” up to twice while a check fails. The improve task starts from the app as the first run left it (scripts/fixtures/hn-dashboard, git-ignored: dummy apps are never committed, so a fresh clone has only the build task). --agent claude runs the same prompts through Claude Code with the same tracker and browser server.

  • First numbers (split Qwen3.6-27B, 64k; five runs each; Claude Code one run each):

    	oppx, before the fixes below	Claude Code
    build: every check passed	4 of 5 runs (3 in the first message), 2.0–3.9 min	1 of 1, 11.8 min, $1.23
    improve: every check passed	1 of 5 runs, 3.1–11.5 min, 44–140 model calls	1 of 1, 10.9 min, $0.90

    The build failure: the page shows nothing until a keyword is typed. The improve failures: “front page by default” reported as done with the request to Algolia unchanged, in 4 of 5 runs. In the run looked at closely a search result two steps earlier read “All stories that are on the front/home page right now: …/search?tags=front_page”; the model then wrote tags=story and called that the front-page endpoint. Claude Code’s apps used tags=front_page both times.

  • Fixes from those runs:

    • A review before the reply (REVIEW_PROMPT, CHO-14): on a request with several parts (numbered or bulleted items, or 400 characters; several_parts), once the model wants to answer, it writes one line per thing asked for with its evidence: an added line of the turn’s diff (Agent._diff, from the texts kept before each file’s first change, 9,000 characters at most), or a command or browser step and what it returned; otherwise NOT DONE. A NOT DONE line is to be done before answering, and the reply may not call it done. One more model call (500 tokens at most) on such turns only.
    • A read after a refused edit is not a repeat: the refusal says “Read the exact lines again and retry”, and that read came back as “You already did exactly this”, three times, until the turn ended stuck.
    • 80 steps for a turn that is changing files (MAX_STEPS_CHANGING; 40 otherwise): 4 of 5 first messages ended at step 40 mid-work, before their own review.
    • A connector’s name as a shell command is refused with what it is (_not_a_connector): with the browser’s tools on offer the model chose run with playwright browser navigate http://localhost:5173/, was declined, and went to the web to find out how to use Playwright.
    • Web searches are bounded (see the web-search topic, Search guard): 41 searches in that turn, the last ones for “a”, “b”, “c”.

  • With the review step (before the other four): improve passed every check in 3 of 5 runs, twice. Where the review ran, it caught the front page each time (“NOT DONE. The initial fetchStories(‘’) still queries tags=story…”); the failed runs never reached it (stuck at the step limit, or in the search loop), or knew it was undone and couldn’t find the parameter with search down.

  • With web search off (--no-web, the eval’s default since the engines blocked the server): the build task scored 0 of 7 twice. The system prompt still listed web_search, the action list didn’t have it, and the model typed web_search --query '…' as a shell command nine times in a row; a declined command was never a repeat, because run is exempt from the repeat guard. Fixes: without web search the prompt doesn’t list the tool and says it is off; one of the agent’s own actions typed as a command is refused with what it is (_not_a_program, which also covers connector names); a command that was declined or refused is a repeat when asked for again in the same request (declined); and only a command that really ran clears the repeat guard (Agent._ran). One run after that: 6 of 7 (the app’s index.html names a /favicon.svg it didn’t keep).

  • What the review can’t do: with web search off, improve failed “front page” in 5 of 5 runs. The review ran every time, and in most runs it accepted “fetches with no query” as the front page: it checks a claim against the diff and against what tools returned, and nothing in view contradicted the belief. One run asked for the parameter to be confirmed instead of claiming it.

  • The eval stops by itself when two runs in a row fail the same checks (--no-stop runs them all): a batch that repeats one failure is GPU time for nothing. Run one run and read it before a batch.

  • A scorer mistake worth knowing: Chrome asks for /favicon.ico when a page names no icon, and the static server’s 404 is logged as a console error. Two “console errors” in the first batch were that; the scorer now puts an empty one in the folder it serves.

  • Still open: a resumed session restores only the messages, not what tools returned; a search step shows “Web results (untrusted reference text):” as its summary instead of a count; a start for a second copy of a server that already runs isn’t caught; none of this was rerun from scratch in one session on the final build, nor on Qwen3.8.