Trace page and the gateway’s request trace

Part of the developer notes; the index is in CLAUDE.md.

The Trace page (under Logs) shows what each request cost and how requests relate, live while the server runs. It reads the gateway’s in-memory trace (docker/server/reqtrace.py, the latest 2,000 requests; nothing is written to disk).

• What a record holds: device, timings (wait for a slot, first token when streamed, total), prompt, cached and written tokens, status, and the labels the client attached. No text: the conversation’s shape is kept as one hash per message prefix. oppx labels each request in the body (oppx_trace: session, turn, step, kind of call, the tool the previous action chose and how long the client ran it, context estimate, events such as tool output cut or a repeat); the gateway strips them, like oppx_utility. /v1/info says "trace": true and oppx sends labels only then (OPPX_TRACE), since an older gateway would pass them on to SGLang. Other clients show up as single requests without turns.
• Capture content (a checkbox, off by default; PUT /admin/trace/content): also keeps the end of the latest message, the reply and tool titles (they name files and patterns), 6,000 characters each, in memory. Switching it off drops them. This is the one place the server holds prompt text, and only while its owner asks for it.
• Views: turns (newest first, with stuck / failed / context-cut badges); where the time went for a turn, as a flame-style chart (a time axis, the turn’s own span, then one span per request placed on the turn’s clock so they step down and to the right, each named beside it and coloured by phase: the model reading the prompt, the model writing, the client running the tool, waiting; under it the phases as a table with their share of the turn, including the time between requests; for calls that aren’t streamed the read/write split is estimated from the tokens written at the streamed replies’ speed, and says so); context per request (prompt tokens stacked as from-cache and computed against the window, ▾ where the client cut context or the cache held under half of what had been sent before); the requests as a table (the same numbers without hovering); prefix cache.
• One colour per meaning across the page: strong blue is the prompt the model had to read (time in the first chart, computed tokens in the second), pale blue the part that came from the cache, orange is writing, green the client. The first version used blue and orange for different things in the two charts. The palette was checked with the dataviz validator on both themes’ surfaces. The page lays itself out by its own width (container queries): under 860 px the turn list moves on top and the tiles go two by two; it was rendered at 1280, 760 and 460 px.
• Prefix cache (RadixAttention): SGLang 0.5.21 has no endpoint for its tree (only pretty_print to its own log), so the page shows two things. SGLang’s own counters from /metrics: tokens cached for reuse (kv_evictable_tokens), in use by running requests, evicted so far, and the recurrent-state pool. And the tree of shared prefixes rebuilt from the requests (Trace.tree: a node is a run of messages every request through it shares; it ends where requests part or one ends), sized in tokens with each request’s own characters-per-token ratio and coloured by how recently it was used. It shows what could be reused; expected_cached (the part of a prompt an earlier request already sent) against SGLang’s reported cached_tokens shows what was. On a real turn they agreed within a few tokens once warm (2,370 expected, 2,368 cached).
• Admin API: GET /admin/trace?after=<seq> (new records and unfinished older ones, plus the pool), GET /admin/trace/tree, PUT /admin/trace/content, DELETE /admin/trace. App commands get_trace, get_trace_tree, set_trace_content, clear_trace.
• Tested: scripts/test_reqtrace.py (7 tests, no dependencies, in CI); the gateway in the backend image against a fake model (streamed, plain, a refusal, content on and off, no admin token: 401); and a second gateway with the new code in front of the running split Qwen3.8-27B with a real oppx -p question (route, action, answer, reply: all labelled, the tool attributed, no text kept). The page itself was checked with made-up data in both themes; not yet seen against a live backend in the app (that needs the server restarted on the new image).
• A new build file: reqtrace.py is in docker::BUILD_FILES and the Dockerfile, so the image tag changes: every cluster member must have the same code, or its split worker can’t build the image the host asks for.