Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Cluster: discovery, roles, joining, split serving, model sync

Part of the developer notes; the index is in CLAUDE.md.

  • Concurrency (2026-10-16): a member’s report (every 5 s) asks its four probes together (the backend’s and the worker’s new log lines, collect_inventory, collect_serving, check_worker), so it takes as long as the slowest. The log lines are still queued backend first. take_over and the host’s invitations register every invite, then send them all at once, so one unreachable peer no longer holds up the others for its timeout. Model sync uses the parallel download (4 files at once).

Cluster

Servers on one network form a cluster: one host and its members. With the split strategy they serve one model together (see Split serving below). The service is cluster::Cluster in core. It runs inside the app, or headless as oppxs run (crates/openphalanx-server); both use ~/.local/share/openphalanx/cluster/, so run one per machine.

  • Discovery: every server broadcasts a beacon (id, name, role, certificate fingerprint) on UDP 9093 every 3 s. A server is listed under Servers on this network only when its beacon is under 10 s old and a TLS health check, pinned to the fingerprint it announced, succeeds. Stopped or unreachable servers drop off within about 10 s. Broadcast only crosses one LAN segment, not Tailscale.

    • Fast join: a server that starts (app opened, or oppxs run) sends a query beacon twice; the others answer by unicast at once and check it immediately, so it is listed in about 0.3 s (measured on the 4090) instead of waiting for the next 3 s round.

  • This machine is always “This machine” in the app (cluster panel, Machines, charts, logs, VRAM pool, the split summary) and in oppxs status. Its name is only what the other servers show for it (oppxs run --name, or Rename in the panel, which now says “seen by others as …”). Two machines with the same hostname had ended up with the same name, and the host looked like the member.

  • Other servers are shown by address, or by a nickname given here (Cluster::label: nickname, else the host part of its URL, else the name it gives itself). Nicknames are per machine, keyed by server id, saved in state.json (nicknames), so they survive restarts and updates; set_nickname with no name removes one. Every place a server is named gets the label from the cluster view (MemberView.name, CandidateView.name, the host in a member’s role, invitations, the Rust-built messages and the split summary), so charts, logs and the VRAM pool follow; announced keeps the server’s own name for a tooltip. In the app, Nickname / Edit beside a server in the cluster panel (NodeName.svelte: save, remove, cancel); headless, oppxs nickname <server> [name]. The same store names paired client devices on the Trace page (by device id; a device has no address there, so its default stays the name it paired with).

  • One server process per machine (core::ports): the app and oppxs share ports 9092, 9093/udp and 9094. A second one used to fail once and stay without a cluster service (invisible to the others). Now the app keeps retrying (serve_cluster: the API once the port is free; discovery retries its own bind), and the leftover can be stopped on request: ports::holders reads ss -H -ltnup, ports::free sends SIGTERM (so it stops its model and says goodbye) and SIGKILL after 8 s, only to processes named oppxs or openphalanx; anything else on those ports is named and left alone. oppxs free-ports [--yes]; the app’s Free the ports… asks first. Listing was run against the live app; the stopping path has unit tests for the parsing only.

  • Roles (state.json): standalone, host or member. A host with no members is standalone again.

    • Clients pair with the host only: members hide the pairing section, refuse new_pairing_code, and skip the start-up pairing code.

  • Joining needs consent on both sides:

    1. Add to cluster: the host sends an invitation (a single-use secret bound to that server, valid for 10 minutes).
    2. Approve on the invited server: in its app, or with oppxs approve, after checking the host’s fingerprint. It then joins over a connection pinned to that certificate and gets a 256-bit member token; the host stores only its hash.
    3. Trusted hosts: a server remembers the hosts it approved, and accepts their invitations without asking next time.

  • Make host (hand-over):

    • The current host asks a server (on the network, or one of its members) to take over. That server approves (“Become host”) and invites everyone in the request.
    • The old host and its members accept automatically: the old host asked for it, and members learn the new host’s fingerprint from the old host in their report replies.
    • Clients must pair with the new host.

  • Strategy (cluster::Strategy, the host decides; members show the host’s choice from report replies):

    • Split model (default): one model across the servers. Bigger models, about single-GPU speed, needs every server up and a wired network.
    • Replicas: a full copy per server. More concurrent users and resilience, no bigger models.

    It’s colour-coded in the cluster panel (violet / blue) with an ⓘ tooltip; oppxs strategy split|replicas does the same.

    • Locked while serving: set_strategy refuses a change while this server is serving (“Stop the server before changing the strategy”), and the panel shows the other option faded with a lock.
    • Pooled VRAM (split): on a host with online members, the Models page measures fits against the sum of each server’s free VRAM (its GPU with the most free memory, as last reported), and every server’s runtime memory is added to the need (vram::split_across). A donut (VramPool.svelte) shows each server’s share and the selected model’s need. The pre-flight VRAM check follows the split plan (plan_split in the app).

  • Split serving (cluster step 3, split.rs; pipeline parallel with SGLang --pp-size N --nnodes N):

    • When: the host, with the split strategy and online members, starts a model that doesn’t fit its own GPU. A model that fits runs on the host alone: faster, and it keeps serving if a member drops out.
    • Plan (split::plan): layers are shared out in proportion to each server’s free VRAM after its fixed costs (runtime memory on every server, the input embedding on the first, the output head on the last), at least one each. Layer count and embedding size come from the model’s config.json (text_config for multimodal models). Passed as SGLANG_PP_LAYER_PARTITION. SGLang takes the minimum KV capacity across ranks, so each server sizes its own --mem-fraction-static from its stage.
    • Rank 0 is the host’s normal backend on the host’s network (--network host; NCCL and torch connect back to the address each rank announces, which a bridge would hide). The gateway then binds the admin API to loopback itself (ADMIN_HOST=127.0.0.1), SGLang moves to 9096, and SearXNG is published on 127.0.0.1:9098 only.
    • Members run a worker: the host puts a WorkerOrder (run id, model, rank, partition, host:9100 rendezvous, stage) in each member’s report reply. The member starts openphalanx-worker (the same image with start-sglang.sh as entrypoint, no gateway, host network, weights read-only) with its own GPU and LAN interface (NCCL_SOCKET_IFNAME/GLOO_SOCKET_IFNAME), and reports starting/running/failed. An order without a worker, or leaving the cluster, stops it. Its log goes to the host’s Logs page prefixed [split worker]. The worker starts in its own task, so the member keeps reporting while it builds the image (the first time, a ~16 GB download): awaited inside the report loop, the host saw no report for 12 s and paused on every first start. A member whose service restarts adopts a running worker with the order’s run id (docker label) instead of recreating it.
    • Before starting the host checks every member: the same OpenPhalanx version as the host (cluster::split_version_problem; a 0.3.0 member used to fail later, inside its worker, with “the backend image … isn’t on this machine”), NVIDIA runtime present, and the model synced (“still downloading (45%)” otherwise). Members need the backend image too; the worker fails with a clear message if it’s missing.
    • While serving: a member dropping out pauses serving; a failed worker stops it with that member’s reason. Stop clears the orders, so workers stop too. The Server page’s model pill shows the split (“rig-3090: 40 layers (…) · laptop-4090: 24 layers (…)”). The orders are saved to cluster/workers.json, so reopening the app on a running split keeps it (it used to withdraw the orders, which stopped the member’s rank and crashed rank 0); a member not heard from since the service started counts from that start, not from when it joined. Orders left from a run that is gone are dropped. While serving, the Models page counts each member’s VRAM as it was before its worker started.
    • Runtime memory per stage: each server is charged runtime memory for the weights it holds (vram::runtime_overhead), not the whole model’s. Charging the whole model’s to each left Qwen3.6-27B with a 30.9k-token KV cache at a 32k context and 5–6 GB unused per GPU. /v1/info also reports min(context, SGLang's KV tokens) as context_length, so clients budget against what the server can actually hold.
    • Same recurrent-state pool on every rank: --max-mamba-cache-size 16 (docker::SPLIT_STATE_SLOTS; read only by models with linear-attention layers). SGLang otherwise sizes the pool from each GPU’s free memory, and rank 0 hands out slot numbers from its own: with 23 slots on the 3090 and 20 on the 4090, Qwen3.8-27B died mid-request with “index out of bounds” (IndexKernel.cu) on the 4090, usually on the first request of a new conversation, and took the backend with it (three times in one afternoon of agent runs). With 16 on both, 20 fresh conversations and the edit eval ran without a crash.
    • Same settings on every rank: --chunked-prefill-size 2048 (docker::SPLIT_PREFILL_CHUNK; SGLang otherwise picks 4096 on a 24 GB card and 2048 on 16 GB, and the smaller rank crashed reshaping a 4096-token chunk), and the model’s dtype override if the catalog has one.
    • Ports between the servers: 9100 and the next 16 on the host, plus NCCL’s ephemeral ports both ways; keep the servers on one LAN without a firewall between them.
    • Trusted LAN only: the rendezvous (9100) and NCCL/gloo traffic aren’t authenticated, and SGLang passes objects between ranks with pickle (broadcast_pyobj in sglang/srt/utils/common.py). Anyone who can reach those ports while a split model starts or runs can potentially execute code on the servers. That’s inherent to multi-node SGLang; the cluster panel’s Split tooltip says so.
    • Planning by hand: cargo run -p openphalanx-core --example split_plan -- rig-3090=22.1 laptop-4090=15.5 prints the plan for the selected model and context (the host first, free GiB per server).
    • Verified by hand (containers with the app’s arguments, 3090 + 4090 laptop on gigabit Ethernet): Qwen3-8B-AWQ 20/16 layers, about 100 tokens/s, a 6.1k-token prompt in 2.3 s; Qwen3.6-27B-AWQ 42/22 layers at 32k context (18.6 of 24 GB and 11.0 of 16 GB used), correct on a 16.8k-token prompt in 13.7 s. The full app flow (Start in the GUI on the host, worker orders, Stop, the app reopening mid-run, a member’s service killed and restarted) was run on v0.5.0: about 4 minutes to serving, including the member building its image.

  • Model sync:

    • The cluster’s model follows the host’s selected model (set_desired_model on select, on every pre-flight check and on Start; repo plus pinned commit; local-folder models are skipped). The host sends it to members in every report reply.
    • A member that already has it (the app’s models folder or the HF cache) reports “ready”; otherwise “missing”. It downloads only after the user agrees on the host: the Server page asks “laptop-4090 doesn’t have … (20.4 GB). Download it there?”, and approve_download lets members fetch it (verified, resumable), reporting progress and errors (retried). Choosing another model resets the approval.
    • Start waits up to 8 s for members to report on a newly chosen model before checking the split, so a fresh selection doesn’t fail on a stale report.
    • The Machines table shows each member’s status; headless, use oppxs model <catalog-id>|none.
    • Verified: the 4090 downloaded Qwen2.5-Coder-7B-AWQ (5.2 GB, about 92 MB/s, verified, marker written), and reported “already on this machine” after a restart.

  • Member logs: members queue log lines (their backend’s docker logs since the last report, plus cluster events such as joins and downloads) and send up to 400 per report. The host keeps 3,000 per member. The Logs page has a machine dropdown; headless, use oppxs logs <member>.

  • One controller (start guard): whoever presses Start first becomes the controller; the other side can’t start too.

    • Cluster::claim_start runs before pre-flight. A standalone server or host just marks itself serving (refused while it is handing control to someone). A member first takes the host role over through POST /cluster/v1/take-over, which the host refuses with 409 while it is serving or already handing over. Both checks happen under one Control mutex on the host, so two simultaneous presses can’t both win.
    • The app’s monitor keeps set_serving in step with the backend (starting or running). The host sends host_serving in report replies; a member then has locked_by_host in its view, and its power button turns violet with a lock and “ is running the cluster”. All it can do is leave the cluster (or close the app).

  • Split pause: with the split strategy, a host that is serving stops its backend when a member has said goodbye or hasn’t reported for 12 s (dropped_members), and the Server page shows “Paused: dropped out of the cluster…” (state paused, red). Start again once it’s back, or remove it. Members send POST /cluster/v1/bye when the app exits (RunEvent::Exit) or oppxs gets Ctrl-C, so a clean exit pauses at once.

  • Members report to the host every 5 s (/cluster/v1/report): inventory (CPUs, RAM, GPUs with load and temperature, Docker/NVIDIA runtime) and, if their backend runs, what it serves (model, gateway and inference metrics). The host shows a member offline after 20 s without a report. A member that gets 401 (removed, or the cluster was dissolved) becomes standalone.

  • App:

    • Cluster panel on the Server page: pending invitations and host requests (Approve/Decline, with the sender’s fingerprint), this server’s role with Rename/Leave/Dissolve, and the servers on the network (Add to cluster / Make host).
    • In this cluster cards: the host has a red Remove on each member, a member has Leave on its own card. A removed server shows up under “Not in this cluster” again, and Add to cluster brings it back without asking there (it trusts the host it approved before).
    • Machines table: every server, with Make host/Remove for members.
    • Charts show one machine at a time (a dropdown); the tiles sum the cluster (six of them, always split evenly: 6 in a row, else 3 + 3, else 2 + 2 + 2, by the dashboard’s own width through a container query; auto-fit left GPU memory alone on a second row at in-between widths). Eight panels (Dashboard.svelte, sampled every 2 s into metrics.svelte.ts, 10 minutes kept): generation throughput, requests in flight, time to first token, queue depth, client requests, prefix cache hit, KV cache in use, GPU load and temperature.
      • Time to first token (p50, p95, p99): measured by the gateway, from a request’s arrival to the first generated token it sends back (Metrics.note_ttft), so waiting for a slot and the search router are included, as the client experiences it. Exact percentiles (nearest rank) over the streamed requests of the last 5 minutes (TTFT_WINDOW_S, at most 2,000); non-streamed requests have no first token and aren’t counted. SGLang’s own time_to_first_token_seconds histogram wasn’t used: its buckets are coarse (0.1, 0.2, 0.4 s…) and it starts counting after our queue.
      • Queue depth: waiting is requests the gateway holds back for a slot (4 in flight per device, 8 server-wide; requests_waiting), queued is SGLang’s own queue (num_queue_reqs). Both are in GET /admin/status under gateway (requests_waiting, ttft_p50/p95/p99, ttft_samples); a gateway from before this reports none and the panels stay empty.
      • Verified on the 27B server with 6 streamed requests at once from one device: 4 in flight and 2 waiting, then p50 0.32 s with the two that waited at 3.8 s and 5.0 s (p95/p99); a non-streamed request added no sample. The panels themselves were type-checked, not looked at in the app.

  • Headless control: oppxs status | servers | invite <name> | make-host <name> | approve [<name>] | decline <name> | remove <name> | leave | dissolve. These talk to the running server on 127.0.0.1:9094 with a per-start token in control.token (0600). Use run --name <name> to set the display name; it’s kept.

  • Verified between the 3090 (app and headless) and the 4090 laptop (headless), on the wired LAN:

    • discovery in both directions; a non-server machine (the HP laptop) never listed;
    • invite and approve; membership surviving restarts of both;
    • reports with the 4090’s GPU;
    • hand-over (the 4090 became host, and the 3090 moved over by itself);
    • dissolve; a stopped server gone from the list within 12 s.