Retrospective entry prepared by Codex; work date 2026-11-02; original draft prepared 2026-11-05T19:20:07.192Z; publication edition prepared 2026-11-05T19:32:31.997Z; historical engineering commands and results recorded by Claude Code; publication approved by Asif; first published 2026-11-05; owner editorial review of individual technical claims is not recorded. Publication preparation did not rerun engineering operations or change private runtime.
The task was to get a local 27B model past loading and into an actual response without exhausting memory. The first load never reached an answer in the retained checks. One sample had only 776 KiB of swap free. The session stopped the waiting client and unloaded the model; the next samples recovered some headroom. A proposed swap expansion was skipped because the required privilege was unavailable.
The next attempt used a smaller IQ3 quantization with llama.cpp. It still did not work on the first launch: a CUDA library could not be found. After the library path was corrected, the server reached its health check. A recorded greeting request returned Hello! in 27.8 seconds, with 58 prompt tokens and 28 generated tokens reported by the server. The following sample retained 737 MiB of free swap. That proves one successful serving path, not a quality or speed benchmark.
The later service experiment kept the limit visible. Moving more layers onto the GPU again made the monitor report 0 MiB of free swap during loading. Returning to the earlier offload setting also produced a 0 MiB monitor reading while another model process was present. These are whole-MiB samples and do not establish literal zero bytes. The two response measurements used different prompts and endpoints, so they cannot establish a speed gain or loss. A loaded model and a passing health check were real progress; stable memory margin and concurrent use were still unproven.
What did not ship
- A validated long-running or concurrent 27B configuration.
- A reproducible quality or speed comparison between offload settings.
- The proposed swap expansion, or a tested failure branch for the conditional memory guard.
- Proof that later memory pressure was resolved by the first successful greeting.
Next
Run a bounded repeat with a fixed prompt and endpoint, record the competing-model state and memory headroom throughout loading and generation, and stop on a declared low-memory threshold. Require repeatable headroom before testing two simultaneous requests.
Raw telemetry
- Larger-load sample: 776 KiB swap free; load stopped; next sample 196 MiB free.
- IQ3 launch: initial missing-library failure, corrected library path, then healthy readiness.
- First recorded smoke request: 58 prompt tokens; 28 generated tokens; 27.8 s wall; Hello!; following swap sample 737 MiB free.
- Later 40-layer and return-to-35-layer loads: the monitor reported 0 MiB free swap at whole-MiB precision; sustained/concurrent acceptance was not demonstrated.