Using Warp Agents with llama.cpp and a Local LLM

If you found Warp’s Bring Your Own API Key (BYOK) page and assumed it was your route to running a local model through Warp’s agents, there’s a catch worth knowing before you start: BYOK is for cloud providers only. It lets you plug in your own Anthropic, OpenAI, or Google keys so inference is billed to your provider account instead of consuming Warp credits. There’s no slot for a local model.

The feature you actually want for a local LLM is Warp’s Custom inference endpoint. It lets Warp call an OpenAI-compatible API, including one served by llama.cpp. This post covers the distinction, the networking detail that trips people up, and the current setup I use with Ornith-1.0-35B on an M1 Max MacBook Pro.

BYOK vs. custom inference endpoint

Warp gives you three ways to bring your own inference. Two are relevant here:

• BYOK: Use your own API key for OpenAI, Anthropic, or Google models. Keys live only on your device. There’s no way to specify an arbitrary endpoint, so local models are out.
• Custom inference endpoint: Point Warp at any OpenAI-compatible endpoint: a router like OpenRouter or LiteLLM, a provider like z.ai, an internal gateway, or a local model server exposed at a public URL.

The third, Bring Your Own LLM (BYOLLM), is Enterprise-only managed inference through a cloud provider and isn’t relevant for a local setup.

llama.cpp exposes the OpenAI-compatible Chat Completions API that Warp expects, so the custom inference endpoint is the right door. Other servers such as Ollama, LM Studio, and vLLM can expose a similar API, but they are not covered in this walkthrough.

The gotcha: Warp’s harness runs server-side

Here’s the important part: Warp’s agent harness, meaning the system that assembles your prompt, system instructions, conversation context, and tool calls, runs on Warp’s backend servers, not on your machine. BYOK and custom endpoints only swap which credential and destination the backend uses. They do not move the harness onto your laptop.

The practical consequence: localhost won’t work. When you send a prompt, your endpoint URL and key travel from your device up to Warp’s servers, and Warp’s backend calls your endpoint from there. So a model listening on 127.0.0.1:8080 is unreachable. Warp explicitly rejects localhost, 127.0.0.1, and other private network addresses.

To use a local model, you have to expose it at a public HTTPS URL first, typically with a tunnel.

Step-by-step: llama.cpp with an authenticated endpoint

llama.cpp gives you direct control over quantization, context size, GPU offload, prompt caching, authentication, and chat templates. It can serve GGUF models through an OpenAI-compatible API, which makes it suitable for Warp’s custom inference endpoint.

1. Choose a GGUF model and quantization

The model file has to leave enough memory for the context and KV cache. Do not decide based only on whether the GGUF file technically fits in memory.

My current choice is unsloth/Ornith-1.0-35B-GGUF with the UD-Q6_K_XL quantization. Ornith-1.0-35B is a 35B-A3B Mixture-of-Experts model based on the Qwen3.5 architecture and post-trained for agentic coding work.

This replaced my earlier wang-yang/Ornith-1.0-35B-MTP-GGUF setup. The MTP build looked attractive because it can use self-speculative decoding, but on my M1 Max and this Warp workload it measured slower than plain autoregressive generation. The model-card style benchmark and a terminal agent request are different workloads. A captured agentic prompt full of tool schemas, shell output, and chat-template structure had lower draft acceptance and more verification overhead than the MTP path was worth.

On a MacBook Pro 18,2 with an M1 Max, 32-core GPU, and 64 GB unified memory, the useful target is not just “does the model load?” The useful target is “can it keep a long context, process repeated Warp prompts, and still respond interactively?” For the current setup, the rough memory budget at -c 131072 is about 28 GiB of weights, about 10 GiB of f16 KV cache, and a few GiB of working buffers. That fits under a raised 56 GB GPU wired-memory ceiling while still leaving room for macOS.

2. Raise the Apple Silicon GPU wired-memory limit

Before starting llama.cpp on Apple Silicon, I raise the maximum amount of unified memory that macOS lets the integrated GPU wire:

sudo sysctl iogpu.wired_limit_mb=57344

57344 is 56 GB in megabytes. On a 64 GB Mac, this leaves roughly 8 GB for macOS and userland while allowing Metal to keep a large model, KV cache, and GPU-side working memory resident. Without this, macOS can refuse or constrain GPU allocations even when Activity Monitor appears to show enough unified memory available.

That matters for this command because it combines full Metal offload, a 131,072-token context, f16 KV cache, and --mlock. The server is asking the system to keep a lot of memory resident instead of paging it away.

This is a runtime setting, not a permanent boot configuration. Reapply it after a reboot, and lower it if the desktop becomes unstable or other applications need more memory. You can inspect the current value with:

sysctl iogpu.wired_limit_mb

3. Start llama-server

This is the command I currently run. The API key below is a placeholder. Do not publish or reuse the real key from your own server configuration.

llama-server -hf unsloth/Ornith-1.0-35B-GGUF:UD-Q6_K_XL \
  --alias ornith-1.0-35b \
  -ngl 999 -fa on -t 8 \
  -c 131072 -b 4096 -ub 1024 \
  --cache-reuse 256 --cache-ram 4096 \
  --parallel 1 --no-mmproj \
  --jinja --reasoning-format deepseek \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --mlock \
  --port 8080 --no-webui \
  --api-key "replace-with-a-long-random-key"

The important arguments are:

• -hf unsloth/Ornith-1.0-35B-GGUF:UD-Q6_K_XL: downloads or loads the UD-Q6_K_XL quantization directly from Hugging Face. I use the Unsloth GGUF because its chat template works with the roles Warp sends.
• --alias ornith-1.0-35b: pins a stable model identifier. Without an alias, the model ID can depend on the GGUF path, which means swapping quantization or repository can break the model name configured in Warp.
• -ngl 999: requests full GPU offload. Using a high value is a practical way to ask llama.cpp to offload all layers it can.
• -fa on: enables Flash Attention.
• -t 8: uses eight CPU threads for CPU-side work.
• -c 131072: allocates a 131,072-token context. Ornith’s trained context is larger than this, but prompt processing at deep context becomes the real bottleneck.
• -b 4096: sets the logical batch size for prompt processing.
• -ub 1024: sets the microbatch size. This is one of the knobs worth sweeping because prompt processing is the long-context limiter.
• --cache-reuse 256: reuses prompt-cache chunks of at least 256 tokens. For an agent client that resends a large overlapping conversation every turn, this is the single most valuable flag.
• --cache-ram 4096: caps the prompt-cache RAM budget at 4096 MiB. I no longer use an unlimited cache here.
• --parallel 1: creates one server slot and gives the whole context window to that one request stream. --parallel N divides the context across slots, so -c 131072 --parallel 2 gives each request only 64K.
• --no-mmproj: prevents llama.cpp from loading a multimodal projector for this text-only workload.
• --jinja: enables the Jinja chat-template engine. This matters because Ornith emits tool calls in an XML-shaped form, and llama.cpp needs the template path to convert them into OpenAI-shaped tool_calls.
• --reasoning-format deepseek: moves <think> content into reasoning_content instead of leaving it inline where it can interfere with tool-call parsing.
• --temp 0.6, --top-p 0.95, and --top-k 20: set the server’s default sampling behavior.
• --mlock: asks macOS to keep the model resident instead of paging it out.
• --port 8080: serves the API on port 8080. llama.cpp binds to 127.0.0.1 by default, which is fine because the tunnel connects from the local machine.
• --no-webui: disables the built-in web UI. If an already-open browser tab still appears to work, it may just be cached. curl -sI http://127.0.0.1:8080/ returning 404 is the better check.
• --api-key: protects the API with Bearer-token authentication. This matters because the server will later be reachable through a public tunnel.

I intentionally do not set MTP flags in this command. On this M1 Max, --spec-type draft-mtp was slower for the captured Warp agent request I tested. At n_max=1, draft acceptance was around 79 percent and generation was slower than no MTP. Higher draft depths were worse or unstable. That does not mean MTP is bad everywhere. It means you should measure it on your own machine and prompt shape before making it part of a Warp endpoint.

I also avoid KV-cache quantization here. In depth-aware tests, q8_0 KV looked almost free at depth 0 but became much slower at real working depth. At 32K, f16 KV token generation was roughly 36 t/s while q8_0 KV was roughly 13 t/s. For long-context Metal inference, f16 KV was the right tradeoff on this machine.

4. Verify health, identity, authentication, and chat

Start with the health endpoint:

curl http://127.0.0.1:8080/v1/health

The health endpoint is public even when --api-key is set. A successful response reports {"status":"ok"}. A 503 response means the model is still loading.

Next, verify the protected endpoints with the same Bearer key you will give Warp:

curl http://127.0.0.1:8080/v1/models \
  -H "Authorization: Bearer replace-with-a-long-random-key"

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer replace-with-a-long-random-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ornith-1.0-35b",
    "messages": [
      {"role": "user", "content": "Reply with exactly: endpoint works"}
    ],
    "temperature": 0
  }'

Because the server command sets --alias ornith-1.0-35b, /v1/models should return that stable ID. Use that same ID in Warp. A 401 response normally means the Authorization header is missing, the Bearer prefix is wrong, or the key does not match.

5. Test tool calling before involving Warp

A normal chat response does not prove the model can drive an agent. Send a small function-calling request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer replace-with-a-long-random-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ornith-1.0-35b",
    "messages": [
      {"role": "user", "content": "What files are in the current directory?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "list_files",
          "description": "List files in a directory",
          "parameters": {
            "type": "object",
            "properties": {
              "path": {"type": "string"}
            },
            "required": ["path"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }'

Inspect the response for a structured tool_calls array. If the model writes a shell command as ordinary prose instead, check the llama.cpp startup log for chat-template warnings, confirm --jinja is active, and verify that the GGUF template supports the roles Warp sends. The official DeepReinforce GGUF template was brittle for this path in my testing, while the Unsloth template handled the needed role patterns.

6. Capture a real Warp request and benchmark against it

Synthetic prompts are useful for smoke tests, but they are a weak benchmark for a terminal agent. Warp sends large system instructions, conversation history, tool schemas, and terminal output. That request shape changes prompt-processing cost, chat-template behavior, and MTP draft acceptance. If you tune against a small code snippet, you may optimize for the wrong workload.

I use two helper scripts for this:

• warp-capture.py: a transparent logging proxy that sits between the public tunnel and llama-server. It forwards requests unchanged and saves inbound chat request bodies as JSON.
• ornith-sweep.sh: a benchmark runner that compares KV cache behavior, MTP draft depths, and server settings. When you pass a captured request with PROMPT_JSON, it replays the real Warp request instead of using a synthetic prompt.

The capture path looks like this:

Warp backend -> public tunnel -> warp-capture.py:8081 -> llama-server:8080

Start llama-server normally on port 8080. Then start the capture proxy on a separate local port:

mkdir -p captures
LISTEN=8081 \
UPSTREAM=http://127.0.0.1:8080 \
OUTDIR=./captures \
python3 warp-capture.py

Point your public tunnel at the capture proxy instead of directly at llama.cpp:

cloudflared tunnel --url http://127.0.0.1:8081

or:

ngrok http 8081

Then configure Warp to use the tunnel URL as usual, with /v1 at the end and the same API key that llama-server expects. Run one representative Warp task. The best task is not “say hello”. Use something that makes Warp load tools and terminal context, such as asking it to inspect a small project, list files, explain a script, or propose a change.

For each captured request, warp-capture.py writes two files:

• captures/req-000-HHMMSS.json: the exact JSON request body. This is the file to replay.
• captures/req-000-HHMMSS.txt: a flattened, human-readable version for inspection or for PROMPT_FILE.

The JSON file contains the prompt, messages, and tool schemas that Warp sent. Treat it like sensitive data. The script does not need to save Authorization headers for replay, but the body can still contain code, paths, command output, and conversation context.

Once you have a capture, replay it through the sweep script:

PROMPT_JSON=./captures/req-000-HHMMSS.json ./ornith-sweep.sh

For longer or more targeted runs, override the sweep variables:

PROMPT_JSON=./captures/req-000-HHMMSS.json \
CTX=131072 \
THREADS=8 \
KVTYPES="f16 q8_0" \
NMAX="1 2 3 4" \
RUNS=3 \
NPREDICT=128 \
./ornith-sweep.sh

The sweep has two phases. Phase 1 uses llama-bench to compare KV cache types across context depths. Phase 2 starts llama-server repeatedly and measures the captured request against different MTP and KV settings. For captured JSON, the script forces comparable request settings such as non-streaming output, temperature: 0, fixed max_tokens, and disabled prompt caching for the timed run.

Read the generated report.md, but also inspect the server logs it keeps. The lines worth checking are the startup facts: n_ctx_train, KV self size, Metal offload, Flash Attention, and any draft or acceptance messages. If swap usage grows during the run, the numbers are not clean. Reduce context, reduce cache pressure, or revisit iogpu.wired_limit_mb before trusting the benchmark.

This is how I found that MTP and q8_0 KV were not helping this particular Warp workload on my M1 Max. The point is not that those settings are universally bad. The point is that agent prompts are their own workload, and you should tune against the real request shape.

7. Expose port 8080 through HTTPS

With ngrok:

ngrok http 8080

With a temporary Cloudflare Tunnel:

cloudflared tunnel --url http://127.0.0.1:8080

Do not remove --api-key just because the tunnel URL is difficult to guess. The URL is public while the tunnel runs. Keep llama.cpp bound to 127.0.0.1, let the tunnel connect locally, use a long random key, and stop the tunnel when you are finished.

Repeat the authenticated /v1/models and /v1/chat/completions tests against the public hostname before opening Warp:

curl https://your-public-host.example/v1/models \
  -H "Authorization: Bearer replace-with-a-long-random-key"

8. Add llama.cpp to Warp

Use these values in Warp’s custom inference endpoint settings:

1. Endpoint URL: https://your-public-host.example/v1
2. Credential: the exact value passed to llama-server --api-key
3. Model identifier: ornith-1.0-35b

Warp sends the credential as an API credential to the OpenAI-compatible endpoint. Do not enter Bearer as part of the key unless Warp’s current settings UI explicitly asks for the full header value. The client normally builds Authorization: Bearer <key> itself.

Select the llama.cpp model explicitly in Warp and run a small task that requires a tool, such as listing files and reporting their sizes. Watch both the tunnel log and the llama-server output. A request reaching the server proves connectivity. A successful structured tool sequence proves considerably more.

What changed after tuning

The first working setup is rarely the final setup. These are the changes I made after benchmarking the endpoint on the actual M1 Max workload:

• Switched from the wang-yang MTP GGUF to unsloth/Ornith-1.0-35B-GGUF:UD-Q6_K_XL.
• Added --alias ornith-1.0-35b so Warp can keep a stable model identifier.
• Removed --spec-type draft-mtp and --spec-draft-n-max because MTP was slower on the captured agent request.
• Requested full Metal offload with -ngl 999 instead of a smaller layer count.
• Increased CPU threads from 4 to 8.
• Kept the 131,072-token context, but added explicit batch controls with -b 4096 -ub 1024.
• Changed --cache-ram -1 to --cache-ram 4096 so prompt-cache RAM has a ceiling.
• Removed --cache-idle-slots; at --parallel 1 on unified memory it can copy a large idle slot out and back without freeing separate VRAM.
• Replaced --no-mmap with --mlock, which better matches the goal of keeping the model resident.
• Added --no-mmproj because Warp agent requests are text-only here.
• Added --reasoning-format deepseek so reasoning text does not pollute tool-call parsing.
• Added explicit sampling defaults: --temp 0.6 --top-p 0.95 --top-k 20.
• Added --no-webui because this endpoint is meant to be an API target, not a browser UI.

The biggest lesson is that long-context performance on Apple Silicon is dominated by prompt processing and cache behavior, not just raw token generation. A cold 100K-token prompt can take long enough to time out somewhere in the client path. --cache-reuse 256 is what makes the large context practical for steady-state agent turns, because Warp often resends overlapping conversation context.

Choosing a model for terminal agents

The best chat model is not automatically the best terminal agent model. For Warp, prioritize:

• Reliable function calling and structured output
• Correct Bash, PowerShell, YAML, Docker, Terraform, and configuration syntax
• Enough context for system instructions, command output, and multi-step work
• Fast prompt processing and acceptable first-token latency
• Stable behavior with the chat template embedded in the GGUF

My primary choice is now Ornith-1.0-35B in UD-Q6_K_XL from Unsloth. The practical lesson is that model selection should follow testing on the actual machine. Model architecture, quantization, context allocation, chat-template behavior, prompt-cache reuse, and responsiveness through Warp matter more than selecting the newest model name or enabling every acceleration feature a model card advertises.

On a 64 GB Apple Silicon machine, this quantization leaves enough headroom for long context, f16 KV cache, the operating system, and other applications once iogpu.wired_limit_mb is raised. If this model is too slow on another Mac, test a smaller or more aggressively quantized tool-capable model rather than cutting the context until agent tasks become unreliable.

Billing and the “Auto” trap

Two things worth internalizing:

• Auto always burns Warp credits. Warp’s Auto model routing depends on Warp’s own infrastructure, so it consumes credits regardless of your BYOK or endpoint config. You have to select your specific endpoint-routed model from the picker to actually use your local LLM.
• Inference itself is free of Warp credits when you use your endpoint. With a local model that means it’s genuinely free; you’re just paying in electricity and unified memory. On Business and Enterprise plans, local agent runs still consume platform credits for run lifecycle and observability, separate from inference.

A few other Warp features, such as Codebase Context and cloud agent runs, keep using Warp’s infrastructure no matter what you configure, since they don’t run through your endpoint.

Is it worth it?

Routing a local model through Warp is a real option, but be honest about the tradeoff. Your prompts still leave your machine: up to Warp’s backend, then back down through a public tunnel to your machine. That way, you get Warp’s polished agent harness and tool use driving a model you control and don’t pay per-token for.

If what you want is Warp’s agent experience without spending credits, and you’ve got a capable local model and a tunnel, it works well. If your goal is air-gapped, nothing-leaves-the-laptop inference, a fully local agent tool is the better fit.

Docs referenced: Warp’s Bring Your Own API Key and Custom inference endpoint pages, the llama.cpp server documentation, and the Unsloth Ornith-1.0-35B-GGUF model card.