Skip to content
Guides /Routing and policies

Guides

Page 3 of 6

Routing and policies

How Charon decides where each request runs, and how to set budgets, aliases and local-only rules.

5 sections · 3 min read

For each request, Charon rules out the places that can’t or mustn’t take it, scores the rest, and runs it on the best one. You steer this with a policy: cloud budgets, named groups of models (aliases), and rules. Edit it in the app’s Policies screen or in ~/.config/charon/policy.yaml.

Which model to ask for

  • auto: your auto alias if you made one, otherwise every model, local first.
  • An alias such as fast: the candidates you listed for it.
  • backend/model, such as bedrock-us-east-1/amazon.nova-lite-v1:0: exactly that.

Keep requests on your machine

A local-only request never goes to a cloud. Mark one with the header X-Charon-Policy: local-only, a client key made with charon keys create <name> --tag local-only, or a policy rule. Local-only mode (force_local_only) does this for every request; Pause cloud (pause_cloud) turns off all clouds.

What rules an option out

Checked in this order; weights never override them.

Check Drops an option when
Local-only The request is local-only and the option is a cloud
Unhealthy 3 health checks in a row failed, or its local worker crashed in the last 30 s
Model It doesn’t serve that model or endpoint (such as embeddings)
Context Prompt plus max_tokens is longer than the model allows
Memory A local model isn’t loaded and won’t fit in free GPU memory (unless partial offload is on)
Busy All slots of a local model are busy past the alias’s max_queue_ms (local-only requests wait instead)
Budget The daily or monthly cloud budget is spent
Battery A rule set local_disallowed and the computer is on battery

If nothing is left, the error lists why for each option:

JSON
{
  "error": {
    "type": "no_backend",
    "message": "no backend can serve this request",
    "reasons": [
      "local/qwen2.5-7b: all 2 slots busy, est wait 2400 ms > 1500 ms",
      "openai-main/gpt-4.1-mini: unhealthy after 3 failed probes (connection refused)"
    ]
  }
}

The status is 403 policy_denied when a local-only request has no local option, 402 budget_exceeded when the budget is spent and nothing local fits, and 503 no_backend otherwise.

How options are scored

Text
score = w_latency·(1 − latency) + w_cost·(1 − cost) + w_preference·preference − w_load·load
  • latency (weight 0.4): expected wait plus time to finish, relative to the slowest option.
  • cost (0.3): estimated from the backend’s prices, relative to the dearest. Local is free.
  • preference (0.2): 1 for an alias’s first candidate, 0.2 less for each one after.
  • load (0.1): how busy the local GPU is. Always 0 for clouds.

Ties go to the earlier candidate.

Fallback: if the winner fails before sending any text (connection error, 5xx, 429, or a local out-of-memory), the runner-up gets one try. A reply that has started never switches, and local-only requests never fall back to a cloud.

policy.yaml

YAML
version: 1
budgets:
  cloud_daily_usd: 5.00
  cloud_monthly_usd: 60.00
aliases:
  fast:
    candidates: # best first
      - { backend: local, model: llama-3.1-8b-instruct }
      - { backend: openai-main, model: gpt-4.1-mini }
  smart:
    weights: { latency: 0.2, cost: 0.2, preference: 0.6 }
    max_queue_ms: 3000 # default 1500
    candidates:
      - bedrock-us-east-1/amazon.nova-pro-v1:0
      - gpu-box/qwen2.5-32b-instruct
  auto:
    candidates: [fast, smart] # aliases can list aliases (no loops)
rules:
  - match: { api_key: proj-clientdata } # a client key's name
    set: { tags: [local-only] }
  - match: { header: { X-Charon-Policy: local-only } }
    set: { tags: [local-only] }
  - match: { on_battery: true }
    set: { weights: { cost: 0.1, latency: 0.2, preference: 0.2, load: 0.5 } }

A rule’s match can use api_key, header and on_battery. Its set can add tags, change weights, or set local_disallowed: true.

Apply changes in the app, with charon policy apply policy.yaml, or with PUT /admin/policy. A policy with a mistake is rejected with its line number, and the old one stays in place.

↑↓to move↵to open