Guides
Page 3 of 6
Routing and policies
How Charon decides where each request runs, and how to set budgets, aliases and local-only rules.
For each request, Charon rules out the places that can’t or mustn’t take it, scores the rest, and runs
it on the best one. You steer this with a policy: cloud budgets, named groups of models (aliases),
and rules. Edit it in the app’s Policies screen or in ~/.config/charon/policy.yaml.
Which model to ask for
auto: yourautoalias if you made one, otherwise every model, local first.- An alias such as
fast: the candidates you listed for it. backend/model, such asbedrock-us-east-1/amazon.nova-lite-v1:0: exactly that.
Keep requests on your machine
A local-only request never goes to a cloud. Mark one with the header X-Charon-Policy: local-only,
a client key made with charon keys create <name> --tag local-only, or a policy rule. Local-only
mode (force_local_only) does this for every request; Pause cloud (pause_cloud) turns off all
clouds.
What rules an option out
Checked in this order; weights never override them.
| Check | Drops an option when |
|---|---|
| Local-only | The request is local-only and the option is a cloud |
| Unhealthy | 3 health checks in a row failed, or its local worker crashed in the last 30 s |
| Model | It doesn’t serve that model or endpoint (such as embeddings) |
| Context | Prompt plus max_tokens is longer than the model allows |
| Memory | A local model isn’t loaded and won’t fit in free GPU memory (unless partial offload is on) |
| Busy | All slots of a local model are busy past the alias’s max_queue_ms (local-only requests wait instead) |
| Budget | The daily or monthly cloud budget is spent |
| Battery | A rule set local_disallowed and the computer is on battery |
If nothing is left, the error lists why for each option:
{
"error": {
"type": "no_backend",
"message": "no backend can serve this request",
"reasons": [
"local/qwen2.5-7b: all 2 slots busy, est wait 2400 ms > 1500 ms",
"openai-main/gpt-4.1-mini: unhealthy after 3 failed probes (connection refused)"
]
}
}The status is 403 policy_denied when a local-only request has no local option, 402 budget_exceeded
when the budget is spent and nothing local fits, and 503 no_backend otherwise.
How options are scored
score = w_latency·(1 − latency) + w_cost·(1 − cost) + w_preference·preference − w_load·load- latency (weight 0.4): expected wait plus time to finish, relative to the slowest option.
- cost (0.3): estimated from the backend’s prices, relative to the dearest. Local is free.
- preference (0.2): 1 for an alias’s first candidate, 0.2 less for each one after.
- load (0.1): how busy the local GPU is. Always 0 for clouds.
Ties go to the earlier candidate.
Fallback: if the winner fails before sending any text (connection error, 5xx, 429, or a local out-of-memory), the runner-up gets one try. A reply that has started never switches, and local-only requests never fall back to a cloud.
policy.yaml
version: 1
budgets:
cloud_daily_usd: 5.00
cloud_monthly_usd: 60.00
aliases:
fast:
candidates: # best first
- { backend: local, model: llama-3.1-8b-instruct }
- { backend: openai-main, model: gpt-4.1-mini }
smart:
weights: { latency: 0.2, cost: 0.2, preference: 0.6 }
max_queue_ms: 3000 # default 1500
candidates:
- bedrock-us-east-1/amazon.nova-pro-v1:0
- gpu-box/qwen2.5-32b-instruct
auto:
candidates: [fast, smart] # aliases can list aliases (no loops)
rules:
- match: { api_key: proj-clientdata } # a client key's name
set: { tags: [local-only] }
- match: { header: { X-Charon-Policy: local-only } }
set: { tags: [local-only] }
- match: { on_battery: true }
set: { weights: { cost: 0.1, latency: 0.2, preference: 0.2, load: 0.5 } }A rule’s match can use api_key, header and on_battery. Its set can add tags, change
weights, or set local_disallowed: true.
Apply changes in the app, with charon policy apply policy.yaml, or with PUT /admin/policy. A policy
with a mistake is rejected with its line number, and the old one stays in place.