Skip to content
NewAzure OpenAI and Vertex AI are here

Your GPU first.The cloud when it counts.

Charon runs your AI requests on your own computer, and only reaches for the cloud when it has to. Private by default, cheaper by design, and it always tells you why.

Free for Mac. No account needed.

Works with anything that speaks the OpenAI API

Works with your code

Change one line. Keep your code.

If your app talks to OpenAI, it already works with Charon. Point it at 127.0.0.1:7777/v1 and Charon picks where each request runs.

  • Streaming and tools
  • JSON mode and embeddings
  • The Responses API
  • No API key needed
from openai import OpenAI

client = OpenAI(
    base_url="https://api.openai.com/v1",
    base_url="http://127.0.0.1:7777/v1",
    api_key="charon",
)
reply = client.chat.completions.with_raw_response.create(
    model="auto",
    messages=[{"role": "user", "content": "Summarize…"}],
)
print(reply.headers["x-charon-backend"])  # local
Response headers200 OK
HTTP/1.1
200 OK
x-charon-backend:
local
x-charon-model:
qwen2.5-7b-instruct
x-charon-locality:
local
x-charon-reason:
local: lowest latency (est 182 ms), cost 0

How it works

Rules first, then the best fit.

Charon first rules out anywhere a request can't or shouldn't go: over budget, too big, or private. Then it picks the fastest, cheapest option that's left, and tells you why.

Set your own rules
Three steps, every request
  1. Step 01Filter

    Rule out what can't work

    Too big, over budget, offline or private? Those options are out before anything else happens.

  2. Step 02Score

    Pick the best of the rest

    Charon scores what's left on speed, cost and your preferences. By default, fast and free wins.

  3. Step 03Fall back

    Catch it if it slips

    If the first choice stalls before answering, the runner-up takes over. Your app never notices.

No black box

Every answer says why.

Each response notes where it ran and the reason, so nothing is a mystery. Try three examples.

Routing decision

0.9 µs

model auto1,240 tokens

  1. 01

    local/qwen2.5-7b-instructChosen

    182 ms · $0
    0.912
  2. 02

    bedrock/amazon.nova-pro-v1:0

    640 ms · $0.0011
    0.604
  3. ec2-gpu/llama-3.1-70b

    Filtered by unhealthy: Instance stopped

    –

x-charon-backend: local

x-charon-reason:local: lowest latency (est 182 ms), cost 0

How Charon routes a requestApps send OpenAI-compatible requests to Charon. Charon drops candidates that fail hard filters (local-only, health, model, context, free VRAM, saturation, budget and battery), scores the rest on latency, cost, preference and load, and sends the request to your GPU or to your cloud, with one fallback before the first token.YOUR APPSOpenAI Python SDKLangChain agentcurl, editorscharon127.0.0.1:7777/v1HARD FILTERSlocal_onlyunhealthymodelcontextvramsaturatedbudgetbatteryWEIGHTED SCORELatency.4Cost.3Preference.2Load.1ONE FALLBACK, BEFORE THE FIRST TOKENWHERE IT RUNSYour GPUllama.cpp, free and privateYour cloudBedrock, EC2, any /v1 APIHow Charon routes a requestApps send OpenAI-compatible requests to Charon. Charon drops candidates that fail hard filters (local-only, health, model, context, free VRAM, saturation, budget and battery), scores the rest on latency, cost, preference and load, and sends the request to your GPU or to your cloud, with one fallback before the first token.YOUR APPSOpenAI SDKLangChaincurlcharon127.0.0.1:7777/v1HARD FILTERSlocal_onlyunhealthymodelcontextvramsaturatedbudgetbatteryWEIGHTED SCORELatency.4Cost.3Preference.2Load.1ONE FALLBACK, BEFORE THE FIRST TOKENYour GPUllama.cpp,free and privateYour cloudBedrock, EC2,any /v1 API
Fig. 02Filters run first and can't be overridden. The survivors are scored; the best one runs the request.

Desktop app

Know where every request ran.

Watch requests as they happen: where each one ran, what it cost, and why.

Fig. 03 — The dashboardEvery request, where it ran, and why.

Features

Everything you need, nothing you don't.

Read the docs

01Hardware

Knows your hardware

Charon watches your memory, load and battery, so it only runs what your computer can handle.

02Privacy

Private means private

Mark a request or an app as local-only and it never leaves your computer. Charon checks twice.

03Budgets

Budgets that hold

Set a daily or monthly cloud limit. Once you hit it, requests stay on your computer.

04Explainable

Every choice explained

Each response says where it ran and why, and the app shows how every option scored.

05Backends

Your cloud, your credits

Bring Amazon Bedrock, your own cloud GPU, Azure OpenAI, Vertex AI or any OpenAI-compatible service.

06Failover

Built to stay up

If a model stalls, Charon quietly hands the request to the next best option in under a second.

Benchmarks

Fast enough to forget it's there.

Measured on an Apple M4 Pro. Charon adds less than a millisecond.

Added to each request
0.15 ms
p50; 0.19 ms at p99
To decide among 5 candidates
0.96 µs
3.0 µs among 20
Daemon memory when idle
< 20 MB
No model loaded
Requests per second
28,000
8 clients, mock backend

Half the cloud bill, none of the failures

A coding agent's bursts of parallel tool calls, planning turns and files too long for a small local context, replayed under three policies. Local is Qwen2.5-0.5B on Metal; the cloud is simulated at Amazon Nova Pro prices and latency, so the costs are projected.

PolicyCloud costRun locallyFailed
all-cloud$0.2000% local0 of 500
all-local$0100% local61 of 500
auto$0.093(−54%)86% local0 of 500

Questions

Still curious? The docs answer the rest in plain English.

01Is it free?

Yes. Download the Mac app and use it, no account or sign-up. If you add a cloud provider, you pay that provider for what you use.

02Does my data leave my computer?

Only when Charon sends a request to a cloud you set up yourself, and never for requests you mark private. Charon has no telemetry and no account.

03Do I need a powerful computer?

No. Any Mac with Apple Silicon (M1 or newer) works. Bigger requests can go to the cloud, and more stays local on faster machines.

04Which apps work with it?

Anything that talks to the OpenAI API: the official SDKs, LangChain, LlamaIndex, editors and agents. Change the address, keep your code.

05How do I keep cloud costs down?

Set a daily or monthly budget, pause the cloud from the menu bar, and let Charon stop an idle cloud GPU for you.

06Windows or Linux?

Coming soon. Today Charon runs on Macs with Apple Silicon.