Guides
Page 4 of 6
Backends
Add places for requests to run: your Mac, Amazon Bedrock, your own EC2 GPU, Azure OpenAI, Vertex AI or any OpenAI-compatible API.
A backend is a place Charon can run a request: your own Mac, or a cloud service you have an account
with. The easiest way to add one is the app’s Backends screen. This page lists the matching
charon backend add command and [[backends]] entry in charon.toml for each kind.
Keys are saved in your macOS Keychain, never in charon.toml (see
Cloud keys).
Local models
Local models run on your Mac’s GPU with llama.cpp. They’re free and private. Add one in the app’s Models screen, or:
charon models catalog # ★ marks the best fit for this Mac
charon models pull qwen2.5-7b-instruct
charon models pull https://…/model.gguf --sha256 <digest> # any GGUF fileDownloads resume if interrupted and are deleted if the checksum doesn’t match. Models are listed in
~/.local/share/charon/models/models.yaml:
version: 1
models:
- id: qwen2.5-7b-instruct
path: Qwen2.5-7B-Instruct-Q4_K_M.gguf
context_length: 8192 # per request
parallel_slots: 2 # requests handled at once
- id: nomic-embed
path: nomic-embed-text-v1.5.Q8_0.gguf
embedding: trueA model loads on first use and unloads after idle_unload_minutes idle. Each runs in its own process, so
a crash never takes Charon down: requests move on and the model restarts. Charon checks that a model fits
in memory before loading it.
Amazon Bedrock
charon backend add bedrock --region us-east-1[[backends]]
id = "bedrock-us-east-1"
kind = "bedrock"
region = "us-east-1"
auth = "auto" # api_key | sigv4 | auto (API key if set, else AWS credentials)
aws_profile = "default"
prices = { "amazon.nova-lite-v1:0" = { input_per_mtok = 0.06, output_per_mtok = 0.24 } }Charon finds the available models itself. Prices are estimates you can edit; the app fills in common ones.
Your own EC2 GPU
A GPU server in your AWS account that Charon starts and stops for you.
-
In the app, open Backends, add EC2, and click Save template. This saves
~/Downloads/charon-ec2-gpu.yaml. Create a stack from it in the CloudFormation console, or:Terminalaws cloudformation deploy --region us-east-1 --stack-name charon-gpu \ --template-file ~/Downloads/charon-ec2-gpu.yaml --capabilities CAPABILITY_IAM \ --parameter-overrides VpcId=<vpc-id> SubnetId=<subnet-id>You get an NVIDIA L40S (
g6e) or L4 (g6) instance running vLLM behind an API key, taggedcharon=true. Port 8000 stays closed unless you setAllowedCidr. -
Run the command in the stack’s
CharonCommandoutput. It looks like this:Terminalcharon backend add ec2 --name gpu-box --instance-id i-0abc… --region us-east-1 \ --hourly-usd 1.861 --tunnel ssm --key-from-keychain --idle-stop-minutes 20 charon backend start gpu-box --wait
Charon starts the instance only when you ask, waits until it’s ready, counts its hourly price against
your budget, and stops it after idle_stop_minutes idle. The default SSM tunnel needs the AWS CLI and
session-manager-plugin. Your AWS user needs Bedrock invoke and list, ec2:DescribeInstances, start
and stop on instances tagged charon=true, and SSM port forwarding to them.
[[backends]]
id = "gpu-box"
kind = "ec2_vllm"
instance_id = "i-0abc…"
region = "us-east-1"
hourly_usd = 1.861
port = 8000
scheme = "http" # http | https
tunnel = "ssm" # ssm | none (connect over HTTPS directly)
local_port = 0 # tunnel's local port; 0 = pick one
idle_stop_minutes = 20 # 0 = never
stop_on_exit = false # stop the instance when Charon quits
base_url = "" # optional fixed address instead of the instance's ownAzure OpenAI
charon backend add azure --base-url https://my-resource.openai.azure.com \
--model gpt-4.1-mini --key-from-keychain[[backends]]
id = "azure"
kind = "azure_openai"
base_url = "https://my-resource.openai.azure.com" # "Keys and Endpoint" in the Azure portal
models = ["gpt-4.1-mini"] # your deployment names (required)
key_ref = "keychain:azure" # or AZURE_OPENAI_API_KEY
prices = { "gpt-4.1-mini" = { input_per_mtok = 0.4, output_per_mtok = 1.6 } }Google Vertex AI
charon backend add vertex --project my-project --model google/gemini-2.5-flash[[backends]]
id = "vertex"
kind = "vertex"
project = "my-project"
region = "global" # or us-central1, europe-west4, …
models = ["google/gemini-2.5-flash", "google/gemini-2.5-flash-lite"] # required
prices = { "google/gemini-2.5-flash" = { input_per_mtok = 0.3, output_per_mtok = 2.5 } }Serves Gemini and Model Garden partner models. Charon signs in with your gcloud auth login session (or
Application Default Credentials) and renews the token itself.
Any OpenAI-compatible API
OpenAI, OpenRouter, Groq, vLLM, llama-server, Ollama and others.
charon backend add openai --name openai-main --key-from-keychain
charon backend add ollama # or llama-server, both on this Mac[[backends]]
id = "openai-main"
kind = "openai_compat"
base_url = "https://api.openai.com/v1"
key_ref = "keychain:openai-main"
models = ["gpt-4.1-mini"] # optional; read from /models otherwise
local = false # true = runs on this machine: free, and allowed for local-only requests
prices = { "gpt-4.1-mini" = { input_per_mtok = 0.4, output_per_mtok = 1.6 } }If Ollama or llama-server is running when Charon starts, the app and charon status offer to add it.
Other keys
Every backend has an id (letters, digits, ., _, -) and a kind. Cloud backends also take
context_length (0 = unknown), timeout_s (default 600), key_env (an environment variable that
holds the key) and key_ref; Bedrock, EC2 and Vertex take endpoint to override the service address.
The mock kind is a fake backend for testing, with slots, latency_ms, tokens_per_sec, fail_rate
and reply.