Skip to content
Guides /Backends

Guides

Page 4 of 6

Backends

Add places for requests to run: your Mac, Amazon Bedrock, your own EC2 GPU, Azure OpenAI, Vertex AI or any OpenAI-compatible API.

7 sections · 4 min read

A backend is a place Charon can run a request: your own Mac, or a cloud service you have an account with. The easiest way to add one is the app’s Backends screen. This page lists the matching charon backend add command and [[backends]] entry in charon.toml for each kind.

Keys are saved in your macOS Keychain, never in charon.toml (see Cloud keys).

Local models

Local models run on your Mac’s GPU with llama.cpp. They’re free and private. Add one in the app’s Models screen, or:

Terminal
charon models catalog                                     # ★ marks the best fit for this Mac
charon models pull qwen2.5-7b-instruct
charon models pull https://…/model.gguf --sha256 <digest>  # any GGUF file

Downloads resume if interrupted and are deleted if the checksum doesn’t match. Models are listed in ~/.local/share/charon/models/models.yaml:

YAML
version: 1
models:
  - id: qwen2.5-7b-instruct
    path: Qwen2.5-7B-Instruct-Q4_K_M.gguf
    context_length: 8192 # per request
    parallel_slots: 2 # requests handled at once
  - id: nomic-embed
    path: nomic-embed-text-v1.5.Q8_0.gguf
    embedding: true

A model loads on first use and unloads after idle_unload_minutes idle. Each runs in its own process, so a crash never takes Charon down: requests move on and the model restarts. Charon checks that a model fits in memory before loading it.

Amazon Bedrock

Terminal
charon backend add bedrock --region us-east-1
TOML
[[backends]]
id = "bedrock-us-east-1"
kind = "bedrock"
region = "us-east-1"
auth = "auto"            # api_key | sigv4 | auto (API key if set, else AWS credentials)
aws_profile = "default"
prices = { "amazon.nova-lite-v1:0" = { input_per_mtok = 0.06, output_per_mtok = 0.24 } }

Charon finds the available models itself. Prices are estimates you can edit; the app fills in common ones.

Your own EC2 GPU

A GPU server in your AWS account that Charon starts and stops for you.

  1. In the app, open Backends, add EC2, and click Save template. This saves ~/Downloads/charon-ec2-gpu.yaml. Create a stack from it in the CloudFormation console, or:

    Terminal
    aws cloudformation deploy --region us-east-1 --stack-name charon-gpu \
      --template-file ~/Downloads/charon-ec2-gpu.yaml --capabilities CAPABILITY_IAM \
      --parameter-overrides VpcId=<vpc-id> SubnetId=<subnet-id>

    You get an NVIDIA L40S (g6e) or L4 (g6) instance running vLLM behind an API key, tagged charon=true. Port 8000 stays closed unless you set AllowedCidr.

  2. Run the command in the stack’s CharonCommand output. It looks like this:

    Terminal
    charon backend add ec2 --name gpu-box --instance-id i-0abc… --region us-east-1 \
      --hourly-usd 1.861 --tunnel ssm --key-from-keychain --idle-stop-minutes 20
    charon backend start gpu-box --wait

Charon starts the instance only when you ask, waits until it’s ready, counts its hourly price against your budget, and stops it after idle_stop_minutes idle. The default SSM tunnel needs the AWS CLI and session-manager-plugin. Your AWS user needs Bedrock invoke and list, ec2:DescribeInstances, start and stop on instances tagged charon=true, and SSM port forwarding to them.

TOML
[[backends]]
id = "gpu-box"
kind = "ec2_vllm"
instance_id = "i-0abc…"
region = "us-east-1"
hourly_usd = 1.861
port = 8000
scheme = "http"          # http | https
tunnel = "ssm"           # ssm | none (connect over HTTPS directly)
local_port = 0           # tunnel's local port; 0 = pick one
idle_stop_minutes = 20   # 0 = never
stop_on_exit = false     # stop the instance when Charon quits
base_url = ""            # optional fixed address instead of the instance's own

Azure OpenAI

Terminal
charon backend add azure --base-url https://my-resource.openai.azure.com \
  --model gpt-4.1-mini --key-from-keychain
TOML
[[backends]]
id = "azure"
kind = "azure_openai"
base_url = "https://my-resource.openai.azure.com"   # "Keys and Endpoint" in the Azure portal
models = ["gpt-4.1-mini"]                            # your deployment names (required)
key_ref = "keychain:azure"                           # or AZURE_OPENAI_API_KEY
prices = { "gpt-4.1-mini" = { input_per_mtok = 0.4, output_per_mtok = 1.6 } }

Google Vertex AI

Terminal
charon backend add vertex --project my-project --model google/gemini-2.5-flash
TOML
[[backends]]
id = "vertex"
kind = "vertex"
project = "my-project"
region = "global"        # or us-central1, europe-west4, …
models = ["google/gemini-2.5-flash", "google/gemini-2.5-flash-lite"]   # required
prices = { "google/gemini-2.5-flash" = { input_per_mtok = 0.3, output_per_mtok = 2.5 } }

Serves Gemini and Model Garden partner models. Charon signs in with your gcloud auth login session (or Application Default Credentials) and renews the token itself.

Any OpenAI-compatible API

OpenAI, OpenRouter, Groq, vLLM, llama-server, Ollama and others.

Terminal
charon backend add openai --name openai-main --key-from-keychain
charon backend add ollama        # or llama-server, both on this Mac
TOML
[[backends]]
id = "openai-main"
kind = "openai_compat"
base_url = "https://api.openai.com/v1"
key_ref = "keychain:openai-main"
models = ["gpt-4.1-mini"]  # optional; read from /models otherwise
local = false              # true = runs on this machine: free, and allowed for local-only requests
prices = { "gpt-4.1-mini" = { input_per_mtok = 0.4, output_per_mtok = 1.6 } }

If Ollama or llama-server is running when Charon starts, the app and charon status offer to add it.

Other keys

Every backend has an id (letters, digits, ., _, -) and a kind. Cloud backends also take context_length (0 = unknown), timeout_s (default 600), key_env (an environment variable that holds the key) and key_ref; Bedrock, EC2 and Vertex take endpoint to override the service address. The mock kind is a fake backend for testing, with slots, latency_ms, tokens_per_sec, fail_rate and reply.

↑↓to move↵to open