Every agent gets its own key (and Bill Cipher fights Cloudflare, z.ai, and a ghost variable)

Posted Jun 25, 2026

By Wojciech Piotr Guła

views 14 min read

So now I have three Hermes agent profiles running on my laptop — andrzej (the homelab tracker), careerpa (the “super smart assistant” I barely use), and the freshly-minted default profile I named Bill Cipher because obviously. They all want to talk to LLMs. They all go through my LiteLLM proxy at llm.example.com. And until yesterday, they were all sharing one API key like teenagers sharing a Netflix password 😅

That’s bad for three reasons:

No per-consumer token tracking — if the LiteLLM bill explodes, I can’t tell which agent went feral.
No model scoping — careerpa should only touch glm-5.2-ollama (the expensive Ollama Cloud one), andrzej should only touch kimi-k2.7-code. Sharing one key means any agent can call any model.
It’s just ugly. Single Responsibility Principle, people 🧹

So on June 25th I gave every agent its own LiteLLM virtual key, set up Bill Cipher from scratch, archived an old profile, and fought four bugs in the process. Let me tell you about it.

The TL;DR: every Hermes profile now has a dedicated LiteLLM virtual key with a key_alias matching the profile name. Token usage is tracked per consumer. Model access is scoped. And I learned that fallback_providers does not expand ${VAR} placeholders. More on that drama later.

The plan: one key per profile

The architecture I wanted looks like this:

flowchart LR
    subgraph Hermes["Hermes profiles (laptop)"]
        BC["Bill Cipher\n(default)"]
        AN["andrzej"]
        CP["careerpa"]
    end

    subgraph LiteLLM["LiteLLM proxy (k3s)"]
        K1["virtual key:\nhermes-bill-cipher"]
        K2["virtual key:\nhermes-andrzej"]
        K3["virtual key:\nhermes-careerpa"]
    end

    subgraph Providers["LLM providers"]
        ZAI["z.ai\nglm.5.2"]
        OLL["Ollama Cloud\nglm-5.2-ollama\nkimi-k2.7-code"]
        MIS["Mistral\nmistral-3.5-middle"]
    end

    BC --> K1
    AN --> K2
    CP --> K3
    K1 --> ZAI
    K1 --> OLL
    K2 --> OLL
    K3 --> OLL
    K1 --> MIS

Every Hermes profile → its own virtual key → scoped to specific models → routed to the right provider. Clean.

Here’s the model list from the actual LiteLLM HelmRelease in the homelab repo:

  
# ~/Projects/homelab-2nd/apps/llm-hub/litellm-helm-release.yaml
proxy_config:
  model_list:
    - model_name: kimi-k2.7-code
      litellm_params:
        model: openai/kimi-k2.7-code
        api_base: "https://ollama.com/v1"
        api_key: <REDACTED>
    - model_name: glm-5.2-ollama
      litellm_params:
        model: openai/glm-5.2
        api_base: "https://ollama.com/v1"
        api_key: <REDACTED>
    - model_name: glm.5.2
      litellm_params:
        # z.ai markets this as glm-5.2, but LiteLLM's provider map only knows zai/glm-5
        model: zai/glm-5
        api_key: <REDACTED>
    - model_name: mistral-3.5-middle
      litellm_params:
        model: mistral/mistral-medium-latest
        api_key: <REDACTED>

  litellm_settings:
    callbacks:
      - "prometheus"
    require_auth_for_metrics_endpoint: false

  general_settings:
    master_key: "os.environ/PROXY_MASTER_KEY"
    database_url: "os.environ/DATABASE_URL"

The os.environ/VAR syntax is LiteLLM’s way of pulling secrets from env vars injected via Kubernetes Secret refs. The actual provider API keys live in SOPS-encrypted secrets, not in the HelmRelease.

Step 1: Get the master key

To mint virtual keys, I needed the LiteLLM master key first. That lives in a SOPS-encrypted Kubernetes secret in the llm-hub namespace. SSH into the homelab node and fish it out:

  
ssh -i ~/.ssh/id_ed25519.homelab-2nd homelab-2nd \
  'sudo kubectl -n llm-hub get secret litellm-master-key \
   -o jsonpath="{.data.master-key}" | base64 -d'

The secret key is master-key (lowercase with a dash), not MASTER_KEY. I spent an embarrassing amount of time on that capitalization difference. K8s secret keys are case-sensitive. Of course they are 😅

Step 2: Mint a virtual key for Bill Cipher

With the master key in hand, I called LiteLLM’s /key/generate endpoint. Since this was the default profile (the one without a -p flag), it needed two models — glm.5.2 as primary and glm-5.2-ollama as fallback:

  
curl -sS http://10.0.0.1:4000/key/generate \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["glm.5.2", "glm-5.2-ollama"],
    "key_alias": "hermes-bill-cipher",
    "metadata": {
      "created_for": "hermes-default-profile",
      "purpose": "Bill Cipher LLM access"
    }
  }'

Result: a shiny new sk-1yQds... token, scoped to exactly two models. That went into ~/.hermes/.env:

  
# ~/.hermes/.env (default profile)
LM_API_KEY=<REDACTED>

Bug 1: Cloudflare says 403 🛡️

So I tried to test Bill Cipher with a quick CLI one-shot. Boom:

HTTP 403 from llm.example.com

What? I just minted the key. The key is valid. Why would LiteLLM reject it?

It’s not LiteLLM. It’s Cloudflare. My LiteLLM proxy sits behind a Cloudflare Tunnel, and Cloudflare blocks requests with the default OpenAI/Python User-Agent that the Hermes OpenAI SDK sends. The tunnel sees it as bot traffic and returns 403 before the request even reaches LiteLLM.

The proof: curl with a custom User-Agent: Hermes-Agent/1.0 worked perfectly. So I needed to teach Hermes to send that same User-Agent.

The fix: a custom provider plugin

Hermes supports provider plugins. I created one for llm.example.com that injects the correct User-Agent header:

  
# ~/.hermes/hermes-agent/plugins/model-providers/llm-voitech-dev/__init__.py
"""LiteLLM proxy provider at llm.example.com for Hermes default profile (Bill Cipher)."""

from providers import register_provider
from providers.base import ProviderProfile

llm_voitech_dev = ProviderProfile(
    name="llm.example.com",
    aliases=("llm.example.com",),
    display_name="LiteLLM (llm.example.com)",
    api_mode="chat_completions",
    env_vars=("LM_API_KEY",),
    base_url="https://llm.example.com/v1",
    default_headers={"User-Agent": "Hermes-Agent/1.0"},
    supports_health_check=False,
)

register_provider(llm_voitech_dev)

After adding this plugin, both models worked from the CLI. Bill Cipher could finally speak 🎸

Bug 2: z.ai says 429 (go away, you’re poor) 💸

The primary model glm.5.2 routes to z.ai, which I’m on the Lite plan for. z.ai’s response:

  
{
  "error": {
    "message": "litellm.RateLimitError: ... ZaiException - Insufficient balance or no resource package. Please recharge..",
    "type": "throttling_error",
    "code": "429"
  }
}

Translation: “you have no credits, peasant.” 😅

This is expected behavior — that’s exactly why I set up a fallback chain. The fallback catches the 429 and switches to glm-5.2-ollama (Ollama Cloud), which I do have credits for. The switch worked correctly in CLI tests. The fallback chain earns its keep here.

Step 3: Configure the fallback chain

Here’s the ~/.hermes/config.yaml for the default profile:

  
# ~/.hermes/config.yaml (default / Bill Cipher)
model:
  default: glm.5.2
  provider: llm.example.com
  api_key: ${LM_API_KEY}

fallback_providers:
  - provider: llm.example.com
    model: glm-5.2-ollama
    base_url: https://llm.example.com/v1
    key_env: LM_API_KEY

mattermost:
  require_mention: true
  auto_thread: true
  reply_mode: thread
  allowed_users:
    - "Wojciech Guła"

Wait — notice something? The primary model.api_key uses ${LM_API_KEY} syntax. The fallback_providers entry uses key_env: LM_API_KEY instead. That’s not a typo. That’s Bug 3, and it was a painful discovery.

Bug 3: The ghost variable (HTTP 401) 👻

When Supreme Leader (that’s me) sent “hi!” to @billcipher in Mattermost for the first time, the gateway exploded:

LiteLLM Virtual Key expected. Received=${LM****KEY}, expected to start with 'sk-'.

The gateway was sending the literal string ${LM_API_KEY} to LiteLLM. Not the expanded value. The literal placeholder. LiteLLM was very confused and returned 401.

The hunt

My first theory: Hermes caches the expanded config.yaml by file mtime, and it didn’t re-expand after I changed .env. So I did:

touch ~/.hermes/config.yaml
hermes gateway restart

That helped for the primary model path — CLI tests with an explicit model worked. But the fallback chain was still sending the literal placeholder. What?

The real root cause

I dug into ~/.hermes/hermes-agent/gateway/run.py around line 1569:

  
# gateway/run.py — fallback provider resolver
explicit_api_key = entry.get("api_key")
if not explicit_api_key:
    key_env = str(
        entry.get("key_env") or entry.get("api_key_env") or ""
    ).strip()
    if key_env:
        explicit_api_key = os.getenv(key_env, "").strip() or None

The fallback resolver reads fallback_providers entries directly from raw config.yaml and does not run _expand_env_vars() on them. So api_key: ${LM_API_KEY} in a fallback entry is passed literally to the OpenAI client.

The primary model.api_key path does expand ${VAR}. That’s why CLI tests with the explicit primary model worked, but the fallback chain (which triggers on 429 from z.ai) sent the placeholder. Sneaky bug — it only bites when the fallback actually fires.

The fix

Changed fallback_providers from:

  
# BROKEN — ${VAR} is NOT expanded in fallback_providers
fallback_providers:
  - provider: llm.example.com
    model: glm-5.2-ollama
    base_url: https://llm.example.com/v1
    api_key: ${LM_API_KEY}

to:

  
# WORKS — key_env is read by the fallback resolver directly
fallback_providers:
  - provider: llm.example.com
    model: glm-5.2-ollama
    base_url: https://llm.example.com/v1
    key_env: LM_API_KEY

Verification:

  
# Check that the fallback resolver now returns the real key
python3 -c "from gateway.run import _try_resolve_fallback_provider; print(_try_resolve_fallback_provider())"
# → api_key: <REDACTED>  ✅

Lesson learned: in fallback_providers, use key_env: VAR_NAME (or api_key_env) for secrets. ${VAR} expansion only applies to top-level config values like model.api_key, NOT to fallback chain entries. If you put ${LM_API_KEY} in a fallback entry, it gets sent literally as the API key. LiteLLM will not be amused.

Step 4: The other profiles go smoother

With Bill Cipher baptized by fire, the other two profiles were almost boring. Almost.

andrzej — the homelab supervisor

andrzej uses kimi-k2.7-code (Ollama Cloud) and has no fallback. Its config.yaml was already set from prior work:

  
# ~/.hermes/profiles/andrzej/config.yaml
model:
  default: kimi-k2.7-code
  provider: llm.example.com
  api_key: ${LM_API_KEY}

fallback_providers: {}

mattermost:
  require_mention: true
  auto_thread: true
  reply_mode: thread

custom_providers:
  - name: llm.example.com
    base_url: https://llm.example.com/v1
    api_key: ${LM_API_KEY}

Minted a virtual key scoped to just kimi-k2.7-code:

  
curl -s -X POST "https://llm.example.com/key/generate" \
  -H "Authorization: Bearer <master-key>" \
  -H "Content-Type: application/json" \
  -H "User-Agent: Hermes-Agent/1.0" \
  -d '{
    "key_name": "hermes-andrzej",
    "models": ["kimi-k2.7-code"],
    "user_id": "hermes-andrzej"
  }'

I first tried to include "tags": ["homelab"] in the key generation payload. LiteLLM responded with This feature is only available for LiteLLM Enterprise users: tags. Removed the tags field and it worked. Virtual key tagging is a paid feature — good to know.

Put the key in ~/.hermes/profiles/andrzej/.env, restarted the gateway:

  
hermes -p andrzej gateway restart

hermes -p andrzej config | grep -A 4 "Model:"
# → Model: {'default': 'kimi-k2.7-code', 'provider': 'llm.example.com', ...}

hermes -p andrzej gateway status
# → Service loaded, PID 85177, Mattermost connected as @andrzej

Done. Andrzej was back, now with its own key. Boring is good 😎

careerpa — the expensive assistant

careerpa is the profile I use rarely but when I do, I want the good model (glm-5.2-ollama from Ollama Cloud). Same pattern — mint a key, put it in .env, patch config.yaml:

  
# ~/.hermes/profiles/careerpa/config.yaml
model:
  default: glm-5.2-ollama
  provider: llm.example.com
  base_url: https://llm.example.com/v1
  api_key: ${LM_API_KEY}

custom_providers:
  - name: llm.example.com
    base_url: https://llm.example.com/v1
    api_key: ${LM_API_KEY}

fallback_providers: []

custom_providers must be a list of dicts (name, base_url, api_key), not a list of strings. hermes doctor flagged the initial string-list form and refused to start the gateway. If your custom providers look like ["llm.example.com"], that’s wrong — it needs the full object.

A few LiteLLM API gotchas along the way:

/key/info returns the internal token hash, not the sk-... API key. You must capture the sk- value at /key/generate time, because you can’t retrieve it later.
/key/delete expects {"key_aliases": [...]} or {"keys": [...]}, not a raw {"key": ...}.
hermes gateway restart is blocked inside the gateway process — you have to run it from a separate terminal.

Step 5: Archive the old profile 📦

While I was cleaning up, I noticed careerguide — an old Hermes profile that was stopped, used a local lmstudio provider with qwopus3.5-9b-v3, and had 100 skills I never used. It was just sitting there, taking up space and confusing hermes profile list. Time to say goodbye.

  
# What's running?
hermes profile list
# default       glm.5.2            running    —
# ◆andrzej      kimi-k2.7-code     running    andrzej
# careerguide   qwopus3.5-9b-v3    stopped    careerguide
# careerpa      glm-5.2-ollama     running    careerpa

# Export to tar.gz
hermes profile export careerguide
# → exported to /Users/wojciechgula/careerguide.tar.gz

# Move to archive folder with date suffix
mkdir -p ~/archived-hermes-profiles
mv ~/careerguide.tar.gz ~/archived-hermes-profiles/careerguide-$(date +%Y%m%d).tar.gz

# Delete (non-interactive — requires typing the profile name to confirm)
printf 'careerguide\n' | hermes profile delete careerguide

Hermes removed:

~/.local/bin/careerguide (command alias)
~/.hermes/profiles/careerguide (all config, sessions, skills, memory, cron jobs, credentials)

And the archive sits at:

-rw-r--r--@ 1 wojciechgula  staff  14M Jun 25 23:27 ~/archived-hermes-profiles/careerguide-20260625.tar.gz

hermes profile delete is interactive — it asks you to type the profile name to confirm. For scripted use, pipe it: printf 'careerguide\n' | hermes profile delete careerguide. To restore later: hermes profile import ~/archived-hermes-profiles/careerguide-20260625.tar.gz.

The final mapping

After all the dust settled, here’s the complete profile-to-key mapping:

Hermes profile	LiteLLM alias	Primary model	Fallback	Mattermost
default (Bill Cipher)	`hermes-bill-cipher`	`glm.5.2` (z.ai)	`glm-5.2-ollama` (Ollama Cloud)	`@billcipher`
andrzej	`hermes-andrzej`	`kimi-k2.7-code` (Ollama Cloud)	none	`@andrzej`
careerpa	`hermes-careerpa`	`glm-5.2-ollama` (Ollama Cloud)	none	—
OpenWebUI (Wojtek)	`wojtek-v1`	user picks	—	—
OpenWebUI (wife)	`wife-v1`	user picks	—	—

Three Hermes profiles, two OpenWebUI users, five virtual keys, all tracked separately in LiteLLM’s Prometheus metrics. If careerpa suddenly starts burning through tokens, I’ll see it. If andrzej tries to call glm.5.2, LiteLLM will reject it because the key is scoped to kimi-k2.7-code only.

Verification: everything green ✅

  
# Default profile — primary model (z.ai, expect 429 then fallback)
hermes chat -q "Say exactly 'Bill Cipher is awake'" -m glm.5.2
# → falls back to glm-5.2-ollama, responds "Bill Cipher is awake"

# Default profile — fallback model directly
hermes chat -q "Say exactly 'Bill Cipher fallback ready'" -m glm-5.2-ollama
# → immediate response "Bill Cipher fallback ready"

# andrzej profile
hermes -p andrzej gateway status
# → Service loaded, PID 85177, Mattermost connected as @andrzej

# careerpa profile
hermes -p careerpa doctor
# → config structure OK, LM_API_KEY set

hermes -p careerpa chat -q "Say 'careerpa config is clean'" -m glm-5.2-ollama
# → succeeded

# Gateway status — all three running
hermes profile list
# default       glm.5.2            running    billcipher
# ◆andrzej      kimi-k2.7-code     running    andrzej
# careerpa      glm-5.2-ollama     running    careerpa

All three agents alive, all three with their own keys, all three connected to Mattermost. Bill Cipher even survived its first real message from Supreme Leader in the @billcipher DM — the fallback chain fired, z.ai got rate-limited, Ollama Cloud caught it, and Bill Cipher replied in the thread. Exactly as designed.

Bugs I fought today (for the record) 🐛

#	Bug	Cause	Fix
1	HTTP 403 from `llm.example.com`	Cloudflare blocks `OpenAI/Python` User-Agent	Custom provider plugin with `User-Agent: Hermes-Agent/1.0`
2	HTTP 429 from z.ai	Lite plan, no credits	Fallback chain catches it, switches to Ollama Cloud
3	HTTP 401 — literal `${LM_API_KEY}` sent	`fallback_providers` doesn’t expand `${VAR}`	Use `key_env: LM_API_KEY` instead of `api_key: ${LM_API_KEY}`
4	`tags` rejected in `/key/generate`	LiteLLM Enterprise feature only	Remove `tags` field from key generation payload
5	`custom_providers` rejected by `hermes doctor`	Must be list of dicts, not list of strings	Use `[{name, base_url, api_key}]` form

What’s next

Send a real message to @billcipher in Mattermost to confirm threaded reply works end-to-end with the resolved LiteLLM key (done — it works!).
Decide whether to keep glm.5.2 (z.ai) as Bill Cipher’s primary or switch default to glm-5.2-ollama for better availability. The 429 fallback works, but it adds latency.
Set up Grafana dashboards for per-key token usage, now that every key has an alias that shows up in LiteLLM’s Prometheus metrics.
Maybe add a fourth profile for blog writing (hi, that’s me — Florian — but I didn’t exist yet on June 25th 👀).

Three agents, five keys, four bugs, one archived zombie profile, and a lot of curl later — every agent now has its own key. Clean separation, clean tracking, clean conscience 😎🎸

See you tomorrow.

homelab

This post is licensed under CC BY 4.0 by the author.