Putting a Passkey in Front of a Kubernetes Agent UI

I wanted to put kagent — Solo.io’s open-source AI agent framework for Kubernetes — on a public URL so I could demo it without firing up a local kind cluster every time. The bar I set: a real subdomain, no password to remember, locked down to only my email.

What I ended up with is kagent.buford.dev, gated behind Cloudflare Access with Google as the identity provider and a passkey on my Google account. No password anywhere in the flow. Touch ID and I’m in.

This post is the war story — k3s on a tiny VPS, the chicken-and-egg between Let’s Encrypt and Cloudflare Access, and the wildcard cert that took down every other subdomain on my server.

The Stack

  • k3s on Hetzner (one node, control plane + worker), with --disable traefik --disable servicelb so it leaves the host’s port 80/443 alone for Caddy
  • kagent v0.9.1 installed via OCI Helm chart, plus its 11 bundled “preview” agents I haven’t yet pruned
  • Caddy in a Docker container, doing TLS termination and reverse proxy
  • Cloudflare Access with Google as the IdP — the actual passkey handler is Google, not Cloudflare
  • Cloudflare Origin Certificate for the TLS handshake between CF’s edge and Caddy

Total downtime added to the VPS during install: about 5 seconds while Caddy was recreated. Total RAM cost: ~700Mi (mostly the bundled agents, which I’ll prune later).

The Network Plumbing Trick

If you set a kagent Service to NodePort, k3s binds it on 0.0.0.0 of every node — meaning anyone hitting <vps-ip>:30080 gets the UI directly, no auth. That defeats the whole point.

The fix is a kubectl port-forward running as a systemd unit, bound only to the Docker bridge gateway IP:

ExecStart=/usr/local/bin/kubectl port-forward -n kagent \
  --address 172.18.0.1 svc/kagent-ui 30080:8080

172.18.0.1 is the gateway of the web Docker network where Caddy lives. Caddy can reach it. The public internet cannot. UFW gets a single scoped rule — allow from 172.18.0.0/16 to any port 30080 — and that’s it. The UI is reachable only through Caddy, which is reachable only through Cloudflare, which is reachable only after passing Access.

Defense in depth, with no extra moving parts.

The Let’s Encrypt Trap

I added a DNS A record for kagent.buford.dev pointing at my VPS, proxied through Cloudflare. Caddy tried to do its usual thing — issue a Let’s Encrypt cert — and immediately failed both ACME challenges in interesting ways.

HTTP-01 failed because Cloudflare Access intercepts every request to that hostname and serves its login page, including the /.well-known/acme-challenge/ paths Let’s Encrypt was probing:

"Invalid response from https://...cloudflareaccess.com/cdn-cgi/access/login/...
\"<!DOCTYPE html>\\n<html>\\n  <head>\\n    <title>Sign in ・ Cloudflare Access</title>"

TLS-ALPN-01 failed because Cloudflare terminates TLS at the edge with its own cert before any handshake reaches Caddy. The ACME-tls/1 ALPN protocol can’t be negotiated end-to-end.

The fix: skip Let’s Encrypt entirely and use a Cloudflare Origin Certificate. CF issues a 15-year cert that only its own edge trusts — perfect for this exact case, where the only HTTPS connections to Caddy are CF→origin. Set the zone’s SSL/TLS mode to “Full (strict)” so CF actually validates the origin handshake.

The Wildcard Cert That Broke Everything

CF’s Origin Certificate dialog defaults to issuing for *.buford.dev, buford.dev. I clicked through. The browser warning a minute later was for skateland.buford.dev — a totally unrelated site:

NET::ERR_CERT_AUTHORITY_INVALID

Caddy’s cert pool now contained a wildcard cert covering *.buford.dev, and when a request came in for any subdomain, Caddy’s matching logic preferred it over the perfectly valid Let’s Encrypt cert it had cached for that specific name. Browsers don’t trust the CF Origin CA, so every site was suddenly broken.

The fix was to revoke the wildcard origin cert and re-issue with hostname kagent.buford.dev only. Once the SAN no longer overlapped with anything else, Caddy fell back to the LE certs for the other subdomains, and the world was right again.

Lesson: never take the default *.example.com SANs on a Cloudflare Origin Cert if any other site on the same origin uses ACME. Issue origin certs scoped exactly to the hostname you’re using them for. Wildcard origin certs are a footgun on multi-tenant origins.

The Final Login Flow

  1. Browser hits https://kagent.buford.dev
  2. Cloudflare’s edge sees an Access app on that hostname → 302 to mosscreekdigital.cloudflareaccess.com/cdn-cgi/access/login/...
  3. The login page offers “Sign in with Google” (the only IdP I configured)
  4. Google does its passkey prompt — Touch ID — and returns a token to CF
  5. CF issues a CF_AppSession cookie scoped to kagent.buford.dev, valid 24h
  6. Browser bounces back to the origin with the cookie
  7. Caddy reverse proxies to 172.18.0.1:30080, which is kubectl port-forward to the kagent-ui Service inside k3s
  8. UI loads

No password typed. The whole flow takes about three seconds the first time, and is essentially invisible on subsequent visits within the session.

What’s Next

The UI is live but the only agents in it right now are the kagent built-ins. The next step is deploying my own vps-mcp Go server — an MCP tool server that gives a kagent agent four DevOps tools (health, Caddy logs, container status, disk usage) over SSH back to the VPS itself. That’ll turn kagent.buford.dev into a proper agentic console for the box it’s running on.

That post is coming next.