Install on Kubernetes

This guide installs one self-hosted Botyard bot's runtime (botyardd) in your own Kubernetes cluster with the botyard-runtime-selfhosted Helm chart. For its own control-plane traffic, the runtime makes only outbound connections, over HTTPS and WSS, to Botyard, and listens on no network port for Botyard. The chart creates no Service and no Ingress. This covers the runtime's own transport only. Commands it starts through exec, and services it supervises, can open listeners and connect on other ports. Having no Service or Ingress does not stop direct connections to the Pod IP. Use the chart's NetworkPolicy (off by default) or your own ingress controls.

Read the self-hosted overview first. It covers what runs where and what you need before you start.

Before you start

You need:

• A Kubernetes cluster at version 1.25 or later, on linux/amd64 or linux/arm64 nodes.
• A default StorageClass, or the name of one, that can provision a ReadWriteOnce volume.
• Outbound HTTPS (TCP 443) from the pod to Botyard, either direct or through an HTTP proxy.
• kubectl, Helm 3.8 or later (for OCI charts), cosign 2.x or later, and jq.
• A self-hosted bot in Botyard, which you will enroll this cluster as. See the Enrollment and devices page.

The chart runs a single-replica StatefulSet. Botyard allows one active device per bot, and the pod's volume holds the device key, so the volume is the device. There is no replicaCount.

The examples below use the namespace botyard, the release name bot and version 1.7.2414. With release name bot, the pod is bot-botyard-runtime-selfhosted-0.

VERSION=1.7.2414
kubectl create namespace botyard

The chart's pod meets the Kubernetes restricted Pod Security Standard: a non-root user, a read-only root filesystem, all capabilities dropped and no service-account token. You can enforce that standard on the namespace:

kubectl label namespace botyard pod-security.kubernetes.io/enforce=restricted

1. Verify what you pull

Botyard publishes the image and the chart to GitHub Container Registry. Both are public:

Each release is signed keyless with cosign. The signing certificate is issued to Botyard's GitHub Actions release workflow, and every signature is recorded in the public Rekor transparency log. The image also carries a signed SPDX SBOM attestation. Botyard holds no long-lived signing key, so you verify the signer's identity, not a key:

IDENTITY='^https://github\.com/Botyard-AI/botyard/\.github/workflows/release-runtime-selfhosted\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+$'
ISSUER=https://token.actions.githubusercontent.com

IDENTITY accepts only the release-runtime-selfhosted.yml workflow in the Botyard-AI/botyard repository, run from a release tag such as v1.7.2414. ISSUER is GitHub Actions' OIDC issuer.

Verify the image by tag, and keep the digest that cosign verified. The block takes the digest only from a successful cosign verify, and accepts it only if it is a well-formed sha256 digest:

IMAGE_DIGEST=
if IMAGE_VERIFIED=$(cosign verify "ghcr.io/botyard-ai/runtime-selfhosted:${VERSION}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"); then
  IMAGE_DIGEST=$(printf '%s' "$IMAGE_VERIFIED" | jq -r '.[0].critical.image."docker-manifest-digest"')
fi
if printf '%s' "$IMAGE_DIGEST" | grep -Eqx 'sha256:[0-9a-f]{64}'; then
  echo "image verified: $IMAGE_DIGEST"
else
  IMAGE_DIGEST=
  echo "STOP: image verification failed. Do not install." >&2
fi

From here on, refer to the image by that digest. A tag can be moved; a digest cannot. Verify the SBOM attestation on the same digest:

if [ -n "$IMAGE_DIGEST" ] && cosign verify-attestation --type spdxjson \
    "ghcr.io/botyard-ai/runtime-selfhosted@${IMAGE_DIGEST}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" > /dev/null; then
  echo "SBOM attestation verified"
else
  echo "STOP: SBOM attestation verification failed. Do not install." >&2
fi

Verify the chart the same way, then pull it and check that the pulled chart has the digest you verified:

CHART_DIGEST=
if CHART_VERIFIED=$(cosign verify "ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted:${VERSION}" \
    --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"); then
  CHART_DIGEST=$(printf '%s' "$CHART_VERIFIED" | jq -r '.[0].critical.image."docker-manifest-digest"')
fi

if ! printf '%s' "$CHART_DIGEST" | grep -Eqx 'sha256:[0-9a-f]{64}'; then
  echo "STOP: chart verification failed. Do not install." >&2
elif ! PULL_OUTPUT=$(helm pull oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted \
    --version "$VERSION" 2>&1); then
  printf '%s\n' "$PULL_OUTPUT" >&2
  echo "STOP: helm pull failed." >&2
elif printf '%s\n' "$PULL_OUTPUT" | grep -Fqx "Digest: ${CHART_DIGEST}"; then
  echo "chart verified and pulled: $CHART_DIGEST"
else
  printf '%s\n' "$PULL_OUTPUT" >&2
  echo "STOP: the pulled chart is not the chart you verified. Delete the .tgz. Do not install." >&2
fi

helm pull saves botyard-runtime-selfhosted-<version>.tgz in the current directory. You install from that file in step 3, so Helm uses exactly the chart you verified.

Each block above ends by printing either a success line or a line starting with STOP. If you see STOP, or any cosign, jq or helm error, for any reason (not only no matching signatures), do not install. Fix the cause and run the whole block again.

Mirroring into a private registry

You can mirror both artifacts into your own registry, for example Amazon ECR, and install from there. cosign copy copies the image or chart together with its signatures and attestations, so the same cosign verify commands, pointed at your mirror, still check Botyard's identity:

MIRROR=registry.example.com/botyard   # your registry and path

cosign copy "ghcr.io/botyard-ai/runtime-selfhosted@${IMAGE_DIGEST}" \
  "${MIRROR}/runtime-selfhosted:${VERSION}"
cosign copy "ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted:${VERSION}" \
  "${MIRROR}/charts/botyard-runtime-selfhosted:${VERSION}"

Mirroring does not change the image digest. Then set image.repository to your mirror. If your nodes need credentials to pull from it, also set imagePullSecrets:

image:
  repository: registry.example.com/botyard/runtime-selfhosted
  digest: sha256:<IMAGE_DIGEST>
imagePullSecrets:
  - name: my-registry-credentials

2. Store the enrollment token in a Secret

The pod enrolls itself as the bot's device on first start, using a single-use enrollment token that is valid for 7 days. Mint it right before you install:

1. In Botyard, open User menu › Settings › Bots › ‹your bot› › Device.
2. Select Generate install command.
3. Botyard shows a one-line install command for a workstation, ending in --token <token>. Do not run that command. Copy only the token, the value after --token.

The Enrollment and devices page covers tokens, revocation and re-enrollment in detail.

Put the token in a Kubernetes Secret. Don't put it in a values file, and don't type it on a command line where it would land in your shell history. This reads it from a hidden prompt:

read -rs BOTYARD_TOKEN   # paste the token, then press Enter
printf '%s' "$BOTYARD_TOKEN" | kubectl -n botyard create secret generic botyard-enrollment \
  --from-file=token=/dev/stdin
unset BOTYARD_TOKEN

If you manage Secrets with a secret store (for example External Secrets Operator), create a Secret named botyard-enrollment with the token under the key token instead.

What happens to the token:

• On the first start, the container enrolls with it, then removes it from its environment before the daemon starts. Neither the container's init process nor the daemon holds it.
• A shell you open with kubectl exec does see BOTYARD_ENROLLMENT_TOKEN. Kubernetes sets a container's environment once, when the container starts, and does not update it later. So every kubectl exec into that container sees the token it started with, even after you delete or change the Secret.
• On every later start, the volume is already enrolled, so the token is not used again. A token that was already used or has expired is inert, so you can leave the Secret in place.

To remove the token from the running pod, wait until the bot is online (step 4), then delete the Secret and replace the pod:

kubectl -n botyard delete secret botyard-enrollment
kubectl -n botyard rollout restart statefulset/bot-botyard-runtime-selfhosted

The chart marks the Secret reference optional, so the new pod starts without the variable. Replacing the pod keeps its PersistentVolumeClaim, so the bot comes back already enrolled. Do the same after any later change to the Secret.

3. Install the chart

Create a minimal values file. It pins the image by the digest you verified, points at the Secret, and turns on the chart's egress-only NetworkPolicy:

image:
  digest: sha256:<IMAGE_DIGEST>   # from step 1
enrollment:
  existingSecret: botyard-enrollment
networkPolicy:
  enabled: true

Install from the chart file you pulled and verified:

helm install bot "./botyard-runtime-selfhosted-${VERSION}.tgz" -n botyard -f values.yaml

When image.digest is set, the chart pulls the image by digest and ignores image.tag.

You can also install straight from the registry with helm install bot oci://ghcr.io/botyard-ai/charts/botyard-runtime-selfhosted --version "$VERSION" -n botyard -f values.yaml. Helm then resolves the version tag again at install time, rather than using the chart file whose digest you checked.

The pod needs no registry credentials: the image and chart are public. The NetworkPolicy only takes effect if your cluster's network plugin enforces NetworkPolicy. On Amazon EKS, that means the VPC CNI with network policy enabled.

4. Check that it worked

Wait for the pod to be ready. The chart defines no probes: the pod is ready once botyardd is running, and the daemon reconnects to Botyard by itself.

kubectl -n botyard rollout status statefulset/bot-botyard-runtime-selfhosted

Read the logs:

kubectl -n botyard logs statefulset/bot-botyard-runtime-selfhosted

On a first start, look for these lines:

• enrolled this device device_id=<id> custody_tier=file …, then botyard-selfhosted: enrollment complete. The token was accepted, and this volume is now the bot's device.
• connected to the Botyard runtime gateway gateway=wss://runtime.botyard.io …. The daemon is connected.

Some INFO and WARN lines about hardware key storage are expected. They say that no Secure Enclave, TPM or similar is available, so the device key is kept on the volume (custody_tier=file).

On every later start, such as after the pod is deleted or rescheduled, these lines are expected and are not an error:

refusing to enroll: this machine is already bound and --replace was not given
botyard-selfhosted: state directory already enrolled; the enrollment token was not used

The volume is already this bot's device, so the token is left unused and the daemon starts and reconnects.

Ask the daemon directly:

kubectl -n botyard exec bot-botyard-runtime-selfhosted-0 -- botyardd status

The Botyard line should read connected. kubectl may also print a line like Unknown stream id 1, discarding message. That message comes from kubectl itself and is harmless.

Finally, in Botyard, User menu › Settings › Bots › ‹your bot› › Device should show the device as Online. The device is named after the pod unless you set enrollment.deviceLabel.

Before the bot can answer a message, it needs a model credential. Assign one with the llm scope on User menu › Settings › Bots › ‹your bot› › Credentials. Without one, a turn fails with No credential is configured for provider scope 'llm'.

If the pod is running but the bot stays offline:

• this machine is not enrolled in the logs means no token reached a new volume. Check that enrollment.existingSecret names your Secret, then restart the pod.
• enrollment failed means the token was already used or has expired. Mint a new one, replace the Secret's value, and restart the pod.
• No connected to the Botyard runtime gateway line and no error usually means the pod can't reach Botyard: a NetworkPolicy, firewall or proxy is in the way, or the proxy URL uses an unsupported scheme. Today the daemon logs failed connection attempts only at debug level, so they don't appear in the default logs. Check outbound TCP 443 from the pod and the proxy settings.

The Operations and troubleshooting page covers crashloops, upgrades and logs.

Values that matter

The chart's values.yaml is the source of truth for every value and its default. These are the ones most installs touch. Add them to values.yaml and run helm upgrade bot ./botyard-runtime-selfhosted-${VERSION}.tgz -n botyard -f values.yaml.

Host policy (hostPolicy)

hostPolicy is this machine's own limit on what the bot may do on the pod: read and write files, run commands, and so on. The bot can only use an operation that Botyard requests and this policy allows.

Empty, the default, means the runtime performs no host operation. No command runs and no file is read or written. The policy does not decide which tools the model is offered. That depends on what the bot is assigned in Botyard, and Botyard does not use your local policy to filter the assignment. So under an empty policy:

• a host tool the bot is not assigned is not offered. The bot may answer that the tool (for example process_start) isn't available;
• a host tool the bot is assigned can still be offered. If the bot calls it, the call reaches the runtime, which refuses it and writes an audit line with decision deny and reason local_policy.

To let the bot use host tools, list the operations in hostPolicy.enabled:

hostPolicy:
  enabled: [read, write, patch]
  root: /var/lib/botyard/workspaces

• root must be on the pod's volume (under /var/lib/botyard/) or under /tmp. It must not be the daemon's state directory, /var/lib/botyard/state, which holds the device key. The chart refuses those paths at install time and creates the root for you.
• root limits the file operations. It does not confine exec: a command runs as the same user and can read the device key. Enable exec only for a bot you would trust with this device's identity.
• Changing hostPolicy restarts the pod.
• To manage the policy yourself, set hostPolicyConfigMap.name to an existing ConfigMap instead. The root directory must then already exist.

The Host policy and security page explains each operation and the trust model.

Outbound proxy (proxy)

If the pod must reach the internet through a proxy, it must be an HTTP proxy (HTTP CONNECT) with an http:// URL. https:// and socks5:// proxies are not supported. The URL needs a real host and either no port (meaning 80) or a port from 1 to 65535.

Set one of these, not both:

• proxy.url — a proxy without credentials, such as http://proxy.corp.example:3128. The chart checks the URL against the rules above when you install, and refuses it with an error naming proxy.url. It may not contain a user name or password.
• proxy.existingSecret — the name of a Secret whose HTTPS_PROXY key holds the URL. Use this for a proxy that needs credentials. The chart can't read Secrets, so nothing checks this value at install time. Check the URL before you store it. A wrong scheme, such as https:// or socks5://, does not stop the pod: it starts and stays running, but never comes online in Botyard, and the failed connections are not logged at the default log level. See the Operations and troubleshooting page if the pod is running but the bot stays offline.

proxy.noProxy lists hosts to reach directly, with curl semantics: domains, .suffix, IP addresses, CIDRs or *.

proxy:
  url: http://proxy.corp.example:3128
  noProxy: .svc,.cluster.local

Extra CA bundle (extraCABundle)

If your proxy inspects TLS, give the pod your proxy's CA certificate. The bundle is added to the built-in public roots; it doesn't replace them. Put the PEM bundle in a ConfigMap or a Secret and set exactly one of these:

extraCABundle:
  configMapName: corp-ca   # or secretName: corp-ca
  key: ca.pem

If the bundle can't be read or isn't valid PEM, the daemon refuses to start.

Egress NetworkPolicy (networkPolicy)

The chart can create an egress-only NetworkPolicy. It denies all ingress, and allows DNS plus TCP 443. Your cluster's network plugin must enforce NetworkPolicy.

networkPolicy:
  enabled: true
  # Optional: allow 443 only to these CIDRs, for example your proxy.
  egressCIDRs: [10.20.0.15/32]
  # Optional: extra ports, for example your proxy's port.
  extraEgressPorts:
    - port: 3128
      protocol: TCP

egressCIDRs restricts only the 443 and extra-port rule. DNS stays allowed to any destination.

Storage (persistence)

The pod keeps its state on a ReadWriteOnce volume: the device key, the enrollment and the bot's workspaces.

persistence:
  storageClassName: gp3   # empty = the cluster default
  size: 10Gi

• persistence.existingClaim reuses an existing PVC, for example to restore a device after recreating the release.
• helm uninstall keeps the volume. Deleting the PVC deletes the device's identity: you would then revoke the device in Botyard and enroll again with a new token.

Resources (resources)

The defaults request 250m CPU and 512Mi memory, with a 2Gi memory limit. Raise them if the bot runs heavy commands on the pod:

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 2Gi

Other values

• enrollment.deviceLabel — the device's name in Botyard. Defaults to the pod name.
• extraEnv — extra, non-secret environment variables. For example, set BOTYARD_HOST_TOOL_AUDIT to off to turn off the per-call audit lines. The chart refuses variables it manages itself, and the identity variables described below.
• podLabels, podAnnotations, nodeSelector, tolerations, affinity, priorityClassName and terminationGracePeriodSeconds work as in most charts.

Don't connect the pod to a service mesh identity

Don't inject SPIFFE_ENDPOINT_SOCKET into this pod, and don't run your own SPIRE agent or SPIFFE-based mesh identity for it.

A self-hosted bot proves who it is with the device key it enrolled with. Botyard's own hosted bots use a different, Botyard-run identity system, which is found through that variable. If the variable is present, today's daemon assumes it is running inside Botyard's hosted platform, tries that identity path instead of its device key, and will not connect.

The chart refuses to set SPIFFE_ENDPOINT_SOCKET (and the related hosted-identity variables) through extraEnv. A mesh admission webhook can still inject it, so exclude this namespace or pod from any SPIFFE/SPIRE injection.

Known limitations

• One replica. One active device per bot. There is no high-availability mode; restarts are covered by the daemon reconnecting and resuming.
• Enrollment is by token only. Workload-identity federation (such as IRSA or a service account's OIDC token) is not supported yet.
• The device key is stored on the volume. KMS- and TPM-backed key storage are not supported yet.
• No customer SPIRE or SPIFFE mesh identity on this pod, as explained above.
• No self-update. You upgrade by installing a newer, verified chart and image version.