Synced from Hive. This page is pulled from kubestellar/hive@v4 during the docs build. Edit the canonical source in the Hive repository.

Provisioning a Hosted Hive

Self-hosting a standalone hive (no hub)? Most of this guide is about hub-attached hosted hives on the KubeStellar-internal fleet (hive-oke / vllm-d, shared hub, placeholder pools). If instead you are standing up your own hub-less hive on your own cluster against your own repos, read Standalone / self-hosted (no hub) below and skip the hub-only Path A / Path B, meta.json, and placeholder-pool sections — none of them apply to you.

This guide documents how hosted hives are provisioned, covering both paths:

• Automated provisioning — the hub creates everything itself, on any cluster the hub can reach with kubectl. This is the normal path.
• Manual provisioning — hand-applied manifests, required when the hub has no network path to the target cluster (a heartbeat-only cluster). This is the situation on vllm-d.

Every command and manifest below was executed and verified while standing up a pool of ten placeholder hives — five on vllm-d (manual) and five on hive-oke (automated) — so the procedure is real, not aspirational. Where a step has a non-obvious failure mode, it is called out as a Gotcha.

Background: how the two clusters differ

The provisioning path is dictated entirely by whether the hub can kubectl to the target cluster.

Why methods map to clusters. Public-method hives run subscription CLIs and need no in-cluster inference, so they live on the hub’s own cluster (hive-oke). Private-method hives point at a self-hosted inference endpoint that lives on vllm-d, so the hive runs there too — and vllm-d is heartbeat-only, which is exactly why it needs the manual path.

Prerequisites (both paths)

• kubectl access with a context per cluster (hive-oke, vllm-d).
• The hub is running on hive-oke in namespace hive-hub, backed by a RWX PVC named hive-hub-data-rwx mounted at /data. The hub’s SaaS store lives at /data/saas/hives/<id>/meta.json; its fleet registry at /data/hub-registry.json.
• For manual (vllm-d) provisioning: the system:openshift:scc:anyuid ClusterRole must exist (it does by default on OpenShift).

Standalone / self-hosted (no hub)

This section is the complete path for the hub-less scenario: an operator (“Joe”) running their own hive on their own firewalled cluster, against their own repos, with self-hosted inference (litellm / vllm / llm-d) and no attachment to hub.kubestellar.io. Everything after this section (Path A, Path B, meta.json, placeholder pools, claiming, upgrade-via-hub) is hub-only and does not apply — skip it.

What “standalone” means

A standalone hive has no hub. In src/pkg/config/config.go the hub: block is a HubConfig whose zero value has Enabled: false, so a config that simply omits hub: is already hub-less. Concretely, a standalone hive needs none of the hub plumbing the rest of this guide describes:

• No HIVE_HUB_SECRET, no HIVE_HUB_URL, no derived HIVE_HEARTBEAT_KEY / HIVE_SESSION_KEY / HIVE_SSO_KEY — those authenticate a spoke to a hub; a hive with nothing to heartbeat to needs none of them.
• No meta.json / My-Hives / hub SaaS record — there is no hub SaaS store.
• No claim / placeholder-pool / Upgrade steps — placeholders and hub-driven auto-upgrade are hub features. You upgrade the image yourself (below).

Because there is no hub auth proxy in front of it, a standalone hive enforces its own dashboard login: dashboard.hub_proxied: false (the default) plus a dashboard.authorized_users allowlist and a github.oauth_client_id for the device-flow login. All three are covered in the swap checklist.

Install method: Kustomize

Use the Kustomize overlays — a peer already built a base and a standalone overlay, so this is the least-effort, git-trackable path (no operator, no Helm values to reverse-engineer). The overlay layers two in-repo bases — the core workload (src/deploy/k8s) and an in-cluster OpenAI-compatible inference backend (src/deploy/inference, llama.cpp serving Qwen2.5-0.5B) — so you get a working model endpoint out of the box. See the overlay and its README here:

• Standalone overlay — github.com/kubestellar/hive/tree/v4/src/deploy/kustomize/overlays/standalone (annotated patch-configmap.yaml, patch-pvc-storageclass.yaml, patch-advisory-mode.yaml, and a README.md with the full swap list).
• Filled-in example (“Joe on Spyre”) — github.com/kubestellar/hive/tree/v4/src/deploy/kustomize/overlays/standalone/example-joe-spyre. Every value there is a placeholder — copy the shape, don’t apply it verbatim.

There are two flows. Pick based on whether you just want to see it run or you’re doing a real deployment.

1. Remote quickstart (see the shape). Kustomize can build straight from GitHub — no clone:

kubectl apply -k "https://github.com/kubestellar/hive/src/deploy/kustomize/overlays/standalone?ref=v4"

This applies the overlay with its PLACEHOLDER values (YOUR_ORG, YOUR_OAUTH_CLIENT_ID, YOUR_RWX_STORAGE_CLASS, …). It is for seeing the manifest shape come up — it will not be a working hive, because the repo, storage class, OAuth app, and inference endpoint are all placeholders you must swap first.

2. Real path (clone → edit → apply). This is the path you actually deploy, because you must swap values before it works:

git clone -b v4 https://github.com/kubestellar/hive.git
cd hive/src/deploy/kustomize/overlays/standalone
# Edit the placeholders — see "What you must swap" below:
#   patch-configmap.yaml       (org/repos, owner login, OAuth client id, litellm endpoint)
#   patch-pvc-storageclass.yaml (your RWX storage class)
# Review the rendered manifests, then apply:
kubectl kustomize .
kubectl apply -k .
kubectl -n hive rollout status deploy/hive
kubectl -n hive-inference rollout status deploy/vllm

Upgrading the image. The base tracks ghcr.io/kubestellar/hive:stable. To pin a reviewed tag or digest (so upgrades are deliberate — there is no hub to auto-upgrade you), run from the overlay directory and re-apply:

kustomize edit set image ghcr.io/kubestellar/hive=ghcr.io/kubestellar/hive:<tag>
# or a digest:  ...=ghcr.io/kubestellar/hive@sha256:<digest>
kubectl apply -k .

On OpenShift. The standalone overlay does not create a Route or grant the anyuid SCC. The overlays/openshift overlay supplies exactly those two OpenShift-only deltas — a Route exposing the dashboard port (3002) and the anyuid SCC RoleBinding the pod needs (it starts as root to set up the ACMM iptables and chown the PVC). A standalone deployment on OpenShift composes standalone + these platform deltas. Two equivalent ways:

• Combined overlay (recommended): in your own copy of the standalone kustomization.yaml, add route.yaml and anyuid-scc-rolebinding.yaml as additional resources — this is exactly what the example-joe-spyre overlay does (it adds its own route.yaml; add the SCC binding the same way). Set the Route spec.host to your cluster’s apps domain (or drop host to let OpenShift auto-generate).
• Apply both: kubectl apply -k <standalone> then kubectl apply -k <openshift> — but note the openshift overlay re-includes the src/deploy/k8s base, so prefer the combined form to avoid re-applying the base twice.

What you must swap

Every value below is a placeholder in the overlay. Cited location is where the key is defined (config struct) and where you edit it (overlay file).

Also populate the base hive-secrets Secret with a GitHub App PEM or HIVE_GITHUB_TOKEN, and — if your inference endpoint needs auth — the key named by governor.litellm.api_key_env (default HIVE_LITELLM_API_KEY).

The NET_ADMIN decision (stated precisely)

The base deployment already requests the NET_ADMIN capability (src/deploy/k8s/deployment.yaml) — the entrypoint uses it to install the forced-proxy-egress iptables REDIRECT that enforces the ACMM MITM egress proxy.

• Cluster GRANTS NET_ADMIN (most do): nothing to do — the full, fail-closed egress gate comes up automatically. This is the recommended default.
• Cluster DENIES NET_ADMIN: the entrypoint FATALs (exit 1, refusing to start with an advisory-only capability model — src/deploy/entrypoint.sh around line 770) and the pod crash-loops. The only escape hatch is setting HIVE_PROXY_ADVISORY_OK=true, provided by the commented-out patch-advisory-mode.yaml — uncomment its - path: patch-advisory-mode.yaml line in your kustomization.yaml.

Security trade-off (one honest sentence): in advisory-only mode the MITM egress gate is best-effort rather than enforced, so an agent holding a raw token could bypass the proxy policy — acceptable for a single-tenant hive whose owner controls every repo and agent, but not a control you should silently disable on a shared.

Firewall / egress (locked-down cluster)

A standalone hive needs no hub egress. The minimal outbound allowlist is:

• Your repo’s GitHub API host — api.github.com, or your GHE instance’s <host>/api/v3.
• ghcr.io (and its blob backend) — for the image pull.
• Your inference endpoint — if it’s in-cluster (the shipped src/deploy/inference backend, or your own litellm/vllm Service) this is cluster-internal traffic, not external egress at all.

Hub hosts (hub.kubestellar.io) are not required. For the full egress matrix (ports, hosts, when each is needed) see network-requirements.md; for the capability rationale see net-admin-requirement.md.

Agent backend

Standalone uses a self-hosted agent backend, not the subscription-CLI backends (copilot / claude) the internal fleet’s base ConfigMap uses. The overlay sets each agent’s backend: litellm (valid enum values: litellm | vllm | llm-d | watsonx). The litellm connection config lives under governor: (LiteLLMConfig on Config.Governor):

governor:
  litellm:
    endpoint: http://vllm-svc.hive-inference.svc.cluster.local:8000  # your OpenAI-compatible proxy
    api_key_env: HIVE_LITELLM_API_KEY   # env var NAME holding the key (default)
    default_model: qwen2.5-0.5b-instruct

Swap backend: vllm to hit HIVE_VLLM_ENDPOINT directly (the base deployment wires both HIVE_VLLM_ENDPOINT and HIVE_LLMD_ENDPOINT). Verify any key against src/pkg/config/config.go before adding it.

Path A — Automated provisioning (hive-oke)

The hub exposes POST /api/saas/hives (handleCreateHive). It generates the hive ID, creates the namespace + RBAC + PVC + Service + Ingress + Deployment, and writes the SaaS meta.json — all automatically.

A.1 Call the API as the hub admin

handleCreateHive authenticates via the hive_hub_user cookie, whose value is simply the username, validated against the SaaS user store. From inside the hub pod (localhost), that is all you need:

HUB_POD=$(kubectl --context hive-oke -n hive-hub get pods -l app=hive-hub \
  --no-headers | grep Running | awk '{print $1}' | head -1)
 
kubectl --context hive-oke -n hive-hub exec "$HUB_POD" -- curl -s -X POST \
  -H "Cookie: hive_hub_user=clubanderson" \
  -H "Content-Type: application/json" \
  -d '{
        "org": "myorg",
        "repos": "myrepo",
        "primary_repo": "myrepo",
        "project_name": "My Hive",
        "acmm_level": 2,
        "cluster_id": "hive-oke",
        "auth_method": "app",
        "app_id": "999999999",
        "installation_id": "",
        "app_private_key": "",
        "is_public": false
      }' \
  http://localhost:80/api/saas/hives

Response:

{"id":"hosted-myorg-myrepo-ab12","status":"provisioning",
 "subdomain":"hosted-myorg-myrepo-ab12.hive.kubestellar.io"}

The generated ID is hosted-<org>-<primary_repo>-<4char>.

app_id: 999999999 above is the placeholder sentinel, not an arbitrary number. It is config.PlaceholderAppID, and the spoke recognises it as “this hive has no GitHub App yet”. Use exactly this value — any other non-zero number is read as a real App ID and the hive will try (and fail) to authenticate as it. See Placeholder app_id below.

CreateHiveRequest fields that matter:

Gotcha — auth_method: app with no key provisions but reports error. The “app later” path leaves the hive with no valid credentials, so its meta.json status becomes "error" with "provisioning failed — check hub logs for details", and it 401-loops until the App is installed. This is expected for a not-yet-claimed hive. If you want it to read as available instead of error, patch the meta.json status (see Placeholder pools).

A.2 What the automated path creates that the manual path does not

The automated provisioner also provisions an OCI file-system export for the PVC and records it in meta.json (oci_file_system_id, oci_export_id). The manual path uses an in-cluster RWX PVC directly and has no OCI export. This is the main structural difference between the two meta.json shapes.

A.3 RBAC the template emits

The provisioning template creates the namespace RBAC for you, including the namespace-scoped read-only hive-route-reader Role and RoleBinding (get/list on networking.k8s.io/ingresses and route.openshift.io/routes).

It is not unused. The spoke reads its own Route/Ingress through it to learn the hostname it actually serves, and reports that to the hub as dashboard_url. Remove it and the hub falls back to synthesising <hiveID>.<hub host>, which 503s for any spoke not fronted by the hub’s wildcard domain — see hive-route-reader under the manual path for the full rationale, the YAML, and the ServiceAccount-derivation caveat.

The template binds hive-sa on RequiresSCC (OpenShift) clusters and default elsewhere. Namespaces provisioned before this Role was added do not have it and need it applied retroactively.

Path B — Manual provisioning (vllm-d)

The hub cannot reach vllm-d, so every object is applied by hand with kubectl --context vllm-d. The full set, in order:

1. Namespace
2. ServiceAccount (hive-sa)
3. RBAC — three Roles (hive-secrets-writer, hive-self-upgrade, hive-route-reader) and four RoleBindings (the three above plus hive-anyuid)
4. PVC (hive-data, RWX cephfs, 50Gi)
5. ConfigMap (hive-config) — the first-boot config seed
6. Secret (hive-secrets) — dashboard token, GitHub App key, LiteLLM key
7. Service (hive, ports 3002 dashboard / 3001 terminal)
8. Routes (hive-dashboard on /, hive-terminal on /terminal)
9. Deployment (hive)
10. Hub SaaS record — meta.json on the hub PVC

Set these shell variables for the target hive:

CTX=vllm-d
ID=hosted-myorg-myrepo          # the hive ID
NS=hive-hosted-$ID              # namespace is always hive-hosted-<id>
ROUTE_HOST=$ID.apps.fmaas-vllm-d.fmaas.res.ibm.com
IMAGE=ghcr.io/kubestellar/hive:v4-latest   # what the hub provisioner stamps by default (saas_provision.go); use `stable` for production
SC=ocs-storagecluster-cephfs
 
# The hub heartbeat secret — the SAME for every spoke on a given hub. Copy it
# from any working hive on the same cluster (see the Deployment gotcha). The
# spoke's heartbeats 401 without it.
HUB_SECRET=$(kubectl --context "$CTX" -n hive-hosted-<any-working-hive> \
  get deploy hive -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep '^HIVE_HUB_SECRET=' | cut -d= -f2-)

B.1 Namespace + ServiceAccount

kubectl --context "$CTX" create ns "$NS"
kubectl --context "$CTX" label ns "$NS" app=hive
kubectl --context "$CTX" -n "$NS" create sa hive-sa

B.2 RBAC — Roles and RoleBindings

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-secrets-writer, namespace: ${NS} }
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get","list","watch","create","update","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-self-upgrade, namespace: ${NS} }
rules:
- apiGroups: ["apps"]
  resources: ["deployments","deployments/scale"]
  verbs: ["get","list","watch","update","patch"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","list","watch","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: hive-route-reader, namespace: ${NS} }
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get","list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-secrets-writer, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-secrets-writer }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-self-upgrade, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-self-upgrade }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-route-reader, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: hive-route-reader }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: hive-anyuid, namespace: ${NS} }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: system:openshift:scc:anyuid }
subjects:
- { kind: ServiceAccount, name: hive-sa, namespace: ${NS} }
YAML

Gotcha — the RoleBinding subject namespace must be the hive’s own namespace. If you build these by copying another hive’s manifests, the subjects[].namespace fields will still point at the source namespace. The bindings then bind a ServiceAccount that does not exist here, the pod does not get the anyuid SCC, it falls to restricted-v2, and it crash-loops with exit 255. Always template subjects[].namespace to $NS. Verify after applying:

kubectl --context "$CTX" -n "$NS" get rolebinding hive-anyuid \
  -o jsonpath='{.subjects[0].namespace}'   # must equal $NS

hive-route-reader — why the dashboard link 503s without it

The spoke discovers the external hostname its own Route/Ingress actually serves and reports it to the hub as dashboard_url (SpokeServedHost, called from the heartbeat path). hive-route-reader is what lets it read them.

Without the Role, that lookup returns “no answer” and the spoke falls back to synthesising <hiveID>.<hub host>. That is correct only for spokes fronted by the hub’s own wildcard domain. Anywhere else it is a guaranteed 503: the wildcard resolves, so DNS looks healthy, but it sends the name to the hub’s router, which has no backend for a hive living on another cluster. This was a live user-visible outage on the vllm-d pool — the hub minted links at <id>.hive.kubestellar.io while the spoke’s Route served <id>.apps.fmaas-vllm-d.fmaas.res.ibm.com.

The spoke is the party that can answer this on a heartbeat-only cluster, since the hub has no kubectl path there by design.

Notes on the shape of the Role:

• Read-only and namespace-scoped. get/list, no write verbs. A compromised spoke learns its own hostname — which it already advertises — and can neither create nor retarget routing.
• Routes are listed unconditionally, not gated on OpenShift. A cluster can serve Routes without requiring an SCC, and a Role naming a CRD-backed resource is inert where that API is absent.
• Ingress is consulted before Route, matching the traffic path, so a cluster carrying both resolves to the same object.

Gotcha — derive the ServiceAccount, do not assume it. The subject is not uniform across the fleet. OpenShift/SCC spokes bind hive-sa; others bind default. Measured live: hive-oke is 22× default; vllm-d is 2× default and 41× hive-sa; a-ks-wec2 is 5× hive-sa. Binding the wrong fails silently — the pod runs, the lookup is denied, and you get the 503 fallback with no error anywhere obvious. Read it off the Deployment:

kubectl --context "$CTX" -n "$NS" get deploy hive \
  -o jsonpath='{.spec.template.spec.serviceAccountName}'

An empty result means the Deployment does not set, which is Kubernetes for default — bind default, not the empty string. The manifests in B.2 above use hive-sa because this manual path creates it in B.1; substitute whatever the command returns if you are patching an existing namespace.

Retrofit — namespaces provisioned before this change do not have it. The Role/RoleBinding is emitted by the provisioning template, so new hosted hives get it automatically. Every namespace created before it was added needs it applied retroactively, or its dashboard link keeps 503ing. Check the fleet:

kubectl --context "$CTX" get rolebinding -A \
  --field-selector metadata.name=hive-route-reader

The spoke also has to be running a build new enough to perform the lookup, so a namespace that already has the RBAC may still need a rollout restart to start reporting.

B.3 PVC

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: hive-data, namespace: ${NS}, labels: { app: hive } }
spec:
  accessModes: ["ReadWriteMany"]
  resources: { requests: { storage: 50Gi } }
  storageClassName: ${SC}
YAML

Use RWX (ReadWriteMany). Both the old and new pod mount the same volume during a rolling upgrade (maxUnavailable: 0), so the volume must support multi-attach — hence cephfs, not RWO block storage.

B.4 ConfigMap (the config seed)

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: ConfigMap
metadata: { name: hive-config, namespace: ${NS}, labels: { app: hive } }
data:
  hive.yaml: |
    project:
      org: myorg                      # a POOL PLACEHOLDER can seed any bootstrap org here; the
      repos: [myrepo]                 # AUTHORITATIVE identity is the meta.json org
      primary_repo: myrepo            # (available-vllmd-<date>-<suffix>), which is what the hub
                                      # renders — see Placeholder pools below
    agents:
      guide:   { backend: copilot, model: claude-sonnet-4-6, enabled: true }
      scanner: { backend: copilot, model: claude-sonnet-4-6, enabled: true }
    governor:
      eval_interval_s: 300
      modes:
        idle: { threshold: 0,  guide: 4h, scanner: 4h }
        busy: { threshold: 10, guide: 2h, scanner: 2h }
    github:
      app_id: 999999999               # the placeholder sentinel (config.PlaceholderAppID)
      installation_id:
      key_file: /secrets/gh-app-key.pem
      oauth_client_id: Ov23ligE2p0gjXg6xAUf   # public device-flow client (no secret)
    dashboard:
      port: 3002
      authorized_users:
        - owner-github-login:owner    # a POOL PLACEHOLDER lists the pool owner: clubanderson:owner
      hub_proxied: false              # vllm-d has no hub auth proxy → direct device-flow
    hub:
      enabled: true
      url: https://hive.kubestellar.io
      dashboard_url: https://${ROUTE_HOST}
      hive_type: hosted
      is_public: false
    acmm_level: 2
YAML

Gotcha — github.app_id cannot be empty. Config validation requires either github.token or github.app_id to be set (pkg/config/config.go: github.token or github.app_id is required). A hive awaiting its real App must carry the placeholder sentinel app_id: 999999999 or it will refuse to boot.

Use 999999999 exactly — it is a recognised sentinel, not a spare number. It is declared as config.PlaceholderAppID, and GitHubConfig.HasApp() reports a hive carrying it as having no GitHub App. Such a hive boots into dashboard-only mode: the dashboard serves, the heartbeat runs, and the “GitHub App required” banner shows the install link. Agents stay idle until a real App is linked.

Any other non-zero placeholder is read as a real App ID. The hive will then try to authenticate as an App that does not exist — and installation_id also becomes non-zero (which happens the moment the owner installs an App), it attempts to load a private key that was never provisioned. Before this was fixed, that combination exited the process before the HTTP listener bound: permanent CrashLoopBackOff, no heartbeat, the hive shown offline on the hub, and any in-flight rollout stuck forever because the new pod never went Ready. The spoke now degrades instead of exiting, but the hive still cannot do any GitHub work until the app_id is corrected.

Replacing the placeholder with the real value is normally automatic — the hub delivers app_id, installation_id and the private key over the heartbeat when the App is installed. To do it by hand, patch the PVC overlay (/data/hive.yaml.dashboard, not the ConfigMap) and restart the pod.

Gotcha — hub_proxied must be false on vllm-d. vllm-d has no hub nginx auth proxy. If hub_proxied is true, dashboard sign-in enters an OAuth redirect loop. On hive-oke it is the opposite — the hub proxy handles auth.

B.5 Secret

DASH_TOKEN=$(openssl rand -hex 32)
cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: Secret
metadata: { name: hive-secrets, namespace: ${NS}, labels: { app: hive } }
type: Opaque
stringData:
  dashboard-token: "${DASH_TOKEN}"
  gh-app-key.pem: "PLACEHOLDER-awaiting-github-app-key"
  litellm_api_key: "sk-PLACEHOLDER"
YAML

B.6 Service

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: v1
kind: Service
metadata: { name: hive, namespace: ${NS}, labels: { app: hive } }
spec:
  selector: { app: hive }
  ports:
  - { name: dashboard, port: 3002, targetPort: 3002 }
  - { name: terminal,  port: 3001, targetPort: 3001 }
YAML

B.7 Routes (OpenShift)

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: route.openshift.io/v1
kind: Route
metadata: { name: hive-dashboard, namespace: ${NS}, labels: { app: hive } }
spec:
  host: ${ROUTE_HOST}
  path: /
  port: { targetPort: dashboard }
  tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
  to: { kind: Service, name: hive, weight: 100 }
  wildcardPolicy: None
---
apiVersion: route.openshift.io/v1
kind: Route
metadata: { name: hive-terminal, namespace: ${NS}, labels: { app: hive } }
spec:
  host: ${ROUTE_HOST}
  path: /terminal
  port: { targetPort: terminal }
  tls: { termination: edge, insecureEdgeTerminationPolicy: Redirect }
  to: { kind: Service, name: hive, weight: 100 }
  wildcardPolicy: None
YAML

B.8 Deployment

cat <<YAML | kubectl --context "$CTX" apply -f -
apiVersion: apps/v1
kind: Deployment
metadata: { name: hive, namespace: ${NS}, labels: { app: hive } }
spec:
  replicas: 1
  strategy: { type: RollingUpdate, rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  selector: { matchLabels: { app: hive } }
  template:
    metadata: { labels: { app: hive } }
    spec:
      serviceAccountName: hive-sa
      # ── init containers — REQUIRED. The ConfigMap is mounted read-only at
      # /etc/hive-seed and copied into a WRITABLE emptyDir at /etc/hive. The hive
      # process must WRITE /etc/hive/hive.yaml at runtime (the entrypoint seeds
      # it, then merges the PVC overlay over it, and the dashboard's Save writes
      # it). Mounting the ConfigMap DIRECTLY at /etc/hive makes it read-only and
      # every config save fails with "open /etc/hive/hive.yaml: read-only file
      # system" — and dashboard-installed GitHub App auth / ACMM changes are lost.
      initContainers:
      - name: copy-config
        image: ${IMAGE}
        command: ["sh","-c","cp /etc/hive-seed/hive.yaml /etc/hive/hive.yaml && echo configmap-copied; if [ -f /data/hive.yaml.runtime ]; then echo runtime-config-exists-for-recovery; elif [ -f /data/hive.yaml.bak ]; then echo legacy-runtime-config-exists-for-recovery; fi"]
        volumeMounts:
        - { name: config,          mountPath: /etc/hive-seed, readOnly: true }
        - { name: config-writable, mountPath: /etc/hive }
        - { name: data,            mountPath: /data }
      - name: init-permissions
        image: ${IMAGE}
        command: ["sh","-c","chown -R 1001:1000 /data 2>/dev/null || true; echo permissions-set"]
        volumeMounts:
        - { name: data, mountPath: /data }
      containers:
      - name: hive
        image: ${IMAGE}
        imagePullPolicy: Always
        ports:
        - { name: dashboard, containerPort: 3002 }
        - { name: terminal,  containerPort: 3001 }
        env:
        - { name: HIVE_ID, value: "${ID}" }
        # HIVE_HUB_SECRET authenticates every heartbeat to the hub. WITHOUT it,
        # the hub rejects the spoke's heartbeats with 401 and the hive shows
        # OFFLINE forever (see the gotcha below). HIVE_HUB_URL points at the hub.
        #
        # SECURITY (C2 domain separation): the hub no longer signs everything with
        # this key. It derives a distinct sub-key per trust domain
        # (heartbeat / session cookie / SSO), so a spoke never needs the master.
        # Automated hub-hosted provisioning injects the derived keys
        # (HIVE_HEARTBEAT_KEY, HIVE_SESSION_KEY, HIVE_SSO_KEY) and NEVER
        # HIVE_HUB_SECRET. Manual provisioning MAY still set HIVE_HUB_SECRET as
        # shown below: the spoke derives the same sub-keys from it locally, so
        # heartbeats/SSO keep working. If you prefer least privilege, set the three
        # HIVE_*_KEY vars instead (the hub derives them per-hive; the exact
        # derivation lives in src/pkg/hub/hub_keys.go — heartbeat is HMAC-based
        # while session/SSO are now Ed25519 asymmetric seeds, so do NOT assume a
        # single HMAC(master, "hive-<domain>-v1") formula) and omit
        # HIVE_HUB_SECRET entirely.
        #
        # HUB-ONLY: this whole HIVE_HUB_SECRET / derived-key block is irrelevant
        # to a standalone (hub-less) hive — see "Standalone / self-hosted (no
        # hub)" at the top of this guide.
        - { name: HIVE_HUB_SECRET, value: "${HUB_SECRET}" }
        - { name: HIVE_HUB_URL,    value: "https://hive.kubestellar.io" }
        # startupProbe keeps the liveness clock from starting until boot
        # finishes, so a slow start can't race liveness into a restart loop.
        startupProbe:
          httpGet: { path: /api/health, port: 3002 }
          initialDelaySeconds: 10
          periodSeconds: 5
          failureThreshold: 30
        livenessProbe:
          # /api/livez (NOT /api/health) — it also fails when the heartbeat
          # loop stops *attempting* sends, so a wedged/dead heartbeat
          # goroutine gets the pod auto-restarted. It does NOT fail merely
          # because the hub is unreachable (see the gotcha below).
          httpGet: { path: /api/livez, port: 3002 }
          periodSeconds: 30
          failureThreshold: 3
          timeoutSeconds: 2
        readinessProbe:
          httpGet: { path: /api/health, port: 3002 }
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
          timeoutSeconds: 2
        volumeMounts:
        - { name: data,            mountPath: /data }
        - { name: config-writable, mountPath: /etc/hive }         # WRITABLE (emptyDir), not the ConfigMap
        - { name: secrets,         mountPath: /secrets, readOnly: true }
      volumes:
      - { name: data,            persistentVolumeClaim: { claimName: hive-data } }
      - { name: config,          configMap: { name: hive-config } }   # seed → /etc/hive-seed
      - { name: config-writable, emptyDir: {} }                       # runtime /etc/hive
      - { name: secrets, secret: { secretName: hive-secrets } }
YAML

maxUnavailable: 0 keeps the current pod serving until the new is ready — uninterrupted upgrades. It works because the PVC is RWX.

Gotcha — HIVE_HUB_SECRET is REQUIRED or the hive is permanently offline. The hub authenticates every heartbeat against a shared secret. A spoke without HIVE_HUB_SECRET in its deployment env sends unauthenticated heartbeats, the hub replies 401 unauthorized, and the hive shows offline in the fleet even though the pod is 1/1 Running. Copy the value from any working hive on the same hub — it is the same for all spokes:

HUB_SECRET=$(kubectl --context "$CTX" -n hive-hosted-<any-working-hive> \
  get deploy hive -o jsonpath='{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{"\n"}{end}' \
  | grep '^HIVE_HUB_SECRET=' | cut -d= -f2-)

Symptom to look for in the spoke logs: hub heartbeat rejected status=401.

Gotcha — point the liveness probe at /api/livez, not /api/health. /api/health checks the HTTP server is up, so a pod whose heartbeat goroutine has silently died still passes it and is never restarted — the hive shows a persistent “offline” dot while 1/1 Running. /api/livez additionally fails (503) when the heartbeat loop stops attempting to send, so the kubelet restarts the pod and the heartbeat revives. It returns 200 unauthenticated healthy. Existing deployments provisioned before this note still point at /api/health — migrate them: kubectl -n <ns> patch deploy hive --type json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/httpGet/path","value":"/api/livez"}]'

Note — /api/livez does not fail on an unreachable hub. It keys off heartbeat attempts, not successes. A hub that is down, firewalled, or rejecting beats leaves the process perfectly healthy, and restarting it cannot fix a network partition — an earlier version gated liveness on heartbeat freshness and crash-looped healthy spokes on firewalled clusters (vllm-d). Heartbeat freshness is reported by /api/health/deep under the hub_heartbeat check, and the hub greys the hive’s dot on its own. Spokes deployed before this fix need a redeploy (or a probe patch) to stop restarting on hub outages.

Gotcha — some clusters enforce an owner label. A ValidatingAdmissionPolicy on vllm-d requires an owner label on PersistentVolumeClaim, ConfigMap, Secret, Service, and Deployment. Objects still apply without it, but each emits a warning. Add owner: <github-login> to every metadata.labels block to silence them.

B.9 Verify the pod comes up on anyuid

kubectl --context "$CTX" -n "$NS" rollout status deploy/hive --timeout=120s
POD=$(kubectl --context "$CTX" -n "$NS" get pods -l app=hive \
  -o jsonpath='{.items[0].metadata.name}')
# MUST be "anyuid" — if it says "restricted-v2", the anyuid binding is wrong
kubectl --context "$CTX" -n "$NS" get pod "$POD" \
  -o jsonpath='{.metadata.annotations.openshift\.io/scc}'

The hub SaaS record (meta.json) — required for both paths

Automated provisioning writes this for you. Manual provisioning does not — you must create it by hand, or the hive is invisible in My Hives and every management action (upgrade, claim) returns hive not found.

The read is loadSaaSHive(id) → /data/saas/hives/<id>/meta.json, read live from disk on every call, so a new or edited file takes effect immediately with no hub restart.

ID=hosted-myorg-myrepo
HUB_POD=$(kubectl --context hive-oke -n hive-hub get pods -l app=hive-hub \
  --no-headers | grep Running | awk '{print $1}' | head -1)
 
cat > /tmp/meta.json <<JSON
{
  "id": "${ID}",
  "owner": "owner-github-login",
  "project_name": "My Hive",
  "org": "myorg",
  "repos": ["myrepo"],
  "primary_repo": "myrepo",
  "acmm_level": 2,
  "status": "",
  "created_at": "",
  "subdomain": "",
  "auto_upgrade": true,
  "is_public": false,
  "cluster_id": "vllm-d"
}
JSON
 
kubectl --context hive-oke -n hive-hub exec "$HUB_POD" -- \
  sh -c "mkdir -p /data/saas/hives/${ID}"
kubectl --context hive-oke -n hive-hub cp /tmp/meta.json \
  hive-hub/"$HUB_POD":/data/saas/hives/${ID}/meta.json

Visibility rule. handleMyHives shows a hive to a user when the user is its owner, has a role on it, or is the hub admin (clubanderson) — even if the hive has no fleet-registry entry (never heartbeated). So a hive with owner: <someone> in its meta.json appears in that person’s My Hives (and the admin’s) as soon as the file exists.

Hub access roles. Manage Access grants four ordered roles (separate from the ClankeR trust tiers shown on /contribute): read < read-write < merger < owner. merger inherits read-write dashboard access and can approve/queue other people’s PRs for the hive auto-merge-on-green flow. The spoke enforces the self-merge ban against the GitHub login bound to the session; owners are exempt because they already have repository-level merge authority.

Queueing a PR approves it as the hive App and applies a label. That label is configurable, because it is applied in a repository the hive does not own and whose review conventions already exist:

governor:
  labels:
    automerge: lgtm

It defaults to lgtm, the long-standing Prow/Kubernetes label for “a second person signed off”, which is the decision the merger tier records. Set it to whatever the managed repositories already use — the label is created on demand if it does not exist.

meta.json requirement is why heartbeats can be accepted but upgrades fail. The heartbeat path is separate from loadSaaSHive. A hive can heartbeat and appear “online” while a missing meta.json makes the Upgrade button (and claim, toggle-auto-upgrade) return hive not found. If you see that, the fix is almost always a missing meta.json.

Config precedence — the PVC overlay is authoritative

At runtime the effective config is not the ConfigMap. The entrypoint:

1. seeds /etc/hive/hive.yaml from the hive-config ConfigMap on boot, then
2. merges the PVC dashboard overlay /data/hive.yaml.dashboard over it, and
3. writes /data/hive.yaml.runtime.

The dashboard’s Config.Save() writes the overlay. Therefore:

• Editing the ConfigMap after first boot does nothing to the running config — the PVC overlay wins.
• To change a running hive’s config, edit /data/hive.yaml.dashboard on the PVC (while scaled to 0, or the running process re-saves from memory), or clear /data/hive.yaml.dashboard + /data/hive.yaml.runtime (and the legacy /data/hive.yaml.bak, if the hive predates the rename) to let it reseed.
• Verify effective config by grep-ing /etc/hive/hive.yaml in the running pod, never the ConfigMap.

Gotcha — access grants (dashboard.authorized_users) take effect on a pod roll. The overlay merge in step 2 runs at boot. When you grant a user in the dashboard’s Manage Access dialog, Config.Save() appends them to /data/hive.yaml.dashboard, but the running device-flow authorizer keeps the access list it built at startup — config hot-reload does not rebuild it. Symptom: the user shows as read-write in Manage Access yet the spoke logs device-flow login rejected: user not authorized for this hive.

This bites hardest right after provisioning or a cross-cluster migration a fresh PVC, where the first grants are written after the pod already booted. Always finish provisioning with the verification below.

# 1. Confirm the grant reached the effective config:
kubectl -n hive-hosted-<id> exec deploy/hive -c hive -- \
  grep -A6 authorized_users /etc/hive/hive.yaml
# 2. If the user is present but still rejected, roll the pod so the authorizer
#    rebuilds from the current overlay:
kubectl -n hive-hosted-<id> rollout restart deploy/hive
# 3. Confirm no post-roll rejections:
kubectl -n hive-hosted-<id> logs deploy/hive -c hive | grep 'not authorized'

Do not hand-create /data/hive.yaml to “fix” a missing config — the effective config is ConfigMap-seed + .dashboard overlay; a stray /data/hive.yaml is never read and adds confusion.

Placeholder pools

A placeholder is a fully-provisioned but idle hive, waiting to be claimed — so a request for access is satisfied in seconds instead of the full 5–10 minute provision. Two pools, per method type:

• vllm-d placeholders serve private methods — provisioned manually (Path B) and left running at replicas: 1.
• hive-oke placeholders serve public methods — provisioned via the API (Path A) and left running at replicas: 1.

Gotcha — placeholder IDs are date + random suffix, NOT sequential integers. The live pool uses hosted-available-vllmd-<YYMMDD>-<suffix>, where <suffix> is a short unique token (4 lowercase base36 chars) — e.g. hosted-available-vllmd-260731-fikn, hosted-available-vllmd-260806-1bo3. Do not use sequential numbers like hosted-available-vllmd-01: they collide with existing pool slots and don’t match the ID convention the fleet uses, so renumbering is needed every time the pool grows. Unique suffixes let you add a batch on any date without touching existing slots. Generate a batch of N suffixes:

tr -dc 'a-z0-9' < /dev/urandom | fold -w4 | head -N | sort -u

Then collision-check each suffix before creating — skip any that already exist — against both the hub meta dirs and the target cluster’s namespaces:

kubectl --context hive-oke -n hive-hub exec "$HUB_POD" -- ls /data/saas/hives/   # existing hive IDs
kubectl --context vllm-d get ns | grep hive-hosted-hosted-available-vllmd        # existing namespaces

Provision each placeholder as above at replicas: 1. The pod boots immediately, lands on the anyuid SCC, and heartbeats to the hub as an available slot ready to be claimed. Both pools stay at replicas: 1:

# manual (vllm-d): apply the Deployment at replicas: 1 (it boots and heartbeats).
# If you applied it at 0, scale it up:
kubectl --context vllm-d -n "$NS" scale deploy/hive --replicas=1
 
# automated (hive-oke): the API already provisions at replicas=1 — leave it there.

Gotcha — do NOT scale placeholders to 0. A replicas: 0 slot is not assignable: the claim/assign path delivers the claimant’s config to a running, heartbeating spoke, so a slot that isn’t booted can never be claimed. Every assignable slot in the live pool runs at replicas: 1, ready 1/1. (Earlier guidance to “scale to 0” was incorrect.)

Give every placeholder a meta.json with owner: clubanderson (admin-only visibility), status: "available", and acmm_level: 2. Set org to a unique value per slot — available-vllmd-<YYMMDD>-<suffix> mirroring the id’s date-suffix — with empty repos: [] and primary_repo: "" (they get their real values when the slot is claimed). The meta.json also carries github_host because vllm-d placeholders target GitHub Enterprise (github.ibm.com), not github.com:

{
  "id": "hosted-available-vllmd-260806-1bo3",
  "owner": "clubanderson",
  "project_name": "Available slot (private) 260806-1bo3",
  "org": "available-vllmd-260806-1bo3",
  "repos": [],
  "primary_repo": "",
  "acmm_level": 2,
  "status": "available",
  "created_at": "",
  "subdomain": "hosted-available-vllmd-260806-1bo3.apps.fmaas-vllm-d.fmaas.res.ibm.com",
  "claim_delivered": false,
  "auto_upgrade": true,
  "is_public": false,
  "cluster_id": "vllm-d",
  "github_host": "github.ibm.com"
}

The org must be unique per slot (mirror the id’s date-suffix) so each renders as a distinct “Available slot (private) <date>-<suffix>” row in the hub. A literal shared org: "placeholder" (with repos: ["placeholder"]) makes every slot render identically as “placeholder / placeholder” — indistinguishable — and is wrong.

Verify a provisioned placeholder

A correct vllm-d placeholder runs at replicas: 1 with its pod up (Running, ready, restartCount: 0) on the anyuid SCC, its hive-anyuid binding points at its own namespace, its PVC is Bound, its livez returns 200, and its hub meta.json exists. This set verified all ten placeholders in the 2026-08-06 batch:

ID=hosted-available-vllmd-<date>-<suffix>; NS=hive-hosted-$ID
kubectl --context vllm-d -n $NS get deploy hive -o jsonpath='{.spec.replicas}'          # must be 1
POD=$(kubectl --context vllm-d -n $NS get pod -l app=hive -o jsonpath='{.items[0].metadata.name}')
kubectl --context vllm-d -n $NS get pod $POD -o jsonpath='{.status.phase}'              # must be Running
kubectl --context vllm-d -n $NS get pod $POD -o jsonpath='{.metadata.annotations.openshift\.io/scc}'  # must be anyuid
kubectl --context vllm-d -n $NS get pod $POD -o jsonpath='{.status.containerStatuses[0].ready}'       # must be true
kubectl --context vllm-d -n $NS get pod $POD -o jsonpath='{.status.containerStatuses[0].restartCount}' # must be 0
kubectl --context vllm-d -n $NS exec $POD -c hive -- curl -s -o /dev/null -w '%{http_code}' localhost:3002/api/livez  # must be 200
kubectl --context vllm-d -n $NS get rolebinding hive-anyuid -o jsonpath='{.subjects[0].namespace}'  # must equal $NS
kubectl --context vllm-d -n $NS get pvc hive-data -o jsonpath='{.status.phase}'         # must be Bound
kubectl --context hive-oke -n hive-hub exec $HUB_POD -- test -f /data/saas/hives/$ID/meta.json  # must exist

The hive-anyuid subject-namespace check is still load-bearing at apply time (it is the same restricted-v2 crash-loop gotcha from B.2): a wrong subject namespace means the pod can’t get the anyuid SCC. But because a placeholder now boots at replicas: 1, you confirm the SCC directly — verify the pod reaches phase=Running, scc=anyuid, ready=true, restartCount=0, and livez == 200.

Note — placeholders carry a fleet-registry entry. Because both pools run at replicas: 1, every placeholder heartbeats and holds a registry entry that pins its version and last-known ACMM and shows it “online”. This is expected and required: the claim/assign path delivers config to the running, heartbeating spoke, so the entry is what makes the slot assignable — do not strip it.

Claiming a placeholder

Until the dashboard “assign” flow lands, claiming is manual:

1. Edit the placeholder’s meta.json: set the real owner, org, repos, primary_repo, acmm_level, and is_public.
2. Update the running config on the PVC overlay (/data/hive.yaml.dashboard) — real project.org / repos, and the claimant’s GitHub App app_id / installation_id.
3. Install the GitHub App key (owner does this from the dashboard’s Install GitHub App flow — the hive is already awake at replicas: 1).
4. Roll the pod so it picks up the new config (kubectl rollout restart deploy/hive) — it is already running, so there is nothing to scale up.

Mapping a namespace back to its hive (identity labels)

A claimed placeholder’s namespace keeps its original placeholder name forever — e.g. hive-hosted-hosted-available-oke-01-placeholder-bb95 — even after the hub API’s claim/assign flow gives the hive a real name and org. Kubernetes has no atomic namespace-rename primitive, and renaming would mean recreating every object inside it (including the PVC), so the namespace name is never changed. Historically this meant there was no way to look at a namespace on the cluster and tell which hive/org it belonged to without cross-referencing the hub registry and reverse-engineering the placeholder slug.

Instead, the hub stamps identity labels and a display-name annotation the namespace itself, keeping it self-describing even though its name never changes:

Sanitization: label values are lower-cased, restricted to [a-z0-9.-] (everything else — spaces, underscores, unicode, punctuation — is dropped outright rather than transliterated), trimmed of leading/trailing separators, and capped at 63 characters (the RFC-1123 label-value limit). So a hive named "TradingAsBuddies" gets the label value tradingasbuddies, while the hive.kubestellar.io/display-name annotation preserves the original TradingAsBuddies exactly. A field that sanitizes to empty (or wasn’t set yet — e.g. a fresh placeholder with no real name/org) is omitted from the labels rather than written as an empty/invalid label.

Given a hive name, find its namespace:

kubectl --context hive-oke get ns \
  -l hive.kubestellar.io/hive-name=tradingasbuddies

Given a namespace, find the hive that owns it:

kubectl --context hive-oke get ns hive-hosted-hosted-available-oke-01-placeholder-bb95 \
  -o jsonpath='{.metadata.annotations.hive\.kubestellar\.io/display-name}'
# -> TradingAsBuddies

When the labels/annotation are (re)written — the hub calls the same stampHostedNamespaceIdentity helper at all three points a hive’s identity is known or changes, so the namespace never goes stale:

1. Automatic provisioning (provisionHive, Path A above) — stamps whatever is known at namespace-creation time. For a freshly-created pool placeholder this is often just hive.kubestellar.io/hive-id, since org/name are still placeholder values at this point.
2. Claim (handleApproveProvision) — the real name/org become known here for the first time, so the labels/annotation are (re)written.
3. Assign / reassign (handleAssignHive) — same, so a placeholder that gets reassigned to a different owner has its namespace identity refreshed too.

The stamp is applied via idempotent kubectl label/kubectl annotate --overwrite (merged into whatever the namespace already carries, not replaced) and is best-effort: a failure to label (no kubectl on PATH, a transient API error) is logged and does not fail the claim/assign request — this is a cosmetic/operability nicety, not a required step for the hive to work.

Name-bearing vanity Route

The same problem — a URL permanently encoding the placeholder slug — applies to the dashboard’s public URL. The assign path (handleAssignHive) also creates an additional OpenShift Route whose host is derived from the hive’s own display name, rather than the org/repo-derived host used previously:

<sanitized-hive-name>-<4-char-suffix>.<cluster-domain>

For example, a hive named TradingAsBuddies on a cluster with domain apps.example.com gets a Route host like tradingasbuddies-a1b2.apps.example.com, adopted as https://tradingasbuddies-a1b2.apps.example.com. The random suffix keeps two hives with similar/identical sanitized names from colliding on the same host.

This is an additional Route, not a rename: the original Route (host derived from org/repo or the placeholder ID) is left completely alone, so any in-flight bookmark or callback against it keeps working. The namespace itself is still never renamed or recreated — a second Route pointing at the same Service is added. If the hive has no display name set, the vanity host falls back to the previous org/repo-derived scheme.

Namespace shown in the dashboards

The namespace is now surfaced in two read-only UI spots, so an operator or hive owner doesn’t need kubectl at all to find it:

• Spoke dashboard → Hub tab. The hive’s own dashboard shows its Kubernetes namespace next to the hub URL/name info. The spoke learns its own namespace from the POD_NAMESPACE env var, set via the Kubernetes downward API (fieldRef: metadata.namespace) on the hive Deployment — falling back to a NAMESPACE env var, then to the in-cluster service-account namespace file, if POD_NAMESPACE isn’t set (e.g. a spoke provisioned before this env var existed). The row is omitted entirely when none of the three resolve.
• Hub dashboard → per-spoke status hover. Hovering a hive in the hub’s fleet view shows its namespace, computed client-side as "hive-hosted-" + <hive-id> — the same convention the hub uses server-side — so it can never disagree with the namespace the identity labels above are stamped on.

Deprovisioning

# delete the workload namespace
kubectl --context "$CTX" delete ns "$NS"
# remove the hub SaaS record
kubectl --context hive-oke -n hive-hub exec "$HUB_POD" -- \
  rm -rf /data/saas/hives/"$ID"
# if it ever heartbeated, also drop its registry entry (see the placeholder gotcha)

Quick failure-mode reference