> ## Documentation Index
> Fetch the complete documentation index at: https://docs.booleinference.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scale Boole AI on Kubernetes: GPU Pods, PVCs, and Probes

> Deploy Boole AI on Kubernetes: schedule GPU pods, persist model weights with a PVC, and add liveness and readiness probes for production use.

Boole AI runs on Kubernetes using standard GPU workload patterns. Use a Deployment with a GPU resource limit to schedule pods onto GPU nodes, and a PersistentVolumeClaim to cache model weights across pod restarts so you avoid re-downloading weights every time a pod is rescheduled.

## Prerequisites

Before applying the manifests below, make sure your cluster has:

* **Kubernetes 1.24+**
* **NVIDIA GPU Operator** installed on the cluster ([installation guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/getting-started.html))
* At least one GPU node available and schedulable

## Basic Deployment Manifest

The manifest below creates a single-replica Deployment and a ClusterIP Service. Adjust the model slug, replica count, and namespace to match your environment.

```yaml boole-deployment.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: boole-inference
  labels:
    app: boole-inference
spec:
  replicas: 1
  selector:
    matchLabels:
      app: boole-inference
  template:
    metadata:
      labels:
        app: boole-inference
    spec:
      containers:
        - name: boole
          image: ghcr.io/boole-ai/boole:latest
          args: ["serve", "--model", "llama-3.3-70b-instruct"]
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
          volumeMounts:
            - name: weights
              mountPath: /weights
      volumes:
        - name: weights
          persistentVolumeClaim:
            claimName: boole-weights-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: boole-inference
spec:
  selector:
    app: boole-inference
  ports:
    - port: 80
      targetPort: 8000
  type: ClusterIP
```

Apply it with:

```bash theme={null}
kubectl apply -f boole-deployment.yaml
```

## PersistentVolumeClaim

Create a PVC to store downloaded model weights. This prevents the pod from re-downloading weights every time it restarts or is rescheduled.

```yaml boole-pvc.yaml theme={null}
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: boole-weights-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
```

Apply the PVC before the Deployment:

```bash theme={null}
kubectl apply -f boole-pvc.yaml
```

<Info>
  50 Gi is sufficient for a single 70B-class model. Increase storage if you plan to cache multiple models on the same volume.
</Info>

## Scaling

Each running pod requires a dedicated GPU. To increase inference concurrency, scale the number of replicas — the Kubernetes scheduler places each new pod on a node that has a free GPU.

```bash theme={null}
kubectl scale deployment boole-inference --replicas=4
```

Keep one model per pod for isolation. Sharing a GPU across multiple pods degrades throughput and makes resource accounting unpredictable.

<Note>
  GPU node pools on managed Kubernetes services (GKE, EKS, AKS) typically require specific node selectors or tolerations to schedule onto GPU nodes. Add a `nodeSelector` or `tolerations` block to the pod spec to match your cloud provider's GPU node labels.
</Note>

## Health Checks

Add liveness and readiness probes to the container spec so Kubernetes can detect and recover from failed inference processes without manual intervention.

```yaml boole-deployment.yaml theme={null}
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 5
```

* **`/health`** — returns `200` when the process is alive. If this probe fails, Kubernetes restarts the container.
* **`/ready`** — returns `200` when the model is loaded and the server can accept requests. Traffic is only routed to the pod after this probe succeeds.

Set `initialDelaySeconds: 30` on both probes to give the server enough time to load the model before Kubernetes starts polling. The cold start time is under 400 ms, but the 30-second buffer accounts for image pull time and volume mount latency on first launch.