> ## Documentation Index
> Fetch the complete documentation index at: https://docs.booleinference.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Boole AI with Docker and NVIDIA GPU Passthrough

> Pull the official Boole AI Docker image, pass through your NVIDIA GPU with a single flag, and start serving LLM inference in one containerized command.

Boole AI is available as a Docker image for teams that prefer containerized deployments. The image bundles the binary and all required CUDA libraries — there is nothing extra to install inside the container. Pass through your GPU and start serving in one command.

## Prerequisites

Make sure the following are in place before you start:

* **Docker 24+** installed on the host
* **NVIDIA Container Toolkit** installed and configured ([installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html))
* **NVIDIA GPU** with CUDA 12.0+ drivers on the host

## Pull and Run

The image is hosted on the GitHub Container Registry. Pull it implicitly by running:

```bash theme={null}
docker run --gpus all \
  -p 8000:8000 \
  ghcr.io/boole-ai/boole:latest \
  serve --model llama-3.3-70b-instruct
```

The `--gpus all` flag passes every available GPU into the container. Replace `all` with a specific device index (e.g. `device=0`) if you want to target a single card on a multi-GPU host.

## Docker Compose

For persistent or multi-service setups, define the Boole service in a Compose file:

```yaml docker-compose.yml theme={null}
version: "3.9"
services:
  boole:
    image: ghcr.io/boole-ai/boole:latest
    command: serve --model llama-3.3-70b-instruct
    ports:
      - "8000:8000"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
```

Start the service with:

```bash theme={null}
docker compose up
```

## Environment Variables

You can configure the server through environment variables instead of CLI flags. Set them with `-e` in `docker run` or under `environment:` in your Compose file.

<ParamField path="BOOLE_MODEL" type="string" required>
  Model slug to serve. Equivalent to the `--model` flag.
</ParamField>

<ParamField path="BOOLE_PORT" type="integer" default="8000">
  Port the server listens on inside the container. Remember to update your `-p` port mapping to match if you change this.
</ParamField>

<ParamField path="BOOLE_HOST" type="string" default="0.0.0.0">
  Bind address. Defaults to `0.0.0.0` inside the container so the port mapping works correctly. Change to `127.0.0.1` if you only want loopback access within the container network.
</ParamField>

## Persistent Weight Cache

By default, the container downloads model weights to a temporary layer that disappears when the container is removed. Mount a host volume to cache weights across container restarts and avoid re-downloading on each run:

```bash theme={null}
docker run --gpus all \
  -v /data/boole-weights:/weights \
  -p 8000:8000 \
  ghcr.io/boole-ai/boole:latest \
  serve --model llama-3.3-70b-instruct --weight-cache /weights
```

On the first run, Boole writes the weights to `/data/boole-weights` on your host. Subsequent runs load from the cache and skip the download entirely.

<Tip>
  Running at scale? See [Kubernetes deployment](/deployment/kubernetes) to orchestrate Boole AI across a GPU node pool with automatic scheduling and persistent volume claims.
</Tip>