Skip to main content
The Boole AI local binary is a single self-contained executable with model weights embedded. It requires no Python runtime, no dependency management, and no model zoo. Download once, run anywhere — on an air-gapped server, a workstation, or any Linux or macOS machine with a supported GPU.

​
System Requirements

Before you download, confirm your machine meets these requirements:
  • GPU: NVIDIA GPU with CUDA 12.0+ drivers installed
  • OS: Linux x86_64 or macOS (Apple Silicon)
  • Disk: ~50 GB per 70B-class model; ~14 GB per 8B-class model
  • RAM: 16 GB system RAM minimum

​
Download and Install

Visit booleinference.com/download and grab the binary for your platform. Always check the download page for the latest release — nightly compiler improvements are shipped as new binaries. Once downloaded, make the binary executable:
Optionally move it onto your PATH so you can invoke it from anywhere:

​
Starting the Server

Run boole serve with the model slug you want to load:
The server starts on port 8000 by default. Cold start takes under 400 ms — the nightly compiler pre-fuses kernels and pre-pages the KV cache so the first request is ready almost immediately.

​
Common Flags

​
string
required
The model slug to load and serve. Browse available slugs at booleinference.com/models.
​
integer
default:"8000"
The port the HTTP server listens on.
​
string
default:"127.0.0.1"
The bind address. Set to 0.0.0.0 to accept connections from other machines on your network.
​
integer
default:"auto"
Maximum number of concurrent requests. Defaults to a value the compiler selects based on available VRAM.
​
string
default:"auto"
Quantization level: int4, int8, or fp16. Defaults to the level the nightly compiler chose when it built this binary. Override only when you have a specific quality-vs-memory trade-off in mind.

​
Testing the Server

Once the server is running, send a chat completion request to confirm everything is working:
A successful response returns a JSON object with a choices array containing the model’s reply.

​
Updating the Binary

The Boole AI nightly compiler continuously improves throughput — the same model weights can run meaningfully faster after a compiler update. To pick up the latest optimizations, download a fresh binary from booleinference.com/download and replace the existing one:
No model re-download is required; the weights are embedded in the new binary.
The local binary runs with zero telemetry. No data leaves your machine — your prompts, responses, and model weights stay entirely local.
Prefer a containerized workflow? See Docker deployment to run the same binary inside a Docker container with GPU passthrough.