Boole AI Local Quickstart: GPU Inference in Minutes

This guide walks you through downloading the Boole binary, starting a local inference server, and making your first request — all in under four minutes. Because the binary ships with weights embedded and has no external dependencies, setup is a single download followed by a single command. Once the server is running, it exposes the same OpenAI-compatible API as the cloud, so any existing code works without modification.

Prerequisites

An NVIDIA GPU with CUDA drivers installed (driver version 520 or later recommended)
Linux (x86_64) or macOS (Apple Silicon)
Approximately 50 GB of free disk space for 70B-parameter models; smaller models require significantly less

Steps

Download the binary

Download the latest Boole binary for your platform. Always check booleinference.com for the most current release before downloading.

curl -L -o boole-latest-linux-x86_64 \
  https://booleinference.com/download/boole-latest-linux-x86_64

Visit booleinference.com to find the latest release URL for your platform, including macOS (Apple Silicon) builds.

Make the binary executable

Grant execute permissions so your shell can run the file directly.

chmod +x boole-latest-linux-x86_64

Start the inference server

Launch the server with the model you want to serve. Boole will load the embedded weights and begin accepting requests.

./boole-latest-linux-x86_64 serve --model llama-3.3-70b-instruct

Cold start is under 400 ms. Once you see Listening on http://localhost:8000, the server is ready. On a single NVIDIA A10, Llama 3.3 70B serves at 312 tokens/sec.

Test with curl

Send a chat completion request to confirm everything is working.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

You should receive a JSON response with the model’s reply in choices[0].message.content.

Using with Existing Code

Because the local server is fully OpenAI-compatible, you can point any OpenAI SDK client at http://localhost:8000/v1 with no other changes. Pass any non-empty string as the API key — the local server does not validate it.

app.py

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local",  # any non-empty string
)

response = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Boole collects zero telemetry and stores no prompts. After the initial download, the binary and its embedded weights run entirely offline — ideal for air-gapped environments or privacy-sensitive workloads.

Next Steps

Advanced Local Configuration

Configure ports, concurrency limits, model caching paths, and multi-GPU setups.

Available Models

Browse all 42 supported models — from Llama and Qwen to Whisper — and check hardware requirements for each.

Get Started

Core Concepts

Deployment

Integration

Account & Billing

Support

Boole AI Local Quickstart: GPU Inference in Minutes

Steps

Using with Existing Code

Next Steps

Advanced Local Configuration

Available Models

​Steps

​Using with Existing Code

​Next Steps

Advanced Local Configuration

Available Models

Steps

Using with Existing Code

Next Steps