> ## Documentation Index
> Fetch the complete documentation index at: https://docs.booleinference.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Boole AI Local Quickstart: GPU Inference in Minutes

> Download the Boole binary, start serving a model on your NVIDIA GPU, and get your first tokens in under 4 minutes — no Python or dependencies needed.

This guide walks you through downloading the Boole binary, starting a local inference server, and making your first request — all in under four minutes. Because the binary ships with weights embedded and has no external dependencies, setup is a single download followed by a single command. Once the server is running, it exposes the same OpenAI-compatible API as the cloud, so any existing code works without modification.

<Note>
  **Prerequisites**

  * An NVIDIA GPU with CUDA drivers installed (driver version 520 or later recommended)
  * Linux (x86\_64) or macOS (Apple Silicon)
  * Approximately 50 GB of free disk space for 70B-parameter models; smaller models require significantly less
</Note>

## Steps

<Steps>
  <Step title="Download the binary">
    Download the latest Boole binary for your platform. Always check [booleinference.com](https://booleinference.com) for the most current release before downloading.

    ```bash theme={null}
    curl -L -o boole-latest-linux-x86_64 \
      https://booleinference.com/download/boole-latest-linux-x86_64
    ```

    <Tip>
      Visit [booleinference.com](https://booleinference.com) to find the latest release URL for your platform, including macOS (Apple Silicon) builds.
    </Tip>
  </Step>

  <Step title="Make the binary executable">
    Grant execute permissions so your shell can run the file directly.

    ```bash theme={null}
    chmod +x boole-latest-linux-x86_64
    ```
  </Step>

  <Step title="Start the inference server">
    Launch the server with the model you want to serve. Boole will load the embedded weights and begin accepting requests.

    ```bash theme={null}
    ./boole-latest-linux-x86_64 serve --model llama-3.3-70b-instruct
    ```

    <Note>
      Cold start is under 400 ms. Once you see `Listening on http://localhost:8000`, the server is ready. On a single NVIDIA A10, Llama 3.3 70B serves at **312 tokens/sec**.
    </Note>
  </Step>

  <Step title="Test with curl">
    Send a chat completion request to confirm everything is working.

    ```bash theme={null}
    curl http://localhost:8000/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
        "model": "llama-3.3-70b-instruct",
        "messages": [{"role": "user", "content": "Hello!"}]
      }'
    ```

    You should receive a JSON response with the model's reply in `choices[0].message.content`.
  </Step>
</Steps>

## Using with Existing Code

Because the local server is fully OpenAI-compatible, you can point any OpenAI SDK client at `http://localhost:8000/v1` with no other changes. Pass any non-empty string as the API key — the local server does not validate it.

```python app.py theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="local",  # any non-empty string
)

response = client.chat.completions.create(
    model="llama-3.3-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```

<Tip>
  Boole collects **zero telemetry** and stores no prompts. After the initial download, the binary and its embedded weights run entirely offline — ideal for air-gapped environments or privacy-sensitive workloads.
</Tip>

## Next Steps

<CardGroup cols={2}>
  <Card title="Advanced Local Configuration" icon="sliders" href="/deployment/local-binary">
    Configure ports, concurrency limits, model caching paths, and multi-GPU setups.
  </Card>

  <Card title="Available Models" icon="layer-group" href="/concepts/models">
    Browse all 42 supported models — from Llama and Qwen to Whisper — and check hardware requirements for each.
  </Card>
</CardGroup>