Skip to main content
Answers to the most common questions about Boole AI — covering local deployment, the cloud API, pricing, data privacy, and model availability. If you don’t find what you’re looking for here, reach out to support@boole.ai.

​
General

Boole AI is a high-throughput LLM inference platform. You can run models locally using the Boole binary on your own GPU, or call the cloud API hosted at https://api.boole.dev/v1. Both surfaces expose the same OpenAI-compatible interface, so switching between local and cloud requires only a base_url change.Boole’s nightly compiler optimizes every supported model for maximum throughput — delivering, for example, 312 tokens/sec on Llama 3.3 70B on a single NVIDIA A10 GPU.
Boole runs a nightly compiler pipeline called Crucible that performs three core optimizations automatically:
  • Kernel fusion — combines adjacent GPU operations to eliminate memory round-trips between kernels.
  • Weight quantization — reduces weight precision where quality impact is negligible, shrinking memory bandwidth requirements.
  • KV cache paging — manages the attention cache in fixed-size pages to prevent memory fragmentation and enable larger effective batch sizes.
To date the compiler has run 2,057 iterations, retaining 217 of the best-performing optimization passes. The result compounds: throughput improves continuously without any changes on your end.
Yes. Boole exposes the same API surface as OpenAI — the same endpoints, request shapes, and response formats. To migrate an existing OpenAI integration, change two values:
No other code changes are required. Streaming, tool calls, and function calling are all supported.
Boole currently supports 42 models, all optimized by the nightly compiler. The catalogue includes Llama 3.3 70B, Qwen 2.5 72B, Mixtral 8x22B, DeepSeek V3, Whisper Large v3, and many others.See the full list and per-model specifications in Models.

​
Pricing & Billing

Local deployment is free — there are no per-token fees when you run Boole on your own GPU.Cloud API pricing starts at $0.08 per million input tokens (DeepSeek V3). Example rates:See the full pricing table at Pricing.
No — Boole’s prices only decrease over time. As the nightly compiler finds more efficient optimization passes, the cost to serve each token falls, and those savings are passed to you.In the event of any price increase (which has not happened), Boole will provide a minimum of 30 days’ advance notice.
Every new account receives $20 in credits on signup — equivalent to approximately 1 million tokens on Llama 3.3 70B. No credit card is required to claim them.Credits apply automatically to cloud API usage and expire after 12 months if unused.
No. When you run the Boole binary on your own hardware, there are no per-token fees, no subscription fees, and no license fees. You pay only for the GPU hardware and electricity — Boole charges nothing.

​
Privacy & Security

No. Boole operates a zero-retention policy: your prompts and completions are processed in memory to generate a response and are not written to disk or logged after the response is returned.This applies to both the cloud API and the local binary.
Yes. Boole AI is SOC 2 Type II certified, covering security, availability, and confidentiality. All cloud API traffic is encrypted with TLS 1.3.Contact security@boole.ai to request a copy of the audit report or a security review questionnaire.
Yes. The local binary is a single self-contained executable with weights embedded. Once downloaded, it requires no internet connection at inference time — no telemetry calls, no license checks, no external requests of any kind.This makes Boole suitable for air-gapped environments and deployments with strict network egress policies.
No. The local binary has zero telemetry by default. It does not send usage data, crash reports, or any other information to Boole’s servers. All inference happens entirely on your machine.

​
Technical

Any NVIDIA GPU with CUDA 12.0 or later is supported. Recommended hardware by model size:The NVIDIA A10 (24 GB) can run 70B models with INT4 quantization enabled via --quantize int4. For full-precision 70B inference, use an A100 (40 GB or 80 GB).
The default rate limit is 60 requests per second per API key on the cloud API. There is no rate limit on the local binary.If your workload requires a higher cloud rate limit, contact support@boole.ai with your account email and expected request volume.
Yes. All text generation models support Server-Sent Events (SSE) streaming. Enable it by setting stream=True in your request:
Streaming is supported on both the cloud API and the local binary.
Yes. Tool calls and function calling are fully supported on all chat models. The request and response format matches the OpenAI specification exactly, so any existing function-calling code works without modification.