API Rate Limits for the Boole AI Cloud: 60 req/sec

The Boole AI cloud API enforces rate limits to ensure fair, stable access for all users. By default, each API key is limited to 60 requests per second. If your application requires higher throughput, you can request an increase — limits are not a hard ceiling for all workloads.

Default Limits

Limit	Default
Requests per second (per API key)	60
Concurrent connections	Unlimited
Max tokens per request	Model-dependent (typically 32,768)
Max requests per day	Unlimited

Rate limits apply per API key, not per account. If you need more than 60 req/sec across a single workload, you can either request a higher limit on one key or distribute traffic across multiple keys.

Rate Limit Errors

When a request exceeds your limit, the API responds with HTTP 429 Too Many Requests:

{
  "error": {
    "message": "Rate limit exceeded. Max 60 requests per second.",
    "type": "rate_limit_error",
    "code": 429
  }
}

Do not treat a 429 as a fatal error. It signals that you should back off briefly and retry — the API does not penalize retries.

Handling Rate Limits

Implement exponential backoff with jitter so your application recovers automatically from bursts that exceed the limit:

retry.py

import time
import random

def call_with_retry(client, **kwargs):
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except Exception as e:
            if "429" in str(e):
                wait = (2 ** attempt) + random.uniform(0, 1)
                print(f"Rate limited. Retrying in {wait:.2f}s (attempt {attempt + 1}/5)")
                time.sleep(wait)
            else:
                raise
    raise RuntimeError("Max retries exceeded after 5 attempts")

The backoff schedule for this implementation:

Attempt	Base wait	Max wait (with jitter)
1	2s	~3s
2	4s	~5s
3	8s	~9s
4	16s	~17s
5	—	RuntimeError raised

Use batching and connection pooling to stay within the limit more easily. Sending multiple messages inside a single chat/completions call counts as one request against your rate limit, not one per message.

Requesting Higher Limits

If 60 req/sec is not enough for your workload, email support@boole.ai with:

Your account email
The model(s) you need higher limits for
Your expected request volume (req/sec and daily total)
A brief description of your use case

The team typically responds within one business day. Increases can be applied to an existing key or a new dedicated key, depending on your preference.

Local binary deployments have no rate limits. Throughput is constrained only by your GPU hardware — for example, 312 tokens/sec on Llama 3.3 70B on a single NVIDIA A10 GPU.

⌘I

Get Started

Core Concepts

Deployment

Integration

Account & Billing

Support

API Rate Limits for the Boole AI Cloud: 60 req/sec

Default Limits

Rate Limit Errors

Handling Rate Limits

Requesting Higher Limits

​Default Limits

​Rate Limit Errors

​Handling Rate Limits

​Requesting Higher Limits

Default Limits

Rate Limit Errors

Handling Rate Limits

Requesting Higher Limits