Bulk & batch

Indexing a document collection means embedding many texts at once. flinq has no special bulk mode today; you batch inputs per request and run a few requests concurrently under the rate limit. This page covers the request cap, a ready concurrency recipe, the limits to stay under, and the asynchronous batch endpoint on the roadmap.

Inputs per request

The cap on inputs per request is per model: 2048 for flinq-pilot-otter and 256 for flinq-pilot-wombat (see Models). A request over the cap returns 422 rather than truncating silently. Keep each request below the serving model’s cap and split a larger corpus across several requests.

Concurrency recipe

The pattern is simple: chunk the corpus into arrays under the cap, then embed a few chunks in parallel so latency overlaps. An embed_many helper does exactly this, and you can copy it verbatim:

from concurrent.futures import ThreadPoolExecutor

MAX_BATCH = 2048  # per-model cap: 2048 (otter), 256 (wombat)

def embed_many(client, texts, *, batch_size=512, concurrency=4):
    batch_size = min(batch_size, MAX_BATCH)
    chunks = [texts[i:i + batch_size] for i in range(0, len(texts), batch_size)]
    out = [None] * len(chunks)
    with ThreadPoolExecutor(max_workers=max(1, concurrency)) as ex:
        futures = {ex.submit(client.embed, c): i for i, c in enumerate(chunks)}
        for fut in futures:
            out[futures[fut]] = fut.result()
    return [vec for chunk in out for vec in chunk]

Start with batch_size=512 and concurrency=4, then raise concurrency until you start seeing 429 responses, and step back one notch. Honor the Retry-After header on any 429 and retry with backoff; the OpenAI SDK does this for you.

Limits to stay under

• Inputs per request: per model (2048 otter / 256 wombat), hard, returns 422 over the cap.
• Requests per minute (RPM): a per-key ceiling. Over it the API returns 429 with a Retry-After header. Pace concurrency against this, not against raw thread count.
• Tokens per minute (TPM) and your credit balance: all usage draws down one EUR meter at EUR 0.40 per 1M tokens. A large ingest can spend a lot of credit quickly, so make sure your balance covers the run, or set up auto-recharge, before you start. See Pricing and Rate limits and errors.

Async batches (roadmap)

An asynchronous POST /v1/batches endpoint is a documented fast-follow. It is OpenAI-shaped: submit a file of requests, poll for completion, and download the results, the same flow as the OpenAI Batch API. Because the work runs offline rather than in your request path, it is billed at a 50% discount on the standard per-token EUR rate, which makes it the cost-efficient path for large, non-interactive ingests. Until it ships, the concurrency recipe above is the recommended way to embed a corpus.