Measuring latency

Every successful API response includes a standard Server-Timing header with server-side latency measurements.

Server-Timing: processing;dur=31.669

Metric	What it measures
processing	Service time from receiving the request until the response headers are ready.

What happens before those headers depends on the response mode:

• Complete JSON response: generation is buffered, so processing includes prompt processing and generation.
• Streaming response: headers are sent before the first token, so processing includes request handling and stream setup but excludes prompt processing and generation that happen after the headers.

For either mode, subtract processing from client-observed time to response headers to estimate everything outside that window: your client, plus the network path in front of the service. Treat the result as an approximation, because the two measurements are taken at different points. Do not subtract processing from streaming time-to-first-token: those spans end at different events, so the residual would mislabel model work as network time.

Streaming latency

Response headers cannot report work that happens after streaming begins. Measure these client-side boundaries instead:

Measurement	How to measure it
Response headers	Request start to response headers.
First-token wait	Response headers to the first content event.
Generation	First content event to the end of the stream.
Total latency	Request start to stream end.

Time to first token is the first two rows combined. Measured client-side, these boundaries cover the whole request; the response itself cannot separate network time from model time.

For short outputs, generation time may be too small to measure reliably. Use total latency when comparing the end-to-end experience.

Reading it

cURL

curl -sS -D - -o /dev/null \
  https://inference.celeris.ai/celeris-1/v1/chat/completions \
  -H "Authorization: Bearer $CELERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"celeris-1","messages":[{"role":"user","content":"ok"}],"max_tokens":256}' \
  | grep -i '^server-timing'

Python

import os

import httpx

with httpx.Client() as http:
    r = http.post(
        "https://inference.celeris.ai/celeris-1/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['CELERIS_API_KEY']}"},
        json={
            "model": "celeris-1",
            "messages": [{"role": "user", "content": "ok"}],
            "max_tokens": 256,
        },
    )
    r.raise_for_status()
    print(r.elapsed.total_seconds() * 1000, "ms wall clock")
    print("Server-Timing:", r.headers["server-timing"])

TypeScript

// Server-Timing is CORS-exposed (Access-Control-Expose-Headers), so browser
// code can read it straight off the fetch Response:
const url = 'https://inference.celeris.ai/celeris-1/v1/chat/completions';
const init = {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CELERIS_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'celeris-1',
    messages: [{role: 'user', content: 'ok'}],
    max_tokens: 256,
    stream: false,
  }),
};
const started = performance.now();
const response = await fetch(url, init);
if (!response.ok) throw new Error(`Celeris request failed: ${response.status}`);
const timeToHeadersMs = performance.now() - started;

const serverTiming = response.headers.get('server-timing') ?? '';
const match = serverTiming.match(/processing;dur=([\d.]+)/);
const processingMs = match ? Number(match[1]) : undefined;
const networkMs = processingMs == null ? undefined : Math.max(0, timeToHeadersMs - processingMs);
console.log({serverTiming, timeToHeadersMs, processingMs, networkMs});

Getting the numbers down

• Deploy close to the region. Typical short requests complete in roughly 50–150 ms within the region. Network distance adds to end-to-end latency. See Models § Regions.
• Reuse connections. Use an HTTP client with connection pooling and keep-alive enabled. OpenAI SDK clients do this by default.
• Cap max_tokens. Completion length is the main thing you control that moves inference time. Any positive integer that fits the context window is accepted. Small budgets do not measurably speed up a short response, so size max_tokens to cap a long one rather than as a fine-grained speed control.
• Watch your 429 rate. Throttling and the retries it forces add latency that your own timings will show. See Rate limits.