# Measuring latency

Every successful API response includes a standard
[`Server-Timing`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing)
header with server-side latency measurements.

```http
Server-Timing: processing;dur=31.669
```

| Metric | What it measures |
| --- | --- |
| `processing` | Service time from receiving the request until the response headers are ready. |

What happens before those headers depends on the response mode:

- **Complete JSON response:** generation is buffered, so `processing` includes
  prompt processing and generation.
- **Streaming response:** headers are sent before the first token, so
  `processing` includes request handling and stream setup but excludes prompt
  processing and generation that happen after the headers.

For either mode, subtract `processing` from client-observed time to response
headers to estimate everything outside that window: your client, plus the
network path in front of the service. Treat the result as an approximation,
because the two measurements are taken at different points. Do not subtract `processing` from streaming
time-to-first-token: those spans end at different events, so the residual would
mislabel model work as network time.

## Streaming latency

Response headers cannot report work that happens after streaming begins. Measure
these client-side boundaries instead:

| Measurement | How to measure it |
| --- | --- |
| Response headers | Request start to response headers. |
| First-token wait | Response headers to the first content event. |
| Generation | First content event to the end of the stream. |
| Total latency | Request start to stream end. |

Time to first token is the first two rows combined. Measured client-side, these
boundaries cover the whole request; the response itself cannot separate network
time from model time.

For short outputs, generation time may be too small to measure reliably. Use
total latency when comparing the end-to-end experience.

## Reading it

**cURL**

```bash
curl -sS -D - -o /dev/null \
  https://inference.celeris.ai/celeris-1/v1/chat/completions \
  -H "Authorization: Bearer $CELERIS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"celeris-1","messages":[{"role":"user","content":"ok"}],"max_tokens":256}' \
  | grep -i '^server-timing'
```

**Python**

```python

with httpx.Client() as http:
    r = http.post(
        "https://inference.celeris.ai/celeris-1/v1/chat/completions",
        headers={"Authorization": f"Bearer {os.environ['CELERIS_API_KEY']}"},
        json={
            "model": "celeris-1",
            "messages": [{"role": "user", "content": "ok"}],
            "max_tokens": 256,
        },
    )
    r.raise_for_status()
    print(r.elapsed.total_seconds() * 1000, "ms wall clock")
    print("Server-Timing:", r.headers["server-timing"])
```

**TypeScript**

```typescript
// Server-Timing is CORS-exposed (Access-Control-Expose-Headers), so browser
// code can read it straight off the fetch Response:
const url = 'https://inference.celeris.ai/celeris-1/v1/chat/completions';
const init = {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${process.env.CELERIS_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'celeris-1',
    messages: [{role: 'user', content: 'ok'}],
    max_tokens: 256,
    stream: false,
  }),
};
const started = performance.now();
const response = await fetch(url, init);
if (!response.ok) throw new Error(`Celeris request failed: ${response.status}`);
const timeToHeadersMs = performance.now() - started;

const serverTiming = response.headers.get('server-timing') ?? '';
const match = serverTiming.match(/processing;dur=([\d.]+)/);
const processingMs = match ? Number(match[1]) : undefined;
const networkMs = processingMs == null ? undefined : Math.max(0, timeToHeadersMs - processingMs);
console.log({serverTiming, timeToHeadersMs, processingMs, networkMs});
```

## Getting the numbers down

- **Deploy close to the region.** Typical short requests complete in roughly
  50–150 ms within the region. Network distance adds to end-to-end latency. See
  [Models § Regions](/models#regions).
- **Reuse connections.** Use an HTTP client with connection pooling and
  keep-alive enabled. OpenAI SDK clients do this by default. Celeris keeps an
  idle connection open for 300 seconds; a client that closes sooner pays a
  fresh TCP and TLS handshake on the next call. Node's built-in `fetch`
  (undici) closes idle sockets after 4 seconds unless the server advertises a
  longer `Keep-Alive` timeout or you set `keepAliveTimeout`.
- **Keep completions short.** Completion length is the main thing you control
  that moves inference time. A smaller `max_tokens` does not speed up a short
  response. See [output budgets](/making-requests#output-budgets).
- **Watch your `429` rate.** Throttling and the retries it forces add latency
  that your own timings will show. See
  [Rate limits](/rate-limits#backing-off-well).

## Test your own workload

Time representative prompts through the same client and retry policy you will
use in production. Start each timer before the first attempt and stop after the
final result, so retries and failures show in your numbers.
