Performance

Every figure on this page comes from an identified qualification campaign, with its hardware, protocol and limits. Nothing is extrapolated.

The Compact profile was functionally qualified on a CPU workstation without GPU: the full stack works, offline, with a real reranker. The measured latencies demonstrate the integration; they are not a service level.

Key results

Compact profile, full stack end to end (from click to cited answer), published synthetic corpus of 8 documents and 10 questions.

Compact profile on CPU — functionally qualified

Tested

x86_64 workstation, 4 cores / 8 threads, 14 GB RAM, no GPU. Revision RC2, campaign of 9 November 2026.

51sFull answer, Compact CPU profile27.8 – 71.3 s

10sClient first token, Compact CPU profile3.9 – 17.9 s

21.5tokens/sGeneration throughput, Compact CPU profile15.8 – 52.0 tokens/s

9.5sReranking per question, Compact CPU profile8.9 s (integration campaign) – max 18.2 s

7.8GBPeak memory, Compact CPU profilereranker up to 5.7 GB; Ollama ≈ 3 GB; steady state (RC2): TEI 3.5 GB, Ollama 3.0 GB

Internal verdict: GO with reservations. All 71 end-to-end steps passed (cold install, real reranking, offline verified, resilience, backup and restore). Reservations concern latency (28 to 71 s per answer) and the quality of the 1.5-billion-parameter model, which is not claimed.

GPU profiles: not qualified

To be qualified

No GPU profile is qualified yet. The 8 GB Compact profile (same 1.5-billion-parameter model) and the 12 GB profile (7-billion-parameter model) are described in the repositories, but no GPU performance is published until a qualification has been run on a dedicated host with the official images.

Useful finding when preparing a GPU machine: the shipped Ollama image (0.30.8) does not work with an NVIDIA 535 driver; it requires NVIDIA driver 550 or newer.

In every current profile, reranking runs on CPU: a GPU would speed up generation, not necessarily the whole answer chain.

Methodology

Full stack deployed as documented for Compact (9 services), models provisioned then run offline.
Corpus: the 8 synthetic files published in compact/fixtures/corpus (TXT, Markdown, a 14-page text PDF, DOCX) and 10 questions with expected sources.
Public script compact/scripts/e2e_qualification.py: upload, wait for indexing, questions in aggregated then streamed mode, citation checks.
Timing on the client side (through the UI) and in the gateway logs (tokens per second, first fragment).
Memory from docker stats; VRAM from nvidia-smi.

Reading the metrics without mixing them up

The following measurements are not comparable with each other.

Full answer: From click to last token: retrieval, reranking, prompt building and generation.
Client first token: Time before the first displayed fragment: it includes retrieval and reranking.
Gateway first fragment: Internal delay between prompt reception and the model’s first token (a few hundred ms).
Generation throughput: Tokens produced per second once generation starts; depends on hardware and prompt length.
Reranking: Duration of the cross-encoder on the 20 best passages; on CPU it is the dominant item.
Ingestion: From upload to "ready": extraction, chunking, embeddings, indexing.
Memory: Container RAM (docker stats) and VRAM (nvidia-smi); peak values.

Limits

Deliberately small, synthetic corpus: latencies grow with document size and the number of passages to rerank.
One user at a time: the gateway accepts two concurrent generations, then answers 503 with Retry-After.
1.5-billion-parameter model: the answers demonstrate the path, not documentary quality.
12 GB GPU profile (7 B model): described, never installed or measured.
Multi-node through the Pool: tested on simulated nodes only.

Public measurement register

All published measurements, with their context. Evidence links point to the scripts and corpus published in the repositories.

Compact profile (CPU)

Full answer, Compact CPU profile51 s

Full answer latency (from click to end of answer)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 10 · median 51.2 s; mean 50.2 s
Observed range: 27.8 – 71.3 s
Measured path: Full RAG stack (9 services), end to end through the web UI: question → hybrid retrieval → reranking → generation → answer with citations.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, synthetic, representative
Revision: RC2 (campaign of 2026-11-09)
Date: 9 November 2026
Limits: Synthetic corpus of 8 documents and 10 questions; 1.5-billion-parameter model; CPU shared between Ollama and the reranker. These figures demonstrate the integration, not a service level.
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/fixtures/corpus/manifest.json compact/docs/limites-et-capacites.md

Client first token, Compact CPU profile10 s

Time to first answer fragment (SSE stream), retrieval and reranking included

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 10 · median 10.1 s
Observed range: 3.9 – 17.9 s
Measured path: Full RAG stack (9 services), end to end through the web UI: question → hybrid retrieval → reranking → generation → answer with citations.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, synthetic, representative
Revision: RC2 (campaign of 2026-11-09)
Date: 9 November 2026
Limits: Includes hybrid retrieval, reranking (≈ 9.5 s) and prompt building; the gateway alone emits its first fragment within 274–377 ms once the prompt is received.
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/fixtures/corpus/manifest.json

Generation throughput, Compact CPU profile21.5 tokens/s

Generated tokens per second (gateway logs, 24 RAG generations)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 24 · mean
Observed range: 15.8 – 52.0 tokens/s
Measured path: Gateway log (generate.ok) within the full stack; long RAG prompts.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, synthetic
Revision: pre-release (integration campaign of 2026-11-09)
Date: 9 November 2026
Limits: Heterogeneous mean over long RAG prompts. Not to be confused with a direct Ollama measurement on a short prompt (34.6 tokens/s, one run).
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/fixtures/corpus/manifest.json

Reranking per question, Compact CPU profile9.5 s

Duration of one reranking call (cross-encoder on CPU)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 22 · mean derived from counters (210 s / 22 calls)
Observed range: 8.9 s (integration campaign) – max 18.2 s
Measured path: Backend → gateway /v1/rerank → TEI CPU, full stack.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, synthetic
Revision: RC2 (campaign of 2026-11-09)
Date: 9 November 2026
Limits: float32 reranker on 8 threads; the Compact profile sets a 20 s timeout. Reranking is the dominant latency item.
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/docs/limites-et-capacites.md

Ingestion of 8 documents, Compact CPU profile13 s

Time from first upload to last document ready (37 chunks, 38 KB of text)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 8 · —
Observed range: 10 s (integration campaign) – 12.9 s (RC2)
Measured path: Upload via the UI → Celery worker → embeddings via the gateway → index.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, synthetic
Revision: RC2 (campaign of 2026-11-09)
Date: 9 November 2026
Limits: Small text corpus; single worker (concurrency 2); no OCR.
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/fixtures/corpus/manifest.json

Peak memory, Compact CPU profile7.8 GB

Cumulated memory of the 9 containers at peak (docker stats)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 1 · peak; mean ≈ 7.1 GB
Observed range: reranker up to 5.7 GB; Ollama ≈ 3 GB; steady state (RC2): TEI 3.5 GB, Ollama 3.0 GB
Measured path: docker stats sampling during the E2E campaign.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware
Revision: pre-release and RC2 (2026-11-09)
Date: 9 November 2026
Limits: An 8 GB machine is not enough; a 6 GB limit is needed for the reranker (3 GB causes an OOM).
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/docs/limites-et-capacites.md

Functional qualification campaign, Compact CPU profile71 / 71 steps

End-to-end steps without failure (cold install, real reranking, offline verified, resilience, backup/restore)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 71 · 0 failure (58 PASS, 11 OBSERVED, 2 INFO)
Observed range: duration 795 s
Measured path: Full RAG stack (9 services), end to end through the web UI: question → hybrid retrieval → reranking → generation → answer with citations.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, reproduced
Revision: RC2 (campaign of 2026-11-09)
Date: 9 November 2026
Limits: Functional qualification, executed three times on the full stack (pre-RC, RC1, RC2). Internal verdict: GO with reservations (latencies, 1.5 B model quality not claimed).
Reproducibility: Compact installation as documented, then compact/scripts/e2e_qualification.py on the fixtures corpus.
Evidence: compact/scripts/e2e_qualification.py compact/scripts/resilience_check.py compact/docs/limites-et-capacites.md

Disk footprint (RC4)779 / 941 / 274 / 236 MB

Size of locally built application images (non-root)

Hardware: Local run of the test suites
Sample: 1 · backend / frontend / gateway / pool
Observed range: models ≈ 3.9 GB; third-party images (pgvector, Redis, TEI, Ollama) ≈ 5 GB more
Measured path: docker images after build.
Context: No corpus (direct probe) · n/a
Nature: local
Revision: RC4 (2026-11-10)
Date: 10 November 2026
Limits: The documentation’s "~3 GB of images" counts only application images and light bases; the total footprint with Ollama and TEI is about 7 GB excluding models.
Reproducibility: docker compose build then docker images.
Evidence: README.md compact/docs/limites-et-capacites.md

GPU (not qualified)

NVIDIA driver compatibility of the shipped Ollama image0 / 2 generations

Successful generations with the official Ollama 0.30.8 image on driver 535

Hardware: Shared GPU workstation — AMD Ryzen Threadripper 2990WX, 128 GB RAM, NVIDIA RTX 3060 Ti 8 GB, driver 535.309.01 / CUDA 12.2, non-dedicated host
Sample: 2 · —
Observed range: error “device kernel image is invalid”
Measured path: Direct Ollama in the container, no gateway, no RAG.
Context: No corpus (direct probe) · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, one-off
Revision: RC2 (2026-11-10)
Date: 10 November 2026
Limits: Useful as a prerequisite: the official Ollama image ≥ 0.30 requires an NVIDIA driver ≥ 550 (compressed CUDA kernels).
Reproducibility: Reproducible: run the shipped Ollama image on a host with driver 535.
Evidence: app/models_registry.yaml compact/compose.gpu-12gb.yml

12 GB GPU profile0 measurements

Available measurements

Hardware: No hardware
Sample: 0 · —
Observed range: never installed
Measured path: None.
Context: No corpus (direct probe) · qwen2.5:7b-instruct
Nature: not generalisable
Revision: RC2 to RC4
Date: 10 November 2026
Limits: Described in compose.gpu-12gb.yml with qwen2.5:7b-instruct; never installed, no digest recorded, no measurement, not qualified.
Reproducibility: Documented procedure (provision --with-12gb) but not executed.
Evidence: compact/compose.gpu-12gb.yml compact/docs/limites-et-capacites.md

Pool, tests and sizes

Direct / Pool parity (single node)32 / 33 operations

Identical operations through the direct gateway and through the Pool (identical embedding vectors and rerank orders)

Hardware: CPU workstation x86_64 — Intel Core i5-1145G7, 4 cores / 8 threads, 14 GB RAM, no GPU
Sample: 33 · 1 expected difference, 0 failure
Observed range: first stream event: direct 269 ms / Pool 216 ms; full RAG through the Pool: 71 steps, 0 failure
Measured path: Client → direct gateway, then client → Pool → same gateway.
Context: Published synthetic corpus (compact/fixtures/corpus): 8 documents, 10 questions · qwen2.5:1.5b-instruct · granite-embedding:278m · bge-reranker-v2-m3
Nature: local, hardware, reproduced
Revision: candidate Pool and RC2 (2026-11-09)
Date: 9 November 2026
Limits: Single node. Multi-node is implemented and tested on simulated nodes only: not hardware-qualified.
Reproducibility: ai-gateway-pool-open: docs/contract-qualification.md, docs/performance.md.
Evidence: docs/contract-qualification.md docs/performance.md

Automated test suites (RC4, publication)≈ 17,100 tests

Tests passed, 0 failure

Hardware: Local run of the test suites
Sample: 1 · backend 13,011 (FULL profile) · frontend 1,307 · gateway 2,036 · pool 795
Observed range: backend FAST 9,953 in 167 s; STANDARD 12,965 in 244 s; FULL 13,011 in 930 s
Measured path: Local run of the published suites; the GitHub CI of the three repositories runs the same entry points.
Context: No corpus (direct probe) · n/a
Nature: local, reproduced
Revision: RC4 and RC4.1 (2026-11-10 and 11)
Date: 10 November 2026
Limits: Counts vary slightly by profile (FAST without database, STANDARD, FULL) and revision.
Reproducibility: make test-backend-all, npm test (frontend), gateway and pool pytest suites.
Evidence: Makefile .github/workflows/ci.yml

Access to evidence

The qualification scripts, the corpus and the limits documents are published in the repositories. Detailed qualification dossiers (raw logs) are internal to the project and are not distributed.

e2e_qualification.py (external link) · Synthetic corpus (external link) · Limits and capabilities (external link)