What we do · expertise
Your models on your side, your data never leaves.
Inference and fine-tuning on your infrastructure: GPU sizing, serving architecture, agents and the MCP protocol. We favour sourced answers — an answer that does not cite its document is unusable in a regulated business.
Method
Sizing is calculated before a model is chosen, and it decides the hardware budget. Weights have to fit in accelerator memory next to the attention cache, which grows with the number of concurrent sessions and the context length. As long as the weights fit on one GPU, a replica is one pod and one card. When they no longer do, they are sharded across several cards in one machine. When they no longer fit in one machine, it takes several machines to serve a single model.
That is where Kubernetes alone stops being enough. A Deployment does not know that four pods are one model: it schedules them separately, restarts one without the others, and a half-placed group serves nothing at all. LeaderWorkerSet treats the group as the unit — one leader, its workers, placed, restarted and scaled together. It is the piece that makes a multi-node replica genuinely operable, and that is why it is installed in the platform before it is needed.
The chain we run is this one: KServe for the service declaration, LeaderWorkerSet for distributed replicas, the NVIDIA GPU Operator for drivers, device plugin and DCGM metrics, vLLM as the inference engine, an Envoy gateway at the front with a tokenizer-aware scheduler. It runs on a client's infrastructure, described in GitOps, alongside Ceph storage, PostgreSQL operated by CloudNativePG and OpenBao.
The manifest opposite is taken from it, and one of its details carries the point: the checkpoint is already FP8-quantised when we stage it. That is not a quality footnote, it is a sizing decision. The on-disk and in-memory footprint falls by roughly half, startup no longer includes a quantisation pass, and the chosen scheme — eight-bit float weights channel by channel, per-token dynamic activations — is tighter than the one the engine would apply on its own. The weights arrive through a download job into an internal volume, then mounted read-only: at startup, the service fetches nothing from the internet.
What this produces is hard to measure, and we measure it anyway. On a document-exploitation engagement, the client team estimated 1 h (1) given back to each equipped employee. It is a self-reported estimate, at a single client, and it does not generalise. ▪
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
name: …
spec:
model:
# Local copy staged by the download Job (avoids the presets' 1Gi
# storage-initializer cap). Mounted at /mnt/models by the controller.
#
# Pre-quantized FP8 build rather than the BF16 repo quantized at load:
# 8-bit float weights (channel-wise, static) with per-token dynamic
# activations, which is a tighter scheme than the per-tensor default
# vLLM applies for --quantization fp8. And no quantization pass
# during startup.
uri: pvc://…
name: …
replicas: 1
# Managed gateway + route + EPP scheduler (the llm-d data path via the
# envoy GatewayClass installed by platform-apps/envoy-gateway).
router:
gateway: {}
route: {}
template:
containers:
- name: main
image: vllm/vllm-openai:v0.24.0
resources:
limits:
nvidia.com/gpu: "1"
# Injected via the llmd preset's VLLM_ADDITIONAL_ARGS launch-script hook.
env:
- name: VLLM_ADDITIONAL_ARGS
value: "--kv-cache-dtype fp8 --enable-auto-tool-choice …"Scope
Everything starts from the hardware you have and from what your documents actually are.
Engagements
Described by the setup that was put in place and by what it produced.
Data & AI · fixed-term engagement
An ingestion pipeline that takes documents as they come, a base of open tables queried in SQL, semantic indexing — and one rule on the answering side: an answer without a source document is not an answer.
Sovereign cloud · ongoing
An internal cloud for data and inference, deployed under public control: the models run on the authority's own infrastructure, and the logs stay on its side.
Get in touch
Before talking about a model, we look at three things: the hardware you have, what your documents actually are, and what your logging policy permits. Sizing is calculated; it is not guessed, and it is not copied from another client.