Private RAG on Air-Gapped Kubernetes: Orchestrating Open-Source Models on Local GPU Nodes

Regulated industries keep asking the same question: can we get retrieval-augmented generation over our internal documents without a single byte leaving the building? Defense contractors, hospitals, banks, and industrial operators often cannot call a hosted LLM API. Their networks are segmented, their egress is closed, and their auditors want proof, not promises.

The open-weight ecosystem is now good enough to make this practical. Llama, Mistral, Qwen, and Gemma-family models run well on a handful of datacenter GPUs, and the serving stack (vLLM, Text Embeddings Inference, pgvector) is mature and Kubernetes-native.

The hard part is not the model. It is everything around it: getting artifacts into a cluster that cannot pull from the internet, scheduling scarce GPUs fairly, keeping retrieval fast and permission-aware, and proving that nothing phones home. This article walks through a reference architecture for a fully air-gapped RAG platform on Kubernetes, with configuration you can adapt.

1. Threat Model and Design Constraints#

Before choosing components, write down what "air-gapped" means for your environment. We use these working assumptions:

• No egress. Cluster nodes have no route to the public internet. Not even DNS forwarding to a public resolver.
• One-way import. Artifacts (container images, model weights, Python wheels, OS packages) arrive through a controlled transfer process: a connected "staging" zone, scanning, signing, then a physical or data-diode transfer.
• No telemetry. Every component must have its usage reporting disabled and verified.
• Document-level authorization. Retrieval must respect the same access controls as the source system. A user must never receive a chunk they could not open directly.

These constraints shape every decision below. In particular, "just pull the model from Hugging Face at startup" is the most common way an air-gapped design silently becomes a connected one.

2. Reference Architecture#

The platform has five planes:

3. Supply Chain: Getting Bits In Without Getting Bits Out#

Mirroring images#

On the connected side, pin everything by digest and export it. skopeo can copy to a directory or OCI archive that you carry across the boundary:

skopeo copy --all \
  docker:400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic">//docker.io/vllm/vllm-openai@sha256:<digest> \
  oci-archive:/transfer/vllm-openai.tar

cosign sign --key cosign.key registry.staging.example/ai/vllm-openai@sha256:<digest>

Inside the cluster, load the archives into Harbor (or any OCI registry) and enforce signature verification at admission with a Kyverno verifyImages policy (or Sigstore policy-controller) scoped to the rag namespace and your internal registry.

Model weights as versioned artifacts#

Treat weights like any other release artifact. On the staging side, download with huggingface-cli, verify checksums, scan for pickled code (prefer safetensors only), and push the directory to an internal store:

huggingface-cli download Qwen/Qwen2.5-14B-Instruct \
  --local-dir ./qwen2.5-14b-instruct \
  --include 400 font-semibold">class="text-emerald-300">"*.safetensors" 400 font-semibold">class="text-emerald-300">"*.json" 400 font-semibold">class="text-emerald-300">"tokenizer*"

sha256sum ./qwen2.5-14b-instruct/* > SHA256SUMS
mc cp --recursive ./qwen2.5-14b-instruct internal/models/qwen2.5-14b-instruct/v1/

Verify each model's license before import; some open weights carry usage restrictions.

Distributing weights to pods#

Large weights (tens of GB) should not be downloaded per pod start. Put them on a ReadWriteMany volume (CephFS, NFS, or a CSI driver with node-local caching) or pre-populate a node-local cache with an init container. Mount read-only and point inference engines at the local path with offline flags set:

env:
  - name: HF_HUB_OFFLINE
    value: 400 font-semibold">class="text-emerald-300">"1"
  - name: TRANSFORMERS_OFFLINE
    value: 400 font-semibold">class="text-emerald-300">"1"
  - name: HF_HUB_DISABLE_TELEMETRY
    value: 400 font-semibold">class="text-emerald-300">"1"
  - name: VLLM_NO_USAGE_STATS
    value: 400 font-semibold">class="text-emerald-300">"1"
  - name: DO_NOT_TRACK
    value: 400 font-semibold">class="text-emerald-300">"1"

4. GPU Nodes on Kubernetes#

The GPU Operator#

The NVIDIA GPU Operator installs drivers, the container toolkit, the device plugin, and DCGM exporter as managed components. In an air-gapped cluster you must point it at your internal registry and, if you use the driver container, provide a pre-built driver image matching your kernel:

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.repository=harbor.internal.example/nvidia \
  --set toolkit.repository=harbor.internal.example/nvidia \
  --set devicePlugin.repository=harbor.internal.example/nvidia \
  --set validator.repository=harbor.internal.example/nvidia

Many air-gapped shops instead bake the driver into the node image and set driver.enabled=false.

Isolating GPU pools#

Taint GPU nodes so that only workloads that ask for GPUs land there. Otherwise a stray log shipper will happily occupy a very expensive machine:

kubectl taint nodes gpu-a100-01 nvidia.com/gpu=present:NoSchedule
kubectl label nodes gpu-a100-01 workload=llm-serving gpu-400 font-semibold">class=a100-80gb

Partitioning: MIG versus time-slicing#

5. Serving the Generation Model with vLLM#

vLLM provides an OpenAI-compatible server with continuous batching and PagedAttention, which is what makes a small number of GPUs serve many concurrent users. A representative Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-generation
  namespace: rag
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm-generation }
  template:
    metadata:
      labels: { app: vllm-generation }
    spec:
      nodeSelector:
        gpu-400 font-semibold">class: a100-80gb
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      containers:
        - name: vllm
          image: harbor.internal.example/ai/vllm-openai@sha256:<digest>
          args:
            - --model=/models/qwen2.5-14b-instruct/v1
            - --served-model-name=rag-llm
            - --max-model-len=16384
            - --gpu-memory-utilization=0.90
            - --enable-prefix-caching
            - --max-num-seqs=64
          env:
            - { name: HF_HUB_OFFLINE, value: 400 font-semibold">class="text-emerald-300">"1" }
            - { name: VLLM_NO_USAGE_STATS, value: 400 font-semibold">class="text-emerald-300">"1" }
          ports:
            - containerPort: 8000
          resources:
            limits:
              nvidia.com/gpu: 1
              memory: 48Gi
            requests:
              cpu: 400 font-semibold">class="text-emerald-300">"8"
              memory: 32Gi
          volumeMounts:
            - { name: models, mountPath: /models, readOnly: 400">true }
            - { name: shm, mountPath: /dev/shm }
          readinessProbe:
            httpGet: { path: /health, port: 8000 }
            initialDelaySeconds: 90
            periodSeconds: 10
      volumes:
        - name: models
          persistentVolumeClaim: { claimName: model-store, readOnly: 400">true }
        - name: shm
          emptyDir: { medium: Memory, sizeLimit: 8Gi }

Notes that matter in practice:

• --max-model-len drives KV-cache cost. Do not set the context window to the model's maximum if your prompts are typically 4 to 8k tokens. The memory you free becomes concurrency.
• --enable-prefix-caching helps RAG because system prompts and shared instructions repeat across requests.
• Model load is slow. Use generous probe delays or a startupProbe.
• Multi-GPU models use --tensor-parallel-size N with nvidia.com/gpu: N on a single node.

Embeddings and reranking#

For embeddings, Hugging Face's Text Embeddings Inference (TEI) is compact and fast. A model such as BAAI/bge-m3 or bge-large-en-v1.5 fits comfortably in a MIG slice:

Request a slice such as nvidia.com/mig-2g.20gb: 1, pass --model-id=/models/bge-m3 and --max-batch-tokens=32768, and mount the same read-only model volume.

Run a second TEI instance with a cross-encoder reranker (for example a bge-reranker model). Reranking the top 30 to 50 candidates down to 5 to 8 is often the single largest quality improvement available after chunking strategy.

6. The Data Plane: pgvector as the Default#

For most enterprise corpora (up to tens of millions of chunks), PostgreSQL with pgvector is a better default than a separate vector database. You already know how to operate, back up, audit, and secure Postgres, and you can join vectors with ACL tables in one query.

400 font-semibold">CREATE EXTENSION IF NOT EXISTS vector;

400 font-semibold">CREATE 400 font-semibold">TABLE documents (
  doc_id       uuid PRIMARY KEY,
  source_uri   text NOT NULL,
  sha256       text NOT NULL,
  classification text NOT NULL,
  ingested_at  timestamptz NOT NULL DEFAULT now()
);

400 font-semibold">CREATE 400 font-semibold">TABLE doc_acl (
  doc_id uuid REFERENCES documents(doc_id) ON 400 font-semibold">DELETE CASCADE,
  principal text NOT NULL,          -- group or user claim 400 font-semibold">from OIDC
  PRIMARY KEY (doc_id, principal)
);

400 font-semibold">CREATE 400 font-semibold">TABLE chunks (
  chunk_id   bigserial PRIMARY KEY,
  doc_id     uuid NOT NULL REFERENCES documents(doc_id) ON 400 font-semibold">DELETE CASCADE,
  ordinal    int  NOT NULL,
  content    text NOT NULL,
  tsv        tsvector GENERATED ALWAYS AS (to_tsvector(400 font-semibold">class="text-emerald-300">'english', content)) STORED,
  embedding  vector(1024) NOT NULL
);

400 font-semibold">CREATE 400 font-semibold">INDEX chunks_embedding_hnsw
  ON chunks USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 128);

400 font-semibold">CREATE 400 font-semibold">INDEX chunks_tsv_gin ON chunks USING gin (tsv);
400 font-semibold">CREATE 400 font-semibold">INDEX chunks_doc_idx ON chunks (doc_id);

Dimension 1024 matches bge-m3; adjust for your embedding model.

Permission-aware hybrid retrieval#

The retrieval query enforces authorization in SQL, so the application cannot forget to filter. This version combines vector similarity and full-text rank using reciprocal rank fusion:

WITH allowed AS (
  400 font-semibold">SELECT c.chunk_id, c.doc_id, c.content, c.embedding, c.tsv
  400 font-semibold">FROM chunks c
  400 font-semibold">JOIN doc_acl a ON a.doc_id = c.doc_id
  400 font-semibold">WHERE a.principal = ANY ($3::text[])          -- caller400 font-semibold">class="text-emerald-300">'s groups
),
vec AS (
  400 font-semibold">SELECT chunk_id,
         row_number() OVER (400 font-semibold">ORDER BY embedding <=> $1::vector) AS r
  400 font-semibold">FROM allowed
  400 font-semibold">ORDER BY embedding <=> $1::vector
  LIMIT 40
),
kw AS (
  400 font-semibold">SELECT chunk_id,
         row_number() OVER (400 font-semibold">ORDER BY ts_rank_cd(tsv, q) DESC) AS r
  400 font-semibold">FROM allowed, websearch_to_tsquery('english', $2) q
  400 font-semibold">WHERE tsv @@ q
  400 font-semibold">ORDER BY ts_rank_cd(tsv, q) DESC
  LIMIT 40
)
400 font-semibold">SELECT c.chunk_id, c.doc_id, c.content,
       COALESCE(1.0 / (60 + vec.r), 0) + COALESCE(1.0 / (60 + kw.r), 0) AS score
400 font-semibold">FROM chunks c
LEFT 400 font-semibold">JOIN vec ON vec.chunk_id = c.chunk_id
LEFT 400 font-semibold">JOIN kw  ON kw.chunk_id  = c.chunk_id
400 font-semibold">WHERE vec.chunk_id IS NOT NULL OR kw.chunk_id IS NOT NULL
400 font-semibold">ORDER BY score DESC
LIMIT 30;

One caveat: with HNSW, applying a restrictive filter after the index scan can return fewer rows than requested. In pgvector 0.8+ you can set hnsw.iterative_scan = relaxed_order and raise hnsw.ef_search to compensate. For corpora where most users see a small fraction of documents, consider partitioning by classification or tenant so the filter prunes before the index is touched. Always test recall with real ACL distributions, not the whole-corpus case.

7. Ingestion Pipeline#

Ingestion is a batch workload and should not compete with serving for GPUs. Model it as Kubernetes Jobs (or Argo Workflows) with its own MIG slice and a lower priority class.

Run the ingestion workers as a Kubernetes Job with a rag-batch PriorityClass, the embedding service URL, chunk size (for example 512 tokens with 64 overlap), and credentials injected from Secrets.

Design points worth getting right:

1. Idempotency by content hash. Store sha256 per document; skip unchanged files and re-chunk only modified ones. Deletes must cascade, which the schema above handles.
2. Structure-aware chunking. Split on headings, tables, and list boundaries before falling back to token windows. Carry the heading path into each chunk as context.
3. ACL sync is a first-class job. Pull permissions from the source system on a schedule, and treat a failed ACL sync as a reason to fail closed (deny) rather than serve stale grants.
4. Embedding version column. When you change embedding models, you must re-embed everything. Add embedding_model text and run old and new side by side during migration.

8. Network Policy: Proving the Air Gap#

The physical air gap is your outer boundary; Kubernetes policy is the inner one. Start from default deny, then open only what is needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: 400 font-semibold">default-deny-all
  namespace: rag
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: gateway-egress
  namespace: rag
spec:
  podSelector:
    matchLabels: { app: rag-gateway }
  policyTypes: [Egress]
  egress:
    - to:
        - podSelector: { matchLabels: { app: vllm-generation } }
      ports: [{ protocol: TCP, port: 8000 }]
    - to:
        - podSelector: { matchLabels: { app: tei-embed } }
        - podSelector: { matchLabels: { app: tei-rerank } }
      ports: [{ protocol: TCP, port: 8080 }]
    - to:
        - podSelector: { matchLabels: { app: postgres } }
      ports: [{ protocol: TCP, port: 5432 }]
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
          podSelector:
            matchLabels: { k8s-app: kube-dns }
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }

Inference pods get no egress policy entries at all except DNS. That means that even if a dependency tries to phone home, the packets are dropped, and a Cilium or Calico flow log gives you evidence for the auditor. Pair this with a test: a CI job that runs each image in a namespace with default-deny and fails if startup requires any external connection.

9. Scheduling GPUs Fairly with Kueue#

When several teams or workloads share a small GPU fleet, quota and preemption matter more than raw capacity. Kueue (a Kubernetes SIG project) provides queueing and fair sharing for batch workloads, and plays well with priority classes:

Interactive serving (vLLM, TEI) stays as regular Deployments with a high PriorityClass, while ingestion, evaluation runs, and fine-tuning go through Kueue queues. The serving workloads can preempt batch jobs when load spikes, not the other way around.

For autoscaling serving replicas, the Horizontal Pod Autoscaler on CPU is useless for GPU inference. Scale on queue depth instead, using vLLM's Prometheus metrics (such as vllm:num_requests_waiting) through KEDA or a Prometheus adapter. Remember that each new replica needs a free GPU and a model-load delay, so autoscaling is a capacity-planning tool, not an instant response.

10. A Worked Capacity Example#

The following is an illustrative sizing exercise with stated assumptions, not a benchmark. Always measure on your own hardware and corpus.

Assumptions:

• About 30 concurrent sessions at peak.
• Average prompt 3,500 tokens (system prompt plus 6 retrieved chunks plus question), average output 350 tokens.
• A 14B-parameter model in 16-bit precision on one 80 GB GPU.

Memory reasoning:

• Weights at 16-bit are roughly 2 bytes per parameter, so about 28 GB.
• With --gpu-memory-utilization=0.90 on an 80 GB card, around 72 GB is usable, leaving roughly 40 GB or more for KV cache after activations and overhead.
• KV-cache bytes per token depend on layers, KV heads (grouped-query attention shrinks this), and head dimension; compute it from your model's config. For a GQA model of this class it is on the order of a few hundred KB per token.
• At 3,850 tokens per active sequence and a few hundred KB per token, 30 concurrent sequences need on the order of tens of GB. That fits, with headroom, which is why one GPU can plausibly serve this workload at interactive speeds.

Decision: deploy two replicas on two GPUs, not for throughput but for availability during node maintenance and model upgrades. If requirements grow, quantizing (FP8 on Hopper-class cards, or AWQ/GPTQ 4-bit) trades a small quality delta for substantially more concurrency, and should be validated with your own evaluation set before adoption.

Capacity is dominated by context length times concurrency, so fewer, better chunks directly reduce GPU spend.

11. Observability and Evaluation Without SaaS#

Everything must be self-hosted: Prometheus, Grafana, Loki, and optionally an OpenTelemetry Collector with Tempo or Jaeger. Track these signals:

• Serving: time to first token, inter-token latency, queue depth, KV-cache usage, preemptions (vLLM metrics); GPU utilization, memory, and ECC errors (DCGM exporter).
• Retrieval: query latency split by embed, search, rerank; empty-result rate; ACL-filtered result counts.
• Quality: a fixed, versioned evaluation set of question and expected-source pairs, run nightly as a Job. Track recall@k of the retriever separately from answer faithfulness so you can tell which half regressed.

Log chunk IDs and document hashes with each answer, not just prompts. When someone disputes an answer, you want to reconstruct exactly which text the model saw.

Treat retrieved text as untrusted input to limit prompt injection, and avoid giving the model side-effecting tools unless required.

Conclusion#

A private RAG platform on air-gapped Kubernetes is an integration problem more than a research problem. The model and serving layers are commodity; the durable engineering lives in the supply chain that safely imports artifacts, GPU scheduling that protects interactive latency, permission-aware retrieval enforced in the database, and network policy that turns "we believe nothing leaves" into evidence.

Start small: one signed model, one vLLM replica, pgvector, default-deny networking, and an evaluation set from day one. Then add reranking, MIG partitioning, and queue-based scheduling as real usage tells you where the bottlenecks are.

If you are designing a sovereign or air-gapped AI platform and want a second set of eyes on the architecture, capacity model, or security posture, the knetwork engineering team offers architecture consultations tailored to regulated environments.