LLM Request Router Load Balancing

Use this guide to deploy and validate Stargate load-balancer configuration in self-managed NVCF. For the complete lb-config.json schema, algorithm behavior, defaults, and tuning fields, see the Stargate load balancer configuration.

This guide covers NVCF deployment ownership and trusted request metadata. It does not redefine the Stargate schema.

Configure the self-managed stack

Set the request-router configuration in the Helmfile environment:

1
addons:
2
  llm:
3
    enabled: true
4
    requestRouter:
5
      loadBalancer:
6
        config: |
7
          {
8
            "default": "power-of-two",
9
            "request_algorithms": {
10
              "round-robin": "round-robin"
11
            }
12
          }

The self-managed stack passes addons.llm.requestRouter.loadBalancer to the request-router chart as llmRequestRouter.loadBalancer.

The chart supports two configuration sources:

Inline config takes precedence when both values are set. When neither value is set, Stargate uses its built-in power-of-two default and accepts a routing-method override when it is in the allowlist of built-in algorithms.

Stargate reads and validates the file only during process startup. The StatefulSet does not include a load-balancer ConfigMap checksum in its pod template. After a ConfigMap-only update, restart the StatefulSet so every replica loads the same configuration.

Distinguish router algorithms from nvcf-cli routing methods

Algorithm availability is enforced at separate layers:

For example, when a configuration is set and power-of-two is the effective algorithm, wait_and_widen requires a wait-and-widen entry in request_algorithms. The legacy groq_multiregion value remains accepted and resolves to the same algorithm.

Use wait-and-widen and pulsar-wait-and-widen in new function metadata, lb-config.json files, request-algorithm maps, and deployment manifests. Existing groq-multiregion and pulsar-multiregion values continue to work through the Stargate and control-plane compatibility aliases.

Keep router headers trusted

The LLM API Gateway resolves router-facing headers from authenticated function metadata and the parsed request:

The chat-completions provider creates a new outbound request from gateway context, so it drops all three request deadline headers. Responses and embeddings use native proxy paths that clone inbound headers and overwrite the routing headers for which the gateway resolved a value. These paths always remove caller-supplied x-priority, but do not unconditionally remove every other router-facing header.

For a separately managed Gateway API route, add this filter to the rule that forwards to llm-api-gateway:

1
filters:
2
  - type: RequestHeaderModifier
3
    requestHeaderModifier:
4
      remove:
5
        - x-request-id
6
        - x-routing-key
7
        - x-model
8
        - x-routing-method
9
        - x-input-tokens
10
        - x-cache-affinity-key
11
        - x-priority
12
        - x-request-slo-ms
13
        - x-max-wait-ms
14
        - x-stargate-max-wait-ms

The stock gateway-routes chart does not expose a value for this filter. Use an equivalent policy at an external edge or maintain a route override. Preserve x-multi-turn-session-id; clients can use it for session affinity. Chat Completions and Responses request bodies can also supply prompt_cache_key. See the Gateway API header modifier guide for filter semantics.

The gateway derives x-cache-affinity-key for chat-completions and Responses requests when affinity applies. A request body can contain the raw prompt_cache_key, but only its SHA-256-derived value appears in the internal header. The router forwards the request body to the model backend. It does not derive affinity for embeddings. Do not set require_cache_affinity_key on a model that serves /v1/embeddings unless another trusted gateway supplies the key.

Stargate returns HTTP 400 for a blank, unknown, or configured-but-unavailable x-routing-method. It also returns HTTP 400 when a required router header is missing or a numeric header is invalid.

Apply and roll out

Render the chart before applying it:

$
helm template llm-request-router \
>
  deploy/helm/llm-request-router/llm-request-router \
>
  --values <request-router-values.yaml>

Confirm that the rendered StatefulSet has the expected --lb-config-path argument and that inline JSON creates one ConfigMap with the lb-config.json key.

If the LLM route accepts untrusted traffic, inspect the rendered or live HTTPRoute and confirm that the trusted-header filter is present:

$
kubectl get httproute -n <gateway-namespace> llm-invocation -o yaml

Apply the self-managed environment from the unpacked stack directory:

$
cd path/to/nvcf-self-managed-stack
$
make apply HELMFILE_ENV=<environment-name>

For an inline configuration, inspect the live file and start argument:

$
kubectl get configmap -n nvcf llm-request-router-lb \
>
  -o jsonpath='{.data.lb-config\.json}'
$
kubectl get statefulset -n nvcf llm-request-router \
>
  -o jsonpath='{.spec.template.spec.containers[0].args}'

Restart after a ConfigMap-only change, then wait for all replicas:

$
kubectl rollout restart statefulset/llm-request-router -n nvcf
$
kubectl rollout status statefulset/llm-request-router -n nvcf
$
kubectl get pods -n nvcf -l app.kubernetes.io/name=llm-request-router

Confirm every listed pod was recreated after the ConfigMap update. For each pod, check for the load balancer config loaded startup log and compare its reported default and model-override count:

$
kubectl logs -n nvcf <request-router-pod> \
>
  | grep 'load balancer config loaded'

Do not continue if a pod predates the update, lacks a successful load log, or reports a different configuration summary. Stable affinity requires the same mounted configuration, seed, and candidate view across replicas.

Validate routing

Use a deployed function whose routingMethod is the effective configured algorithm or is present in request_algorithms.

1. Invoke without changing the function routing method and confirm success.
2. Update the function to a method present in request_algorithms and confirm success.
3. Try a method accepted by nvcf-cli that is neither the configured algorithm nor present in request_algorithms; confirm that Stargate returns HTTP 400.
4. For an affinity-aware method, repeat a supported multi-turn request with the same prompt_cache_key or the returned x-multi-turn-session-id.
5. Exercise a failed or saturated backend and confirm selection and retry counters change.

Observe the request router

The chart exposes llm-request-router:9090/metrics when request-router metrics are enabled. The current chart passes --metrics-port and uses Stargate’s default stargate_ metric prefix.

See LLM Request Router Metrics for metric names, labels, and scrape configuration.

Troubleshoot

Use these logs together:

$
kubectl logs -n nvcf statefulset/llm-request-router \
>
  --all-pods=true --tail=100
$
kubectl logs -n nvcf deploy/llm-api-gateway --tail=100
$
kubectl logs -n nvcf-backend <function-pod> -c llm-worker --tail=100