ArgoCD CLI broken with Cilium Gateway? Use TLS passthrough

ArgoCD UI worked fine at https://lab.sofianedjerbi.com/argo/. The CLI failed:

argocd login lab.sofianedjerbi.com --grpc-web --grpc-web-root-path /argo
# Error: POST https://lab.sofianedjerbi.com/argo/session.SessionService/Create failed with status code 404

Direct requests via port-forward returned HTTP 200. Same requests through Cilium Gateway returned 404. The gateway was doing something to the traffic

The hidden filter

Cilium Gateway includes envoy.filters.http.grpc_web enabled by default. This filter:

1. Detects gRPC-web requests via Content-Type: application/grpc-web+proto
2. Converts them to native gRPC before forwarding
3. Expects the backend to speak native gRPC over HTTP/2

ArgoCD with server.insecure: true runs in HTTP mode. It only accepts gRPC-web over HTTP/1.1, not native gRPC. When Cilium converts the protocol, ArgoCD doesn’t understand it and returns 404

Found this after digging: cilium/cilium#31933

What didn’t work

Approach	Result	Why
URLRewrite /argo/* to /*	404 for gRPC	Filter still converts protocol
Dedicated subdomain	Still 404	Same filter, path wasn’t the issue
Patch CiliumEnvoyConfig	Rejected	I wanted everything as code
HTTPS with HTTPRoute	Redirect loops	Cilium doesn’t support BackendTLSPolicy

TLS passthrough fixes it

Make the traffic opaque to Cilium. It can’t inspect or modify encrypted packets

1. ArgoCD runs with TLS enabled (server.insecure: false) on port 443
2. Gateway listener uses protocol: TLS with mode: Passthrough
3. TLSRoute forwards encrypted traffic directly to ArgoCD
4. ArgoCD terminates TLS and handles native gRPC over HTTP/2
5. cert-manager provides the Let’s Encrypt certificate

The configuration

Gateway listener:

listeners:
  - name: argo-tls
    protocol: TLS
    port: 443
    hostname: argo.lab.sofianedjerbi.com
    tls:
      mode: Passthrough
    allowedRoutes:
      kinds:
        - kind: TLSRoute

TLSRoute:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TLSRoute
metadata:
  name: argocd
spec:
  parentRefs:
    - name: cilium-gateway
      sectionName: argo-tls
  hostnames:
    - argo.lab.sofianedjerbi.com
  rules:
    - backendRefs:
        - name: argocd-server
          port: 443

ArgoCD Helm values:

configs:
  params:
    server.insecure: false

server:
  certificate:
    enabled: true
    domain: argo.lab.sofianedjerbi.com
    issuer:
      kind: ClusterIssuer
      name: letsencrypt-prod

Why it works

HTTPRoute (TLS termination)	TLSRoute (passthrough)
Gateway terminates TLS	Gateway forwards encrypted packets
Envoy inspects HTTP headers	Envoy can’t inspect content
grpc-web filter activates	No L7 processing possible
Converts gRPC-web to native gRPC	Traffic passes unchanged
ArgoCD (HTTP mode) confused	ArgoCD terminates TLS, handles gRPC

The key: ArgoCD in TLS mode supports native gRPC over HTTP/2. HTTP/2 is naturally available over TLS connections. The same ArgoCD that fails with gRPC-web in HTTP mode works perfectly with native gRPC in TLS mode

The pattern

When a Layer 7 proxy interferes with your protocol, drop to Layer 4

Cilium’s grpc-web filter can’t be disabled via Gateway API. Rather than patching Envoy configs, TLS passthrough makes the traffic invisible. The encrypted stream passes through untouched

This applies to WebSockets, gRPC, custom binary protocols. If your L7 proxy breaks a protocol, consider L4 passthrough. You trade header-based routing and gateway-level logging for protocol integrity

Sometimes the fix is to stop the proxy from helping