You spin up a Kubernetes cluster on Hetzner. One node pulls images fine. Another node gets 403 Forbidden. Same network, same config, different result

unexpected status from HEAD request to
https://registry.k8s.io/v2/pause/manifests/3.7: 403 Forbidden

Google Cloud Armor is blocking your Hetzner IP because someone else who had that IP range did something Google didn’t like. Maybe months ago. Maybe years ago

registry.k8s.io sits behind Google’s load balancer with Cloud Armor protection. Some Hetzner IP ranges are on the blocklist. GitHub issue #138 documents this since 2023. Issue #1664 shows it still happens in 2025

The randomness makes it worse. You build a 3-node cluster. Node 1 works. Node 2 works. Node 3 gets 403s. Same network, same region. Doesn’t matter

Why this happens

Hetzner recycles IPs. You inherit the reputation of whoever had your IP before. They ran a botnet? You’re blocked. They scraped Google? You’re blocked

The Kubernetes registry team knows. From issue #138: “we’re using cloud armor, configured here”

They haven’t fixed it. Google’s position: those IPs showed abusive behavior. Doesn’t matter if it was you

What works

Use Quay or Docker Hub

Most Kubernetes images are mirrored on other registries:

# /etc/rancher/k3s/registries.yaml
mirrors:
  registry.k8s.io:
    endpoint:
      - "https://quay.io/kubernetes"

Or update your manifests directly. Instead of registry.k8s.io/pause:3.7, use quay.io/kubernetes/pause:3.7

Test before deploying

Check if your IP is blocked:

curl -I https://registry.k8s.io/v2/

403 means blocked. Get a different IP before you build your cluster

For existing clusters, test all nodes:

for node in $(kubectl get nodes -o name); do
  kubectl debug $node -it --image=busybox -- \
    wget -O- https://registry.k8s.io/v2/ 2>&1 | grep -q 403 && \
    echo "$node: BLOCKED" || echo "$node: OK"
done

Contact Hetzner support

Open a ticket at https://console.hetzner.cloud/support. Tell them your IP is blocked by Google Cloud Armor

They might swap it. They might not. From issue #138, some users got new IPs, others didn’t

Last resort: run a mirror

If images don’t exist on other registries, mirror them yourself:

# /etc/rancher/k3s/registries.yaml
mirrors:
  registry.k8s.io:
    endpoint:
      - "https://your-mirror.example.com"

Try Quay first. Self-hosting is overkill for most cases

Why this won’t get fixed

Google should allowlist major cloud providers. Hetzner hosts thousands of production clusters

Blocking entire IP ranges because of historical abuse is lazy security. The error message should say “blocked by Cloud Armor” instead of “403 Forbidden”

Neither will happen. Container registries are critical infrastructure. Kubernetes can’t function without them. But the project uses Google Cloud Armor anyway. Issues get closed. Users implement workarounds

This affects any provider that recycles IPs. DigitalOcean, Vultr, OVH, Linode. Shared hosting means shared reputation. Your IP’s history isn’t yours to control

Plan accordingly. Use Quay. Test your IPs. Don’t assume registry.k8s.io will work