Automating Active Directory Test User Resets with PowerShell

Summary

A routine node shutdown on a self-hosted Talos/Cilium Kubernetes cluster triggered a multi-layer failure cascade that ended with no new pods being created anywhere in the cluster. The visible symptoms shifted several times during the investigation, NotReady node, CNI timeouts, a crash-looping Cilium operator, stuck ArgoCD, and finally a global context deadline exceeded on every pod create, but they were links in a single chain.

The root causes were three independent latent problems, each harmless in isolation, that only became fatal when a node restart forced them to surface at once:

1. A Cilium security-identity explosion (~217k → 36k identities) caused by high-cardinality Kubernetes Job labels being treated as identity-relevant, which overflowed the per-endpoint BPF policy map.
2. A leftover CRD stored-version mismatch from a Cilium 1.19 → 1.20 upgrade that crash-looped the Cilium operator.
3. A dead admission webhook (Vault Agent Injector) scoped to every pod in the cluster with a 30-second timeout, which delayed every pod create past the controller-manager's internal deadline.

This article walks the diagnostic path in the order it actually unfolded, because the misleading symptom-hopping is itself the lesson.

Environment

• OS: Talos Linux (immutable, API-driven; no SSH, no shell on nodes)
• Virtualization: Proxmox VE
• CNI: Cilium (kube-proxy replacement, native routing)
• GitOps: ArgoCD with automated: { prune: true, selfHeal: true }
• Topology: 3 control-plane nodes, ~10 workers
• Cilium version: freshly synced from 1.19.6 → 1.20.0 (19 days prior)

Phase 1, The misleading start: a "broken" node

A worker node (talos93) showed NotReady. The node condition was explicit:

MemoryPressure   Unknown   NodeStatusUnknown   Kubelet stopped posting node status.

talosctl to the node failed with connection refused on port 50000 (the Talos API). ICMP worked (ttl=63, one hop, right host), but apid was dead.

The Proxmox VNC console showed what looked like a live Talos boot log, but the timestamps never advanced. Key insight: a powered-off VM keeps its last framebuffer. The "live" console was a frozen screenshot of the final shutdown sequence. Scrolling up revealed:

[talos] shutdown via ACPI received
[talos] shutdown sequence: 10 phase(s)

The node had been intentionally shut down via the Proxmox UI. Nothing was broken. qm list reporting running reflects the QEMU process, not the guest OS state; a halted guest can still show running.

Lesson: For liveness on a Talos node, check port 50000 or kubectl get node. Never trust the VNC framebuffer, and never trust qm list alone.

A clean qm stop / qm start brought the node back and it rejoined the cluster. This phase was a red herring, but restarting the node is what shook loose the three real problems.

Phase 2, CNI timeouts and a crash-looping operator

As the node rejoined, pods across the cluster began failing with:

failed to setup network for sandbox ...: plugin type="cilium-cni" failed (add):
unable to create endpoint: Cilium API client timeout exceeded

The cilium-cni binary calls the local Cilium agent to wire up each pod. A timeout means the agent didn't answer in time. Initially this looked like a simple post-reboot race (agents restart, pods land before the agent is ready), and for most nodes it self-healed.

But the Cilium operator was crash-looping. Its logs showed:

Start hook failed ... Failed to start when elected leader ...
Leader election lost, shutting down

The underlying cause was a CRD update rejection:

CustomResourceDefinition.apiextensions.k8s.io "ciliumnodeconfigs.cilium.io"
is invalid: status.storedVersions[0]: Invalid value: "v2alpha1":
missing from spec.versions

Root cause: CRD stored-version mismatch after upgrade

Cilium 1.20 removed the v2alpha1 API version of CiliumNodeConfig. The 1.20 operator ships a CRD manifest with only v2 in spec.versions, but the live CRD still listed v2alpha1 in status.storedVersions. Kubernetes enforces an invariant: a version cannot be removed from spec.versions while it is still listed in storedVersions. So every operator reconcile was rejected, the operator lost leader election, restarted, and repeated.

The state:

spec.versions:    v2 v2alpha1
storedVersions:   ["v2alpha1","v2"]
storage version:  v2 (storage=true), v2alpha1 (storage=false)
objects:          0 CiliumNodeConfig objects exist

With v2 as the storage version and zero objects persisted at v2alpha1, pruning the stale version was safe (no storage migration needed).

Fix

kubectl patch crd ciliumnodeconfigs.cilium.io --subresource=status --type=merge \
  -p '{"status":{"storedVersions":["v2"]}}'
kubectl -n kube-system rollout restart deploy cilium-operator

The operator recovered. But CNI timeouts persisted, which meant the operator was a symptom, not the disease.

Phase 3, The real Cilium problem: identity explosion

Chasing the persistent CNI timeouts to a specific agent revealed the actual datapath error:

Failed to add PolicyMap key ... error="update map cilium_policy_v3_00024:
update: no space left on device"
...
A policy map update had to delete changes before it added them in order to
prevent a map overflow, a transient drop may have occurred.

no space left on device here is not disk: it is the kernel's ENOSPC for a full BPF map. Each endpoint has a per-endpoint policy map (cilium_policy_v3_*) with a fixed maximum (bpf-policy-map-max, default 16384). The endpoint needed more policy entries than the map could hold, so every insert failed → policy sync failed → endpoint regeneration failed → cilium-cni timed out. That was the true source of the CNI errors.

Why the map overflowed

Each policy-map entry corresponds to an allowed peer identity × port × protocol. If a cluster has tens of thousands of identities, a single endpoint's policy can approach the 16384 ceiling. So the question became: how many identities exist?

# security identities (line count, over-counts labels)
cilium-dbg identity list | wc -l      → 217234

# authoritative object count
kubectl get ciliumidentities --no-headers | wc -l   → 36906

# type split (CIDR identities have id >= 16777217)
total: 36920, cidr: 0, clusterlocal: 36920

A healthy cluster has hundreds to low thousands. 36,906 cluster-local identities is an explosion, and cidr: 0 ruled out FQDN/CIDR policy fan-out. The cause had to be label churn.

Finding the churning label

Cilium assigns one identity per unique set of identity-relevant labels. Ranking label keys by how many identities carry them:

36908 k8s:io.kubernetes.pod.namespace
36908 k8s:io.cilium.k8s.policy.serviceaccount
...
35137 k8s:job-name
35137 k8s:batch.kubernetes.io/job-name
 7794 k8s:velero.io/repo-name
 1442 k8s:velero.io/pod-volume-backup

95% of all identities carried job-name / batch.kubernetes.io/job-name. The cluster runs many scheduled Jobs/CronJobs (Velero/kopia maintenance, GitLab, security scans). Every Job run creates a pod with a unique job-name label. Since those labels were identity-relevant, every Job run since install minted a brand-new, never-reused Cilium identity. Over months, tens of thousands accumulated.

Cilium's default identity-relevant label filter already excludes pod-template-hash, controller-revision-hash, and pod-template-generation, but not the Job labels.

Fix

Exclude the high-cardinality Job/Velero labels from identity computation. The labels Helm value appends to Cilium's defaults, and using exclusion patterns (!) means no other labels are dropped:

labels: "k8s:!job-name k8s:!batch.kubernetes.io/job-name
         k8s:!velero.io/repo-name k8s:!velero.io/pod-volume-backup
         k8s:!controller-uid k8s:!batch.kubernetes.io/controller-uid"

Regex dots in the Helm string must be escaped (\\. in YAML). Existing identities do not change; after an agent restart, new identities stop carrying Job labels and the operator garbage-collects the stale ones as their pods terminate. A temporary bpf.policyMapMax: 65536 bump absorbs the overflow while the ~35k stale identities drain.

Lesson: Any label with unbounded cardinality (Job names, UUIDs, timestamps, revision hashes) must be excluded from Cilium identity computation. Otherwise identity count grows without bound and eventually overflows the BPF policy map.

Phase 4, The show-stopper: a cluster-wide create deadlock

Meanwhile, ArgoCD would not come up, and eventually no pods were being created at all, not stuck scheduling, not Pending, simply never materialized. Deployments showed READY 0/1, UP-TO-DATE 0, AVAILABLE 0 across dozens of namespaces. Every controller in kube-controller-manager logged:

Timeout: request did not complete within requested timeout -
context deadline exceeded

Eliminating the wrong suspects

• Control plane healthy: all 3 apiservers, controller-managers, schedulers 1/1 Running. kubectl get --raw=/readyz returned in 81 ms, etcd and apiserver were fast.
• Controller-manager leader valid: the kube-controller-manager lease was renewed 2 seconds prior. Not a leadership wedge.
• Nodes healthy: all Ready.
• The controller-manager logs were full of pods.metrics.k8s.io errors, a red herring from a down metrics-server (broken HPA metrics), not the create failure.

The definitive test

A raw pod create with HTTP tracing pinned it down exactly:

POST .../namespaces/cert-manager/pods    (16:03:51)
Response status="504 Gateway Timeout"    (16:04:25)   ← 34 seconds

A bare pause pod, no labels, no HPA, no matching NetworkPolicy, took 34 seconds and returned 504. The stall was inside the apiserver's admission chain. And 34 seconds ≈ 30-second webhook timeout + overhead was the tell.

Root cause: a dead webhook that fails "open", slowly

A webhook was firing on the pod create. It had been overlooked because it used failurePolicy: Ignore, which sounds safe. The trap:

vault-agent-injector-cfg
  webhook: vault.hashicorp.com
  failurePolicy: Ignore
  timeoutSeconds:  30
  namespaceSelector: {}          ← ALL namespaces
  objectSelector:  everything except the injector's own pods

The Vault Agent Injector backend had zero endpoints (its pod could not be created, the same cascade). Here is the critical mechanism:

failurePolicy: Ignore does not mean "skip a dead webhook instantly." When the backend is unreachable, the apiserver waits the full timeoutSeconds (30s) before deciding the call failed and applying the Ignore policy.

So every pod create in every namespace stalled for 30 seconds. kubectl tolerates that, but kube-controller-manager creates pods with a shorter request context. The 30-second webhook wait blew that deadline first → FailedCreate: context deadline exceeded → the ReplicaSet controller backed off and retried → same 30-second wall → nothing was ever created.

This was a genuine single point of failure: one dead 1-replica Deployment, whose webhook was scoped to every pod in the cluster, froze all pod creation cluster-wide.

Fix (break-glass)

Remove the dead webhook from the admission path so creates stop waiting on it. Deleting (not just patching failurePolicy) is required, because a timeout blocks regardless of policy until the deadline elapses:

kubectl get validatingwebhookconfiguration cert-manager-webhook -o yaml \
  > /tmp/cert-manager-webhook.yaml           # backup
kubectl delete validatingwebhookconfiguration cert-manager-webhook

kubectl get mutatingwebhookconfiguration vault-agent-injector-cfg -o yaml \
  > /tmp/vault-injector-webhook.yaml          # backup
kubectl delete mutatingwebhookconfiguration vault-agent-injector-cfg

A verification probe confirmed the fix, the same pod create that took 34 seconds now returned instantly. The controller-manager immediately began draining the backlog, and dependent components (Vault, cert-manager, ArgoCD) started to recover.

The deleted webhooks are restored, either by ArgoCD selfHeal once its pods are back, or by re-applying the backups, but only after their backends are 1/1 with live endpoints, or the deadlock re-forms instantly.

The GitOps chicken-and-egg

A subtle trap ran through the whole incident: the durable fixes lived in Git, applied by ArgoCD, but ArgoCD itself was down, unable to get pod endpoints (Cilium) and unable to be created (webhook deadlock). The fix needs ArgoCD; ArgoCD needs the fix.

The escape is to apply the critical fixes manually to live objects, bypassing GitOps, then let ArgoCD reconcile to the Git state once it is healthy:

• Cilium identity fix → kubectl patch configmap cilium-config + agent restart.
• Policy-map bump → same ConfigMap.
• Webhook deadlock → delete the offending webhook configs directly.

Because ArgoCD is down, it will not fight the manual changes; when it recovers, the same changes committed to Git make the manual and desired states converge.

Recovery order

The correct sequence to unwind the cascade:

1. Break the create deadlock first, delete the dead all-namespaces webhook(s). Nothing else can recover while pod creation is frozen.
2. Apply the Cilium identity fix to the live ConfigMap; restart agents. Keep the policyMapMax bump as a crutch during identity drain.
3. Let controllers drain the backlog, Vault, cert-manager, metrics-server, ArgoCD come up as pods finally get created and networked.
4. Restore the deleted webhooks once their backends are 1/1 with live endpoints.
5. Commit the manual changes to Git so ArgoCD's desired state matches reality and nothing gets reverted on the next sync.
6. Verify identity count falls from ~37k toward a few hundred; then revert policyMapMax to default in a later commit.

Prevention

Admission webhooks are the highest-risk component

They sit synchronously in the create path of every matching object. A single dead webhook can freeze the cluster. Hardening rules:

• Scope narrowly. Never leave namespaceSelector: {} (all namespaces) on a webhook. Use opt-in labels so it fires only where actually needed.
• Exempt system namespaces explicitly. Do not assume defaults exclude kube-system, for the Vault injector, they do not.
• Cut timeouts hard. Drop timeoutSeconds from 30 to ~2. With Ignore, a dead backend then delays each create by 2s instead of 30s: survivable instead of fatal.
• Make webhook backends resilient. Run ≥2 replicas with a PodDisruptionBudget so a single node loss cannot zero out the endpoints. (For the Vault injector at >1 replica with auto-TLS, enable leader election to avoid CA-bundle races between replicas.)
• Alert on empty webhook endpoints. A webhook service with zero ready endpoints is a latent cluster-wide outage:

  kube_endpoint_address_available{endpoint=~".*webhook.*"} == 0   for > 2m

Cilium identity hygiene

• Exclude unbounded-cardinality labels (Job names, UUIDs, timestamps) from identity computation via the labels filter. Left unchecked, identity count grows without bound and overflows the per-endpoint BPF policy map.
• Alert on cilium_identity count and on policy-map utilization before they reach the 16384 ceiling.

GitOps upgrade discipline

• Pin chart versions. targetRevision: '*' silently pulled Cilium to 1.20.0 and introduced the CRD stored-version mismatch. Pin to explicit versions and upgrade deliberately.
• Set upgradeCompatibility on Cilium to the originally installed version so upgrades do not silently change map sizing or defaults.
• Validate Helm values indentation. A malformed values block can be silently dropped, leaving a component on unintended defaults (which is how the injector kept its dangerous all-namespaces / 30s configuration).

Operations

• On Talos, verify node liveness via port 50000 or kubectl get node, never the Proxmox VNC framebuffer (it shows the last frame of a halted guest) and never qm list alone (it reports the QEMU process, not the guest OS).

Timeline of shifting symptoms

Each symptom pointed somewhere plausible but wrong. The discipline that resolved it was refusing to fix by inference, reading the node condition, the operator log, the datapath error, the identity breakdown, and finally a raw timed pod create, letting each artifact name its own cause before acting.

If you are running a self-hosted Kubernetes platform that has outgrown the person who built it, or a migration that keeps surfacing problems like these, reach out to Deviqon Labs, because this is the kind of work we do.

About the Author

Robert Rotter is the founder of Deviqon Labs, where he works on video and cloud infrastructure for broadcast and streaming teams.