Kubernetes Namespace Stuck in 'Terminating'

Question

I'm encountering an issue where a Kubernetes namespace is stuck in the 'Terminating' state. Running kubectl get ns cattle-monitoring-system -o json|jq produces error messages related to custom.metrics.k8s.io/v1beta1 and shows a DiscoveryFailed condition in the namespace status:

E1213 08:02:39.979034  953148 memcache.go:287] couldn't get resource list for custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request
…
{
  "apiVersion": "v1",
  "kind": "Namespace",
  …
  "status": {
    "conditions": [
      {
        "lastTransitionTime": "2023-12-12T14:53:40Z",
        "message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: the server is currently unable to handle the request",
        "reason": "DiscoveryFailed",
        "status": "True",
        "type": "NamespaceDeletionDiscoveryFailure"
      },
      …
    ]
  }
}

How can I resolve this issue to successfully delete the namespace?

karlsebal · Accepted Answer · 2024-02-12 17:58:36Z

The issue you're encountering is related to the Kubernetes API server's inability to discover the custom.metrics.k8s.io/v1beta1 API, which is blocking the namespace from terminating. Here are the steps to troubleshoot and resolve this:

Check for Associated APIService

Investigate if there's an APIService for custom.metrics.k8s.io/v1beta1:

kubectl get apiservices.apiregistration.k8s.io -o json|jq  '.items[]|select(.metadata.name=="v1beta1.custom.metrics.k8s.io")'

Output should be something like

{
  "apiVersion": "apiregistration.k8s.io/v1",
  "kind": "APIService",
  ...
  "status": {
    "conditions": [
      {
        "message": "service/example-service in \"example-namespace\" is not present",
        "reason": "ServiceNotFound",
        "status": "False",
        "type": "Available"
      }
    ]
  }
}

Verify No Active Components Related to APIService:

Issue the following command. Adjust example-service according to the output you generated earlier.

for c in configmaps secrets deployments statefulsets pods services; do
    echo "Checking for $c related to example-service"
    kubectl get $c -A | grep example-service
done

If there's no output, it confirms there are no active components related to the APIService in the cluster.

Delete the APIService

kubectl delete apiservice v1beta1.custom.metrics.k8s.io

After deleting the APIService, the namespace should proceed to terminate. Monitor the cluster for any unexpected issues.

Stack Exchange Network

Kubernetes Namespace Stuck in 'Terminating'

1 Answer 1

Check for Associated APIService

Verify No Active Components Related to APIService:

Delete the APIService

You must log in to answer this question.

Hot Network Questions