All Pods Across Namespaces: NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system aws-node-ds5gz 2/2 Running 0 32m 10.0.29.164 ip-10-0-29-164.ec2.internal kube-system aws-node-q9nm5 2/2 Running 0 32m 10.0.34.70 ip-10-0-34-70.ec2.internal kube-system aws-node-tgszq 2/2 Running 0 32m 10.0.6.30 ip-10-0-6-30.ec2.internal kube-system coredns-54d6f577c6-fjnj5 1/1 Running 0 37m 10.0.4.101 ip-10-0-6-30.ec2.internal kube-system coredns-54d6f577c6-khmhv 1/1 Running 0 37m 10.0.3.184 ip-10-0-6-30.ec2.internal kube-system eks-pod-identity-agent-82gdj 1/1 Running 0 32m 10.0.6.30 ip-10-0-6-30.ec2.internal kube-system eks-pod-identity-agent-ngxvl 1/1 Running 0 32m 10.0.34.70 ip-10-0-34-70.ec2.internal kube-system eks-pod-identity-agent-ntcqj 1/1 Running 0 32m 10.0.29.164 ip-10-0-29-164.ec2.internal kube-system karpenter-6999d88d44-l8xj6 1/1 Running 0 32m 10.0.27.233 ip-10-0-29-164.ec2.internal kube-system kube-proxy-hgv8f 1/1 Running 0 33m 10.0.34.70 ip-10-0-34-70.ec2.internal kube-system kube-proxy-mjnzv 1/1 Running 0 33m 10.0.6.30 ip-10-0-6-30.ec2.internal kube-system kube-proxy-vw5ql 1/1 Running 0 33m 10.0.29.164 ip-10-0-29-164.ec2.internal monitoring alertmanager-prometheus-operator-kube-p-alertmanager-0 2/2 Running 0 26m 10.0.46.0 ip-10-0-34-70.ec2.internal monitoring prometheus-operator-kube-p-operator-6b795b97b6-95h56 1/1 Running 0 26m 10.0.36.133 ip-10-0-34-70.ec2.internal monitoring prometheus-operator-kube-state-metrics-7d7756cc6-76crn 1/1 Running 0 26m 10.0.34.210 ip-10-0-34-70.ec2.internal monitoring prometheus-operator-prometheus-node-exporter-cthpz 1/1 Running 0 26m 10.0.29.164 ip-10-0-29-164.ec2.internal monitoring prometheus-operator-prometheus-node-exporter-n9md2 1/1 Running 0 26m 10.0.34.70 ip-10-0-34-70.ec2.internal monitoring prometheus-operator-prometheus-node-exporter-xtbpv 1/1 Running 0 26m 10.0.6.30 ip-10-0-6-30.ec2.internal monitoring prometheus-prometheus-operator-kube-p-prometheus-0 2/2 Running 0 26m 10.0.7.162 ip-10-0-6-30.ec2.internal Karpenter Pods Status: NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-ds5gz 2/2 Running 0 31m kube-system aws-node-q9nm5 2/2 Running 0 31m kube-system aws-node-tgszq 2/2 Running 0 31m kube-system coredns-54d6f577c6-fjnj5 1/1 Running 0 36m kube-system coredns-54d6f577c6-khmhv 1/1 Running 0 36m kube-system eks-pod-identity-agent-82gdj 1/1 Running 0 31m kube-system eks-pod-identity-agent-ngxvl 1/1 Running 0 31m kube-system eks-pod-identity-agent-ntcqj 1/1 Running 0 31m kube-system karpenter-6999d88d44-l8xj6 1/1 Running 0 30m kube-system kube-proxy-hgv8f 1/1 Running 0 32m kube-system kube-proxy-mjnzv 1/1 Running 0 32m kube-system kube-proxy-vw5ql 1/1 Running 0 32m monitoring alertmanager-prometheus-operator-kube-p-alertmanager-0 2/2 Running 0 25m monitoring prometheus-operator-kube-p-operator-6b795b97b6-95h56 1/1 Running 0 25m monitoring prometheus-operator-kube-state-metrics-7d7756cc6-76crn 1/1 Running 0 25m monitoring prometheus-operator-prometheus-node-exporter-cthpz 1/1 Running 0 25m monitoring prometheus-operator-prometheus-node-exporter-n9md2 1/1 Running 0 25m monitoring prometheus-operator-prometheus-node-exporter-xtbpv 1/1 Running 0 25m monitoring prometheus-prometheus-operator-kube-p-prometheus-0 2/2 Running 0 25m CAS Pods Status: NAMESPACE NAME READY STATUS RESTARTS AGE cas cas-aws-cluster-autoscaler-5bd47d57d6-ndfx6 1/1 Running 0 6m51s kube-system aws-node-fqp9v 2/2 Running 0 28m kube-system aws-node-pbt26 2/2 Running 0 6m31s kube-system coredns-54d6f577c6-6qghr 1/1 Running 0 33m kube-system coredns-54d6f577c6-wqxm4 1/1 Running 0 33m kube-system eks-pod-identity-agent-tz9xf 1/1 Running 0 6m31s kube-system eks-pod-identity-agent-w6zv7 1/1 Running 0 28m kube-system kube-proxy-69t7d 1/1 Running 0 6m31s kube-system kube-proxy-q68pm 1/1 Running 0 29m monitoring alertmanager-prometheus-operator-kube-p-alertmanager-0 2/2 Running 0 6m43s monitoring prometheus-operator-kube-p-operator-6b795b97b6-jkw98 1/1 Running 0 7m11s monitoring prometheus-operator-kube-state-metrics-7d7756cc6-jm9fq 1/1 Running 0 6m44s monitoring prometheus-operator-prometheus-node-exporter-8wdxv 1/1 Running 0 24m monitoring prometheus-operator-prometheus-node-exporter-ssp6k 1/1 Running 0 6m31s monitoring prometheus-prometheus-operator-kube-p-prometheus-0 2/2 Running 0 24m Karpenter Pods in Namespace: NAMESPACE NAME READY STATUS RESTARTS AGE kube-system karpenter-6999d88d44-l8xj6 1/1 Running 0 32m CAS Pods in Namespace: Karpenter Logs: {"level":"INFO","time":"2024-12-13T17:27:06.684Z","logger":"controller","message":"found provisionable pod(s)","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","Pods":"homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-ltcx7, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-qwrpz, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-68p5q, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-fvp99, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-pxvzj and 22 other(s)","duration":"99.523063ms"} {"level":"INFO","time":"2024-12-13T17:27:06.684Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","nodeclaims":1,"pods":27} {"level":"INFO","time":"2024-12-13T17:27:06.707Z","logger":"controller","message":"created nodeclaim","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","NodePool":{"name":"default"},"NodeClaim":{"name":"default-z9bjs"},"requests":{"cpu":"6900m","memory":"6912Mi","pods":"31"},"instance-types":"c5.2xlarge, c5.4xlarge, c5a.2xlarge, c5a.4xlarge, c5ad.2xlarge and 55 other(s)"} {"level":"INFO","time":"2024-12-13T17:27:09.192Z","logger":"controller","message":"launched nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"86ec4df7-6cbf-4251-a6ea-9dad72cf73e3","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","instance-type":"c5d.2xlarge","zone":"us-east-1c","capacity-type":"spot","allocatable":{"cpu":"7910m","ephemeral-storage":"17Gi","memory":"14162Mi","pods":"58","vpc.amazonaws.com/pod-eni":"38"}} {"level":"INFO","time":"2024-12-13T17:27:32.010Z","logger":"controller","message":"registered nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"83350423-d042-4011-80b3-1b1f3780ee1a","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","Node":{"name":"ip-10-0-35-120.ec2.internal"}} {"level":"INFO","time":"2024-12-13T17:27:42.416Z","logger":"controller","message":"initialized nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"5ebfdef3-8147-40f6-bdd0-294687af865a","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","Node":{"name":"ip-10-0-35-120.ec2.internal"},"allocatable":{"cpu":"7910m","ephemeral-storage":"18181869946","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"14853408Ki","pods":"58"}} {"level":"INFO","time":"2024-12-13T17:38:33.782Z","logger":"controller","message":"disrupting nodeclaim(s) via delete, terminating 1 nodes (0 pods) ip-10-0-35-120.ec2.internal/c5d.2xlarge/spot","commit":"5bdf9c3","controller":"disruption","namespace":"","name":"","reconcileID":"8b57ac63-0cac-4a96-971d-bca686c7c7f1","command-id":"601f7237-358e-4ee5-97b2-b4a3ce4bc4c1","reason":"empty"} {"level":"INFO","time":"2024-12-13T17:38:34.434Z","logger":"controller","message":"tainted node","commit":"5bdf9c3","controller":"node.termination","controllerGroup":"","controllerKind":"Node","Node":{"name":"ip-10-0-35-120.ec2.internal"},"namespace":"","name":"ip-10-0-35-120.ec2.internal","reconcileID":"8636073b-a1f3-4fee-bb9d-d83fe344a1ab","taint.Key":"karpenter.sh/disrupted","taint.Value":"","taint.Effect":"NoSchedule"} {"level":"INFO","time":"2024-12-13T17:39:47.686Z","logger":"controller","message":"deleted node","commit":"5bdf9c3","controller":"node.termination","controllerGroup":"","controllerKind":"Node","Node":{"name":"ip-10-0-35-120.ec2.internal"},"namespace":"","name":"ip-10-0-35-120.ec2.internal","reconcileID":"2f26c62c-e0e8-4ef0-b7a9-71fa128fcd79"} {"level":"INFO","time":"2024-12-13T17:39:47.928Z","logger":"controller","message":"deleted nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.termination","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"6ce7834c-fc6a-467f-bece-c0bd9f3bdb26","Node":{"name":"ip-10-0-35-120.ec2.internal"},"provider-id":"aws:///us-east-1c/i-03883828477d0baa1"} CAS Logs: Failed to get logs Karpenter Pod Details: Name: karpenter Namespace: kube-system CreationTimestamp: Fri, 13 Dec 2024 18:18:38 +0100 Labels: app.kubernetes.io/instance=karpenter app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=karpenter app.kubernetes.io/version=1.0.0 helm.sh/chart=karpenter-1.0.0 Annotations: deployment.kubernetes.io/revision: 1 meta.helm.sh/release-name: karpenter meta.helm.sh/release-namespace: kube-system Selector: app.kubernetes.io/instance=karpenter,app.kubernetes.io/name=karpenter Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable StrategyType: RollingUpdate MinReadySeconds: 0 RollingUpdateStrategy: 1 max unavailable, 25% max surge Pod Template: Labels: app.kubernetes.io/instance=karpenter app.kubernetes.io/name=karpenter Service Account: karpenter Containers: controller: Image: public.ecr.aws/karpenter/controller:1.0.0@sha256:1eb1073b9f4ed804634aabf320e4d6e822bb61c0f5ecfd9c3a88f05f1ca4c5c5 Ports: 8080/TCP, 8001/TCP, 8443/TCP, 8081/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP SeccompProfile: RuntimeDefault Limits: cpu: 1 memory: 1Gi Requests: cpu: 1 memory: 1Gi Liveness: http-get http://:http/healthz delay=30s timeout=30s period=10s #success=1 #failure=3 Readiness: http-get http://:http/readyz delay=5s timeout=30s period=10s #success=1 #failure=3 Environment: KUBERNETES_MIN_VERSION: 1.19.0-0 KARPENTER_SERVICE: karpenter WEBHOOK_PORT: 8443 WEBHOOK_METRICS_PORT: 8001 DISABLE_WEBHOOK: false LOG_LEVEL: info METRICS_PORT: 8080 HEALTH_PROBE_PORT: 8081 SYSTEM_NAMESPACE: (v1:metadata.namespace) MEMORY_LIMIT: 1073741824 (limits.memory) FEATURE_GATES: SpotToSpotConsolidation=false BATCH_MAX_DURATION: 10s BATCH_IDLE_DURATION: 1s CLUSTER_NAME: karpenter-eks CLUSTER_ENDPOINT: https://8E9E8B7FC120D19AF470E6CBEA68AFF8.gr7.us-east-1.eks.amazonaws.com VM_MEMORY_OVERHEAD_PERCENT: 0.075 INTERRUPTION_QUEUE: Karpenter-karpenter-eks RESERVED_ENIS: 0 Mounts: Volumes: Topology Spread Constraints: topology.kubernetes.io/zone:DoNotSchedule when max skew 1 is exceeded for selector app.kubernetes.io/instance=karpenter,app.kubernetes.io/name=karpenter Priority Class Name: system-cluster-critical Node-Selectors: kubernetes.io/os=linux Tolerations: CriticalAddonsOnly op=Exists Conditions: Type Status Reason ---- ------ ------ Available True MinimumReplicasAvailable Progressing True NewReplicaSetAvailable OldReplicaSets: NewReplicaSet: karpenter-6999d88d44 (1/1 replicas created) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 32m deployment-controller Scaled up replica set karpenter-6999d88d44 to 1 CAS Pod Details: Failed to get pod details Karpenter Resources: