All Pods Across Namespaces:
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE   IP            NODE                          NOMINATED NODE   READINESS GATES
kube-system   aws-node-ds5gz                                           2/2     Running   0          32m   10.0.29.164   ip-10-0-29-164.ec2.internal   <none>           <none>
kube-system   aws-node-q9nm5                                           2/2     Running   0          32m   10.0.34.70    ip-10-0-34-70.ec2.internal    <none>           <none>
kube-system   aws-node-tgszq                                           2/2     Running   0          32m   10.0.6.30     ip-10-0-6-30.ec2.internal     <none>           <none>
kube-system   coredns-54d6f577c6-fjnj5                                 1/1     Running   0          37m   10.0.4.101    ip-10-0-6-30.ec2.internal     <none>           <none>
kube-system   coredns-54d6f577c6-khmhv                                 1/1     Running   0          37m   10.0.3.184    ip-10-0-6-30.ec2.internal     <none>           <none>
kube-system   eks-pod-identity-agent-82gdj                             1/1     Running   0          32m   10.0.6.30     ip-10-0-6-30.ec2.internal     <none>           <none>
kube-system   eks-pod-identity-agent-ngxvl                             1/1     Running   0          32m   10.0.34.70    ip-10-0-34-70.ec2.internal    <none>           <none>
kube-system   eks-pod-identity-agent-ntcqj                             1/1     Running   0          32m   10.0.29.164   ip-10-0-29-164.ec2.internal   <none>           <none>
kube-system   karpenter-6999d88d44-l8xj6                               1/1     Running   0          32m   10.0.27.233   ip-10-0-29-164.ec2.internal   <none>           <none>
kube-system   kube-proxy-hgv8f                                         1/1     Running   0          33m   10.0.34.70    ip-10-0-34-70.ec2.internal    <none>           <none>
kube-system   kube-proxy-mjnzv                                         1/1     Running   0          33m   10.0.6.30     ip-10-0-6-30.ec2.internal     <none>           <none>
kube-system   kube-proxy-vw5ql                                         1/1     Running   0          33m   10.0.29.164   ip-10-0-29-164.ec2.internal   <none>           <none>
monitoring    alertmanager-prometheus-operator-kube-p-alertmanager-0   2/2     Running   0          26m   10.0.46.0     ip-10-0-34-70.ec2.internal    <none>           <none>
monitoring    prometheus-operator-kube-p-operator-6b795b97b6-95h56     1/1     Running   0          26m   10.0.36.133   ip-10-0-34-70.ec2.internal    <none>           <none>
monitoring    prometheus-operator-kube-state-metrics-7d7756cc6-76crn   1/1     Running   0          26m   10.0.34.210   ip-10-0-34-70.ec2.internal    <none>           <none>
monitoring    prometheus-operator-prometheus-node-exporter-cthpz       1/1     Running   0          26m   10.0.29.164   ip-10-0-29-164.ec2.internal   <none>           <none>
monitoring    prometheus-operator-prometheus-node-exporter-n9md2       1/1     Running   0          26m   10.0.34.70    ip-10-0-34-70.ec2.internal    <none>           <none>
monitoring    prometheus-operator-prometheus-node-exporter-xtbpv       1/1     Running   0          26m   10.0.6.30     ip-10-0-6-30.ec2.internal     <none>           <none>
monitoring    prometheus-prometheus-operator-kube-p-prometheus-0       2/2     Running   0          26m   10.0.7.162    ip-10-0-6-30.ec2.internal     <none>           <none>

Karpenter Pods Status:
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE
kube-system   aws-node-ds5gz                                           2/2     Running   0          31m
kube-system   aws-node-q9nm5                                           2/2     Running   0          31m
kube-system   aws-node-tgszq                                           2/2     Running   0          31m
kube-system   coredns-54d6f577c6-fjnj5                                 1/1     Running   0          36m
kube-system   coredns-54d6f577c6-khmhv                                 1/1     Running   0          36m
kube-system   eks-pod-identity-agent-82gdj                             1/1     Running   0          31m
kube-system   eks-pod-identity-agent-ngxvl                             1/1     Running   0          31m
kube-system   eks-pod-identity-agent-ntcqj                             1/1     Running   0          31m
kube-system   karpenter-6999d88d44-l8xj6                               1/1     Running   0          30m
kube-system   kube-proxy-hgv8f                                         1/1     Running   0          32m
kube-system   kube-proxy-mjnzv                                         1/1     Running   0          32m
kube-system   kube-proxy-vw5ql                                         1/1     Running   0          32m
monitoring    alertmanager-prometheus-operator-kube-p-alertmanager-0   2/2     Running   0          25m
monitoring    prometheus-operator-kube-p-operator-6b795b97b6-95h56     1/1     Running   0          25m
monitoring    prometheus-operator-kube-state-metrics-7d7756cc6-76crn   1/1     Running   0          25m
monitoring    prometheus-operator-prometheus-node-exporter-cthpz       1/1     Running   0          25m
monitoring    prometheus-operator-prometheus-node-exporter-n9md2       1/1     Running   0          25m
monitoring    prometheus-operator-prometheus-node-exporter-xtbpv       1/1     Running   0          25m
monitoring    prometheus-prometheus-operator-kube-p-prometheus-0       2/2     Running   0          25m

CAS Pods Status:
NAMESPACE     NAME                                                     READY   STATUS    RESTARTS   AGE
cas           cas-aws-cluster-autoscaler-5bd47d57d6-ndfx6              1/1     Running   0          6m51s
kube-system   aws-node-fqp9v                                           2/2     Running   0          28m
kube-system   aws-node-pbt26                                           2/2     Running   0          6m31s
kube-system   coredns-54d6f577c6-6qghr                                 1/1     Running   0          33m
kube-system   coredns-54d6f577c6-wqxm4                                 1/1     Running   0          33m
kube-system   eks-pod-identity-agent-tz9xf                             1/1     Running   0          6m31s
kube-system   eks-pod-identity-agent-w6zv7                             1/1     Running   0          28m
kube-system   kube-proxy-69t7d                                         1/1     Running   0          6m31s
kube-system   kube-proxy-q68pm                                         1/1     Running   0          29m
monitoring    alertmanager-prometheus-operator-kube-p-alertmanager-0   2/2     Running   0          6m43s
monitoring    prometheus-operator-kube-p-operator-6b795b97b6-jkw98     1/1     Running   0          7m11s
monitoring    prometheus-operator-kube-state-metrics-7d7756cc6-jm9fq   1/1     Running   0          6m44s
monitoring    prometheus-operator-prometheus-node-exporter-8wdxv       1/1     Running   0          24m
monitoring    prometheus-operator-prometheus-node-exporter-ssp6k       1/1     Running   0          6m31s
monitoring    prometheus-prometheus-operator-kube-p-prometheus-0       2/2     Running   0          24m

Karpenter Pods in Namespace:
NAMESPACE     NAME                         READY   STATUS    RESTARTS   AGE
kube-system   karpenter-6999d88d44-l8xj6   1/1     Running   0          32m

CAS Pods in Namespace:


Karpenter Logs:
{"level":"INFO","time":"2024-12-13T17:27:06.684Z","logger":"controller","message":"found provisionable pod(s)","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","Pods":"homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-ltcx7, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-qwrpz, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-68p5q, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-fvp99, homogeneous-workload/homogeneous-workload-karpenter-6f598db84f-pxvzj and 22 other(s)","duration":"99.523063ms"}
{"level":"INFO","time":"2024-12-13T17:27:06.684Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","nodeclaims":1,"pods":27}
{"level":"INFO","time":"2024-12-13T17:27:06.707Z","logger":"controller","message":"created nodeclaim","commit":"5bdf9c3","controller":"provisioner","namespace":"","name":"","reconcileID":"c448aa7a-f92f-4c97-a7ba-afddd76bbe03","NodePool":{"name":"default"},"NodeClaim":{"name":"default-z9bjs"},"requests":{"cpu":"6900m","memory":"6912Mi","pods":"31"},"instance-types":"c5.2xlarge, c5.4xlarge, c5a.2xlarge, c5a.4xlarge, c5ad.2xlarge and 55 other(s)"}
{"level":"INFO","time":"2024-12-13T17:27:09.192Z","logger":"controller","message":"launched nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"86ec4df7-6cbf-4251-a6ea-9dad72cf73e3","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","instance-type":"c5d.2xlarge","zone":"us-east-1c","capacity-type":"spot","allocatable":{"cpu":"7910m","ephemeral-storage":"17Gi","memory":"14162Mi","pods":"58","vpc.amazonaws.com/pod-eni":"38"}}
{"level":"INFO","time":"2024-12-13T17:27:32.010Z","logger":"controller","message":"registered nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"83350423-d042-4011-80b3-1b1f3780ee1a","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","Node":{"name":"ip-10-0-35-120.ec2.internal"}}
{"level":"INFO","time":"2024-12-13T17:27:42.416Z","logger":"controller","message":"initialized nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"5ebfdef3-8147-40f6-bdd0-294687af865a","provider-id":"aws:///us-east-1c/i-03883828477d0baa1","Node":{"name":"ip-10-0-35-120.ec2.internal"},"allocatable":{"cpu":"7910m","ephemeral-storage":"18181869946","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"14853408Ki","pods":"58"}}
{"level":"INFO","time":"2024-12-13T17:38:33.782Z","logger":"controller","message":"disrupting nodeclaim(s) via delete, terminating 1 nodes (0 pods) ip-10-0-35-120.ec2.internal/c5d.2xlarge/spot","commit":"5bdf9c3","controller":"disruption","namespace":"","name":"","reconcileID":"8b57ac63-0cac-4a96-971d-bca686c7c7f1","command-id":"601f7237-358e-4ee5-97b2-b4a3ce4bc4c1","reason":"empty"}
{"level":"INFO","time":"2024-12-13T17:38:34.434Z","logger":"controller","message":"tainted node","commit":"5bdf9c3","controller":"node.termination","controllerGroup":"","controllerKind":"Node","Node":{"name":"ip-10-0-35-120.ec2.internal"},"namespace":"","name":"ip-10-0-35-120.ec2.internal","reconcileID":"8636073b-a1f3-4fee-bb9d-d83fe344a1ab","taint.Key":"karpenter.sh/disrupted","taint.Value":"","taint.Effect":"NoSchedule"}
{"level":"INFO","time":"2024-12-13T17:39:47.686Z","logger":"controller","message":"deleted node","commit":"5bdf9c3","controller":"node.termination","controllerGroup":"","controllerKind":"Node","Node":{"name":"ip-10-0-35-120.ec2.internal"},"namespace":"","name":"ip-10-0-35-120.ec2.internal","reconcileID":"2f26c62c-e0e8-4ef0-b7a9-71fa128fcd79"}
{"level":"INFO","time":"2024-12-13T17:39:47.928Z","logger":"controller","message":"deleted nodeclaim","commit":"5bdf9c3","controller":"nodeclaim.termination","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-z9bjs"},"namespace":"","name":"default-z9bjs","reconcileID":"6ce7834c-fc6a-467f-bece-c0bd9f3bdb26","Node":{"name":"ip-10-0-35-120.ec2.internal"},"provider-id":"aws:///us-east-1c/i-03883828477d0baa1"}

CAS Logs:
Failed to get logs

Karpenter Pod Details:
Name:                   karpenter
Namespace:              kube-system
CreationTimestamp:      Fri, 13 Dec 2024 18:18:38 +0100
Labels:                 app.kubernetes.io/instance=karpenter
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=karpenter
                        app.kubernetes.io/version=1.0.0
                        helm.sh/chart=karpenter-1.0.0
Annotations:            deployment.kubernetes.io/revision: 1
                        meta.helm.sh/release-name: karpenter
                        meta.helm.sh/release-namespace: kube-system
Selector:               app.kubernetes.io/instance=karpenter,app.kubernetes.io/name=karpenter
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=karpenter
                    app.kubernetes.io/name=karpenter
  Service Account:  karpenter
  Containers:
   controller:
    Image:           public.ecr.aws/karpenter/controller:1.0.0@sha256:1eb1073b9f4ed804634aabf320e4d6e822bb61c0f5ecfd9c3a88f05f1ca4c5c5
    Ports:           8080/TCP, 8001/TCP, 8443/TCP, 8081/TCP
    Host Ports:      0/TCP, 0/TCP, 0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      1
      memory:   1Gi
    Liveness:   http-get http://:http/healthz delay=30s timeout=30s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/readyz delay=5s timeout=30s period=10s #success=1 #failure=3
    Environment:
      KUBERNETES_MIN_VERSION:      1.19.0-0
      KARPENTER_SERVICE:           karpenter
      WEBHOOK_PORT:                8443
      WEBHOOK_METRICS_PORT:        8001
      DISABLE_WEBHOOK:             false
      LOG_LEVEL:                   info
      METRICS_PORT:                8080
      HEALTH_PROBE_PORT:           8081
      SYSTEM_NAMESPACE:             (v1:metadata.namespace)
      MEMORY_LIMIT:                1073741824 (limits.memory)
      FEATURE_GATES:               SpotToSpotConsolidation=false
      BATCH_MAX_DURATION:          10s
      BATCH_IDLE_DURATION:         1s
      CLUSTER_NAME:                karpenter-eks
      CLUSTER_ENDPOINT:            https://8E9E8B7FC120D19AF470E6CBEA68AFF8.gr7.us-east-1.eks.amazonaws.com
      VM_MEMORY_OVERHEAD_PERCENT:  0.075
      INTERRUPTION_QUEUE:          Karpenter-karpenter-eks
      RESERVED_ENIS:               0
    Mounts:                        <none>
  Volumes:                         <none>
  Topology Spread Constraints:     topology.kubernetes.io/zone:DoNotSchedule when max skew 1 is exceeded for selector app.kubernetes.io/instance=karpenter,app.kubernetes.io/name=karpenter
  Priority Class Name:             system-cluster-critical
  Node-Selectors:                  kubernetes.io/os=linux
  Tolerations:                     CriticalAddonsOnly op=Exists
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   karpenter-6999d88d44 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  32m   deployment-controller  Scaled up replica set karpenter-6999d88d44 to 1

CAS Pod Details:
Failed to get pod details

Karpenter Resources:
