# Karpenter IBM Cloud Provider Bug Report: Missing Kubeconfig in cluster-info ConfigMap

## Summary
Karpenter IBM Cloud Provider fails to create IKS worker nodes due to missing `kubeconfig` key in the `cluster-info` ConfigMap, despite the provider correctly attempting to use IKS APIs.

## Environment
- **Karpenter Version**: Latest (installed via Helm chart)
- **Provider**: karpenter-provider-ibm-cloud
- **Kubernetes**: IKS cluster v1.29.15+IKS
- **Region**: us-south
- **Date**: 2025-07-05

## Expected Behavior
When `bootstrapMode: iks-api` is configured, Karpenter should:
1. Successfully retrieve cluster configuration from the `cluster-info` ConfigMap
2. Generate proper bootstrap user data for IKS worker nodes
3. Add new worker nodes to the existing IKS cluster

## Actual Behavior
Karpenter correctly attempts to use IKS APIs but fails during bootstrap user data generation:
1. ✅ Finds compatible instance types (60 types found)
2. ✅ Attempts to create instance through IKS path ("Creating instance" log)
3. ❌ Fails with: `kubeconfig not found in cluster-info configmap`
4. ❌ NodeClaim remains in `Unknown` state indefinitely

## Configuration

### IBMNodeClass (Correctly Configured)
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: auto-placement
spec:
  bootstrapMode: iks-api                                    # ✅ IKS API mode
  iksClusterID: d1klma8d0omd29i9ik1g                       # ✅ Valid cluster ID
  image: ibm-ubuntu-24-04-2-minimal-amd64-4
  instanceProfile: bx2-2x8
  region: us-south
  subnet: 0717-197e06f4-b500-426c-bc0f-900b215f996c
  vpc: r006-a8efb117-fd5e-4f63-ae16-4fb9faafa4ff
  zone: us-south-1
```

## Evidence

### 1. Successful Progress Until Bootstrap
```json
{"level":"INFO","message":"Found compatible instance types","count":60,"types":["bx2d-4x16","bx3d-4x20",...]}
{"level":"INFO","message":"Creating instance"}
{"level":"ERROR","message":"Failed to create instance","error":"generating bootstrap user data: getting cluster info: kubeconfig not found in cluster-info configmap"}
```

### 2. Current cluster-info ConfigMap Structure
```yaml
apiVersion: v1
data:
  cluster-config.json: |
    {
      "cluster_id": "d1klma8d0omd29i9ik1g",
      "cluster_name": "prod-dal10-d1klma8d0omd29i9ik1g",
      "cluster_type": "vpc-gen2_cruiser",
      "master_public_url": "https://c111-e.us-south.containers.cloud.ibm.com:30409",
      "master_url": "https://c111.us-south.containers.cloud.ibm.com:30409",
      "crn": "crn:v1:bluemix:public:containers-kubernetes:us-south:..."
    }
  etcd_host: c111.us-south.containers.cloud.ibm.com
  etcd_private_host: c111.private.us-south.containers.cloud.ibm.com
  kube_1_29: 1.29.15_1591
kind: ConfigMap
metadata:
  name: cluster-info
  namespace: kube-system
```

### 3. Missing kubeconfig Key
The provider expects a `kubeconfig` key in the ConfigMap, but the standard IKS `cluster-info` ConfigMap contains:
- ✅ `cluster-config.json` - Has cluster metadata
- ✅ `etcd_host` - Has etcd endpoints  
- ✅ `kube_1_29` - Has version info
- ❌ `kubeconfig` - **MISSING** - Expected by Karpenter

### 4. NodeClaim Status
```bash
$ kubectl describe nodeclaim default-psxwt
Status:
  Conditions:
    Message: creating instance, generating bootstrap user data: getting cluster info: kubeconfig not found in cluster-info configmap
    Reason:  LaunchFailed
    Status:  Unknown
    Type:    Launched
```

## Root Cause Analysis

### Provider Expectation vs IKS Reality
- **Karpenter expects**: `cluster-info` ConfigMap with `kubeconfig` key containing cluster access configuration
- **IKS provides**: `cluster-info` ConfigMap with `cluster-config.json` containing cluster metadata

### Bootstrap Process Flow
1. ✅ Karpenter identifies need for new node
2. ✅ Finds compatible instance types from NodePool requirements  
3. ✅ Attempts IKS instance creation (correct API path)
4. ❌ **FAILS**: Cannot generate bootstrap user data due to missing kubeconfig format

## Potential Solutions

### Option 1: Provider Adaptation
Update the provider to:
- Parse `cluster-config.json` instead of expecting `kubeconfig`
- Extract `master_url` and cluster credentials from IKS-standard format
- Generate appropriate bootstrap configuration for IKS worker nodes

### Option 2: ConfigMap Enhancement
Modify cluster setup to add `kubeconfig` key to `cluster-info` ConfigMap:
```yaml
data:
  kubeconfig: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        server: https://c111.us-south.containers.cloud.ibm.com:30409
        certificate-authority-data: <ca-data>
      name: minimal-karpenter-test
    # ... rest of kubeconfig
```

## Impact
- **Severity**: Medium - IKS scaling blocked but provider architecture is correct
- **Scope**: All IKS users using `bootstrapMode: iks-api`
- **Progress**: Major improvement from previous VPC instance creation bug

## Comparison to Previous Issue
- **Previous**: Provider created standalone VPC instances (completely wrong)
- **Current**: Provider correctly uses IKS APIs but fails on bootstrap configuration (nearly working)

## Workaround
Currently investigating if the `cluster-info` ConfigMap can be manually enhanced with the expected `kubeconfig` format.

## Steps to Reproduce
1. Create IKS cluster 
2. Install latest Karpenter IBM Cloud Provider
3. Configure IBMNodeClass with `bootstrapMode: iks-api`
4. Create NodePool with resource requirements exceeding current capacity
5. Deploy pod requiring scaling
6. Observe NodeClaim creation failure with kubeconfig error

## Additional Information
- ✅ No more standalone VPC instances created
- ✅ Provider respects `bootstrapMode: iks-api` setting
- ✅ Instance type selection works correctly
- ❌ Bootstrap user data generation fails on kubeconfig lookup

This represents significant progress from the previous issue where the provider was creating VPC instances instead of using IKS APIs.