# Getting Started with Karpenter IBM Cloud Provider

This guide walks you through setting up Karpenter IBM Cloud Provider from installation to your first auto-scaled workload.

## Prerequisites

Before starting, ensure you have:

### Required Access
- **IBM Cloud Account** with VPC Infrastructure Services access
- **Kubernetes Cluster** (IKS or self-managed on IBM Cloud VPC)
- **kubectl** configured for your cluster
- **Helm 3** for installation

### Required Tools
```bash
# Verify tools are available
kubectl version --client
helm version
ibmcloud version

# Install IBM Cloud CLI if needed
curl -fsSL https://clis.cloud.ibm.com/install/linux | sh
ibmcloud plugin install vpc-infrastructure
```

## IBM Cloud Setup

### Step 1: Create Service ID and API Keys
For production environments, use Service IDs for better security:

```bash
# Login to IBM Cloud
ibmcloud login

# Create Service ID for Karpenter
ibmcloud iam service-id-create karpenter-provider \
  --description "Service ID for Karpenter IBM Cloud Provider"

# Get Service ID
SERVICE_ID=$(ibmcloud iam service-ids --output json | jq -r '.[] | select(.name=="karpenter-provider") | .id')

# Assign VPC Infrastructure Services role
ibmcloud iam service-policy-create $SERVICE_ID \
  --roles "VPC Infrastructure Services" \
  --service-name is

# Create API keys
ibmcloud iam service-api-key-create karpenter-general $SERVICE_ID \
  --description "General IBM Cloud API access for Karpenter"

ibmcloud iam service-api-key-create karpenter-vpc $SERVICE_ID \
  --description "VPC-specific API access for Karpenter"
```

**Save the API keys securely - they won't be shown again!**

### Step 2: Gather Required Resource Information

```bash
# Set your target region
export REGION=us-south
ibmcloud target -r $REGION

# List available VPCs
ibmcloud is vpcs --output json

# Choose your VPC and list subnets
export VPC_ID="your-vpc-id"
ibmcloud is subnets --vpc $VPC_ID --output json

# List security groups in your VPC
ibmcloud is security-groups --vpc $VPC_ID --output json

# List available images
ibmcloud is images --visibility public --status available | grep ubuntu
```

**Collect the following information:**
- **VPC ID**: `vpc-12345678` (from VPC list)
- **Subnet ID**: `subnet-12345678` (choose one per zone you want to use)
- **Security Group ID**: `sg-12345678` (existing or create new)
- **Image ID**: `r006-12345678` (Ubuntu 20.04 recommended)
- **Region**: `us-south` (or your preferred region)
- **Zone**: `us-south-1` (subnet's availability zone)

## Installation

### Step 1: Create Kubernetes Secrets
Store your API keys securely in Kubernetes:

```bash
# Create namespace
kubectl create namespace karpenter

# Create secret with API keys
kubectl create secret generic karpenter-ibm-credentials \
  --from-literal=api-key="your-general-api-key" \
  --from-literal=vpc-api-key="your-vpc-api-key" \
  --namespace karpenter

# Verify secret creation
kubectl get secret karpenter-ibm-credentials -n karpenter
```

### Step 2: Install Karpenter IBM Cloud Provider

**Option A: Using Helm (Recommended)**
```bash
# Add Helm repository
helm repo add karpenter-ibm https://pfeifferj.github.io/karpenter-provider-ibm-cloud
helm repo update

# Install with environment variables
helm install karpenter karpenter-ibm/karpenter \
  --namespace karpenter \
  --create-namespace \
  --set controller.env.IBM_REGION="us-south" \
  --set controller.env.IBM_API_KEY.valueFrom.secretKeyRef.name="karpenter-ibm-credentials" \
  --set controller.env.IBM_API_KEY.valueFrom.secretKeyRef.key="api-key" \
  --set controller.env.VPC_API_KEY.valueFrom.secretKeyRef.name="karpenter-ibm-credentials" \
  --set controller.env.VPC_API_KEY.valueFrom.secretKeyRef.key="vpc-api-key"
```

**Option B: Using kubectl and manifests**
```bash
# Download manifests
curl -O https://raw.githubusercontent.com/pfeifferj/karpenter-provider-ibm-cloud/main/charts/karpenter/templates/deployment.yaml

# Apply with your values
kubectl apply -f deployment.yaml
```

### Step 3: Verify Installation
```bash
# Check if pods are running
kubectl get pods -n karpenter

# Check controller logs for startup
kubectl logs -n karpenter deployment/karpenter -f

# Look for successful controller startup messages
# Expected: "Starting Controller" for nodepool, nodeclaim, nodeclass controllers
```

**✅ Expected Output:**
```
Starting Controller {"controller": "nodepool", "controllerGroup": "karpenter.sh"}
Starting Controller {"controller": "nodeclaim", "controllerGroup": "karpenter.sh"}
Starting Controller {"controller": "nodeclass.hash", "controllerGroup": "karpenter.ibm.sh"}
Starting Controller {"controller": "pricing", "controllerGroup": "karpenter.ibm.sh"}
```

### Step 4: Create Your First NodeClass

Choose the configuration that matches your environment:

**📝 Replace the placeholder values with your actual resource IDs:**

```bash
# Create basic NodeClass
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: default-nodeclass
  annotations:
    karpenter.ibm.sh/description: "Basic NodeClass for getting started"
spec:
  # REQUIRED: Replace with your actual values
  region: us-south                    # Your IBM Cloud region
  zone: us-south-1                    # Target availability zone
  vpc: vpc-12345678                   # Your VPC ID
  image: r006-12345678                # Ubuntu 20.04 image ID
  
  # REQUIRED: Security and access
  securityGroups:
  - sg-12345678                       # Your security group ID
  
  # OPTIONAL: Specific subnet (auto-selected if not specified)
  subnet: subnet-12345678             # Your subnet ID
  
  # OPTIONAL: SSH access (recommended for troubleshooting)
  sshKeys:
  - key-12345678                      # Your SSH key ID
  
  # Bootstrap mode - fully automatic
  # No userData required - Karpenter handles everything!
EOF
```

### Step 5: Create NodePool for Auto-Scaling

```bash
cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default-nodepool
  annotations:
    karpenter.sh/description: "General purpose NodePool"
spec:
  # Template for nodes created by this pool
  template:
    metadata:
      labels:
        provisioner: karpenter
        environment: getting-started
    spec:
      # Reference to our NodeClass
      nodeClassRef:
        apiVersion: karpenter.ibm.sh/v1alpha1
        kind: IBMNodeClass
        name: default-nodeclass
      
      # Instance requirements
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values: ["bx2-2x8", "bx2-4x16", "cx2-2x4", "cx2-4x8"]
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
  
  # Resource limits (adjust based on your needs)
  limits:
    cpu: 1000    # Max 1000 vCPUs across all nodes in this pool
    memory: 1000Gi
  
  # Disruption settings for cost optimization
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
    # Remove nodes when they're empty for 30 seconds
EOF
```

### Step 6: Verify Configuration

```bash
# Check if NodeClass is ready
kubectl get ibmnodeclasses
kubectl describe ibmnodeclass default-nodeclass

# Check if NodePool is ready  
kubectl get nodepools
kubectl describe nodepool default-nodepool

# Expected STATUS: Ready for both resources
```

### Step 7: Test Auto-Scaling

Deploy a workload that requires node provisioning:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auto-scaling-test
  labels:
    app: scaling-test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: scaling-test
  template:
    metadata:
      labels:
        app: scaling-test
    spec:
      containers:
      - name: resource-consumer
        image: busybox:1.35
        command: ["sleep", "3600"]
        resources:
          requests:
            cpu: "2"      # Request 2 vCPUs per pod
            memory: 4Gi   # Request 4GB RAM per pod
          limits:
            cpu: "2"
            memory: 4Gi
      # Ensure pods spread across different nodes
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: scaling-test
              topologyKey: kubernetes.io/hostname
EOF
```

### Step 8: Monitor Auto-Scaling

Watch Karpenter provision nodes automatically:

```bash
# Watch NodeClaims being created
kubectl get nodeclaims -w

# Watch nodes being added to cluster
kubectl get nodes -w

# Check pod scheduling
kubectl get pods -o wide

# Monitor Karpenter controller logs
kubectl logs -n karpenter deployment/karpenter -f
```

**🎉 Expected Behavior:**
1. Pods will initially show as `Pending` (no suitable nodes)
2. Karpenter detects unschedulable pods
3. Creates NodeClaims based on pod requirements
4. Provisions IBM Cloud VPC instances
5. Instances join cluster automatically (no manual setup!)
6. Pods get scheduled on new nodes

**⏰ Typical Timeline:**
- NodeClaim creation: ~10 seconds
- IBM Cloud instance provisioning: 2-3 minutes
- Bootstrap and cluster join: 1-2 minutes
- **Total**: 3-5 minutes for new nodes to be ready

## Common Configurations

### IKS Integration Mode
For IBM Kubernetes Service clusters, use IKS mode for seamless integration:

```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: iks-nodeclass
spec:
  region: us-south
  vpc: vpc-iks-12345
  image: r006-ubuntu-20-04
  
  # IKS-specific configuration
  iksClusterID: "cluster-12345678"
  iksWorkerPoolID: "pool-default"  # Optional: specific worker pool
  
  securityGroups:
  - sg-iks-workers
```

### Multi-Zone Production Setup
For high availability, create separate NodeClasses per zone:

```yaml
# Zone 1 NodeClass
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: prod-us-south-1
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-prod-12345
  image: r006-rhel-8
  instanceProfile: bx2-4x16
  securityGroups:
  - sg-prod-web
  - sg-prod-monitoring
---
# Zone 2 NodeClass  
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: prod-us-south-2
spec:
  region: us-south
  zone: us-south-2
  vpc: vpc-prod-12345
  image: r006-rhel-8
  instanceProfile: bx2-4x16
  securityGroups:
  - sg-prod-web
  - sg-prod-monitoring
```

### GPU Workloads
For GPU-accelerated workloads:

```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: gpu-nodeclass
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-gpu-12345
  image: r006-ubuntu-20-04
  instanceProfile: gx2-8x64x1v100  # GPU instance
  securityGroups:
  - sg-gpu-workloads
  userData: |
    #!/bin/bash
    # GPU drivers will be installed automatically
    # Custom setup can be added here
```

### Custom Bootstrap Configuration
For specialized setups:

```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: custom-nodeclass
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-custom-12345
  image: r006-ubuntu-20-04
  userData: |
    #!/bin/bash
    # Custom pre-bootstrap setup
    echo "Installing custom packages..."
    apt-get update && apt-get install -y htop monitoring-agent
    
    # Custom environment configuration
    echo "ENVIRONMENT=production" >> /etc/environment
    
    # The bootstrap script is automatically appended
```

## Troubleshooting Quick Fixes

### NodeClass Not Ready
```bash
# Check validation errors
kubectl describe ibmnodeclass default-nodeclass

# Look for validation messages in status
kubectl get ibmnodeclass default-nodeclass -o yaml

# Common issues and solutions:
# ❌ Invalid VPC ID → Verify VPC exists in region
# ❌ Invalid subnet ID → Check subnet exists in specified zone  
# ❌ Invalid security group → Ensure security group exists in VPC
# ❌ Invalid image ID → Use valid IBM Cloud image ID
# ❌ Missing API permissions → Check Service ID roles
```

### Nodes Not Provisioning
```bash
# Check NodeClaim status and events
kubectl get nodeclaims -o wide
kubectl describe nodeclaim <nodeclaim-name>

# Check controller logs for specific errors
kubectl logs -n karpenter deployment/karpenter --tail=100 | grep ERROR

# Common issues:
# ❌ Quota exceeded → Contact IBM Cloud support for quota increase
# ❌ Instance type unavailable → Try different instance types in NodePool
# ❌ Subnet has no available IPs → Use larger subnet or multiple subnets
```

### Pods Still Pending
```bash
# Check pod resource requirements
kubectl describe pod <pending-pod-name>

# Verify NodePool can satisfy requirements
kubectl describe nodepool <nodepool-name>

# Check if instance types can handle the workload
kubectl get nodepools -o yaml | grep -A 10 requirements

# Common solutions:
# ✅ Increase NodePool limits (cpu/memory)
# ✅ Add more instance types to requirements
# ✅ Check pod resource requests are reasonable
# ✅ Verify anti-affinity rules aren't too restrictive
```

### Bootstrap Failures
```bash
# Check instance bootstrap logs (requires SSH access)
ssh ubuntu@<node-ip> "sudo journalctl -u cloud-final"
ssh ubuntu@<node-ip> "sudo tail -f /var/log/cloud-init-output.log"

# Check kubelet status on node
ssh ubuntu@<node-ip> "sudo systemctl status kubelet"

# Common issues:
# ❌ Network connectivity → Check security groups allow cluster communication
# ❌ Bootstrap token expired → Check controller logs, automatic renewal should handle this
# ❌ DNS resolution → Verify VPC DNS settings
```

## Next Steps

### Scale Testing
Test your setup with larger workloads:

```bash
# Scale up the test deployment
kubectl scale deployment auto-scaling-test --replicas=10

# Create varying workload sizes
kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mixed-workload
spec:
  replicas: 5
  selector:
    matchLabels:
      app: mixed-workload
  template:
    metadata:
      labels:
        app: mixed-workload
    spec:
      containers:
      - name: variable-consumer
        image: busybox:1.35
        command: ["sleep", "3600"]
        resources:
          requests:
            cpu: "500m"     # Smaller requests
            memory: 1Gi
EOF
```

### Monitoring Setup
Set up monitoring for production use:

```bash
# Check Karpenter metrics
kubectl get --raw /metrics | grep karpenter

# Monitor resource usage
kubectl top nodes
kubectl top pods

# Set up alerts for node provisioning failures
kubectl get events --field-selector type=Warning
```

### Production Hardening
For production deployments:

1. **Resource Limits**: Set appropriate NodePool limits
2. **Security Groups**: Configure restrictive security rules
3. **Monitoring**: Set up alerting for provisioning failures  
4. **Backup**: Document your configuration for disaster recovery
5. **Testing**: Validate auto-scaling behavior under load

### Further Reading

#### Integration-Specific Guides
- **[IKS Integration](./iks-integration.md)** - Complete guide for IBM Kubernetes Service clusters
- **[VPC Integration](./vpc-integration.md)** - Complete guide for self-managed clusters on VPC

#### Advanced Topics
- **[Bootstrap Methods](./bootstrap-methods.md)** - Deep dive into automatic bootstrap configuration
- **[Security Considerations](./security-considerations.md)** - Production security best practices
- **[Limitations](./limitations.md)** - Current constraints and workarounds
- **[Troubleshooting Guide](../TROUBLESHOOTING.md)** - Comprehensive troubleshooting

---

**🚀 Congratulations!** You now have a working Karpenter IBM Cloud Provider setup that automatically provisions and scales nodes based on your workload demands.