# Bootstrap Methods

The Karpenter IBM Cloud Provider provides automatic node bootstrap capabilities to seamlessly join IBM Cloud VPC instances to your Kubernetes cluster. This document explains the available bootstrap methods and their configurations.

## Overview

The provider supports three bootstrap approaches:

1. **Auto Bootstrap** (Default) - Intelligent automatic method selection (Experimental)
2. **VPC Bootstrap** - Direct cloud-init integration for self-managed clusters  
3. **IKS Bootstrap** - Native IBM Kubernetes Service integration (Experimental)

**Key Feature**: **Zero Configuration Required** - The provider automatically detects your cluster configuration and generates appropriate bootstrap scripts with no manual userData needed.

## Auto Bootstrap (Recommended)

### When to Use
- **Default choice** for most deployments
- **Simplified configuration** without manual bootstrap decisions
- **Automatic fallback** between different bootstrap methods

### How It Works
The Auto Bootstrap intelligently selects the best bootstrap method based on your configuration:

1. **IKS Detection**: If `iksClusterID` is provided and accessible → Uses IKS Bootstrap
2. **VPC Fallback**: Otherwise → Uses VPC Bootstrap with automatic cluster discovery

### Configuration
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: auto-bootstrap-nodeclass
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-12345678
  image: r006-ubuntu-20-04
  
  # Auto bootstrap (default - no bootstrapMode needed)
  # Optionally provide IKS cluster ID for IKS preference
  iksClusterID: "cluster-12345678"  # Optional
  
  # No userData required - fully automatic!
```

### Automatic Features
- ✅ **Cluster Discovery**: Automatically detects cluster API endpoint and CA certificate
- ✅ **Token Management**: Generates and refreshes bootstrap tokens automatically  
- ✅ **Network Detection**: Discovers cluster CIDR and DNS configuration
- ✅ **System Configuration**: Enables IP forwarding, disables swap, configures hostname
- ✅ **Runtime Selection**: Auto-detects and configures container runtime (containerd/crio)

## VPC Bootstrap (Cloud-Init)

### When to Use
- **Self-managed Kubernetes clusters** running on IBM Cloud VPC
- **Custom cluster configurations** requiring specific setup
- **Non-IKS environments** where you control the Kubernetes installation

### Configuration
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-bootstrap-nodeclass
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-12345678
  image: r006-ubuntu-20-04
  
  # Explicit VPC bootstrap mode (optional - auto-detected)
  bootstrapMode: vpc
  
  # Optional custom pre-bootstrap setup
  userData: |
    #!/bin/bash
    echo "Custom pre-bootstrap configuration"
    # Bootstrap script automatically appended
```

### Automatic Features

#### **Intelligent Cluster Discovery**
- 🔍 **API Endpoint Detection**: Automatically finds internal cluster API server endpoint
- 🔐 **Certificate Authority**: Extracts cluster CA certificate from existing nodes
- 🌐 **DNS Configuration**: Discovers cluster DNS service IP and domain
- 📡 **Network Setup**: Detects cluster pod and service CIDR ranges

#### **Container Runtime Management**
- 📦 **Containerd** (Default): Installs and configures containerd runtime
- 🏗️ **CRI-O Support**: Alternative container runtime option
- 🔧 **Auto-Detection**: Analyzes existing cluster nodes to match runtime

#### **CNI Plugin Integration**
- 🌐 **Calico**: Full support with automatic configuration
- 🔄 **Cilium**: Advanced networking with eBPF support  
- 🏃 **Flannel**: Lightweight overlay networking
- 🎯 **Auto-Detection**: Matches CNI plugin used by existing cluster nodes

#### **Complete Kubernetes Setup**
- ⚙️ **System Preparation**: Configures system requirements (swap, IP forwarding, hostname)
- 📦 **Package Installation**: Installs kubelet, kubeadm, kubectl with correct versions
- 🔧 **Service Configuration**: Sets up systemd services and startup scripts
- 🏷️ **Node Labeling**: Applies proper Karpenter and workload labels
- 🏃 **Bootstrap Process**: Executes kubeadm join with proper configuration

### Customization Options

#### **Custom User Data**
```yaml
spec:
  userData: |
    #!/bin/bash
    # Your custom pre-bootstrap configuration
    echo "Installing custom packages..."
    apt-get update && apt-get install -y htop vim
    
    # Custom environment variables
    echo "CUSTOM_VAR=value" >> /etc/environment
    
    # Custom service configuration
    systemctl enable my-custom-service
```

#### **Environment Variables**
```bash
# Override container runtime
export CONTAINER_RUNTIME=crio

# Custom CNI configuration
export CNI_PLUGIN=cilium

# Debug mode
export DEBUG=true
```

## 🏢 IKS Bootstrap

### When to Use
- **IBM Kubernetes Service (IKS) clusters** with existing worker pools
- **Enterprise environments** requiring IBM-managed infrastructure
- **Consistent worker pool management** across teams

### Configuration
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: iks-bootstrap-nodeclass
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-iks-12345
  image: r006-ubuntu-20-04
  
  # IKS-specific configuration
  iksClusterID: "cluster-12345678"        # Required: Your IKS cluster ID
  iksWorkerPoolID: "pool-default"         # Optional: specific worker pool
  
  # Optional: Custom post-registration setup
  userData: |
    #!/bin/bash
    echo "Post-IKS registration customization"
    # Custom configuration after node joins IKS cluster
```

### Features

#### **Native IKS Integration**
- 🏢 **Worker Pool API**: Uses IBM Kubernetes Service worker pool resize APIs
- 🔄 **Automatic Registration**: Nodes automatically join IKS cluster through worker pools
- 📊 **Consistent Management**: Maintains consistency with IKS operational patterns

#### **Hybrid API/CLI Approach**
- 🌐 **Primary**: Uses IBM Cloud REST APIs for worker pool operations
- 🖥️ **Fallback**: Automatically switches to IBM Cloud CLI on E3917 API restrictions
- 🔀 **Transparent**: Fallback is automatic and transparent to users

#### **Enterprise Features**
- 🔐 **IBM-Managed Security**: Leverages IKS security policies and compliance
- 📈 **Built-in Monitoring**: Integration with IBM Cloud monitoring and logging
- 🛡️ **Automatic Updates**: Consistent with IKS update and patch management

### Important Constraints

#### **Instance Type Limitations**
- **Constraint**: Cannot dynamically select instance types in IKS mode
- **Reason**: IKS Worker Pool API uses pre-configured instance types
- **Impact**: `instanceProfile` and `instanceRequirements` are ignored
- **Solution**: Pre-create worker pools for different instance types

```yaml
# Example: Multiple NodeClasses for different instance types
---
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: iks-small-instances
spec:
  iksClusterID: "cluster-12345678"
  iksWorkerPoolID: "pool-small"     # Pre-configured with bx2-2x8
---
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: iks-large-instances
spec:
  iksClusterID: "cluster-12345678"  
  iksWorkerPoolID: "pool-large"     # Pre-configured with bx2-8x32
```

### Requirements

#### **IKS Cluster Access**
- ✅ Valid IKS cluster ID in same region as nodes
- ✅ API key with IKS cluster access permissions
- ✅ Worker pools pre-configured with desired instance types

#### **E3917 Error Handling**
The provider automatically handles E3917 "Cluster provider not permitted" errors:

```bash
# Provider logs will show automatic CLI fallback
kubectl logs -n karpenter deployment/karpenter | grep "CLI fallback activated"
```

**🔧 No configuration required** - fallback is automatic when needed.

## Advanced Configuration

### Environment Variables
All bootstrap modes support environment variable customization:

```bash
# Bootstrap behavior
export KARPENTER_LOG_LEVEL=debug         # Enhanced logging
export BOOTSTRAP_TIMEOUT=600             # Bootstrap timeout in seconds

# Container runtime preferences
export CONTAINER_RUNTIME=containerd      # or crio
export CONTAINERD_VERSION=1.7.2          # Specific version

# CNI plugin preferences  
export CNI_PLUGIN=calico                 # or cilium, flannel
export CNI_VERSION=v3.26.0               # Specific CNI version

# System configuration
export ENABLE_IP_FORWARDING=true         # Enable IP forwarding
export DISABLE_SWAP=true                 # Disable swap
export HOSTNAME_STRATEGY=ibm-cloud       # Hostname configuration strategy
```

### Multi-Environment Deployment

**Scenario**: Different bootstrap strategies for different environments

```yaml
# Production: IKS for enterprise compliance
---
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: production-nodes
  labels:
    environment: production
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-prod-12345
  iksClusterID: "prod-cluster-id"
  instanceProfile: mx2-4x32
  securityGroups:
  - sg-prod-strict
  
# Development: VPC for flexibility
---
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: development-nodes
  labels:
    environment: development
spec:
  region: us-south  
  zone: us-south-1
  vpc: vpc-dev-12345
  instanceProfile: bx2-2x8
  securityGroups:
  - sg-dev-permissive
  userData: |
    #!/bin/bash
    # Development-specific tools
    apt-get update && apt-get install -y htop vim git
    echo "ENVIRONMENT=development" >> /etc/environment
```

### Specialized Bootstrap Configurations

#### **GPU Workloads with Custom Drivers**
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: gpu-workload-nodes
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-gpu-12345
  image: r006-ubuntu-20-04
  instanceProfile: gx2-8x64x1v100
  userData: |
    #!/bin/bash
    # Install NVIDIA drivers before bootstrap
    apt-get update
    apt-get install -y nvidia-driver-470 nvidia-container-toolkit
    
    # Configure containerd for GPU support
    mkdir -p /etc/containerd
    cat > /etc/containerd/config.toml <<EOF
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
        BinaryName = "/usr/bin/nvidia-container-runtime"
    EOF
    
    # Bootstrap script automatically appended
```

#### **Security-Hardened Environments**
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: security-hardened-nodes
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-secure-12345
  image: r006-rhel-8  # RHEL for enhanced security
  userData: |
    #!/bin/bash
    # Security hardening before bootstrap
    
    # Configure SELinux
    setenforce 1
    sed -i 's/SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config
    
    # Install security tools
    yum install -y aide tripwire fail2ban
    
    # Configure firewall
    systemctl enable firewalld
    systemctl start firewalld
    
    # Kernel hardening
    echo "kernel.dmesg_restrict = 1" >> /etc/sysctl.conf
    echo "net.ipv4.conf.all.log_martians = 1" >> /etc/sysctl.conf
    
    # Bootstrap process continues with hardened system
```

#### **High-Performance Computing (HPC)**
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: hpc-compute-nodes
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-hpc-12345
  image: r006-ubuntu-20-04
  instanceProfile: cx2-32x64  # High-performance instance
  userData: |
    #!/bin/bash
    # HPC optimizations before bootstrap
    
    # CPU performance tuning
    echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
    
    # Memory optimizations
    echo 'vm.swappiness = 1' >> /etc/sysctl.conf
    echo 'vm.dirty_ratio = 15' >> /etc/sysctl.conf
    
    # Install HPC libraries
    apt-get update && apt-get install -y \
      openmpi-bin openmpi-common libopenmpi-dev \
      intel-mkl-full libblas3 liblapack3
    
    # Network optimizations
    echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
    echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
```

## Troubleshooting Bootstrap Issues

### Common Problems and Solutions

#### **🔍 Bootstrap Script Debugging**
```bash
# Check cloud-init logs on the instance
ssh ubuntu@<instance-ip> "sudo journalctl -u cloud-final"
ssh ubuntu@<instance-ip> "sudo tail -f /var/log/cloud-init-output.log"

# View the generated bootstrap script
ssh ubuntu@<instance-ip> "sudo cat /var/lib/cloud/instance/scripts/*"

# Check bootstrap script execution status
ssh ubuntu@<instance-ip> "sudo systemctl status cloud-final"
```

#### **🔗 Cluster Join Failures**
```bash
# Check kubelet status and logs
ssh ubuntu@<instance-ip> "sudo systemctl status kubelet"
ssh ubuntu@<instance-ip> "sudo journalctl -u kubelet --no-pager -n 50"

# Verify cluster connectivity
ssh ubuntu@<instance-ip> "curl -k https://CLUSTER_ENDPOINT/healthz"

# Check kubeadm join process
ssh ubuntu@<instance-ip> "sudo journalctl | grep kubeadm"
```

#### **🌐 Network Connectivity Issues**
```bash
# Test DNS resolution
ssh ubuntu@<instance-ip> "nslookup kubernetes.default.svc.cluster.local"

# Check if required ports are accessible
ssh ubuntu@<instance-ip> "nc -zv CLUSTER_ENDPOINT 6443"

# Verify security group rules allow cluster communication
ibmcloud is security-group <security-group-id> --output json
```

#### **🔑 Authentication Problems**
```bash
# Check if bootstrap token is valid
kubectl get secrets -n kube-system | grep bootstrap-token

# Verify token hasn't expired
kubectl describe secret <bootstrap-token-secret> -n kube-system

# Check if automatic token renewal is working
kubectl logs -n karpenter deployment/karpenter | grep "token"
```

### Best Practices

#### **Monitoring Bootstrap Success**
```bash
# Watch NodeClaims progress
kubectl get nodeclaims -w

# Monitor bootstrap in real-time
kubectl logs -n karpenter deployment/karpenter -f | grep bootstrap

# Check for successful node registration
kubectl get nodes --watch
```

#### **Debug Mode**
Enable debug logging for detailed bootstrap information:

```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: debug-nodeclass
spec:
  # ... other configuration
  userData: |
    #!/bin/bash
    # Enable debug mode
    export DEBUG=true
    export KARPENTER_LOG_LEVEL=debug
    
    # Your custom configuration
```

## 📚 Related Documentation

- **[Getting Started](./getting-started.md)** - Complete setup walkthrough
- **[Supported CNI/CRI](./supported-cni-cri.md)** - Compatible network and runtime plugins
- **[Security Considerations](./security-considerations.md)** - Security best practices
- **[Limitations](./limitations.md)** - Current constraints and workarounds
- **[Troubleshooting Guide](../TROUBLESHOOTING.md)** - Comprehensive issue resolution

## Key Takeaways

### ✨ **Zero Configuration Required**
- The provider automatically generates bootstrap scripts
- No manual userData needed for basic setups
- Intelligent cluster discovery handles most scenarios

### **Automatic Fallback**
- Auto bootstrap selects the best method for your environment
- IKS mode falls back to CLI when API restrictions occur
- VPC mode works with any self-managed Kubernetes cluster

### **Production Ready**
- Comprehensive error handling and retry logic
- Automatic token management and renewal
- Support for custom configurations when needed

---

**💡 Quick Start**: For most users, simply omit `userData` and let the provider handle bootstrap automatically. Custom configurations are available when you need specialized setups.