# VPC Integration Guide

This guide focuses on using Karpenter IBM Cloud Provider with self-managed Kubernetes clusters running on IBM Cloud VPC infrastructure.

## Overview

VPC integration provides flexible node provisioning for self-managed Kubernetes clusters with full control over cluster configuration and automatic bootstrap capabilities.

### Key Benefits
- **Automatic Bootstrap**: Zero-configuration node joining with intelligent cluster discovery
- **Dynamic Instance Selection**: Full flexibility in instance type selection based on workload requirements
- **Custom Configurations**: Support for specialized setups (GPU, HPC, security hardening)

## Prerequisites

### Infrastructure Requirements
- **Self-Managed Kubernetes**: Running on IBM Cloud VPC instances
- **VPC Infrastructure**: VPC with subnets, security groups, and network configuration
- **API Access**: Service ID with VPC Infrastructure Services permissions
- **Network Connectivity**: Proper security groups allowing cluster communication

### Required Information
Gather the following before starting:
```bash
# List your VPCs
ibmcloud is vpcs --output json

# List subnets in your VPC
ibmcloud is subnets --vpc <vpc-id> --output json

# List security groups
ibmcloud is security-groups --vpc <vpc-id> --output json

# List available images
ibmcloud is images --visibility public --status available | grep ubuntu
```

## Quick Setup

### Step 1: Install Karpenter
```bash
# Create namespace and secrets
kubectl create namespace karpenter

kubectl create secret generic karpenter-ibm-credentials \
  --from-literal=api-key="your-general-api-key" \
  --from-literal=vpc-api-key="your-vpc-api-key" \
  --namespace karpenter

# Install via Helm
helm repo add karpenter-ibm https://pfeifferj.github.io/karpenter-provider-ibm-cloud
helm install karpenter karpenter-ibm/karpenter \
  --namespace karpenter \
  --create-namespace \
  --set controller.env.IBM_REGION="us-south"
```

### Step 2: Create VPC NodeClass
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-nodeclass
  annotations:
    karpenter.ibm.sh/description: "VPC self-managed cluster NodeClass"
spec:
  # REQUIRED: Replace with your actual values
  region: us-south                      # Your IBM Cloud region
  zone: us-south-1                      # Target availability zone
  vpc: vpc-12345678                     # Your VPC ID
  image: r006-12345678                  # Ubuntu 20.04 LTS recommended
  
  # Security and networking
  securityGroups:
  - sg-k8s-workers                      # Security group allowing cluster communication
  
  # Optional: Specific subnet (auto-selected if not specified)
  subnet: subnet-12345678               # Your subnet ID
  
  # Optional: SSH access for troubleshooting
  sshKeys:
  - key-admin-access
  
  # Optional: Resource group
  resourceGroup:
    name: "kubernetes-resources"
  
  # VPC mode uses automatic bootstrap - no userData required!
```

### Step 3: Create NodePool
```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: vpc-nodepool
spec:
  template:
    metadata:
      labels:
        provisioner: karpenter-vpc
        cluster-type: self-managed
    spec:
      nodeClassRef:
        apiVersion: karpenter.ibm.sh/v1alpha1
        kind: IBMNodeClass
        name: vpc-nodeclass
      
      # Full flexibility in instance requirements
      requirements:
      - key: node.kubernetes.io/instance-type
        operator: In
        values: ["bx2-2x8", "bx2-4x16", "cx2-2x4", "cx2-4x8", "mx2-2x16"]
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
  
  limits:
    cpu: 1000
    memory: 1000Gi
  
  disruption:
    consolidationPolicy: WhenEmpty
    consolidateAfter: 30s
```

## VPC Bootstrap Features

### Automatic Cluster Discovery
The VPC integration automatically discovers your cluster configuration:

- **API Endpoint**: Finds internal cluster API server endpoint for secure communication
- **CA Certificate**: Extracts cluster CA certificate from existing nodes
- **DNS Configuration**: Discovers cluster DNS service IP and search domains
- **Network Settings**: Detects cluster pod and service CIDR ranges
- **Runtime Detection**: Matches container runtime used by existing nodes

### Zero Configuration Bootstrap
```yaml
# Minimal configuration - everything else is automatic
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: minimal-vpc
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-12345678
  image: r006-ubuntu-20-04
  securityGroups:
  - sg-default
  # No userData needed - bootstrap is fully automatic!
```

## Advanced VPC Configurations

### Multi-Zone VPC Setup
```yaml
# Zone 1
---
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-us-south-1
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-12345678
  subnet: subnet-zone1-12345
  image: r006-ubuntu-20-04
  securityGroups:
  - sg-k8s-workers
---
# Zone 2
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-us-south-2
spec:
  region: us-south
  zone: us-south-2
  vpc: vpc-12345678
  subnet: subnet-zone2-12345
  image: r006-ubuntu-20-04
  securityGroups:
  - sg-k8s-workers
```

### GPU Workloads
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-gpu
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-gpu-12345
  image: r006-ubuntu-20-04
  instanceProfile: gx2-8x64x1v100      # GPU instance type
  securityGroups:
  - sg-gpu-workloads
  userData: |
    #!/bin/bash
    # GPU drivers and configuration
    apt-get update
    apt-get install -y nvidia-driver-470 nvidia-container-toolkit
    
    # Configure containerd for GPU support
    mkdir -p /etc/containerd
    cat > /etc/containerd/config.toml <<EOF
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
        BinaryName = "/usr/bin/nvidia-container-runtime"
    EOF
    
    # Bootstrap script automatically appended
```

### High-Performance Computing (HPC)
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-hpc
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-hpc-12345
  image: r006-ubuntu-20-04
  instanceProfile: cx2-32x64           # High-performance instance
  securityGroups:
  - sg-hpc-cluster
  userData: |
    #!/bin/bash
    # HPC optimizations
    echo performance > /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
    
    # Memory optimizations
    echo 'vm.swappiness = 1' >> /etc/sysctl.conf
    echo 'vm.dirty_ratio = 15' >> /etc/sysctl.conf
    
    # Install HPC libraries
    apt-get update && apt-get install -y \
      openmpi-bin openmpi-common libopenmpi-dev \
      libblas3 liblapack3
    
    # Network optimizations for high-throughput
    echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
    echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
```

### Custom CNI Configuration
```yaml
apiVersion: karpenter.ibm.sh/v1alpha1
kind: IBMNodeClass
metadata:
  name: vpc-custom-cni
spec:
  region: us-south
  zone: us-south-1
  vpc: vpc-custom-12345
  image: r006-ubuntu-20-04
  userData: |
    #!/bin/bash
    # Custom CNI setup before cluster join
    
    # Install Cilium CNI
    curl -L -o /opt/cni/bin/cilium-cni \
      https://github.com/cilium/cilium/releases/download/v1.14.0/cilium-linux-amd64.tar.gz
    
    # Custom CNI configuration
    mkdir -p /etc/cni/net.d
    cat > /etc/cni/net.d/05-cilium.conf <<EOF
    {
      "cniVersion": "0.4.0",
      "name": "cilium",
      "type": "cilium-cni",
      "enable-debug": false
    }
    EOF
    
    # Bootstrap script handles the rest
```

## Dynamic Instance Selection

Unlike IKS mode, VPC integration provides full flexibility in instance type selection:

```yaml
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: flexible-nodepool
spec:
  template:
    spec:
      nodeClassRef:
        name: vpc-nodeclass
      requirements:
      # Karpenter will choose the best instance type based on pod requirements
      - key: node.kubernetes.io/instance-type
        operator: In
        values: [
          "bx2-2x8", "bx2-4x16", "bx2-8x32",    # Balanced instances
          "cx2-2x4", "cx2-4x8", "cx2-8x16",     # Compute optimized
          "mx2-2x16", "mx2-4x32", "mx2-8x64"    # Memory optimized
        ]
      - key: karpenter.ibm.sh/instance-family
        operator: In
        values: ["bx2", "cx2", "mx2"]
      # Resource-based selection
      - key: kubernetes.io/arch
        operator: In
        values: ["amd64"]
      - key: karpenter.sh/capacity-type
        operator: In
        values: ["on-demand"]
```

## VPC-Specific Troubleshooting

### Bootstrap Issues

#### Cluster Discovery Failures
```bash
# Check if controller can reach cluster API
kubectl logs -n karpenter deployment/karpenter | grep "cluster discovery"

# Verify internal API endpoint is accessible
ssh ubuntu@<node-ip> "curl -k https://INTERNAL_API_ENDPOINT/healthz"

# Check security group rules
ibmcloud is security-group <sg-id> --output json | jq '.rules'
```

#### Bootstrap Script Problems
```bash
# View generated bootstrap script
ssh ubuntu@<node-ip> "sudo cat /var/lib/cloud/instance/scripts/*"

# Check cloud-init logs
ssh ubuntu@<node-ip> "sudo journalctl -u cloud-final"

# Monitor bootstrap execution
ssh ubuntu@<node-ip> "sudo tail -f /var/log/cloud-init-output.log"
```

### Network Connectivity
```bash
# Test cluster communication
ssh ubuntu@<node-ip> "nc -zv CLUSTER_ENDPOINT 6443"

# Check DNS resolution
ssh ubuntu@<node-ip> "nslookup kubernetes.default.svc.cluster.local"

# Verify route table
ibmcloud is vpc-routes <vpc-id>
```

### Instance Provisioning
```bash
# Check available instances in zone
ibmcloud is instance-profiles --output json | jq '.[] | select(.family=="bx2")'

# Monitor quota usage
ibmcloud is instances --output json | jq 'length'

# Check subnet capacity
ibmcloud is subnet <subnet-id> --output json | jq '.available_ipv4_address_count'
```