# OpenShift Virtualization Low-Level Design

## 1. Introduction

This Low-Level Design (LLD) document provides detailed technical specifications for implementing OpenShift Virtualization as a replacement for the existing OpenStack environment. It includes specific configurations, component interactions, and implementation details with references to official Red Hat documentation.

This document focuses on the overall OpenShift platform architecture and x86_64 implementation. Detailed specifications for LinuxOne (s390x) worker node integration are documented separately in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

For a comprehensive understanding of the implementation process, see [Section 5: Installation and Bootstrapping Process](#5-installation-and-bootstrapping-process). For ongoing management details, refer to [Section 8: Day 2 Operations](#8-day-2-operations).

## 1.1 Assumptions and Constraints

### 1.1.1 Assumptions

1. **Infrastructure Assumptions**:
   - Existing FC-SAN infrastructure is available and properly configured
   - Network infrastructure supports VLAN tagging and LACP bonding
   - All hardware meets minimum requirements specified in Section 2.1
   - Red Hat Enterprise Linux 8 or later is available for the jumphost
   - Red Hat OpenShift Container Platform 4.16 is the target version

#### 1.1.1.1 Personnel Skill and Training Assumptions

Personnel responsible for the OpenShift cluster implementation and operations must possess or acquire proficiency in multiple technical domains. The required skills encompass:

**Essential Competencies:**
- **Linux System Administration**: RHCSA-level proficiency in system administration, command-line operations, and troubleshooting
- **OpenShift Platform Administration**: Comprehensive knowledge of OpenShift architecture, installation, configuration, and day-to-day operations
- **Storage Administration**: OpenShift Data Foundation (ODF) and Ceph administration for storage management
- **Automation**: Ansible proficiency for infrastructure automation and configuration management
- **Specialized Platform Components**: OpenShift Virtualization (future implementation) and s390x/LinuxONE integration

**Key Training Requirements:**
- **Primary Courses**: DO280 (OpenShift Administration), DO370 (ODF Administration), RH294 (Ansible Automation)
- **Foundation Courses**: RH124/134 (Linux Administration), DO180 (Container Fundamentals)
- **Specialized Courses**: DO316 (OpenShift Virtualization), LX0103 (LinuxONE), DO425 (Security)

**Role-Specific Training Duration:**
- Platform Administrators: 12-16 days
- Storage Administrators: 16-20 days  
- Automation Engineers: 12-16 days

> **Detailed Training Curriculum**: Comprehensive training requirements, course descriptions, certification paths, and implementation strategies are documented in [Appendix A: Personnel Training Requirements](#appendix-a-personnel-training-requirements).

1. **Operational Assumptions**:
   - Operations team has basic Kubernetes/OpenShift knowledge
   - Network team can provide required VLAN configurations
   - Storage team can provision FC-SAN LUNs
   - Sufficient IP address space is available for all networks
   - DNS services are available for cluster name resolution

2. **Security Assumptions**:
   - Hardware supports AMD Secure Execution (SEV)
   - BIOS/firmware can be updated to required versions
   - Network security policies allow required traffic
   - Red Hat subscriptions are available for all components

### 1.1.2 Constraints

1. **Technical Constraints**:
   - Must maintain compatibility with existing FC-SAN infrastructure
   - Network changes must be non-disruptive to existing services
   - Storage migration must be performed without data loss
   - Must support both x86_64 and s390x architectures
   - Must maintain existing security compliance requirements

2. **Operational Constraints**:
   - Limited maintenance windows for infrastructure changes
   - Must maintain existing backup and recovery procedures
   - Must support existing monitoring and alerting systems
   - Must maintain existing disaster recovery capabilities
   - Must support existing change management processes

3. **Resource Constraints**:
   - Limited number of available IP addresses
   - Limited FC-SAN storage capacity
   - Limited network bandwidth for migration
   - Limited number of available VLANs
   - Limited number of available physical servers

## 2. Infrastructure Specifications

> **Note**: Cluster-specific and environment-specific details such as VLAN numbers, IP ranges, and host specifications are documented in environment-specific documents maintained in the customer's Confluence. These documents are agreed to be kept up-to-date by the customer to reflect the current production environment.
> 
> **Environment-Specific Documentation:**
> - **Test01**: [test01.kvant.cloud Environment](https://phoenixsystems.atlassian.net/wiki/spaces/RedKvant/pages/3424059393/test01.kvant.cloud+Environment)
> - **Test02**: [test02.kvant.cloud Environment](https://phoenixsystems.atlassian.net/wiki/spaces/RedKvant/pages/3395977220/test02.kvant.cloud+Environment)
> - **Basel**: [Basel Environment](https://phoenixsystems.atlassian.net/wiki/spaces/RedKvant/pages/3666968577/Basel)
> - **Zürich**: [Zurich - Green Lupfig Environment](https://phoenixsystems.atlassian.net/wiki/spaces/RedKvant/pages/4188471332/Zurich+-+Green+Lupfig)

### 2.1 Hardware Requirements

> **Note**: The requirements listed below represent **Red Hat's official minimum specifications** for OpenShift Container Platform, OpenShift Virtualization, and OpenShift Data Foundation. **Actual production sizing** depends heavily on specific use cases, workload characteristics, and performance requirements that vary by environment.
>
> **Environment-Specific Implementation**: Specific hardware models, configurations, and sizing for each environment are documented in environment-specific Confluence pages (referenced above) and are maintained by Phoenix Systems to reflect current production deployments.

#### 2.1.1 Control Plane Nodes (Masters)

| Component | Red Hat Minimum Requirements | Production Considerations |
|-----------|------------------------------|---------------------------|
| CPUs | 4 vCPUs or cores | Actual sizing depends on cluster scale and workload characteristics |
| Memory | 16 GB RAM | Additional memory required for larger clusters (500+ nodes) |
| Storage | 100 GB for `/var` | SSD recommended for etcd performance |
| Network | 1 Gbps NIC | Redundant 10+ Gbps NICs recommended for production |

**Reference**: [OpenShift Control Plane Requirements](https://docs.openshift.com/container-platform/4.16/installing/installing_bare_metal/installing-bare-metal.html#installation-requirements-user-infra_installing-bare-metal)

> **Implementation**: Production deployments use dual 100 Gbps NICs with LACP bonding for redundancy and performance. See Section 2.2.1 for network configuration details.

#### 2.1.2 Worker Nodes (x86_64)

| Component | Red Hat Minimum Requirements | OpenShift Virtualization Requirements |
|-----------|------------------------------|--------------------------------------|
| CPUs | 2 vCPUs or cores | 8+ cores with nested virtualization support |
| Memory | 8 GB RAM | 32+ GB RAM (varies significantly by VM workloads) |
| Storage | 100 GB for `/var` | Additional storage for VM disk images |
| Network | 1 Gbps NIC | Redundant 10+ Gbps NICs for VM live migration |

**References**: 
- [OpenShift Worker Node Requirements](https://docs.openshift.com/container-platform/4.16/installing/installing_bare_metal/installing-bare-metal.html#installation-requirements-user-infra_installing-bare-metal)
- [OpenShift Virtualization Hardware Requirements](https://docs.openshift.com/container-platform/4.16/virt/install/preparing-cluster-for-virt.html#virt-hardware-requirements_preparing-cluster-for-virt)

> **Production Sizing**: Worker node sizing is **highly dependent on use cases**:
> - **CPU**: Varies by VM density and CPU overcommit ratios
> - **Memory**: Depends on VM memory allocations and host OS overhead
> - **Storage**: Varies by persistent volume requirements and VM disk sizes
> - **Network**: Live migration and storage traffic requirements drive bandwidth needs

#### 2.1.3 LinuxOne Nodes (s390x)

LinuxOne (s390x) worker nodes have specific hardware requirements and LPAR configuration considerations that differ significantly from x86_64 nodes. These nodes are added as Day 2 operations after the initial cluster is operational.

> **Detailed Specifications**: Complete hardware requirements, LPAR configuration details, and implementation specifications for s390x nodes are documented in the dedicated [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

**Reference**: [RHCOS on IBM Z and LinuxONE](https://docs.openshift.com/container-platform/4.16/installing/installing_ibm_z/installing-ibm-z.html)

#### 2.1.4 Storage Requirements (OpenShift Data Foundation)

| Component | Minimum Requirements | Production Considerations |
|-----------|----------------------|---------------------------|
| **Initial Setup** | 3 nodes minimum for Ceph | Using worker nodes with FC-SAN backend |
| **Storage per Node** | 500 GB minimum per OSD | Production sizing depends on data volume and performance requirements |
| **OSDs per Node** | 1 minimum, 2+ recommended | Initial deployment: 2 OSDs per node from FC-SAN LUNs |
| **Network** | 1 Gbps for storage traffic | Dedicated storage network recommended for production, >25Gbps for production |

**Reference**: [OpenShift Data Foundation Planning](https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/index)

> **Migration Strategy**: Initial deployment uses worker nodes with FC-SAN storage. Future migration to dedicated storage nodes allows for better resource allocation and performance optimization. See [Section 2.3.2](#232-migration-strategy-to-dedicated-hardware) for details.

#### 2.1.5 Jumphost (Bastion/Management Node)

| Component | Minimum Requirements | Production Recommendations |
|-----------|----------------------|----------------------------|
| CPUs | 4 cores | Additional cores for complex automation tasks |
| Memory | 16 GB RAM | 32+ GB for multiple concurrent operations |
| Storage | 100 GB | SSD for better I/O performance |
| Network | 1 Gbps connection | Redundant connections to management networks |
| OS | RHEL 8+ | Latest supported RHEL version |

> **Purpose**: Hosts Ansible automation, `oc` CLI tools, installation artifacts, and serves as the operational control center for cluster management.

### 2.1.6 Hardware Sizing Summary

> **Important**: The above specifications represent **minimum requirements only**. Production deployments require comprehensive capacity planning based on:
>
> - **Workload Characteristics**: VM sizes, CPU/memory requirements, I/O patterns
> - **Performance Requirements**: Response times, throughput, availability SLAs  
> - **Growth Projections**: Expected scaling over time
> - **Security Requirements**: Isolation needs, compliance mandates
> - **Operational Requirements**: Backup, monitoring, maintenance windows
>
> **Environment-Specific Sizing**: Each deployment environment has unique requirements documented in environment-specific Confluence pages. These documents contain:
> - Actual hardware models and specifications used
> - Environment-specific network configurations  
> - Performance benchmarks and sizing decisions
> - Lessons learned and optimization notes

### 2.2 Networking Details

> **Note**: See [Network Design](img/Network%20Design.drawio) for a detailed view of the network architecture, including VLANs and network services.
<&nbsp;/>
> **Note**: Specific networking details such as VLAN IDs, IP address ranges, and network segmentation specifics are documented in the environment-specific Confluence documentation referenced above and are kept up-to-date by the customer.

#### 2.2.0 Customer Network Infrastructure Requirements

This implementation requires specific network connectivity from the customer's infrastructure. The following specifications define the **minimum network interface requirements** that must be provided by the customer's network team.

**Server Network Interface Requirements:**
- **Physical Connectivity**: 2x 10Gbps ports per server configured as LACP port-channel
- **VLAN Transport**: Ability to transport multiple VLANs over the port-channel
- **Required VLANs**: Customer must provide the following VLAN services:
  - TODO: Add VLANs

**Network Service Dependencies:**
- **Layer 2**: VLAN tagging and LACP support on customer switches
- **Layer 3**: Customer-provided gateway services for external connectivity
- **DNS**: Customer-provided DNS resolution for cluster FQDNs
- **Time Services**: NTP server accessibility for time synchronization

> **Important**: All other network infrastructure details (switching, routing, security policies) remain under Phoenix Systems control and are documented in environment-specific Confluence pages. This design ensures maximum flexibility while clearly defining the interface requirements.

#### 2.2.1 Red Hat OpenShift Network Prerequisites

The following network prerequisites are required by Red Hat OpenShift Container Platform as documented in the official installation guides:

**Red Hat Documented Network Requirements:**
- **Connectivity**: All cluster nodes must have network connectivity to each other
- **DNS Resolution**: Forward and reverse DNS resolution for all cluster nodes and services
- **Time Synchronization**: All nodes must maintain synchronized time via NTP
- **Port Requirements**: Specific ports must be accessible as per [Red Hat Port Requirements](https://docs.openshift.com/container-platform/4.16/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal)

**Customer Network Infrastructure Checklist:**
*The following items align with Red Hat's documented prerequisites:*

- [ ] LACP bonding support on customer switches
- [ ] VLAN tagging capability across required VLANs  
- [ ] DNS resolution for cluster FQDNs
- [ ] NTP server accessibility
- [ ] Required ports accessible per Red Hat documentation
- [ ] Sufficient bandwidth for cluster operations

> **Reference**: [OpenShift Bare Metal Network Requirements](https://docs.openshift.com/container-platform/4.16/installing/installing_bare_metal/installing-bare-metal.html#installation-network-user-infra_installing-bare-metal)

**Note**: These prerequisites are established by Red Hat for successful OpenShift deployment. Environment-specific implementation details are documented in customer-maintained Confluence pages.

#### 2.2.2 Network Infrastructure Assumptions

This implementation operates under specific assumptions about the customer's network infrastructure capabilities. These assumptions are based on typical enterprise network deployments and Red Hat OpenShift requirements.

**Core Network Assumptions:**
- Customer network infrastructure supports enterprise-grade switching with LACP
- VLAN segmentation capabilities are available and configured
- Customer maintains DNS and NTP services accessible to the cluster
- Network security policies allow required OpenShift traffic flows

**Infrastructure Service Assumptions:**
- [TO BE DETAILED] - Routing and gateway services
- [TO BE DETAILED] - Firewall and security policy management  
- [TO BE DETAILED] - Load balancing and external connectivity
- [TO BE DETAILED] - Monitoring and network management integration

> **Future Enhancement**: This section will be expanded with detailed assumptions as implementation experience is gained across different customer environments.

#### 2.2.3 Physical Network Configuration

The network architecture will implement redundancy through dual 100Gbit NICs configured in active-active bonding using LACP for x86_64 nodes. This configuration provides both high availability in case of link failure and increased bandwidth through link aggregation. For LinuxOne (s390x) nodes, network configuration will leverage native s390x capabilities.

> **Rationale**: The active-active LACP bonding configuration was chosen for x86_64 nodes to provide both redundancy and increased bandwidth. This is particularly important for virtual machine live migration on these nodes. s390x nodes utilize their own robust networking features suitable for mainframe environments. See [Section 3.2](#32-livemigration-network-considerations) for live migration network considerations (applicable to x86_64).

Unlike traditional network configuration via network-scripts, the network setup for OpenShift x86_64 nodes will be configured using:

1. **MachineConfig**: For base OS-level network configuration on RHCOS-based x86_64 nodes.
2. **NMState Operator**: For declarative network state management on x86_64 nodes.
3. **MachineSet configurations**: For x86_64 node-specific network setup if using Machine API for scaling.

For LinuxOne (s390x) nodes, network configuration is typically managed at the RHEL OS level using standard Linux networking tools or s390x-specific configuration utilities, applied during their Day 2 addition process.

Networking must be configured through NMState resources for x86_64 nodes, which provides a declarative way to manage the host network settings in a Kubernetes-native manner. This follows the OpenShift best practice for network configuration management on these nodes.

The network configuration is managed through the `nmstate_config` role, which applies network configurations using Kubernetes-native resources. The role ensures that network settings are applied consistently across all nodes and can be managed through GitOps practices.

The networking components will include:

- Bond configuration across the dual NICs
- VLAN tagging for network segmentation
- Bridge setup for VM connectivity

Reference: [OpenShift Networking with NMState](https://docs.openshift.com/container-platform/4.16/networking/k8s_nmstate/k8s-nmstate-updating-node-network-config.html)

#### 2.2.4 VLAN Configuration

The following VLANs will be configured on top of the bonded interfaces. Specific VLAN IDs and IP ranges are documented in the environment-specific Confluence documentation:

1. **Provider-Int VLAN** 
   - Purpose: Internal provider network for cluster services
   - IP Range: Environment-specific (from RFC1918 space)
   - Traffic types: Internal cluster communications, service traffic

2. **Provider-Ext VLAN** 
   - Purpose: External provider network for external connectivity
   - IP Range: Environment-specific (from RFC1918 space)
   - Traffic types: External services access, load balancer VIP traffic

3. **Storage Network VLAN**
   - Purpose: Dedicated network for storage traffic (ODF/Ceph)
   - IP Range: Environment-specific (from RFC1918 space)
   - Traffic types: Ceph cluster communication, storage replication

4. **PXE Boot VLAN**
   - Purpose: Network for PXE booting during installation
   - IP Range: Environment-specific (from RFC1918 space)
   - Traffic types: DHCP, TFTP, PXE boot traffic

5. **Additional Environment-Specific VLANs**
   - Purpose: As defined per environment requirements
   - Details: Refer to environment-specific Confluence documentation

The VLAN configuration is managed through the `nmstate_config` role, which applies network configurations using Kubernetes-native resources. This approach enables declarative configuration and reconciliation of network states across all nodes.

The network configuration will be applied through a task like this:

1. **Node Network Configuration**:

   ```ansible
   # From nmstate_config role
   - name: Deploy storage network on cluster nodes
     delegate_to: localhost
     kubernetes.core.k8s:
       kubeconfig: "{{ ocp_cluster_auth }}/kubeconfig"
       apply: true
       state: present
       template:
         - path: "nncp.yaml.j2"
   ```

   > **Example from**: [roles/odf_config/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/odf_config/tasks/main.yaml)

2. **Network Configuration Template**:
   The network configuration is defined in templates that are applied to the nodes. These templates can be customized per environment and node type.

3. **Network State Reconciliation**:
   The NMState operator ensures that the actual network state matches the desired state defined in the configuration. Any deviations are automatically corrected.

This approach ensures that network configuration is declarative, consistent, and managed through Kubernetes resources rather than directly manipulating configuration files on nodes.

```yaml
# Network configuration using NMState
interfaces:
  - name: bond0
    type: bond
    state: up
    ipv4:
      address:
      - ip: "10.153.11.109"
        prefix-length: "24"
      enabled: true
    link-aggregation:
        options:
          miimon: '100'
        mode: 802.3ad
        ports:
            - ens9f0
            - ens9f1
            - ens9f2
            - ens9f3
  - name: pxe
    type: vlan
    state: up
    ipv4:
      address:
      - ip: "10.153.15.109"
        prefix-length: "24"
      enabled: true
    vlan:
      base-iface: "bond0"
      id: "1186"
```
> **Example from**: [roles/nmstate_config/templates/nncp.yaml.j2](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/nmstate_config/tasks/main.yaml)

#### 2.2.5 Load Balancer Configuration

The implementation uses the MetalLB operator for load balancing services in a bare metal environment. MetalLB provides network load balancing capabilities without requiring external hardware load balancers, allowing the assignment of external IPs to Kubernetes services.

The deployment will use Layer 2 mode configuration for MetalLB with service IP allocation from designated address pools. Load balancer functionality is thoroughly tested as part of the deployment validation process.

#### 2.2.6 OVN CNI Configuration

OVN (Open Virtual Network) will be configured as the default CNI plugin during the initial cluster installation. OVN provides advanced networking features required for both x86_64 and s390x workloads, including:

- Network policy enforcement
- Distributed L2/L3 networking
- NAT and floating IP support
- Integration with external networks

The OVN configuration will be specified in the OpenShift installation manifest:

```yaml
# OVN configuration in install-config.yaml
networking:
  machineNetwork:
  - cidr: "10.153.11.0/24"
  networkType: "OVNKubernetes"
```
> **Example from**: [roles/install_ocp/templates/install-config.yaml.j2](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/install_ocp/templates/install-config.yaml.j2)

During the bootstrapping phase, network configuration is particularly challenging since it must be established before the cluster is fully operational. The approach will involve:

1. Configuring jumphost networking first
2. Preparing the OCP installation environment
3. Installing OCP prerequisites
4. Configuring NMState resources post-installation

Reference: [OVN-Kubernetes Configuration](https://docs.openshift.com/container-platform/4.16/networking/ovn_kubernetes_network_provider/about-ovn-kubernetes.html)

### 2.3 Storage Configuration

> **Note**: See [ODF Architecture Overview](img/odf-architecture-overview.drawio) for a visual representation of the storage components and their relationships.

#### 2.3.1 Initial ODF Configuration with FC-SAN Backend

OpenShift Data Foundation will be deployed with Rook/Ceph using FC-SAN as the backing store, primarily to support persistent storage requirements, including those managed by OpenShift Virtualization. This approach involves several phases:

> **Rationale**: FC-SAN was chosen as the initial storage backend to leverage existing infrastructure and provide high-performance storage for virtual machines. The migration to dedicated hardware is planned to improve performance and reduce complexity. See [Section 2.3.2](#232-migration-strategy-to-dedicated-hardware) for the migration strategy.

1. **Installation of ODF Operator**:
   - Deploy via OperatorHub
   - Configure the operator with appropriate permissions
   - Create necessary namespace and operator group

        ```ansible
        # ODF Operator installation
        - name: Bootstrap ODF
        hosts: jumphost
        roles:
            - role: odf_config
            tags: odf
        ```

        > **Example from**: [playbooks/10_bootstrap_ocp.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/10_bootstrap_ocp.yaml)

2. **Configuration for FC-SAN Integration**:
   - ODF will be configured to use existing FC-SAN volumes as backing storage
   - Storage is implemented with explicit static LUN mapping on the FC-SAN
   - LUN pre-provisioning is automated using the IBM Storage Virtualize module:

        ```ansible
        # LUN provisioning from add_luns role
        - name: Create the disks
          delegate_to: localhost
          ibm.storage_virtualize.ibm_svc_manage_volume:
            clustername: "{{ flash_system.management_address }}"
            name: "redhat_{{ inventory_hostname }}_odf_ceph_osd_{{ item }}"
            state: present
            size: "{{ odf_flash_disks.disk_size }}"
            unit: "{{ odf_flash_disks.disk_size_unit }}"
            pool: 2
          loop: "{{ range(0, odf_flash_disks.disk_count) }}"

        - name: Attach the disks
          delegate_to: localhost
          ibm.storage_virtualize.ibm_svc_vol_map:
            clustername: "{{ flash_system.management_address }}"
            volname: "redhat_{{ inventory_hostname }}_odf_ceph_osd_{{ item }}"
            state: present
            host: "{{ flash_system_host_name | default(inventory_hostname | upper) }}"
          loop: "{{ range(0, odf_flash_disks.disk_count) }}"
        ```

        > **Example from**: [roles/add_luns/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/add_luns/tasks/main.yaml)

3. **Storage Cluster Creation**:
   - StorageCluster custom resource will define the ODF configuration
   - Initial deployment will use worker nodes for Ceph services
   - OSD (Object Storage Daemon) pods will use FC-SAN volumes
   - Node labeling for ODF nodes is automated:

        ```ansible
        # Node labeling from ocp_label_nodes role
        - name: Add labels to the node
          delegate_to: localhost
          ansible.builtin.command: "oc --kubeconfig {{ ocp_cluster_auth }}/kubeconfig label node {{ k8s_node_name | default('worker-'+inventory_hostname) }} {{ item }}"
          loop: "{{ labels }}"
        ```

        > **Example from**: [roles/ocp_label_nodes/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/ocp_label_nodes/tasks/main.yaml)

        ```ini
        …
        # Storage configuration in inventory
        [ceph_nodes]
        master-0
        master-1
        master-2
        …
        ```

        > **Example from**: [inventory/*/hosts](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/inventory)

   - StorageCluster configuration uses templates for flexibility:

        ```yaml
        # StorageCluster configuration from storagecluster.yaml.j2
        apiVersion: ocs.openshift.io/v1
        kind: StorageCluster
        metadata:
          name: ocs-storagecluster
          namespace: openshift-storage
        spec:
          arbiter: {}
          enableCephTools: true
          encryption:
            keyRotation:
              schedule: '@weekly'
          flexibleScaling: true
          managedResources:
            cephBlockPools: {}
            cephCluster: {}
            cephConfig: {}
            cephDashboard: {}
            cephFilesystems:
              dataPoolSpec:
                application: ""
                erasureCoded:
                  codingChunks: 0
                  dataChunks: 0
                mirroring: {}
                quotas: {}
                replicated:
                  size: 0
          monDataDirHostPath: /var/lib/rook
          resourceProfile: balanced
          storageDeviceSets:
          - config: {}
            count: {{ odf_osd_count }}
            dataPVCTemplate:
              spec:
                accessModes:
                - ReadWriteOnce
                resources:
                  requests:
                    storage: "1"
                storageClassName: ceph-osd-local-storage
                volumeMode: Block
            name: ocs-deviceset-local
            replica: 1
        ```

        > **Example from**: [roles/odf_config/templates/storagecluster.yaml.j2](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/odf_config/templates/storagecluster.yaml.j2)

The storage implementation considers several factors:

- Performance requirements for VM workloads
- Resilience needs for high availability
- Transition strategy to dedicated storage nodes
- Integration with existing FC-SAN infrastructure

Storage functionality must be thoroughly tested as part of the deployment process through validation of storage classes and ODF resources.

Reference: [ODF Installation Documentation](https://docs.openshift.com/container-platform/4.16/storage/persistent_storage/persistent-storage-ocs.html)

#### 2.3.2 Migration Strategy to Dedicated Hardware

The migration to dedicated storage hardware will involve a staged approach to ensure data integrity and service continuity:

1. **Preparation of Dedicated Storage Nodes**:
   - Commission new hardware nodes specifically for storage
   - Ensure network connectivity and FC-SAN access if needed
   - Label nodes appropriately for ODF storage usage

2. **Expansion of the Storage Cluster**:
   - Add new OSDs to the storage cluster using the dedicated nodes
   - Configure placement rules to prefer the new nodes
   - Allow data rebalancing to occur gradually

        ```ansible
        # Example of node management for storage expansion
        - name: Add node
        hosts: jumphost
        roles:
            - ocp_add_node
        ```

        > **Example from**: [playbooks/11_add_node.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/11_add_node.yaml)

3. **Data Migration and Decommissioning**:
   - Monitor rebalancing progress until data is distributed to new nodes
   - Gradually remove old OSDs from worker nodes
   - Ensure data integrity throughout the process

        ```ansible
        # ODF removal playbook
        - name: Remove ODF
        hosts: jumphost
        roles:
            - role: odf_config
            state: absent
        ```

        > **Example from**: [playbooks/12_delete_node.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/12_delete_node.yaml)

4. **Verification and Cleanup**:
   - Verify performance and availability after migration
   - Release FC-SAN resources that are no longer needed
   - Update storage class configurations if necessary

This migration approach allows for a controlled transition without service disruption. By incrementally adding new storage and gradually removing old storage, data is safely migrated while maintaining availability.

The implementation must include procedures for removing ODF components if needed, which can be adapted for the migration process.

Reference: [ODF Storage Device Management](https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.16/html/managing_and_allocating_storage_resources/managing-persistent-volume-claims_rosa)

## 3. OpenShift Virtualization Configuration

### 3.1 OpenShift Virtualization Design Considerations

> **Remark**: OpenShift Virtualization does **currently** not support multiple Architectures in the same cluster. This is a known limitation and will be addressed in future releases. Therefore it has been designed to run exclusively on x86_64 architecture.
<&nbsp;/>
> **Rationale**: The design separates the infrastructure foundation from the OpenShift Virtualization implementation to allow for a phased approach. This separation of concerns enables the customer to implement OpenShift Virtualization according to their specific requirements while ensuring the x86_64 infrastructure meets all prerequisites. OpenShift Virtualization will currently not be deployed or supported on s390x worker nodes.

OpenShift Virtualization extends Kubernetes functionality to support virtual machines alongside containers **on x86_64 architecture only**. While the installation and configuration of OpenShift Virtualization will be performed by the customer on the x86_64 worker nodes, this project establishes the foundation to support this use case on that specific architecture. 

LinuxOne (s390x) nodes will focus on running secure containerized workloads natively. For detailed information on s390x workload design and security considerations, see the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

The architecture is designed to provide:

- Sufficient resources for virtualized workloads
- Network configurations suitable for VM traffic patterns
- Storage performance required for virtualized applications
- Security capabilities for VM isolation

The customer will be responsible for:

- Installing the OpenShift Virtualization Operator
- Configuring the HyperConverged custom resource
- Setting up VM templates and policies
- Managing the VM lifecycle

Reference: [OpenShift Virtualization Overview](https://docs.openshift.com/container-platform/4.16/virt/about-virt.html)

### 3.2 LiveMigration Network Considerations

The infrastructure includes a dedicated LiveMigration network to support VM migration between x86_64 hosts. This provides several benefits:

- Separates migration traffic from other cluster traffic
- Provides dedicated bandwidth for memory state transfer
- Reduces impact on application performance during migrations

The customer will leverage this network when configuring OpenShift Virtualization's live migration capabilities. This specialized network is configured as part of the VLAN setup described in [Section 2.2.4](#224-vlan-configuration).

## 4. Security Implementation

### 4.1 AMD Secure Execution Considerations

> **Rationale**: AMD Secure Execution provides hardware-based memory encryption, which is essential for protecting virtual machines in a multi-tenant environment. The implementation requires specific hardware and BIOS configurations, which are detailed in this section.

AMD Secure Execution (SEV) provides hardware-based memory encryption for virtual machines, protecting them from host-based attacks. This security feature will be available for customer implementation with OpenShift Virtualization. The infrastructure is designed to support:

- AMD CPUs with SEV capabilities
- Appropriate BIOS/firmware configurations
- Hardware requirements for memory encryption

The customer will be responsible for configuring VMs to use SEV when OpenShift Virtualization is implemented. These considerations factor into the hardware requirements specified in [Section 2.1.2](#212-worker-nodes-x86_64).

Secure boot must be disabled in certain scenarios, which is handled through the provided Ansible role:

```ansible
# Disable SecureBoot configuration
- name: Disable SecureBoot on the servers
  hosts: ocp
  gather_facts: false
  tasks:
    - name: Disable SecureBoot
      ansible.builtin.include_role:
        name: disable_secureboot
      vars:
        secureboot_enabled: false
```

> **Example from**: [playbooks/02_install_ocp.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/02_install_ocp.yaml)

### 4.2 Secure Containers on s390x

LinuxOne (s390x) nodes provide enhanced security capabilities for containerized workloads. Support for these nodes is a key part of the architecture and their integration is typically a Day 2 operational task.

> **Detailed Security Implementation**: Complete security design, implementation details, and configuration procedures for s390x workloads are documented in the dedicated [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document, specifically in Section 4 (Security Implementation for s390x).

The s390x platform provides a different security architecture that can be leveraged for workloads with specific security requirements, complementing the x86_64 virtualization capabilities.

## 5. Installation and Bootstrapping Process

### 5.1 Installation Workflow Overview

> **Note**: Detailed documentation of the Ansible repository, playbooks, roles, and inventory structure is maintained in separate documents. These technical documents, along with a comprehensive explanation of the installation process and automation, will be part of the knowledge transfer to the customer's operations team.
<&nbsp;/>
> **Rationale**: The initial installation process is designed to be non-disruptive to existing services while ensuring all core OpenShift components for the control plane and initial x86_64 worker nodes are properly configured. The safety guard mechanism prevents accidental cluster destruction, which is critical in a production environment. The addition of LinuxOne (s390x) worker nodes is a distinct Day 2 operation detailed in [Chapter 8](#8-day-2-operations).

The installation process for the initial OpenShift cluster (control plane and x86_64 worker nodes) must follow a structured sequence:

1. **Jumphost Setup**:
   - Prepare the environment for installation
   - Configure jumphost networking
   - Set up required tools and repositories

2. **OpenShift Installation**:
   - Prepare prerequisites
   - Cache RHCOS images
   - Install the OpenShift cluster

3. **Post-Installation Configuration**:
   - Bootstrap ODF storage (on x86_64 workers)
   - Perform smoke tests on the initial x86_64 cluster
   - Configure additional services relevant to the initial deployment

4. **Node Management** (as needed for x86_64 during initial phase):
   - Add x86_64 nodes (if required beyond initial deployment)
   - Delete x86_64 nodes
   (Addition of s390x nodes is covered in Day 2 Operations)

This workflow ensures that components for the initial x86_64 based cluster are installed and configured in the correct order, with validation steps between major phases.

### 5.1.1 Installation Failure Handling and Rollback Procedures

The Ansible playbooks used for cluster deployment are designed specifically for bootstrapping a new OpenShift cluster, not for maintaining or modifying an existing one. There are several important characteristics of these playbooks to understand:

1. **Non-Idempotent Nature**: The installation playbooks are not idempotent. They are designed to be run once to create a fresh installation, not to be safely re-run against an existing cluster.

2. **Safety Guard Mechanism**: A deliberate safety guard mechanism has been implemented to prevent accidental execution of the installation playbook against an existing production cluster. This protection is crucial because running the installation playbook against an existing cluster would completely destroy all workloads, configurations, and persistent data. For backup and recovery procedures in case of major failures, refer to [Section 6.2](#62-backup-and-disaster-recovery).

    ```ansible
    # Safety guard implementation example
    - name: Fail if safety guard is not passed
      when: ocp_safety_guard is not defined
      failed_when: true
      vars:
        message: |
          You have invoked a playbook that could potentially disrupt or even destroy your cluster.
          `02_install_ocp.yaml` will not check in advance if your cluster already exists but simply re-install it from scratch.
          The `ocp_{add|delete}_node` playbooks work on the entire inventory so make sure to limit it selection via `-l <host>`.
          If you're unsure, trigger the play with `--check`.

          If you absolutely know what you're doing, set the `ocp_safety_guard` variable on runtime (and runtime only).
          I'm not telling you here how to do this to make sure you know your craft.
      ansible.builtin.debug:
        msg: "{{ message.split('\n') }}"
    ```

    > **Example from**: [roles/ocp_safety_guard/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/ocp_safety_guard/tasks/main.yaml)

3. **Failure Recovery Approach**: If the installation playbook fails during execution, the recommended approach is:
   - Investigate and correct the root cause of the failure by examining logs and error messages
   - Clean up any partial installation artifacts that may have been created
   - Restart the installation process from the beginning

   Since the playbook is only used during initial bootstrapping when no production data exists, this approach ensures a clean starting point without the need for complex rollback procedures.

4. **No Partial Updates**: The playbook does not support partial updates or resuming from a failure point. This is by design, as a partially installed cluster could have unpredictable behavior or security issues.

This approach aligns with the immutable infrastructure paradigm of OpenShift, where components are replaced rather than modified in place, and underscores the importance of having separate processes for initial deployment versus day-to-day operations and updates.

```ansible
# Installation workflow from 02_install_ocp.yaml
- name: Check the safety guard
  hosts: localhost
  roles:
    - role: ocp_safety_guard
      tags: always

- name: Install OCP prerequisites
  hosts: jumphost
  roles:
    - name: install_ocp_prereqs
      tags: prereqs

- name: Disable SecureBoot on the servers
  hosts: ocp
  gather_facts: false
  tags:
    - secureboot
    - prereqs
  tasks:
    - name: Disable SecureBoot
      ansible.builtin.include_role:
        name: disable_secureboot
      vars:
        secureboot_enabled: false

- name: Provision & Attach LUNs for ODF
  hosts: ocp_odf
  tasks:
    - name: Provision & Attach LUNs for ODF
      when: odf_flash_disks is defined
      ansible.builtin.include_role:
        name: add_luns

- name: Prepare OCP installation environment
  hosts: jumphost
  roles:
    - prepare_ocp_installation_environment

- name: Install OCP
  hosts: jumphost
  roles:
    - install_ocp
```

> **Example from**: [playbooks/02_install_ocp.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/02_install_ocp.yaml)

The `ocp_safety_guard` role implements a critical safety mechanism to prevent accidental destruction of existing clusters. The installation playbook does not check if a cluster already exists before proceeding with a fresh installation. If run against an existing production cluster, it will destroy all workloads, configurations, and persistent data.

This guard forces operators to explicitly acknowledge they understand the consequences of their actions by requiring a runtime variable that is intentionally not documented in detail. In enterprise environments where clusters host mission-critical applications, this safeguard is essential to prevent costly mistakes, especially when multiple teams share management responsibilities or during maintenance windows where multiple changes are being executed under time pressure.

### 5.2 Jumphost Configuration

The jumphost serves as the central management point for cluster deployment and administration. Its preparation must involve:

1. **System Setup**:
   - Install required operating system (RHEL 8 or later)
   - Configure network access to all cluster nodes
   - Set up storage for installation artifacts

2. **Software Installation**:
   - Install required tools for cluster management
   - Set up configuration management software
   - Prepare repositories and dependencies

        ```ansible
        # Jumphost networking configuration (from configure_jumphost_networking role)
        interface_bonding:
        name: bond0
        members:
            - ens9f1
            - ens9f2
            - ens9f3
        mode: 802.3ad
        miimon: 100

        interface_IPs:
        baremetal:
            description: "OCP Baremetal Network"
            interface_name: baremetal
            interface: bond0
            ipv4: 10.153.11.108/24
            routes4:
            - 192.168.208.0/24 10.153.11.1
        # Additional networks...
        ```

        > **Example from**: [roles/configure_jumphost_networking/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/configure_jumphost_networking/tasks/main.yaml)

3. **Repository Management**:
   - Clone installation repositories
   - Configure access to Red Hat content
   - Set up local mirrors if required

Special network configurations must be implemented on the jumphost to ensure proper connectivity to all cluster components.

### 5.3 Ansible Inventory Structure

The inventory structure must define the cluster topology and configuration parameters. The implementation will include groups for initial x86_64 deployment and potentially separate groups or host definitions for s390x nodes to be added in Day 2 operations.

1. **Environment-Specific Configuration**:
   - Settings specific to each environment (test, prod, etc.)
   - Node definitions and variables

        ```ini
        # Inventory structure (hosts file)
        jumphost.init

        [jumphost]
        srv08d

        [ocp:children]
        master
        worker

        [master]
        srv09d
        srv10d
        srv11d

        [worker]

        # Additional groups...
        ```

        > **Example from**: [inventory/*/hosts](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/inventory)

2. **Global Configuration**:
   - Shared settings across environments
   - Common variables and defaults

        ```yaml
        # Global variables (from 20_ocp.yaml)
        ocp_base_domain: example.domain
        ocp_release: 4.16.34
        ocp_name: test01
        ocp_architecture: amd64
        ocp_controller_replicas: 3
        ocp_worker_replicas: 0

        ocp_machine_network: 10.153.11.0/24
        ocp_network_type: OVNKubernetes
        ocp_api_vip:
        - 10.153.11.10
        ocp_ingress_vip:
        - 10.153.11.11
        ```

        > **Example from**: [inventory/_global/ocp/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/inventory/_global/ocp/main.yaml)

This two-level structure allows for environment-specific customization while maintaining common configurations in a central location.

### 5.4 Node Preparation

Preparing x86_64 nodes for OpenShift UPI installation involves several critical steps:

1. **Hardware Configuration (x86_64)**:
   - Configure UEFI/BIOS settings via automation
   - Enable hardware virtualization features
   - Set up hardware RAID if applicable
   - Configure out-of-band management

2. **Network Bootstrapping**:
   - Initial network configuration is particularly challenging
   - Configuration will be applied via Ignition and NMState
   - Network settings must be available at first boot
   - DHCP reservations or static IP configuration must be prepared
   - Playbook to check IPMI is reachable (for x86_64 bare metal)

        ```ansible
        # Node preparation (from preflight role)
        - name: Check connectivity to all servers via the BMC network
        ansible.builtin.command: "ping -c 2 {{ hostvars[item].ipmi.ip_address }}" 
        changed_when: false
        register: ping_status
        until: ping_status.rc == 0
        retries: 5
        delay: 2
        when: (hostvars[item].ipmi is defined) and (hostvars[item].ipmi.ip_address is defined)
        loop: "{{ groups['all'] }}"
        ```

        > **Example from**: [roles/preflight_network/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/preflight_network/tasks/main.yaml)

3. **Storage Preparation**:
   - Configure local disks for OS installation
   - Set up FC-SAN zoning and LUN presentation
   - Prepare OSD devices for Ceph

Preflight checks must be performed to validate the environment before proceeding with installation, ensuring that all prerequisites are met.

Preparation for LinuxOne (s390x) nodes follows a different procedure, typically involving LPAR configuration on the mainframe and RHCOS installation, detailed in Day 2 operations (Chapter 8).

### 5.5 OpenShift Installation

The User-Provisioned Infrastructure (UPI) installation method for the initial x86_64 cluster requires specific configuration files - this configfile is generated by the ansible playbook `02_install_ocp.yaml`:

1. **install-config.yaml**:
   - Define cluster name and domain
   - Specify infrastructure providers (baremetal for x86_64)
   - Configure networking parameters for the initial cluster
   - Set up authentication details
   - This configuration pertains to the control plane and initial set of x86_64 worker nodes. s390x nodes are not defined in this initial `install-config.yaml` as they are added in Day 2 operations detailed in [Section 8.1](#81-adding-linuxone-s390x-worker-nodes)

        ```yaml
        # install-config.yaml template
        apiVersion: v1
        baseDomain: "{{ ocp_base_domain }}"
        compute:
        - name: worker
        architecture: "{{ ocp_worker_architecture }}"
        platform:
            baremetal: {}
        replicas: {{ ocp_worker_replicas }}
        controlPlane:
        name: master
        architecture: "{{ ocp_controller_architecture }}"
        platform:
            baremetal: {}
        replicas: {{ ocp_controller_replicas }}
        metadata:
        name: "{{ ocp_name }}"
        networking:
        machineNetwork:
        - cidr: "{{ ocp_machine_network }}"
        networkType: "{{ ocp_network_type }}"
        platform:
        baremetal:
            apiVIPs: {{ ocp_api_vip }}
            ingressVIPs: {{ ocp_ingress_vip }}
            # Additional configuration...
        ```

        > **Example from**: [roles/install_ocp/templates/install-config.yaml.j2](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/install_ocp/templates/install-config.yaml.j2)

2. **Ignition Configurations (for x86_64 UPI)**:
   - Bootstrap node configuration
   - Master node configurations
   - Worker node configurations (for initial x86_64 RHCOS workers)

3. **Installation Process**:
   - Generate Ignition configs
   - Set up temporary bootstrap node
   - Initialize control plane nodes
   - Join worker nodes

        ```ansible
        # Installation process (from install_ocp role)
        - name: Render the installation manifests
        ansible.builtin.command: "openshift-baremetal-install create manifests --dir {{ ocp_cluster_artifacts }}"
        tags:
            - config

        # Additional steps...

        - name: Install the cluster (asynchronously)
        tags:
            - install
        block:
            - name: Trigger the cluster installation
            ansible.builtin.command: "openshift-baremetal-install create cluster --dir {{ ocp_cluster_artifacts }} --log-level debug"
            async: 6000
            poll: 0
            register: install_status
        ```

        > **Example from**: [roles/install_ocp/tasks/main.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/roles/install_ocp/tasks/main.yaml)

RHCOS images (for x86_64) must be cached locally for faster deployment, reducing external dependencies during installation.
RHCOS images for s390x will be build by the Day 2 process when installing the LinuxOne (s390x) worker nodes.

Reference: [OpenShift Bare Metal UPI Installation](https://docs.openshift.com/container-platform/4.16/installing/installing_bare_metal/installing-bare-metal.html)

### 5.6 Post-Installation Configuration

After the base x86_64 cluster is operational, additional components must be configured:

1. **Operator Deployment**:
   - Install required operators from OperatorHub
   - Configure operator subscriptions
   - Create required CRDs and resources

        ```ansible
        # Post-installation configuration example
        - name: Bootstrap ODF
        hosts: jumphost
        roles:
            - role: odf_config
            tags: odf
        ```

        > **Example from**: [playbooks/10_bootstrap_ocp.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/10_bootstrap_ocp.yaml)

2. **Networking Setup**:
   - Configure additional network attachments
   - Set up load balancers and ingress
   - Implement network policies

3. **Storage Configuration**:
   - Deploy ODF operator
   - Configure storage classes
   - Set up persistent volume provisioning

The post-installation phase transforms the base OpenShift cluster into a platform ready for customer workloads on x86_64 and prepares for the integration of s390x worker nodes as a Day 2 operation. For detailed monitoring configuration after installation, see [Section 6.1](#61-monitoring-configuration).

### 5.7 Validation and Testing

The implementation must include several testing components to validate the deployment:

1. **Smoke Testing (Initial x86_64 Cluster)**:
   - Verify basic cluster functionality
   - Validate core services
   - Check API availability

        ```ansible
        # Smoke test examples (from 20_smoketest_ocp.yaml)
        - name: Run OpenShift smoke tests
        hosts: jumphost
        roles:
            - test_storage_class
            - test_odf_resources
            - test_loadbalancer
        ```

        > **Example from**: [playbooks/20_smoketest_ocp.yaml](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/playbooks/20_smoketest_ocp.yaml)

2. **Storage Testing**:
   - Validate storage provisioning
   - Test ODF functionality
   - Verify data persistence

3. **Load Balancer Testing**:
   - Validate load balancer configuration
   - Test service exposure
   - Verify functionality using end-to-end test cases

Validation for s390x nodes and their specific workloads (e.g., secure containers) will be performed after their successful integration as a Day 2 operation.

These validation steps ensure that the platform meets requirements before being put into production use.

## 6. Monitoring and Operations

### 6.1 Monitoring Configuration

> **Rationale**: Comprehensive monitoring is essential for maintaining platform health and performance. The monitoring configuration is designed to provide visibility into all aspects of the infrastructure while maintaining compatibility with existing monitoring systems.

Comprehensive monitoring is essential for maintaining platform health and performance:

1. **Prometheus Integration**:
   - Configure ServiceMonitors for component metrics
   - Set up alerting rules for critical conditions
   - Implement custom metrics for infrastructure components

        ```yaml
        # Monitoring configuration
        apiVersion: monitoring.coreos.com/v1
        kind: ServiceMonitor
        metadata:
        name: kubevirt-monitoring
        namespace: openshift-cnv
        spec:
        endpoints:
        - port: metrics
        selector:
            matchLabels:
            app: virt-handler
        ```

        > **Example from**: [manifests/base/monitoring/](https://git.kvant.cloud/phoenix/red.kvant/src/branch/main/manifests/base/monitoring/)

2. **Dashboard Development**:
   - Create Grafana dashboards for key metrics
   - Set up infrastructure-specific monitoring views
   - Implement storage performance dashboards

3. **Log Management**:
   - Configure centralized logging
   - Set up log retention policies
   - Implement log analysis for troubleshooting

The monitoring system will provide visibility into cluster operations, enabling proactive management and quick resolution of issues.

Reference: [OpenShift Monitoring Stack](https://docs.openshift.com/container-platform/4.16/monitoring/monitoring-overview.html)

### 6.2 Backup and Disaster Recovery

Protecting infrastructure and configuration requires a comprehensive backup strategy. The customer must implement several backup procedures in line with Red Hat's official recommendations:

#### 6.2.1 Cluster Backup Components

1. **Control Plane Backup (etcd)**:
   - Etcd contains all cluster configuration data, resource definitions, and states
   - Regular etcd backups must be performed using the cluster-backup script
   - Procedure:

     ```bash
     # SSH to a master node
     ssh core@<master-node>
     
     # Create an etcd backup
     sudo /usr/local/bin/cluster-backup.sh /home/core/backup
     
     # Copy backup to external storage
     scp core@<master-node>:/home/core/backup/* /path/to/backup/location/
     ```

     > **Example from**: Red Hat OpenShift documentation - operational procedures

   - Frequency: Daily automated backups plus before major changes
   - Retention: Minimum of 7 days, with monthly backups kept for 12 months

2. **Kubernetes Resources Backup**:
   - Critical namespace and resource exports
   - Procedure:

     ```bash
     # Export critical namespaces
     oc get -o yaml -n openshift-storage cm,secret,sa,sc,pv,pvc,ds,deploy > odf-backup.yaml
     
     # Export cluster configuration
     oc adm migrate storage-version > cluster-storage-version.yaml
     ```

     > **Example from**: Red Hat OpenShift documentation - operational procedures

   - Frequency: Weekly and before major configuration changes

3. **OpenShift Infrastructure Configuration**:
   - Machine configurations, node settings, and operator states
   - Machine ConfigMaps and CRDs must be backed up regularly
   - Use a backup solution that can capture Kubernetes custom resources

#### 6.2.2 Storage Backup Considerations

1. **ODF/Ceph Backup Strategy**:
   - ODF/Ceph maintains internal replication for data on x86_64 nodes, but this is not a substitute for backups
   - Procedures for backing up the ODF configuration:

     ```bash
     # Backup ODF operator resources
     oc get -n openshift-storage storagecluster,backingstore,bucketclass -o yaml > odf-config-backup.yaml
     
     # Verify Ceph health before backing up
     oc rsh -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) ceph status
     ```

     > **Example from**: Red Hat OpenShift documentation - operational procedures

2. **Application Data Backup**:
   - Recommend the OpenShift API for Data Protection (OADP) operator to protect stateful applications
   - Installation and configuration reference: [OADP Documentation](https://docs.openshift.com/container-platform/4.16/backup_and_restore/application_backup_and_restore/installing/about-installing-oadp.html)
   - Provide external backup repository (S3-compatible storage)

#### 6.2.3 Disaster Recovery Planning

1. **Recovery Time Objectives (RTO)** and **Recovery Point Objectives (RPO)**:
   - Define service-specific RTOs and RPOs based on criticality
   - Tier 1 services: RPO < 24 hours, RTO < 4 hours
   - Tier 2 services: RPO < 48 hours, RTO < 8 hours
   - Tier 3 services: RPO < 1 week, RTO < 24 hours

2. **Recovery Procedures**:
   - Document detailed recovery procedures for different scenarios:
     - Control plane failure
     - Worker node failures
     - Storage loss
     - Data corruption
   - Control plane recovery procedure:

     ```bash
     # Stop the static pods on the recovered master
     sudo mv /etc/kubernetes/manifests/etcd-pod.yaml /tmp/
     sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp/
     sudo mv /etc/kubernetes/manifests/kube-controller-manager-pod.yaml /tmp/
     sudo mv /etc/kubernetes/manifests/kube-scheduler-pod.yaml /tmp/
     
     # Restore etcd backup
     sudo /usr/local/bin/cluster-restore.sh /home/core/backup
     
     # Restart the kubelet service
     sudo systemctl restart kubelet.service
     ```

     > **Example from**: Red Hat OpenShift documentation - operational procedures

3. **Recovery Testing**:
   - Implement regular recovery testing (quarterly)
   - Create a disaster recovery test environment
   - Document and practice recovery procedures
   - Test and validate restored backups
   - Include recovery test reports in compliance documentation

#### 6.2.4 Cluster Failure Scenarios

1. **Single Master Node Failure**:
   - Non-disruptive; cluster continues to operate
   - Replace the failed node using the documented procedures
   - Recovery process:

     ```bash
     # Remove the failed node from etcd membership
     oc patch etcd cluster -p='{"spec": {"unsupportedConfigOverrides": {"useUnsupportedUnsafeNonHANonProductionUnstableEtcd": true}}}' --type=merge
     
     # Create a new master node
     # Details depend on infrastructure platform
     ```

     > **Example from**: Red Hat OpenShift documentation - operational procedures

2. **Multiple Master Node Failures**:
   - Requires etcd quorum recovery
   - Follow Red Hat's documented disaster recovery procedure
   - Contact Red Hat support for assistance

3. **Worker Node Failures**:
   - Replace failed worker nodes using the node management procedures
   - Use the ocp_add_node role for replacement

All backup and recovery procedures must be regularly tested and documented in the organization's operational runbooks. The customer is responsible for maintaining backup procedures and ensuring they meet business continuity requirements for all platform components. Specific backup considerations for s390x workloads are detailed in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

Reference: [OpenShift Backup and Restore](https://docs.openshift.com/container-platform/4.16/backup_and_restore/index.html)

## 7. Conclusion

This Low-Level Design document provides the technical details for implementing the foundation for OpenShift Virtualization as a replacement for OpenStack. The implementation follows Red Hat best practices and leverages the capabilities of OpenShift 4.16 and OpenShift Data Foundation.

> **Rationale**: The design decisions made in this document are based on the requirements to provide a high-performance, secure, and maintainable platform for virtualized workloads. The phased approach allows for a controlled migration from the existing OpenStack environment while maintaining service availability.

The design addresses key requirements including:

- High availability for critical workloads
- Future integration of x86_64 and s390x architectures
- Storage implementation with static FC-SAN LUN mapping
- Network segmentation and performance
- Foundation for secure execution capabilities

Implementation will proceed according to the outlined approach, with detailed procedures and configurations to be developed during the deployment phase. The implementation plan establishes a solid foundation with clear separation of concerns and a logical deployment sequence.

## 8. Day 2 Operations

Once the initial OpenShift Container Platform cluster (Control Plane and x86_64 worker nodes) is installed and validated, several Day 2 operations are critical for extending functionality, maintaining the cluster, and managing its lifecycle. This chapter details these operations, with a particular focus on integrating and managing LinuxOne (s390x) worker nodes.

### 8.1 Adding LinuxOne (s390x) Worker Nodes

The addition of LinuxOne (s390x) worker nodes is a distinct Day 2 operation that involves specific procedures for LPAR provisioning, RHCOS image preparation, and cluster integration.

> **Detailed Implementation Guide**: Complete procedures for adding s390x worker nodes, including LPAR configuration, RHCOS image preparation, network setup, CSR approval processes, and troubleshooting, are documented in the dedicated [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document, specifically in Section 5 (Day 2 Operations: Adding LinuxOne (s390x) Worker Nodes).

**Key Characteristics:**
- Image-based approach using custom RHCOS for s390x
- Ansible-driven automation for configuration preparation
- Simplified mainframe operations team involvement
- Automated cluster joining with CSR approval workflow

### 8.2 Scaling x86_64 Worker Nodes

Scaling the x86_64 worker node pool can be achieved using the Machine API if the initial UPI deployment was configured to support it, or by manually provisioning new x86_64 nodes and joining them to the cluster using a similar CSR approval process as for s390x nodes (though often with RHCOS and Ignition).

- **Using MachineSets:**
  - If MachineSets were configured for x86_64 workers, scaling is as simple as increasing the `replicas` count in the relevant MachineSet CRD.

    ```bash
    oc scale machineset <machineset_name> --replicas=5
    ```

    > **Example from**: Day 2 operational procedures

  - The Machine API will then handle the provisioning of new machines based on the defined provider spec (e.g., creating new VMs on a hypervisor if the baremetal UPI was for management hosts of a virtualized environment, or interacting with BMCs for new baremetal servers if using a baremetal provider spec).
- **Manual Addition of x86_64 Nodes:**
  - Provision a new x86_64 baremetal server or VM.
  - Install RHCOS using an Ignition file generated for adding a worker to the existing cluster.
  - Approve CSRs as they appear.
  - Label the node as needed.

### 8.3 Cluster Upgrades

OpenShift Container Platform upgrades are a critical Day 2 operation to apply bug fixes, security patches, and new features.

- **Upgrade Channels & Versions:** Select an appropriate upgrade channel (e.g., stable, fast) and target version through the OpenShift web console or `oc adm upgrade`.
- **Pre-Upgrade Checks:** Review release notes, run `oc adm upgrade` with `--check-only` (if available, or follow pre-flight checklists), and ensure cluster health.
- **Upgrade Process:** The Cluster Version Operator (CVO) manages the upgrade process, updating operators and then cluster nodes (masters first, then workers). Node draining and cordoning are handled automatically for RHCOS nodes.
- **Monitoring:** Monitor upgrade progress via `oc adm upgrade` or the web console.
- **Post-Upgrade Validation:** Perform health checks and validate application functionality.

> **Note**: Upgrades for s390x worker nodes, especially if running RHEL, might require manual intervention for OS patching and Kubelet updates, coordinated with the overall cluster upgrade if the Kubelet version skew becomes an issue. See [Section 8.1](#81-adding-linuxone-s390x-worker-nodes) for more details on s390x node management.

### 8.4 ODF Storage Management (for x86_64 workloads)

Ongoing management of OpenShift Data Foundation for x86_64 workloads includes:

- **Monitoring ODF Health:** Regularly check Ceph health, OSD status, pool capacity, and performance metrics using `oc` commands, the ODF dashboard, and Prometheus/Grafana.
- **Capacity Management:** Monitor storage utilization and expand ODF capacity by adding more OSDs (potentially by adding more storage to existing nodes or adding new storage nodes, as per the migration strategy in [Section 2.3.2](#232-migration-strategy-to-dedicated-hardware)).

#### 8.4.1 Critical Capacity Management and Operational Limits

> **⚠️ CRITICAL OPERATIONAL REQUIREMENT**: Storage capacity management is essential for ODF/Ceph cluster stability and data availability. Exceeding capacity thresholds will result in cluster health degradation and potential data loss.

**Official Ceph Capacity Thresholds:**

The Ceph storage system implements three critical capacity thresholds that must be actively monitored and managed:

1. **Near Full Warning Threshold - 85%** (`mon_osd_nearfull_ratio: 0.85`)
   - **Status**: `OSD_NEARFULL` warning state
   - **Impact**: Warning alerts generated, no operational impact yet
   - **Action Required**: Begin planning for capacity expansion

2. **Backfill Full Threshold - 90%** (`mon_osd_backfillfull_ratio: 0.90`)
   - **Status**: `OSD_BACKFILLFULL` warning state  
   - **Impact**: Ceph stops backfill operations (data rebalancing/recovery)
   - **Action Required**: **Immediate capacity expansion required**

3. **Full Threshold - 95%** (`mon_osd_full_ratio: 0.95`)
   - **Status**: `OSD_FULL` error state
   - **Impact**: **All write operations are prevented** - cluster becomes read-only
   - **Action Required**: **Emergency capacity expansion**

> **Reference**: [Official Ceph Documentation - Monitor Configuration](https://docs.ceph.com/en/latest/rados/configuration/mon-config-ref/#storage-capacity) and [Ceph Health Checks](https://docs.ceph.com/en/latest/rados/operations/health-checks/#osd-nearfull)

**Recommended Operational Practices:**

- **Soft Operational Limit**: **75%** - Initiate capacity planning and procurement
- **Hard Operational Limit**: **80%** - Begin immediate capacity expansion procedures
- **Emergency Threshold**: **85%** - Execute emergency capacity expansion

**Capacity Monitoring Commands:**

```bash
# Check overall cluster capacity
oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph df

# Check OSD utilization
oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph osd df

# Check cluster health status
oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph health detail

# Monitor capacity alerts
oc get prometheusrule -n openshift-storage -o yaml | grep -A 10 -B 5 "CephCluster"
```

**Emergency Capacity Expansion Procedures:**

If capacity exceeds 85% (Near Full threshold):

1. **Immediate Assessment:**
   ```bash
   # Identify OSDs approaching full
   oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph osd df | sort -k8 -nr
   
   # Check for stuck PGs that might be preventing cleanup
   oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph pg stat
   ```

2. **Capacity Expansion Options:**

   **Option A: Add LUNs to Existing Nodes**
   - Provision additional FC-SAN LUNs using the `add_luns` role
   - Add new OSDs to existing worker nodes
   - Monitor rebalancing progress continuously

   **Option B: Deploy Additional ODF Worker Nodes**
   - Add new worker nodes to the cluster using the `ocp_add_node` role:
   ```ansible
   # Add new ODF worker nodes
   - name: Add new worker nodes for ODF
     hosts: jumphost
     roles:
       - ocp_add_node
     vars:
       node_labels:
         - "cluster.ocs.openshift.io/openshift-storage="
         - "node-role.kubernetes.io/worker="
   ```
   - Provision LUNs for the new nodes using `add_luns`
   - Update the StorageCluster to include new nodes

3. **Emergency Mitigation (if approaching 90%):**
   - Temporarily increase the full ratio (use with extreme caution):
   ```bash
   # EMERGENCY ONLY - Increase thresholds temporarily
   oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph osd set-nearfull-ratio 0.90
   oc exec -n openshift-storage $(oc get pods -n openshift-storage -l app=rook-ceph-operator -o name) -- ceph osd set-backfillfull-ratio 0.93
   ```

> **Warning**: Temporarily modifying capacity thresholds should only be done as an emergency measure while adding capacity. Reset to default values immediately after capacity expansion.

#### 8.4.2 ODF Node Sizing and OSDs per Node Guidelines

**Official Ceph Hardware Sizing Recommendations:**

Based on official Ceph documentation, the following guidelines should be followed when adding new ODF worker nodes:

**CPU Requirements per OSD:**
- **HDD OSDs**: 0.5-1 CPU cores per HDD OSD (assuming 2GHz CPU)
- **SSD OSDs**: 2 CPU cores per SSD OSD  
- **NVMe OSDs**: 4+ CPU cores per NVMe OSD
- **Performance Factor**: Modern deployments should optimize for IOPS per core rather than just cores per OSD

**RAM Requirements per OSD:**
- **Minimum**: 4GB per OSD daemon (BlueStore default `osd_memory_target`)
- **Recommended**: 5-8GB per OSD for production workloads
- **Rule of Thumb**: ~1GB RAM per 1TB of storage capacity
- **Total Node RAM**: Number of OSDs × (4-8GB) + 16GB for OS overhead

**Network Bandwidth per Node:**
- **Minimum**: 10 Gbps for basic operations
- **Recommended**: 25+ Gbps for production
- **Constraint**: Aggregate OSD throughput must not exceed available network bandwidth
- **Replication Factor**: Consider 3x replication traffic for data writes

**OSDs per Node Recommendations:**

| Storage Type | OSDs per Node | CPU Cores Required | RAM Required | Network Consideration |
|--------------|---------------|-------------------|--------------|----------------------|
| **HDD-based** | 8-12 OSDs | 4-12 cores | 48-128GB | 10+ Gbps adequate |
| **SSD-based** | 4-8 OSDs | 8-16 cores | 48-96GB | 25+ Gbps recommended |
| **NVMe-based** | 2-4 OSDs | 8-16 cores | 32-64GB | 25+ Gbps required |
| **Mixed (HDD + SSD WAL/DB)** | 6-10 OSDs | 6-10 cores | 48-112GB | 10-25 Gbps |

> **Reference**: [Official Ceph Hardware Recommendations](https://docs.ceph.com/en/reef/start/hardware-recommendations/) and [Red Hat ODF Hardware Guide](https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.16/html/planning_your_deployment/index)

**Additional Considerations for Node Planning:**

1. **Fault Tolerance**: No single node should exceed ~15% of total cluster storage
2. **Failure Impact**: More OSDs per node = greater impact if node fails
3. **Rebalancing Time**: Consider network bandwidth for recovery operations  
4. **Power and Cooling**: Factor in datacenter constraints for dense configurations
5. **Future Growth**: Reserve capacity for additional OSDs as storage needs grow

**Deployment Best Practices:**

- Start with fewer OSDs per node for better fault tolerance
- Monitor resource utilization and gradually increase density if needed
- Ensure adequate network bandwidth before adding more OSDs per node
- Plan for 3x network traffic during recovery/rebalancing operations

#### 8.4.3 Critical Capacity Planning for Node Failure Resilience

> **⚠️ CRITICAL PLANNING REQUIREMENT**: Cluster capacity must be planned to ensure that the failure of the **largest storage node** can be tolerated while maintaining data availability and performance. This requires maintaining sufficient free capacity for rebalancing operations after node failure.

**Resilience Planning Formula:**

When planning cluster capacity, the following calculation must be applied:

```
Maximum Usable Space = (Total Raw Capacity - Largest Node Capacity) × 0.8
```

Where:
- **Total Raw Capacity**: Sum of all OSD storage across all nodes
- **Largest Node Capacity**: Storage capacity of the node with most storage
- **0.8**: Safety factor (80% utilization limit to maintain performance)

> **Note**: This calculation shows the maximum usable space before considering replication overhead. The actual data storage will be further reduced by the replication factor (typically 3x for standard ODF deployments).

**Example Calculations:**

**Scenario 1a: 3 Nodes with 4 OSDs Each**
- Configuration: 3 nodes with 4×5TB OSDs each = 60TB total
- Largest node: 20TB
- Calculation: (60TB - 20TB) × 0.8 = 40TB × 0.8 = 32TB usable
- **Maximum safe utilization: 53.3% of total capacity**

**Scenario 1b: 3 Nodes with 6 OSDs Each**
- Configuration: 3 nodes with 6×5TB OSDs each = 90TB total
- Largest node: 30TB
- Calculation: (90TB - 30TB) × 0.8 = 60TB × 0.8 = 48TB usable
- **Maximum safe utilization: 53.3% of total capacity**

**Scenario 2: 4 Equal Nodes**
- Configuration: 4 nodes with 5×5TB OSDs each = 100TB total
- Largest node: 25TB
- Calculation: (100TB - 25TB) × 0.8 = 75TB × 0.8 = 60TB usable
- **Maximum safe utilization: 60% of total capacity**

**Scenario 3: 5 Equal Nodes**
- Configuration: 5 nodes with 5×5TB OSDs each = 125TB total
- Largest node: 25TB
- Calculation: (125TB - 25TB) × 0.8 = 100TB × 0.8 = 80TB usable
- **Maximum safe utilization: 64% of total capacity**

**Scenario 4: 10 Equal Nodes**
- Configuration: 10 nodes with 5×5TB OSDs each = 250TB total
- Largest node: 25TB
- Calculation: (250TB - 25TB) × 0.8 = 225TB × 0.8 = 180TB usable
- **Maximum safe utilization: 72% of total capacity**

**Key Insights from Capacity Planning:**

| Cluster Size | Node Configuration | Max Used Space at 80% | Resilience Level |
|-------------|-------------------|----------------------|------------------|
| **3 nodes (small)** | 3×20TB (4×5TB OSDs) | **53.3%** | Minimum viable |
| **3 nodes (large)** | 3×30TB (6×5TB OSDs) | **53.3%** | Minimum viable |
| **4 nodes** | 4×25TB | **60%** | Better |
| **5 nodes** | 5×25TB | **64%** | Good |
| **10 nodes** | 10×25TB | **72%** | Excellent |

**Critical Operational Implications:**

1. **Larger clusters are more efficient**: More nodes allow higher utilization percentages (53% → 72%)
2. **Even node sizing is preferred**: Mixed node sizes reduce efficiency (largest node becomes bottleneck)
3. **Replication overhead is additional**: These percentages show usable space before 3x replication overhead
4. **3-node minimum limitation**: 3-node clusters are limited to ~53% utilization regardless of node size
5. **Planning buffer**: Always plan for 15-20% additional capacity beyond calculated maximums

> **Warning**: Exceeding these calculated capacity limits risks cluster performance degradation or potential data unavailability during node failures. Regular capacity monitoring against these thresholds is essential for maintaining cluster health.

### 8.5 Node Maintenance

Performing maintenance on cluster nodes (both x86_64 and s390x) requires careful planning to avoid service disruption.

- **Draining Nodes:** Before maintenance (e.g., hardware replacement), safely drain workloads from the node:

  ```bash
  oc adm drain <node_name> --ignore-daemonsets --delete-emptydir-data
  ```

  > **Example from**: Day 2 operational procedures

- **Performing Maintenance:** Execute the required maintenance tasks.
- **Uncordoning Nodes:** After maintenance, make the node schedulable again:

  ```bash
  oc adm uncordon <node_name>
  ```

  > **Example from**: Day 2 operational procedures

- **RHCOS Updates:** For RHCOS-based nodes (both x86_64 and s390x), OS updates are managed by the Machine Config Operator (MCO) as part of OpenShift updates or by applying custom MachineConfigs. s390x-specific update procedures are detailed in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

### 8.6 Backup and Restore Operations

Refer to [Section 6.2](#62-backup-and-disaster-recovery) for detailed backup and disaster recovery strategies. Day 2 operations include:

- **Regularly Executing Backups:** Ensuring automated etcd backups, ODF configuration backups (if applicable), and application data backups (using OADP or other tools) are running successfully.
- **Testing Restore Procedures:** Periodically testing the restoration process in a non-production environment to validate backup integrity and recovery time objectives (RTOs).
- **Managing Backup Storage:** Monitoring and managing the capacity of backup storage locations.

### 8.7 Security and Compliance Monitoring

Ongoing security operations include:

- **Applying Security Patches:** Timely application of OCP, ODF, RHCOS, and s390x firmware security updates.
- **Vulnerability Scanning:** Regularly scanning container images and nodes for vulnerabilities using tools like OpenSCAP, Quay security scanner, or third-party solutions.
- **Audit Log Review:** Monitoring and reviewing OpenShift audit logs and node audit logs for suspicious activity.
- **Compliance Checks:** Running compliance checks against defined benchmarks (e.g., CIS Benchmarks for OpenShift and Kubernetes) using tools like OpenSCAP or configuration management tools.
- **Updating Security Policies:** Reviewing and updating NetworkPolicies, SecurityContextConstraints (SCCs), and RBAC configurations as application requirements evolve or new threats emerge.

This security monitoring complements the security implementation described in [Section 4](#4-security-implementation), ensuring ongoing protection of both x86_64 and s390x workloads. Specific security monitoring for s390x workloads is detailed in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

### 8.8 Network Configuration Updates

Modifications to network configurations post-installation might be needed.

- **NMState for x86_64:** For x86_64 nodes, changes to network interfaces, bonding, or VLANs can be managed declaratively via NodeNetworkConfigurationPolicy (NNCP) custom resources applied by the NMState operator.
- **s390x Network Changes:** For RHCOS on s390x nodes, network configuration changes are managed through specific procedures detailed in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document, maintaining consistency with the immutable infrastructure approach.
- **MetalLB Configuration:** Updating IP address pools for MetalLB if more external IPs are required for LoadBalancer services.
- **OVN-Kubernetes:** Advanced OVN-K configurations (e.g., external gateways, hybrid networking) might be implemented or adjusted.

This chapter provides a foundational overview of common Day 2 operations. Specific procedures should be further detailed in operational runbooks.

## 9. Conclusion

This Low-Level Design document provides the technical details for implementing the foundation for OpenShift Virtualization as a replacement for OpenStack. The implementation follows Red Hat best practices and leverages the capabilities of OpenShift 4.16 and OpenShift Data Foundation.

The design addresses key requirements including:

- High availability for critical workloads
- Future integration of x86_64 and s390x architectures
- Storage implementation with static FC-SAN LUN mapping
- Network segmentation and performance
- Foundation for secure execution capabilities

Implementation will proceed according to the outlined approach, with detailed procedures and configurations to be developed during the deployment phase. The implementation plan establishes a solid foundation with clear separation of concerns and a logical deployment sequence.

LinuxOne (s390x) integration details and procedures are comprehensively documented in the [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document.

> **Rationale**: The design decisions made in this document are based on the requirements to provide a high-performance, secure, and maintainable platform for virtualized workloads. The phased approach allows for a controlled migration from the existing OpenStack environment while maintaining service availability.

## Appendices

### Appendix A: Personnel Training Requirements

Personnel responsible for the OpenShift cluster implementation and operations must possess or acquire proficiency in multiple technical domains. The following training requirements are organized by role and skill level, with specific Red Hat training courses recommended to achieve these competencies.

#### A.1 Core Foundation Skills

**1. Linux System Administration (RHCSA Level)**
- **Skills Required**: Command-line proficiency, system administration, file systems, user management, package management, systemd services, network configuration, troubleshooting
- **Training Courses**:
  - **RH124**: [Red Hat System Administration I](https://www.redhat.com/en/services/training/rh124-red-hat-system-administration-i) - Foundation skills
  - **RH134**: [Red Hat System Administration II](https://www.redhat.com/en/services/training/rh134-red-hat-system-administration-ii) - Advanced administration
  - **RH199**: [RHCSA Rapid Track Course](https://www.redhat.com/en/services/training/rh199-rhcsa-rapid-track-course) - Accelerated path for experienced administrators
- **Certification**: Red Hat Certified System Administrator (RHCSA)

**2. Container and Kubernetes Fundamentals**
- **Skills Required**: Container concepts, Kubernetes architecture, pods, services, deployments, storage, networking
- **Training Courses**:
  - **DO180**: [Red Hat OpenShift I: Containers & Kubernetes](https://www.redhat.com/en/services/training/do180-red-hat-openshift-i-containers-kubernetes) - Container and Kubernetes basics
  - **DO188**: [Red Hat OpenShift Development I: Introduction to Containers with Podman](https://www.redhat.com/en/services/training/do188-red-hat-openshift-development-i-introduction-containers-podman) - Container development fundamentals

#### A.2 OpenShift Platform Administration

**3. OpenShift Administration (Essential)**
- **Skills Required**: OpenShift architecture, cluster installation, configuration, monitoring, troubleshooting, user management, security policies
- **Training Courses**:
  - **DO280**: [Red Hat OpenShift Administration II: Operating a Production Kubernetes Cluster](https://www.redhat.com/en/services/training/do280-red-hat-openshift-administration-ii-operating-production-kubernetes-cluster) - **Primary course for cluster administrators**
  - **DO380**: [Red Hat OpenShift Administration III: Scaling Kubernetes Deployments in the Enterprise](https://www.redhat.com/en/services/training/do380-red-hat-openshift-administration-iii-scaling-kubernetes-deployments-enterprise) - Advanced scaling and enterprise features
- **Certification**: Red Hat Certified OpenShift Administrator (RHCOA)

**4. OpenShift Data Foundation (Storage Administration)**
- **Skills Required**: ODF architecture, Ceph administration, storage classes, persistent volumes, performance tuning, troubleshooting
- **Training Courses**:
  - **DO370**: [Red Hat OpenShift Data Foundation Administration](https://www.redhat.com/en/services/training/do370-red-hat-openshift-data-foundation-administration) - **Essential for storage administrators**
  - **CEPH125**: [Red Hat Ceph Storage Administration](https://www.redhat.com/en/services/training/ceph125-red-hat-ceph-storage-administration) - Deep Ceph knowledge (supplementary)
- **Certification**: Red Hat Certified Specialist in OpenShift Data Foundation

#### A.3 Automation and Infrastructure as Code

**5. Ansible Automation (Required for Deployment)**
- **Skills Required**: Playbook development, role creation, inventory management, variable handling, templating, troubleshooting
- **Training Courses**:
  - **RH124**: [Red Hat System Administration I](https://www.redhat.com/en/services/training/rh124-red-hat-system-administration-i) - Includes basic Ansible
  - **RH294**: [Red Hat Linux Automation with Ansible](https://www.redhat.com/en/services/training/rh294-red-hat-linux-automation-ansible) - Comprehensive Ansible training
  - **DO374**: [Red Hat Ansible Automation Platform](https://www.redhat.com/en/services/training/do374-developing-advanced-automation-red-hat-ansible-automation-platform) - Advanced automation and AAP
- **Certification**: Red Hat Certified Engineer (RHCE) with Ansible

#### A.4 Specialized Platform Components

**6. OpenShift Virtualization (Future Implementation)**
- **Skills Required**: KubeVirt concepts, VM lifecycle management, virtual machine storage and networking, live migration, integration with OpenShift
- **Training Courses**:
  - Currently covered within DO280 and DO380 OpenShift courses
  - **DO316**: [Managing Virtual Machines with Red Hat OpenShift Virtualization](https://www.redhat.com/en/services/training/do316-managing-virtual-machines-red-hat-openshift-virtualization) - **Dedicated OpenShift Virtualization course**
- **Additional Resources**: [OpenShift Virtualization Documentation](https://docs.openshift.com/container-platform/4.16/virt/about-virt.html)

**7. s390x/LinuxONE Integration (Specialized)**
- **Skills Required**: s390x architecture, LinuxONE systems, RHCOS on s390x, mainframe integration concepts
- **Training Courses**:
  - **LX0103**: [IBM Z and LinuxONE: Technical Introduction](https://www.redhat.com/en/services/training/lx0103-ibm-z-and-linuxone-technical-introduction) - Foundation
  - Integration aspects covered in DO280 and DO380
- **Additional Resources**: IBM Z and LinuxONE documentation, Red Hat on IBM Z documentation, [LinuxOne s390x LPAR Design](LinuxOne-s390x-LPAR-Design.md) document

#### A.5 Advanced and Specialized Skills

**8. Security and Compliance**
- **Skills Required**: SecurityContextConstraints (SCCs), network policies, RBAC, compliance frameworks, security hardening
- **Training Courses**:
  - **DO425**: [Red Hat Security: Linux in Physical, Virtual, and Cloud](https://www.redhat.com/en/services/training/do425-red-hat-security-linux-physical-virtual-and-cloud) - Security fundamentals
  - Security topics integrated into DO280 and DO380

**9. Monitoring and Observability**
- **Skills Required**: Prometheus, Grafana, alerting, log management, metrics collection, troubleshooting
- **Training Courses**:
  - Monitoring topics covered in DO280
  - **DO480**: [Multicluster Management with Red Hat Advanced Cluster Management for Kubernetes](https://www.redhat.com/en/services/training/do480-multicluster-management-red-hat-advanced-cluster-management-kubernetes) - Includes monitoring across clusters

#### A.6 Role-Based Training Paths

| Role | Essential Courses | Supplementary Courses | Duration | Priority |
|------|------------------|----------------------|----------|----------|
| **Platform Administrator** | RH124/134, DO180, DO280 | DO380, DO370 | 12-16 days | High |
| **Storage Administrator** | RH124/134, DO180, DO280, DO370 | CEPH125 | 16-20 days | High |
| **Automation Engineer** | RH124/134, RH294, DO280 | DO374 | 12-16 days | High |
| **Operations Engineer** | RH124/134, DO180, DO280 | DO380, DO316 | 12-20 days | Medium |
| **Security Administrator** | RH124/134, DO180, DO280, DO425 | DO380 | 16-20 days | Medium |
| **LinuxONE Specialist** | RH124/134, DO180, DO280, LX0103 | DO380 | 16-20 days | Medium |

#### A.7 Continuous Learning and Certification Maintenance

**Ongoing Requirements**:
- Regular participation in Red Hat Learning Subscription for updates
- Annual recertification for time-limited certifications
- Attendance at Red Hat Summit for latest developments
- Engagement with Red Hat Customer Portal and documentation updates

**Learning Resources**:
- **Red Hat Learning Subscription**: [Unlimited access to Red Hat training](https://www.redhat.com/en/services/training/learning-subscription)
- **Red Hat Developer Program**: Free developer resources and training
- **Red Hat Customer Portal**: Documentation, KCS articles, and support resources
- **Partner Training**: Additional specialized training from technology partners (IBM, etc.)

#### A.8 Training Implementation Strategy

**Phase 1 - Foundation (Immediate)**:
- Core Linux administration (RH124/134)
- Basic OpenShift administration (DO180, DO280)
- Essential Ansible skills (RH294)

**Phase 2 - Specialization (3-6 months)**:
- Advanced OpenShift (DO380)
- Storage administration (DO370)
- Advanced automation (DO374)

**Phase 3 - Advanced Topics (6-12 months)**:
- OpenShift Virtualization (DO316)
- LinuxONE integration (LX0103)
- Security and compliance (DO425)

This comprehensive training plan ensures personnel have the necessary skills to implement, operate, and maintain the OpenShift platform with all its specialized components.

### Appendix B: Additional Design Considerations

#### B.1 Risk Assessment

A thorough risk assessment is crucial for the successful deployment and operation of the OpenShift Virtualization platform. The following table outlines potential risks, their impact, likelihood, and mitigation strategies. These risks should be considered in conjunction with the general implementation approach detailed in [Section 5](#5-installation-and-bootstrapping-process) and the backup and recovery procedures described in [Section 6.2](#62-backup-and-disaster-recovery).

| Risk Category          | Risk Description                                                                 | Potential Impact                                     | Likelihood | Mitigation Strategies                                                                                                                                                                                                                                                           |
|------------------------|----------------------------------------------------------------------------------|------------------------------------------------------|------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| **Technical Risks**    |                                                                                  |                                                      |            |                                                                                                                                                                                                                                                                               |
|                        | Hardware/Software incompatibility (e.g., SEV, NIC drivers, OCP/ODF versions)       | Service instability, deployment delays, feature loss | Medium     | Thorough pre-deployment testing in a dedicated lab environment; Adherence to Red Hat's Hardware Compatibility List (HCL); Vendor consultation for firmware/driver recommendations; Phased rollout with pilot testing.                                                              |
|                        | Network configuration complexity (NMState, bonding, VLANs, OVN-K) leading to errors | Network outages, service disruption, VM connectivity issues | Medium     | Utilization of validated and version-controlled Ansible roles for network configuration; Comprehensive pre-flight checks and post-deployment validation scripts; Peer review of network designs and configurations; Detailed rollback plans for network changes.                     |
|                        | Storage performance bottlenecks (FC-SAN limitations, ODF tuning, I/O contention) | Degraded VM performance, application slowdowns, data access issues | Medium     | Rigorous performance testing with realistic workload simulations prior to production; Continuous monitoring of storage latency and throughput; Optimization of ODF parameters (e.g., Bluestore settings, OSD placement); Phased migration to dedicated ODF hardware as planned. |
|                        | Data migration issues (from OpenStack or during ODF transition to dedicated nodes)   | Data loss, extended downtime, migration failures     | Low        | Development of a detailed, step-by-step migration plan with rollback procedures; Use of proven and tested migration tools and scripts; Pilot migrations for representative workloads; Full backups before initiating migration activities; Thorough data validation post-migration. |
|                        | OpenShift Virtualization operator or component bugs/limitations                  | VM instability, feature unavailability, security vulnerabilities | Medium     | Strict adherence to Red Hat supported versions and patch levels for OpenShift Virtualization and its dependencies; Active engagement with Red Hat support and community forums for known issues; Thorough testing of virtualization features in a non-production environment.       |
| **Operational Risks**  |                                                                                  |                                                      |            |                                                                                                                                                                                                                                                                               |
|                        | Lack of skilled personnel for OpenShift/ODF/Virtualization management            | Misconfigurations, slow incident resolution, inefficient operations | Medium     | Comprehensive training programs for operations staff covering OpenShift, ODF, and KubeVirt; Development of detailed operational runbooks and SOPs; Establishment of a clear Red Hat support escalation path; Phased handover to the operations team with ongoing SME support. |
|                        | Backup and Recovery failures or inadequacies                                     | Data loss, inability to meet RTO/RPO objectives       | Low        | Regular, automated testing of backup and restore procedures (including full DR drills); Versioning and offsite storage of etcd and critical configuration backups; Validation of OADP compatibility with application workloads; Clear documentation of recovery steps.              |
|                        | Monitoring gaps leading to undetected issues or delayed alerting                   | Prolonged outages, performance degradation, security incidents missed | Medium     | Implementation of a comprehensive monitoring solution covering all layers (hardware, OS, OpenShift platform, ODF, VM workloads); Definition of granular alerting rules with appropriate thresholds; Regular review and tuning of dashboards and alerts; Integration with existing enterprise monitoring. |
| **Resource Risks**     |                                                                                  |                                                      |            |                                                                                                                                                                                                                                                                               |
|                        | Delays in hardware procurement or delivery                                       | Project timeline slippage, delayed service availability | Low        | Early engagement with procurement teams and hardware vendors; Establishment of clear delivery timelines and SLAs; Identification of alternative suppliers or hardware models if feasible.                                                                                 |
|                        | Insufficient infrastructure resources (IPs, VLANs, SAN storage, power/cooling)   | Inability to scale, deployment failures, service limitations | Medium     | Thorough resource auditing and capacity planning during the design phase; Reservation of required resources in advance; Phased deployment strategy to align with resource availability; Regular review of resource utilization and proactive scaling.                       |
| **Security Risks**     |                                                                                  |                                                      |            |                                                                                                                                                                                                                                                                               |
|                        | Misconfiguration of security features (SEV, NetworkPolicies, RBAC, SCCs)         | Unauthorized access, data breaches, compliance violations | Medium     | Security-focused design reviews and threat modeling; Adherence to security best practices from Red Hat and industry standards (e.g., CIS Benchmarks); Implementation of least privilege principles for all access; Regular security audits and penetration testing.             |
|                        | Vulnerabilities in OpenShift, ODF, KubeVirt, or underlying OS/firmware           | System compromise, data exfiltration, service disruption | Medium     | Implementation of a robust patch management process with timely application of security updates; Continuous vulnerability scanning of container images and host systems; Subscription to Red Hat security advisories; Secure software development lifecycle (SSDLC) for any custom components. |
| **Project Risks**      |                                                                                  |                                                      |            |                                                                                                                                                                                                                                                                               |
|                        | Scope creep or significant changes in requirements mid-project                   | Budget overruns, timeline extensions, resource strain | Medium     | Establishment of a formal change management process with impact assessment for all requested changes; Clear initial scope definition and stakeholder agreement; Regular project status reviews to manage expectations.                                                       |
|                        | Unforeseen technical challenges or integration complexities                      | Project delays, increased costs, potential for rework  | Medium     | Adoption of an agile project management approach with iterative development and testing cycles; Allocation of contingency buffers in timelines and budgets; Proactive technical deep-dives and proof-of-concepts for high-risk areas.                                           |

#### B.2 Dependencies

The successful implementation and ongoing operation of the OpenShift Virtualization platform rely on several key internal and external dependencies. These dependencies must be coordinated with the installation workflow described in [Section 5.1](#51-installation-workflow-overview) and ongoing operations detailed in [Chapter 8](#8-day-2-operations).

##### B.2.1 External Dependencies

1. **Hardware Vendors:**
    - Servers (Compute, Storage): For timely delivery, firmware updates, driver support, and hardware troubleshooting.
    - Network Equipment (Switches, NICs): For compatibility, firmware, and driver support.
    - Storage Arrays (FC-SAN): For IBM FlashSystem support, microcode updates, and troubleshooting.
2. **Software Vendors:**
    - Red Hat: Crucial for OpenShift Container Platform, OpenShift Data Foundation, RHEL subscriptions, official documentation, security advisories, bug fixes, and enterprise support services.
3. **Network Infrastructure Providers:**
    - Core Network: Availability and correct configuration of physical network infrastructure, including switches, routers, and firewalls supporting LACP, VLAN tagging, and required routing protocols.
    - Internet Service Provider (ISP): Reliable internet connectivity for access to external container registries, software repositories, and Red Hat support services (unless fully air-gapped).
4. **Utility Services:**
    - Data Center Facilities: Reliable power, cooling, and physical security for the hardware.
5. **Standard Services:**
    - DNS Services: Robust and reliable internal (and potentially external) DNS resolution for cluster hostnames, API endpoints, service discovery, and application FQDNs.
    - NTP Services: Accurate and consistent time synchronization across all cluster nodes and the jumphost, critical for certificate validity, log correlation, and distributed system operations.
    - Certificate Authority (CA): If using an internal or third-party CA, its availability for issuing and renewing SSL/TLS certificates for OpenShift services and applications.

##### B.2.2 Internal Dependencies (Organizational & Technical)

1. **Network Team:**
    - Provisioning and configuration of VLANs, IP subnets, and LACP on physical switches.
    - Firewall rule implementation and management for ingress/egress traffic.
    - Assistance with network troubleshooting and performance analysis.
2. **Storage Team:**
    - Provisioning and zoning of FC-SAN LUNs for ODF.
    - Monitoring and management of the FC-SAN infrastructure.
    - Expertise in IBM Storage Virtualize for troubleshooting and optimization.
3. **Server/Infrastructure Team:**
    - Physical racking, cabling, and initial power-on of servers.
    - Configuration of out-of-band management (BMC/IPMI).
    - Assistance with hardware diagnostics and component replacement.
4. **Security Team:**
    - Approval of security designs and configurations (e.g., SEV enablement, NetworkPolicies, SCCs).
    - Conducting security assessments, vulnerability scans, and penetration tests.
    - Guidance on compliance with organizational security policies and industry regulations.
5. **Operations Team:**
    - Availability for training on OpenShift, ODF, and KubeVirt.
    - Assumption of day-to-day operational responsibilities post-handover, including monitoring, patching, backup, and user support.
6. **Project Management & Stakeholders:**
    - Ongoing project sponsorship, budget allocation, and resource prioritization.
    - Approvals for project milestones and change requests.
7. **Application Development Teams (Platform Consumers):**
    - Providing clear requirements for VM resources and performance.
    - Participation in UAT and performance testing of their virtualized applications.
    - Adherence to platform usage guidelines and security policies.
8. **Existing Infrastructure & Tooling:**
    - Jumphost: Properly configured and maintained with required tools (Ansible, `oc` CLI, Helm, etc.) and network connectivity.
    - Ansible Environment: Availability of the correct Ansible version, collections, and playbooks/roles stored in a version control system (e.g., Git).
    - Identity Management (IdM): Integration with enterprise IdM (e.g., LDAP, Active Directory) for user authentication and authorization if required.
    - Monitoring & Logging Systems: Potential integration with existing centralized monitoring (e.g., Prometheus federation, SNMP traps) and logging (e.g., Splunk, ELK) systems.

---

*End of Document*