## Step 0: Obtain RHCOS Images Using CoreOS Installer

The [CoreOS Installer](https://coreos.github.io/coreos-installer/) is a utility that can download, verify, and customize CoreOS images. We'll use it to obtain the required files.

### Install CoreOS Installer

First, install the CoreOS Installer:

```bash
# Install using cargo (Rust package manager)
cargo install coreos-installer

# Or use a container image
podman pull quay.io/coreos/coreos-installer:release
alias coreos-installer='podman run --rm --pull=always --privileged -v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data -w /data quay.io/coreos/coreos-installer:release'
```

### Download RHCOS ISO and Components

Download the RHCOS ISO for s390x architecture:

```bash
# List available RHCOS streams to see available versions
coreos-installer list-stream --stream=rhcos-4.16

# Download the RHCOS ISO
coreos-installer download -s rhcos-4.16 -p s390x -f iso -C ./rhcos/

# Download the RHCOS kernel, initramfs, and rootfs separately
coreos-installer download -s rhcos-4.16 -p s390x -f kernel -C ./rhcos/
coreos-installer download -s rhcos-4.16 -p s390x -f initramfs -C ./rhcos/
coreos-installer download -s rhcos-4.16 -p s390x -f rootfs -C ./rhcos/
```

This will download all necessary components to the `./rhcos/` directory.

### Create a Custom ISO with Embedded Ignition (Optional)

If you want to create a custom ISO with an embedded Ignition config:

```bash
# First, create or obtain your ignition config
# For example, for a worker node:
oc extract -n openshift-machine-api secret/worker-user-data --keys=userData --to=worker.ign

# Embed the ignition config into the ISO
coreos-installer iso ignition embed -i worker.ign ./rhcos/rhcos-*.s390x.iso -o ./rhcos/rhcos-custom.s390x.iso

# You can also add live kernel arguments
coreos-installer iso kargs modify ./rhcos/rhcos-custom.s390x.iso \
  --append "ip=10.153.1.20::10.153.1.1:255.255.255.0:RHCOSLPAR01.test02.kvant.cloud:vlan1171:none nameserver=10.153.1.1 rd.znet=qeth,0.0.0001,0.0.0002,0.0.0003,layer2=1,portno=0,portname=enc1 vlan=vlan1171:enc1"
```

### Create PXE Boot Files (Alternative Method)

For PXE booting, you can create an initrd with embedded ignition:

```bash
# Create an initrd with the ignition config
coreos-installer pxe ignition wrap -i worker.ign -o worker.ign.img

# This can be appended to the initial ramdisk during PXE boot
# Example: use worker.ign.img as an additional initrd for LPAR boot
```

## Step 1: Extract and Prepare Boot Files

mount the RHCOS ISO and extract necessary files:

```bash
# Mount the ISO
mkdir -p ~/mnt/rhcos
sudo mount -o loop ./rhcos/rhcos-416.94.202410211619-0-live.s390x.iso ~/mnt/rhcos

# Create a directory for the extracted files
mkdir -p ~/boot-rhcos
cp -r ~/mnt/rhcos/* ~/boot-rhcos/
```

## Step 2: Set Up Web Server on Jumphost

Set up a web server on the jumphost (10.153.7.108) to serve the required files:

```bash
# On jumphost (10.153.7.108)
sudo mkdir -p /var/www/html/rhcos
sudo mkdir -p /var/www/html/ignition

# Copy rootfs image to HTTP server
sudo cp ~/mnt/rhcos/images/pxeboot/rootfs.img /var/www/html/rhcos/rhcos-416.94.202410211619-0-live-rootfs.s390x.img

# Make sure Apache is running
sudo systemctl start httpd
```

## Step 3: Create Boot Files

### Create generic.ins file

This file already exists in the ISO, but verify its contents match the following:

```
images/kernel.img 0x00000000
images/initrd.img 0x02000000
images/genericdvd.prm 0x00010480
images/initrd.addrsize 0x00010408
```

If you need to modify it, create a custom INS file:

```bash
cat > ~/boot-rhcos/custom.ins << 'EOF'
images/pxeboot/kernel.img 0x00000000
images/pxeboot/initrd.img 0x02000000
images/custom.prm 0x00010480
images/initrd.addrsize 0x00010408
EOF
```

### Create custom.prm file

Create a custom parameter file with network settings using the jumphost as the HTTP server:

```bash
cat > ~/boot-rhcos/images/custom.prm << 'EOF'
rd.neednet=1 console=ttysclp0 coreos.inst.install_dev=sda 
coreos.live.rootfs_url=http://10.153.7.108/rhcos/rhcos-416.94.202410211619-0-live-rootfs.s390x.img 
coreos.inst.ignition_url=http://10.153.7.108/ignition/worker.ign 
ip=10.153.1.20::10.153.1.1:255.255.255.0:RHCOSLPAR01.test02.kvant.cloud:vlan1171:none
nameserver=10.153.1.1 
cio_ignore=all,!condev 
zfcp.allow_lun_scan=0 
rd.znet=qeth,0.0.0001,0.0.0002,0.0.0003,layer2=1,portno=0,portname=enc1
vlan=vlan1171:enc1
rd.zfcp=0.0.YOUR_FCP_DEV,0xYOUR_WWPN,0xYOUR_LUN
EOF
```

Note: You need to replace the FCP parameters:
- `YOUR_FCP_DEV` - FCP device number (e.g., 4000)
- `YOUR_WWPN` - World Wide Port Name of your storage (e.g., 5005076300C213E9)
- `YOUR_LUN` - Logical Unit Number of your storage (e.g., 5022000000000000)

## Step 4: Create Ignition Config

For a worker node, create/obtain an ignition config:

```bash
# Example command to copy an existing worker.ign file
# Replace with actual source of your ignition config

# To retrieve the Master Ignition Configuration run the following command:
oc extract -n openshift-machine-api secret/master-user-data --keys=userData --to=-

# To retrieve the Worker Ignition Configuration run the following command:
oc extract -n openshift-machine-api secret/worker-user-data --keys=userData --to=-

sudo cp /path/to/your/worker.ign /var/www/html/ignition/worker.ign
sudo chmod 644 /var/www/html/ignition/worker.ign
```

## Step 5: FTP Boot Files to the LPAR

You need to make the boot files available to the LPAR, typically through FTP:

```bash
# Start FTP server if not already running
sudo systemctl start vsftpd

# Make sure files are in an accessible location
mkdir -p /var/ftp/pub/rhcos-boot
cp -r ~/boot-rhcos/* /var/ftp/pub/rhcos-boot/
chmod -R 755 /var/ftp/pub/rhcos-boot/
```

## Step 6: Boot the LPAR

From the HMC:

1. Select your LPAR
2. Select "Load" from the menu
3. Choose "FTP" as the source
4. Enter your FTP server details:
   - Host: 10.153.7.108 (your jumphost IP)
   - User/Password: FTP credentials
   - File location: `/pub/rhcos-boot/custom.ins` (or `generic.ins` if unchanged)
5. Click "OK" to start the boot process

## Step 7: Monitor Installation

Once booted, you can monitor the installation by:

1. Connecting to the LPAR console through the HMC
2. The installer should download the rootfs image and ignition config
3. It will then install RHCOS to the specified disk
4. If using an OpenShift ignition config, the node should automatically join your cluster

## Alternative Boot Method: Direct Console Input

As an alternative to creating files, you can input boot parameters directly at the console:

1. From the "Operating System Messages" console in HMC
2. Enter the following commands line by line:

```
ro ramdisk_size=40000 cio_ignore=all,!condev
rd.neednet=1 console=ttysclp0 coreos.inst.install_dev=sda
coreos.live.rootfs_url=http://10.153.7.108/rhcos/rhcos-416.94.202410211619-0-live-rootfs.s390x.img
coreos.inst.ignition_url=http://10.153.7.108/ignition/worker.ign
ip=10.153.1.20::10.153.1.1:255.255.255.0:RHCOSLPAR01.test02.kvant.cloud:vlan1171:none
nameserver=10.153.1.1
rd.znet=qeth,0.0.0001,0.0.0002,0.0.0003,layer2=1,portno=0,portname=enc1
vlan=vlan1171:enc1
zfcp.allow_lun_scan=0
rd.zfcp=0.0.YOUR_FCP_DEV,0xYOUR_WWPN,0xYOUR_LUN
generic.ins
.
```

Replace only the placeholders for the FCP parameters with your specific values.

## Troubleshooting

- If the boot fails, check console output for error messages
- Verify network connectivity from the LPAR to your jumphost (10.153.7.108)
- Ensure FCP devices are properly configured and visible
- Check that storage LUNs are accessible
- Verify the Ignition config is valid
- Check the web server logs on the jumphost for any failed requests

## Network Diagram

```
+-------------+           +-----------------+            +----------------+
|    LPAR     |           |  Network Switch |            |   Jumphost    |
|             |           |                 |            | 10.153.7.108  |
| 10.153.1.20 |-----------|                 |------------|                |
|             |           |                 |            | - HTTP Server  |
+-------------+           +-----------------+            | - FTP Server   |
                                                         +----------------+
```

## References

- [Red Hat Documentation: Boot Parameters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/automatically_installing_rhel/preparing-a-rhel-installation-on-64-bit-ibm-z_rhel-installer#customizing-boot-parameters_preparing-a-rhel-installation-on-64-bit-ibm-z)
- [Installing Red Hat OpenShift with LPAR on IBM Z](https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/gerald-hosch1/2024/07/26/installing-red-hat-openshift-with-lpar-on-ibm-z?communityKey=fd56de68-d38b-499b-a1f4-51010f4eee66)
- [Installing OCP in a Mainframe Z Series](https://www.redhat.com/en/blog/installing-ocp-in-a-mainframe-z-series)