////
Purpose
-------
This page gives an overview of Red Hat Enterprise Linux CoreOS (RHCOS)
////

= Red Hat Enterprise Linux CoreOS (RHCOS)

Red Hat Enterprise Linux CoreOS (RHCOS) represents the next generation of single-purpose container operating system technology by providing the quality standards of Red Hat Enterprise Linux (RHEL) with automated, remote upgrade features.

RHCOS is supported only as a component of OpenShift Container Platform {ocp_version} for all OpenShift Container Platform machines. RHCOS is the only supported operating system for OpenShift Container Platform control plane, or master, machines. While RHCOS is the default operating system for all cluster machines, {customer} can create compute machines, which are also known as worker machines, that use RHEL as their operating system. There are two general ways RHCOS is deployed in OpenShift Container Platform {ocp_version}:

* If {customer} installs its cluster on infrastructure that the cluster provisions, RHCOS images are downloaded to the target platform during installation, and suitable Ignition config files, which control the RHCOS configuration, are used to deploy the machines.

* If {customer} installs its cluster on infrastructure that {customer} manages, {customer} must follow the installation documentation to obtain the RHCOS images, generate Ignition config files, and use the Ignition config files to provision its machines.

== Key RHCOS features

The following list describes key features of the RHCOS operating system:

* *Based on RHEL*: The underlying operating system consists primarily of RHEL components. The same quality, security, and control measures that support RHEL also support RHCOS. For example, RHCOS software is in RPM packages, and each RHCOS system starts up with a RHEL kernel and a set of services that are managed by the systemd init system.

* *Controlled immutability*: Although it contains RHEL components, RHCOS is designed to be managed more tightly than a default RHEL installation. Management is performed remotely from the OpenShift Container Platform cluster. When {customer} sets up its RHCOS machines, {customer} can modify only a few system settings. This controlled immutability allows OpenShift Container Platform to store the latest state of RHCOS systems in the cluster so it is always able to create additional machines and perform updates based on the latest RHCOS configurations.

* *CRI-O container runtime*: Although RHCOS contains features for running the OCI- and libcontainer-formatted containers that Docker requires, it incorporates the CRI-O container engine instead of the Docker container engine. By focusing on features needed by Kubernetes platforms, such as OpenShift Container Platform, CRI-O can offer specific compatibility with different Kubernetes versions. CRI-O also offers a smaller footprint and reduced attack surface than is possible with container engines that offer a larger feature set. At the moment, CRI-O is the only engine available within OpenShift Container Platform clusters.

* *Set of container tools*: For tasks such as building, copying, and otherwise managing containers, RHCOS replaces the Docker CLI tool with a compatible set of container tools. The podman CLI tool supports many container runtime features, such as running, starting, stopping, listing, and removing containers and container images. The skopeo CLI tool can copy, authenticate, and sign images. {customer} can use the `crictl` CLI tool to work with containers and pods from the CRI-O container engine. While direct use of these tools in RHCOS is discouraged, {customer} can use them for debugging purposes.

* *rpm-ostree upgrades*: RHCOS features transactional upgrades using the `rpm-ostree` system. Updates are delivered by means of container images and are part of the OpenShift Container Platform update process. When deployed, the container image is pulled, extracted, and written to disk, then the bootloader is modified to boot into the new version. The machine will reboot into the update in a rolling manner to ensure cluster capacity is minimally impacted.

* *bootupd firmware and bootloader updater*: Package managers and hybrid systems such as `rpm-ostree` do not update the firmware or the bootloader. With `bootupd`, RHCOS users have access to a cross-distribution, system-agnostic update tool that manages firmware and boot updates in UEFI and legacy BIOS boot modes that run on modern architectures, such as x86_64, ppc64le, and aarch64.

For information about how to install bootupd, see the documentation for Updating the bootloader using bootupd for more information.

Updated through the Machine Config Operator: In OpenShift Container Platform, the Machine Config Operator handles operating system upgrades. Instead of upgrading individual packages, as is done with `yum` upgrades, `rpm-ostree` delivers upgrades of the OS as an atomic unit. The new OS deployment is staged during upgrades and goes into effect on the next reboot. If something goes wrong with the upgrade, a single rollback and reboot returns the system to the previous state. RHCOS upgrades in OpenShift Container Platform are performed during cluster updates.

For RHCOS systems, the layout of the rpm-ostree file system has the following characteristics:

* `/usr` is where the operating system binaries and libraries are stored and is read-only. We do not support altering this.

* `/etc`, `/boot`, `/var` are writable on the system but only intended to be altered by the Machine Config Operator.

* `/var/lib/containers` is the graph storage location for storing container images.

== Choosing how to configure RHCOS

RHCOS is designed to deploy on an OpenShift Container Platform cluster with a minimal amount of user configuration. In its most basic form, this consists of:

* Starting with a provisioned infrastructure, such as on AWS, or provisioning the infrastructure itself.

* Supplying a few pieces of information, such as credentials and cluster name, in an `install-config.yaml` file when running `openshift-install`.

Because RHCOS systems in OpenShift Container Platform are designed to be fully managed from the OpenShift Container Platform cluster after that, directly changing an RHCOS machine is discouraged. Although limited direct access to RHCOS machines cluster can be accomplished for debugging purposes, {customer} should not directly configure RHCOS systems. Instead, if {customer} needs to add or change features on its OpenShift Container Platform nodes, consider making changes in the following ways:

* *Kubernetes workload objects, such as DaemonSet and Deployment*: If {customer} needs to add services or other user-level features to its cluster, consider adding them as Kubernetes workload objects. Keeping those features outside of specific node configurations is the best way to reduce the risk of breaking the cluster on subsequent upgrades.

* *Day-2 customizations*: If possible, bring up a cluster without making any customizations to cluster nodes and make necessary node changes after the cluster is up. Those changes are easier to track later and less likely to break updates. Creating machine configs or modifying Operator custom resources are ways of making these customizations.

* *Day-1 customizations*: For customizations that {customer} must implement when the cluster first comes up, there are ways of modifying its cluster so changes are implemented on first boot. Day-1 customizations can be done through Ignition configs and manifest files during `openshift-install` or by adding boot options during ISO installs provisioned by the user.

Here are examples of customizations {customer} could do on day 1:

* *Kernel arguments*: If particular kernel features or tuning is needed on nodes when the cluster first boots.

* *Disk encryption*: If its security needs require that the root file system on the nodes are encrypted, such as with FIPS support.

* *Kernel modules*: If a particular hardware device, such as a network card or video card, does not have a usable module available by default in the Linux kernel.

* *Chronyd*: If {customer} wants to provide specific clock settings to its nodes, such as the location of time servers.

To accomplish these tasks, {customer} can augment the `openshift-install` process to include additional objects such as `MachineConfig` objects. Those procedures that result in creating machine configs can be passed to the Machine Config Operator after the cluster is up.

NOTE: The Ignition config files that the installation program generates contain certificates that expire after 24 hours, which are then renewed at that time. If the cluster is shut down before renewing the certificates and the cluster is later restarted after the 24 hours have elapsed, the cluster automatically recovers the expired certificates. The exception is that {customer} must manually approve the pending `node-bootstrapper` certificate signing requests (CSRs) to recover kubelet certificates. See the documentation for link:https://docs.openshift.com/container-platform/{ocp_version}/backup_and_restore/disaster_recovery/scenario-3-expired-certs.html[Recovering from expired control plane certificates] for more information.

== Choosing how to deploy RHCOS
Differences between RHCOS installations for OpenShift Container Platform are based on whether {customer} is deploying on an infrastructure provisioned by the installer or by the user:

* *Installer-provisioned*: Some cloud environments offer pre-configured infrastructures that allow {customer} to bring up an OpenShift Container Platform cluster with minimal configuration. For these types of installations, {customer} can supply Ignition configs that place content on each node so it is there when the cluster first boots.

* *User-provisioned*: If {customer} is provisioning its own infrastructure, {customer} have more flexibility in how {customer} add content to a RHCOS node. For example, {customer} could add kernel arguments when {customer} boots the RHCOS ISO installer to install each system. However, in most cases where configuration is required on the operating system itself, it is best to provide that configuration through an Ignition config.

The Ignition facility runs only when the RHCOS system is first set up. After that, Ignition configs can be supplied later using the machine config.

== Ignition
Ignition is the utility that is used by RHCOS to manipulate disks during initial configuration. It completes common disk tasks, including partitioning disks, formatting partitions, writing files, and configuring users. On first boot, Ignition reads its configuration from the installation media or the location that {customer} specifies and applies the configuration to the machines.

Whether {customer} are installing its cluster or adding machines to it, Ignition always performs the initial configuration of the OpenShift Container Platform cluster machines. Most of the actual system setup happens on each machine itself. For each machine, Ignition takes the RHCOS image and boots the RHCOS kernel. Options on the kernel command line identify the type of deployment and the location of the Ignition-enabled initial RAM disk (initramfs).

=== How Ignition works
To create machines by using Ignition, {customer} needs Ignition config files. The OpenShift Container Platform installation program creates the Ignition config files that {customer} needs to deploy its cluster. These files are based on the information that {customer} provides to the installation program directly or through an `install-config.yaml` file.

The way that Ignition configures machines is similar to how tools like cloud-init or Linux Anaconda kickstart configure systems, but with some important differences:

* Ignition runs from an initial RAM disk that is separate from the system {customer} are installing to. Because of that, Ignition can repartition disks, set up file systems, and perform other changes to the machine’s permanent file system. In contrast, cloud-init runs as part of a machine init system when the system boots, so making foundational changes to things like disk partitions cannot be done as easily. With cloud-init, it is also difficult to reconfigure the boot process while {customer} are in the middle of the node boot process.

* Ignition is meant to initialize systems, not change existing systems. After a machine initializes and the kernel is running from the installed system, the Machine Config Operator from the OpenShift Container Platform cluster completes all future machine configuration.

* Instead of completing a defined set of actions, Ignition implements a declarative configuration. It checks that all partitions, files, services, and other items are in place before the new machine starts. It then makes the changes, like copying files to disk that are necessary for the new machine to meet the specified configuration.

* After Ignition finishes configuring a machine, the kernel keeps running but discards the initial RAM disk and pivots to the installed system on disk. All of the new system services and other features start without requiring a system reboot.

* Because Ignition confirms that all new machines meet the declared configuration, {customer} cannot have a partially configured machine. If a machine setup fails, the initialization process does not finish, and Ignition does not start the new machine. its cluster will never contain partially configured machines. If Ignition cannot complete, the machine is not added to the cluster. {customer} must add a new machine instead. This behavior prevents the difficult case of debugging a machine when the results of a failed configuration task are not known until something that depended on it fails at a later date.

* If there is a problem with an Ignition config that causes the setup of a machine to fail, Ignition will not try to use the same config to set up another machine. For example, a failure could result from an Ignition config made up of a parent and child config that both want to create the same file. A failure in such a case would prevent that Ignition config from being used again to set up an other machines until the problem is resolved.

* If {customer} has multiple Ignition config files, {customer} gets a union of that set of configs. Because Ignition is declarative, conflicts between the configs could cause Ignition to fail to set up the machine. The order of information in those files does not matter. Ignition will sort and implement each setting in ways that make the most sense. For example, if a file needs a directory several levels deep, if another file needs a directory along that path, the later file is created first. Ignition sorts and creates all files, directories, and links by depth.

* Because Ignition can start with a completely empty hard disk, it can do something cloud-init cannot do: set up systems on bare metal from scratch using features such as PXE boot. In the bare metal case, the Ignition config is injected into the boot partition so that Ignition can find it and configure the system correctly.

=== The Ignition sequence

The Ignition process for an RHCOS machine in an OpenShift Container Platform cluster involves the following steps:

* The machine gets its Ignition config file. Control plane machines (also known as the master machines) get their Ignition config files from the bootstrap machine, and worker machines get Ignition config files from a master.

* Ignition creates disk partitions, file systems, directories, and links on the machine. It supports RAID arrays but does not support LVM volumes.

* Ignition mounts the root of the permanent file system to the /sysroot directory in the initramfs and starts working in that /sysroot directory.

* Ignition configures all defined file systems and sets them up to mount appropriately at runtime.

* Ignition runs systemd temporary files to populate required files in the /var directory.

* Ignition runs the Ignition config files to set up users, systemd unit files, and other configuration files.

* Ignition unmounts all components in the permanent system that were mounted in the initramfs.

* Ignition starts up the init process of the new machine, which in turn starts up all other services on the machine that run during system boot.

At the end of this process, the machine is ready to join the cluster and does not require a reboot.

