// Module included in the following assemblies:
//
//

[id="looking-inside-openshift-nodes_{context}"]
= Looking inside {product-title} nodes

Directly accessing a node is strongly discouraged. Nodes are meant to be managed entirely from the cluster and that are considered tainted if you log in to a node and change anything. That said, there might be times when you want to troubleshoot a problem on a node or simply go onto a node in a test environment to see how things work.

For debugging purposes, the oc debug command lets you go inside any pod and look around. For nodes, in particular, you open a tools pod on the node, then chroot to the node’s host filesystem. At that point, you are effectively working on the node. Here’s how to do that:

----
$ oc get nodes

NAME                                     STATUS  ROLES  AGE    VERSION

ip-10-0-0-1.us-east-2.compute.internal   Ready   worker 3h19m  v1.25.0

ip-10-0-0-39.us-east-2.compute.internal  Ready   master 3h37m  v1.25.0

…

$ oc debug nodes/ip-10-0-138-39.us-east-2.compute.internal

Starting pod/ip-10-0-138-39us-east-2computeinternal-debug …​
----

----
$ oc debug nodes/ip-10-0-138-39.us-east-2.compute.internal

Starting pod/ip-10-0-138-39us-east-2computeinternal-debug …​

To use host binaries, run chroot /host

If you don’t see a command prompt, try pressing enter.

sh-4.3#
----

As noted, you can change to the root of the node’s filesystem by typing chroot /host and running commands from the host on that filesystem as though you were logged in directly from the host. Here are some examples of commands you can run to see what is happening on the node:

* crictl: This CRI-O client command provides many of the same operations for examining images images and containers that the docker CLI offers the Docker Container Engine. One difference is that crictl can also act on pods. If you are debugging issues with containers run the {product-title} users or the {product-title} control plane, crictl is the best tool to use.
* podman: Provides many of the same features as crictl and docker CLI tools, but requires no container engine. On a node, podman can be useful for debugging container issues if the CRI-O runtime isn’t working.
* skopeo: Copy, delete, and inspect container images with skopeo.
* rpm-ostree: Use e.g. rpm-ostree status to look at the operating system state.
* journalctl: The standard journalctl command can be very useful for querying the system journal for messages that provide information about applications running on the system.

Because the nodes are {op-system} Linux-based systems, you can use standard Linux commands to explore the nodes as well. These include ps, netstat, ip, route, rpm, and many others. You can change to the /etc directory on the host and look into configuration files for services running directly on the host. For example, look at /etc/crio/crio.conf for CRI-O settings, /etc/resolv.conf for DNS server settings, and /etc/ssh for SSH service configuration and keys.

If you are unable to reach the nodes with oc debug, because something is wrong with the {product-title} cluster, you might be able to debug the nodes by setting up a bastion host on the cluster. For information on setting up a bastion host for {product-title}, see https://github.com/eparis/ssh-bastion[ssh-bastion].
