# Kernel Crash Dumps (kdump)

## Overview

- **kdump** captures kernel memory when the system crashes
- Uses **kexec** to boot a secondary "capture kernel" from reserved memory
- Crash dump stored in **vmcore** file for analysis

## Installation and Setup

```bash
# Install (usually pre-installed)
yum install kexec-tools

# Enable and start kdump service
systemctl enable kdump
systemctl start kdump
systemctl status kdump
```

## Memory Reservation

Edit `/etc/default/grub`:

```bash
GRUB_CMDLINE_LINUX="... crashkernel=auto ..."
```

Regenerate GRUB config:

```bash
# BIOS systems
grub2-mkconfig -o /boot/grub2/grub.cfg

# UEFI systems
grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
```

Reboot to apply changes.

## kdumpctl Commands

```bash
# Check status
kdumpctl status

# Show reserved memory
kdumpctl showmem

# Rebuild initramfs
kdumpctl rebuild

# Propagate SSH keys (for remote dump targets)
kdumpctl propagate
```

## Configuration (/etc/kdump.conf)

### Dump Target Options

```bash
# Local filesystem (default)
path /var/crash

# Raw partition
raw /dev/sda5

# NFS share
nfs server:/export/crash

# SSH/SCP remote
ssh user@server
sshkey /root/.ssh/kdump_id_rsa
path /var/crash
```

### Core Collection Settings

```bash
# Default collection with compression and filtering
core_collector makedumpfile -l --message-level 1 -d 31

# For SSH targets, use scp
core_collector scp
```

### Compression Options (-c, -l, -p)

| Option | Algorithm |
|--------|-----------|
| `-c` | zlib |
| `-l` | lzo |
| `-p` | snappy |

### Dump Levels (-d)

| Level | Description |
|-------|-------------|
| 0 | Include all pages |
| 1 | Exclude zero pages |
| 31 | Exclude zero, cache, user, free pages (smallest) |

## Crash Dump Files

Default location: `/var/crash/`

```bash
ls /var/crash/
# <hostname>-<date>/
#   vmcore           - Full crash dump
#   vmcore-dmesg.txt - Kernel log at crash time
#   kexec-dmesg.log  - Capture kernel log
```

Send `vmcore-dmesg.txt` first to Red Hat Support for preliminary analysis.

## Triggering Crash Dumps

### Test with SysRq

```bash
# Enable SysRq
echo 1 > /proc/sys/kernel/sysrq

# Trigger crash (WARNING: crashes system!)
echo c > /proc/sysrq-trigger
```

### Configure Panic Triggers

```bash
# Panic on OOM (out of memory)
echo 1 > /proc/sys/vm/panic_on_oom

# Panic on hung tasks
echo 1 > /proc/sys/kernel/hung_task_panic

# Panic on soft lockup
echo 1 > /proc/sys/kernel/softlockup_panic

# Panic on NMI
echo 1 > /proc/sys/kernel/panic_on_io_nmi
```

### Make Permanent (/etc/sysctl.conf)

```bash
vm.panic_on_oom=1
kernel.hung_task_panic=1
kernel.softlockup_panic=1
kernel.panic_on_io_nmi=1
kernel.sysrq=1

# Apply
sysctl -p
```

## SysRq Commands

Key sequence: `Alt + PrintScreen + [key]`

| Key | Action |
|-----|--------|
| m | Dump memory info |
| t | Dump thread state |
| p | Dump CPU registers |
| c | Crash system |
| s | Sync filesystems |
| u | Remount read-only |
| b | Reboot |
| o | Power off |
| f | Start OOM killer |
| w | Dump hung processes |

## Early kdump (RHEL 8+)

Captures crashes during early boot:

```bash
# Ensure kdump initramfs exists
kdumpctl rebuild

# Rebuild boot initramfs with early kdump
dracut -f --add earlykdump

# Add kernel parameter
grubby --update-kernel=ALL --args="rd.earlykdump"
```

## Analyzing Crash Dumps

```bash
# Install crash utility
yum install crash

# Analyze vmcore
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore

# Inside crash shell
crash> bt       # backtrace
crash> log      # kernel log
crash> ps       # process list
crash> sys      # system info
crash> help     # available commands
```

## Web Console

kdump can also be configured via the web console (cockpit):
1. Access https://hostname:9090
2. Navigate to "Kernel Dump" tab
3. Configure settings graphically

## Troubleshooting kdump

| Issue | Check |
|-------|-------|
| kdump not starting | `kdumpctl status`, check crashkernel param |
| No dump generated | Check target permissions, disk space |
| Dump incomplete | Increase reserved memory |
| SSH dump fails | `kdumpctl propagate`, check SSH keys |

## Red Hat Resources

- Kdump Helper Lab: https://access.redhat.com/labs/kdumphelper/
- Kernel Oops Analyzer: https://access.redhat.com/labs/kerneloopsanalyzer/