# Linux Storage Stack and Troubleshooting

## Storage Stack Layers (Top to Bottom)

1. **VFS** - Virtual File System (common API)
2. **File Systems** - XFS, ext4, etc.
3. **Device Mapper** - LVM, LUKS, RAID
4. **Block Layer / Schedulers** - I/O ordering
5. **SCSI Mid-layer** - Protocol abstraction
6. **Low-level Drivers** - Hardware communication

## VFS and Caching

```bash
# View memory/cache usage
free -h
cat /proc/meminfo

# Drop caches (1=pagecache, 2=dentries/inodes, 3=all)
echo 3 > /proc/sys/vm/drop_caches

# Cache tunables in /proc/sys/vm/
```

## Device Mapper

```bash
# List device mapper devices
dmsetup ls

# Show mapping table
dmsetup table /dev/mapper/myvg1-mylv1

# Device info
dmsetup info /dev/mapper/myvg1-mylv1

# View device major:minor
ls -l /dev/mapper/
ls -l /dev/dm-*
```

## LVM Commands

```bash
# Physical volumes
pvs          # list PVs
pvdisplay    # detailed view
pvcreate /dev/vdb1

# Volume groups
vgs          # list VGs
vgdisplay    # detailed view
vgcreate myvg /dev/vdb1 /dev/vdb2

# Logical volumes
lvs          # list LVs
lvdisplay    # detailed view
lvcreate -L 10G -n mylv myvg
lvextend -L +5G /dev/myvg/mylv
lvreduce -L -5G /dev/myvg/mylv  # dangerous!
```

## Disk Schedulers

```bash
# View current scheduler
cat /sys/block/sda/queue/scheduler

# Change scheduler temporarily
echo mq-deadline > /sys/block/sda/queue/scheduler

# Available schedulers:
# - none      : FIFO, simple
# - mq-deadline : sorted batches, good for reads
# - kyber     : latency-focused, self-tuning
# - bfq       : bandwidth fairness, low latency
```

## Multipath

```bash
# Show multipath topology
multipath -ll

# Flush and reconfigure
multipath -F
multipath -r

# Service
systemctl status multipathd

# Config file
/etc/multipath.conf
```

## Stratis Storage

```bash
# Install and enable
yum install stratisd stratis-cli
systemctl enable --now stratisd

# Prepare devices
wipefs --all /dev/vdb

# Create pool
stratis pool create my-pool /dev/vdb /dev/vdc

# List pools and block devices
stratis pool list
stratis blockdev

# Create filesystem
stratis fs create my-pool my-fs

# Mount
mount /dev/stratis/my-pool/my-fs /mnt

# Create snapshot
stratis fs snapshot my-pool my-fs my-fs-snap

# List filesystems
stratis fs list my-pool

# Destroy filesystem
stratis fs destroy my-pool my-fs
```

## XFS Repair

```bash
# Check XFS filesystem (read-only)
xfs_repair -n /dev/vda1

# Repair XFS (unmount first!)
umount /dev/vda1
xfs_repair /dev/vda1

# Force repair if log is dirty
xfs_repair -L /dev/vda1  # WARNING: loses log data
```

## ext4 Repair

```bash
# Check/repair (unmount first!)
fsck.ext4 /dev/vda1

# Force check even if clean
fsck.ext4 -f /dev/vda1

# Auto-fix all issues
fsck.ext4 -y /dev/vda1
```

## Block Device Info

```bash
# List block devices
lsblk
lsblk -f  # show filesystems

# Block device attributes
blkid

# Disk usage
df -h
df -i  # inodes

# Detailed partition info
fdisk -l /dev/sda
parted /dev/sda print
```

## I/O Monitoring

```bash
# Real-time I/O stats
iostat -x 1

# Per-process I/O
iotop

# Block trace
blktrace -d /dev/sda -o trace
blkparse -i trace
```

## Common Storage Issues

| Symptom | Possible Cause | Diagnosis |
|---------|---------------|-----------|
| Mount fails | Wrong filesystem type | `blkid`, `file -s /dev/xxx` |
| Read-only mount | Filesystem errors | `dmesg`, `journalctl -k` |
| Slow I/O | Wrong scheduler | Check `/sys/block/*/queue/scheduler` |
| No space left | Full disk or inodes | `df -h`, `df -i` |
| LV not visible | VG not activated | `vgchange -ay` |
| Multipath issues | Path down | `multipath -ll` |

## Emergency Read-Only Remount

```bash
# Remount root read-only
mount -o remount,ro /

# Remount read-write
mount -o remount,rw /
```