Deep-Dive
TROUBLESHOOTING REFERENCE
Click any failure domain to explore diagnostic commands, root causes, and fixes.
HIGH LOAD & CPU
The load average number everyone misreads under pressure
❯uptime
load average: 8.42, 6.10, 4.05
❯nproc
🟠Load average > core count sustained over 5-15 min = the box is genuinely saturated, not a blip
🔵High load with low CPU% usually means processes stuck in D state (uninterruptible I/O wait) — check disk, not CPU
❯top -o %CPU
❯ps aux --sort=-%cpu | head -10
❯pidstat 2 5
Process States
🔵Z (zombie) — finished but parent hasn't called wait() to reap it
🟠D (uninterruptible sleep) — usually stuck on disk/network I/O, can't even be killed with -9
Fix
1
Zombies can't be killed directly — restart or fix the parent process
2
Many zombies + one parent PID = that parent has a reaping bug — file it as a real bug, not a one-off restart
MEMORY & THE OOM KILLER
"Available" memory almost never means what you think it means
total used free shared buff/cache available
Mem: 15Gi 9.2Gi 512Mi 128Mi 5.3Gi 5.8Gi
🟠Don't panic at low "free" — look at "available." A box with 500MB free and 5.8GB available is fine.
❯dmesg -T | grep -i "out of memory"
Out of memory: Killed process 8821 (java) total-vm:4200000kB
❯journalctl -k | grep -i oom
1
The killed process usually isn't the cause — it's just whatever the kernel's OOM-score heuristic picked
2
Check /var/log/messages or journal around the same timestamp for what was actually consuming memory beforehand
❯watch -n 60 'ps -o pid,rss,cmd -p 8821'
❯smem -tk
DISK & I/O
"No space left on device" when df says there's plenty free — yes, that's real
❯df -h
❯du -sh /var/* | sort -rh | head
❯lsof +L1
🟠If du and df disagree wildly, a process is holding a deleted file open — lsof +L1 finds it, restarting that process frees the space
❯df -i
❯find /var/spool -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head
❯iostat -xz 2
❯iotop -oPa
NETWORK ISSUES
DNS, routing, and firewalls — in that order, almost every time
❯dig billing-api.internal +short
❯cat /etc/resolv.conf
❯systemd-resolve --status
❯ip route get 10.20.4.5
❯traceroute -T -p 443 billing-api.internal
❯ss -tulpn | grep 8080
❯nft list ruleset
❯iptables -L -n -v --line-numbers
❯firewall-cmd --list-all
BOOT & SERVICE FAILURES
When the box won't come back up after a reboot — the scariest category, and usually the most methodical to fix
🔴Bad /etc/fstab entry (typo'd UUID, missing device) is the #1 cause of "boots to emergency shell"
🟠Corrupt initramfs after a kernel update that didn't rebuild it properly
From Emergency Shell
1
Mount root read-write: mount -o remount,rw /
2
Fix the offending line in /etc/fstab, or comment it out temporarily
3
systemctl reboot and confirm it comes up clean this time
❯systemctl --failed
❯systemctl status billing-worker -l
❯journalctl -u billing-worker -b
❯systemd-analyze blame
Rescue vs Emergency
🔵rescue.target — most local filesystems mounted, base system running
🔵emergency.target — bare minimum, root filesystem often read-only, use when rescue itself won't boot
Boot Into One
1
At GRUB, press e to edit, append systemd.unit=rescue.target to the kernel line
2
Ctrl+X or F10 to boot with that one-time change
Decision Guide
WHERE DO I LOOK FIRST?
A quick lookup for the first 60 seconds of any "the server is having issues" page.
| Symptom | Check First | Command |
|---|
| Everything feels slow | Load average vs core count | uptime + nproc |
| App crashed unexpectedly | OOM killer | dmesg -T | grep -i oom |
| Writes failing | Disk space and inodes | df -h && df -i |
| Can't reach another service | DNS resolution first | dig +short |
| Box won't come back after reboot | fstab and failed units | systemctl --failed |