HowToCodePage

Monitoring and troubleshooting

ps and top for a snapshot, /proc and /sys for the details, strace for what a process is asking for, lsof for what it holds, and how to diagnose a hung process.

From a symptom to a process

# who is using the CPU and memory right now
top -b -n1 | head -20
ps -eo pid,ppid,stat,pcpu,pmem,rss,etime,comm --sort=-pmem | head

# memory: RSS is resident, VSZ is virtual and often meaningless
smem -rs rss | head -10
pmap -x 12345 | tail -3

# I/O: who is reading and writing
iostat -xz 1 3
pidstat -d 1 5
iotop -oP

# open files and sockets held by a process
lsof -p 12345 | wc -l
lsof -i :8080
SymptomFirst checkThen
High CPUtop -H for the busy threadperf top or a profiler
High memoryps sorted by RSSHeap profile or pmap for mapping growth
Disk fulldf -h, du -xsh *Find deleted-but-open files
I/O waitiostat -xLook for a saturated device or a failing disk
Process unresponsivecat /proc/PID/stackgdb for a thread backtrace
Port already in usess -tlnpFind the owning PID and decide
Too many open filesCount descriptors with lsofRaise the limit or fix the leak

/proc and /sys as the ground truth

cat /proc/12345/status | grep -E "State|VmRSS|Threads|FDSize"
cat /proc/12345/limits | grep -i "open files"
ls -l /proc/12345/fd | head                # what the process has open
cat /proc/12345/wchan; echo                # what kernel function it is blocked in
cat /proc/12345/stack                      # needs privilege; kernel stack
cat /proc/12345/io                         # bytes read and written

cat /proc/meminfo | grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree"
cat /proc/loadavg
cat /proc/pressure/io                      # pressure stall information
  • /proc/PID/fd shows every descriptor, including files that were deleted while still open — a frequent cause of a full disk with nothing to find.
  • wchan names the kernel function where a thread is sleeping, which points directly at the wait reason.
  • /proc/pressure reports the share of time tasks were stalled on CPU, memory or I/O, which is far more useful than utilisation alone.
  • Everything here is per-process and needs no extra tooling installed, which makes it reliable on a minimal host.

Diagnosing a hung process

  1. Confirm it is alive and in which state: ps -o pid,stat,wchan,etime -p PID.
  2. Check what it is waiting on: /proc/PID/wchan and /proc/PID/stack.
  3. Look at its descriptors to see what it holds open.
  4. Attach a debugger and capture all thread backtraces before changing anything.
  5. Trace the system calls to see the last thing it asked for.
  6. If it is in D state, suspect the device or the file system, not the application.
ps -o pid,stat,wchan:32,etime -p 12345
cat /proc/12345/wchan; echo
gdb -p 12345 -batch -ex "thread apply all bt" 2>/dev/null | head -80
strace -p 12345 -f -tt -T -o /tmp/attach.txt &
sleep 3; kill %1

# kernel-side: hung task reports
dmesg -T | tail -40 | grep -i "blocked for more than"
⚠️
Capture evidence before restarting. A restarted process destroys the only copy of its state, and a problem that happens once a week will not recur on demand. Dump the backtraces, the descriptors and a short system call trace first.

FAQ

Why is free memory so low but nothing is swapping?
Unused memory is wasted memory. Linux uses free RAM for the page cache, and MemAvailable is the figure that matters, not MemFree.
What does D state mean?
Uninterruptible sleep, usually a blocked I/O request. The thread cannot be interrupted, so a process stuck there cannot even be killed cleanly until the I/O completes or times out.

CPU scheduling: how the CPU is shared Storage and file systems

Last refreshed 2026-09-18.