HANDBOOK · JAVA THREAD DUMP ANALYSIS

Notes on Thread Dump analysis

Capture and analyze Java thread dumps — detect deadlocks, lock contention, pool starvation, and blocked I/O in production.

16
Modules
6
War Stories
6
Labs
jstack & jcmd Thread.print workflows
Deadlock and BLOCKED thread patterns
Tomcat, Spring, HikariCP thread analysis
Correlate with JFR and strace

When you need a snapshot of JVM state

Deadlocks

jstack reports Java deadlocks explicitly — instant RCA for hung services.

Pool Starvation

See all 200 threads BLOCKED on one lock or I/O wait.

Fast & Safe

Thread dumps have minimal overhead — capture every few minutes during incidents.

17 MODULES

Guide sections

Module 1

Introduction to Thread Dumps

What thread dumps are and when to capture them.

Module 2

Why Every SRE Needs Thread Dump Skills

Hangs and latency without error logs — thread dumps reveal the wait.

Module 3

Java Thread Fundamentals

Platform vs virtual threads, thread pools, and monitors.

Module 4

Capturing Thread Dumps

jstack, jcmd, kill -3, and automation.

Module 5

Reading Thread Dump Format

Thread headers, stack frames, and lock sections.

Module 6

Thread States Deep Dive

RUNNABLE, BLOCKED, WAITING, TIMED_WAITING meanings.

Module 7

Deadlock Detection

Automatic jstack detection and manual cycle analysis.

Module 8

Lock Contention Analysis

Many threads BLOCKED on one monitor — the hot lock.

Module 9

Thread Pool Starvation

Tomcat, @Async, ForkJoin, and custom executors.

Module 10

Correlating with JFR and strace

Three-layer diagnosis: JVM threads, JFR events, kernel syscalls.

Module 11

Kubernetes Thread Dump Capture

kubectl exec and ephemeral debug containers.

Module 12

Spring Boot Thread Issues

@Async, @Scheduled, Tomcat, WebFlux event loops.

Module 13

Database & I/O Blocked Threads

JDBC, HTTP client, and messaging waits in stack traces.

Module 14

Production Best Practices

Capture frequency, automation, and storage.

Module 15

SRE Incident War Stories

Real thread dump investigations.

Module 16

Thread Dump vs Other Tools

jstack vs JFR vs strace decision guide.

Module 17

SRE Production Playbook

Thread incident response workflow.

Detailed Module Content

Production SRE playbooks, examples, and incident patterns

MODULE 1

Introduction to Thread Dumps

A thread dump is a snapshot of every thread's stack trace, state, and lock information at one instant. Essential for hangs, deadlocks, and pool exhaustion.

  • Thread name and ID
  • State: RUNNABLE, BLOCKED, WAITING, TIMED_WAITING
  • Full stack trace per thread
  • Locked monitors and lock waiters
  • Deadlock detection (jstack)
MODULE 2

Why Every SRE Needs Thread Dump Skills

IncidentMetricsThread Dump Shows
API hungThreads maxed, no CPU200 threads BLOCKED on HikariCP.getConnection
DeadlockZero throughputFound Java-level deadlock: thread A ↔ B
Slow recoveryGradual latencyTIMED_WAITING on HTTP client read
MODULE 3

Java Thread Fundamentals

  • Platform threads: 1:1 OS thread (traditional)
  • Virtual threads (JDK 21+): lightweight, still dumpable
  • Thread pools: ExecutorService, Tomcat workers
  • Monitors: synchronized blocks; AQS: ReentrantLock, Semaphore
MODULE 4

Capturing Thread Dumps

terminal
jcmd <PID> Thread.print > /tmp/td-$(date +%s).txt
jstack -l <PID> > /tmp/td.txt   # -l = extra lock info
kill -3 <PID>                   # SIGQUIT — dumps to stdout/logs
SRE TIP

Capture 3 dumps 30 seconds apart during incidents — distinguishes stuck vs slow-progress threads.

MODULE 5

Reading Thread Dump Format

terminal
"http-nio-8080-exec-12" #45 daemon prio=5 os_prio=0 cpu=123ms elapsed=45s tid=0x... nid=0x... waiting on condition
   java.lang.Thread.State: BLOCKED (on object monitor)
   at com.zaxxer.hikari.pool.HikariPool.getConnection(...)
   - waiting to lock <0x00000007b2c4a8f0> (a com.zaxxer.hikari.pool.HikariPool)
Read Top to Bottom

Thread name → State → Stack frames (most recent at top) → Lock info at bottom of thread block.

MODULE 6

Thread States Deep Dive

RUNNABLE
Executing or ready for CPU
BLOCKED
Waiting to enter synchronized block
WAITING
Object.wait() or LockSupport.park() no timeout
TIMED_WAITING
sleep, park with timeout, poll(timeout)
MODULE 7

Deadlock Detection

jstack prints Found one Java-level deadlock with the cycle of threads and locks.

  1. Capture jstack immediately on hang
  2. Search for 'deadlock' (case insensitive)
  3. Map threads to business operations
  4. Fix lock ordering — always acquire locks in same global order
  5. Add timeout locks where appropriate
MODULE 8

Lock Contention Analysis

Search dump for BLOCKED and group by lock address <0x...>. The lock with most waiters is your bottleneck.

synchronized method
Entire method serialized
ReentrantLock.tryLock
Check if timeout configured
MODULE 9

Thread Pool Starvation

  • All http-nio-*-exec-* BLOCKED → Tomcat workers stuck
  • All pool-N-thread-M WAITING → custom pool idle or blocked on task
  • Queue growing in metrics + all threads busy → undersized pool or slow tasks
SRE TIP

If every worker is BLOCKED on external I/O, increasing pool size only delays failure — fix the dependency.

MODULE 10

Correlating with JFR and strace

ToolShows
Thread dumpWhich Java method / lock threads wait on
JFRJavaMonitorEnter duration, thread timeline
stracefutex wait, socket read block at kernel
MODULE 11

Kubernetes Thread Dump Capture

terminal
kubectl exec <pod> -- jstack 1 > thread-dump.txt
kubectl exec <pod> -- jcmd 1 Thread.print
# distroless:
kubectl debug -it <pod> --image=amazoncorretto:17 --target=app --share-processes
MODULE 12

Spring Boot Thread Issues

  • Tomcat http-nio-* — servlet blocking calls kill throughput
  • @Async pool threads — separate from HTTP workers
  • WebFlux should have few BLOCKED — if many, blocking call in reactive chain
  • Scheduler threads stuck on cron job holding global lock
MODULE 13

Database & I/O Blocked Threads

socketRead0 + PG JDBC
Threads waiting on slow PostgreSQL query
HttpClient.read
Downstream API timeout or slowness
MODULE 14

Production Best Practices

  • Automate capture on health-check failure
  • Store dumps with timestamp in incident ticket
  • Never restart before capturing dump on hang
  • Use lightweight scripts — jcmd preferred
  • Correlate thread name with MDC/request ID in logs
MODULE 15

SRE Incident War Stories

Payment Service Deadlock

SEV1

Impact: Complete payment outage 18 min

Symptoms: Zero throughput, CPU idle, no errors

Dump evidence: jstack: deadlock between PaymentService.lock and LedgerService.lock

Root cause: Inconsistent lock ordering across two services in same JVM

Resolution: Global lock order: always Ledger then Payment; deployed hotfix

Lesson: jstack deadlock section gave exact threads — fix in 15 min

Tomcat Thread Pool Exhausted

SEV2

Impact: API timeouts, 503 errors

Symptoms: 200/200 Tomcat threads busy

Dump evidence: All http-nio-exec BLOCKED on HikariPool.getConnection — pool timeout 30s

Root cause: DB failover caused 30s connection attempts holding all workers

Resolution: Reduced connection timeout to 3s; circuit breaker on DB

Lesson: Thread dump showed pool issue faster than DB team escalation

Kafka Consumer Thread Hang

SEV2

Impact: Consumer lag 5M messages

Symptoms: One consumer pod stuck, others fine

Dump evidence: kafka-consumer-thread BLOCKED on ReentrantLock held by same thread — recursive lock bug

Root cause: Custom interceptor re-entered synchronized block

Resolution: Replaced synchronized with ReentrantLock + removed recursion

Lesson: Single stuck thread in dump — not always all threads

Scheduled Job Global Lock

SEV3

Impact: API slow every hour on the hour

Symptoms: Predictable latency spike

Dump evidence: scheduling-1 RUNNABLE in heavy loop; 50 http threads BLOCKED on shared StaticLock

Root cause: @Scheduled job held global synchronized lock for 4 minutes during report generation

Resolution: Moved report to @Async with dedicated pool; removed global lock

Lesson: Correlate latency spike time with scheduler thread in dump

HTTP Client Read Block

SEV2

Impact: Checkout dependency timeout cascade

Symptoms: TIMED_WAITING on socket read

Dump evidence: 80 threads in sun.nio.ch.SocketDispatcher.read waiting on inventory API

Root cause: Inventory service degraded; no circuit breaker on caller

Resolution: Added resilience4j timeout + fallback; reduced client read timeout

Lesson: TIMED_WAITING + socket read = external dependency — not JVM bug

False Alarm — GC Pause

SEV4

Impact: Brief hang reported by QA

Symptoms: All threads RUNNABLE in dump but service unresponsive momentarily

Dump evidence: Dump taken during long GC pause — threads runnable but JVM paused

Root cause: Full GC 8s pause — not thread issue

Resolution: Tuned G1; used JFR for GC not thread dump

Lesson: Thread dump during STW GC is misleading — check GC logs too

MODULE 16

Thread Dump vs Other Tools

Hang/deadlock → jstackLock timing → JFROS wait → straceMemory → heap dump
MODULE 17

SRE Production Playbook

Alert: timeouts3x thread dump 30s apartDeadlock?BLOCKED group?External I/O?Fix + verify
terminal
# Incident script
for i in 1 2 3; do jcmd $PID Thread.print > /tmp/td-$i.txt; sleep 30; done
diff /tmp/td-1.txt /tmp/td-3.txt  # stuck = identical stacks
PRACTICAL EXPERIENCE

Hands-on Labs

LAB 1

Capture thread dump with jcmd Thread.print

Safe capture during live incident.

LAB 2

Identify deadlock from jstack output

Parse 'Found one Java-level deadlock' section.

LAB 3

Diagnose Tomcat thread pool exhaustion

All workers BLOCKED — find the monitor.

LAB 4

Correlate BLOCKED threads with slow DB

Threads waiting on socket read to PostgreSQL.

LAB 5

Capture from Kubernetes pod

kubectl exec + jstack workflow.

LAB 6

Compare 3 thread dumps 30s apart

Stuck vs progressing threads.

TEST YOUR KNOWLEDGE

Interactive Quiz

10 questions · Instant feedback

INTERVIEW PREP

Common Interview Questions

Q1

How to capture thread dump in production?

jcmd <PID> Thread.print or jstack <PID> — redirect to file. Repeat 2-3 times 30s apart.

Q2

How to find deadlock?

Search for 'Found one Java-level deadlock' or analyze lock chains manually in dump.

Q3

What causes all Tomcat threads BLOCKED?

Slow DB, external API, lock contention, or pool misconfiguration.

Q4

THREAD dump during high CPU?

Still useful — see if RUNNABLE threads are in tight loop vs blocked.

Q5

How to analyze thread pool sizing?

Compare active vs max threads in dump; count WAITING on queue vs RUNNABLE.

Q6

jstack vs jcmd Thread.print?

Equivalent output on modern JDK; jcmd preferred unified tool.

Q7

Thread dump in Kubernetes?

kubectl exec + jstack 1 or jcmd 1 Thread.print; ephemeral container if needed.

TROUBLESHOOTING

Investigation Flowcharts

Service Hung

No progress on requests → Thread dump x3 → All BLOCKED? → Find lock/socket → Deadlock? → Fix ordering or timeout

High Latency

P99 up → Thread dump → Pool exhausted? → Scale pool or fix slow dep → Lock contention?

Deadlock

Complete hang → jstack → Deadlock section? → Fix lock order → Verify with new dump

DB Slowness

Threads BLOCKED → Stack shows socket read JDBC → DB metrics → Query or pool fix

Production Cheat Sheet

Capture

jcmd <PID> Thread.print > /tmp/td.txt
jstack <PID> > /tmp/td.txt

Thread States

  • RUNNABLE
  • BLOCKED (monitor)
  • WAITING (park)
  • TIMED_WAITING

Red Flags

  • Found Java-level deadlock
  • All threads BLOCKED same lock
  • pool exhausted

Analysis Tips

  • Take 3 dumps 30s apart
  • Search for BLOCKED
  • Correlate with JFR lock events