Capture and analyze Java heap dumps with Eclipse MAT — find memory leaks, OOM root causes, and dominator retainers in production incidents.
See exactly which objects retain heap — JFR shows allocation rate, heap dumps show retainers.
Post-mortem analysis when pods OOMKill — no reproduction needed.
Dominator tree, GC roots, leak suspects, and histogram in Eclipse MAT.
What hprof files are, when to use them, and how they differ from JFR.
OOM incidents, memory leaks, and K8s OOMKilled pods require heap analysis.
Young/old gen, metaspace, and what appears in hprof vs native memory.
jcmd, jmap, OOM flags, and safe production timing.
Eclipse MAT, VisualVM, jhat, and when to use each.
Opening hprof, histogram, dominator tree, and leak suspects report.
Understanding retention paths and why objects weren't collected.
Step-by-step SRE playbook from alert to fix.
Kubernetes OOM workflow and heap artifact recovery.
When to use each tool in the JVM observability stack.
exec, ephemeral containers, volume mounts for hprof files.
Common leaks in Spring apps: caches, sessions, actuator, class loaders.
Connection wrappers, result sets, and in-memory caches.
When to capture, security, size, and storage.
Real heap dump investigations with MAT evidence.
Decision tree with JFR, jstack, strace, and NMT.
End-to-end heap incident response.
Production SRE playbooks, examples, and incident patterns
A heap dump is a snapshot of every object in the JVM heap at a point in time — classes, instances, references, and retained sizes. Saved as .hprof files.
Heap dumps answer who is holding memory — not just how fast allocations happen (that's JFR).
| Scenario | Metrics Show | Heap Dump Reveals |
|---|---|---|
| OOMKilled pod | Memory at limit, restart | Exact objects retaining 2GB |
| Slow memory growth | Heap % climbing over days | Cache class dominating dominator tree |
| Post-incident RCA | GC logs show Full GC | Leak suspect: SessionMap |
Enable -XX:+HeapDumpOnOutOfMemoryError on every production Java service.
# Recommended (JDK 11+) jcmd <PID> GC.heap_dump /tmp/heap.hprof # On OOM (set at startup) -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/app/heap.hprof # Legacy jmap -dump:live,format=b,file=heap.hprof <PID>
Live heap dumps trigger a stop-the-world pause. For a 8GB heap this can take 30–90 seconds. Capture during maintenance window or low traffic.
| Tool | Best For | Production Use |
|---|---|---|
| Eclipse MAT | Leak analysis, dominator tree | Download hprof, analyze offline |
| VisualVM | Quick histogram | Smaller dumps, dev/staging |
| jhat | Server-side quick view | Large dumps on analysis server |
| JFR | Live allocation rate | Complement — not replacement |
Increase MAT heap: MemoryAnalyzer -vmargs -Xmx8g for large production dumps.
The dominator tree shows objects that dominate memory — if you remove a dominator, its retained subtree becomes unreachable.
GC roots are starting points: static fields, thread stacks, JNI, class loaders. Path from root to object explains why GC kept it alive.
kubectl describe pod <pod> | grep -A5 OOM kubectl logs <pod> --previous # If heap path mounted: kubectl cp <pod>:/var/log/app/heap.hprof ./oom.hprof
Distinguish heap OOM (Java heap space) from container memory limit (RSS includes native + metaspace + heap).
kubectl exec -it <pod> -- jcmd 1 GC.heap_dump /tmp/heap.hprof kubectl cp <pod>:/tmp/heap.hprof ./heap.hprof # Ephemeral debug container (distroless) kubectl debug -it <pod> --image=amazoncorretto:17 --target=app --share-processes
Mount emptyDir or PVC at HeapDumpPath so OOM dumps survive pod termination.
@Cacheable without evictionImpact: Checkout API down 22 min, pods OOMKilled
Symptoms: Heap 95%, Full GC every 2 min, P99 timeout
Dump evidence: MAT: SessionCache retained 4.2GB — 890k session objects, no TTL
Root cause: New feature cached full cart per session without eviction
Resolution: Added Caffeine maxSize=10k + expireAfterAccess=30m
Lesson: Leak Suspects report flagged SessionCache in 2 minutes
Impact: Memory climb over 5 days, eventual OOM
Symptoms: Gradual heap growth, normal traffic
Dump evidence: Dominator: ThreadLocalMap → UserContext 2.8GB across 200 threads
Root cause: UserContext set in filter, never removed in finally block
Resolution: threadLocal.remove() in filter finally; pool threads reused
Lesson: ThreadLocal + thread pools = classic leak — always remove()
Impact: Single service OOM every 48h
Symptoms: Predictable OOM cycle after deploy
Dump evidence: Histogram: EventDTO 12M instances in static HashMap
Root cause: Debug logging stored every event in static map 'for troubleshooting'
Resolution: Removed static map; use bounded ring buffer
Lesson: Search dominator for static fields holding collections
Impact: Metaspace + heap growth on hot redeploys
Symptoms: Only after multiple deploys without restart
Dump evidence: GC roots: WebappClassLoader retained 1.1GB old app classes
Root cause: Thread holding reference to old classloader via shutdown hook
Resolution: Proper lifecycle on shutdown hooks; restart pod on deploy
Lesson: ClassLoader leaks show as metaspace + old gen class objects
Impact: Pods killed at 512Mi limit, heap only 256MB
Symptoms: OOMKilled but small hprof — native memory issue
Dump evidence: Heap dump only 180MB; RSS was 510MB — direct buffers
Root cause: Container limit too low for heap + metaspace + Netty direct memory
Resolution: Increased limit to 1Gi, -XX:MaxDirectMemorySize=256m
Lesson: OOMKilled ≠ Java heap full — check native and direct memory
Impact: Ingestion service OOM during bulk indexing spike
Symptoms: byte[] dominates histogram after traffic spike
Dump evidence: Retained byte[] 3.1GB from unreleased bulk response buffers
Root cause: Exception path skipped buffer release in custom client wrapper
Resolution: try-finally on buffer release; added leak test
Lesson: byte[] top of histogram → buffer or serialization leak
Use heap dump when you need object-level retention. Use JFR for live allocation rate. Use jstack for thread state. Use strace for OS boundary.
# Emergency checklist jcmd <PID> VM.flags | grep HeapDump jcmd <PID> GC.heap_dump /tmp/incident.hprof # Open in MAT → Leak Suspects → Dominator Tree
GC.heap_dump on a running Spring Boot service safely.
Find top memory retainers and leak suspects.
HeapDumpOnOutOfMemoryError for post-mortem.
Baseline vs incident to find growing object types.
kubectl exec + jcmd + kubectl cp workflow.
Find ThreadLocalMap entries retaining sessions.
10 questions · Instant feedback
When do you take a heap dump vs JFR?
JFR during live incident for allocation rate; heap dump when you need object retainers or post-OOM analysis.
How to capture heap dump without jmap?
jcmd <PID> GC.heap_dump /path/heap.hprof or -XX:+HeapDumpOnOutOfMemoryError.
What is dominator tree?
Tree showing which objects retain the most heap — if removed, how much memory becomes unreachable.
Common production leak patterns?
Unbounded caches, ThreadLocal, static collections, session maps, connection wrappers.
How to analyze OOMKilled pod?
Check if heap dump was written, download hprof, open in MAT, check dominator tree and leak suspects.
Heap dump too large to download?
Capture with live objects only, use MAT on server, or analyze subset with jhat remotely.
Difference shallow vs retained heap?
Shallow = object itself; retained = object + everything only reachable through it.
GC roots in MAT?
Objects reachable from JVM roots — dominator paths explain why objects weren't collected.
Heap growing? → JFR allocation rate → Still leaking? → Capture hprof → MAT dominator tree → Fix retainers
Pod restart OOMKilled → Heap dump exists? → Open in MAT → Leak suspects → Fix + tune -Xmx
Heap > 80% → Live hprof or JFR → Old gen full? → Leak vs legit cache → Tune or fix
Open hprof → Leak Suspects report → Dominator tree → Path to GC roots → Identify fix
jcmd <PID> GC.heap_dump /tmp/heap.hprof