Production-grade JVM diagnostics — CPU, memory, GC, locks, JDBC, Kafka, and Kubernetes. The definitive low-overhead profiler built into the JDK.
Metrics tell you what. Logs tell you when. JFR tells you why — inside the JVM at the moment of the incident.
Hot methods, JIT storms, infinite loops — flame graphs from production.
Allocation hotspots before OOMKill — find leaking classes early.
Capture JFR from pods with ephemeral containers and kubectl workflows.
What JFR is, architecture, and how it differs from other JVM tools.
Metrics show what; logs show when; JFR shows why inside the JVM.
Heap, metaspace, GC, safepoints, JIT, and TLAB fundamentals.
jcmd, startup flags, emergency capture during incidents.
CPU, memory, GC, threads, I/O, and exception event categories.
Automated analysis, flame graphs, locks, and memory views.
Hot methods, JIT storms, and CPU spike investigation.
Allocation hotspots, TLAB events, and leak patterns.
Pause times, promotion failures, and GC tuning signals.
Runnable, blocked, waiting states and pool starvation.
JavaMonitorEnter, ThreadPark, and contended locks.
JDBC latency, connection pool exhaustion, HikariCP.
End-to-end JVM time breakdown for slow requests.
ExceptionThrow storms and hidden retry loops.
kubectl exec, ephemeral containers, OOMKilled pods.
Controllers, Jackson, @Async, and startup bottlenecks.
Deserialization, rebalance storms, producer latency.
Real production incidents solved with JFR evidence.
Decision tree: JFR vs async-profiler, jstack, perf, heap dumps.
Incident response flow and emergency JFR commands.
Full explanations from the SRE JFR training course
Java Flight Recorder (JFR) is a low-overhead, production-grade profiling and diagnostics tool built into the JVM. It continuously records detailed events about the JVM's internal behavior — CPU usage, memory allocations, garbage collection, thread activity, locks, I/O, exceptions, and more — with minimal performance impact (typically < 1% overhead).
JFR was originally developed by Oracle for JRockit and later integrated into HotSpot/OpenJDK. It is now a standard part of the JDK.
Traditional profiling tools (VisualVM, JProfiler, YourKit) often introduce high overhead, making them unsuitable for production. JFR was designed from the ground up for always-on, low-overhead observability in mission-critical systems.
| Tool | Layer | Overhead | Production Safe | Best For |
|---|---|---|---|---|
| JFR | JVM | Very Low | Yes | Production profiling & diagnostics |
| JMC | Analysis UI | N/A | N/A | Analyzing JFR recordings |
| jstack | Thread dumps | Low | Yes (short) | Quick thread state snapshots |
| jcmd | JVM commands | Low | Yes | Starting/stopping JFR |
| async-profiler | Native + JVM | Low-Medium | Yes | Flame graphs & CPU profiling |
| VisualVM | GUI | Medium | Limited | Development & light profiling |
| JConsole | JMX | Low | Yes | Basic monitoring |
Application Code
↓
JVM (HotSpot)
↓
JFR Event Producers
(GC, JIT, Threads, Memory, Locks, I/O, etc.)
↓
JFR Recording Engine (Ring Buffer)
↓
.jfr File (Binary Recording)
↓
JMC / JDK Mission Control / Custom Tools
↓
Analysis & Visualization
JFR works by instrumenting the JVM internally at key points and emitting structured events that are written to a highly efficient ring buffer.
JFR writes events to a ring buffer — old events are overwritten unless you dump to a .jfr file. This is why duration and maxsize matter in production.
Metrics tell you WHAT is happening.
Logs tell you WHEN something happened.
JFR tells you WHY — at the exact moment the problem occurred inside the JVM.
| Incident | What Metrics/Logs Show | What JFR Reveals |
|---|---|---|
| Slow API / High Latency | P95/P99 high | Which methods, locks, DB calls, or GC pauses are causing it |
| CPU Spike | CPU at 90-100% | Hot methods, JIT compilation storms, infinite loops |
| GC Pauses / Latency Spikes | GC time increasing | Allocation rate, promotion failures, humongous objects |
| Memory Leak / OOMKill | Heap usage growing | Allocation hotspots, leaking objects, ThreadLocal leaks |
| Thread Pool Starvation | Threads blocked / queue growing | Which threads are blocked on what locks |
| Database Latency | DB response time high | JDBC calls, connection acquisition time, query execution |
| Kafka Consumer Lag | Lag increasing | Deserialization time, processing time per record |
| Connection Pool Exhaustion | Timeouts on DB calls | Connection acquisition latency & contention |
Key Insight: JFR gives you ground truth from inside the JVM during the exact time window of the incident.
When Prometheus shows high latency but logs are clean, JFR often reveals GC pauses, lock waits, or JDBC acquisition time that APM agents miss.
To effectively use JFR, SREs must understand these core JVM components:
JFR can observe and record events from all of these components with very low overhead because the instrumentation is built directly into the JVM.
TLAB allocations are fast path — when you see high ObjectAllocationOutsideTLAB, allocation pressure is severe and GC will follow.
# Start a recording jcmd <PID> JFR.start name=MyAppRecording duration=300s filename=/tmp/app.jfr # Dump current buffer (without stopping) jcmd <PID> JFR.dump filename=/tmp/emergency.jfr # Stop recording jcmd <PID> JFR.stop name=MyAppRecording
java -XX:StartFlightRecording=filename=/tmp/app.jfr,dumponexit=true,settings=profile \
-jar myapp.jar
Production-safe settings:
settings=default or settings=profile (profile has slightly higher detail)maxage, maxsize)dumponexit=true in very long-running services unless neededWhen an application is slow or about to OOM:
jcmd <PID> JFR.start name=Emergency duration=60s filename=/tmp/emergency.jfr # Wait 30-60 seconds then jcmd <PID> JFR.dump filename=/tmp/emergency.jfr
jcmd JFR.start name=R duration=300s filename=/tmp/app.jfr Timed recordingjcmd JFR.dump filename=/tmp/snapshot.jfr Dump buffer nowjcmd JFR.check List active recordingsjcmd JFR.stop name=R Stop named recording-XX:StartFlightRecording=settings=profileHigh-detail profile at boot-XX:StartFlightRecording=maxsize=250MRing buffer cap-XX:StartFlightRecording=delay=60sSkip startup noise-XX:FlightRecorderOptions=stackdepth=256Deeper stack tracesJFR records hundreds of event types. Key categories:
jdk.CPULoadjdk.ExecutionSamplejdk.Compilationjdk.ObjectAllocationInNewTLABjdk.ObjectAllocationOutsideTLABjdk.JavaMonitorEnterjdk.GarbageCollectionjdk.GCPhasePausejdk.PromotionFailedjdk.ThreadStartjdk.ThreadParkjdk.JavaMonitorEnterjdk.ThreadSleepjdk.SocketReadjdk.SocketWritejdk.FileReadjdk.FileWritejdk.ExceptionThrowFor each event, JFR captures timestamp, thread, stack trace (when enabled), and event-specific fields.
Java Mission Control (JMC) is the official (and best) tool for analyzing JFR recordings.
Pro Tip: Always start with the Automated Analysis page — it often points directly to the root cause.
Always correlate JMC timestamps with your Grafana incident window — misaligned recordings lead to wrong conclusions.
Scenario: CPU at 90%+, response time increasing.
- Methods consuming > 5-10% of CPU
- Excessive JIT compilation (jdk.Compilation)
- Hot loops or expensive operations (JSON, regex, serialization)
Common Culprits:
JFR excels at finding leaks before OOMKill.
jdk.ObjectAllocationInNewTLABjdk.ObjectAllocationOutsideTLABCommon Leaks:
ConcurrentHashMap or ArrayListThreadLocal not removed after useJFR provides deep visibility into GC behavior.
jdk.GCPhasePause)JFR helps you decide if you need to tune heap size, change GC, or fix allocation patterns.
JFR shows thread states over time:
Lock contention is one of the most common hidden causes of latency.
jdk.JavaMonitorEnterjdk.ThreadParkjdk.LockInstancesReentrantLockImpact: Even small contention on hot locks can destroy throughput under load.
If JavaMonitorEnter duration exceeds request SLA on hot path, even microsecond locks aggregate to seconds under load.
JFR can show JDBC-level activity (with some configuration).
JFR helps distinguish between pool acquisition latency vs actual database query slowness.
All threads waiting for connection. Fix pool size, leak, or DB connectivity.
Connections acquired fast but JDBC/socket time high. Index/query/DB issue.
Scenario: P95 latency increased suddenly.
- Thread pool utilization
- Lock contention
- Database / external call latency
- Serialization / deserialization time
- GC impact on request threads
JFR lets you build a complete picture of where time is being spent inside the JVM for slow requests.
Frequent exceptions can silently destroy performance.
jdk.ExceptionThrow events with full stack tracesExamples:
NullPointerException in hot pathsSocketTimeoutException / connection resetsSQLException from connection issuesBusiness impact: Increased CPU, latency, and noisy logs.
# Exec into pod kubectl exec -it <pod> -- /bin/sh # Start JFR jcmd 1 JFR.start name=prod duration=120s filename=/tmp/app.jfr # Copy recording out kubectl cp <pod>:/tmp/app.jfr ./app.jfr
kubectl debug -it <pod> --image=amazoncorretto:17 --target=<container> --share-processes jcmd 1 JFR.start name=k8s duration=120s filename=/tmp/pod.jfr settings=profile jcmd 1 JFR.dump filename=/tmp/pod.jfr exit kubectl cp <pod>:/tmp/pod.jfr ./pod-incident.jfr
For distroless images, never install JDK into the app container — ephemeral debug containers are the production-safe pattern.
JFR is excellent for finding slow endpoints or initialization bottlenecks in Spring Boot applications.
-XX:StartFlightRecording=filename=/tmp/boot.jfr,delay=10sProfile Spring Boot startupspring.jmx.enabled=trueEnable JMX for jcmd in some setupsmanagement.endpoints.web.exposure.include=healthKeep actuator minimal in prodJFR helps you see whether lag is caused by slow processing, deserialization, or network.
Kafka lag triage with JFR:
ObjectAllocation on deserialize → payload/schema issue@KafkaListener → business logic slowImpact: P99 checkout latency 800ms → 4.2s
Metrics/Logs: Grafana: latency spike, normal CPU. Logs: no errors.
JFR evidence: JFR: jdk.GCPhasePause events 1.8–2.1s every 45s. Humongous byte[] allocations during cart serialization.
Root cause: G1 humongous objects + undersized heap. Promotion failures during peak traffic.
Resolution: Increased heap 4G→8G, tuned -XX:G1HeapRegionSize, fixed oversized session cart caching.
Lesson: Always check GC pauses in JFR when latency spikes without error logs.
Impact: 30% payment failures, HikariCP timeouts
Metrics/Logs: Metrics: pool active=50/50, pending threads growing. Logs: getConnection timed out.
JFR evidence: JFR: threads blocked 8–12s on HikariCP.getConnection. JDBC events near zero until timeout.
Root cause: Connection leak in failed-payment rollback path — connections not returned to pool.
Resolution: Hotfix: try-with-resources on Connection. Pool recovered in 3 min.
Lesson: Distinguish pool starvation from slow queries using JFR thread + JDBC timing.
Impact: Pod restart loop in Kubernetes
Metrics/Logs: K8s: OOMKilled. Heap metrics showed growth over 6 hours.
JFR evidence: JFR (pre-OOM): ConcurrentHashMap allocations 2.4 GB/hour. Top class: InventoryCacheEntry.
Root cause: Unbounded local cache without TTL/eviction on inventory SKU map.
Resolution: Added Caffeine cache with maxSize + expireAfterWrite. Memory flatlined.
Lesson: JFR allocation events find leaks days before OOMKill.
Impact: Lag 2M messages, consumer group stalled
Metrics/Logs: Metrics: lag ↑, CPU moderate. Logs: rebalance noise.
JFR evidence: JFR: 68% sample time in Avro deserialization. ObjectAllocation char[] and byte[] dominant.
Root cause: Schema change increased payload 4x. Deserialization became bottleneck.
Resolution: Scaled consumers + optimized Avro reader. Consider payload slimming.
Lesson: Kafka lag is not always broker — JFR shows per-record processing cost.
Impact: API errors, DB 'too many connections'
Metrics/Logs: DB metrics: connections at max. App: thread pool exhausted.
JFR evidence: JFR: 200 threads in TIMED_WAITING on socket read; connection storm to PG.
Root cause: Missing connection pool limit on new microservice — each pod opened 100+ direct connections.
Resolution: Enforced HikariCP maxPoolSize=20 per pod. Added PgBouncer.
Lesson: JFR thread view exposes connection storms faster than DB metrics alone.
Impact: Elevated latency across 12 services
Metrics/Logs: Redis latency P99 high. Dependent services timing out.
JFR evidence: JFR on caller service: threads WAITING on Jedis connection pool; ExceptionThrow SocketTimeoutException.
Root cause: Redis single-node CPU saturated after bad key pattern (KEYS in cron).
Resolution: Removed KEYS usage, added read replica, increased pool with backoff.
Lesson: ExceptionThrow frequency in JFR reveals retry/timeout storms early.
Impact: New deployment pods slow to pass readiness
Metrics/Logs: K8s: readiness probe failures for 240s. No runtime errors.
JFR evidence: JFR from boot: jdk.ClassLoad + jdk.Compilation dominated first 180s. Bean init for 400+ beans.
Root cause: Monolith-style classpath scan + eager @PostConstruct on heavy clients.
Resolution: Lazy init, split actuator, reduced @ComponentScan scope. Startup → 45s.
Lesson: JFR startup recordings justify JVM tuning and Spring refactor with data.
Impact: Checkout API P99 200ms → 1.8s
Metrics/Logs: CPU normal, DB healthy. APM showed gap in 'application' time.
JFR evidence: JFR: jdk.JavaMonitorEnter on com.acme.PromoService.applyPromo — 40 threads blocked avg 400ms.
Root cause: synchronized block around promo DB call — accidental serialization of all requests.
Resolution: Replaced with per-key locking + async cache. P99 restored.
Lesson: Lock Instances view in JMC finds hidden serialization under load.
CPU Issue?
perf for kernel-level CPUMemory / Leak?
GC Issue?
-Xlog:gc*)Lock / Thread Issue?
Container / Kubernetes?
Alert triggered ↓ Check Metrics (Prometheus/Grafana) ↓ Check Logs (ELK / Loki) ↓ Capture JFR (emergency or continuous) ↓ Analyze in JMC (start with Automated Analysis) ↓ Correlate with system tools (strace, top, iostat if needed) ↓ Identify Root Cause ↓ Apply Fix / Mitigation ↓ Document + Improve Runbook
# Quick 60-second capture jcmd <PID> JFR.start name=Incident duration=60s filename=/tmp/incident.jfr # Dump immediately jcmd <PID> JFR.dump filename=/tmp/incident.jfr
Right-size heap, thread pools, and connection pools from real data.
Low-overhead always-on recordings in production.
Export recordings and use LLMs for RCA drafts.
Use jcmd to start a 5-minute profile recording.
Find methods consuming >10% CPU during a spike.
Use allocation events to identify growing object types.
Correlate JavaMonitorEnter with Lock Instances view.
Compare jdk.GCPhasePause across collectors.
kubectl exec + jcmd + kubectl cp workflow.
Overlay JVM events with latency spike window.
Labs 1–3 include interactive simulations. Labs 4–7 are designed for hands-on practice in your own JVM or Kubernetes environment.
10 questions · Instant feedback
Covers jcmd, JMC, GC, locks, Kubernetes capture, and tool selection.
What makes JFR safe for production compared to traditional profilers?
Very low overhead (<1%), built into JVM, ring buffer design, no bytecode injection.
How do you start a 5-minute JFR recording on PID 12345?
jcmd 12345 JFR.start name=incident duration=300s filename=/tmp/incident.jfr settings=profile
What JMC view do you open first and why?
Automated Analysis — rules engine surfaces GC, memory, and hot method issues immediately.
Name three JFR events for memory leak investigation.
ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, and heap usage trends in Memory view.
How do you capture JFR from a distroless Kubernetes pod?
kubectl debug ephemeral container with JDK image, share-processes, jcmd on JVM PID, kubectl cp .jfr out.
JFR vs heap dump — when to use each?
JFR live for allocation rate and hotspots; heap dump post-mortem for object retainers and dominators.
What does high ObjectAllocationOutsideTLAB indicate?
Severe allocation pressure — objects too large for TLAB or TLAB exhausted.
How do you correlate JFR with an incident window?
Note recording start/end UTC, align with Grafana alert time, filter JMC events to that range.
What JDK versions include JFR for free?
OpenJDK 11+ (flight recorder enabled by default in modern builds).
JFR vs async-profiler?
JFR: broad JVM events, GC, allocations, low overhead. async-profiler: excellent CPU/native flame graphs.
CPU alert? → Capture JFR 60–120s → JMC flame graph → Hot Java method? → Fix code / JIT warm-up → Still high? → strace/perf
Heap growing? → JFR allocation events → Top class growing? → Leak in app code → OOMKilled? → heap dump + JFR
P95/P99 up, no errors? → JFR during window → GC pauses? → Lock contention? → JDBC/pool wait? → External timeout?
GC pause alert? → jdk.GCPhasePause in JFR → Full GC frequent? → Tune heap/GC or fix allocation → Humongous objects?
Throughput collapsed? → JFR Threads view → All blocked? → jstack correlate → Fix deadlock/pool size
Pod slow/OOM? → kubectl debug + jcmd → JFR dump + cp → JMC analysis → Adjust limits/heap/code
jcmd <PID> JFR.start name=Prod duration=300s filename=/tmp/recording.jfr settings=profile
jcmd <PID> JFR.dump filename=/tmp/emergency.jfr
Alert triggered → Metrics (Prometheus) → Logs (ELK/Loki) → Capture JFR → Analyze in JMC → Correlate (strace/top) → Root cause → Fix → Postmortem