FOR SREs & PLATFORM ENGINEERS

Notes on Java Flight Recorder

Production-grade JVM diagnostics — CPU, memory, GC, locks, JDBC, Kafka, and Kubernetes. The definitive low-overhead profiler built into the JDK.

20
Modules
8
War Stories
7
Labs
jcmd + JMC production playbooks
Spring Boot, Kafka, K8s workflows
8 detailed incident war stories
Interview prep + troubleshooting flows

When metrics and logs aren't enough

Metrics tell you what. Logs tell you when. JFR tells you why — inside the JVM at the moment of the incident.

CPU Spikes

Hot methods, JIT storms, infinite loops — flame graphs from production.

Memory Leaks

Allocation hotspots before OOMKill — find leaking classes early.

Kubernetes

Capture JFR from pods with ephemeral containers and kubectl workflows.

20 MODULES + 3 BONUS

Guide sections

Module 1

Introduction to JFR

What JFR is, architecture, and how it differs from other JVM tools.

Module 2

Why Every SRE Should Learn JFR

Metrics show what; logs show when; JFR shows why inside the JVM.

Module 3

JVM Internals Required for JFR

Heap, metaspace, GC, safepoints, JIT, and TLAB fundamentals.

Module 4

Capturing JFR Recordings

jcmd, startup flags, emergency capture during incidents.

Module 5

Understanding JFR Events

CPU, memory, GC, threads, I/O, and exception event categories.

Module 6

Using Java Mission Control (JMC)

Automated analysis, flame graphs, locks, and memory views.

Module 7

CPU Troubleshooting with JFR

Hot methods, JIT storms, and CPU spike investigation.

Module 8

Memory Leak Analysis

Allocation hotspots, TLAB events, and leak patterns.

Module 9

Garbage Collection Analysis

Pause times, promotion failures, and GC tuning signals.

Module 10

Thread Analysis

Runnable, blocked, waiting states and pool starvation.

Module 11

Lock Contention Analysis

JavaMonitorEnter, ThreadPark, and contended locks.

Module 12

Database Troubleshooting

JDBC latency, connection pool exhaustion, HikariCP.

Module 13

API Latency Investigation

End-to-end JVM time breakdown for slow requests.

Module 14

Exception Analysis

ExceptionThrow storms and hidden retry loops.

Module 15

Kubernetes and JFR

kubectl exec, ephemeral containers, OOMKilled pods.

Module 16

JFR and Spring Boot

Controllers, Jackson, @Async, and startup bottlenecks.

Module 17

Kafka Troubleshooting with JFR

Deserialization, rebalance storms, producer latency.

Module 18

SRE Incident War Stories

Real production incidents solved with JFR evidence.

Module 19

JFR vs Other Tools

Decision tree: JFR vs async-profiler, jstack, perf, heap dumps.

Module 20

SRE Production Playbook

Incident response flow and emergency JFR commands.

Detailed Module Content

Full explanations from the SRE JFR training course

MODULE 1

Introduction to JFR

What is JFR?

Java Flight Recorder (JFR) is a low-overhead, production-grade profiling and diagnostics tool built into the JVM. It continuously records detailed events about the JVM's internal behavior — CPU usage, memory allocations, garbage collection, thread activity, locks, I/O, exceptions, and more — with minimal performance impact (typically < 1% overhead).

JFR was originally developed by Oracle for JRockit and later integrated into HotSpot/OpenJDK. It is now a standard part of the JDK.

Why Oracle Created It

Traditional profiling tools (VisualVM, JProfiler, YourKit) often introduce high overhead, making them unsuitable for production. JFR was designed from the ground up for always-on, low-overhead observability in mission-critical systems.

History & Evolution

  • JRockit Mission Control (original commercial tool)
  • 2018: Open-sourced and integrated into OpenJDK 11+
  • Today: Available in all modern JDKs (OpenJDK, Oracle JDK, Amazon Corretto, Azul Zulu, etc.)

JFR vs Related Tools

ToolLayerOverheadProduction SafeBest For
JFRJVMVery LowYesProduction profiling & diagnostics
JMCAnalysis UIN/AN/AAnalyzing JFR recordings
jstackThread dumpsLowYes (short)Quick thread state snapshots
jcmdJVM commandsLowYesStarting/stopping JFR
async-profilerNative + JVMLow-MediumYesFlame graphs & CPU profiling
VisualVMGUIMediumLimitedDevelopment & light profiling
JConsoleJMXLowYesBasic monitoring

JFR Architecture

terminal
Application Code

      ↓

JVM (HotSpot)

      ↓

JFR Event Producers

   (GC, JIT, Threads, Memory, Locks, I/O, etc.)

      ↓

JFR Recording Engine (Ring Buffer)

      ↓

.jfr File (Binary Recording)

      ↓

JMC / JDK Mission Control / Custom Tools

      ↓

Analysis & Visualization

JFR works by instrumenting the JVM internally at key points and emitting structured events that are written to a highly efficient ring buffer.

Key Insight

JFR writes events to a ring buffer — old events are overwritten unless you dump to a .jfr file. This is why duration and maxsize matter in production.

MODULE 2

Why Every SRE Should Learn JFR

Metrics tell you WHAT is happening.

Logs tell you WHEN something happened.

JFR tells you WHY — at the exact moment the problem occurred inside the JVM.

Common Production Incidents Where JFR Shines

IncidentWhat Metrics/Logs ShowWhat JFR Reveals
Slow API / High LatencyP95/P99 highWhich methods, locks, DB calls, or GC pauses are causing it
CPU SpikeCPU at 90-100%Hot methods, JIT compilation storms, infinite loops
GC Pauses / Latency SpikesGC time increasingAllocation rate, promotion failures, humongous objects
Memory Leak / OOMKillHeap usage growingAllocation hotspots, leaking objects, ThreadLocal leaks
Thread Pool StarvationThreads blocked / queue growingWhich threads are blocked on what locks
Database LatencyDB response time highJDBC calls, connection acquisition time, query execution
Kafka Consumer LagLag increasingDeserialization time, processing time per record
Connection Pool ExhaustionTimeouts on DB callsConnection acquisition latency & contention

Key Insight: JFR gives you ground truth from inside the JVM during the exact time window of the incident.

Ground Truth

When Prometheus shows high latency but logs are clean, JFR often reveals GC pauses, lock waits, or JDBC acquisition time that APM agents miss.

MODULE 3

JVM Internals Required for JFR

To effectively use JFR, SREs must understand these core JVM components:

Key Areas

  • Heap Memory: Young (Eden, Survivor) + Old Generation
  • Metaspace: Class metadata (replaced PermGen)
  • Thread Stacks: Per-thread call stacks
  • Garbage Collectors: G1GC, ZGC, Shenandoah, Parallel, Serial
  • Safepoints: Global JVM pauses for certain operations
  • JIT Compiler: C1/C2 compilation of hot methods
  • Class Loading: ClassLoader hierarchy and loading time
  • TLAB / PLAB: Thread-Local Allocation Buffers (critical for allocation performance)

JFR can observe and record events from all of these components with very low overhead because the instrumentation is built directly into the JVM.

SRE TIP

TLAB allocations are fast path — when you see high ObjectAllocationOutsideTLAB, allocation pressure is severe and GC will follow.

Heap Generations (G1)

Young Gen (Eden + Survivor) → promotion → Old Gen Metaspace (class metadata, unbounded by default) Thread stacks + native memory (not in heap, but affects RSS)
MODULE 4

Capturing JFR Recordings

Using jcmd (Recommended for Running Processes)

terminal
# Start a recording

jcmd <PID> JFR.start name=MyAppRecording duration=300s filename=/tmp/app.jfr



# Dump current buffer (without stopping)

jcmd <PID> JFR.dump filename=/tmp/emergency.jfr



# Stop recording

jcmd <PID> JFR.stop name=MyAppRecording

Startup Recording (via JVM Flags)

terminal
java -XX:StartFlightRecording=filename=/tmp/app.jfr,dumponexit=true,settings=profile \

     -jar myapp.jar

Production-safe settings:

  • Use settings=default or settings=profile (profile has slightly higher detail)
  • Limit duration or use ring buffer (maxage, maxsize)
  • Avoid dumponexit=true in very long-running services unless needed

Emergency Capture During Incidents

When an application is slow or about to OOM:

terminal
jcmd <PID> JFR.start name=Emergency duration=60s filename=/tmp/emergency.jfr

# Wait 30-60 seconds then

jcmd <PID> JFR.dump filename=/tmp/emergency.jfr

Live Process (jcmd)

jcmd JFR.start name=R duration=300s filename=/tmp/app.jfrTimed recording
jcmd JFR.dump filename=/tmp/snapshot.jfrDump buffer now
jcmd JFR.checkList active recordings
jcmd JFR.stop name=RStop named recording

Startup Flags

-XX:StartFlightRecording=settings=profileHigh-detail profile at boot
-XX:StartFlightRecording=maxsize=250MRing buffer cap
-XX:StartFlightRecording=delay=60sSkip startup noise
-XX:FlightRecorderOptions=stackdepth=256Deeper stack traces
MODULE 5

Understanding JFR Events

JFR records hundreds of event types. Key categories:

CPU & Compilation

  • jdk.CPULoad
  • jdk.ExecutionSample
  • jdk.Compilation

Memory & Allocation

  • jdk.ObjectAllocationInNewTLAB
  • jdk.ObjectAllocationOutsideTLAB
  • jdk.JavaMonitorEnter

GC Events

  • jdk.GarbageCollection
  • jdk.GCPhasePause
  • jdk.PromotionFailed

Threads & Locking

  • jdk.ThreadStart
  • jdk.ThreadPark
  • jdk.JavaMonitorEnter
  • jdk.ThreadSleep

I/O & Network

  • jdk.SocketRead
  • jdk.SocketWrite
  • jdk.FileRead
  • jdk.FileWrite

Exceptions

  • jdk.ExceptionThrow

For each event, JFR captures timestamp, thread, stack trace (when enabled), and event-specific fields.

jdk.ExecutionSample + CPULoad
CPU hotspots and JVM load
jdk.ObjectAllocationInNewTLAB
Allocation rate per class
jdk.GCPhasePause
Stop-the-world pause duration
jdk.JavaMonitorEnter
Synchronized block contention
jdk.ExceptionThrow
Exception frequency + stacks
MODULE 6

Using Java Mission Control (JMC)

Java Mission Control (JMC) is the official (and best) tool for analyzing JFR recordings.

Key Views in JMC

  • Automated Analysis — Rules engine highlights problems automatically
  • Event Browser — Filter and search all events
  • Threads — Thread states over time + stack traces
  • Memory — Heap usage, allocations, GC pauses
  • Method Profiling / Flame Graphs — Hot methods
  • Lock Instances — Contended monitors

Pro Tip: Always start with the Automated Analysis page — it often points directly to the root cause.

JMC Analysis Workflow

  1. Open .jfr file → start with Automated Analysis
  2. Review flagged problems (GC, memory, hot methods)
  3. Drill into Method Profiling / flame graph for CPU
  4. Check Memory → Allocations for leak suspects
  5. Use Threads + Lock Instances for contention
  6. Export findings for postmortem / ticket
SRE TIP

Always correlate JMC timestamps with your Grafana incident window — misaligned recordings lead to wrong conclusions.

MODULE 7

CPU Troubleshooting with JFR

Scenario: CPU at 90%+, response time increasing.

Investigation Workflow

  • Capture JFR during the spike
  • Open in JMC → Method Profiling or Flame Graph
  • Look for:

- Methods consuming > 5-10% of CPU

- Excessive JIT compilation (jdk.Compilation)

- Hot loops or expensive operations (JSON, regex, serialization)

Common Culprits:

  • Infinite loops / busy waiting
  • Excessive object creation + serialization
  • Heavy regex or XML/JSON parsing
  • Inefficient algorithms in hot paths

Investigation Workflow

  1. Capture JFR during CPU spike (60–120s)
  2. Open Method Profiling / flame graph in JMC
  3. Sort by self-time and total-time
  4. Check jdk.Compilation event rate (JIT storm?)
  5. Correlate hot methods with deployment / config change
com.fasterxml.jackson.* 35%
JSON serialization storm
java.util.regex.* 22%
Catastrophic backtracking
jdk.Compilation burst
Cold start or new code path
while(true) tight loop
Bug or missing sleep/backoff
MODULE 8

Memory Leak Analysis

JFR excels at finding leaks before OOMKill.

Key Events

  • jdk.ObjectAllocationInNewTLAB
  • jdk.ObjectAllocationOutsideTLAB

Investigation Steps in JMC

  • Go to Memory view
  • Look at Allocation tab
  • Identify classes with highest allocation rate
  • Check for growing collections, caches, or ThreadLocals

Common Leaks:

  • Unbounded caches
  • Growing ConcurrentHashMap or ArrayList
  • Session objects not cleaned up
  • ThreadLocal not removed after use
byte[] growing
Buffer leak or unbounded cache
ConcurrentHashMap entries
Unbounded in-memory map
ThreadLocal + pool threads
Classic leak in Tomcat/Spring
char[] from String concat
Logging or XML building in loop

Investigation Steps

  1. Memory view → Allocations tab → sort by allocated bytes
  2. Identify top 3 classes by rate and total
  3. Check if rate grows linearly over recording window
  4. Take heap dump only if you need object retainers
MODULE 9

Garbage Collection Analysis

JFR provides deep visibility into GC behavior.

What to Analyze

  • GC pause times (jdk.GCPhasePause)
  • Allocation rate vs promotion rate
  • Humongous object allocations (G1)
  • Full GC frequency
  • Concurrent mark failures

Modern GCs

  • G1GC (default in JDK 9+)
  • ZGC & Shenandoah (low-pause)
  • Parallel GC

JFR helps you decide if you need to tune heap size, change GC, or fix allocation patterns.

GC Red Flags in JFR

  • Full GC more than once per hour in steady state
  • GCPhasePause > 200ms on latency-sensitive services
  • PromotionFailed events (G1 cannot evacuate in time)
  • Humongous allocations dominating old gen (G1)
G1GC
Default JDK 9+. Balance throughput & pauses.
ZGC
Sub-ms pauses, large heaps, JDK 17+.
Shenandoah
Low pause, concurrent compaction.
MODULE 10

Thread Analysis

JFR shows thread states over time:

  • Runnable
  • Blocked (waiting for monitor)
  • Waiting (Object.wait / park)
  • Timed Waiting

Common Issues Found

  • Deadlocks (multiple threads blocked on each other)
  • Lock contention (many threads blocked on same monitor)
  • Thread pool saturation (all threads blocked or waiting)
  • Executor starvation
All threads BLOCKED
Deadlock or pool exhaustion
RUNNABLE but low throughput
CPU burn in tight loop
WAITING on pool queue
Thread pool too small
TIMED_WAITING on HTTP
Slow downstream dependency
MODULE 11

Lock Contention Analysis

Lock contention is one of the most common hidden causes of latency.

Key JFR Events

  • jdk.JavaMonitorEnter
  • jdk.ThreadPark
  • jdk.LockInstances

Analysis in JMC

  • Lock Instances view shows which locks are most contended
  • Correlate with thread stacks to find the exact synchronized block or ReentrantLock

Impact: Even small contention on hot locks can destroy throughput under load.

SRE TIP

If JavaMonitorEnter duration exceeds request SLA on hot path, even microsecond locks aggregate to seconds under load.

MODULE 12

Database Troubleshooting

JFR can show JDBC-level activity (with some configuration).

What You Can See

  • Time spent acquiring connections from pool
  • Time spent in actual query execution (if using JDBC instrumentation)
  • Connection pool exhaustion patterns

Popular Pools

  • HikariCP (recommended)
  • C3P0, DBCP, Tomcat JDBC Pool

JFR helps distinguish between pool acquisition latency vs actual database query slowness.

Pool vs Query — Decision Guide

Pool acquisition slow

All threads waiting for connection. Fix pool size, leak, or DB connectivity.

Query execution slow

Connections acquired fast but JDBC/socket time high. Index/query/DB issue.

MODULE 13

API Latency Investigation

Scenario: P95 latency increased suddenly.

End-to-End Workflow

  • Capture JFR during the degradation window
  • Analyze:

- Thread pool utilization

- Lock contention

- Database / external call latency

- Serialization / deserialization time

- GC impact on request threads

JFR lets you build a complete picture of where time is being spent inside the JVM for slow requests.

Latency Budget Breakdown

Request thread timeline (example P99 = 2.4s): 0–50ms Controller + validation 50–80ms Jackson deserialize 80–2100ms HikariCP acquire + JDBC (ROOT CAUSE) 2100–2300ms Jackson serialize 2300ms Response sent
MODULE 14

Exception Analysis

Frequent exceptions can silently destroy performance.

What JFR Shows

  • jdk.ExceptionThrow events with full stack traces
  • Frequency of specific exception types
  • Hidden retry loops causing exception storms

Examples:

  • NullPointerException in hot paths
  • SocketTimeoutException / connection resets
  • SQLException from connection issues

Business impact: Increased CPU, latency, and noisy logs.

NullPointerException × 50k/min
Hot path bug — CPU waste
SocketTimeoutException burst
Downstream or pool issue
SQLException in retry loop
Amplifies load during incident
MODULE 15

Kubernetes and JFR

Capturing JFR from Pods

terminal
# Exec into pod

kubectl exec -it <pod> -- /bin/sh



# Start JFR

jcmd 1 JFR.start name=prod duration=120s filename=/tmp/app.jfr



# Copy recording out

kubectl cp <pod>:/tmp/app.jfr ./app.jfr

Advanced Techniques

  • Ephemeral containers (recommended for distroless)
  • Sidecar containers with JFR tools
  • Init containers or debug containers

Kubernetes-Specific Issues JFR Helps Solve

  • OOMKilled pods (memory leak vs heap sizing)
  • CPU throttling vs actual JVM CPU usage
  • Startup delays (class loading, JIT warm-up)
  • HPA scaling problems caused by GC pauses
  • Kafka consumer lag in containerized workloads
kubectl debug (ephemeral container)
kubectl debug -it <pod> --image=amazoncorretto:17 --target=<container> --share-processes
jcmd 1 JFR.start name=k8s duration=120s filename=/tmp/pod.jfr settings=profile
jcmd 1 JFR.dump filename=/tmp/pod.jfr
exit
kubectl cp <pod>:/tmp/pod.jfr ./pod-incident.jfr
SRE TIP

For distroless images, never install JDK into the app container — ephemeral debug containers are the production-safe pattern.

MODULE 16

JFR and Spring Boot

Common Areas to Analyze

  • Controller method execution time
  • Serialization (Jackson) performance
  • Bean initialization & startup time
  • @Async / thread pool usage
  • Reactive (WebFlux) vs Servlet performance

JFR is excellent for finding slow endpoints or initialization bottlenecks in Spring Boot applications.

-XX:StartFlightRecording=filename=/tmp/boot.jfr,delay=10sProfile Spring Boot startup
spring.jmx.enabled=trueEnable JMX for jcmd in some setups
management.endpoints.web.exposure.include=healthKeep actuator minimal in prod
MODULE 17

Kafka Troubleshooting with JFR

Key Things to Investigate

  • Record deserialization time
  • Processing time per record
  • Consumer rebalance storms
  • Producer send latency
  • Serialization bottlenecks on producer side

JFR helps you see whether lag is caused by slow processing, deserialization, or network.

Kafka lag triage with JFR:

  1. High ObjectAllocation on deserialize → payload/schema issue
  2. Long method samples in @KafkaListener → business logic slow
  3. Many threads in rebalance/wait → consumer group instability
  4. SocketWrite blocked → broker or network bottleneck
MODULE 18

SRE Incident War Stories

eCommerce Checkout Slowness

SEV2

Impact: P99 checkout latency 800ms → 4.2s

Metrics/Logs: Grafana: latency spike, normal CPU. Logs: no errors.

JFR evidence: JFR: jdk.GCPhasePause events 1.8–2.1s every 45s. Humongous byte[] allocations during cart serialization.

Root cause: G1 humongous objects + undersized heap. Promotion failures during peak traffic.

Resolution: Increased heap 4G→8G, tuned -XX:G1HeapRegionSize, fixed oversized session cart caching.

Lesson: Always check GC pauses in JFR when latency spikes without error logs.

Payment Gateway Timeouts

SEV1

Impact: 30% payment failures, HikariCP timeouts

Metrics/Logs: Metrics: pool active=50/50, pending threads growing. Logs: getConnection timed out.

JFR evidence: JFR: threads blocked 8–12s on HikariCP.getConnection. JDBC events near zero until timeout.

Root cause: Connection leak in failed-payment rollback path — connections not returned to pool.

Resolution: Hotfix: try-with-resources on Connection. Pool recovered in 3 min.

Lesson: Distinguish pool starvation from slow queries using JFR thread + JDBC timing.

Inventory Service OOMKilled

SEV2

Impact: Pod restart loop in Kubernetes

Metrics/Logs: K8s: OOMKilled. Heap metrics showed growth over 6 hours.

JFR evidence: JFR (pre-OOM): ConcurrentHashMap allocations 2.4 GB/hour. Top class: InventoryCacheEntry.

Root cause: Unbounded local cache without TTL/eviction on inventory SKU map.

Resolution: Added Caffeine cache with maxSize + expireAfterWrite. Memory flatlined.

Lesson: JFR allocation events find leaks days before OOMKill.

Kafka Consumer Lag Storm

SEV2

Impact: Lag 2M messages, consumer group stalled

Metrics/Logs: Metrics: lag ↑, CPU moderate. Logs: rebalance noise.

JFR evidence: JFR: 68% sample time in Avro deserialization. ObjectAllocation char[] and byte[] dominant.

Root cause: Schema change increased payload 4x. Deserialization became bottleneck.

Resolution: Scaled consumers + optimized Avro reader. Consider payload slimming.

Lesson: Kafka lag is not always broker — JFR shows per-record processing cost.

PostgreSQL Connection Storm

SEV2

Impact: API errors, DB 'too many connections'

Metrics/Logs: DB metrics: connections at max. App: thread pool exhausted.

JFR evidence: JFR: 200 threads in TIMED_WAITING on socket read; connection storm to PG.

Root cause: Missing connection pool limit on new microservice — each pod opened 100+ direct connections.

Resolution: Enforced HikariCP maxPoolSize=20 per pod. Added PgBouncer.

Lesson: JFR thread view exposes connection storms faster than DB metrics alone.

Redis Cache Timeout Cascade

SEV3

Impact: Elevated latency across 12 services

Metrics/Logs: Redis latency P99 high. Dependent services timing out.

JFR evidence: JFR on caller service: threads WAITING on Jedis connection pool; ExceptionThrow SocketTimeoutException.

Root cause: Redis single-node CPU saturated after bad key pattern (KEYS in cron).

Resolution: Removed KEYS usage, added read replica, increased pool with backoff.

Lesson: ExceptionThrow frequency in JFR reveals retry/timeout storms early.

Spring Boot Startup 4 Minutes

SEV3

Impact: New deployment pods slow to pass readiness

Metrics/Logs: K8s: readiness probe failures for 240s. No runtime errors.

JFR evidence: JFR from boot: jdk.ClassLoad + jdk.Compilation dominated first 180s. Bean init for 400+ beans.

Root cause: Monolith-style classpath scan + eager @PostConstruct on heavy clients.

Resolution: Lazy init, split actuator, reduced @ComponentScan scope. Startup → 45s.

Lesson: JFR startup recordings justify JVM tuning and Spring refactor with data.

API P99 Degradation — Lock Contention

SEV2

Impact: Checkout API P99 200ms → 1.8s

Metrics/Logs: CPU normal, DB healthy. APM showed gap in 'application' time.

JFR evidence: JFR: jdk.JavaMonitorEnter on com.acme.PromoService.applyPromo — 40 threads blocked avg 400ms.

Root cause: synchronized block around promo DB call — accidental serialization of all requests.

Resolution: Replaced with per-key locking + async cache. P99 restored.

Lesson: Lock Instances view in JMC finds hidden serialization under load.

MODULE 19

JFR vs Other Tools

Decision Tree

CPU Issue?

  • Use JFR (Method Profiling + Flame Graphs) first
  • Fall back to async-profiler for native code or when JFR overhead is concern
  • Use perf for kernel-level CPU

Memory / Leak?

  • JFR (Allocation events) → Best for production
  • Heap dump (jcmd GC.heap_dump) for post-mortem

GC Issue?

  • JFR (GC events + pause times)
  • GC logs (-Xlog:gc*)

Lock / Thread Issue?

  • JFR (Thread states + Monitor events)
  • jstack for quick snapshot

Container / Kubernetes?

  • JFR + kubectl + ephemeral containers
  • Combine with Prometheus + node metrics
CPU? → JFR flame graph first, async-profiler if native-heavy
Memory leak? → JFR allocations live, heap dump post-mortem
GC pauses? → JFR GC events + correlate GC logs
Threads/locks? → JFR + jstack snapshot
OS/syscall? → strace at kernel boundary
MODULE 20

SRE Production Playbook

Recommended Incident Response Flow

terminal
Alert triggered

   ↓

Check Metrics (Prometheus/Grafana)

   ↓

Check Logs (ELK / Loki)

   ↓

Capture JFR (emergency or continuous)

   ↓

Analyze in JMC (start with Automated Analysis)

   ↓

Correlate with system tools (strace, top, iostat if needed)

   ↓

Identify Root Cause

   ↓

Apply Fix / Mitigation

   ↓

Document + Improve Runbook

Emergency JFR Commands

terminal
# Quick 60-second capture

jcmd <PID> JFR.start name=Incident duration=60s filename=/tmp/incident.jfr



# Dump immediately

jcmd <PID> JFR.dump filename=/tmp/incident.jfr

Production Incident Playbook

1. Alert fires 2. Grafana dashboards 3. Log correlation 4. JFR capture 5. JMC analysis 6. strace if needed 7. Fix + postmortem
BONUS

Bonus Modules

JVM Tuning Using JFR

Right-size heap, thread pools, and connection pools from real data.

  • Tune -Xmx from allocation rate trends in JFR
  • Size thread pools from actual Runnable vs Waiting ratio
  • Set HikariCP maxPoolSize from connection wait events

Continuous Profiling with JFR

Low-overhead always-on recordings in production.

  • Run settings=default continuously with maxage=1h
  • Dump on alert via webhook + jcmd sidecar
  • Store .jfr in object storage for trend analysis

AI-Assisted JFR Analysis

Export recordings and use LLMs for RCA drafts.

  • Export event summary to JSON for LLM input
  • Prompt: summarize hot methods, GC, and lock contention
  • Generate draft RCA for human review — never auto-close incidents
PRACTICAL EXPERIENCE

Hands-on Labs

LAB 1

Capture JFR from a running Spring Boot app

Use jcmd to start a 5-minute profile recording.

LAB 2

Analyze CPU hotspot using JMC Flame Graphs

Find methods consuming >10% CPU during a spike.

LAB 3

Find a memory leak before OOM

Use allocation events to identify growing object types.

LAB 4

Diagnose lock contention

Correlate JavaMonitorEnter with Lock Instances view.

LAB 5

Investigate GC pauses in G1 vs ZGC

Compare jdk.GCPhasePause across collectors.

LAB 6

Capture JFR from a Kubernetes pod

kubectl exec + jcmd + kubectl cp workflow.

LAB 7

Correlate JFR with Prometheus metrics

Overlay JVM events with latency spike window.

Labs 1–3 include interactive simulations. Labs 4–7 are designed for hands-on practice in your own JVM or Kubernetes environment.

TEST YOUR KNOWLEDGE

Interactive JFR Quiz

10 questions · Instant feedback

Ready to test your JFR expertise?

Covers jcmd, JMC, GC, locks, Kubernetes capture, and tool selection.

INTERVIEW PREP

Common Interview Questions

Q1

What makes JFR safe for production compared to traditional profilers?

Very low overhead (<1%), built into JVM, ring buffer design, no bytecode injection.

Q2

How do you start a 5-minute JFR recording on PID 12345?

jcmd 12345 JFR.start name=incident duration=300s filename=/tmp/incident.jfr settings=profile

Q3

What JMC view do you open first and why?

Automated Analysis — rules engine surfaces GC, memory, and hot method issues immediately.

Q4

Name three JFR events for memory leak investigation.

ObjectAllocationInNewTLAB, ObjectAllocationOutsideTLAB, and heap usage trends in Memory view.

Q5

How do you capture JFR from a distroless Kubernetes pod?

kubectl debug ephemeral container with JDK image, share-processes, jcmd on JVM PID, kubectl cp .jfr out.

Q6

JFR vs heap dump — when to use each?

JFR live for allocation rate and hotspots; heap dump post-mortem for object retainers and dominators.

Q7

What does high ObjectAllocationOutsideTLAB indicate?

Severe allocation pressure — objects too large for TLAB or TLAB exhausted.

Q8

How do you correlate JFR with an incident window?

Note recording start/end UTC, align with Grafana alert time, filter JMC events to that range.

Q9

What JDK versions include JFR for free?

OpenJDK 11+ (flight recorder enabled by default in modern builds).

Q10

JFR vs async-profiler?

JFR: broad JVM events, GC, allocations, low overhead. async-profiler: excellent CPU/native flame graphs.

TROUBLESHOOTING

Investigation Flowcharts

CPU Spike

CPU alert? → Capture JFR 60–120s → JMC flame graph → Hot Java method? → Fix code / JIT warm-up → Still high? → strace/perf

Memory / OOM

Heap growing? → JFR allocation events → Top class growing? → Leak in app code → OOMKilled? → heap dump + JFR

High Latency

P95/P99 up, no errors? → JFR during window → GC pauses? → Lock contention? → JDBC/pool wait? → External timeout?

GC Problems

GC pause alert? → jdk.GCPhasePause in JFR → Full GC frequent? → Tune heap/GC or fix allocation → Humongous objects?

Thread Issues

Throughput collapsed? → JFR Threads view → All blocked? → jstack correlate → Fix deadlock/pool size

Kubernetes

Pod slow/OOM? → kubectl debug + jcmd → JFR dump + cp → JMC analysis → Adjust limits/heap/code

JFR Production Cheat Sheet

Start Recording

jcmd <PID> JFR.start name=Prod duration=300s filename=/tmp/recording.jfr settings=profile

Emergency Dump

jcmd <PID> JFR.dump filename=/tmp/emergency.jfr

Key JMC Views

  • Automated Analysis (start here)
  • Method Profiling / Flame Graph
  • Memory → Allocations
  • Threads · Lock Instances

Events to Watch

  • jdk.ObjectAllocation*
  • jdk.GCPhasePause
  • jdk.JavaMonitorEnter
  • jdk.ThreadPark
  • jdk.ExceptionThrow

SRE Production Playbook

Alert triggered → Metrics (Prometheus) → Logs (ELK/Loki) → Capture JFR → Analyze in JMC → Correlate (strace/top) → Root cause → Fix → Postmortem