Kubernetes in Production: Troubleshooting & Incident Response
Fix real Kubernetes outages on a lab on your laptop: one triage method, 20+ hands-on drills, from the page to the post-mortem.
You know the basics of Kubernetes. Then something breaks in production, the pager goes off, and every pod looks fine. This course teaches what to do in that moment.
- hands-on drills
- 20+
- sections
- 9
- triage questions
- 5
- cloud accounts needed
- 0
Five questions for any broken cluster
- 1What is broken for users?
- 2Since when?
- 3What changed?
- 4Where does it fail?
- 5What do the signals say?
9 sections, one method
Each section ends with a short quiz. This is the plan for launch and details may still change.
- 01
A triage method
Five questions that work on any cluster, and the commands that answer each one.
- 02
Pods that won't run
CrashLoopBackOff, OOMKilled (exit code 137), Pending, CreateContainerConfigError, and a liveness probe that kills a healthy app.
- 03
Traffic that doesn't arrive
A Service with no endpoints, a network policy blackhole, and DNS blocked by an egress policy.
- 04
Releases
A stuck rollout, rolling back properly, and restoring a broken Deployment.
- 05
Capacity
Noisy neighbours, CPU throttling, and autoscaling under load.
- 06
Day-2 operations
Draining a node under load, upgrading a cluster, and what a backup really restores.
- 07
From the dashboards
Prometheus, Grafana, Loki and Tempo during an incident, and an SLO challenge.
- 08
Blind drills
Drills where you don't know the fault, and a final timed on-call shift.
- 09
The human side
Communicating during an incident, post-mortems that change something, and game days.
By the end of the course you can
- Triage any Kubernetes incident with five questions: what's broken for users, since when, what changed, where it fails, what the signals say
- Diagnose pods that won't run (CrashLoopBackOff, OOMKilled, Pending, config errors, probe kills) with get, describe, logs and events
- Find why traffic doesn't arrive: Service selectors and endpoints, network policies, DNS
- Roll back a bad release safely, and read rollout history before changing anything
- Spot capacity problems: CPU throttling, noisy neighbours, autoscaling that can't keep up
- Drain nodes, upgrade a cluster and test backups without breaking your SLO
- Use Prometheus, Grafana, Loki and Tempo to follow one slow request from metric to trace
- Write the status update, the post-mortem and the game-day plan that come with every incident
Every lecture runs the same way
The lab is SnowOps Labs. It runs a production-shaped Kubernetes cluster on your laptop (macOS, Linux or Windows with WSL2), or in GitHub Codespaces.
- Step 1
We break it
Each lecture starts by breaking a real cluster on purpose.
- Step 2
You fix it
You work it out yourself, with hints if you need them.
- Step 3
The lab checks
A check against the cluster decides whether your fix actually held.
- Step 4
We walk through it
Our path, command by command, and why each step works on any cluster.
Requirements
- The basics of Kubernetes: Pods, Deployments, Services, and running kubectl
- A computer with 4 GB of memory and 2 CPUs free for Docker, and about 10 GB of disk (macOS, Linux, or Windows with WSL2). A GitHub Codespace works instead
- No cloud account and no paid tools
Who it’s for
Developers and junior DevOps engineers
You run services on Kubernetes and want to be calm on call.
Interview candidates
SRE, DevOps and platform interviews ask "a pod keeps crashing, what do you check first?" This is the practice for that question.
Course and certification graduates
You passed a Kubernetes course or certification and want real troubleshooting practice.
Frequently asked
When does the course launch?
The course is in production now. Join the waitlist and you'll get an email the day it opens.
How much will it cost?
The price will be announced at launch. Waitlist members hear it first.
What do I need to know first?
The basics of Kubernetes: Pods, Deployments, Services, and running kubectl. The course starts from the moment something breaks, not from what a Pod is.
What computer do I need?
4 GB of memory and 2 CPUs free for Docker, and about 10 GB of disk, on macOS, Linux or Windows with WSL2. A GitHub Codespace works instead.
Do I need a cloud account?
No. The lab runs on your own machine, and there are no paid tools.
Is the lab free?
Yes. SnowOps Labs is free and open source under Apache-2.0, and you can install it today from labs.snowops.net.
Can I try it before the course opens?
Yes. Watch the walkthroughs on YouTube, play Night Shift in your browser, or install the lab and break your first cluster.