Tunes for when the servers go down
Recently I was fortunate enough to be a guest on the Talk Python Podcast with host Michael Kennedy.
That episode has been live for a little bit now.
This weekend I saw an email from Michael saying he had been playing around with turning episodes into songs.
It was a bit surreal because it did use a lot of direct quotes and references to things I commonly say.
Things that, if you are not an AWS nerd like me, you might not normally understand.
I could imagine blaring this at 3 AM during a server fire would be less than soothing, but perhaps I will crank it up next time I am called in to fight a major outage.
I enjoyed it and hope you will too.
Here are the lyrics:
[INTRO - PHONE BUZZ, DISTANT ALARM]
Three A.M.
Your site's down.
Everybody's awake.
One question...
[DRUMS CRASH IN]
WHAT DID YOU BREAK?
[VERSE 1 - FAST TALK-SUNG]
Tuesday afternoon, everything was fine. Clicked around the console, said, "I'll document it sometime."
SSH in, change a thing, little fix nobody knows.
Six months later, server's burning...
How'd we configure those?
Terraform plan. Read the lines. Don't just hit apply.
"Small database rename..."
[BEAT STOPS]
Why does that say DESTROY?
[PRE-CHORUS]
Every shortcut feels okay when nothing's on fire.
Then your phone lights up and your heart rate climbs higher.
[CHORUS - HUGE]
IT'S THREE A.M.!
Is it a BUG? Is it a BILL? Is it a BREACH?
Everything you needed now was decided weeks ago in peace.
THREE A.M.!
No sleep. No clue. Everybody watching you.
Write it down. Lock it down. Build it so you know what to do...
AT THREE A.M.!
[VERSE 2 - RAP]
Access key in a dot-env, seems harmless enough.
Push it public, eight minutes later...
SIXTEEN MILLION SPAMS!
[BACKING VOCAL] WHAT?!
Rotate the key. Don't delete the commit and pretend that makes it clean.
If it touched the public internet, consider that secret seen.
I-A-M roles. Least privilege. Give it only what it needs.
And that little star means EVERYTHING.
[SHOUT] ASTERISK IS THE ENEMY!
[CHORUS]
IT'S THREE A.M.!
Is it a BUG? Is it a BILL? Is it a BREACH?
One wildcard permission puts the whole account in reach.
THREE A.M.!
No sleep. No clue. Everybody watching you.
Scope it down. Lock it down. Make the boring choices at two...
So you're ready at THREE A.M.!
[BREAKDOWN - HALF TIME]
Public database?
No.
Public bucket?
No.
One security group for everything?
Come on, man.
[RIFF]
Load balancer out front. Database stays inside.
Private subnet means there isn't even a road from the outside.
[VERSE 3 - FAST PUNK]
Servers aren't puppies, don't give 'em little names.
Don't hand-feed one E-C-two box and pray it stays the same.
Build the image. Ship the image. Kill it, bring it back.
Cattle, not puppies.
[BACKING VOCAL] SORRY, FIDO!
Pipeline on commit. No hand deploy.
CloudWatch tells you why prod got destroyed.
But query five years of logs before you check the price...
You might solve the outage and create a second crisis.
[BRIDGE - QUIET, TENSE]
Then the bill comes in.
Four thousand extra.
Twenty thousand QA.
Ninety-five grand somewhere else because nobody set the brakes.
Sometimes traffic means your product finally won.
Sometimes traffic means...
[PAUSE]
RUN.
[HEAVY DROP]
[VERSE 4]
S-three stays private. CloudFront at the gate.
Signed URL, two minutes, then it expires away.
W-A-F up front. Rate-limit cheap.
Block obvious garbage before the fancy models eat.
And when the agent says, "Don't worry, fixed your stack,"
Have another agent try to break its back.
Read-only first. Human approves writes.
Don't give the robot production because it asked you nicely.
[FINAL BUILD]
And if you want to know how your team reacts...
Don't wait for real life to launch the attack.
Break it on purpose. Let everybody learn.
Rebuild with Terraform.
Then do it again.
[FINAL CHORUS - MASSIVE]
IT'S THREE A.M.!
Is it a BUG? Is it a BILL? Is it a BREACH?
You don't learn incident response when the flames are at your feet.
THREE A.M.!
You practiced this. You know the route. Nobody's freaking out.
Logs are there. Backups work. The playbook's coming out.
THREE A.M.!
Break it now so later you can sleep.
Least privilege. Private networks. Infrastructure you can repeat.
THREE A.M.!
Site comes back. Everybody breathe.
[OUTRO - PHONE SILENCES]
Postmortem tomorrow.
[PAUSE]
No blame.
Just...
please stop putting asterisks in I-A-M.
[CLICK]