Incident Response Runbook

1. Find out what changed.
2. Rollback everything.

FAQ

My change was in a different datacenter.

Roll that back.

I changed the internal network and this is an external problem.

Please be rolling that back.

I only changed staging.

Oh really? Roll it back.

My change couldn't possibly be related!

Don't care, roll it back.

I didn't change code, it's just a backfill

Don't care, roll it back. Turn off the backfill.

Nothing changed! I just added some additional app nodes!

Turn them off.

Seriously it would be impossible for this change to be related.

You keep using that word, I do not think it means what you think it means. Roll it back.