Most teams have a plan for when a server falls over. Alarms fire, a runbook opens, everyone knows their role. Almost nobody has a plan for when the server is perfectly healthy and the data is simply wrong. And that is the situation that gets pointed at in a leadership meeting, live, while the room slowly turns to look at you. That moment is what data incident response is for, and most teams improvise it.
I have been the person the room turned to look at. The instinct is to start changing things immediately, to be seen doing something. Resist it. Panicked changes are how a small data problem becomes a big one, and how a one-hour fix becomes a three-day cleanup.
Slow down for the first few minutes so you can move fast for the rest. Here is the playbook I wish someone had handed me.
The data incident response playbook, step by step
Six steps, in order. The first two are the ones people skip, and skipping them is what turns the one-hour fix into the three-day cleanup.
Step one: triage before you touch anything
Before you change a single value, answer three questions. What exactly is wrong? How far has it spread? Who is already acting on it?
That last one gets skipped and later regretted. A wrong number already sitting in a report someone sent to a customer is a very different incident than a number only your team has seen. You cannot size the response until you know the blast radius.
Resist the urge to fix. Diagnose.
Step two: contain the spread
Now that you know roughly what broke, your job is still not to fix it. It is to stop it getting worse. If a bad feed is pumping duplicate rows into a table every few minutes, pausing that feed matters more than cleaning the rows already there. Clean up while the tap is still running and you will be cleaning forever.
Containment buys the one thing you actually need: time to fix the real thing properly instead of frantically.
Step three: find the source, do not just patch the symptom
Here the temptation peaks. You can see the wrong number. You could just update it. Set the value to what it should be, and the meeting is happy.
Then it comes back next week, because you treated the symptom.
The value you can see is the end of a chain, not the start. Walk it backward. The dashboard read a published table, which was built by a transformation, which read a source feed. Follow the lineage upstream until you find the actual break. This is where good lineage and metadata earn their keep, turning a whole-pipeline mystery into a single suspect stage. Fix it there and it stays fixed.
Step four: fix, and then verify the fix
The fix is not done when you make the change. It is done when you have proven the change worked and broke nothing else. Two different things, and people declare victory after the first one constantly.
Re-run the check that caught the problem. Confirm the number is right, not merely different. Then glance at the neighbors, because the classic incident-inside-an-incident is the well-meaning fix that quietly breaks the table next door.
Verification is boring. Verification is the job. A fix you have not verified is a hopeful edit.
Step five: tell the people who trusted the bad number
Somebody made a decision, sent a report, or built a forecast on that value. They need to hear it from you, in plain language, before they hear it from someone else. Say what was wrong, what is right now, and whether they need to do anything.
It is an uncomfortable message, and it is the difference between a team people trust and a team people quietly stop believing. Nobody expects perfection. Everybody remembers who told them straight.
Step six: the short, blameless review
Once the fire is out, spend fifteen minutes on the question that stops the next one. What broke, why did it break, and what single control would have caught it earlier?
Keep it blameless. The moment it becomes a hunt for who to blame, people stop telling you what really happened, and you lose the only information that prevents a repeat. And keep it short. A wandering two-hour meeting produces nothing. Fifteen focused minutes and one follow-up action beats it every time.
Write it on one page. What happened, the timeline, the cause, the fix, and the one thing you are changing so it does not happen again. A one-page record beats a heroic memory, because heroic memories leave the company and take the knowledge with them.
The real measure of a data team
I used to think a good data team was one that never had incidents. I do not believe that anymore. Data breaks. Sources change, feeds fail, someone fat-fingers a value.
The teams I trust are not the ones who never have a bad day. They are the ones who can take a wrong number in a live meeting and close it out calmly, quickly, without drama, then quietly make sure that exact break never happens again.
Changing the data is the middle of an incident, not the end. The end is when the fix is verified, the right people know, and you have learned enough to prevent the next one. Get there calmly and you become the person everyone is glad they turned to look at.
If you want the full walkthrough, including recovery verification and the post-incident review, I cover the whole response process in my Pluralsight course, Respond to Data Quality Incidents.
Reference: Pinal Dave (https://blog.sqlauthority.com/), X
