Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /homepages/10/d214220746/htdocs/thecrankydba/wp-content/plugins/jetpack/_inc/lib/class.media-summary.php on line 77

Warning: "continue" targeting switch is equivalent to "break". Did you mean to use "continue 2"? in /homepages/10/d214220746/htdocs/thecrankydba/wp-content/plugins/jetpack/_inc/lib/class.media-summary.php on line 87
Lesson Learned: Only Restoring WHERE you want | Mike Hillwig
Warning: count(): Parameter must be an array or an object that implements Countable in /homepages/10/d214220746/htdocs/thecrankydba/wp-includes/post-template.php on line 284

Lesson Learned: Only Restoring WHERE you want

Leave a comment

September 17, 2009 by Mike Hillwig

As a DBA, my job is to protect the company’s data at all costs. But what happens when it’s the DBA who causes data loss?

I wrote about this a couple of years ago when I accidentally restored a backup copy on a production instance.

One of the things I do on a regular basis is to restore a copy of our databases onto test and development servers. I have this down to a science.  Today, I did the ultimate DBA f**kup. I ran the restore process on the production database server instead of the test server. 30 seconds into it, one of my coworkers came running into my office asking what was happening. I saw my career flash before my eyes. It couldn’t have happened at a worse time. All three of our business units are in the midst of month-end reconciliations, known as CTCs. This happened about 2:30 and we were down for just about an hour. Worse yet, we had to go back to our noon backup, meaning we lost about two hours worth of data.

My boss at the time was an incredibly good guy. I had honestly expected him to show me the door, and he would have been completely justified in doing so. Instead, he asked how we prevented this from happening in the future. Yes, he was using this as a learning/teaching experience.

Today, any time I have a script that does a restore like this, it has a section at the top that looks something like this:

IF @@servername = ‘prodserver’
PRINT ‘This is the production server. You can”t do that!’

ELSE IF @@servername <> ‘devserver’
PRINT ‘This isn”t the DEV server, you can”t do that!’

ELSE
BEGIN

— Rest of restore script goes here
END

I have to admit this little code snippit has has saved my tail a few times since then. I’m also a lot more careful when it comes to doing restores. Mother History is one hell of a teacher.