July 25, 2010 by Mike Hillwig
Tomorrow, my company moves into our new facility. It’s about 1.5 miles down the road from our old facilities. We were in two different buildings about 100 yards apart. Now, we’re moving into a single building.
We’re a technology company who makes networking software and quite a bit of the hardware that runs our software .We also make management software. On top of that, we do final assembly and testing in-house for the majority of our products. That means our IT group had two server rooms, engineering had two test labs, customer support had a test lab, and manufacturing had a major facility. For all intents and purposes, we had six data centers. That’s a whole heck of a lot of networking. In our new facility, that is consolidated down to four. The engineering “lab” is massive. Our cabling contractor tells us they probably ran 150 miles of CAT6 cable in the new facility.
Tomorrow’s move meant we migrated our data center Friday night. It was bumpy but successful.
Six months ago, we bought a pair of EMC ClarIIon CX-120 SANs and a pair of IBM H series blade centers with HS22 blades. My coworker and I have spent the past six months consolidating servers, virtualizing servers, and migrating systems to new virtual machines. About 80% of our corporate systems are virtual now. I have to say that it made this weekend’s migration much easier. Well, almost.
In working with the people from EMC, they noticed that we were replicating the LUNs that contained my backup files as well as my tempdb files. This was causing a lot of replication traffic between the two SANs. While they were sitting next to each other, this was fine. But going across a 100 megabit connection, they thought it might be too much. They suggested that we not replicate those LUNs. Foolishly, I agreed. After the migration (which was bumpy itself due to a few self-inflicted issues), the virtual machines that run all four of my primary database servers failed to start. I saw my career flash before my eyes.
Fortunately, EMC sent us an amazing engineer who actually made sure the SANs were completely in sync before moving. All he had to do was present the LUN as a VMWare storage group and my servers came up. The process took about 90 minutes longer than we had expected, but it was still quite successful. And now we have a better idea what to expect when we move our DR to a colo facility next month.
One of the blessings of this move is that we got a new alarm system. And that means my only remaining SQL 2000 instance got retired and replaced with a brand new SQL 2008 instance. That made me happy.