When Bad Things Happen to Good People

When Bad Things Happen to Good People

Sometimes good customer service requires us to be a good customer.

Back in October, I was having a really rough time at work. My client’s project wasn’t going all that well, and the development team at our parent company was being incredibly demanding. I wasn’t sleeping and I wasn’t eating well. My emotional state was in pretty rough shape as well. Clearly, I needed a break.

I walked into my boss’ office and told him that I needed to take a few vacation days. Brian, my boss, didn’t even blink. He saw the pressure I was under, and he was completely supportive of my taking time off. He asked where I was going. “I’m going to Europe for a week.” was my response. His response was brilliant. “Good. Leave your phone with me.” He was not so subtly telling me that he wanted me to take the time off.

I had some frequent flyer miles on Delta that I needed to burn, so I booked an award ticket to Berlin with a layover in Paris.

This is what a tired Mike looks like. It was taken right before I left for Berlin.

This is what a tired Mike looks like. It was taken right before I left for Berlin.

By the time I reached Berlin, I was completely wiped. Between the exhaustion from work and the lack of sleep on the flights, I needed some rest. When I got to the Westin Berlin Grand, I was ready for a nap. My head hit the pillow, and I was out cold. I think I slept for 36 hours. It was glorious.

When I woke from 36 hours of sleep, I found myself surrounded by pillows. 36 hours of sleep was pretty glorious.

When I woke from 36 hours of sleep, I found myself surrounded by pillows. 36 hours of sleep was pretty glorious.

For the next few days, I ate well, rested, and lost myself in one of my favorite cities on the planet. It’s exactly what time off is for.

On my return flight, I was flying Amsterdam to Berlin on Delta. Not sure that I wanted to do an eight hour flight in coach again, I went to the Delta/KLM desk and used some miles to upgrade. Delta’s business class service across the Atlantic is exceptional.

At one point during the flight, Heather, the flight attendant in the business cabin spilled a glass of wine on my shirt. She was completely horrified at the situation. She worked really hard to help me clean up and was completely apologetic. The purser came over to apologize as well. I kept my cool about the situation, knowing that I’ve been through a rough spell at work myself. If I got upset because someone else was having a bad time at work, it wouldn’t do anything for me. Heather worked hard to fix a bad situation, and I changed into a clean shirt while she poured me a fresh glass of wine. In my mind, she make the situation right.

On New Year’s Eve, I was flying from Boston to Mumbai via Amsterdam to visit my offshore teams. As we boarded the Delta A330, a familiar face offered me a glass of champagne. It was Heather from my October flight! I could see that her demeaner changed, and she was going back into “fix it” mode and started apologizing for our earlier encounter. I just told her that “bad things happen to good people” and that she should forget about it.

This was taken with Boston-based Delta flight attendant Heather on our second trip together. This one was taken enroute from Boston to Amsterdam on my way to India.

This was taken with Boston-based Delta flight attendant Heather on our second trip together. This one was taken enroute from Boston to Amsterdam on my way to India.

Shortly before takeoff, the Purser, Mary, came through the cabin to introduce herself. I asked her if she could come back with Heather. She looked at me quizzically, but did so. When she came back, I presented Heather with one of Delta’s “Job Well Done” certificates. Those are tools that Delta gives to top-tier frequent flyers to reward employees who provide great service. Heather went out of her way to make a bad situation right. I asked Heather if we could take a picture. I knew that the story would be worth repeating someday.

Delta Job Well Done Certificate

I always tell my team that it’s not the mistakes we make that define us. It’s how we handle them that is more important. Heather completely proved my belief. Bad things happen to good people, and good people make bad situations right.

Every once in a while, I have bad days at work. How I adapt and fix those mistakes is often more important than the mistakes themselves.

Do More With Less: SQL CMS and MSX

Do More With Less: SQL CMS and MSX

I’m not the typical DBA. The environment I support is in hosting, meaning I support a larger number of environments per DBA than nmost people I know. My team focuses more on the infrastructure and less on the application. I have dozens of the same application across many servers. The next DBA to have my job will look at the environment and say “This is not normal.”

When I first started this job, I realized that we had a problem with how we were doing things. We had maintenance plans everywhere. When we tried to change anything, it would take hours or days to make the smallest change across dozens of servers. That had to change. That’s when I stumbled across SQL Agent Multi-Server Administration. When I figured out how this thing worked, my world forever changed.

There are two out-of-the box tools that SQL Server provides that make my job a lot easier, and that’s the Central Management Server and the SQL Agent Mutli-Server Administration. The premise of this presentation is that these two tools and minimize the administrative overheead for DBAs.

These two tools, while completely independent, compliment each other incredibly well. What they do is completely independent of each other, but when you use them right, they pack a one-two punch.

The Central Management Server

This is one of the most handy tools I work with on a daily basis. If you need to query multiple servers, that’s precisely what this tool is made to do. I’d love to know how this tool evolved, but I’d love to find out. My guess is that some developer at Microsoft needed to query a few servers at once, realized that it was difficult. They probably spent a day or two writing the core internals and it became the tool we have today.

The use cases are many. I’ve used it for security tasks like creating a new SQL login across multiple instances, providing reports for auditors, and even finding a specific SQL Agent job that I knew I put somewhere.  Just recently, I had an incident come into my queue of our service desk software stating that a SQL Agent job had failed. The problem is that it was missing the server where it had failed. I knew the name of the job that failed, and I knew the date of the failure. I just didn’t know what server it had run on. With a few lines of code, I was able to query msdb.dbo.sysjobhistory on every one of my servers to find that thing.

What makes this tool even more powerful is that it allows you to break your servers into categories. In my production environment, we categorize based on the application and then the environment type, such as dev, test, or prod. Within a few clicks, I can query every one of my SSRS servers.
An example of querying multiple servers with the CMS

But wait… There’s more! Microsoft had made it incredibly easy for us to automate the population of our CMS catalog. With two stored procedures,  sp_sysmanagement_add_shared_server_group and sp_sysmanagement_add_shared_registered_server, we can do this without digging to far into the GUI. This data lives in sysmanagement_shared_server_groups and sysmanagement_shared_registered_servers. This makes it much easier for you to integrate with a Configuration Management Database (CMDB) or some other type of master catalog. In fact, I’m in the process of writing the automation to do just that in my environment.

One of the added benefits of the Central Administration Server is that the security is handled by each target server. Even if someone has access to your CMS, it doesn’t necessarily guarantee they have access to the targets. Your security and auditing are still valid.

There is the ability to configure CMS to use SQL logins with earlier versions of the CMS. Don’t. Just don’t.

And if you use Policy Based Management, the CMS plays really nicely with PBM. Make sure you check out the Enterprise Policy Management Framework on Codeplex.

SQL Agent Multi-Server Administration

SQL Agent Multi-Server Administration can be a controversial tool among people who have played with it. I’m actually one of the few people I know who have used this tool in a production environment. We live and die by this thing.

With only three DBAs, we manage over 130 instances. With that ratio of servers to DBAs, it’s pretty clear that I don’t have time to log into every server every day to do a health check. I have a very simple philosophy with server checklists. We automate that crap out of that. If we need to check things on a regular basis, we have a SQL Agent job that does it for us. I have all kinds of SQL scripts to check the craziest things. I was showing my environment to someone at the PASS Summit last year, and they were amazed at the number of jobs we deploy just to monitor our servers.

The problem with SQL Agent jobs is that they tend to sprawl. And keeping them consistent becomes brutal. That’s where Multi-Server Administration (referred to as MSX) comes in very handy. It allows you to define one server as the master server and it owns all of the jobs. From there, you can enroll target servers, and then chose what targets what get what jobs.

There are two problems with MSX. First, it tends to be a little quirky. If something doesn’t go quite right, it can put your MSX-TSX relationship for that target server in a blocked state. Second, when you deploy a job to a target server, the WHOLE job gets deployed, including the schedule. This can be tricky if you deploy IO-intensive jobs to multiple servers at once. Your storage administrator (or the SAN itself) may get a little bent out of shape.

The good news is that both of these limitations can be easily worked around. Give me some time, and I’ll document more about the blocking situation. It’s not as dire as it sounds. And as far as the scheduling thing goes, I’m a fan of using a two-job approach. I will frequently deploy IO-intensive jobs without a schedule. Then I will create a job locally that uses msdb.dbo.sp_start_job to call the master job. That allows me the benefit of keeping my scripts consistent while still having the benefit of scheduling flexibility at the instance level.

One of the questions I often hear about MSX is about the inevitable looming disaster of the master server disappearing. First of all, you should practice recovering that server, including the MSDB database. However, the good news here is that when you deploy a job to the target server, that target server will keep doing what it’s supposed to do, even when the master server goes down. That means if your master server does go south, you can rebuild a new one and then migrate each of your targets to the new master. In three years, that’s not yet been necessary in my environment. However, should it happen, my team will be ready to recover the master server and migrate if necessary. As part of my CMDB integration, I’m moving to a new server. That will will include a new MSX server.

An example of the SQL Agent job targets


There are two gotchas of MSX that I really want to stress. One is that it absolutely requires AD authentication. This simply will not work with SQL Server authentication, nor should it. The other one is security. Be very careful who has the ability to manage SQL Agent jobs on your master server. Someone who can create a SQL Agent job on the master server will be able to deploy that job to a target server, even if they don’t have access to the target. That means if their job has a CREATE LOGIN or GRANT statement, it will run as the target server’s SQL Agent service account. That could be a security gap if you have segregation of duty issues.

Is Your SQL Server Blogger a Drag Queen?

Is Your SQL Server Blogger a Drag Queen?

My friends will tell you that I’m a reality TV junkie. I’m what the people at Bravo TV would call a super fan. Bravo’s Andy Cohen is my hero. He’s the creative genius behind Top Chef and the Real Housewives franchise. I’m completely obsessed with the Real Housewives. From Beverly Hills to New Jersey, I love the housewives.

But my favorite reality TV show isn’t on Bravo. It’s on Logo. My one true reality TV obsession is RuPaul’s Drag Race. For those of you who aren’t familiar, drag queens are male performers who pass themselves off as women, and RuPaul is perhaps one of the most famous drag queens ever. These men have taken this to a true art form, almost to the point where you sometimes can’t tell if you’re looking at an actual woman. They’re incredibly good as passing themselves off as something they’re not–women.

RuPaul in both forms. It's hard to believe that one person can pass themself off as a man AND as a woman.

RuPaul in both forms. It’s hard to believe that one person can pass themself off as a man AND as a woman.

RuPaul’s Drag Race is in the survivor vein of reality TV shows where someone is eliminated every week. The show is campy and a lot of fun.

This is a technology blog. Why am I talking about drag queens?

When I blog or present about SQL Server, I stick to one solid rule. I only talk about things where I have practical expeience. These are things I’ve done in a production environment. It’s stuff where I know what I’m talking about.

Some bloggers will post things that they’ve only seen in a lab environment and pass it off as if they’re a foremost expert on the topic. That frightens me. These bloggers, so desperate to create content and drive traffic to their sites, will come up with scenarios that give credibility that simply isn’t due. Those are the drag queens of the blogging world. They’re passing themselves off as something they’re not.

The good news is that most of the MVPs I know write about stuff they have seen in client environments or in their own practical experience. We then recreate that in a more sterile enviornment so we don’t post confidential or proprietary data.

The next time you read a SQL Server blog, try to look past the makeup. Does the blogger really understand the topic, or is it a blogger in drag?

Drinking the Microsoft KoolAid

I’ve been using Microsoft Windows since version 3.0 and have worked with SQL Server in some capacity for the past twelve years or so. But I’ve been a Mac guy, too. I love my Macs. However, one thing was always missing–Outlook. Yes, Microsoft has had a Mac version of Office for years. But it just wasn’t the same. But I still loved my Macs–until Windows 8 came out.

People seem to hate major updates to Windows, but I think Windows 8 (or Windows 8.1 to be more specific) was a game changer for me. I don’t know why, but it just works for me. And when I present at user groups or SQL Saturdays, it just felt weird presenting on my Macbook Air.

My Mac Mini in the home office was starting to show its age, so I decided it was time for a bit of a technology refresh. I bought myself an HP Pavilion from Woot! and I splurged on a new Surface Pro 3. This is in addition to the Lenovo ThinkPad T440S I have for work. They’re all running Windows 8.1 with Office 2013/Office 365. So far, the only thing I feel like I’m missing is a good replacement for iPhoto.

While I was at it, it was time to retire GMail. I’ve been using GMail for years and was actually paying for Google Apps to get GMail to use my own domain. So I recently switched to my mail hosting to Office 365 as well. Google has a great webmail interface, but Office 365 is better. And using GMail with Outlook via IMAP kind of sucked. Now, I’m using Outlook with Office 365, and I’m loving it. It’s just like what I use for work. It’s perfect. Using a webmail interface works great when you can’t use Outlook. But if you can use Outlook as a desktop app, why not?

As for replacing iPhoto, I have more than 15 years of digital pictures and that library is close to 70 GB. This requires something fairly robust to manage it. I’m giving Adobe Lightroom a spin.

At this point, I still have a lot of love for Apple. I’m heavily invested in the iOS ecosystem. I have an iPhone, an iPad Mini, three Apple TVs, and the entire wireless network in my house is built around the Apple Airport. Last summer, I was having problems with my iPhone 5S and decided to give the Windows Phone a spin. I hated it. I went running back to the iPhone as soon as the iPhone 6 came out.

Some of my friends don’t understand why I would abandon my Mac. The simple fact is that I have made a career working with Microsoft technologies. Today, the desktop and server operating systems are so incredibly similar that keeping one platform just makes sense for me.

My Windows SysAdmin Team: A Key to my Success

DBAs have a natural conflict with developers. I like to think of it as the battle between good and evil. Developers like to break stuff, and our job is to prevent them from doing something stupid in production. Another group we often have an antagonistic relationship it’s is the sysadmin team, whether it’s servers, storage, or network. In my current position, that’s. not exactly the case.

Speaking for myself here, I will gladly admit that I’m only as good as the sysadmins who manage my servers. At this company, we’re in silos. I’m responsible for the database server, but not the windows machine, the storage, or network. That means I have to have a great partnership with my sysadmin team. Notice that I call it a partnership. I truly mean that.

Within my company, my relationship with our sysadmin team is legendary. Chris, our senior Windows guy is nothing short of awesome. I work just as well with Jon and most of their offshore people. Several months ago, my old manager was quite miffed when I got a phone call and dropped everything to help Chris with a problem. She was appalled by the fact that I didn’t follow the process to the letter and wait for him to put in a ticket first. I just looked at her and said “That’s because Chris does the exact same thing for me.” I had to explain that if it weren’t important, he would have put in a ticket or emailed. In this case, he had called my mobile. Without him needing to explain, I knew this was important.

Chris and I have a unique method of communicating. If the IM conversation starts with “Do you have a minute?,” we answer honestly. Then we triage and prioritize. But if the conversation starts with “I need help” then we drop everything for each other.

Our Windows team builds and maintains the servers where I run SQL Server. If those servers aren’t stable, neither are the databases on it. I manage the SQL instances that host many of their tools, including VMWare Virtual Center. Their tools are only as good as the databases I’m managing for them. Our success depends on each other. I recently found a careless mistake made by my own team on a VCenter server. I went ballistic. I look after those guys and they do the same for me.

When I want to try something new on a server, whether it’s a change in permissions, a group policy, or a radically different configuration, I have a conversation with my Windows team before just putting in a request. It gives us an opportunity to collaborate on things instead of just executing someone else’s vision. They do the same for me. One of the greatest lessons I learned in college is that people support what they help create. That is entirely true in my world. Our Oracle DBAs don’t have the same partnership with the Unix team and it shows. Sometimes I think they’re jealous of just how well our relationship works.

Chris and I have an unspoken agreement: we never blindly throw things over the wall at each other. When my boss wants do do something that impacts our Windows team, my response is predictable–“I don’t think it will be a problem, just let me run it past Chris first.”

Sometimes being a DBA means being good with the technology. More often, though, it means being good with people.

Turning Around the Recruitment Call

I’m a regional mentor for PASS. That means I help user groups in the northeastern US with whatever their needs are. This can be anything including marketing, membership, speakers, and sponsors.

The easy part is finding speakers. With a single tweet or email, we can usually help groups find speakers. What I consistently hear is that it’s hard to find sponsors. 

Most groups don’t need thousands of dollars a month to operate. They need enough to pay for the pizza and sodas each month. That money is more readily available than you think. How? 

If you’re like me, your phone rings at least once a week (or once a day) from a recruiter looking to place you with a client. These people spend a ton of money on memberships to sites like Monster and LinkedIn. The hits they get are soft hits at best. At the end of these calls, the recruiter invariably asks if I know anybody who might be looking for job, my response is always “You bet I do.” 

This is where the conversation completely changes. I’m no longer the prey. I’m the hunter. 

I explain to the recruiter that I help organize a SQL Server user group, and I explain the demographics of the group, such as the number of people who participate in our Meetup site, or the number of people who attend our meetings. Every time I do this, I imagine the recruiter salivating. Every member of your user group is a hard hit to them. It’s a recruiting goldmine. 

This is the best part. I explain that I can put them in front of my user group at our next meeting if they’re interested. The recruiter is no longer salivating. She or he is downright drooling. If you want to get in front of 50 SQL Server professionals (or the number that usually attend your group) in one night, we ask you to sponsor us for the night. That’s the cost of the pizza for one or two meetings.  I’m always happy to put them in touch with the member of our group who handles our sponsorship. 

If you’re getting money from a sponsor, you need to help them. We encourage them to do some type of raffle at our meetings, such as an Amazon gift card or some type of a tech toy. Let them collect business cards (or index cards with contact info) from members to do the raffle. Those cards are pure gold for recruiters. It’s absolutely worth the cost of a few pizzas. 

Sometimes this works. Sometimes it doesn’t. But it’s worth that extra two minutes on a call to pull in the occasional sponsor for your group. 

« Previous Entries