Gotta love a customer escalation

Guys and Gals, let me give you a tale of doing the right thing the wrong way.

LJ cut and Friends only. RHAT employees, please respect that. Names ahve been changed to protect me. A few more details than what I privately emailed last night.

First Citizens Bank has had a series of zSeries issues open with GSS.
Two dated November 2004. GSS has been slow to respond.

First week of Feb, they call their sales rep and we start the escalation
process. I think I told a GSS rep that “closed unsupported” wasn’t an option. I may have been rude. He deserved it. It gets escalated to the third tier and engineering group in support (SEG).

GSS is still slow to respond. Early March and it’s still in limbo. I escalate again via a phone call to my manager. We talk to the SEG manager. He’s a good guy, and starts running some triage. Takes him almost a week to get everyone on the same page, in a room and an action plan. This is now really an engineering issue.

Engineering is slow to respond. We escalate again. We get cvommits for fixes for some items near, some items in 4-6 months.

So Customer is still not fixed. Customer is starting to make SuSE noises.
We have a large pitch comming for x86, GFS, RHN, the works. GSS and
engineering are making soothing noises, but still no fixes. All of it is scheduled for later.

Friday FCB leaves a voicemail for the sales rep that basically says they’re loosing confidence, and suse is looking like the right thing for their mainframe.

We loose the zSeries, guess what else we’re gonna lose.

I forward the voicemail to the SUpport VP, SEG Manager, Services Exec VP, and Matthew Szulick. I cc my manager and my sales rep on it.

And wait for the hammer to fall.

I am exhillerated. I admit it now. I am having fun stirring up the hornet’s nest, at this point, and walk around in a brief daze knowing full well I am gonna get called out for this, caring and not caring at the same time.

I appologize to SEG Manager, ’cause it’s nothing personal. We’re fedexing our only mainframe QA engineer to the customer (or some equivalent of FedEx).

I get called into Support VP’s Office. She is aggitated.

I am not afraid. I am terrified. I try very hard not to show it. I don’t
think this helps.

She is asking her assistant to find my manager’s cel phone number. I offer it and my phone to her. I think this doesn’t help either.

She asks me point blank what I thought I was doing lobbing this
voicemail to her second and third line managers. I am not appologetic or
penitent. I don’t remember what else was said, but my old manager (now head of GPS) is just outside the door, so he may remember better than I.

She is PISSED. Every answer I give her seem sot make her madder. I bet my parents know how she feels.

I think I said she should feel free to call my manager. I don’t think this
helped either.

I may have had the cat who ate the cannary smile on my face. I do not
expect that helped things.

I am dismissed after she accepts an appology I don’t remember making.

It lasted all of 3-5 mins. I’m not sure what I was supposed to be, but aparently I was Rude. Maybe I didn’t cower. Maybe I was flippant. I now know why the prior Support Director got the hell into sales. She is fucking scary when she’s not happy with you. And she has a very commanding presence.

As I said to Renee in IM, at that point I feel like I’m invincible and fragile as glass at the same time.

I try to call my manager again, and leave a voicemail. Can’t find him or
my regional supervisor. I call the west suprervisor in desperation, leave a voicemail.

Find my regional guy, give him the 10000 foot overview, ticket numbers, etc.

Wait for the adrenaline rush to die off. It didn’t start to run down until about 7pm. I am still wacked out physically.

Supervisor manages to get Manager. We concall. Manager is not pleased. Supervisor is lost. I ask him if this is a repremand when he’s trying to say what there is to say. Poor guy got the manager challenges with me and the South East outside engineer.

Manager and Supervisor are defending me, I think. We are gonna regroup on Monday.

Inside Sales Directory may not have my back, but he’s watching out for me.

Have not heard from EVP or Matthew yet. Monday is another day. I do not expect to hear from Matthew.

Hashed it over with West Coast supervisor on the drive home SE to SE, not SE to manager, off the record. He thinks I did the right thing, but then he claims he almost got into a fistfight with a former RHAT VP. He is a very cool guy, and I wish he was in Raleigh and not Texas.


I don’t know how bad it is, and what kind of mess is awaiting me on Monday. My manager seems to think I burned a bridge and made the whole group look bad.

In the end, The customer will get fixed, so I expect they think I did
the right thing.

I will probably never be a manager or supervisor at Red Hat. I think I
am proud of this. I beleive I did the right thing. I don’t think I made
any new friends, though.

I have been told off the record by some reps that I have big brass ones.

I think I won some points with the reps on the sales floor.
It was a good day for that.

I dunno if VP respects me because I am not afraid of her, or if I have
made an Enemy. Time will tell. Most of my former enemies are now some of my best resources.

I just want to help my customers. I just want someone to take ovwnership of these things and get them addressed.

I probably need to appologize to VP for escalating around her. I have two days to figure out how to do it without getting fired. If I’m not already fired and just don’t know yet.

I have to try very hard not to call bullshit when she tries to shift blame to engineering. Customer doesn’t see or care. Customer sees GSS. Customer perception is truth to the customer.

I must try very hard not to blast the escalation process I bypassed when it failed. It is flawed and doesn’t let anyone take ownership or responsability for an issue. It has a very wear enforcement policy, and doesn’t take customer needs into the equation.

I must not scream how sticking to process is going to fail us, because the process doesn’t care about customers, just engineers.

This what being the rabble rouser feels like. It takes a pebble to start an avelanche. I am a pebble. I do not know if I will be ground to dust on the way down the mountain, but I am trying very hard not to be.

About Kevin Sonney

Kevin Sonney - who, contrary to popular opinion was NOT raised by wolves - grew up in central North Carolina. He fell into the technology field by accident in 1991, when he gave up the wild and crazy lifestyle of an on-air AM radio DJ to become a mundane technical support monkey. The technology industry has never really recovered from this. Kevin has worked for such names as IBM, Red Hat, webslingerZ, and Lulu Technologies (we won't mention the ones that didn't survive the experience). He currently works as a Linux Administrator for Apptio. In his spare time he rescues stray animals and plays video games with his two sons. His wife, we're sad to say, helps him get past the really hard bits. Kevin is still not very mundane, he just got better at hiding it.
This entry was posted in Uncategorized. Bookmark the permalink.

9 Responses to Gotta love a customer escalation

  1. jay says:

    The Customer is the -only- reason the Company exists.

    Inside sales is (all too often) the only advocate for the Customer.

    Support groups, by nature, are reactive at best and at worst, interrupt driven.

    You blew a whistle, called bullshit, etc… but you had the Customer in mind.

    Going around someone is perfectly acceptable once you have shown the attempt at escalation.

    In this case, GSS puts the Company at risk through perceived inaction or a flat refusal to acknowledge the Customer concern.

    I’ve had customer service beaten into me at all stages of my career. If you ask me what part of the vendor relationship is the first to be strained, the answer is easy. Support.

    I don’t give a damn how well the inital engineering or sales process and delivery happen. Everything you do has a lifecycle — and the care and feeding through that lifecycle is a function of Support.

    Sales is how you -get- the Customer. Support is why you -keep- the Customer.

    So, I say this without knowing the full deatils:

    Have your manager go out and buy the team a bottle of this to put on the desk next to the phones:

    Because hey, if you have a set… keep em shiny.

  2. alchemist says:

    About the only details left out were the deliverable dates and a pain with engineering trying to get them.

    Manager seem to think I should ahve given VP annother chance, then EVP, then CEO as last resort. And that he shoudl have been involved the whole way.

    RHAT is so full of beurocracies now it’s killing me. Almost 1000 people, and I know some of this is gonna be needed, but damn, this is killing me.

    Need someone with my Skills at NeoNova? ;)

  3. z_kungfu says:

    Regardless of anything that happens you did the right thing for the customer… Good luck…

  4. jewelyaz says:

    Kev, back when we were doing support together at Da Vinci (gulp) fifteen years ago, I bought deeply into the help-the-customer customer advocate role. It probably ultimately got me my layoff ticket last fall as part of the Windows Server Division 93-person layoff at Microsoft. And yeah, I’m sure I’m not supposed to be talking about this too officially but here we are.

    I went into documentation from Da Vinci, technical writing and editing, first at Northern Telecom, which I mostly loved, and then at M$FT, which I mostly hated. I never forgot those painful eMail 2.0 phone calls, where everyone on the line knew that we’d screwed them and our goal was to figure out how to get them mostly working.

    At Northern Telecom, I instituted some changes to the docs I owned that brought customer satisfaction from 41% to 96% in one year. No shit. What did I do? The documents were “Change Application Procedures” or CAPs. They were the long sets of complicated instructions to be used when updating telco hardware. Some CAPs took 24-36 hours to complete. Nowhere in these documents were there any expected timelines, and though there were plenty of cautions about not leaving a time-sensitive process unattended, there were no built-in breaks AT ALL.

    I insisted that we do (which meant, *I* did, either in the lab or on the alpha sites) time logging so that we could provide estimates for each task. Then I advised them at the start of each time-sensitive process what the expected time-to-completion was, and HOW MANY BREAKS THEY WOULD GET. The breaks were actually in the document; “45 minute meal break expected in 15 minutes”. Ten steps later, “Now go take your 45 minute meal break.” Simple human-being shit that made the telco guys BEG to get the CAP work out in the field. :-) That made me feel good.

    Now let’s go to MS. “Love the customer” culture abounds, but from what I saw, actual awareness of WHO our customers actually WERE and the tasks they were trying to accomplish was pretty thin. None of us had ever met a real customer. With this disconnect, good intentions, and very smart people, we did manage to do pretty well by our customers. I continually fought for the IT people that were forced into the job, the ones without any serious background knowledge, but I was often ignored.

    About two years ago, I got involved in facilitating live customer chats with documentation and dev folks. I loved it! Here were my customers, the IT folks I used to talk to on the phone! They had real problems, and we were not solving them all, either via products OR documentation. I developed a feedback loop into the documentation planning process. Many people (grunts) were excited about it. My brand-new manager was excited about it. Apparently, people above her thought it made them look bad, as if we’d been ignoring the customers for years (well, duh). Just as my new process was about to be implemented, 20 people in the documentation group, including me, got our pink slips.

    It is SOOOOOOOOOOOOOOOOOooooooooooooo good to be out of there. Health insurance might drive me back, but never again into Windows Server — thankfully, MS is huge.

    So, get your Brasso, and stick to your position, one that makes total sense. Apologize, stay professional, but just remind them, “customer, customer, customer”. That’s who pays the salaries and the rent. Customers don’t give a flying fuck about RH’s broken inside processes, the only problems they can see are THEIR problems. So get together and fix them, get the sale. Right on. And THEN work with that VP that you pissed off to address the internal-process problems, because it sounds like you have good insight into what they are. Offer some possible solutions, so that the crisis won’t develop again. Good luck and keep us posted!

  5. jay says:

    Heh. Ya just never know… ;)

    It’s a little different in a sub-20 person company as you can imagine. Yet, there is still a pecking order and toes to step on or, in my cast, stomp on until they “get it”.

    This week I started having dreams about work again but it’s due to weird travel and poor diet. That’s the part of the job I don’t like so much. It’s good for the top line revenue (we’re kicking ass) but it involves paying a price. Our biggest arguments (if you call them that) are logistics related to costs on last minute travel. If that’s the worst we have to worry about, we’re doing quite all right.

    Then again, I’m extremely demanding, focused, and have a hard time accepting anything less than near perfect. Luckily, I report to and deal with mostly type A aggressives… so we can shout, stomp feet, yell, then go get lunch afterwards. I won’t say it gets old though. Everything does.

    But it is -always- about the Customer and for the benefit of the Customer.

  6. clubjuggler says:

    As Jay said, if you don’t have customers, you don’t have a company. You tried it within the system to start with. That didn’t work. Therefore you did what needed to be done. I think the Support VP is just mad because you showed how bad they were doing. That’s not your fault and if there’s any justice in the world they won’t shoot the messenger. (Of course, history isn’t a very good indicator of that, so I’ll wish you lots of luck!)

  7. sandypar says:

    You know that I stand behind you 100%. I know that so many higher-ups at the Hat ignore problems until their managers are aware of the problem. I have played the nice guy who tiptoes around management for 8+ years. I learned this last week that I will never do it again. When VP level management learns of failures, fixes happen.

    You go boy!!!!! Brasso for everyone in the SE group. Koerv will be proud! :-) (She called and left me a voicemail, by the way. She is very curious to know what happened. I may call her back and read your post.)

  8. ovrclokd says:

    *hug* you did the right thing. i’m proud of you. and you definitely have some big ones!

    my one suggestion: do your manager – and yourself – a favor. sit down and document, in writing, every single stupid step of this process. every time you talked with GSS, every time you talked with the customer, everything. doesn’t have to be complicated; just make a bullet-list timeline of how this came about (and, of course, be prepared to back it up with original email messages where you still have them).

    and then give it to your manager. it’ll give him two things: a better perspective on how you got to this point and why you took the actions you did, and a much stronger position from which to back you up.

    it may be an exercise in redundancy at this point. but it can’t hurt. and it lets him know that you really did try to work within the process before you decided that the process was broken & was going to cost you business…

    just my $.02. i hope if i’m ever in your situation that i have the intestinal fortitude to do the right thing. :)

  9. h_postmortemus says:

    Whew.

    Ok, in order:

    1) Do what ovrclokd said and make a bulleted timeline of everything that happened. Not too late for that. At the very least, a good reference for yourself in the future.

    2) Don’t doubt yourself, you did the correct thing. You short-circuited a broken process to get a customer fixed.

    3) The reason you have pissed people off is because you did the correct thing AND they were doing the incorrect thing. You exposed someone as being lazy.

    4) There are many, many more places you can work than just RedHat, so don’t go boo-hooing over that. I sincerely doubt RedHat will be around for much more than another 5-10 years.

Comments are closed.