• Amazon / Kindle
• iTunes / iPad
• Barnes & Noble / Nook
• Sony / Sony Reader

Note the online versions are in color if the devices support color (it looks great on an iPad!).  We think the color content matters quite a bit, so if your device is black and white we recommend buying the print version (feel free to buy both!) to get the full experience.

Enjoy!  The eBook price is $9.99.  Cheers, -Chris

Continuing on with my four- part series on how to create a world class problem management function and proactively take control of service quality and get back that precious time you're currently spending on fire-fighting.

Once you’ve created a work environment where problems are freely revealed and you’ve achieved teamwork across functions via clear leadership and shared goals, the next key ingredient is making technical problem solving a core competency.  An effective work environment isn’t enough without your own CSI team who can roll up their sleeves and figure out tough problems.  In our book we describe these as the detective Columbos of your org – the Crackerjack detectives that always get their man.  Calling out the need for technical problem solving expertise may seem like obvious guidance but it’s actually a common gap I see in many problem management functions.  Few organizations do it really well.

I previously blogged about the “Car Talk” guys from National Public Radio (Tom and Ray Magliozzi, also known as Click and Clack) and how they leverage an encyclopedic knowledge of automotive engineering to solve their callers’ issues without ever even seeing the car.  They’re both MIT grads which provides their foundational understanding of engineering mechanics.  They also have an innate knowledge of how things work that comes from years of taking cars apart and putting them back together again in the garage.  Combining the two together, it’s remarkable how quickly they’re able to figure out a caller’s problem rapidly and explain it clearly. 

Another analogy that helps make this point is from the world of medicine.  Doctors have a substantial baseline of scientific knowledge they acquire before ever seeing a patient.  By having an in-depth understanding of physiology, pharmacology, biology, etc., they possess a foundational expertise they’re able to call upon when confronted with a difficult diagnosis.  Web MD’s Boolean logic is fine for common ailments but insufficient for more complex and challenging illnesses. 

Same goes for technical troubleshooting.  There are way too many IT administrators or engineers who think troubleshooting is about calling up the vendor support line (I can save you some time, they’ll tell you to upgrade to the latest patch level) or tinkering with settings in a console without a true understanding of the technology itself.  As technology becomes more mature these skills seem to be in even shorter supply since many new IT professionals haven’t known a world where you have to build solutions from the base elements of technology. 

In my experience the best IT problem solvers operate the same way as good mechanics or doctors.  They tend to have a foundation of classical technical knowledge but almost always possess a never-ending curiosity about the way systems work.  They’re easy to spot.  They like to get their hands dirty.  They’re the ones that have built their own computers, run open source technology on their home PC, can hack just about anything and insist upon seeing for themselves what the code is actually trying to do (as opposed to what the developers say it does).  At corporate IT shops they are the ones who know where all the log files are for the relevant systems and how to interpret them.  They go to the logs first before dragging in other SMEs.

In the end, what type of police detective, auto mechanic or doctor do you want working for you? Think about that when there’s a temptation to skimp on you your problem management investment.  It’s important to hire for this competency and to develop it through training.  Don’t assume it’s going to come naturally.

Next up, the final post in the series, the Holy Grail: Determining meta-problems and making calculated strategic improvements.

-Mike Hagan-

The "Excellent Nervine"... soothes, quiets, and strengthens the Nerves.When the first cases of illness were diagnosed by doctors, no obvious cause could be found. Apparently, during that period when the docs didn't know what they were dealing with, they often concluded that  patients were simply suffering from "The nerves". That was a good catch-all for saying there was something wrong with them but we didn't know what exactly. The doctors would prescribe some miracle syrup, such as root beer, Coca Cola, or Dr Guretin's Nerve Syrup (pictured). Did they really expect those remedies to solve the problem? Probably not, and that might have been okay if there was really no way to understand the cause of the problems and a possible solution.

Unfortunately, in IT we too often see the same attitude towards system problems. Administrators taking the easy path of rebooting a server, or service desk analysts asking a user to reboot their PC to take care of that unstable desktop. Do they really expect the problem to go away? Are they crossing their fingers that the same user won't call back with the same issue?

Fortunately, in 1952 a couple of astute detectives in Sydney started applying methodical investigation techniques to try to understand why so many people were dropping like flies. They quickly started noticing some commonalities between cases, and suspected foul play. They were open minded and explored all possibilities. When they suspected that poisoning might be the cause, and stared sampling the cakes and tea that some women were serving their "loved ones" they found that the samples tested positive for Thallium, the active ingredient in rat poison. Colorless, tasteless, and with no smell, Thallium was the perfect murder ingredient. The product was banned in most of the world, but Sydney had an estimated rat population of 1 million at the time so it was still allowed to be used.

If it wasn't for this great detective work (and the eventual conviction of three serial murderers), more people might have been poisoned. In IT, we've all been faced with problems that seem to poison our days too. Yet too often bogus remedies are applied and the underlying problems are allowed to continue disrupting the flow of business, and in some cases cause crises. In the Opposite of Luck, we advocate treating problems like criminals, even serial killers. It's amazing how applying a methodical approach of preserving the "crime scene", searching for and analyzing evidence, collecting facts, listing suspects, and zeroing in on the culprit can help make your business a better world, and IT a less scary place.

Astronaut Ken Mattingly (played by Gary Sinise in “Apollo 13”) set aside his own personal disappointment to play a critical leadership role during that crisis. Teamwork across multiple engineering teams was key to addressing difficult power issues and bringing the spacecraft home safely.Continuing the series on problem management as a critical means of producing consistent quality and keeping that damned Blackberry from dancing around on your night stand at 4am!  Last week I laid out the four essentials to succeeding at problem management and creating a living culture of constant improvement.  We've covered #1:

Working cross-functionally is the key to getting to root cause of complex problems.  Without teamwork across all teams the hardest problems go unresolved and linger out there like unpaid parking tickets in your glove compartment – every day growing more and more dangerous.  The days of discreet mainframe applications are mostly behind us and today’s distributed computing solutions are a sprawling web of complexity spanning all areas of responsibility in the typical IT org. 

How many times have you seen an incident response where one by one the various on-call representatives chime in and declare their team’s piece isn’t the culprit (database, network, app, etc.)?  After everyone's said their piece is working fine all you hear is cricket’s chirping … yet, the system is still down! 

This is human nature and I see it all the time in organizations where teams don’t share the same quality objectives.  There’s more concern about proving what the issue isn’t rather than figuring out what's wrong.  The preceding example comes from the world of incident management but it’s equally true for problem management.  To achieve the teamwork and partnership needed teams have to share common quality goals and have effective leadership.  Without those things, they’ll just bat the issues back and force rather than dig in and really figure things out.

To ensure accountability, prioritization and focus it’s essential to have an over-arching quality metric and goal shared by the entire IT organization.  Quality must be everyone’s business – not just the operations teams.  However, rather than just tracking availability which is a limited and one-dimensional quality measure at best, we recommend a Business Impact Index that quantifies the true pain inflicted on the enterprise by system outages so that they’re better understood and can be more easily compared across different systems. 

Ideally, the BII metric output would be lost dollars but in most companies that not easily done so a simple point system can suffice.  The important thing is to measure in terms of the business damage caused (lost revenue, customer dissatisfaction. lost productivity, etc.) so that you know which incidents hurt the most.  Without that focus it’s impossible to mass your limited resources on the root causes most in need of redress.  The BII also improves IT credibility and business alignment as it shows you’re holding yourself accountable for the things that hurt them the most.  There’s lots of additional detail about the BII and how to implement it in our book.

Once you know the true impact of incidents you can get to work on the most critical problems.  However, doing so requires teamwork and coordination that will span the entire department.  It’s important to have a senior level leader serve as the virtual owner for the problem management function to ensure accountability, productivity and conflict resolution regardless of where root cause lies.

The senior sponsor should assure that problems are being prioritized; addressed and needed progress is being made.  She should also ensure all functions are making the needed contribution so that resource gaps can be worked.  The sponsor should be prepared to intervene on stalemates, conflicts and vendor escalations. Magic can happen when folks come together to bear down of the most wanted list. 

Come back next time as I discuss how to make technical troubleshooting a core competency.

-Mike Hagan-

As NASA Flight Director, Gene Kranz (played by Ed Harris in "Apollo 13") led a remarkably effective problem solving organization. Not coincidentally, finger pointing wasn't part of the culture.The recent high profile cloud service outages at Amazon and Microsoft have generated a lot of inquiries to me about the challenges of effective problem management.  I frequently give talks on the subject and it’s something a lot of organizations really struggle to do well.

I’m here to tell you though; it’s an absolutely essential component of ensuring customer satisfaction, managing costs and maintaining high employee morale.  Only by being great at getting to the true root cause of quality impacting issues can an IT organization reach its full potential.  Solving problems and taking preventative action frees up the resource cycles and focus to allow you to concentrate on innovating for the business rather than reacting to recurring outages and crises. It also gets you off of the treadmill of reactive behavior that forces long hours and a wildly eratic work schedule. Sleeping with a smart phone under your pillow is just no way to live and it doesn't have to be that way.

These are the four essentials to succeeding at problem management and creating a living culture of constant improvement:

1)      Celebrate the revelation of problems

2)      Drive cross-functional teamwork via clear ownership and shared goals

3)      Invest in and develop technical troubleshooting as a core competency

4)      Classify your problems into meta-problems and make bold targeted institutional improvements where they’ll have the biggest impact

For today’s post we’ll focus on #1.

Celebrating the revelation of problems is first and paramount.  You simply can’t fix problems you don’t know about and you can’t know about problems if people aren’t inclined to bring them into the light of day.  This is a difficult challenge for a great many organizations that struggle to achieve the right balance of ensuring accountability while at the same time maintaining a work environment where employees feel empowered or even safe to bring issues forward.

In organizations that do this poorly, upper management goes looking for scalps after an incident or lower/middle management looks for sacrificial scalps to proactively keep the senior leader wolves at bay. In other more passive aggressive cultures, folks don’t get overtly punished for identifying problems but if they do raise something they get stuck with full ownership without the needed resources or partnership to actually address it. 

In these types of work environments it’s not surprising that employees do whatever they have to do to protect themselves.  At best, they avoid acknowledging issues that they know we’ll bring recriminations or they’ll get stuck owning with little chance of resolution.  At worst, they’ll deny, obfuscate, deflect, filibuster and sometimes even destroy critical data.

It’s the responsibility of leadership to create an environment where employees are held accountable for their actions and results but also where problems aren’t hidden or concealed.  I always tell my teams or clients to be mad at the problem, not the person.  I also tell them they can’t get in trouble for revealing a problem but they sure can get in trouble for concealing one.

It’s also important for your deeds to match you words.  Employees listen to what you say but will be much more focused on what you do.  Sometimes it takes a lot of discipline and restraint to avoid the blame game.  If you have a one-strike-and-you’re-out policy – even if it’s only once in a while, don’t be surprised if significant issues are not being brought to light.

In the end, blame wastes crucial energy that could have gone towards getting to root cause and making proactive improvements. It’s counterproductive, ineffective and bad for morale.

Tune in for my next post on problem management essential #2:  Drive cross-functional problem solving teamwork via clear ownership and shared goals.

-Mike Hagan-

Submarines have water tight compartments to contain damage and help prevent catastrophic failure. Cloud solutions need the same thing.

Although not a good thing in the short term for Amazon or its impacted customers, I expect this incident will ultimately be a step forward for overall cloud service quality and reliability (and ultimately adoption).  It revealed weaknesses that will be successfully and creatively addressed in a number of different ways.  One of those ways is something discussed in our book – proactively building in mechanisms to prevent cascading and escalating system damage from relatively small instigating incidents.

Most every submarine movie I’ve seen has that now clichéd scene where the captain makes the hard call to close a water tight door to prevent the entire boat from filling up with water and going straight to the bottom.  The drama is always heightened by there being crew members still alive inside that flooding compartment.  Men are sacrificed to protect the boat and the mission. The captain only has that horrible but necessary option, however, because the designers saw fit to create the series of compartments and hatches in the boat when it was first built.

From what I’ve learned so far about Amazon’s incident, it appears they had no equivalent of water tight doors to contain the impact of the cascading EBS remirroring that caused the substantial multi-day event.  I’m sure this wasn’t negligence but rather something they just hadn’t anticipated.  After all, we’re still in the infancy of the cloud era and these services are very new and generally reliable. Also, it has been noted by others that customers of these types of cloud services need to be aware of the inherent limitations and architect  to be more fault tolerant.

At its core, the real promise of the cloud is the ability to turn computer processing and data storage into a large scale utility that is more reliable and cost effective.  However, just as with the creation of other large scale utilities like electricity, complexities and dependencies we don’t yet understand will require compartmentalization to prevent widespread outages when smaller local incidents occur.  For example, in the 70’s major power outages occurred where a single instigating event propagated regionally; infamously causing major blackouts in New York City and the east coast of the United States.

Examples abound in other realms as well.  In the geo-political world the assassination of the Arch Duke Ferdinand in Sarajevo was the first domino of unexpected events that ultimately led to WW I.  Hidden relationships in the form of secret alliances between various world powers made it all possible.

The recent global economic crisis shows us how the complexities and dependencies of mortgage backed securities and their incomprehensible derivatives led to a chain of events that almost caused another depression.

In each of these circumstances-  the electrical grid, WW I or the economic crisis - improvements were made not just to prevent the instigating events themselves, but to minimize the potential damage when those same things or other unforeseen or unanticipated events happen again.  Circuit breakers, the elimination of secret treaties between countries and the recent banking regulatory reforms are examples of how the risk is now better contained.

This isn’t the first major outage of cloud services and it certainly won’t be the last but I expect the cloud will become more and more reliable over time.  The service providers will learn lessons and proactively implement governors and system controls as the complicated relationships and dependencies become better understood.  Often those improvements will come proactively from study and analysis but, inevitably, they’ll also come in the form of outage related post-mortems like this one.  The point is, though, they will come.

I’m confident that Amazon and others will create their own equivalent of water tight compartments or circuit breakers and these services will get stronger and better as the natural cycle of learning and improving does for the cloud what’s it’s done for so many other large and complex systems.

-Mike Hagan-

Second, when my ordering experience is so good, I tend to let the little things slide.  Did my pizza take 5 min longer than promised?  Does my salad have a bit too much dressing?  Are they out of Dr. Pepper?  I'm so impressed by the ordering experience that I am easily willing to forgive small transgressions, and in fact I am very optimistic that the next time I go there it'll be better.  I'm a believer.

The last thing is loyalty. Next time my family wants to have pizza, it is so easy to call Pagliacci's that I need to have a really really good reason to go anywhere else.  They have me hooked.

When your business interfaces with IT, what is their experience like?  If it's not as good as this pizza chain, how hard would it be for you to make it first-rate?  How does the business currently react when you are out of Dr. Pepper?

-Chris Oleson-

Now you see it - Now you don't

When I visited Seattle for the first time back in the 80’s I was somewhat amused by the signs on Interstate 5 warning that you’ve entered the apple maggot quarantine area.  If you’ve ever been to Seattle you’ve probably seen them.  I had no idea what an apple maggot was but I had a pretty good idea that I didn’t want to get caught bringing any of them into the state!

Years later, long after I had made Seattle my home, I was picking up a friend from the airport and was a little taken back when he commented about one of those signs.  It startled me a bit because I didn’t realize the signs were still there.  Somewhere along the way I’d stopped noticing the ubiquitous warnings I once had thought were so odd.   They had become so “normal” that at some point in time I had just started filtering them out even though they were still there faithfully doing their job  - defending the region’s apples from slimy invaders.  They had more or less become invisible to me.

This same thing happens in business operations all the time.  The weird or unusual can become so “normal” that we no longer notice because no consequences resulted from the initial occurrences.  This is a phenomenon that can lead to the accumulation of quality issues and extensive customer dissatisfaction or even catastrophe.  IT departments are particularly notorious for making their customers ignore or work around technical issues or limitations.  Often they’re better at explaining these things away than addressing the issues themselves.  Sometimes these things are even given cute names which strangely legitimize there inevitability (think BSD: Blue Screen of Death).

I’m talking about nuisances and productivity killers like painfully long boot up times, random error messages, erratic system behavior, bad IVR’s, three or four mouse clicks to do things that should only require one or two, cumbersome work flows, convoluted request processes and so much more.  Just like with those strange signs on I-5, we stop noticing these workplace issues and get used to them.  They become “normal.”   But, we overlook them at our own peril because customers are becoming more and more insistent upon proactive attention to these things and more.

This phenomenon was labeled as the “normalization of deviance” by sociologist Diane Vaughan in her investigation of the Space Shuttle Challenger disaster where known issues were dismissed or rationalized away until they resulted in an enormous system failure.  My friend John Helm has done tremendous work helping his consulting clients address this type of quality drift within their organizations.

So, what is the key to ensuring the abnormal doesn’t become normal?  It first starts with a focus on measuring service excellence and productivity from the customer’s perspective and holding yourself accountable.  Measure the things they care about in order to ensure you’re focused on the things that really matter most.  Relentlessly seek and action customer feedback.  Aggressive but realistic improvement goals should be established and rewards and recognition should be aligned with the targets to incent the right behaviors.  Make it clear that standing still with quality is the same as moving backwards.  Create a culture that demands constant improvement, allows for revealing problems without incrimination or penalty and takes folks out of their comfort zone making them confront problems and issues before they ever become invisible.

-Mike Hagan-

Greatness Can Happen When Everything Comes TogetherFor me, I’ve always defined an HPT as a group that generates results far beyond what the combined sum of the parts (team members) would seem to indicate is likely or even possible.  It’s that mystical combination of talent and chemistry that can generate extraordinary results.  A great example would be the 1980 U.S. Olympic hockey team where a newly assembled collection of collegiate amateurs defeated a phalanx of international all-star teams jam packed with more talented and experienced professionals.

I think in part the reason why VCU and Butler’s stories along with the 1980 Olympic hockey team thrill us so much is because of how seldom they happen and the enviably tight bonds that seem to exist between the players and the players and coaches.  In business, unfortunately, my experience has been HPT’s are just as rare as they are in sports – perhaps even more so.

Continuing with the sports theme, with baseball opening day upon us it seems timely to reflect on the accomplishments of the New York Yankees the past ten years.  In spite of always having the largest payroll in baseball by far – sometimes five times as much as other franchises – Yanks fans have only seen one World Series championship since 2000.  In 2008 the Bronx bombers had a whopping payroll of $209 million but lost the American League East to the lowly Tampa Bay Devil Rays who only spent a relatively paltry $44 million.  On paper the Yanks assemble the best, highest paid personnel and consistently produce victories but winning it all has proven to be elusive.   Given they have the means to get just about any player they desire not only would you expect them to be winning championships most years but to do so with relative ease.  That simply hasn’t proven to be the case.

Creating a high performing team in business can be every bit as difficult and elusive.  Too frequently we see teams or are part of teams made up of highly talented individuals who don’t meet their full potential or even fail completely when success seemed so likely.  Challenges working effectively together, personal agendas and ambitions, misaligned goals, personality conflicts, ineffective communication, lack of cohesion, the ever elusive lack of chemistry and so many more are often the reasons.  These issues don’t necessarily doom teams to failure and much success can and does happen in spite of these common challenges but they certainly make the realization of extremely high performance impossible.

HPT’s are difficult to create because so much more is required than merely the right combination of individual talents, which, by the way, is the New York Yankees’ approach.  Finding the talent can often be the easiest part of building a team.  Making it all come together in that magical way that coach Herb Brooks accomplished at the Winter Olympics at Lake Placid in 1980 is so much more difficult.  HPT’s require common values, common goals, some sort of cultural cohesion, the willingness to put the needs of team first and, above all, trust.  Trust is so frequently the missing ingredient because highly talented individuals often struggle with making themselves vulnerable.  It’s been my experience that showing vulnerability is an essential step to creating trust in groups

One thing beneficial about the quest for service excellence is the power it can have to help make a team congeal.  Aspiring and aligning around constant improvement for the purposes of driving quality and customer satisfaction can be an effective way to bring a team together in that ongoing quest to create an HPT.  Creating a work environment where the revelation of problems and issues is not only appreciated but celebrated can help make performers feel safe so that they’re comfortable showing vulnerability and gaining each other’s trust.  

In the words of Vince Lombardi, "Individual commitment to a group effort - that is what makes a team work, a company work, a society work, a civilization work."

-Mike Hagan-

The Doctor (1991)

In my own life the best lessons have been about humility.  In “The Doctor,” William Hurt plays a skilled surgeon whose view of his profession changes when he finds himself on the other side of the scalpel.  This film says a lot about how empathy and understanding for those on the receiving side of your work can make you more effective and personally fulfilled.  There’s also a lot here about openness to new ideas.

Charlie Wilson’s War (2007)

I often hear the word “politics” used as a pejorative in business but in truth the absence of politics in any organization would be as unnatural as the absence of bacteria in your body.  In the workplace there are good and bad politics just like there are good and bad germs in nature.  Influencing others is essential to success in business and here is a great lesson in the form of a political film based upon a true story.  CWW is a terrific example of how an otherwise unremarkable leader can accomplish amazing things by understanding organizational structures, who decision makers are, what they care about and how to reach them with your message (note that only the first 60 seconds of this clip is relevant).  See my previous post from 1/4/2011 for more.

It’s a Wonderful Life (1946)

A fine example of leading thru service.  Seeing himself as a failure and wishing he was never born, George Bailey learns that true success comes from making others successfull.  Leadership is not about how many people work for you but how many people you are serving which makes George the “richest man in town”.  For me, the emotional payoff in the final scene is still unsurpassed in any film I’ve seen.

Truman (1995)

An example of how character matters.  He wasn’t an expert on economics or foreign affairs but Harry Truman relied heavily on honesty, integrity, tenacity and just plain common sense to lead the country through the end of World War II and into the start of the cold war.   In spite of the lowest approval ratings by any president when he left office, President Truman is now judged very favorably by historians in part because many of his decisions, although unpopular at the time, have stood up to the test of time.  An excellent bio-pic based on David McCullough’s award winning book with Gary Sinise in the title role.

Master and Commander: The Far Side of the World (2003)

A highly entertaining film about using resourcefulness, ingenuity and leveraging your team’s strengths to overcome major disadvantages.  Previously, I wrote that “12 O’clock High” is the “Citizen Kane” of leadership films.  If that’s true then “Master and Commander” is the “Casablanca.”  It's loaded with great examples and lessons.  Military films can make for excellent case studies because so much is at stake which crystallizes the importance of actions and decisions.  Being on a wooden ship in the middle of the ocean adds even more clarity.  Russell Crowe stars as Captain “Lucky Jack” Aubrey who uses creativity, cunning and leadership by example to overcome a formidable foe on the high seas during the Napoleonic wars.

The Godfather and The Godfather Part II (1972 and 1974)

“For what shall it profit a man, if he shall gain the whole world, and lose his own soul” –Mark 8:36-.  Immensely smart and remarkably composed under pressure, Michael Corleone’s ruthlessness, thirst for vengeance and belief that he’s always at war turns him into everything he vowed never to become. There are some good lessons to be extracted from Michael and Vito's style (keeping your friends close but your enemies closer was practiced by Abraham Lincoln).  That being said,  I don't recommend the horse head in the bed as an effective negotiation tool.  In contrast, there are lessons in failure to be learned from Fredo and Sonny - the former being weak and the latter impulsive and hotheaded.  Overall, however, this is a story about the misuse of power, corruption and the hardening of the heart that comes from both.

-Mike Hagan-