When Defects Go Big Time

I remember a conversation with a test manager many years ago who worked for a major airline. This gentleman told me about the great challenge of ensuring ticket prices are correct. He said that if the price is off by just a few dollars, millions of dollars can be lost in a day - and this was in the 90s!

I'm sure by now you have heard about the free tickets issued by United Airlines recently. Those of us in testing are thinking, "Man, I'm glad that wasn't on my watch."

Regular readers of this blog and my newsletter know that I don't attack companies over public defects. I prefer to use them as learning experiences and let people make up their own minds.

Sometimes in class case studies, we make a list of risks. One of the risks is often "bad PR" or "loss of credibility/trust". I often tell clients the two places they don't want to wind up are on the front page or evening news. It's hard to measure that kind of impact.

In the recent United case, there are measurable costs, as United as decided to honor the bookings.

El Al Airlines had a similar defect on August 8 of last year. According to reports immediately after the defect, the airline said it would honor the fares. But, a day later they walked that decision back. They did eventually honor the prices.  "In all, about 5,000 tickets were sold before the error was fixed. El Al blamed an outside contractor for the mistake." (http://newyork.cbslocal.com/2012/08/08/website-glitch-offers-travelers-a-big-airfare-bargain/)

A few years back when I was working with one of the big travel websites, they had a defect in the currency conversion rates. People were booking rooms in the U.K. that could cost up to $1,000/night for a penny per night. The company only honored those which were part of a package deal because the people could rightfully say they didn't know there was an obvious problem. I think they honored something like 4,000 reservations!  (By the way, the big travel websites have a huge data quality challenge because they depend on the vendors to provide pricing.)

At the end of the day, it often becomes a PR decision. There have also been legal cases on these kind of problems where the case is based on whether or not the website is an order entry system or not. Now, if you or I book the wrong dates, they will charge us a fee to change the ticket. 

However, in researching this article, I did discover that, "In January, the Department of Transportation enacted a new regulation to help protect consumers when they’re buying airline tickets. It states: 'The seller of the air transportation cannot increase the price of that air transportation to that consumer, even when the fare is a mistake.' But that regulation has never been tested in court. So as far as consumer rights attorney Brian Bromberg is concerned it’s still really up to the passenger to take action." (http://newyork.cbslocal.com/2012/08/09/el-al-to-honor-all-tickets-in-price-snafu/)

Back in 2003, Sheraton had a similar defect where they sold $850/night hotel rooms in Bora Bora for $85. http://usatoday30.usatoday.com/travel/news/2003/2003-01-14-starwood2.htm

Sheraton chose to take the PR hit and not honor the bookings. "Over two days, 136 people booked 2,631 rooms at the cheap rate and some made multiple reservations covering more than two months of vacation, Starwood says. If all the reservations were kept, the glitch would cost the resort $2 million."

In the above referenced article you will also see this little factoid, "United Airlines has had several glitches on its United.com that let some passengers pay $25 for San Francisco-Paris flights and, more recently, $5 for Chicago-Denver flights. In each case, United honored the cheap fares." And, there have been other United pricing defects, such as in July of 2012, tickets to Hong Kong for $40.

"It's deja vu all over again," to quote Yogi Berra.

The other side of the argument is like if you or I to the store and an item scans for .01, chances are either most people would not feel right about paying the incorrect price. Plus, the cashier would probably call the manager and they would take 10 minutes to find out the right price.

We don't know the reason yet for the recent United Airline ticket pricing defect, so I can't say much beyond speculation. I would love to see the root cause analysis. I hope United tells the public the cause much like NASDAQ did on the Facebook IPO performance defect.

The part that troubles me is that system defects of all sorts are becoming a pattern with the airline industry. From scheduling systems to ticketing systems and website problems, the stories are almost expected. My real concern is when the safety line will be crossed. I predict it will happen. With today's "systems of systems", there are extremely high levels of system integration. These systems are very difficult to test, to say the least.

I was on a flight once that was delayed because the database on the plane wouldn't work. The mechanic came on board with a CD to fix the problem! Avionics are one thing. The integration between systems is another.

The one lesson I know for sure is that software defects can get expensive, either in direct losses or intangible losses in image and confidence. People are getting used to minor defects in software, and we know there will always be bugs. But just like in Jurassic Park (I recommend the book over the movie), we need to be very careful. Some of these defects can grow into monsters.

I would love to hear your thoughts on this one!

Randy

Software Testing Master Class and Testing Mobile Applications - Salt Lake City, UT

I'm excited to announce two events in October in Salt Lake City:

Software Testing Master Class (Advanced 4-day workshop) October 21 - 24, 2013

This is a unique session that is project-based and covers advanced topics in software test management and software test analysis and design. We will be learning by testing an actual project in four days.

Click here for more details.

Testing Mobile Applications (Full-day tutorial)
Friday, October 25, 2013

Mobile applications are not only the future, they are here, now, and need to be tested. The big question is "How?"  In this tutorial, bring your mobile device(s) and we will explore a framework for testing mobile applications, look at some of the tools and generally get a view of the mobile testing landscape. Click here for more details.

Book Review - Peopleware, 3rd Edition

Click to buy on Amazon.com

I have been a fan of this book from the first edition in 1987 because it brings weight to the human factors in computing. Peopleware, first edition, caused me to think about the relationship between workspace and productivity.

Unfortunately, these “people issues” are prioritized at the bottom of the stack in IT. However, most people in the trenches know that people make or break what we do in IT both long-term and short-term. The most critical and chronic problems are not technology-related, they are people-related!

A minority of managers fully understand the impact of people in IT projects. The rest of the management population tends to treat people like interchangeable components that can be located anywhere in the company and become instantly productive.

This is one of those books that you wish your manager would read and adopt. The problem is that too often, the management solution of people issues is reorganization or layoffs. A few rare and valuable companies that do value people and their long-term value have learned that people require time, care and feeding to be productive.

The value in the third edition of Peopleware is that DeMarco and Lister have had about 25 years to validate the insightful book they originally wrote in 1987. For sure, a lot has changed in the workplace since the 80’s, especially the IT workplace. Cubicles come and cubicles go, and some dysfunction is very much the same. The third edition clarifies many key issues in short and concise chapters that not only point out the problems, but also offer solutions. The third edition is definitely a value-added update to a classic.

One insight in particular I took away from the book is the long-term cost and effectiveness impact of employee turnover. Some companies seem to totally ignore this impact. DeMarco and Lister start with a learning curve assumption of three months for a role with moderate complexity. This is in addition to the existing experience and knowledge a person might have. The curve can be from six months to two years in some companies, which places the net capital investment at $200,000 per person.

So when a valuable person (like yourself) leaves a company, most managers won’t do what it takes to keep you, or even to fix the issues after you leave. Instead, they continue to pay out this cost without even knowing the actual costs incurred. And this doesn’t even include the cost of delayed projects, mistakes made by the replacement person, etc.

In case I haven’t made the point - In my opinion after 35 years in the IT profession, this is the one book I think every IT professional should own, read, and hopefully, apply.