The Ever-Expanding Nasdaq Performance Crash Costs

Keeping track of the stated costs of the Nasdaq performance problems during the Facebook IPO of May 18, 2012 is like trying to contain Gorilla Glue - it just keeps growing. This may be the largest single software defect cost in history. The total to date is $72 Million.

First reports were that Nasdaq had set aside $40 million to pay investment firms for their losses, then the number increased to $62 million on March 25, 2013. http://www.nydailynews.com/news/national/sec-approves-nasdaq-plan-dole-62m-reimbursements-facebook-ipo-day-glitches-article-1.1298521

Most recently, the SEC announced an additional $10 million fine on May 29th, 2013. Mention was also made of $2 million paid to investors, but in this meatloaf pile of money, I'm not sure if that was part of the $62 million or in another pile somewhere.

One of the best accounts of what happened is here: http://www.cnbc.com/id/100736915.

This account shows that not only was there a technical problem causing severe performance issues, but alleged securities laws violations resulting from its "poor systems and decision-making".

From the CNBC story, "The SEC said several members of Nasdaq's senior leadership team convened a "Code Blue" conference call at the SEC's request and, thinking that they had fixed the problem by removing a few lines of code, chose not to delay the start of secondary market trading in Facebook shares. But they had not grasped the root cause, the SEC said. The decision to resume trading without fully understanding the problem resulted in violations of several rules, according to the SEC, including Nasdaq's own rule governing the price/time priority for executing trade orders." (emphasis mine)

There were also numerous other rules broken by Nasdaq, which are detailed in the CNBC story.

This is a sad story of: 1. the impact of poor performance in the real world, 2. how just one or two lines of code can bring down a system and cause extensive losses, 3. how making a decision without knowing the true (root) cause can make things even worse, 4. how the cost of the impact of a software defect (not a bug or a glitch) can be extraordinarily high and continue to grow due to lawsuits, fines and penalties.

Unfortunately, the real losers in all this are the individual investors who are still underwater in their initial investment in Facebook.

In this story from NBC (http://www.nbcnews.com/business/nasdaq-pay-10-million-fine-poor-systems-facebook-ipo-6C10109547):

"Facebook's IPO on May 18, 2012 was marred by technical glitches that left the market makers — who facilitate trades for brokers and are crucial to the smooth operation of stock trading — in the dark for hours as to which trades had gone through.

Nasdaq said it tested its systems before the IPO but the testing did not reveal the "design flaw" that caused the glitches."

I hate it when that happens!

Glitches???  That is system failure due to defective code. The story did say "dark for hours." A glitch is a momentary occurrence, but sounds better than it really is. Maybe if we called these failures what they really are - defects, that would start to convey some seriousness of the situation.

Here is an interesting quote from George S. Canellos, co-director of the SEC's Division of Enforcement, "This action against Nasdaq tells the tale of how poorly designed systems and hasty decision-making not only disrupted one of the largest IPOs in history, but produced serious and pervasive violations of fundamental rules governing our markets."

Now, the question is, "Will lessons truly be learned form this?"

A Little Distance Helps You See the Details?

It sounds odd, but sometimes to see details, you need some distance.

Yesterday I was at the pharmacy (for those of you in the U.K., the chemist...) to pick up a prescription. I also needed to get a small jar of Mentholatum.

There were rows and rows of items just for colds and flu!

I wasn't seeing Mentholatum anywhere. As a tester, I realized I was in a "needle in the haystack" problem. I was about to give up, then I noticed the line at the counter had only one person in line, so I got in line.

While I was waiting, I looked back at one of the rows of shelves and from about 20 feet away started to scan each shelf. Then, I spotted the small jars on the bottom shelf, almost hidden from view.

Standing one foot away, I couldn't see the lower shelf at all.

It reminded me of a time when in college, I had an engineering professor who would always tell us, "You are too close to the problem. Step back. Think about it from a distance." Most of the time that was exactly way we couldn't solve the problem!

It's the same with testing. Many times, we don't see defects because we are too deep in the details. We need the bigger picture sometimes.

Just something to think about...

Have a great weekend!

Randy

ISTQB Foundation Level Training - Miami/Ft. Lauderdale, FL, July 16 - 18, 2013

We don't conduct many public courses, but are going to host an ISTQB Training event in July in the Miami/Ft. Lauderdale area (closer to Ft. Lauderdale). If you have been thinking about getting certified in software testing, here's your chance!

There is a 10% discount for groups of 3 or more. Plus, the price includes the cost of the exam ($250 value).

The instructor will be Dr. Tauhida Parveen, an authorized ISTQB trainer and author of two testing books. (And a great trainer!)

For more information and to register, click here:
https://www.mysoftwaretesting.com/ISTQB_Foundation_Level_Course_in_Software_Testing_p/ctflpub.htm

We hope to see you there!