Wednesday, May 12, 2010

The 900+ Point Drop in the Dow - A Real-Life Root Cause Analysis Challenge

Just a random thought today...

The other day when the Dow dropped over 900 points, it was blamed on some trader someplace entering an order to sell "billions" instead of "millions" of P&G stock. To date, they still cannot produce the trade or the trader. Something doesn't smell right.

1. I would think that level of trade would require some sort of secondary approval.
2. Isn't there an audit trail of trades that would lead back to the trader?
3. If a billion dollar trade could do this, shouldn't there be an edit or at least warning message, "You have entered an amount in the billions. Click OK to bring down the entire global financial system."

This makes me question if this was really the case. Other possibilities could be:

1. A run on stocks that was truly panic selling and this was a way to explain it away without spooking everyone else in the country. I guess this is the "conspiracy theory" view.
2. A software defect...and maybe not a simple one. This could be one of those deeply-embedded ones. I know a little about how the Wall St. systems and processes work and believe me, this is not beyond the realm of possibility.

It may be impossible to know for certain. One would have to perform a deep dive root cause analysis, go through the change logs (if they exist), look at the exact version of code for everything going on at that time, look at a highly dynamic data get the idea. That probably won't happen. If someone does manage to isolate this as a software defect and can show it, I nominate them for the Root Cause Analysis Hall of Fame, located in Scranton, Pa. (Don't go looking for that...but there is a Tow Truck Museum in Chattanooga, TN.)

Just a random thought...


Cem Kaner said...

It now appears that a significant factor was a legitimate (poorly designed) sell order by one trader.

One problem in "the system" is the stop-loss order. The order works like this. You buy a stock at (say) $10. Then set a 10% stop-loss so if the stock drops 10% (to $9), your broker is to sell "at market" (the current going price.) The broker enters the order into her system (or you enter the order online yourself) and when the stock drops, your stock is automatically sold. There is nothing unreasonable about automating trades like this--being able to define and enter a trade yourself is the basis for stock sale commissions dropping from as $100 a few years ago to $5 today.

I teach courses in quantitative investment modeling (evaluating investment models is, to a large degree, highly automated exploratory testing). Books my students and I read routinely tell us trailing stops are Very Wise And Proper. Stock market cheerleaders (in this case, NOT Cramer) on TV shows will often suggest that people who don't set trailing stops are Reckless Idiots. Several books will tell you that trailing stops are essential to prevent Emotionally-Driven Bad Decisions.

I've seen several simulations (and run a few) that suggest that trailing stops are a poor exit (sale) strategy for controlling losses but the mainstream advice is that "big drop” = “sell at market.”

People aren’t required to use unprotected trailing stops. You can set a trailing stop with a limit (if this drops $1 to $9, put in an order to sell it at $9) or a trading stop with a discretionary limit (try to sell at $9 but accept as little as $8.75).

If you put in a market order, you sell at the current price. If that's 28 cents because the market drops like an exploding rocket, you sell -- and you get your 28 cents. If you put in a limit order, you might not sell if the market drops too fast.

When the market has already been dropping, a large sell order can temporarily drive stock prices down further. Supply exceeds demand. If the trader sells a group of stocks together (e.g. a security tied to all the stocks in the S&P 500 index, together), there is pressure on all those stocks at the same time. Price spikes (rapid rises or drops) in prices of individual stocks that reflect large orders are not uncommon.
In a market that has been declining a big sell can spike a stock lower, so that a few people will have lost 10% from their purchase price. If they use trailing stop-loss orders, they sell. More supply. Market drops. More stop-loss orders. More supply. Sell sell sell.

Imagine you are a buyer in this situation. You see the stock dropping. Do you buy? Why try to catch a crashing rocket? Maybe you should set a trailing order that says, wait while this stock drops and keep waiting until it goes back up 10 cents and THEN buy?

So if the sellers are using trailing-stop-loss-at-market orders (sell at any price) and the buyers are using their heads or their software (trailing-stop-buy-at-a-limit), the market can fall pretty far because almost no one buys until it stops falling.

This is not a problem of high-volume or algorithmic trading. This is a problem in how people specify their sell orders. They sell in a risky way. More prudent, limit, sellers get slower trades (sometimes not as profitable) but take on less catastrophic-sale risk. It's a choice, a decision, it has costs and benefits, including sometimes nasty consequences.

I think the exchanges should have halted trading sooner, but the system, as far as I can tell from the news reports, was probably working as designed and responding predictably.

A lot of people get bad advice on how to trade stocks. How much the software should protect them from the sometimes-nasty-consequences of that advice is an interesting question.

-- Cem Kaner

Randy Rice said...

Hi Cem,

Thanks for your comment. That's a great explanation. It's interesting that many years ago I kept hearing from media sources that dramatic drops in the market weren't possible because of built-in "safety measures" to prevent market crash situations like the 1930's. Now we've seen that it takes a little time for those to kick in.

Also interesting is that the requirement has been met correctly, yet the real-world risk exists. It reminds me of my work with one of the exchanges when various end-users would request modifications that were in direct conflict with each other.

Thanks again for posting your thoughts here, Cem.