Drilling for Gold

I was watching Gold Rush last Friday night on Discovery channel. The crew felt that they needed assurance that there was some gold in the ground below them. As one of the guys remarked, "Unfortunately, there is no arrow on the map that says, 'dig for gold here'".

In fact, one veteran gold miner said, "No drill, no dig."

So, they hired a guy to bring a drilling truck out to the site to drill all the way down to bedrock. Then, they removed the dirt from the drill bit, placed it into individual buckets (one per hole), and panned the dirt for gold.

By doing this, they confirmed there was gold in the ground where they were currently digging, and no gold at all in their "plan B" location.

Does this sound familiar? They were "testing" for gold by taking samples. The results of their tests were very valuable in guiding the next steps of their work. The risk is that it costs about $1,000 per day just in fuel costs to run the earth movers.

I think this is a really great analogy of what we do as testers, especially in black-box, top-down testing. The defects are like the gold. In fact, a software defect is more valuable than one ounce of gold when you consider the cost savings of post-implementation re-work! Then, when we learn from the defects and improve our practices, the defects have even greater value.

The main difference between gold mining and testing is that we're not always sure when something is a defect, but you can't miss gold.

My approach in top-down testing is to cover the critical workflow processes (scraping the surface). When defects are found, drill down to find others. This form of sampling makes use of the principle that defects tend to cluster.

Like gold mining, in testing, there are no indicators that say, "Look here for defects." We have to sample wisely to gain insight where to focus our efforts. There's gold (or defects) in them there hills!

Small Business Saturday and Cyber Monday Specials

We are excited to be a part of Small Business Saturday!

This is a special opportunity for you to support us as a small business and to get major savings on our e-learning training courses in software testing and software quality.

Every e-learning course will be discounted from Saturday, Nov. 26 through midnight on Monday, Nov. 28th. This even includes our ISTQB Foundation-level course, which includes the exam and additional text book!

There are no limits. Buy as many courses as you like!

Plus, if you pay with American Express, you can get an extra $25 back!

American Express is promoting Small Business Saturday again this year. Register your American Express when it opens November 1st, and then use your registered card to shop at a participating small business on Saturday, November 26th. You’ll receive a $25 statement credit when you spend $25 or more at participating small businesses. From this press release:

"For the second year in a row, American Express will help drive traffic through the doors of small businesses with a special incentive: a $25 statement credit offer for Cardmembers when they register their Card and spend $25 or more on Small Business Saturday at any qualifying small business that accepts the American Express Card. Cardmembers will be able to register their cards in early November."

To register your American Express card, visit http://www.facebook.com/SmallBusinessSaturday

Here's how to get these savings on our e-learning courses:

1. If the training is for your team, get your budget allocation and approvals from your management. This is a great way to make the best use of remaining 2011 training funds!

2. If the training is just for you, don't spend all your money on Friday!

3. Visit our e-commerce site at http://www.mysoftwaretesting.com and buy the courses you like. The discounted prices will be shown.

Remember that you have 12 months of access to the courses, even after you complete the course! To see other benefits and features:

http://riceconsulting.com/home/index.php/e-Learning/benefits-of-e-learning.html

http://riceconsulting.com/home/index.php/e-Learning/features.html

To see demos of the courses:

http://softwaretestingtrainingonline.com/moodle/course/category.php?id=11

Of course, I am always happy to answer any questions:

http://riceconsulting.com/home/index.php/component/option,com_chronocontact/Itemid,90/

I hope you take advantage of this rare opportunity!

Randy

Book Review - The Economics of Software Quality

Publisher: Addison-Wesley
ISBN-10: 0132582201 ISBN-13: 978-0132582209
Publication Date: August 3, 2011
587 pages, hardcover


In this book, authors Capers Jones and Olivier Bonsignour quantify the factors that influence software quality and provide information for people to gain insight into how their projects might compare to others. The measurements in this book are based on thousands of software projects.

One of my frequent complaints about the software industry is that we just don't measure very many things. However, thankfully there are people like Jones and Bonsignour that do have a rich source of metrics from enough projects that we can learn from them.

Capers Jones has long been considered the source for software quality metrics. To me, Capers is the "numbers guy" of our profession. With over 40 years in the field, Jones has a wealth of information he has maintained and published over many years.

Olivier Bonsignour is responsible for Research & Development and Product Management in a continual effort to build the world’s most advanced Application Intelligence technology. Prior to joining CAST, Mr. Bonsignour was the CIO for DGA, the advanced research division of the French Ministry of Defense.

For example, the authors state that "high quality levels are invariable associated with shorter-than-average development schedules and lower-than-average development costs." This finding is based on over 13,000 projects between 1973 and today.

The authors maintain that the real economic value of high quality software is not the cost to fix defects, but rather:

• the reduced likelihood of canceled projects
• the reduced risk of litigation
• shortened development schedules
• lower development costs
• reduced warranty costs
• increased customer satisfaction

This book addresses:

• What is software quality and how do we define its value?
• How can we estimate and measure software quality?
• How can software defects be prevented?
• How can we find and remove defects before testing?
• What are effective ways to test software and measure its effectiveness?
• What is the current state of post-delivery software defects?
• How do projects of various characteristics (low, average and high-quality) compare?
• How can technical debt be addressed from a business value perspective?

You will find a multitude of data from projects in a variety of industries, at various levels of quality, and at various levels of practice maturity. You will see by the numbers which project approaches work and which ones don't work very well.

By reading this book, you will gain insight not only into the current state of software quality, but you will also learn about measurement and metrics of software. These are critical things for any software quality professional to learn. In fact, after reading this book, you will know more about software measurement than 95% (that's my estimate) of testers and QA professionals.

I highly recommend this book, not only as a guide for software quality efforts, but also a benchmark for your own efforts.

The Greatest Risk of All

I perform quite a few risk assessments on projects and in organizations. There is one risk common in almost every situation although it is not always identified. This risk is also something I have experienced generally in life. In fact, it is a risk so familiar it’s easy to miss. People get burned by this risk all the time.

Hans Solo understood this risk when he admonished young Luke Skywalker, “Don’t get too cocky, kid.”

There are many ways this risk can be seen:

• People who tune out mentally in training because they think they already know the material. However, these people hardly ever ask any questions, either.
• Project planners who indicate all contingencies are considered.
• Everyday situations where we buy something, then learn the details about the product that are discovered only after trying to resolve a problem.

What is this risk that is so common and so elusive?

It is the “unknown unknowns.” There is nothing new about this. British Mathematician Alfred North Whitehead, who lived from 1861 to 1947, said “Not ignorance, but ignorance of ignorance is the death of knowledge”.

In an article published recently in the journal Science Communication, Jerome Ravetz writes:

“The idea of ignorance of ignorance is quite unfamiliar. Indeed, scientific culture generally suppresses awareness of ignorance. But ignorance of ignorance was quite well-known from Plato and Socrates onward; it became unpopular in the scientific revolution with Galileo and Descartes. Since then, the triumphalist faith that science would provide the good and the true has put ignorance to one side, and led scientists to the sin of pride in their scientific conquests.” 1

We know some things pretty well. For example, we know we need to test software. We also know we can’t find all the defects in software. We know where we have missed defects in the past.

The problem is, when we tackle something new, the rules change and no one is there to tell us what the rules are and how others play to the rules or cheat the rules. The risk comes into play when we ignore that fact or minimize it.

I find much of the programming on television to be lacking, but sometimes I do get hooked on a show. One “reality” show in particular, “Gold Rush” on the Discovery channel, has grabbed my attention. This show demonstrates the unknown unknowns very well.

Looking for Gold in All the Wrong Places

“Gold Rush Alaska ” is about a team of rookie gold miners who go to Alaska to find gold. The oldest member of the team tried many years ago to find gold, but that’s about all the experience that exists on the team. It seems that at almost every step in their gold finding effort, they discover what they didn’t know. The first hurdle came when they had to drive their main earth mover through a wide river because the bridge was too old and weak to handle the weight. Then came the time when they called in a gold mining expert who showed them how their dirt washer was letting much of the gold pass right out the exit and into the river. Once again, an “unknown unknown”. There were also dozens of other things these guys learned and are still learning. I admire their tenacity and faith. I hope they succeed before they run out of money and hope.

It is often said that “Beginners make dumb mistakes.” I prefer the way that Jerry Weinberg says it instead – “Beginners make beginner mistakes.” That’s a good thing for us software people to remember.

Phases of Realization

I have observed that people tend to go through phases when faced with a new situation:

1. Cluelessness (Blissful ignorance)

At this level, people are so unaware of their situation and risks, they “don’t know that they don’t know what they don’t know.” This is total cluelessness, but also very common. It’s like deciding to build a house without really understanding all the problems and stress. It’s very exciting at first to be thinking of your dream home, but then the decisions arise, delays are encountered, things may be done incorrectly and so forth. Ask anyone who has ever had a house built and they probably have a list of ten or more major things they would do differently. One of those things might be “not to have a house built.”

2. Curiosity

This is where you have enough objectivity to at least ask, “What are we missing?” You try to analyze and plan as much as possible, but it’s still only partial understanding of the possibilities. It’s much like going on a trip, thinking through all the clothes you will need, only to learn that the weather at your destination is much different than what you expected it to be.

3. Cautious awareness

In this phase, you are aware of the possibility of that “unknown unknowns” exist. You try your best to anticipate the unknown, but you hit the wall everyone hits. That is, you can’t prepare for everything. You might also think of this as “risk acceptance”.

4. Contingency planning

Realizing that you can’t prepare for everything totally, you put in place reserves to account for the unexpected. These are your Plan B, C and D for dealing with the unknown. In fact, some of the contingencies may seem totally outrageous.

In Oklahoma, where I live, most people have some type of tornado precautions and contingencies. Not so much for earthquakes until this week. Last Saturday evening, November 5, 2011, about 11 P.M. we had a 5.7 earthquake in central Oklahoma. In the past year, we have had a sharp increase in earthquakes. It’s odd to think that I have experienced more earthquakes in Oklahoma than I did while working nine months in San Francisco a couple of years ago! Sure, we’re small time compared to other places, but I’m thinking that more Oklahoma businesses are thinking about earthquake drills. 2

5. Confidence

Now that you understand the risk of the unknowns and have some back-up plans in place, you start to feel more confident in your ability to succeed. However, it is a false level of confidence in the contingencies.

Next to the early phases of cluelessness and curiosity, this may be the most dangerous phase because it easy for us to think we know more than we actually do know. We think, “Don’t worry, we have contingency plans in place.”

This reminds me of the time many years ago when one of my clients needed to restore the main database to recover from a power outage. Then they learned that the operator had failed to mount the second tape during the backup process. For some reason, the operator had misinterpreted the message that prompted for the next tape. This had been going on for months!

6. Comprehension

There is an old adage about judgment and experience. “To avoid mistakes, you need to have good judgment. To get good judgment, you have to make some mistakes.” Comforting, isn’t it?

There are two basic ways to learn: From your own mistakes and from other people’s mistakes. The good thing about learning from other people’s mistakes is that you avoid their pain. The bad thing is that we tend to observe these mistakes and think, “Oh, that not going to happen to us.”

A good thing that happens in this phase is that people may have the foresight to get an external opinion. This would have really helped the gold miners early on.

7. Calibration

Once you have asked the key questions, developed some contingencies, and reached out to others, you are ready to make adjustments. The question is, what will those adjustments look like?

Another bit of wisdom I learned from Jerry Weinberg is the “ Hudson Bay Start.” Back in the days of the Hudson Bay Trading Company, when the Canadian frontier was being settled, part of the plan was to travel west for one day. Then the next day, they would go back to the start and pick up the things they realized they missed the first time.

This is why I like pilot projects with low risk. You get a chance to learn what you don’t know. You can make mistakes without a lot of the downside risks.

It’s important to understand in this phase that the risk of the unknowns still exist. Going forward, you have to rely on your ability to adapt to new challenges and find a way to overcome them. This is where all of your experience and wisdom are needed.

8. Compilation and completion

It’s amazing how quickly we forget things. A good practice is to actually write down the things you learn and review them from time to time. My own practice for this is to keep a journal of the important lessons I experience. I may experience a lesson several times before I really “learn” it.

As someone has said, “All lessons will be repeated until learned.” I don’t know who said that, maybe I read it on a bumper sticker. But, I know it’s true.

Albert Einstein is often quoted, “Insanity: doing the same thing over and over again and expecting different results.”

My point here is that if we don’t learn from our mistakes and omissions, then we deserve the pain from them.

I’ll admit that achieving this kind of learning at an organizational level is difficult and rare. This is not about sponsoring training programs – it is much more experience-based than training. I think this level of learning must be personal. Organizations and institutions don’t learn – people do. It takes individuals to think, reflect and change their behavior.

What Can We Do?

This is the troubling thing about the unknown unknowns. You never totally mitigate this risk. However, there are some things we can do that may or may not help:

• Stay humble and recognize that unknown unknowns exists.
• Be flexible and adaptive when mistakes and problems occur.
• Learn from yourself and others.
• Reach out to others to ask questions before you embark on something new.
• Keep a record of things you learn.
• Have contingency plans and resource reserves, just don't place total trust in them.
  

Summary

The recent master of the concept of the unknown unknowns is Donald Rumsfeld. Unfortunately, people failed to follow the message and it was the brunt of jokes.

Here is how he stated it, transformed into poetic form:

The Unknown

As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.

—Feb. 12, 2002, Department of Defense news briefing

Here is a challenge for you. Check out the blog youarenotsosmart.com written by David McRainey. He has recently published his blog as a book titled, You are Not so Smart. It’s an interesting read. The premise as stated on the blog: "The central theme here is that you are unaware of how unaware you are. There is branch of psychology and an old and growing body of research with findings that suggest you have little idea why you act or think the way you do. Despite this, you create narratives to explain your own feelings, thoughts and behaviors, and these narratives become the story of your life."

I hope this article helps us all to realize the inherent risk of the unknown unknowns. I would like to hear your comments. How have you seen this risk?

---

1. Jerome R. Ravetz , “The Sin of Science - Ignorance of Ignorance”, Science Communication, September 2011

2. In fact, as I am writing this article, we experienced an F4 tornado and 4.7 earthquake within just a few hours of each other on November 7, 2011.