Software Defects of the Week

This has been one of crazy weeks, but I have to write about these two incidents. I'm one of those people who, like a bug collector, love to study these problems to learn from them for future prevention.

The first one happened right here in my home city of Oklahoma City. We have this awesome NBA team, The Thunder, that is in the playoffs.

When the tickets went on sale online, some people in Oklahoma City were not allowed to buy a ticket. They received an error message to the effect they didn't live in Oklahoma, Kansas or Arkansas. OK....Turns out, there is a business rule that said priority for tickets go to people who live in the area. Fine. But, these people live in OKC. What happened was that the front office used a file from the U.S. Postal Service to validate the zip codes from customers. I mean, who better to know which zip codes are valid, right? However, there were a few zip codes missing.

This brings up an interesting and common testing problem. How does one find such a defect? In this case, it's a conundrum.

You could say, use equivalence partitioning and test each zip code as a class. But, how do you identify all the zip codes? Especially, if the postal service info is incomplete to begin with! Plus, that would take a long time to do manually. Automation could do the job quickly, but you need a secondary source to compare the correct codes.

My solution would be to either 1) do an automated test using the known customer zip codes (like from a mailing list), or 2) do a manual test that samples from the customer list. This still would not assure the defect would have been found.

In the end, the team did the right thing and sold the affected people seats at a discounted price, once they learned of the problem. Now, perhaps, hopefully, this can be a test case for the future!

The second issue occurred in Auburn, California. Here's the account according to NPR:

"Traffic jams in California - well, they're nothing new. But one recent tie-up on Interstate 80 was noteworthy since it was caused by a computer glitch. The Placer County court accidentally summoned 1,200 people to jury duty on the same morning. Taking their duty seriously, residents tried to be on time at 8:00 a.m. and were in a line of traffic with other would-be jurors. The court apologized but said a real jury summons could be coming soon."

"We apologize profusely," court executive officer Geoff Brandt said of the error.

My take: I'm skeptical. Sounds like there could be some human error involved! I mean, if I input the wrong selection criteria in for the summons, I wouldn't want to take the blame either.

I doubt we will hear of the root cause, but it would be interesting to know.

Keep taking those backups!

Mac Test Automation Tool

I had a brain freeze today during my live online interview with James Lyndsay. D'oh! The cheap Mac test automation tool that slipped my mind at the time is called "Fake". It's $29 USD and found at www.fakeapp.com.

Risk Assessment Slides from Star East 2012 Tutorial

Thanks to everyone who attended my tutorial at StarEast 2012 on Becoming an Influential Test Team Leader. In that session, I briefly discussed my risk assessment method. Here are the slides, plus the flipchart notes for the challenges of testing (yes, all of them!) we listed. For those not there, you will see in the fourth picture, the "human" or "both" (technical and human), were the big winners.

























This is the chart for short-term, mid-term and long-term goals.

Software QA Ranks #3 in the Robert Half Technology Salary Guide

I normally don't get excited about being #3, and sharing the honor as well. But after years of being almost ignored on salary surveys, QA and Business Analysts hit it big in the Robert Half 2012 Technology Salary Guide. Thanks to Hubert Lee (The Dashboard Spy) for publicizing this!

According to the study, QA and Business Analysts are #3 in the "Top 5 Tech Professionals Businesses Want Now" category:

"With more dollars available to IT projects, managers are focusing on quality control and assembling more accurate project requirements. Quality assurance professionals can relieve developers so they can focus on coding, while business analysts can help build trust among stakeholders and serve as go-betweens for technology and business."

You can download the report here:

http://s3.amazonaws.com/DBM/M3/2011/Downloads/RHT_SalaryGuide_2012.pdf

Want to get into QA or Business Analysis? Check out these courses:

http://riceconsulting.com/home/index.php/Table/Basic-Training-Courses/

Or, if you want to get certified in testing, we have the training for you!

http://riceconsulting.com/home/index.php/Table/ISTQB-Training-for-Software-Tester-Certification/

Add value to your company by getting the knowledge to support better project results!

Random Thoughts After the Conference

I had the pleasure of ducking in and out of STPcon today. So, the conference is still going on, I just had to leave early. I've been doing the testing conference thing for 24 years, speaking for 23 of those years. So, I've been to the rodeos quite a bit.

So many ideas are swirling about, which is a good thing. The challenge is sorting them out to make sense in your own situation.

I was observing the panel discussion about agile testing and some things struck me:

1) What happened to XP... and then, Crystal and then...? What I mean is they apparently went out of style. In 2001 I was at a conference where XP was the solution to all problems. Now, it seems there is a new kid on the software block, Kanban. So, the discussion is around Scrum vs. Kanban.

This all makes me think there is "nothing new under the sun." People seem to want to follow the new thing, which I totally get, but why? Typically, it's because the present methods have problems. Why?

The reasons go to deeper issues, I think, than the flavor of the methods being used. I keep going back to my mantra that "the magic is not in the method." It is something deeper and more esoteric than a methodology.

I am convinced I was on an agile team in 1979.

2) People, tools and process...if you gravitate to just one or two of these, you will be out of balance.

OK, so you love the people and the team. That's great and the right people, all being able to get along is key to good projects. But...even the best teams need the right tools and the right methods to get the job done.

3) The customer is king, regardless what your process is. Your customer may be the end-user or the CEO. If they want something, you better get it to them (thanks, Scott Barber, for making this emphatic point today!). If you stick with your process in opposition to what your customer wants or needs, you will be replaced.

I once was teaching a class when on the afternoon of day 1, half of the class had to leave. Some of you cynics are thinking "I can totally see that" but no, it wasn't due to the teacher. It seems they had to restore functionality of a change back to the original version. They got the requirement totally right. They implemented the change 100% correctly. However, a powerful stakeholder got so much push-back on the change, he said "pull it." So, even though everyone had done their job correctly, there was still a problem that had to be dealt with. (By the way, this was all based on a new state law that was passed and was being implemented. The stakeholder was a state senator.)

4) Software development is a creative process and has to be managed as such. It's also a knowledge-based process and knowledge-based workers are not the same as other types of people. Otherwise, we could create software with software all the time, not just some of the time. I know, we have visual tools and the like, but remember CASE tools? There are some principles of manufacturing that can be applied well in software, but we must remember that software is intellectual in nature.

5) Release processes drive almost everything else. If you get this wrong, you can get really bound up. Some of us can recall the famous "I Love Lucy" episode where she and Ethel work in a chocolate factory wrapping the candy as it travels down the conveyor belt. The candy keeps coming faster and faster. They can't keep up and start eating it, stuffing it...funny stuff. I see this in software projects all the time. Too much work, too fast pace, people just try to get the stuff out of the door. Then when things don't work, the testers get blamed.

No matter which method you use, if you can't control scope and pace, you won't succeed. You will always be behind and your customer will always be complaining.

Well, those are some thoughts in the airport. I would like to hear your thoughts.

New Podcast - Making it Big in Software

Check out the newest podcast where I interview Sam Lightstone, author of the book, Making it Big in Software.

http://riceconsulting.com/home/index.php/Library/software-quality-podcasts.html

Webinar Recording - Everything You Wanted to Know About ISTQB Advanced Certifications

Thanks to everyone that attend the webinar on Friday.

Here is the recording:

http://www.anymeeting.com/riceconsulting/EA50DB838849

Here are the slides:

http://www.softwaretestingtrainingonline.com/cc/public_pdf/Everything%20You%20Wanted%20to%20Know%20About%20ISTQB%20Certification.pdf

There was a question about the need to read/understand code for the Advanced Technical Test Analyst certification. Here are two resources for those that would like to know about coding:






Here are my ISTQB Advanced course links:

http://riceconsulting.com/home/index.php/Events/istqb-advanced-level-public-courses.html

http://riceconsulting.com/home/index.php/ISTQB-Software-Tester-Certification/istqb-advanced-study-groups-forming.html

http://www.mysoftwaretesting.com/ISTQB_Advanced_Test_Analyst_Course_Public_Course_p/ctaltapub.htm

Syllabus:

http://www.astqb.org/documents/AdvancedSyllabus2007.pdf

Thanks again!

Randy

Webinar - Everything You Wanted to Know About ISTQB Advanced Certifications

Many people who hold the ISTQB Foundation Level certification are curious about what is involved in the ISTQB advanced level certifications. This session will walk you through the structure of the advanced certifications, what you need to know, the pre-requisites, ways to prepare and the value of holding an advanced certification. There will be plenty of time for questions along the way.

I hold all three ISTQB advanced certifications and will explain the things you need to know about ISTQB advanced certification.

Time: 1/20/2012 1:00 PM (UTC-06:00) Central Time (US & Canada) for 60 minutes

Register: http://www.anymeeting.com/PIID=EB54DE818249

I hope to see you there!

Randy

Book Review - Making it Big in Software

There is a great "secret" I tell software professionals. That "secret" is that if you want to rise to the top of your field, it's not that hard to do because so few people do the simple things to rise to the top. You will learn those things in this book.

Making it Big in Software brings a great perspective to the idea of breaking through to becoming an elite thought leader in the software profession. The first thing that caught my eye was the stellar nature of the people Sam Lightstone interviewed for this book. These include James Gosling, the inventor of Java, Steve Wozniak, inventor of the Apple computer, Grady Booch, co-founder of Rational Software, and many other luminaries in our field.

Lightstone anticipates the question most people have right out of the gate, "Why bother?"

"But with long hours, considerable stress, and no guarantees, the obvious question is whether it’s even worth trying to make it big. I believe the answer is unequivocally yes. The most compelling reason is that, in most cases, you have to show up to the office and work like a lunatic anyway—it’s really not optional (if you want to eat). So if the difference between being a midlevel career programmer and making it big is an incremental strategic investment of time and energy, then it’s more than worth it for you and for your family. In the long run, the benefits are significant: a more satisfying career, greater influence and impact within your company and the industry, more fun, and more money. And while there may not be less “crap” to do, at least it’s strategic work rather than “grunt” work."

Some of the enticing things about making it big are:

• Fun and interesting work
• Corporate and industrial influence
• The betterment of society
• Freedom to work on what you want, when you want (Lightstone makes clear that this is what you want to work on, not how much you have to work!)
  
• Fame
• Travel

(As an aside, I humbly say that as a minor software testing celebrity, I experience these things on a regular basis and it is good. While travel can become a chore, it is cool to teach in Rome twice a year.)

Software testers and QA professionals will appreciate Chapter 2, "What Good Software is Really About." This chapter is a concise case for better software and why we need to refine our craft, profession, or whatever you call what we do as software people.

Lightstone lays out all kinds of practical, real-world advice, such as "What to Look for in a Company." I like this list and it reinforces a key idea that you do not have to strike out on your own to make it big.

"1. Is this a company that has experience in building professional, high-quality systems?

2. Are there really talented people here I can learn from?

3. Is the position I’m being offered one that is interesting, with long-term growth potential on something I can believe in?

4. Do they have savvy business executives who really understand the business requirements for success and have a track record for delivering it?

5. Does the company have clarity of vision for the product it produces?

6. Is there an independent research arm?

7. How does the company innovate, and how profound has their innovation been?

8. Is the work environment pleasant and flexible, and does it suit my lifestyle?

9. Does the company seem stable? Do I believe it will still be around in ten years?

10. Is the pay in line with industry standards?"

The book goes back and forth between interviews and practical guidance, which is a good thing. The interviews let you get inside the heads and hearts of these gurus, while the guidance gives you a plan of attack.

Good economy or bad economy, it doesn't matter in terms of the importance of standing out and making your mark. Economies rise and fall. We all need to learn to not let our jobs distract us from our careers. This book helps light the way to do that. I highly recommend it to anyone in the software field!

Killer Weeds and Killer Defects

Recently I was listening to a radio interview about genetically modified crops. During the discussion, the guest mentioned the fact that farmers are starting to see “killer weeds.” These weeds are resilient to known herbicides which makes them very difficult to control or eradicate.

Being the tester I am, I immediately thought of the “Pesticide Paradox” that Boris Beizer wrote about in the 1980’s in his book, Software Testing Techniques. That principle says that just like bugs that become resilient over time to pesticides, software tests can become ineffective at finding new defects.

Another way to express this principle is to realize that your tests will grow weaker over time in terms of finding new defects.

My way of saying this is that when you repeat tests with the same conditions and test data, the tests become more confirmatory in nature than being discovery-oriented.

Similarly, there is the minefield example in which we can see that the safest way to cross a minefield is to walk in the footsteps of someone who has been successful previously in crossing from one side to the next. Seeing software defects as a mine, if we go down proven paths we don’t hit the mines.

Of course, in testing our goal is to hit the mines. As I say, “Failure is not an option, it is an objective.”

This discussion about killer weeds got me thinking about some other tie-ins with software testing and the “killer defects” we deal with.

Weeds are Ubiquitous

I don’t have a green thumb, I have a killer thumb. I’m sorry, but plants and I don’t get along that well. I have been successful in growing some things, like tomatoes, but my climate is harsh and the main things I can grow are grass and weeds.

However, I seem to have no problem with growing weeds. Have you ever noticed how resilient weeds are? They can grow in the cracks of a sidewalk!

Similarly, software defects are everywhere. Those of us in the weed control business of software (testers), have an abundant supply of defects to find. But like the killer weeds, it seems that no matter how often we spray, or pull, or mow, the weeds return.

Even when I pull the weeds out by the roots, they return because the seeds from neighboring yards (and my own yard) float into the yard or garden. Before long, there they are again.

What are the seeds of your defects and where do they originate? Inadequate requirements? Bad code? Inadequate testing?

The good news is that in software development we can actually control the seeds of defects at their source. It’s not always easy. In fact it hardly ever is easy, which leads me to the next point.

Weed Control is Maintenance

I really wish I could just spray or weed once a year, but that doesn’t work. In Oklahoma last summer, we had 63 days that were 100 degrees or more, plus we were in a very severe drought. Basically, most of the vegetation died, then caught on fire. I told people, “Welcome to Hell.”

Every blade of grass in the vacant lot next door to me was brown. There were inch-wide cracks in the clay soil that had the same consistency of bricks. There were also bright green weeds! Yes, the weeds thrived even in the absence of water in soil as hard as bricks.

Have you ever asked why, despite all your testing and other efforts, the defects seem to occur? The best answer I have is that people are human and humans make mistakes. These mistakes can be misinterpretation of needs, incorrect implementation of solutions, or just forgetfulness.

Another reason for mistakes may be the rate and nature of change. The faster we go, the easier it is to forget something or to simply become careless.

Perhaps the number one thing I hear testers complain about is maintaining their tests. This causes me to ask, “Where did this false expectation start that says tests are maintenance-free?”

The fact is that most tests will require maintenance, just like the software we are testing. However, the thing we deal with as testers that is unique is that changing one line of code may require changes hundreds of tests.

Some may say that this is why agile testing is great. I agree that agile approaches help. However, even in agile there are tests you want to remember and repeat. Yes, these may be automated. However, that may not matter in terms of maintenance since changes ripple through automated tests as well as manual ones.

Timing is Critical

Since I am horticulturally-challenged, I hire my friend Marty to spray and fertilize my yard five times a year. The first spraying in late winter is a pre-emergent. This treatment is very important to kill the weeds while they are still in the seed stage. Failure to catch weeds early means you play catch-up the rest of the year!

In software projects, early defect prevention and detection methods are key to stopping defects before they “bloom” and become larger and nastier. Activities such as reviews, walkthroughs and inspections can be your pre-emergent detection of defects.

Know Your Weeds

Marty doesn’t spray for all possible weeds in my yard. Instead, he sprays for those weeds common to our area. If rogue weeds appear, he sprays them individually as needed.

If you have ever ventured out to the hardware or garden store to buy weed killer, you know that there are many kinds, each oriented to specific types of weeds – grassy weeds, broadleaf weeds, and so on. There are treatments such as Roundup that will kill every bit of vegetation, but his is not a good option if you want to keep the good plants. That is overkill except when you want to clear out everything.

In testing, we need to know the types of defects we most often encounter. We also know that defects tend to cluster and we also know that people fall into patterns of repeating mistakes.

If we orient our tests to find the defects we most commonly encounter, that’s a good first pass. In the past, I have called this “starting a bug collection.” Now, I can also call it “a weed collection.”

Then, we can drill-down down as needed to branch out and find new types of defects.

Conclusion

Basically, we can pull weeds (a.k.a software defects), or we can work on preventing them. Pulling the weeds is hard work and seems to never end (sound familiar?). Preventing weeds takes discipline as well, but spraying in advance is easier and cheaper in the long run. Similarly, preventing defects with process improvement and finding defects early with reviews pays off in the long run.

If you know your source and types of defects, you can test wider and then focus on trouble spots. Just remember that the tests that work well today, may not be as productive in finding new defects the next time you test.

In the second installment of this article, I’ll explore why these killer defects are so hard to eradicate and the results of failing to find and eliminate them.

Video on How to De-Motivate Your Team

Hi Folks,

The featured video this week on Techwell is from the lighting keynote I did at StarEast 2011:



A little context to help understand my remarks: 1) Osama Bin Laden had just been killed and 2) there was a DECA conference at the hotel at the same time with many teenagers running around. They were well-behaved for the most part, but at the same time, we were happy to see them leave.

Enjoy!

Randy

Drilling for Gold

I was watching Gold Rush last Friday night on Discovery channel. The crew felt that they needed assurance that there was some gold in the ground below them. As one of the guys remarked, "Unfortunately, there is no arrow on the map that says, 'dig for gold here'".

In fact, one veteran gold miner said, "No drill, no dig."

So, they hired a guy to bring a drilling truck out to the site to drill all the way down to bedrock. Then, they removed the dirt from the drill bit, placed it into individual buckets (one per hole), and panned the dirt for gold.

By doing this, they confirmed there was gold in the ground where they were currently digging, and no gold at all in their "plan B" location.

Does this sound familiar? They were "testing" for gold by taking samples. The results of their tests were very valuable in guiding the next steps of their work. The risk is that it costs about $1,000 per day just in fuel costs to run the earth movers.

I think this is a really great analogy of what we do as testers, especially in black-box, top-down testing. The defects are like the gold. In fact, a software defect is more valuable than one ounce of gold when you consider the cost savings of post-implementation re-work! Then, when we learn from the defects and improve our practices, the defects have even greater value.

The main difference between gold mining and testing is that we're not always sure when something is a defect, but you can't miss gold.

My approach in top-down testing is to cover the critical workflow processes (scraping the surface). When defects are found, drill down to find others. This form of sampling makes use of the principle that defects tend to cluster.

Like gold mining, in testing, there are no indicators that say, "Look here for defects." We have to sample wisely to gain insight where to focus our efforts. There's gold (or defects) in them there hills!

Small Business Saturday and Cyber Monday Specials

We are excited to be a part of Small Business Saturday!

This is a special opportunity for you to support us as a small business and to get major savings on our e-learning training courses in software testing and software quality.

Every e-learning course will be discounted from Saturday, Nov. 26 through midnight on Monday, Nov. 28th. This even includes our ISTQB Foundation-level course, which includes the exam and additional text book!

There are no limits. Buy as many courses as you like!

Plus, if you pay with American Express, you can get an extra $25 back!

American Express is promoting Small Business Saturday again this year. Register your American Express when it opens November 1st, and then use your registered card to shop at a participating small business on Saturday, November 26th. You’ll receive a $25 statement credit when you spend $25 or more at participating small businesses. From this press release:

"For the second year in a row, American Express will help drive traffic through the doors of small businesses with a special incentive: a $25 statement credit offer for Cardmembers when they register their Card and spend $25 or more on Small Business Saturday at any qualifying small business that accepts the American Express Card. Cardmembers will be able to register their cards in early November."

To register your American Express card, visit http://www.facebook.com/SmallBusinessSaturday

Here's how to get these savings on our e-learning courses:

1. If the training is for your team, get your budget allocation and approvals from your management. This is a great way to make the best use of remaining 2011 training funds!

2. If the training is just for you, don't spend all your money on Friday!

3. Visit our e-commerce site at http://www.mysoftwaretesting.com and buy the courses you like. The discounted prices will be shown.

Remember that you have 12 months of access to the courses, even after you complete the course! To see other benefits and features:

http://riceconsulting.com/home/index.php/e-Learning/benefits-of-e-learning.html

http://riceconsulting.com/home/index.php/e-Learning/features.html

To see demos of the courses:

http://softwaretestingtrainingonline.com/moodle/course/category.php?id=11

Of course, I am always happy to answer any questions:

http://riceconsulting.com/home/index.php/component/option,com_chronocontact/Itemid,90/

I hope you take advantage of this rare opportunity!

Randy

Book Review - The Economics of Software Quality

Publisher: Addison-Wesley
ISBN-10: 0132582201 ISBN-13: 978-0132582209
Publication Date: August 3, 2011
587 pages, hardcover


In this book, authors Capers Jones and Olivier Bonsignour quantify the factors that influence software quality and provide information for people to gain insight into how their projects might compare to others. The measurements in this book are based on thousands of software projects.

One of my frequent complaints about the software industry is that we just don't measure very many things. However, thankfully there are people like Jones and Bonsignour that do have a rich source of metrics from enough projects that we can learn from them.

Capers Jones has long been considered the source for software quality metrics. To me, Capers is the "numbers guy" of our profession. With over 40 years in the field, Jones has a wealth of information he has maintained and published over many years.

Olivier Bonsignour is responsible for Research & Development and Product Management in a continual effort to build the world’s most advanced Application Intelligence technology. Prior to joining CAST, Mr. Bonsignour was the CIO for DGA, the advanced research division of the French Ministry of Defense.

For example, the authors state that "high quality levels are invariable associated with shorter-than-average development schedules and lower-than-average development costs." This finding is based on over 13,000 projects between 1973 and today.

The authors maintain that the real economic value of high quality software is not the cost to fix defects, but rather:

• the reduced likelihood of canceled projects
• the reduced risk of litigation
• shortened development schedules
• lower development costs
• reduced warranty costs
• increased customer satisfaction

This book addresses:

• What is software quality and how do we define its value?
• How can we estimate and measure software quality?
• How can software defects be prevented?
• How can we find and remove defects before testing?
• What are effective ways to test software and measure its effectiveness?
• What is the current state of post-delivery software defects?
• How do projects of various characteristics (low, average and high-quality) compare?
• How can technical debt be addressed from a business value perspective?

You will find a multitude of data from projects in a variety of industries, at various levels of quality, and at various levels of practice maturity. You will see by the numbers which project approaches work and which ones don't work very well.

By reading this book, you will gain insight not only into the current state of software quality, but you will also learn about measurement and metrics of software. These are critical things for any software quality professional to learn. In fact, after reading this book, you will know more about software measurement than 95% (that's my estimate) of testers and QA professionals.

I highly recommend this book, not only as a guide for software quality efforts, but also a benchmark for your own efforts.

The Greatest Risk of All

I perform quite a few risk assessments on projects and in organizations. There is one risk common in almost every situation although it is not always identified. This risk is also something I have experienced generally in life. In fact, it is a risk so familiar it’s easy to miss. People get burned by this risk all the time.

Hans Solo understood this risk when he admonished young Luke Skywalker, “Don’t get too cocky, kid.”

There are many ways this risk can be seen:

• People who tune out mentally in training because they think they already know the material. However, these people hardly ever ask any questions, either.
• Project planners who indicate all contingencies are considered.
• Everyday situations where we buy something, then learn the details about the product that are discovered only after trying to resolve a problem.

What is this risk that is so common and so elusive?

It is the “unknown unknowns.” There is nothing new about this. British Mathematician Alfred North Whitehead, who lived from 1861 to 1947, said “Not ignorance, but ignorance of ignorance is the death of knowledge”.

In an article published recently in the journal Science Communication, Jerome Ravetz writes:

“The idea of ignorance of ignorance is quite unfamiliar. Indeed, scientific culture generally suppresses awareness of ignorance. But ignorance of ignorance was quite well-known from Plato and Socrates onward; it became unpopular in the scientific revolution with Galileo and Descartes. Since then, the triumphalist faith that science would provide the good and the true has put ignorance to one side, and led scientists to the sin of pride in their scientific conquests.” 1

We know some things pretty well. For example, we know we need to test software. We also know we can’t find all the defects in software. We know where we have missed defects in the past.

The problem is, when we tackle something new, the rules change and no one is there to tell us what the rules are and how others play to the rules or cheat the rules. The risk comes into play when we ignore that fact or minimize it.

I find much of the programming on television to be lacking, but sometimes I do get hooked on a show. One “reality” show in particular, “Gold Rush” on the Discovery channel, has grabbed my attention. This show demonstrates the unknown unknowns very well.

Looking for Gold in All the Wrong Places

“Gold Rush Alaska ” is about a team of rookie gold miners who go to Alaska to find gold. The oldest member of the team tried many years ago to find gold, but that’s about all the experience that exists on the team. It seems that at almost every step in their gold finding effort, they discover what they didn’t know. The first hurdle came when they had to drive their main earth mover through a wide river because the bridge was too old and weak to handle the weight. Then came the time when they called in a gold mining expert who showed them how their dirt washer was letting much of the gold pass right out the exit and into the river. Once again, an “unknown unknown”. There were also dozens of other things these guys learned and are still learning. I admire their tenacity and faith. I hope they succeed before they run out of money and hope.

It is often said that “Beginners make dumb mistakes.” I prefer the way that Jerry Weinberg says it instead – “Beginners make beginner mistakes.” That’s a good thing for us software people to remember.

Phases of Realization

I have observed that people tend to go through phases when faced with a new situation:

1. Cluelessness (Blissful ignorance)

At this level, people are so unaware of their situation and risks, they “don’t know that they don’t know what they don’t know.” This is total cluelessness, but also very common. It’s like deciding to build a house without really understanding all the problems and stress. It’s very exciting at first to be thinking of your dream home, but then the decisions arise, delays are encountered, things may be done incorrectly and so forth. Ask anyone who has ever had a house built and they probably have a list of ten or more major things they would do differently. One of those things might be “not to have a house built.”

2. Curiosity

This is where you have enough objectivity to at least ask, “What are we missing?” You try to analyze and plan as much as possible, but it’s still only partial understanding of the possibilities. It’s much like going on a trip, thinking through all the clothes you will need, only to learn that the weather at your destination is much different than what you expected it to be.

3. Cautious awareness

In this phase, you are aware of the possibility of that “unknown unknowns” exist. You try your best to anticipate the unknown, but you hit the wall everyone hits. That is, you can’t prepare for everything. You might also think of this as “risk acceptance”.

4. Contingency planning

Realizing that you can’t prepare for everything totally, you put in place reserves to account for the unexpected. These are your Plan B, C and D for dealing with the unknown. In fact, some of the contingencies may seem totally outrageous.

In Oklahoma, where I live, most people have some type of tornado precautions and contingencies. Not so much for earthquakes until this week. Last Saturday evening, November 5, 2011, about 11 P.M. we had a 5.7 earthquake in central Oklahoma. In the past year, we have had a sharp increase in earthquakes. It’s odd to think that I have experienced more earthquakes in Oklahoma than I did while working nine months in San Francisco a couple of years ago! Sure, we’re small time compared to other places, but I’m thinking that more Oklahoma businesses are thinking about earthquake drills. 2

5. Confidence

Now that you understand the risk of the unknowns and have some back-up plans in place, you start to feel more confident in your ability to succeed. However, it is a false level of confidence in the contingencies.

Next to the early phases of cluelessness and curiosity, this may be the most dangerous phase because it easy for us to think we know more than we actually do know. We think, “Don’t worry, we have contingency plans in place.”

This reminds me of the time many years ago when one of my clients needed to restore the main database to recover from a power outage. Then they learned that the operator had failed to mount the second tape during the backup process. For some reason, the operator had misinterpreted the message that prompted for the next tape. This had been going on for months!

6. Comprehension

There is an old adage about judgment and experience. “To avoid mistakes, you need to have good judgment. To get good judgment, you have to make some mistakes.” Comforting, isn’t it?

There are two basic ways to learn: From your own mistakes and from other people’s mistakes. The good thing about learning from other people’s mistakes is that you avoid their pain. The bad thing is that we tend to observe these mistakes and think, “Oh, that not going to happen to us.”

A good thing that happens in this phase is that people may have the foresight to get an external opinion. This would have really helped the gold miners early on.

7. Calibration

Once you have asked the key questions, developed some contingencies, and reached out to others, you are ready to make adjustments. The question is, what will those adjustments look like?

Another bit of wisdom I learned from Jerry Weinberg is the “ Hudson Bay Start.” Back in the days of the Hudson Bay Trading Company, when the Canadian frontier was being settled, part of the plan was to travel west for one day. Then the next day, they would go back to the start and pick up the things they realized they missed the first time.

This is why I like pilot projects with low risk. You get a chance to learn what you don’t know. You can make mistakes without a lot of the downside risks.

It’s important to understand in this phase that the risk of the unknowns still exist. Going forward, you have to rely on your ability to adapt to new challenges and find a way to overcome them. This is where all of your experience and wisdom are needed.

8. Compilation and completion

It’s amazing how quickly we forget things. A good practice is to actually write down the things you learn and review them from time to time. My own practice for this is to keep a journal of the important lessons I experience. I may experience a lesson several times before I really “learn” it.

As someone has said, “All lessons will be repeated until learned.” I don’t know who said that, maybe I read it on a bumper sticker. But, I know it’s true.

Albert Einstein is often quoted, “Insanity: doing the same thing over and over again and expecting different results.”

My point here is that if we don’t learn from our mistakes and omissions, then we deserve the pain from them.

I’ll admit that achieving this kind of learning at an organizational level is difficult and rare. This is not about sponsoring training programs – it is much more experience-based than training. I think this level of learning must be personal. Organizations and institutions don’t learn – people do. It takes individuals to think, reflect and change their behavior.

What Can We Do?

This is the troubling thing about the unknown unknowns. You never totally mitigate this risk. However, there are some things we can do that may or may not help:

• Stay humble and recognize that unknown unknowns exists.
• Be flexible and adaptive when mistakes and problems occur.
• Learn from yourself and others.
• Reach out to others to ask questions before you embark on something new.
• Keep a record of things you learn.
• Have contingency plans and resource reserves, just don't place total trust in them.
  

Summary

The recent master of the concept of the unknown unknowns is Donald Rumsfeld. Unfortunately, people failed to follow the message and it was the brunt of jokes.

Here is how he stated it, transformed into poetic form:

The Unknown

As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.

—Feb. 12, 2002, Department of Defense news briefing

Here is a challenge for you. Check out the blog youarenotsosmart.com written by David McRainey. He has recently published his blog as a book titled, You are Not so Smart. It’s an interesting read. The premise as stated on the blog: "The central theme here is that you are unaware of how unaware you are. There is branch of psychology and an old and growing body of research with findings that suggest you have little idea why you act or think the way you do. Despite this, you create narratives to explain your own feelings, thoughts and behaviors, and these narratives become the story of your life."

I hope this article helps us all to realize the inherent risk of the unknown unknowns. I would like to hear your comments. How have you seen this risk?

---

1. Jerome R. Ravetz , “The Sin of Science - Ignorance of Ignorance”, Science Communication, September 2011

2. In fact, as I am writing this article, we experienced an F4 tornado and 4.7 earthquake within just a few hours of each other on November 7, 2011.

Self-Assessment Files from Becoming an Influential Test Team Leader at StarWest 2011

Hi Folks,

I promised the people in my Tuesday tutorial at StarWest 2011 the Word versions of the two self-assessments: One on the people issues in testing and the other on building core competencies in testing. Here they are!

People self-assessment
Core competency self-assessment

Feel free to modify for your own purposes. Thanks to everyone who attended my tutorial. It was a great time!

Randy

Giving the Customer a Little Extra

It's 108 today in Oklahoma City, so I find myself blogging, working indoors, and playing some guitar.

I had a revelation awhile back at Five Guys Burgers. I know, for many people, revelations happen in church, during a sunset, etc. Mine happen while I'm stuffing my face.

If you have ever eaten at Five Guys, you know the throw an extra handful of fries in your bag on top of everything. There's a good chance you may get full on the fries before you ever reach the actual container of fries. This got me thinking about why they did this. Then it struck me, this is a Purple Cow. How many other fast food places do that?

In contrast, I got a burger and large fries from one of our local chains (Braum's) recently. When I looked at the fries, I could count them. There were 15 I think. Plus, they were those frozen ones, unlike Five Guys which are cut in the store (like In-n-Out).










It probably costs five cents or less to throw in the extra handful of fries. In return, Five Guys gets loyal and raving customers. I've never seen an ad - print, TV, web, anything, but I've talked to many people who brag on them.

So, what's the attraction? What makes the difference?

Isn't it nice when you get more than you expect, not less?

This is our new Five Guys in Moore. Someone apparently thought it was a drive-in.










It's sad that in our society, people are cynics about deals. I see this all the time in dealing with our customers at Rice Consulting. I quote a price and people ask, "What's the catch?". It's simple. There are no catches.

For example, I give 12 months for people to finish an e-learning course. Most of our competitors give 90 days. If someone needs more time, I always extend the time to take a course.

After a class, I'm always available to answer post-class questions for free. During an e-learning class, I always answer questions as part of the class fee. After a consulting engagement, I always answer post-engagement questions.

In the past weeks, here are some other great examples:

We bought a 2010 Ford Explorer recently from Reynolds Ford in Norman, OK. I didn't haggle on the price, bought the extra protection package, got Ford financing, the works. The whole process took a little over an hour and went as smooth as silk. Our salesmen, Brent was helpful, friendly and knowledgeable. Then, I discovered there was only one set of keys. So, I called Jayna at the dealership for an extra set since they cost about $200 these days for the chipped keys. When I picked up my car from the service area, they had an extra set of keys for me. Oh, and when we took possession of the car, it had a full tank of gas, was washed and the tires were glossed. That's extra for the customer and I'm a fan now. Reynolds Ford knows the value of a lifetime customer is more than $200.

Today, I bought two pounds of coffee at a Starbucks in Norman. The barista offered me a free cup of coffee. Even at 108 degrees out, I accepted. Let it never be said that I ever turned down a free cup of coffee. As James Garner (a Norman native) said in Support Your Local Sheriff, "I never turned down a cup of coffee in my life."

I have actually asked for a free cup of coffee when buying several bags of coffee at other Starbucks and the response was "Sorry, we don't do that anymore." That left me unimpressed. I'm much more inclined now to buy from the Norman store.

There used to be a time when giving a little extra to the customer was common practice. Now it seems that the trend is to reduce the product and charge the same. I believe that's a short-sighted approach.

It appears to me that giving added value is the result of both good policy (that's the role of management) and good performance (that's the role of the person delivering the service.)

At Five Guys, their corporate policy is to top off the fries. At Braum's, their policy is to not under any circumstances exceed the size of the container.

In other cases, like at Reynold's Ford and Starbucks, it's the performance of the right people making the right decision by the customer.

In software testing, we need to be showing our added value. One way to do that is to give a little extra to your customers. Be helpful to them. Ask what they need. If they don't know what they need, help them see the possibilities.

Some people see their customers as "those which must be endured." For sure, there are challenges. I choose to see my customers as the reason I'm in business - to serve and to stay in business.

You may not have a "business" but I suggest that you have many of the same reasons to please your customers. Without them, you wouldn't have a job.

OK, enough for now. I'll go practice Black Dog (Led Zeppelin) and Bach Cello Prelude 1. What can I say? I have a wide range of musical tastes!

Have a great weekend!

Book Review - Documenting Software Architecture, 2nd. ED

Authors: Clements, Bachmann, Bass, Garlan, Ivers, Little, Merson, Nord, Stafford

Addison-Wesley, 537 pages

As a software tester, I rely heavily on system documentation. Unfortunately, documentation is often missing, obsolete, or never created in the first place.

Also, I have a great appreciation for software architects and the work they produce. As a former developer, I used to struggle with the best ways to express system architecture diagrams. After all, there are so many methods available to document systems – UML being a major one, but there are others.

When I started reading this book, I was struck by its practicality, beautiful simplicity, and integration between authors. Everything I read in this book is written in a clear and understandable way. The authors understand that different audiences will read this book, so they give graphical (of course) guidance in which chapters are most applicable to architects, stakeholders and novices.

This book covers the basics, such as module views and module styles, component and connector views, allocation views and styles.

Part two of the book goes beyond the basics and gets into issues regarding levels of detail, deciding among alternatives, documenting interfaces and documenting behavior. Part three is devoted to building the architecture documentation.

There are appendices devoted to UML, SysML and AADL to show how architectural documentation is shown in each of these.

It would be tempting to say that this book is needed for new technologies, such as SOA and the cloud, which is true, but too narrow. Actually, this book can be applied to any technology or approach – traditional, agile, iterative, or anything. That’s because the one thing people ask for and struggle with is documentation. This is especially true for architectural documentation.

Read this book and apply the things in it and you will stand out on projects - in a good way. And, of course, that’s a good thing!

Test Automation is Not Automatic

Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations.

Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, management many times does not know or accept that reality.

There are some test tools, such as unit test tools, that are practically automatically applied. My remarks in this article are aimed at the capture/playback and scripting tools for test automation.

The issues are that:

1) Not every test can or should be automated

2) For those tests that can be automated, it takes time and effort to build the automation

3) For those tests that have been automated, the tests must be maintained

4) It takes time to lean how to use a tool

5) It takes effort and planning to implement a test automation framework

None of these are automatic, even with the best of tools.

Not Every Test Can or Should be Automated

Think about the things you test that are not very repeatable. Or, they may be prone to constant change. Perhaps the things you test are developed in a technology that has little or no tool support.

Then, there are tests such as user acceptance tests that need people’s evaluation to judge acceptance.

Some tests require creativity and adaptation to perform. You may have to make judgments during the test, which may be too complex to describe in a script. Test automation leverages the mundane testing to give more time and attention to the unique tests.

Your job is to identify the tests that can be automated. (They don’t come labeled!) Then, you must understand the nature of the test, such as the pre-requisites, the steps to perform it, the exceptions and where to find the expected results. None of this is automatic. It’s all test design and test implementation.

For Those Tests That Can Be Automated, It Takes Time And Effort To Build The Automation

It’s one thing to automate a function, but another to design a good test of the function. That’s why capture/playback is so appealing, yet lacking. The issues are: What are you testing in the capture session? Then, how can you extend those tests to add value?

You have to apply approaches like data-driven testing and keyword-driven testing, which take time and effort to understand and implement. These are not “out of the box” deliverables.

For Those Tests That Have Been Automated, The Tests Must Be Maintained

This has been one of the consistent issues in test automation. There are things you can do to ease the maintenance burden, but it doesn’t do away with the issue.

For example, modular and reusable test scripts are very helpful in reducing the number of tests that must be maintained. Still, you must maintain the scripts you have.

This means you must know when the application has changed, what the changes were, and create new tests for those changes. This implies the presence of configuration management and traceability.

It Takes Time To Lean How To Use A Tool

Of course, the learning curve varies by tool, but the fact remains that a tool with sophisticated features will take some time to learn. The learning curve also varies by person. Some people with deep experience in automation may be able to learn very fast, but when you consider the wider deployment, most people will fall on the lower end of the experience scale.

The time also varies by the approach used to get training. Some people try self-learning which is noble, but typically takes longer than classroom training. Mentoring from an experienced person may be the best approach.

You must assess your organization’s skills and abilities to know even if mentoring will help. The best mentor in the world can’t help the person who isn’t ready to learn. Is that another “Randyism?”

It Takes Effort, Money And Planning To Implement A Test Automation Framework

The framework can take a variety of forms, but the context here is the organizational framework which provides a way to efficiently build and control test automation. Tools are only one-third of the picture. You also need processes with trained and motivated people to make everything work together.

Out of the box, tools are just software. A process framework helps with reuse and the maintenance of test automation.

Frameworks aren’t automatic, either. They must be adapted to fit each situation, therefore they require time, effort and funding to design and implement.

Conclusion

This article is certainly not an in-depth treatment of this topic. Entire books and training courses have been written to address these and other aspects of test automation.

I hope this expands on my simple statement “Test automation is not automatic” to provoke your own thinking on this reality.

For over twenty years now, I have seen a wide variety of test tool companies promote their tools as being effort-free, script-less, or however they choose to position their products. The evidence shows these are often empty claims. Just compare the numbers of people who have been successful in using test automation tools to those who have given up using the tools. It’s about 25% successful to 75% unsuccessful in my research.

I hope this article is an encouragement to those who have tried to implement test automation as well as to those who haven’t. It’s important to go into these projects with your eyes open and expectations at a realistic level. Once you get past the idea that the tools do all the work, you can do the planning and other work needed to increase your chances of success.

Test Manager Workshop with Randall Rice and William Perry

Name your software testing challenge and other test mangers and the workshop faculty will give you solutions!

As a software test manager, you face pressure from all directions. Customers expect high-quality products, project managers want to get the projects delivered on time, your management expects you to do more with less, and your team needs your leadership.

You could take a class in software test leadership or management, but much of the knowledge comes from the front of the room. What if you could determine the topics based on your own needs?

This is not a class or a conference, but a highly interactive and personal event. You will spend time personally with William Perry and Randall Rice to discuss your specific testing needs.

You need a mentor! Personal mentoring is one of the most effective ways to get guidance and build knowledge. Mentoring fills the gap that training cannot fill by giving you face-to-face, personal feedback about how to deal with tough issues.

In this unique opportunity, you can spend two days with two respected test management consultants, William E. Perry and Randall W. Rice to gain insight and solve your specific testing problems. Perry and Rice literally wrote the book on people issues in software testing. William E. Perry has written over 30 books on software testing and quality assurance. They share over 80 years of professional software quality and testing experience in world-class organizations.

Any and all topics are on the table. As an example, some of the topics suggested by test managers are:

• People issues in software development and testing
• Testing large and complex legacy systems
• Testing SOA and cloud-based applications
• Test metrics and measurements for showing your value
• How to grow as a test team leader
• How to communicate your value and your team’s value
• How to grow your team
• Test tools and test automation
• Software QA processes
• Software forensics
• Testing internal controls
• Testing information security
• Applying statistical methods to software quality
• Gathering and defining testable user requirements
• Managing expectations
• The Ins and Outs of software test certifications
• Any topic of interest to you!

However, the final topics will be determined by the attendees during the conference. That ensures your topic will be discussed!

You will leave this event with confidence and knowledge in how to best approach your situations.

Location – Orlando, FL (exact venue to be announced later)
Dates – Thursday, October 27 and Friday, October 28, 2011
Cost - $1,295 per person

Seats are limited, so reserve yours early to avoid disappointment.

Bonus Option!

Attend one day earlier to have an extra opportunity with Randy and Bill to learn how to test dirty systems.

The day prior to the workshop, William Perry and Randall Rice will be conducting a one-day interactive workshop based on their new book, Testing Dirty Systems. In this workshop you will learn:

• How to apply statistical process control (SPC) to software development and maintenance
• How to acquire system knowledge
• How to plan and perform the testing of large, complex and undocumented systems
• How to measure and report the test results in value-added ways
• How to use test results to clean a dirty system

Location – Orlando, FL (exact venue to be announced later)
Date: Wednesday, October 26, 2011
Cost - $550 per person if attended as part of the two-day Software Test Managers’ Workshop, $690 if attending only the one-day event.

Click here to register.

Click here to ask any questions.

Credibility

For those of you who have been in my tutorial Becoming an Influential Test Team Leader, you know I'm big on the topic of credibility. If people don't find you credible, they don't find your information credible either. So, credibility is huge for testers and test leaders.

Today, John Maxwell's "Leadership Word of the Day" is credibility. He has a nice one-minute video and you can also sign up to get these free at:

http://johnmaxwellteam.com/credibility/

If you want to view my 45 minute conference presentation on credibility, you can find it here:

http://softwaretestingtrainingonline.com/moodle/course/view.php?id=30

(When asked for a login, you can click the button that reads "login as a guest".)

I hope you listen to one or both of these and think about what it means to be credible as a person, a leader and a tester.

Have a great weekend!

Randy

Ten Proven Ways to De-Motivate Your Team

Thanks everyone for your kind words about my lightning keynote today at StarEast.



Here is the complete script:

10. Set unreasonable “stretch” goals just to see how hard people will work.

It really doesn’t matter what the goals are, or what the deadlines are, just make them really hard to achieve. If you really want to wear people down, do this at least once every 3 months. Overlapping stretch goals are especially fun.

9. Never explain your rationale for decisions.

Reasons? You don’t need no stinking reasons! “Because I said so” works just fine. In fact, it’s good mental exercise for your team to try to figure out your irrational actions.

8. Assign meaningless tasks.

The most important thing about work is that people look busy at all times. Whether it’s writing a PowerPoint presentation for you to impress your boss, or just to test until 6 in the evening, make sure everyone always looks busy.

7. No matter how good something is, criticize it.

Especially the first time you review it. Get a nice, big, red marker and go crazy. Forget about the main point of the content and focus on sentence structure or specific words you don’t like (such as “the” or “that”). Before long, people will give up and stop trying to make something right the first time.

6. Take all the credit for yourself.

After all, this team is your creation, right? This is especially important to remember at bonus time. Otherwise, you may have to suffer financially. (Don’t forget to attend all senior management meetings alone.)

5. Solve problems by building a new bureaucracy.

Remember, there are no simple solutions, only simple people. You are a much more complex person than that. You can design forms, approval processes and even spreadsheets. Of course, once you implement this required bureaucracy, you need to police it some way, so you will need a special team of spies to make sure everyone follows the “new way of doing things.”

4. Listen…like a brick wall.

The trick is to make people think you are listening. So, look them right in the eye, nod approvingly, but let your mind roam. Then, you can fulfill all your ADD fantasies. NBA scores, weekend plans, lunch plans….you name it.

3. Refuse to consider ways to do the job more effectively.

Tools? We don’t need no stinking tools, either! Besides, we can’t afford all those fancy tools. Free tools? We can’t use those – we do have rules to follow, you know. Training? We had a class 5 years ago. Can’t you people remember anything? Learn at lunch? Heck no, your team is too busy working through lunch. (See point #8 – always look busy.)

2. Treat your team like they are machines that should never break down.

I mean really. Why do these people call in sick and let you down when you least expect it. Who’s going to write your status report? And don’t even get me started about bathroom breaks!!

1. Never, ever, in any circumstance, give anyone praise or recognition.

Otherwise, people would start to feel hope and happiness, like when we saw all the teenagers leaving for home today. Remember, if you never praise anyone, you don’t have to insult them, just ignore them. Eventually they will leave and you can hire someone else to de-motivate.