Recently I was listening to a radio interview about genetically modified crops. During the discussion, the guest mentioned the fact that farmers are starting to see “killer weeds.” These weeds are resilient to known herbicides which makes them very difficult to control or eradicate.
Being the tester I am, I immediately thought of the “Pesticide Paradox” that Boris Beizer wrote about in the 1980’s in his book, Software Testing Techniques. That principle says that just like bugs that become resilient over time to pesticides, software tests can become ineffective at finding new defects.
Another way to express this principle is to realize that your tests will grow weaker over time in terms of finding new defects.
My way of saying this is that when you repeat tests with the same conditions and test data, the tests become more confirmatory in nature than being discovery-oriented.
Similarly, there is the minefield example in which we can see that the safest way to cross a minefield is to walk in the footsteps of someone who has been successful previously in crossing from one side to the next. Seeing software defects as a mine, if we go down proven paths we don’t hit the mines.
Of course, in testing our goal is to hit the mines. As I say, “Failure is not an option, it is an objective.”
This discussion about killer weeds got me thinking about some other tie-ins with software testing and the “killer defects” we deal with.
Weeds are Ubiquitous
I don’t have a green thumb, I have a killer thumb. I’m sorry, but plants and I don’t get along that well. I have been successful in growing some things, like tomatoes, but my climate is harsh and the main things I can grow are grass and weeds.
However, I seem to have no problem with growing weeds. Have you ever noticed how resilient weeds are? They can grow in the cracks of a sidewalk!
Similarly, software defects are everywhere. Those of us in the weed control business of software (testers), have an abundant supply of defects to find. But like the killer weeds, it seems that no matter how often we spray, or pull, or mow, the weeds return.
Even when I pull the weeds out by the roots, they return because the seeds from neighboring yards (and my own yard) float into the yard or garden. Before long, there they are again.
What are the seeds of your defects and where do they originate? Inadequate requirements? Bad code? Inadequate testing?
The good news is that in software development we can actually control the seeds of defects at their source. It’s not always easy. In fact it hardly ever is easy, which leads me to the next point.
Weed Control is Maintenance
I really wish I could just spray or weed once a year, but that doesn’t work. In Oklahoma last summer, we had 63 days that were 100 degrees or more, plus we were in a very severe drought. Basically, most of the vegetation died, then caught on fire. I told people, “Welcome to Hell.”
Every blade of grass in the vacant lot next door to me was brown. There were inch-wide cracks in the clay soil that had the same consistency of bricks. There were also bright green weeds! Yes, the weeds thrived even in the absence of water in soil as hard as bricks.
Have you ever asked why, despite all your testing and other efforts, the defects seem to occur? The best answer I have is that people are human and humans make mistakes. These mistakes can be misinterpretation of needs, incorrect implementation of solutions, or just forgetfulness.
Another reason for mistakes may be the rate and nature of change. The faster we go, the easier it is to forget something or to simply become careless.
Perhaps the number one thing I hear testers complain about is maintaining their tests. This causes me to ask, “Where did this false expectation start that says tests are maintenance-free?”
The fact is that most tests will require maintenance, just like the software we are testing. However, the thing we deal with as testers that is unique is that changing one line of code may require changes hundreds of tests.
Some may say that this is why agile testing is great. I agree that agile approaches help. However, even in agile there are tests you want to remember and repeat. Yes, these may be automated. However, that may not matter in terms of maintenance since changes ripple through automated tests as well as manual ones.
Timing is Critical
Since I am horticulturally-challenged, I hire my friend Marty to spray and fertilize my yard five times a year. The first spraying in late winter is a pre-emergent. This treatment is very important to kill the weeds while they are still in the seed stage. Failure to catch weeds early means you play catch-up the rest of the year!
In software projects, early defect prevention and detection methods are key to stopping defects before they “bloom” and become larger and nastier. Activities such as reviews, walkthroughs and inspections can be your pre-emergent detection of defects.
Know Your Weeds
Marty doesn’t spray for all possible weeds in my yard. Instead, he sprays for those weeds common to our area. If rogue weeds appear, he sprays them individually as needed.
If you have ever ventured out to the hardware or garden store to buy weed killer, you know that there are many kinds, each oriented to specific types of weeds – grassy weeds, broadleaf weeds, and so on. There are treatments such as Roundup that will kill every bit of vegetation, but his is not a good option if you want to keep the good plants. That is overkill except when you want to clear out everything.
In testing, we need to know the types of defects we most often encounter. We also know that defects tend to cluster and we also know that people fall into patterns of repeating mistakes.
If we orient our tests to find the defects we most commonly encounter, that’s a good first pass. In the past, I have called this “starting a bug collection.” Now, I can also call it “a weed collection.”
Then, we can drill-down down as needed to branch out and find new types of defects.
Conclusion
Basically, we can pull weeds (a.k.a software defects), or we can work on preventing them. Pulling the weeds is hard work and seems to never end (sound familiar?). Preventing weeds takes discipline as well, but spraying in advance is easier and cheaper in the long run. Similarly, preventing defects with process improvement and finding defects early with reviews pays off in the long run.
If you know your source and types of defects, you can test wider and then focus on trouble spots. Just remember that the tests that work well today, may not be as productive in finding new defects the next time you test.
In the second installment of this article, I’ll explore why these killer defects are so hard to eradicate and the results of failing to find and eliminate them.
