StarEast 2010

This has been a great StarEast conference! Thanks to everyone that attended my sessions - the full-day tutorial on Monday and my two track sessions on Wednesday. In the track sessions I offered some resources:

A Deeper Dive into Dashboards

PowerPoint slides of the presentation (PDF)

Excel dashboard with macros
Excel dashboard without macros

The Xcelsius dashboard example
The full 16 minute video of how I created the dashboard

(The reason I give both versions is because the macro-enabled one will update if you are connected to the Internet.)

Here are a couple of the websites I mentioned:

www.dashboardspy.com
www.datapigtechnologies.com

The Elusive Tester to Developer Ratio

We were discussing how to leverage high ratio situations, like 1 tester to 10 developers and higher. The question was asked about how to get testers more on-board with doing unit testing. One suggestion I had was to provide with with a checklist. Here is a unit test workbench with two checklists. This is in Word format, so feel free to modify it to meet your needs.

By the way, I plan to have narrated versions of both presentations posted soon on my website.

Stay in Touch With Me!

There are several ways to stay in touch:

1) Sign up for my newsletter
2) Follow me on Twitter - @rricetester
3) Become a Facebook fan

I hold a monthly drawing for free books for my newsletter subscribers and Facebook fans. The next one is on May 1.

Thanks and have safe travels home!

Oklahoma City Test Automation Training Workshop - May 13 and 14, 2010

We still have seats left for the new hands-on test automation workshop I am presenting May 13 and 14 in downtown Oklahoma City. If you live nearby, say in Dallas, Little Rock, Wichita, Lawton, Tulsa, Amarillo, heck, even Kansas City or Topeka, it is worth the drive and two night's hotel stay. At $400 per person, this is a crazy deal. Especially since this kind of training doesn't happen often in our part of the country. However, this is the first presentation and I need feedback before presenting the workshop in Rome in June.

This workshop is sponsored by the Red Earth QA SIG and hosted by Devon Energy. Thanks greatly to those organizations.

This workshop is not oriented to any specific tool, but the lessons and concepts are transferable to any tool. We will be using some free and inexpensive tools for the exercises, so bring your own computer.

Nervous about scripting? That's OK. You will be able to work at a pace that suits you and I along with others in the workshop will be there to help.

To learn more and to register, just visit http://www.mysoftwaretesting.com/product_p/okcauto.htm

If you would like for me to come to your company and present the workshop, just call me at 405-691-8075.

Thanks!

Randy

My Take on the McAfee Mess

OK, first let me say I wasn't there, so like Will Rogers is quoted as saying, "All I know is what I read in the papers." In my case, the paper is Computerworld.

On April 21, McAfee released an update to their anti-virus application that disabled systems running Microsoft XP SP3. This caused a bad day at McAfee, but even worse days at all the companies and homes that were impacted.

You can read the early account here:

http://www.computerworld.com/s/article/9175928/The_McAfee_update_mess_explained?source=toc

Then, today, a more detailed explanation and apology was issued:

http://www.computerworld.com/s/article/9175940/McAfee_apologizes_for_crippling_PCs_with_bad_update?source=CTWNLE_nlt_dailyam_2010-04-23

The bottom line was a critical defect was missed and made its way to customers. Here are my observations as an interested bystander and software testing consultant:

1) The apology was cryptic for a technical audience. "We recently made a change to our QA [quality assurance] environment that resulted in a faulty DAT making its way out of our test environment and onto customer systems." However, no explanation of the change was given. Was a platform removed, or skipped? Was a test case skipped? Was there a rush to get the update out? Why was the environment changed? The blame seems to be on the change to the environment.

2) The risk was very high. This is the tester's worst nightmare and an example of what you don't want your company to go through, or your customers. This is a credibility-basher. I work with some software companies that say "We don't care about the risk. If there are problems, we'll just post a hotfix on the web site." Right.....

3) Testing can't find all the defects. However, testing is an easy role to place blame. At least they didn't blame the testers - they blamed the process and the environment, which is probably the appropriate place to focus.

4) This points out a big risk for COTS applications - applying an update without testing it. I know the updating process is automated for large companies. However, one of the things I teach in my COTS testing class is to test the updates before rolling them out to the entire company. I suspect this will be one of those lessons learned for many people. The really troubling thing is for individuals who get impacted. They have no "test" PCs.

5) The customers want to hear from the CEO over this. So, your CEO doesn't seem to care about testing? This is a good case study to show why they should care.

6) Your test is only as good as your environment. You may have great tools, great testers and great processes, but if you have gaps in your environment, you don't know for sure what you are testing.

7) This is one of those head-bangers. Apparently, this was not one of those deeply-embedded defects, but one that could have been found just in a simple update to a commonly-used platform. This is one of those defects that leaves management and customers asking, "Why didn't you guys test that? (I refer you back to observation #1) They aren't saying for sure.

8) There will be more defects escape in the future. The only question is, what will the impact be? If you really want a scare, take this scenario out to medical devices, aircraft, automobiles, utilities and other safety-critical applications. No matter how hard we try, there will still be defects because we can't test everything. That's not an easy reality to embrace because many people have grown to trust that software just works - mostly. Testers know better

OK, enough of being the armchair quarterback. This is serious and frustrating, but not the end of the world. A few weeks from now, it will all be forgotten. Actually, that's part of the problem. We experience the pain, the pain goes away, then we experience it again...and again.

Your thoughts?

Striving for Quality has Real Payoffs

Here is an interesting article about the tie-in between IT, quality and business growth:

http://www.computerworld.com/s/article/9175836/Striving_for_quality_has_real_payoffs?taxonomyId=14&pageNumber=1

The common misconception is that the 20th century quality gurus, like Deming, Juran and Crosby were out there teaching about quality for the sake of virtue. The reality is that they taught quality concepts as a way to build the bottom line in a company by gaining customers and serving existing customers better. That's why I get really annoyed at people who dismiss quality improvement as an optional thing. I agree that "quality" has been so overused is fails to resonate anymore with people, but so does "safety" until you get hurt or killed. The problem is not the concept but our thinking (or lack of attention).

Read this article and learn how to make the message to management. The great thing about this article was that management made the mandate to IT. And that is exactly the direction the quality mandate should flow.

Have a quality day!

Randy

Technology is Just the Tool

"The more elaborate our means of communication, the less we communicate."
— Joseph Priestley: Was an 18th-century theologian and educator

Isn't it interesting that this was written in the 18th century? It's like the idea that instead of technology making our lives easier, it has done just the opposite. Technology is just the tool. You can use any tool relentlessly and still wear yourself out. You can use any tool in an ineffective way.

One of my favorite lines from a movie is when Pappy O'Daniel says in "Oh Brother, Where Art Thou?" as he and his entourage are walking into the little radio station, "We ain't one-at-a-timin' here. We're MASS communicating!"

Sure, technology has changed the means of communication. What it has not changed is how we really connect with someone.

10 Reasons Why You Aren't Done Yet

Here is a great post by Michael Hyatt on personal productivity and the work habits (and sins) that beset us. I found this article helpful and if you are like me, with too many "not dones" on my "to-dos", you will also find it helpful.

http://michaelhyatt.com/2010/03/10-reasons-why-you-aren%E2%80%99t-done-yet.html

It's About Communication

I've been teaching for a long time now that I believe communication is the basis for everything we do in IT. However, communication is the one thing we ignore the most - both in practice and skill building. I dislike the term "soft skills" because "soft" implies easy or optional. The reality is that unless you communicate well with everyone in the organization, you lack the knowledge to solve problems and keep a project on track.

I ran across an interesting article this morning at computerworld.com about how one CIO does this :

CIO says communication is key to buy-in

No matter which role you are in, communication is a big part of your job, whether you practice it or not.

Here are some quick tips:

1) Be intentional

You have to keep communication in mind and be thinking all the time, "Who needs to know this?" You also have to be thinking "What do I need to know?" and "Whom should I be speaking with today?"

2) Do not confuse e-mail and meetings with communication

These can be vehicles for communication, but they have flaws. E-mail fails to convey the tone of our communication and meetings can drift into discussions of many topics. Just pick up the phone or walk down the hall if you really want to touch base with someone.

3) Practice

Yes, practice how you say things. Learn the words that carry meaning and power. Learn how to control your body language and read the body language of others. Do you have a big presentation in your future? Do you have an important conversation with someone soon? Practice!

4) Read before sending

So you've written an e-mail, status report or some other piece of written communication. Take an extra 60 seconds and read back through it. Place yourself in the role of the people who will receive it. How would be feel if you were the recipient? Are there any obvious gaps, typos, or other mistakes? Testers get nailed on this because we are highlighting others' mistakes. People aren't perfect, but try to find mistakes the best you can.

A really good book on this subject is Naomi Karten's "Communication Gaps and How to Close Them".



I hope I have communicated my thoughts on this well! Your thoughts?

Have a great day.

Randy

Being a Linchpin in a Process-Oriented Environment

I had lunch with my friend Frank today and I asked him his perspective on the Linchpin idea of being an artist and how that fits or conflicts with the ideas taught by Dr. Deming. Frank and I both like Dr. Deming's ideas a lot, so I am trying to set a context that makes sense.

In Linchpin, Seth Godin writes about becoming indispensable at our jobs by becoming more unique. We become more unique by letting the artist in us shine forth in what we do. Godin admits there are times you would not want this, like for airline pilots, etc. However, that specific example caused me to think about Captain Sullenberger who skillfully guided the U.S. Airways jet to a safe landing in the Hudson River. That was artistry. My friend Frank pointed out this was not because of his training from the airline, but because he knew how to fly gliders. Interesting.

In a factory setting, you want people making the same things in the same way. Dr. Deming taught people to remove variations in the process. He taught that processes should be so well defined that anyone with the right training could perform them with identical results. We can apply that in some ways to software development. So, how does one let their artistry shine in that kind of process-oriented environment?

Frank observed that Quality Circles was a technique Dr. Deming taught to let the workers in a process have a voice in improving it. Those people who are good at seeing problems and suggesting changes to the process that save the company money are linchpins and artists. You would not want people to start changing the process whenever or however they want, but you do want to know how to improve the process at the right time, in the right way.

Certainly Dr. Deming valued the person and how they contribute to quality. However, he was not a proponent of surviving by heroic efforts. The problem of heroic efforts is that is rewards those who do sloppy work and then swoop into to fix it, while those who consistently do good work get little recognition.

Unfortunately, too many good people suggest good ideas to management and those ideas are ignored. These people often leave for companies that do value their ideas. They can do that and get more money in the process because they add value. They are linchpins. This applies to leaders as well. Leaders who listen to ideas will rise and succeed, while those who don't will continue to struggle.

Your thoughts?

Randy

Book Review - Linchpin by Seth Godin

For a long time I have been telling software testers there is someone out there, either in their city or halfway around the world, that can test faster and cheaper than they can. Like it or not, software testing has become a commodity for many organizations. It doesn’t matter who is doing the testing, how they are doing it, only that testing is being done at a cost they are willing to pay.

These are the same companies who call me up to get “two days of training on testing” then have no idea what their needs are, what will happened after the class, or even who will be the instructor. They just want “a pound of training” as I call it. This type of attitude means you need a point of distinction. That’s what this book is all about – how to become indispensable by letting the artist in you emerge.

Think of the companies that lay off hundreds or thousands of people all at once. How were those people chosen? They didn’t lay off everyone, so who got to stay and why? It wasn’t the people who obediently played by the rules and didn’t make waves. The ones who stayed were difference makers – linchpins.

This book takes you on a journey. It starts with some background of how we got here. After all, for all of human history except the last 60 years or so, people lived without the idea of being taken care of by a company. Now, we find ourselves in a different ballgame with a whole new set of rules. Having a good job is no longer the definition of success. Lots of “good jobs” have gone away, never to return.

The journey continues to discuss how to become that person that the company values so highly, they will do anything to keep you – the linchpin upon which there success depends.

Of course, becoming and staying a linchpin isn’t easy. There is a lot of resistance to stay in your comfort zone. Godin goes into detail about the choice required to become a linchpin and why it’s the choice between being remarkable and being a cheap drone.

The brilliant thing about this book is that Seth goes into depth about “the resistance”, the thing that keeps us from being unique and taking the risk to let our special gifts shine. It is so easy to want to blend in and go with the flow, but Seth makes a compelling case that this is the path toward extinction.

After reading the book and thinking of my experiences in consulting in many companies, I realize that the only difference between many companies and a chicken slaughterhouse is the lack of chickens. Every day people dutifully report for duty, keep their heads down, do what they are told to do (or not to do), then go home and do it all again the next day. The work is mindless, requires no creativity and at the end of the day, people leave unfulfilled and unappreciated – and a bit bloody.

At times I felt conflicted as I read Linchpin. I identify with being the artist. I tend to procrastinate, so the idea of setting a ship date for something and then shipping no matter what has actually helped to get some things done. On the other hand, I have seen so many companies suffer from the deadline mentality I can’t totally embrace that idea.

I think there is balance in the decision to ship or not to ship. In some cases, shipping bad stuff on the deadline can be very bad. In other cases, it can be the first faltering steps toward a great eventual product. I think it all depends on how understanding people are. For things like cars, airplanes and pacemakers, it’s a bad idea to ship just based on the deadline being reached.

The other point I have difficulty with is the idea of not being attached to your worldview. I agree that we need to be able to see and understand other people’s worldviews. My point of departure comes in being so unattached that you have no stable set of beliefs. I think it’s important to know why you hold your worldview and even be willing to question it. I also think it’s fine to be passionate about your beliefs. The key is whether or not your beliefs are based in truth. I know, I know, this opens up a great philosophical discussion about “what is truth?” I just think you can be a linchpin and still be true to your beliefs.

Finally, perhaps the most troubling realization of all is the contradiction of on my closely held beliefs in the idea of systematizing of processes. I really think Dr. Deming had it right about defining a process so that anyone could perform it without error. This is a good thing for factories, hospitals and fast food places that rely on consistency of results. However, when it comes to intellectual work, we need the creativity of the artist.

This leads me to the application of this idea to software testing. On one hand, we need high levels of accuracy and repeatability for some types of testing. On the other hand, we need the creativity of the artist for discovering new defects and ways of performing testing.

This is a great book, an essential book, for anyone in today’s marketplace. Both young and not-so-young will be better prepared to deal with the world by reading Linchpin. This is an easy read and I especially like the way Seth fleshes out his ideas in a stream of coherent small sections of writing. This alone has changed how I plan to write my future books - many of which now have publication dates! Thanks, Seth, for another insightful and timely book.

I will be continuing these thoughts here on my blog in the coming days, so come on and join the discussion!

Dirty Little Secrets About Software Test Estimation

I've been doing a lot of test estimation lately and it causes me to recall things that I have taught in the past about project estimation and test estimation, in particular. I call these things “Dirty Little Secrets” because they are things that many people know, but few people admit.

The context of this article is about software test estimation, but some of these secrets can be seen in other types of estimating as well. This list is not intended to be complete, but rather the things I see most often in trying to estimate software testing efforts.

Secret 1 – Your Estimate is Probably Wrong

Take comfort in this. Too many people work and work to get “just right” estimates. The problems are:

1) As a tester, your estimates often depend on other estimates, such as the time needed for development, people available, etc. These numbers are often inaccurate, so that means yours will be also.

On one project, I was asked to come up with some quick estimates on about 15 projects for a given business. To help in this effort, I was sent a set of 15 documents, each of which contained estimates for one of the projects to be estimated. My approach was to apply a factor of 25% of the total development effort. Now this factor could be considered low, since the actual percentages can range between 20 and 50%.

However, there was one thing that made my test effort factor practically irrelevant. Many of the prior estimates had “TBD” on the design and code efforts. This gap in data rendered my assumptions to the level of sheer guesswork. I had a bigger problem, though. My client still wanted estimates. What's a consultant to do?

In this case, I made some assumptions with a warning that the estimates could be way off. I estimated the effort to design the project equal to that of gathering and documenting requirements. I also made the effort to code the projects the same as the effort to design the project. Yes, this gave the same number for person-weeks for requirements, design and coding. Keep in mind I had no resource numbers at this point, only estimated weeks of effort. I documented my assumptions and also the low confidence I had in the estimate. I assured my client the estimate would change once we had numbers for design and coding. Plus, we would need resource estimates before we could estimate costs.

About a week later, I got the missing data. Not surprisingly, the newly supplied estimates for design and coding were different from the ones I assumed. So, I used the new numbers and my estimates changed. I have yet to know how accurate any of the estimates are at this point.

(By the way, as things turned out, none of the projects were ever started!)

2) The Project Scope Will Probably Change.

There's not much you can do about this. Scope change happens for a variety of reasons, some foreseen and some unforeseen.

You never know when new requirements will emerge. New opportunities can appear as well.

Last year, I had a fence installed at my home. When we first measured the dimension of the area to be fenced, I used the area that I had assumed was my property. Since there is not a house on the other side of us, I could not use the adjoining property as a basis. After reading the plat map, I learned that our property is actually about ten feet narrower than I had believed. For years, I have been watering, fertilizing and mowing about 1,000 square feet of property that did not belong to me!

This new information changed the scope and therefore, the estimate and final costs of the project.

Software projects are more complex than home improvements, and this makes the challenge even greater. Our assumptions get challenged and often proven wrong. Whether it is the scope of requirements or the number of resources needed, scope can change either up or down.

3) You Don't Know What You Don't Know.

And...you won't find that out until it's too late. This is one of the biggest project risks and impacts much more than estimates. There's not much you can do about this one either. Just keep an open mind, keep questioning decisions and information throughout the project. Realize that you don't know everything and what you do know will change.

Here are a couple of prime examples of project “surprises” :

• People will fudge numbers to make themselves look better. Realize this happens and always question information, whether it be bug counts, estimates or the time spent on testing a feature.

• Deliverables will be late. In fact, they will be later than you think they will. Some things will be early, but more often than not, your work will be delayed by delays experienced by others.

Secret 2 – Your Estimate Needs to be Wrong – In the Right Direction

Why do I say something so outlandish? You only know how close your estimate is when the work is done. When my focus is on the accuracy of the estimate, I am at risk of conforming the work to the estimate, not what is in the best interest of the project.

When you truly understand your estimate is just that – an estimate, you can be free to adjust it. At least, you can adjust it for your own purposes if your management is not accepting of reality.

So what is the “right” direction? It's high.

There's an interesting debate among project managers as to whether or not it is ethical to “pad” estimates for the purpose of dealing with future unknown needs. Some believe you must have pads and others believe it is unethical to give an estimate you know is higher than you think is accurate.

I fall into the camp of overestimating. However, I don't call this extra a “pad”. Instead, I call it a “reserve” that is the safety net should extra time, money and/or people be needed. Even Scotty never gave Captain Kirk the real estimate!

I don't see this as deception. I see it as contingency planning and a necessary part of any estimate. In my experience, it's better to come in “under” than “over.”

It's funny that the people I know to be the best estimators have been doing it a long time and their estimates seem very high at first. I think these people have learned the lesson of having reserves in the estimate.

Secret 3 – The Recipient of the Estimate Already Has a Number in Mind

This is why I say that estimation of anything is an inherently human activity. There's a lot of feeling involved and not much logic at times.

Understand that your customer has an expectation already in mind. If your estimate goes too far above or below the customer's early expectation, silent alarms go off in their head.

In the case of my fence project, I had a number in mind. It wasn't a low price and it wasn't too far off from the estimate I received. When I did get the estimate, at least I didn't have sticker shock. Had my expectation been a lot lower that the estimate, I would have probably looked for a cheaper option.

Once I had an experience in which I gave an estimate that was so much lower than a competitor's and I almost didn't get the work! The prospective client felt that because of the lower price I must have been leaving something out. Thankfully, they asked me to explain why my quote was so much lower and I explained it was because I had a faster and more effective way of doing the job.

Secret 4 - Your Customer Assumes Your Estimate is Too High

This is one of the biggest hindrances to good estimation. Let's say that you do your best job of estimation with no reserves included. You present these to your management or customer and they take one look and immediately reduce it by one-half because they believe you must have inflated it. After all, the last person they got an estimate from did so. Therefore, the assumption is that everyone who estimates has excess in the numbers.

What does this do? It forces people to pad their estimates because they know management will assume they have been inflated. Therefore, it is a self-fulfilling prophecy.

Secret 5 – Your Estimates are Incomplete

Most people estimate time and cost. How many people do you know that also estimate the quality of the product? Not many I would guess.

You have probably heard the old adage, “cost, schedule, quality – pick two.”

While it's true that it's very hard to deliver all three aspects, it's still possible. High quality will probably cost more and may take longer than lower quality – at least in the short term. In the long run, high quality costs less because of lower maintenance.

In software development and testing, we often fail to consider the quality levels in a given estimate. For example, how much would a test cost that found a few defects and resulted in a buggy product versus one that found many defects (or found the right ones) and resulted in a high-quality product? More often than not we estimate how long testing will take, how many people we need, and how much testing will cost. A lot fewer people estimate how many defects they find at the various levels of criticality.

Conclusion

As you go forth and estimate, keep these little secrets in mind and you will have less stress over your estimates. If you look at how people respond to your estimates, you will see many of the secrets in action. These are things you can't control. However, you can understand that the dynamics exist and be prepared to keep expectations reasonable.