Test Automation is Not Automatic

Recently while teaching a workshop on Testing Dirty Systems, I uttered this “Randyism” off the top of my head, “Test automation is not automatic.” I realized immediately that I had just concisely stated the problem in making test automation a reality in many organizations.

Most testers know that test automation is not automatic. (Wouldn’t it be great?) However, management many times does not know or accept that reality.

There are some test tools, such as unit test tools, that are practically automatically applied. My remarks in this article are aimed at the capture/playback and scripting tools for test automation.

The issues are that:

1) Not every test can or should be automated

2) For those tests that can be automated, it takes time and effort to build the automation

3) For those tests that have been automated, the tests must be maintained

4) It takes time to lean how to use a tool

5) It takes effort and planning to implement a test automation framework

None of these are automatic, even with the best of tools.

Not Every Test Can or Should be Automated

Think about the things you test that are not very repeatable. Or, they may be prone to constant change. Perhaps the things you test are developed in a technology that has little or no tool support.

Then, there are tests such as user acceptance tests that need people’s evaluation to judge acceptance.

Some tests require creativity and adaptation to perform. You may have to make judgments during the test, which may be too complex to describe in a script. Test automation leverages the mundane testing to give more time and attention to the unique tests.

Your job is to identify the tests that can be automated. (They don’t come labeled!) Then, you must understand the nature of the test, such as the pre-requisites, the steps to perform it, the exceptions and where to find the expected results. None of this is automatic. It’s all test design and test implementation.

For Those Tests That Can Be Automated, It Takes Time And Effort To Build The Automation

It’s one thing to automate a function, but another to design a good test of the function. That’s why capture/playback is so appealing, yet lacking. The issues are: What are you testing in the capture session? Then, how can you extend those tests to add value?

You have to apply approaches like data-driven testing and keyword-driven testing, which take time and effort to understand and implement. These are not “out of the box” deliverables.

For Those Tests That Have Been Automated, The Tests Must Be Maintained

This has been one of the consistent issues in test automation. There are things you can do to ease the maintenance burden, but it doesn’t do away with the issue.

For example, modular and reusable test scripts are very helpful in reducing the number of tests that must be maintained. Still, you must maintain the scripts you have.

This means you must know when the application has changed, what the changes were, and create new tests for those changes. This implies the presence of configuration management and traceability.

It Takes Time To Lean How To Use A Tool

Of course, the learning curve varies by tool, but the fact remains that a tool with sophisticated features will take some time to learn. The learning curve also varies by person. Some people with deep experience in automation may be able to learn very fast, but when you consider the wider deployment, most people will fall on the lower end of the experience scale.

The time also varies by the approach used to get training. Some people try self-learning which is noble, but typically takes longer than classroom training. Mentoring from an experienced person may be the best approach.

You must assess your organization’s skills and abilities to know even if mentoring will help. The best mentor in the world can’t help the person who isn’t ready to learn. Is that another “Randyism?”

It Takes Effort, Money And Planning To Implement A Test Automation Framework

The framework can take a variety of forms, but the context here is the organizational framework which provides a way to efficiently build and control test automation. Tools are only one-third of the picture. You also need processes with trained and motivated people to make everything work together.

Out of the box, tools are just software. A process framework helps with reuse and the maintenance of test automation.

Frameworks aren’t automatic, either. They must be adapted to fit each situation, therefore they require time, effort and funding to design and implement.

Conclusion

This article is certainly not an in-depth treatment of this topic. Entire books and training courses have been written to address these and other aspects of test automation.

I hope this expands on my simple statement “Test automation is not automatic” to provoke your own thinking on this reality.

For over twenty years now, I have seen a wide variety of test tool companies promote their tools as being effort-free, script-less, or however they choose to position their products. The evidence shows these are often empty claims. Just compare the numbers of people who have been successful in using test automation tools to those who have given up using the tools. It’s about 25% successful to 75% unsuccessful in my research.

I hope this article is an encouragement to those who have tried to implement test automation as well as to those who haven’t. It’s important to go into these projects with your eyes open and expectations at a realistic level. Once you get past the idea that the tools do all the work, you can do the planning and other work needed to increase your chances of success.