Best Practices in Software Test Automation

What are Best Practices?

Best practices are guidelines and advice on the best way to do something; collected over time and based on experience with previous projects. Best practices in an organization should come from the bottom of the organization up, rather than being mandated by management. These best practices might even differ from organization to organization, but certain key factors are present in successful organizations and projects. Discussed in this article are twelve of the most important practices that can assist you in ensuring successful implementation of functional test automation on your projects and in your organization.

• Best Practice 1: Know your objective

There are many good reasons for doing test automation.

Test automation can save time, make testing easier, and improve the testing coverage. It can also help keep testers motivated; I can list a page of other benefits that can be derived from doing functional test automation. However, it is not likely that your organization will need to do all these things at the same time. Different groups typically have different hopes and ideas of what they want test automation to offer them. These need to be stated, or else disappointment is likely.

A clear objective to work towards during test automation is highly important. It provides us with a compass heading towards which planning and implementation will be directed. Eventually, the results of the test automation implementation will be compared against the original objective to establish whether we did reach our goal or not.

Mistakes made...

1. · "Test automation is the answer to our testing problems"

Test automation assists in the testing process, and can therefore provide value to a test team. Certain benefits can be derived, but we need to know and understand exactly what they are without having false expectations or hopes. Automation is not a replacement for an inadequate test process or an inefficient test team. Address the testing problems prior to embarking on a costly test automation implementation. Automation's objective is definitely not to solve all testing problems in existence.

1. · "Since we have limited time to test, let's use test automation."

Test automation takes 3 to 128 times longer to implement than manual testing, and therefore it should be properly planned to achieve the expected benefits. If time is of the essence, consider adding more resources to the test process (this also has limited benefits, and might not always be a solution) rather than embarking on the lengthy process of implementing test automation. Functional test automation is definitely not a quick solution to a test project in distress. Multiple other solutions are better suited to address this need.

• Best Practice 2: Test Automation requires a manual test process

"Success in test automation requires careful planning and design work, and it's not a universal solution. ... automated testing should not be considered a replacement for hand testing, but rather as an enhancement." (James Bach [pg. vii] in his foreword: Software Testing with Visual Test 4.0, Thomas Arnold II, IDG Books Worldwide Inc. Foster City, CA.: 1996. ISBN 0-7645-8000-0.)

The words of James Bach and many others are quite clear on this subject. Test automation will not give you a correct testing process or methodology. It will not enforce a methodology or process. Test automation merely supports the test process; it will not replace the manual test process.

Mistakes made ...

1. · "We need to implement testing in our projects. Let's buy a tool and automate the testing."

You can script to your heart's content, if you do not have a proper manual test process in place the benefits of test automation will never be seen. It will only assist you in doing the incorrect process a lot faster. Test automation merely assists and enhances the test process itself. Ensure that a proper test methodology is implemented before attempting to add test automation.

1. · "We can just add the test automation to our current test process and start automating our current test scripts."

Test Automation requires test cases to support the automation process. If my test cases do not incorporate the all-important logon to the SUT (system under test), the automation script will need a lot of manual intervention, and would therefore be more manual than automated. The current test process might have to be changed, or adapted, to be implemented for the tool suite selected. Review and adapt the test process to ensure maximum benefit is realized from the tool suite to be used in your test process.

• Best Practice 3: The earlier the involvement in the Software Development Life Cycle (SDLC) the greater the benefits

We all know how testing has its own phases aligned with the phases of the SDLC. The earlier testing gets involved in the life cycle of the project the better. Many articles have been written on this subject, and it holds true. Since test automation supports the testing process, the test team can get involved very early in the life cycle of a project. If the test team uses a test automation tool suite, they will most likely have a requirements management tool or test management tool included in the suite. These tools can assist the test team in various ways from an early stage in the development life cycle, even before a line of code is developed.

The more mature techniques and methods of test automation are closely tied to the testing methodology being used. An early start to the test life cycle, at the start of the development project life cycle, is vital to achieving the best results, and ensuring maximum return on investment.

Mistakes made…

1. · "The test automators cannot get involved in the project before the developers release the first build."

A software development project does not start with code development. Since functional test automation is also a development project, as we will discuss in Best Practice 9, it follows the same phases as a software development project and does not start at the coding phase. Proper planning and design are essential to the success of the functional test automation implementation, and need to be conducted hand-in-hand with the test life cycle.

1. · "Testing is only a single phase in the SDLC and therefore testers and test automation only need to be part of the project for the testing phase."

This 'prehistoric' misconception in the IT industry has been proven one of the major reasons for bad software quality.

• Best Practice 4: Test Automation is a fulltime effort

When people are allowed to work on test automation on their own time, or as a back burner project whenever the test schedule allows, test automation becomes a sideline effort. Test Automation does not get the time and focus it needs.

Getting into automated testing requires preparing not only yourself, but also your company and environment. This requires focus and dedicated resources.

Mistakes made…

1. · "We have a test automation tool. Can the test team see if they can use it?"

It is clear that the expectation and an understanding of test automation is lacking. I can guarantee that the tool implementation will not be successful, and would result in almost no benefit to the test team. The test team is usually confronted with manual test work for development projects that does not meet the code delivery date for testing. The test automation implementation would probably be of the capture-record-and-playback form, and therefore the benefit would be low.

Test Automation will require training for the staff involved with it. Time is required to design and implement the architecture. Referring to the development nature of functional test automation implementation, the team will have to learn a new skill set. Test automation implementation requires dedication and focus to ensure success. In this instance, it is better to not implement test automation than to allocate ad hoc time for experimentation.

• Best Practice 5: Select the correct product or suite of products

Buying the wrong tool is listed as one of the major challenges in test automation, because - no matter what kind of process, methodology, or organisation you have - if the tool is not a good technical and business fit, it will not be used.

We know that a good process and organisation are also essential for test automation. However, if the tool will not function at a basic level, the people who should use the tool, and thereby gain the benefit from the tool, will not use it.

Unfortunately, too few people do adequate research before buying a test tool. Adequate research includes defining a set of tool requirements based on what the intended users of the tool need to accomplish, developing a set of evaluation criteria by which candidate tools will be judged, and taking the experience of other people who have used the tools into consideration.

Select a tool that will allow you to implement automated testing in a way that conforms to your long-term testing strategy and test methodology.

Mistakes made…

1. · "A vendor demonstrated a tool to us and it seems as if the tool will add value. I think we should buy it."

Take time to define the tool requirements in terms of technology, process, applications, people skills, and organisation. Then look at multiple tool vendors and select the right fit for your circumstances. A good idea would be to do a proof of concept project with the tool that best fits your criteria.

1. · "We purchased the whole suite of tools from the vendor, but are only using the functional test automation tool."

Test Automation tools are expensive, and to justify the Return On Investment (ROI) it is normally not effective and efficient to use only the functional test automation tool in the product suite. Other benefits can be derived from using the full suite of products; for example, proper defect management, metric reporting on test status, etc. Purchase what you need, or use what you have fully.

1. · "The yearly license fees are so expensive we cannot afford to use the tool suite any more."

Too often test automation tools end up as 'shelf-ware' because of the cost of the licenses compared to the benefit derived. Refer to mistake two above for one of the reasons why the ROI is not being realised. Before signing the purchase order, be aware of the impact of annual licensing fees.

• Best Practice 6: Get executive management commitment

Test automation can yield very substantial results, but requires a substantial commitment to be successful. Do not start without commitments in all areas, the most important being executive management.

'Thomas R. Arnold II, VP at ST Labs, Inc., sums it up fairly well:
"I would like to say up front ... that no test automation tool is a silver bullet. Automation takes time, effort, and commitment from all involved, including an understanding from management about the realities of what automation can and cannot do." ' 1

Management support is needed in designing and deploying test processes that will support the effective use of test tools, reinforce the role and use of automated test tools in the organisation, and allow time for tools to be integrated in the testing process.

Without management support, the entire test automation effort is at risk. If management does not - clearly and consistently - show their support for test automation, people will be less inclined to show interest in using the tools. This is a major concern, especially considering that overcoming the learning curve of some tools requires dedication.

Perhaps the greatest challenge seen in management support is balancing the high expectations of tool benefits against the time, effort, and discipline it takes to implement the tool. Management may become impatient about the lack of tool progress and shift their support to other initiatives.

Mistakes made…

1. · "We thought these tools were going to solve your testing problems."

When making the case to management for acquiring test tools, present the challenges as well as the benefits. Make sure that management understand their influence on how others will accept automated test tools.

1. · "We purchased a tool for you 3 months ago, and yet we cannot see any benefits in the project."

Communicate to management from the start that it takes time and planning to build a firm foundation of people, processes, and the right tools. Keep management informed of tool progress, and any issues that arise.

• Best Practice 7: Select the appropriate technique/s for the project

As maturity levels improve in the functional test automation field, we see a decrease in cost for test automation implementations, and an increase in benefits derived for the project or organisation.

The increase in maturity levels has provided us with more mature techniques for implementing test automation. These techniques vary from the simplest record-and-playback to the most mature, viz., keyword or action based. These techniques each have their own benefits and drawbacks, appropriate for certain specific circumstances.

During a small data conversion project, a few lookup data tables' contents in the old system need to be compared to the new lookup data tables' contents after the batch conversion. If this can be done via the enquiry functions of both systems, the fastest method might be record-and-playback. This is a once off test of the conversion, and writing specific scripts to do this might take longer than the record-and-playback method. In terms of cost and timing, the record-and-playback technique would be the most appropriate for this project. This is one of the few instances where I would suggest using the record-and-playback technique; most of the time the benefits of this technique are few compared to the more mature techniques like data-driven and keyword test automation.

It is important to know, and be able to use, multiple techniques appropriately. If we do not use the various techniques when appropriate, our test automation will not provide us with the benefits we are aiming for.

Mistakes made…

1. · "Record-and-playback is sufficient for our needs."

The benefits of this simplistic technique are few, and its application is widely misused. More mature techniques are available with exponentially more benefits in terms of ease of use, cost saving, and time saving if implemented correctly.

1. · "We have created our test cases for the project and are approaching the release of the first build. Can we implement keyword test automation before the end of the project."

Keyword test automation is a very advanced test automation technique, and is closely tied to the testing methodology being used for the project. The preparation work for implementing keyword test automation requires more time than some other techniques. The framework needs to be established, and test design needs to support the method. I would not suggest implementing the method from scratch at this late stage of the project.

• Best Practice 8: Do not attempt to automate all tests

Test Automation does not work well for all situations or tests. Some tests are better left for manual execution. If a test will not be repeated more than three times, it is probably not a good candidate to automate. It takes a lot longer to automate a test than to execute it manually. The benefit of the automated tests is linked to how many times it would be repeated. For example, multiple builds, or testing on multiple platforms, or executing the same test for multiple data sets.

Software that is tested manually will be tested with a randomness that helps find bugs in more varied situations. Since a software program usually will not vary, automated testing may not find some bugs that manual testing will. Automated software testing is never a complete substitute for manual testing.

Proper analysis needs to be done to identify the correct test cases to automate to ensure ROI for the effort. How to select tests for automation is a subject on its own, but the following suggestions can be helpful:

1. Perform a proof-of-concept (POC) on a cross section of test cases for the system to determine technical applicability and return on investment (ROI) metrics.

2. Perform an ROI analysis for each testing regimen to select automation candidates.

3. Perform a risk analysis on the automation candidates to prioritise what should be automated first.

If the tests selected for functional test automation are identified and prioritised, the test cases with the best potential to provide benefit will be automated first. If time constraints affect the test automation delivery, at least we know the largest possible benefit would be derived.

Mistakes made…

1. · "It was so much fun that we automated all the tests."

It might have been fun, but the ROI was definitely not optimal. Time spent on automation of certain tests would never be recovered. A general rule of thumb would be never to automate more than 60% of your test cases to ensure maximum benefit with the lowest investment cost.

• Best Practice 9: Manage the automation as a development project

Test Automation development is a software development process, although it is seldom treated that way. Following software development practices can make the difference between the success and failure of an automated testing project.

We need to run test automation projects just as we do our other software development projects. The following pertains to test automation projects, and is true for development projects:

· Projects need developers dedicated to developing test automation.

· Test automation automates a task in which the programmer is probably not an expert. Therefore, expert testers should be consulted and provide the requirements.

· Test automation benefits can be seen if we design our approach before we start coding.

· Test automation code needs to be tracked and safeguarded. Therefore, we need to use source code management.

· Test automation will have bugs. Therefore, we need to plan to track them, and test for them.

· Users will need to know how to use it. Therefore, we need user documentation.

The type of work done during test automation is software code development, and since these are programs, they must be managed in the same way that application code is managed.

Mistakes made…

1. · "When the code is complete, give it to the testers so they can implement test automation."

Why would we follow a phased approach to software development, but not with test automation script development? We should enforce the same stringent processes and phases on test automation scripters as on developers… and the scripts need to be tested.

1. · "You don't need to follow all the phases of the software development life cycle with test automation."

The fact that we are writing software to test software, and not to perform some business scenario, does not mean the method is different. What is important for the one is important for the other; both are software development projects.

• Best Practice 10: Use the correct resources

One area consistently missed by organisations desiring to automate testing is the staffing issue.

Automated testing is different from other types of testing:

1. Most tools use procedural languages to create their scripts.

2. These tools are very similar to IDEs (Integrated Development Environments).

3. Debugging scripts is the same as debugging traditional software.

…and therefore, resources are going to need:

· academic-level knowledge of procedural coding languages,

· basic knowledge of coding practices and procedures, and

· in-depth knowledge and practical experience with the tool being used for the test automation.

Because of this and because 'record-and-playback' (which generally does not require learning the back-end scripting) is not a valid way to automate most tests, the use of these tools becomes like that of a development effort. Given that such is the case, there must be support within the organisation to realise this and hire staff accordingly, as well as giving the necessary time for the development.

To accomplish this, a Test Automation Tool Specialist, or similar, position must be created, and staffed with at least one senior-level programmer. It does not really matter in what languages the programmer is proficient; what is important is that this person must be capable of designing, developing, testing, debugging, and documenting code.

Test Automation is a combination of testing and development. To mentor and teach your team and resources, use consultants as appropriate to bootstrap your effort. Learn as much as possible from theses consultants to enable you to avoid the stumbling blocks, and proceed successfully with your automation effort after they depart.

Mistakes made…

1. · "We sent our testers on a test automation course, and they still can't implement proper test automation scripts."

Learning the test tool Integrated Development Environment (IDE) does not make you a good scripter. A 3-day course will not teach you programming principles. Mentoring by a developer can accelerate the learning process. Add a person to the test team who is a test scripter. This person should be comfortable working with code, and be able to take the basic test designed by a test analyst and convert it into an automated script.

1. · "The testers did attend the vendor training."

The vendor training will train your staff in the product. Teaching someone Microsoft's development studio will not make them a code writer or programmer. The person will merely know how to use the IDE to write code. It is going to require more than just vendor training to turn your staff into test automators. I suggest hiring a test automation consultant to assist with getting the test team out of the starting block. The consultant can train and mentor your staff, and teach them how to avoid the most common pitfalls. Start the team with simple basic scripting concepts, and add complexity later.

• Best Practice 11: Develop for maintenance

Without proper automation architecture planning and design the test team will soon find itself with hundreds or thousands of test scripts, thousands of separate run result files/logs, and the combined work of maintaining the existing scripts for various versions of the system, as well as creation of new scripts for new enhancements. Soon they will find themselves with a maintenance nightmare.

To avoid the maintenance problems that so many organisations succumb to, the following pointers can help:

· Create a reusable architecture.

· Develop generic, cross application initialisation and configuration parameters.

· Create reusable, cross application functions.

· Design test scripts by assembling components.

Avoid the creation of disposable automation through proper planning and design of your test automation implementation.

Mistakes made…

1. · "We did not design with maintenance in mind."

If you did not start your test automation with a design for maintenance, you have probably already become a victim of the maintenance nightmare. Review your current scripts and determine what amount of script/code snippets can be reused if you implement a different approach (maintenance focused). If the amount of rework is large, it might be better to restart your effort with the correct approach in mind.

1. · "No standards were followed in script creation."

This will also result in difficulty during the maintenance phase; or if the resource that created the scripts leaves for greener pastures. Coding and scripting standards allow for ease of maintenance for the whole team. Different people can understand each other's scripts with ease, and when someone leaves the team it is not the end of the world. When test automators return to scripts that they have not worked on for a while it is also easier to understand what they did the first time round. This leads to a reduction in script maintenance time.

Start to implement standards and good coding practices for the test automators or scripters in your project/organisation. It is never too late to start with this practice.

• Best Practice 12: Review and improve implementation process after each project

Development practices suggest having 'post-mortems' at the end of a project. This holds true for the test automation project as well. The following information will surface during a 'post-mortem':

· New ways to deal with specific issues that were discovered during the project may be suggested.

· Experiences gained will provide insight into changes required for the current process or architecture in use.

· Ideas may emerge that may enhance and make the current process more efficient.

· Metrics to be reviewed may provide valuable information for the enhancement of processes.

· Further training or mentoring needs may emerge.

Use the information from the 'post-mortem' to learn and adapt your processes and best practices to gain maximum benefit from your test automation implementation.

Mistakes made…

1. · "We are still implementing test automation in the same manner as we did on our first test automation project in the organisation."

It is clear that the team/organisation is not learning from its experiences. Learning from our experiences allows us to change our ways for the better, adapt processes for maximum benefit, and make life easier for ourselves. The test team needs to review what works for them and what does not, and adapt their methods to improve for future projects.

1. · "What benefits did we derive from this automation effort."

This should be easy to answer if we followed Best Practice 1: Know your objective. We set out with a goal in mind and, while we implemented the test automation, we worked towards our goal. At the end of the project, we should know if we achieved what we set out to do. If we cannot answer, it is time to implement Best Practice no. 1.

Conclusion

Once you determine your company profile, perfect your processes, establish test specialists, give the team members appropriate testing tools, and follow the best practices laid out in this article; your company can realise the benefits of automated software testing. When compared to manual testing, properly applied automation will result in higher quality products, lower risk to your company, faster approval, and decreased time to market.

The higher level you reach on the automated software testing maturity ladder, the more benefits you will realise. Whatever level you choose, however, keep in mind a major lesson of the past few decades: No matter what tools you buy, your largest investment by far will be in the processes and people you put in place to use those tools.

Use the best practices as specified by those who have travel-led the path to successful test automation implementation, and adapt these practices to fit your organisation's individual needs by learning from your test automation implementation.

By Henk Coetzee

--end