Automation test criteria: From withdrawing cash from ATMs, taking medical appointments, to buying parking lot tickets and groceries, automation is everywhere. The automation of straightforward tasks as well as complex activities like self-driving cars promises to save you a lot of time and costs.
The same is the case with testing practices. Back in the old days, software testing was only about finding application errors. But today, the scope of testing has broadened like never. Most common testing approaches like functional testing, regression testing, integration testing, are being replaced with automation tools.
Automation has surely changed the whole testing scenario but having said that, the most difficult task is to decide whether test automation is required or not. Of course, everything can be automated, but it is important to consider automation test criteria before you rush to the decision.
So, the smartest move is to automate the right test cases to maximize your testing journey. Before we move ahead to the selection of test cases, let’s quickly understand what a test case means.
How to identify the right test cases for automation?
If you wish to select just the right automation test criteria, you need to have a plan that helps you to get the most benefit out of your automation testing efforts. Well, you do not have to start from scratch while determining the test cases for automation but there are a few crucial factors that need to be considered while identifying the right automation test cases.
If the testing frequency and execution time are significant, automation is a must-have for these test cases. In context to resource requirements, it is crucial to check the involvement of devices, operating systems, browsers, databases, and platforms while running the test cases. It also depends on the involvement level of users (the higher involvement means less likelihood of automation). Another factor is the identification of test cases that require complex and flaky features or have a defined outcome for testing software (the more complex features lead to more likelihood of automation to avoid human error). Lastly, the automation downsides should not outweigh the value derived by automation testing. Here, the value includes the saved manual resources, time, and insights. And the downsides of automation include the cost of implementation and maintenance, human flexibility.
Now that you know the automation test criteria, it is time to choose the right test cases for automation. These test cases need to be compared carefully with the criteria. Check below to understand which test cases you should automate and when to test to avoid automation.
Test types fit for automation
Regression test (sanity and smoke test): These tests consume a lot of resources and time as they are the backbone of every release of the testing process.
Performance tests (stress and load test): These tests are very repetitive and consume a lot of time to reach the desired test coverage.
Data-driven tests: These tests are conducted on the crucial features of the application under test (AUT). It utilizes automation to minimize the human error potential on the critical components and data related to the product.
Repetitive test runs: This is the golden rule of automation that if the test run is repetitive, it should be automated. It will help you derive precise and accurate results, every time.
High-risk test cases: If the involvement of risk is high, the test cases should be automated. It will help you prioritize the test cases based on the potential of human error hazards.
Web application with critical parts: It is recommended to automate the critical parts of the application as it will help you to avoid possibilities of human errors. It is especially important for test cases that can jeopardize the overall web application.
Extensive tests: The test cases that imply large data sets need to be automated. It also increases the possibility of making mistakes in the case of manual testing.
Complex test cases: Automation is almost a rule when it comes to complex test cases. It helps you to be on the safe side while saving a lot of time.
Other tests: Some test cases that need to be automated are API tests, integration tests, cross-browser tests, unit tests, etc.
Test types not suitable for automation
Although automation is a promising solution for the speedy and high-quality testing process, automating everything will do you more harm than good. To overdo or misuse automation testing, stick to these four principles: always automate repetitive tests, risky tests, complex tests, and time-consuming tests. This will help you focus on specific automation test criteria and leave some valuable time for dealing with tasks that cannot be automated. Here are some tests you should not automate:
Ad-hoc tests: There are no concrete criteria for evaluating the software application and therefore, automation can result in false outcomes.
Usability testing or User experience tests: Automation cannot mimic the exact emotions and expressions of humans. For the same reason, these test cases should not be automated.
Intermittent or low-risk tests: These test cases produce unreliable results when they are automated.
Anti-automation: Some anti-automation features like CAPTCHA should never be automated as they can hamper the whole purpose of the feature.
Steps to identify the right test cases for Automation
Step 1: Identification of the parameters on which the test case is based for automation. Below are some identified parameters. However, you can identify your specific parameters based on your application.
Tests that can cause human errors
Repetitive tests that are used for multiple builds
Tests that are performed on multiple software or hardware platforms, and configurations
Tests that require different multiple data sets
Frequently used functionality with high-risk conditions
Tests that take a lot of manual testing efforts and time
Tests that are stable, low risk, and unlikely to change often
Tests that have a significant downtime between steps.
Step 2: Break the application into specific modules. For every module, it is important to analyze and identify the test cases that should be automated. The below list will vary depending on the project and can be further enhanced to suit your needs:
Y – Yes
N – No
Step 3: Consolidate various test cases for every module as shown below. Here, the details are quantified and provide an estimate to finish the testing process manually.
Step 4: Once the granular level details are identified, they can be presented in the below manner.
Step 5: Consider the below attributes that form the basis for determining the return on investment.
Time to develop and maintain the scripts
Purchasing and licensing cost of the automation tool
Management of overheads
Cost and time to train the resources
Based on the above factors, below is an example of test automation ROI calculation for 5 years.
*ROI = (Cumulative saving / Investment through automation) *100
Defining entry and exit criteria for test automation
Entry and exit criteria are specific conditions that need to be satisfied to enter or exit the project cycle. In context to automation test criteria, entry criteria are the conditions that should be satisfied for the testing to begin. And the exit criteria are the conditions that should be satisfied to stop the testing process.
Entry criteria: Once the design document is approved, the test plans need to be prepared in the planning phase. The necessary hardware should be installed and configured, and their functionality should be checked properly. The software code should be unit tested and signed off by the developer team. The test data and test cases should be prepared in advance and the resources should be well trained about the tools and their functionalities.
Exit criteria: When at least 95% of the test cases are passed, the tester can exit the testing phase. However, it is very challenging to determine when software testing can be stopped. Some of the primary criteria include:
When all bugs are fixed.
When more than 95% of test cases are passed.
When the deadline is reached, or agreement is signed off.
When alpha and beta testing is completed.
Well, there is no thumb rule for deciding the exit criteria. When the testing of primary functions is achieved, the testing usually stops, and the team looks for minor bugs that can create trouble later.
Test automation tool selection criteria
The selection of a test automation tool is one of the most important steps as it greatly affects the whole automation effort. If you select the right tool, it leads to an easier and effective automation process. Here is the list of the most important aspects to consider while selecting the automation tool.
Technology and platform support – Are you testing WPF, C#, or .NET application and their respective operating systems? Do you need mobile application or web application support? Do you work with different operating systems, Android or iOS?
Skill level flexibility – Is your QA department comfortable in writing automated test scripts or is there any need for keyword testing?
Feature-rich – Does the automation tool support record and playback test creation and manual creation of automated tests. Does the tool have the right set of features for implanting the checkpoints to verify the database, values, and key functionality of the application?
Maintenance – Create automated tests that are maintainable, reusable, and resistant to change in the user interface of the application. Will the automated tests break if there are any changes in the UI of the application?
Integration with the ecosystem – Does the automation testing tool integrate with CI/CD pipelines such as Azure DevOps? Does the tool support test management frameworks like Zephyr and defect management systems like Jira?
Testing of enterprise applications – Does the tool offer out-of-the-box support to test the enterprise applications like Oracle, SAP, Salesforce, etc.
Automate your testing strategy with QAonCloud
Just because you can automate every test, does not mean you should always do it. If everything will be automated, it will require a lot of software maintenance, additional money, and time. It is crucial to take great care of automation test criteria while selecting the test cases to ensure sustainable automation.
No doubt, automation testing is one of the best ways to accomplish all the testing goals and effective utilization of time and resources. But you need to be very cautious about selecting the right test cases and automation tools otherwise you will end up remaining on the shelf and with no return on investment. If you are looking for just the right match for your automation testing requirements, QAonCloud is your one-stop solution. It makes the automation, execution, and maintenance of automation seamless – all from the cloud. With QAonCloud, you can easily automate advanced test scenarios on multiple browsers and devices in the cloud anywhere across the world.
See how we work to build a cutting-edge automation process and ensure maximum test coverage at scale. Contact our QA experts today.
