How do People Select Test Automation Tools?
A large majority of experienced testers and managers prioritize licensing and support costs and good test reports when deciding which test automation tools to use for their projects. In the Test Automation Challenges survey conducted with 2,000+ test professionals, more than 60% of respondents with 3+ years of experience in test automation agreed that these criteria are the most important priorities in deciding a test automation tool. Moreover, respondents with less experience in test automation prioritize training and support materials over other criteria.
The top six priorities considered by more than 50% of respondents with 3+ years of experience are:
(1) Licensing and support costs
(2) Good test reports
(3) Training, documentation, tutorials, guidelines
(4) CI, DevOps support
(5) Level of programming skills required
(6) Level of skills and experience required
Open-source tools (coding experience required)
Free tools (limited community’s support)
Commercial tools (high support & license cost)
Software automation tools and frameworks are crucial for the success of test automation projects. But it is a difficult task to select a right set of test automation tools for a new software project. There are many tools with each having several unique good features. One single tool may not satisfy all project requirements and constraints, resulting in multiple automation tools to be used. Chosen tools seem to be working at the beginning, but troubles occur later as hidden problems are overlooked at the time of tool selection.
A tool vendor often provides a list of key features and characteristics when comparing their tool with others in the market. They tend to highlight their tool’s strengths while ignoring important features that may pinpoint its weaknesses. This practice confuses not only new test automation adopters but also experienced ones in making decision about which tools and frameworks for new test automation projects.
To select a suitable tool, one can base on a set of top priorities for which every interested tool is checked. Such checklist may come from experience of professionals as well as specific project requirements and constraints.
Understanding the priorities in determining automation tools is part of our survey which questioned software test professionals about their experience with test automation. The survey’s questionnaire was posted on several software testing websites and shared via several mailing lists including that of Katalon Studio’s.
We received answers from 2,291 respondents of which 1,877 (82%) have ever applied test automation on their projects. Of all respondents, 970 (or 42%) and 484 (22%) have at least one and more than three years of experience in test automation, respectively. The results reported here exclude the respondents who said to have yet to apply test automation.
Tool Selection Priorities by all Respondents
The top four attributes chosen by at least 50% of respondents as their most important priorities in tool selection include:
(1) Training, documentation, tutorials, guidelines: the highest percentage of respondents chose this attribute, which is concerned with training and materials for learning and using the tool, as a priority for their tool selection.
(2) Good test reports: tool generated test documents and logs for status report, test execution analysis, and defect diagnosis.
(3) Licensing and support costs: all costs associated with acquiring, maintenance, and support the use of tools.
(4) Level of programming skills required: if testers on the team do not possess good programming skills, learning programming to use a tool is a real concern for them.
The highest priority is identified by 56% of all respondents. Many respondents, 58%, have less than 1 year of experience in test automation. So, it is reasonable that training and materials for learning and using tools is the most important attribute for choosing a tool.
The fifth highest priority chosen by almost 50% of respondents is the level of skills and experience required by tools. This attribute, which is related to the 4th highest priority or the level of programming skills required, refers to the need of skills and experience to use the tool effectively. For example, Selenium is a popular test automation framework but it requires adopters to possess a high level of technical skills and experience to get started.
It is quite surprising to find that a majority of respondents did not consider user interface/user experience (UI/UX), support services, long-term commitments, and application lifecycle management (ALM) tool integration as the priorities in selecting an automation tool. Noticeably, less than 20% of respondents chose the ability to integrate with the ALM tools as an important priority.
It is worth noting that although functionality/feature-rich of a tool is often cited as a strength, it is not appreciated by many respondents with just 46% of them using it as a priority.
Tool Selection Priorities Evaluated by Experienced Respondents
We considered two respondent groups, one with at least 3 years of experience in test automation and the other with 1 year or less. It is intuitive to believe that the feedback on the top priorities for selecting tools by experienced test automation professionals is more reasonable than that by less experienced ones.
➤➤➤ Continue reading “How do People Select Test Automation Tools?”