Test Automation: Trends, Benefits, and Costs

  in Special Issue   Posted on September 3, 2020

Information for the Special Issue

Submission Deadline: Wed 31 Mar 2021
Journal Impact Factor : 2.450
Journal Name : Journal of Systems and Software
Journal Publisher:
Website for the Special Issue: https://www.journals.elsevier.com/journal-of-systems-and-software/call-for-papers/special-issue-on-test-automation-trends-benefits-and-costs
Journal & Submission Website: https://www.journals.elsevier.com/journal-of-systems-and-software

Special Issue Call for Papers:

Today software has a significant impact on all aspects of our society, and its functioning is crucial for a multitude of economic, social, and educational activities. As a consequence, assuring the correctness and quality of software systems and components becomes paramount.

Automation across the Software-Testing Process is a powerful asset: while originally conceived for test execution, nowadays it is increasingly used for test generation, test prioritization and selection, test repair and evolution, among others, as well as for automatically comparing the actual with the expected outcome. Investments on automation of both tests synthesis and their execution is pursued to help ensure the adequate quality of software systems/applications while reducing the high effort and costs incurred in the testing of complex systems. De-facto, test automation is central in many modern approaches to software development such as model-driven engineering, agile frameworks, or TDD (Test Driven Development). It also represents a cornerstone for evolutionary development life-cycle focusing on “Continuous Practices” (i.e., continuous integration, testing, delivery, and deployment), or more recently on DevOps.

On the other hand, as with any piece of software, development and maintenance of test code require considerable effort and skills. Besides, tests themselves need to be kept aligned with the ever-evolving system/application under test. Therefore, the potential benefits aimed by test automation have to be weighted against its costs and drawbacks.

In today’s competitive world, companies demand a good Return On Investment (ROI) for every task concerning software development, and software testing is no exception. And in complex technical endeavours such as software testing, calculation of investments and returns cannot be reduced to just a balancing of money flow, but should also consider other non-financial, yet very concrete and critical aspects, such as social acceptance, developer’s reputation, security and privacy, etc.

On the one side, research in test automation is very active, and in the last years, several solutions and tools have emerged that claim to improve the cost/effectiveness of testing. For instance, design patterns used for structuring test code, refactoring strategies and tools helping in development and maintenance. However, we are not yet able to foresee the investment costs versus money saved when a proposed test automation solution is introduced in a Company.

On the other side, managers demand evidence that proposed solutions actually improve the ROI and need applicable metrics to estimate the concrete costs/benefits of test automation. They would also need support in deciding which solution is best for their context, and often this is impeded by the large gap between an academic solution and the many concrete issues encountered in transferring it to practice, e.g., lack of scalability, flakiness, tight time-to-market. Moreover, inadequate testing and wrong decisions in test automation may be clearly the cause of a technical debt, and being able to reason on costs and benefits of test automation solutions can help prevent and reduce it.

Based on the above reasoning, we believe that research in test automation needs: i) to better investigate how to integrate means to assess the usefulness and applicability of new techniques and approaches; ii) to understand what are the issues that prevent broader adoption of existing tools or solutions, and iii) to collect evidence of (financial, technical and social) costs and benefits of the current trend and practices in test automation.

This special issue welcomes contributions regarding approaches, techniques, tools and experience reports about adopting and improving test automation cost/effectiveness, as well as empirical studies and case studies investigating the cost and benefits of test automation.



Topics of interest include, but are not limited to, the following:

● Approaches and tools for improving test automation effectiveness and ROI, for example by

      ○ Reducing development and maintenance efforts/costs (e.g., automated generation of test code and self-healing test scripts)

      ○ Increasing test coverage and bug detection

● Costs and benefits of Test Automation in modern systems and applications, for example:

     ○ Web applications

     ○ Mobile applications (e.g., Android, iOS)

     ○ Internet of things systems

     ○ Machine learning and artificial intelligence based systems

● Costs and benefits of Test Automation at any testing level, for example:

    ○ End-to-End, acceptance, integration, functional, unit

● Measurement frameworks relevant for Test Automation

    ○ Software and process metrics

    ○ Technical debt in software testing

● Empirical studies concerning costs and benefits analysis and evaluating the cost-benefit trade-off of Test Automation (note that Surveys, (Systematic) Literature Reviews or Mapping Studies are out of the scope of the special issue and will be desk-rejected)

    ○ Experiments

    ○ Case studies

    ○ Comparative experiences and evaluations



● Manuscript submission deadline: 31 March 2021

● First notification: 30 June 2021

● Revised manuscript submission: 01 October 2021

● Notification of the final decision: 30 December 2021



Guest Editors

● Maurizio Leotta, University of Genova, Italy (maurizio.leotta@unige.it)

● Guglielmo De Angelis, CNR-IASI, Italy (guglielmo.deangelis@iasi.cnr.it)

● Filippo Ricca, University of Genova, Italy (filippo.ricca@unige.it)

● Antonia Bertolino, CNR-ISTI, Italy (antonia.bertolino@isti.cnr.it)



Paris Avgeriou and David Shepherd


Special Issue Editors

Wing-Kwong Chan and Raffaela Mirandola


Submission Guidelines

All manuscripts and any supplementary material should be submitted through the Elsevier Editorial System for the Journal of Systems and Software at: http://ees.elsevier.com/jss/. Please, follow the submission instructions given on this site. Authors should select “VSI:TestAutomation” from the “Choose Article Type” pull-down menu during the submission process.

All submitted papers should adhere to the general principles of the Journal of Systems and Software articles. Submissions have to be prepared according to the Guide for Authors, available on the journal website. The submitted paper must follow the format specified in the JSS Guide for Authors (https://www.elsevier.com/journals/journal-of-systems-and-software/0164-1212/guide-for-authors).

Manuscripts must not have been published previously or be currently under consideration for publication in any other journal or conference. Note that an extended version of previously published papers has to contain at least 30% of new material different from the original work. Also if a submission includes significant portions taken from previous works then those sentences must be paraphrased or modified otherwise the manuscript will be desk rejected. Authors are requested to attach to the submitted paper their relevant, previously published articles and a summary document explaining the enhancements made in the journal version. Furthermore, the manuscript must explicitly include a proper citation to the previous work highlighting how the new contribution has been improved.



Review Process

To ensure the highest quality and the timeliness of the review process a board of distinguished reviewers has been established. Each submission will be reviewed by at least two experts including members of the board of distinguished reviewers. The final decision about each submission will be taken by the Guest Editors in collaboration with both the Editors-in-Chief and the Special Issue Editors.



Distinguished Reviewers

Jonathan Bell Northeastern University (USA)

Lionel Briand University of Ottawa (Canada), and University of Luxembourg (Luxembourg)

Gabriella Carrozza Accenture Italia (Italy)

Marcelo d’Amorim Federal University of Pernambuco (Brasil)

Serge Demeyer University of Antwerp (Belgium)

Sigrid Eldh Ericsson AB (Sweden)

Emelie Engström Lund University (Sweden)

Michael Felderer University of Innsbruck (Austria), and Blekinge Institute of Technology (Sweden)

Robert Feldt Chalmers | University of Gothenburg (Sweden), and Blekinge Institute of Technology (Sweden)

Moonzoo Kim Korea Advanced Institute of Science and Technology (South Korea)

Mika Mäntylä University of Oulu (Finland)

Leonardo Mariani University of Milano-Bicocca (Italy)

Dusica Marijan Simula Research Laboratory (Norway)

Mike Papadakis University of Luxembourg (Luxembourg)

Patrizio Pelliccione University of L’Aquila (Italy), and Chalmers | University of Gothenburg (Sweden)

John Penix Google, Inc (USA)

Silvia R.Vergilio Federal University of Paraná (Brasil)

Rudolf Ramler Software Competence Center Hagenberg GmbH (Austria)

Daniel Sundmark Mälardalen University (Sweden)

Hong Zhu Oxford Brookes University (UK)




If you have any questions concerning the topics and scope of this special issue, please do not hesitate to contact the guest-editors.

Other Special Issues on this journal

Closed Special Issues