•   When: Monday, November 28, 2022 from 10:00 AM to 12:00 PM
•   Speakers: Kesina Baral
•   Location: ENGR 4801
•   Export to iCal

A significant change in software development over the last decade has been the growth of test automation. Automated tests that can reveal software failures not only save time and money, but also increase reproducibility, reduce errors during testing, and ultimately lead to software that is better and cheaper. However, developing and maintaining high quality test suites that can evaluate software quality and ensure reliability is challenging. One reason for this challenge is the difficulty of creating automated test oracles. Automated tests must include code to check whether software behaves correctly on individual tests by comparing expected behavior with actual behavior, thus revealing incorrect behavior. This code is called test oracle. To write test oracles, developers must first understand the software requirements and identify correct behaviors. Next, developers must implement code for automated comparison on expected and actual behavior. However, the identified software behavior may evolve as the software requirements evolve, and the test suite needs to evolve as well. Managing the growing test suite by hand can result in test bloat, where the test suite has more tests than are needed to effectively test the software. Failure to create correct test oracles and effectively maintain test suites can result in low quality test suites and in turn low quality software.

In this research, we designed, implemented, and evaluated approaches and tools to improve test quality by tackling the challenges of test oracles and test maintenance. More specifically, the research presented a study of blind tests–a common but not widely studied test oracle problem. We conducted three empirical studies with students and professionals to identify the prevalence and root causes of blind tests. We found that blind tests is a serious problem for test automation and is prevalent in tests written by students as well as professionals. Second, we developed a behavioral oracle taxonomy for Android apps that characterizes GUI failures in different levels of abstraction. Next, we designed Magneto, an approach for automating Android test oracles. Our evaluation illustrates that Magneto has 94.6% accuracy on average. Finally, we also presented a framework to tackle test maintenance problem. We developed a framework to automate test management by enhancing automated tests to track software changes and re-run as needed. Our evaluation shows that the framework takes 6.87 seconds to analyze software changes and the accuracy of the framework recommended action is 94% on average.

The goal of this research is to improve software quality, through effective and efficient testing. We demonstrate that the approaches and tools presented in this research can improve software quality through a series of empirical studies and evaluations. We developed techniques and strategies to automatically create automated test oracles that are more effective than existing test oracles. We also designed techniques to effectively automate test maintenance activities that are currently done by hand.