Everyone knows ‘record and replay’ is an immature approach to test automation that leads to fragile automated test suites. Right?
Well the reason I’ve thought this, for a very long time, is that most record and replay tools (for example, Selenium IDE) duplicate element locators throughout the scripts they create.
For example, if I record five variances of signing up for a WordPress.com site, then I will have five ‘copies’ of how Selenium has located elements on those various signup screens, which, if any of these elements change, will mean all five tests will be broken. This is test suite fragility.
Your web application regression tests created using record/replay tools are fragile and keep breaking. Hammoudi et al. set out to find out why. If we knew that, perhaps we could design mechanisms to automatically repair broken tests, or to build more robust tests. The authors look at 300 different versions of five open source web applications, creating test suites for their initial versions using Selenium IDE and then following the evolution of the projects.
Over 70% of all breakages are due to locator fragility in the face of change, and over 50% of all breakages are further due to attribute-based locators.
Adam Colyer – Why do record/replay tests of web applications break?
This also highlights a second reason why record and replay fails; as well as those locators being duplicated they may also not be the best locators.
For example, a record and replay tool like Selenium IDE may select elements based upon id, or xpath, even though these may change with every version of the application so using class which may be more static and therefore more appropriate. Or even better, using custom data-attributes might be the best solution of all. But a record and replay tool isn’t going to suggest using these data attributes that may not even exist (until you add them yourself).
My conclusion from reading this study is that it confirms the validity of my suspicion that record and replay tools create fragile test suites.
By using a time-proven thoughtful approach, like defining page and component models with non-duplicated and least-brittle hand-picked element locators, you will realize long term benefits of automated test suite maintainability.