Not Ruining your Test Automation Strategy

I was very curious yesterday when I read a quote on twitter taken from a blog post by Uncle Bob Martin.

“The net result is that GUI driven automated tests are fragile & the process of maintaining them is expensive & unreliable” @unclebobmartin

I clicked through to the blog post but I wasn’t impressed with what else I saw.

“The bottom line is that automated tests through the GUI are inherently unstable, and will drive you to one or the other of those two undesirable states.”

I immediately thought, I have made automated GUI testing successful, so who’s Uncle Bob Martin to state all automated GUI testing is wrong.

Whilst I understand there are better ways of automated testing without using a GUI, I also understand that sometimes it’s not possible to test anything but the GUI, for example: poorly written, old, legacy systems.

But having to use the GUI doesn’t mean you can’t make a maintainable solution, I have this at many places, it just takes skill. I see what Bob thinks of this:

“Now, clearly, automated tools can be made to be clever enough to avoid simple cosmetic issues like the moving of a button, or a change in the spelling of a menu. But you have to work at making the tests tolerant of such changes. Do you think that outsourced team of test writers care much about that?”

Whether outsourced or not, the people creating automated tests do need to care about this. Somewhat still amazed at what I was reading, I had an aha moment when I read the following (emphasis added):

“What’s more the cost of re-recording the tests is high, and the re-recording process itself is error-prone.”

Aha! Bob might have only ever worked on a project where GUI automated testing has been done through record/playback. No wonder he thinks poorly of it.

So, to conclude, I would like to make the following points clear:

  • Automated testing through non-GUI means is smart, but sometimes you have no choice.
  • I have made automated testing through the GUI reliable and maintainable, but it required skill on my part.
  • Automated GUI tests can be used to deliberately show discrepancies in the GUI, often highlighting unintended GUI changes.
  • It’s generally not a good idea to completely write something off because you may have seen it done poorly yourself. It’s like saying Agile is wrong because you worked somewhere where Agile was done poorly.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

5 thoughts on “Not Ruining your Test Automation Strategy”

  1. Note also that Uncle Bob was really targeting the idea of outsourcing your test automation. The point being that it could be done better when the responsibility was taken as part of the development team.


  2. I read that article from Uncle Bob yesterday too and had a similar reaction to Alister. I hope most people would agree that record/playback has not shown a lot of promise for long term automation strategies. On the other hand writing good maintainable loosely coupled GUI automation code works great. We need to write and maintain solid code to test our products at all layers.


  3. My beef is not with GUI testing tools per se. Rather it is with teams that test their entire app through the GUI. You are correct in that sometimes you have no choice. In such cases very careful test construction can mitigate the fragility problem. But no amount of care can come close to competing with an approach that runs the majority of tests through the API.


Comments are closed.