The value of automated testing

I recently saw an email thread discussing the value of test automation and asking how to justify it to management. It just seems such a strange concept to me. Justifying the value of something that is so essential to writing good quality software, is like justifying the value of exercise to a human body: it’s benefits are so obvious it’s almost a waste of time attempting to justify it.

But the main reason I find justifying the value of automated testing so strange is I don’t really see any viable alternative. It’s like justifying to your boss the benefit of flying from Australia to the USA and back for a conference (~30 hours return) instead of taking a ship (50-80 days).

If we have a critical bug in our production system we need to turn around a fix in less than an hour. To turnaround a fix in less than an hour we need full regression test coverage that can give us feedback that we’re all good and haven’t broken anything else in half that time. To get the same coverage that our automated regression test suite has through 30 minutes of manual test execution would require having about 184 QA staff always available to do regression testing in parallel: that’s not going to happen.

Unless you can wait days/weeks for manual regression testing, or have a sufficiently large team of QA resources always available to test, you really can’t release software quickly with confidence that it’s high quality. With automated tests you can; and that’s the value of automated testing.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

8 thoughts on “The value of automated testing”

  1. I haven’t seen the discussion, but I personally stumble upon a similar problem – the “return on investment”. Managers want everything quantified, so even if automation is essential, many people struggle with providing estimates (me too :( ) and therefore the automation is never implemented.

    Have you ever been in such situation? How do you estimate automation in a new company/ for a new project where you have no other estimates to rely on?

    PS: Huge fan of your blog, I’ve been using Selenium WebDriver with java and C# but watir made it much easier to create POCs and maintain frameworks for fast agile projects :)

    Like

    1. I don’t estimate automation separately for a new project, it’s included in the estimate to build and test a user story. The developer is resposible for not only developing new functionality but also developing appropriate level automated tests for that functionality.

      Like

  2. “Unless you can wait days/weeks for manual regression testing, or have a sufficiently large team of QA resources always available to test, you really can’t release software quickly with confidence that it’s high quality. With automated tests you can; and that’s the value of automated testing.”
    Automated regression does not give confidence of high quality software, it is more of “an illusion of confidence”. 100% passed automated regression tests does not equal high quality software, but adding some human testers to it can give you good (or better) software. Automation is more about reducing the risk, but there still is some risk that your automation misses something.

    Like

    1. I am not sure what you’re proposing Rafal. We have human testers (2), but there’s no way they can do manual regression testing within half an hour. And they shouldn’t have to.

      Like

      1. What i wanted to say is that machines are not perfect (and as long as humans are making them, they will never be perfect). That is why it is good to have someone to just test the software manually and look for issues that machine could have missed (even if it is for 0,5-1 hour).

        One time i had automatic regression miss an issue with sorting content. Automation was checking if items are sorted in HTML (which was correct order), but we had an algorithm to position those on page (to get 3 items per row). Issue was that at one point position was calculated incorrectly and instead of filling the row from left to right, it placed 1st item in the middle, 2nd item on the right and 3rd item on the left. Machine couldn’t see the issue, but few minutes of human tester just looking around the application found it.

        Like

        1. Sounds like you had missing or incomplete automated tests, plain and simple. I always extend such testsuites afterwards with the missing tests, to begin closing the blind spots and bring up their quality.

          Another great technique to verify that you have high quality automated tests is called “mutation analysis”. It’s a very effective way to *test your tests* and I talk about it here: http://ortask.com/why-your-mindset-might-be-setting-your-software-for-failure/

          Cheers,
          Mario G

          Like

  3. Well put. I have been dealing with the “cost benefit” (ROI) problem for a long time and have used different analogies to move away from the “bean counting” some people want to do. I typically say it is like having catastrophic event insurance, its there when you really need it. Or that how do you justify the cost of a developer and the time they spend building the application.

    Both has an upfront cost that can pay back in a very big way later on, it is a long term vision of the benefit of the work, not a short term one. Test automation is just like the software under test itself, it is fluid and unique to each situation. Sure there are certain repeatable methods/patterns to use to build it, but each time you have a unique solution particular to the situation/software at hand.

    Like

Comments are closed.