Why I do automated testing

I often get asked the question: “so why do you do automated testing?” both in job interviews and from people I meet. My answer is always pretty much the same; being that it provides me with exposure to a variety of business domains and technologies, and allows me to write code.

Let me expand a bit further on this. I like writing code but I’ve chosen not to be a software engineer. The reason is that I have met lots of different software engineers that have responsibility for one area of a system. They become the expert in this area of the system, hence when you find a problem in that area, you know who to speak to. Often time these people don’t really know much outside their area of expertise in the system, so when the problem is outside their own area, you need to speak with someone else. There’s nothing wrong with this, but it just isn’t me.

I like to know, or attempt to ‘work out’, how the system functions as a whole. I usually achieve this on a project I join by writing an automated build verification test, or ‘smoke’ test, that verifies the system functions as a whole. I love writing these kinds of tests because you need to know how the entire thing works, not just the isolated components. Software testing is one of the few areas where you get this broad level of exposure, and this keeps things interesting.

Another added bonus of doing software testing, is that your skills are pretty transferable, both across business domains and different technologies. For example, so far in my career I have tested systems that:

  • manage job advertisements, job seeker matches and interviews;
  • process immigration and visa applications;
  • manage disability support and funding;
  • produce electronic school report cards;
  • optimise large scale open cut mines;
  • monitor health of heavy machinery; and
  • manage large funds for people’s retirements.

These systems were developed in various technologies including:

  • Java;
  • Microsoft .NET;
  • Cobol;
  • Powerbuilder;
  • Web;
  • Active X;
  • Terminal Emulators;
  • SOAP web services and;
  • CICS.

The point I am trying to make is that if I decided to be a Java Software Engineer when I finished university, because that’s why I studied, I wouldn’t have had exposure to so many technologies and business areas that I have had as an automated software tester.

Sure, I could become a business analyst, because they work across business domains and technologies, but business analysts don’t code, and I do like writing code, that’s why automated testing is perfect.

The final benefit I see in automated testing is that you can apply your skills in particular testing tools to other tools fairly easily, as the concepts are the same. The biggest issue I see with people doing automated testing isn’t picking up a new tool, but rather developing a maintainable and easy to use test framework. Once you have developed a few frameworks for various business domains and technologies, it is easy to apply the concepts to other testing tools.

So that’s why I do automated testing, and I proud of it.

Creating a Watir framework using Test::Unit & Roo

One common challenge I see over and over again is people figuring out how to design a logical and maintainable automated testing framework. I have designed quite a few frameworks for various projects, but one thing that has consistently been a win for me is purposely separating test case and test execution design.

It’s therefore logical that the design of my Watir framework deliberately separates test case design and test execution design so that:

  • test case design is done visually in spreadsheets; and
  • test execution design is done in ruby methods, because code is the most efficient and maintainable way.

Since I last published details about my framework on this blog, I have started doing assertions using the Test::Unit ruby library. The reasons I chose Test::Unit are:

  • it is easy to ‘mix-in’ Test::Unit assertions into modules of ruby code using include Test::Unit::Assertions;
  • it is included with ruby;
  • ruby scripts with Test::Unit::TestCase are instantly executable, in my case, from SciTE;
  • its assertions are easy to understand and use.

I have also made some other improvements to my framework code, including:

  • the ability to specify browser types, and spreadsheet sources, as command line arguments (with defaults);
  • logging test output to a file;
  • no longer attaching to an open browser, the same browser instance is used completely for all tests (and elegantly closed at the end).

The main design has been kept the same, in that a spreadsheet (either excel, openoffice or Google Docs) contains tests grouped by functional area, which call a method in a particular module.

The great thing about my framework is that adding a new test is a matter of designing the test case, and then writing the ruby method: as the methods are called dynamically from the spreadsheet, no extra glue is needed!

Enough talk, here’s the code. The Google spreadsheet is here. You can find a .zip file of all the required files to run it here. It runs on the depot app, which you get here. You will need two gems: Watir (oh duh), and Roo.

Test Driver tc_main.rb


$:.unshift File.join(File.dirname(__FILE__), ".", "lib")
require 'watir'
require 'roo'
require 'test/unit'
require 'customer'
require 'admin'
$stdout = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)
$stderr = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)

class TC_WatirMelon < Test::Unit::TestCase
  @@colmap = {:module_name=>0, :method_name=>1, :comments=>2, :exp_outcome=>3, :exp_error=>4, :first_param=>5}
  @@ss_format = ARGV[0]
  @@specified_browser = ARGV[1]

  def setup
    puts "[Starting at #{Time.now}]\n"
    case @@ss_format
      when "excel"
        @ss = Excel.new("watirmelon.xls")
      when "wiki"
        @ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
      when "gdocs"
        @ss = Google.new("0AtL3mPY2rEqmdEY3XzRqUlZKSmM5Z3EtM21UdFdqb1E")
      else
        @ss = Openoffice.new("watirmelon.ods")
      end
    @ss.default_sheet = @ss.sheets.first
    case @@specified_browser
      when "firefox"
        Watir::Browser.default = 'firefox'
        @browser = Watir::Browser.new
      else
        Watir::Browser.default = 'ie'
        @browser = Watir::Browser.new
        @browser.speed = :zippy
        @browser.visible = true
      end
  end

  def test_run_sheet()
    @ss.first_row.upto(@ss.last_row) do |row|
      #Read row into array
      line = Array.new
      @ss.first_column.upto(@ss.last_column) do |column|
        line << @ss.cell(row, column).to_s.strip
      end

      module_name = line[@@colmap[:module_name]]
      if module_name != "Function" then #if not a header
        method_name = line[@@colmap[:method_name]].downcase.gsub(' ','_') #automatically determine ruby method name based upon data sheet
        exp_outcome = line[@@colmap[:exp_outcome]]
        exp_error = line[@@colmap[:exp_error]]
        first_param = @@colmap[:first_param]
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0 # arity is negative when there is a 'catch all'
        arity = arity-1 # Ignore the first browser parameter
        unless arity == 0
          parameters = line[first_param..first_param+(arity-1)]
        else
          parameters = []
        end
        begin
          act_outcome, act_output = required_method.call(@browser, *parameters)
        rescue Test::Unit::AssertionFailedError => e
          self.send(:add_failure, e.message, e.backtrace)
          act_outcome = false
          act_output = e.message
        end
        if (exp_outcome == 'Success') and act_outcome then
          assert(true, "Expected outcome and actual outcome are the same")
          result = 'PASS'
        elsif (exp_outcome == 'Error') and (not act_outcome) and (exp_error.strip! == act_output.strip!)
          assert(true, "Expected outcome and actual outcome are the same, and error messages match")
          result = 'PASS'
        else
          result = 'FAIL'
          begin
            assert(false,"Row: #{row}: Expected outcome and actual outcome for #{method_name} for #{module_name} do not match, or error messages do not match.")
          rescue Test::Unit::AssertionFailedError => e
            self.send(:add_failure, e.message, e.backtrace)
          end
        end
        puts "###########################################"
        puts "[Running: #{module_name}.#{method_name}]"
        puts "[Expected Outcome: #{exp_outcome}]"
        puts "[Expected Error: #{exp_error}]"
        puts "[Actual Outcome: Success]" if act_outcome
        puts "[Actual Outcome: Error]" if not act_outcome
        puts "[Actual Output: #{act_output}]"
        puts "[RESULT: #{result}]"
        puts "###########################################"
        end
      end
  end

  def teardown
    @browser.close
    puts "[Finishing at #{Time.now}]\n\n"
  end

end

Customer Module customer.rb

require 'test/unit'
include Test::Unit::Assertions

module Customer

  TITLE = 'Pragprog Books Online Store'
  URL = 'http://localhost:3000/store/'

  # Description:: Adds a book named 'book_title' to cart
  def Customer.add_book(browser, book_title)
    browser.goto(URL)
    # Check if title is already in cart - so we can check it was added correctly
    browser.link(:text,'Show my cart').click
    prev_cart_count = 0
    prev_cart_total = 0.00
    if not browser.div(:text,'Your cart is currently empty').exist? then
     # We have a non-empty cart
      for row in browser.table(:index,1)
        if row[2].text == book_title then
          prev_cart_count = row[1].text.to_i
          break
        end
      end
      prev_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f #remove $ sign
      browser.link(:text, 'Continue shopping').click
    end

    found = false
    book_price = 0.00
    1.upto(browser.divs.length) do |index|
      if (browser.div(:index,index).attribute_value('className') == 'catalogentry') and (browser.div(:index,index).h3(:text,book_title).exists?) then
        book_price = browser.div(:index,index).span(:class, 'catalogprice').text[1..-1].to_f #remove $ sign
        browser.div(:index,index).link(:class,'addtocart').click
        found = true
        break
      end
    end
    if not found then
      return false,'Could not locate title in store'
    end

    new_cart_count = 0
    for row in browser.table(:index,1)
      if row[2].text == book_title then
        new_cart_count = row[1].text.to_i
        break
      end
    end
    new_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f # remove $ sign
    assert_equal(new_cart_count,(prev_cart_count+1), "Ensure that new quantity is now one greater than previously")
    assert_equal(new_cart_total,(prev_cart_total + book_price), "Ensure that new cart total is old cart total plus book price")
    browser.link(:text, 'Continue shopping').click
    return true,new_cart_total
  end

  def Customer.check_out(browser, customerName, customerEmail, customerAddress, customerPaymentMethod)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    browser.link(:text,"Checkout").click
    browser.text_field(:id, 'order_name').set(customerName)
    browser.text_field(:id, 'order_email').set(customerEmail)
    browser.text_field(:id, 'order_address').set(customerAddress)
    begin
      browser.select_list(:id, 'order_pay_type').select(customerPaymentMethod)
    rescue Watir::Exception::NoValueFoundException
      flunk('Could not locate customer payment method in drop down list: '+customerPaymentMethod)
    end
    browser.button(:name, 'commit').click
    if browser.div(:id,'errorExplanation').exist? then
      error = ''
      1.upto(browser.div(:id,'errorExplanation').lis.length) do |index|
        error << (browser.div(:id,'errorExplanation').li(:index,index).text + ",")
      end
      browser.link(:text,'Continue shopping').click
      return false, error
    end
    assert_equal(browser.div(:id,'notice').text, 'Thank you for your order.',"Thank you for your order should appear.")
    return true,''
  end

  def Customer.empty_cart(browser)
    browser.goto(URL)
    browser.link(:text,"Show my cart").click
    if browser.div(:text,"Your cart is currently empty").exist? then
      assert('Cart was never empty')
    else
      browser.link(:text,'Empty cart').click
      assert_equal(browser.div(:id, 'notice').text,'Your cart is now empty')
    end
    return true,''
  end

  def Customer.check_cart_total(browser, exp_total)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    act_total = browser.cell(:id, 'totalcell').text[1..-1].to_f
    assert_equal(act_total,exp_total.to_f,"Check that cart total is as expected.")
    return true,act_total
  end
end

Admin Module admin.rb


require 'test/unit'
include Test::Unit::Assertions

module Admin
  TITLE = 'ADMINISTER Pragprog Books Online Store'
  URL = 'http://localhost:3000/admin/'

  def Admin.log_on(browser, username, password)
    browser.goto(URL)
    if browser.link(:text,'Log out').exist? then #if already logged in
      browser.link(:text,'Log out').click
    end
    browser.text_field(:id, 'user_name').set username
    browser.text_field(:id, 'user_password').set password
    browser.button(:value, ' LOGIN ').click
    if browser.div(:id, 'notice').exist? then
      return false,browser.div(:id, 'notice').text
    else
      return true,''
    end
  end

  def Admin.ship_items(browser, name)
    browser.goto(URL)
    browser.link(:text, 'Shipping').click
    num_orders = 0
    index = 0
    browser.form(:action,'/admin/ship').divs.each do |div|
      if div.class_name == "olname"
        index+=1
        if div.text == name then
          browser.form(:action,'/admin/ship').checkbox(:index, index).set
          num_orders+=1
        end
      end
    end

    browser.button(:value, ' SHIP CHECKED ITEMS ').click

    if num_orders == 1 then
      assert_equal(browser.div(:id,"notice").text, "One order marked as shipped","Correct notice")
    elsif num_orders > 1 then
      assert_equal(browser.div(:id,"notice").text, "#{num_orders} orders marked as shipped","Correct notice")
    end
    return true, num_orders.to_s
  end

end

Celerity: first impressions

Last week at the Test Automation Workshop (TAW) here in Australia, the topic of being able to run automated tests ‘headless’ came up, and I mentioned the Celerity project: a headless Watir port. I decided to have a play with Celerity tonight to see how easy it is to get up and running, and also look at it as a way to quickly run Watir scripts.

Installation

The installation was fairly straightforward. You need a Java 6 JDK, as well as the JRuby binaries. You’ll need to update two environment variables, namely add JRuby to your path, and update your JAVA_HOME variable. Installing Celerity involves a JRuby gem: "jruby -S gem install celerity". You can also install it from github, but I don’t know the difference between the two.

Availability of Gems

Ruby gems are supported in JRuby, but only if they don’t use C libraries. This means that Watir won’t work, nor will Roo. I use Roo to define test cases in spreadsheets, so it means I can’t do a comparison of execution times at the moment, at least until I get these test cases out of spreadsheets.

Using Celerity as a simple load testing tool instead

At TAW, Kelvin Ross thought Celerity sounded promising as a load testing tool, considering it was headless and lightweight. I thought I would give a simple google search script a try, running in both Celerity, and Watir, with 30 concurrent users.

Running under Watir

CPU peaked at 100% for the entire run, and each page varied considerable but took on average 10 seconds to load.

Running under Celerity

CPU use was normal, and each page took just over 1 second to load (minimal variance).

Conclusion

Celerity seems a promising way to execute basic load tests using a headless browser. The benefit of Celerity is support for javascript execution in the browser, the downside at the moment is lack of support for some ruby gems. If you could run Watir under JRuby, I could have used a single script.

Scripts Used
Celerity Script

require 'thread'
require "rubygems"
require "celerity"

def test_google
  browser = Celerity::Browser.new
  browser.goto('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}

Watir Script

require 'thread'
require 'watir'
require 'watir/ie'

def test_google
  browser = Watir::IE.start('http://www.google.com')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time = Time.now
  browser.button(:name, 'btnG').click
  end_time = Time.now
  puts end_time - start_time
  browser.close
end

threads = []
30.times do
  threads << Thread.new {test_google}
end
threads.each {|x| x.join}

Australian Test Automation Workshop Slides

Here are some slides I presented yesterday at the Australian Test Automation Workshop, specifically on building an automated test framework using Watir.

An example data sheet can be viewed on Google Docs.

Version control your automated tests, quickly, easily, today for free

Why don’t testers version control their tests?

I am still surprised at how many organizations don’t version control their automated test scripts. I put it down to the following reasons:

  • Developers may use expensive version control tools, but sometimes there aren’t enough licenses for testers;
  • People don’t realize there are free version control tools available;
  • Setting up version control might be considered too difficult for the test team;
  • Some people believe you need to own a version control server to version your test scripts; or
  • Any combination of the above.

In reality:

  • If the software developers’ version control system is available for testers great, but if not, test scripts can be versioned separately;
  • There are many different free version control tools available. TortoiseSVN, which uses the Subversion (SVN) protocol, is very popular and very easy to use;
  • Setting up a new SVN repository using TortoiseSVN only takes a few minutes; and
  • You can set up a SVN repository on a shared network drive, so you don’t need a server (but a server is cool).

How to quickly set up a new SVN respository on a shared network drive (using Windows)

If you haven’t version controlled your test scripts yet, here’s how to do so.

  1. Download and install TortoiseSVN from http://tortoisesvn.tigris.org/ (it’s about 19MB, and requires a reboot: bummer :( )
  2. Find a location on a shared network drive where you can store your SVN repository. For example, it could be Q:\SVN  Repositories , create a new directory for your repository (eg. Q:\SVN  Repositories\WatirMelon\ and right click within this new directory in Windows Explorer, and choose TortoiseSVN, and then ‘create repository here’. The path to your new directory will be your SVN repository path. Create SVN Repository Here
  3. The repository should be created in a matter of seconds, and filled with some directories and files. These files/directories should never be touched, under any circumstances. Repository created successfully
  4. Now you need to create a local repository and check out the new repository (which will be blank initially). Create a directory on your local drive for your repository, for example C:\watirmelon, and right click within this directory and choose ‘SVN Checkout’. SVN Checkout
  5. You will need to specify the location of your repository you created in Step 2, but importantly you will need to add file:/// to the front, and change the backslashes into forward slashes. Checkout Dialog
  6. Once you click OK you have a repository (albeit blank) checked out. You would then simply add all your automated test scripts, then do an ‘SVN Add’, and ‘SVN commit’. If you want to use your automated tests on another machine, you simply checkout the repository following steps 4 & 5 above.

Conclusion

So there it is. Now there’s really no excuse not to version control your automated tests, considering it’s free, quick, easy and doesn’t require a server. So go and do it now (if you haven’t already).

Australian Test Automation Workshop (TAW) 2009

TAW 2009 is coming up on August 27 & 28 and I have already confirmed my attendance (GTAC is a very long flight!) and created a LinkedIn event. TAW is held every year at Bond University on the Gold Coast in Australia.

I did a quick presentation last year, and I think I might do something a bit different this year.

You can download the presentation in full here.

My ANZTB SIGIST Watir Slides

It’s been a while since I presented at the Brisbane ANZTB SIGIST but here are my slides: note no bullets (as usual).

(it’s a shame you can’t embed Google Docs presentations here yet)