AMA: adopting a TDD/BDD approach

João Monteiro asks…

I recently joined a small company where I am the only QA and the test automation suite was already written in a given/when/then style but rarely (if not at all) gets read by the rest of the team (product or developers). Any tips on how to mentor the team to adopt a BDD approach? Do you recommend any tools / framework to share the features in a centralised place easily accessible by the rest of the team an not just on the tests repository?

Continue reading “AMA: adopting a TDD/BDD approach”

Where is the ‘story’ in user stories?

There’s an old Jerry Weinberg quote:

“no matter what the problem is, it’s always a people problem”

which pretty much describes every project I’ve worked on over the years. Lately, I’ve particularly noticed that most of the tough problems evident on projects are more people or business problems than technology problems, which makes me think it’s worthwhile for me to continue my exploration of the business/user end of my list of software development roles.

BA = Business Analyst
UX = User Experience
ET = Exploratory Tester
TT = Technical Tester
Dev = Software Developer

In this vein, I’ve recently been trying to understand how to better articulate user stories, in that one day I’d love to work as a business analyst.

Most nights I read stories to my (almost) three year-old son as a nice way to end the day. Lately I have been making up my own impromptu stories to keep things interesting. I have really enjoyed making up stories on the spot; I think I’d be a good BA.

But thinking about user stories along with bedtime stories immediately raises a question: where is the ‘story’ in user stories?

Most user stories at work sound something like this: “As a user, I want to log onto the system, so that I can access my information”. What a shitty story! Sorry, but seriously, if I told this story to my two year old son, he’d die of boredom!

I’ve spent a fair amount of time reading about user stories but I still can’t find out why they’re actually called stories, because I don’t think they are actual stories:

sto·ry/ˈstôrē/
Noun:
An account of imaginary or real people and events told for entertainment: “an adventure story”.

The closest thing I have found to actual user stories is the concept of ‘soap opera‘ testing scenarios which outline implausible yet possible scenarios:

“A man (Chris Patterson) and his wife (Chris Patterson) want to take their kids (from previous marriages, Chris Patterson, a boy, and Chris Patterson, a girl) on a flight from San Francisco to Glasgow to San Jose (Costa Rica) to San Jose (California) back to San Francisco. He searches for flights by schedule. He’s a Super-Elite-Premium frequent flyer, but he doesn’t want the upgrade that the system automatically grants him so that he can sit with his wife and kids in economy class. He requires a kosher meal, his wife is halal, the boy is a vegetarian, and the girl is allergic to wheat. He has four pieces of luggage per person (including two pairs of skis, three sets of golf clubs, two 120 lb. dogs, and three overweight suitcases), where his frequent flyer plan allows him (but only him) to take up to four checked items, but the others can only take two each. He gets to the last step on the payment page before he realizes that he has confused San Jose (California) for San Jose (Costa Rica), so the order of the itinerary is wrong. The airline cancels the flight after it has accepted his bags, and reroutes him on a partner. The partner cancels the flight (after it has accepted the bags) to San Jose (California) so it reroutes him to another competitor, who cancels the flight (after accepting the bags) to San Jose (Costa Rica) and reroutes him to another partner, who goes bankrupt after it has accepted the bags for the San Francisco flight.”

~ Michael Bolton

Now that’s a real user story!

So, I think we have two choices on the user stories front. We can either make our user stories actually like real juicy stories, or at least start calling them something else!

The color of acceptance is gray

James Shore recently wrote some brillant words about acceptance testing:

I think “acceptance” is actually a nuanced problem that is fuzzy, social, and negotiable. Using tests to mediate this problem is a bad idea, in my opinion. I’d rather see “acceptance” be done through face-to-face conversations before, after, and during development of code, centering around whiteboard sketches (earlier) and manual demonstrations (later) rather than automated tests.

To rephrase: “acceptance” should be a conversation, and it’s one that we should allow to grow and change as the customer sees the software and refines her understanding of what she wants. Testing is too limited, and too rigid. Asking customers to read and write acceptance tests is a poor use of their time, skill, and inclinations.

This is pretty much where my head is at right now around automating acceptance tests. Automated tests are black and white, acceptance is gray.

“The color of truth is gray.”
~ André Gide

I prefer to have a handful of end to end automated functional tests that cover the typical journey of a user than a large set of acceptance tests constantly in a state of flux as the system is being developed and acceptance is being defined and changed.

We need to take feedback from the customer that we are building the right thing and ensure our automated tests model this, not make them responsible for specifying the actual tests.

CukeSalad: not so yummy yummy

I’ve been playing around with CukeSalad: Cucumber specs / tests without step-definitions:

Cucumber, washed and ready to eat for Friction-free ATDD/BDD

In creating an example of my simple WatirMelonCucumber framework using CukeSalad, here’s some of the things I’ve found. Please note: these are my personal observations and are not direct criticisms of the huge effort that has gone into making this tool.

It’s not really no step definitions: they’re just called different things. Instead of writing steps, you’re writing roles and tasks, both of which are mandatory, so I found I actually wrote more code (in more files) than if I just wrote step definitions in the first place.

CukeSalad forces you to use predefined step syntax: which to me defeats the purpose of having features in business language. For example, as far as I can tell, Given statements are purely used for setting the roles, and therefore a business person can’t use it to define a precondition which I would normally use Given statements for. Another example is when you want to capture some data from the test, this must be on the end of a sentence following a semicolon (or comma)

When I search for: the phrase 'Watir'

I spent ages trying to work out why steps don’t match: as you still use Cucumber to run your features, Cucumber will only tell you that there’s no matching steps. This doesn’t help when there’s no steps whatsoever! At some point, I realized that Then steps follow a different syntax, and they can’t use placeholders like Whens, but must have an expected value on the end in single quotes.

Then I should see results greater than '30,000'

It’s very confusing what you’re actually asserting on a Then step: I couldn’t work out a way to capture the expected value from the feature (in the example above ‘30,000’), and CukeSalad just asserts the task returns the actual expected result, which doesn’t help if you’re not testing for equality!

It’s very confusing trying to convert an existing Cucumber test suite: I wouldn’t bother. I found I had to completely rewrite my feature to comply to the syntactical requirements of CukeSalad, so I can’t this working on an existing feature set.

Cucumber world is not supported: so just do a normal include instead in env.rb.

Summary

Sadly, CukeSalad doesn’t offer me anything over using Cucumber. Whilst it sells itself on having no step definitions, I found there was greater effort in writing the roles and tasks. I don’t like the way it forces me to write sentences using somewhat awkward constraints, and how Given, When, & Then statements all behave differently. It certainly gave me a lot of WTF moments, which isn’t what you want when writing automated tests.

I’d be very interested to hear whether anyone has had ongoing success with CukeSalad.

Turnip: trying to solve Cucumber’s problems with ruby BDD

I’ve been playing around with Turnip: a library that was designed to solve Cucumber’s problems in four main areas:

  1. Having a separate test framework is annoying
  2. Mapping steps to regexps is hard
  3. Cucumber has a huge, messy codebase
  4. Steps are always global

I don’t really care about having a Cucumber test framework as I don’t often use RSpec, I actually find regexen quite powerful, I also don’t really care about Cucumber’s code base but I can see some merit in having scoped steps.

I converted my existing demo test suite across to use Turnip and this is what I found.

Turnip follows Spinach in trying to move away from using regular expressions to define steps. Fortunately, unlike Spinach, Turnip allows you to capture values using placeholders in your steps (this was an instant NO to using Spinach):

  step "I should see at least :exp_num_results results" do |exp_num_results|
    on :results do |page|
      page.number_search_results.should >= exp_num_results.to_i
    end
  end

Unfortunately there are a number of limitations to this approach:

  • You can’t capture more than a single word in a placeholder without using double quotes in your feature file – making them less readable; and
  • You can’t capture values like “100,000” as it doesn’t appear to support special characters in placeholders.

You can also do some magic using custom step placeholders, where you define regular expressions, but to me that kinda defeats the point of not using regexen in the first place.

The scoped steps are quite neat and ruby like, and I can see these being very useful for large projects where multiple people are writing the steps.

Running Turnip files is as simple as using the RSpec test runner, but unfortunately the output of pending steps is less useful than Cucumber in that there are no code snippets, and only the first undefined step is displayed as undefined (unlike the Cucumber which shows you every step that is undefined).

Summary

I like the approach that Turnip takes in being a simple way to write business like tests and run them using RSpec. Unfortunately that simplicity comes at a price, and losing the power of regexen in step definitions is not something I would like to give up hastily. I am hoping that the scoped steps someday make their way into the Cucumber code base.

WatirMelon Spinach

Hello Spinach!

“Spinach is a new awesome BDD framework that features encapsulation and modularity of your step definitions.”

Seems like a good idea, so I thought I’d give it a try, and convert my existing simple Watirmelon Cucumber tests to use Spinach instead (link to my source code). It was very easy to do, here’s some observations:

  • It’s easy to get existing Cukes running using Spinach, but I imagine if you were starting out using Spinach you’d design things a lot differently
  • Steps belong in their own class, but you can include mixins to reuse steps – the ruby way
  • Steps are in a steps directory under features – shorter than step_definitions
  • Goodbye regular expressions in step definitions, which is a bit of a shame, as you can no longer capture values from the step name
  • As you can’t have regular expressions in step names, I find myself repeating steps that are similar but slightly different, this means my example steps have gone from 41 to 57 lines of code
  • Scenario Outlines aren’t supported at the moment, but I have raised this as a feature request
  • Hooks are dramatically improved, so I found them very easy to use and understand
  • There is no cucumber world, so you do a normal include instead, and env.rb is still supported
  • It displays really cool ticks on the command line when you’re running your spins

Well done to Codegram for releasing Spinach. If anything, it creates some great innovations that I imagine may find their way into Cucumber in the future.

Creating a Watir framework using Test::Unit & Roo

One common challenge I see over and over again is people figuring out how to design a logical and maintainable automated testing framework. I have designed quite a few frameworks for various projects, but one thing that has consistently been a win for me is purposely separating test case and test execution design.

It’s therefore logical that the design of my Watir framework deliberately separates test case design and test execution design so that:

  • test case design is done visually in spreadsheets; and
  • test execution design is done in ruby methods, because code is the most efficient and maintainable way.

Since I last published details about my framework on this blog, I have started doing assertions using the Test::Unit ruby library. The reasons I chose Test::Unit are:

  • it is easy to ‘mix-in’ Test::Unit assertions into modules of ruby code using include Test::Unit::Assertions;
  • it is included with ruby;
  • ruby scripts with Test::Unit::TestCase are instantly executable, in my case, from SciTE;
  • its assertions are easy to understand and use.

I have also made some other improvements to my framework code, including:

  • the ability to specify browser types, and spreadsheet sources, as command line arguments (with defaults);
  • logging test output to a file;
  • no longer attaching to an open browser, the same browser instance is used completely for all tests (and elegantly closed at the end).

The main design has been kept the same, in that a spreadsheet (either excel, openoffice or Google Docs) contains tests grouped by functional area, which call a method in a particular module.

The great thing about my framework is that adding a new test is a matter of designing the test case, and then writing the ruby method: as the methods are called dynamically from the spreadsheet, no extra glue is needed!

Enough talk, here’s the code. The Google spreadsheet is here. You can find a .zip file of all the required files to run it here. It runs on the depot app, which you get here. You will need two gems: Watir (oh duh), and Roo.

Test Driver tc_main.rb


$:.unshift File.join(File.dirname(__FILE__), ".", "lib")
require 'watir'
require 'roo'
require 'test/unit'
require 'customer'
require 'admin'
$stdout = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)
$stderr = File.new('log.txt',File::WRONLY|File::APPEND|File::CREAT)

class TC_WatirMelon < Test::Unit::TestCase
  @@colmap = {:module_name=>0, :method_name=>1, :comments=>2, :exp_outcome=>3, :exp_error=>4, :first_param=>5}
  @@ss_format = ARGV[0]
  @@specified_browser = ARGV[1]

  def setup
    puts "[Starting at #{Time.now}]\n"
    case @@ss_format
      when "excel"
        @ss = Excel.new("watirmelon.xls")
      when "wiki"
        @ss = Excel.new("http://localhost:8080/download/attachments/2097153/watirmelon.xls")
      when "gdocs"
        @ss = Google.new("0AtL3mPY2rEqmdEY3XzRqUlZKSmM5Z3EtM21UdFdqb1E")
      else
        @ss = Openoffice.new("watirmelon.ods")
      end
    @ss.default_sheet = @ss.sheets.first
    case @@specified_browser
      when "firefox"
        Watir::Browser.default = 'firefox'
        @browser = Watir::Browser.new
      else
        Watir::Browser.default = 'ie'
        @browser = Watir::Browser.new
        @browser.speed = :zippy
        @browser.visible = true
      end
  end

  def test_run_sheet()
    @ss.first_row.upto(@ss.last_row) do |row|
      #Read row into array
      line = Array.new
      @ss.first_column.upto(@ss.last_column) do |column|
        line << @ss.cell(row, column).to_s.strip
      end

      module_name = line[@@colmap[:module_name]]
      if module_name != "Function" then #if not a header
        method_name = line[@@colmap[:method_name]].downcase.gsub(' ','_') #automatically determine ruby method name based upon data sheet
        exp_outcome = line[@@colmap[:exp_outcome]]
        exp_error = line[@@colmap[:exp_error]]
        first_param = @@colmap[:first_param]
        required_module = Kernel.const_get(module_name)
        required_method = required_module.method(method_name)
        arity = required_method.arity() # this is how many arguments the method requires, it is negative if a 'catch all' is supplied.
        arity = ((arity * -1) - 1) if arity < 0 # arity is negative when there is a 'catch all'
        arity = arity-1 # Ignore the first browser parameter
        unless arity == 0
          parameters = line[first_param..first_param+(arity-1)]
        else
          parameters = []
        end
        begin
          act_outcome, act_output = required_method.call(@browser, *parameters)
        rescue Test::Unit::AssertionFailedError => e
          self.send(:add_failure, e.message, e.backtrace)
          act_outcome = false
          act_output = e.message
        end
        if (exp_outcome == 'Success') and act_outcome then
          assert(true, "Expected outcome and actual outcome are the same")
          result = 'PASS'
        elsif (exp_outcome == 'Error') and (not act_outcome) and (exp_error.strip! == act_output.strip!)
          assert(true, "Expected outcome and actual outcome are the same, and error messages match")
          result = 'PASS'
        else
          result = 'FAIL'
          begin
            assert(false,"Row: #{row}: Expected outcome and actual outcome for #{method_name} for #{module_name} do not match, or error messages do not match.")
          rescue Test::Unit::AssertionFailedError => e
            self.send(:add_failure, e.message, e.backtrace)
          end
        end
        puts "###########################################"
        puts "[Running: #{module_name}.#{method_name}]"
        puts "[Expected Outcome: #{exp_outcome}]"
        puts "[Expected Error: #{exp_error}]"
        puts "[Actual Outcome: Success]" if act_outcome
        puts "[Actual Outcome: Error]" if not act_outcome
        puts "[Actual Output: #{act_output}]"
        puts "[RESULT: #{result}]"
        puts "###########################################"
        end
      end
  end

  def teardown
    @browser.close
    puts "[Finishing at #{Time.now}]\n\n"
  end

end

Customer Module customer.rb

require 'test/unit'
include Test::Unit::Assertions

module Customer

  TITLE = 'Pragprog Books Online Store'
  URL = 'http://localhost:3000/store/'

  # Description:: Adds a book named 'book_title' to cart
  def Customer.add_book(browser, book_title)
    browser.goto(URL)
    # Check if title is already in cart - so we can check it was added correctly
    browser.link(:text,'Show my cart').click
    prev_cart_count = 0
    prev_cart_total = 0.00
    if not browser.div(:text,'Your cart is currently empty').exist? then
     # We have a non-empty cart
      for row in browser.table(:index,1)
        if row[2].text == book_title then
          prev_cart_count = row[1].text.to_i
          break
        end
      end
      prev_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f #remove $ sign
      browser.link(:text, 'Continue shopping').click
    end

    found = false
    book_price = 0.00
    1.upto(browser.divs.length) do |index|
      if (browser.div(:index,index).attribute_value('className') == 'catalogentry') and (browser.div(:index,index).h3(:text,book_title).exists?) then
        book_price = browser.div(:index,index).span(:class, 'catalogprice').text[1..-1].to_f #remove $ sign
        browser.div(:index,index).link(:class,'addtocart').click
        found = true
        break
      end
    end
    if not found then
      return false,'Could not locate title in store'
    end

    new_cart_count = 0
    for row in browser.table(:index,1)
      if row[2].text == book_title then
        new_cart_count = row[1].text.to_i
        break
      end
    end
    new_cart_total = browser.cell(:id, 'totalcell').text[1..-1].to_f # remove $ sign
    assert_equal(new_cart_count,(prev_cart_count+1), "Ensure that new quantity is now one greater than previously")
    assert_equal(new_cart_total,(prev_cart_total + book_price), "Ensure that new cart total is old cart total plus book price")
    browser.link(:text, 'Continue shopping').click
    return true,new_cart_total
  end

  def Customer.check_out(browser, customerName, customerEmail, customerAddress, customerPaymentMethod)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    browser.link(:text,"Checkout").click
    browser.text_field(:id, 'order_name').set(customerName)
    browser.text_field(:id, 'order_email').set(customerEmail)
    browser.text_field(:id, 'order_address').set(customerAddress)
    begin
      browser.select_list(:id, 'order_pay_type').select(customerPaymentMethod)
    rescue Watir::Exception::NoValueFoundException
      flunk('Could not locate customer payment method in drop down list: '+customerPaymentMethod)
    end
    browser.button(:name, 'commit').click
    if browser.div(:id,'errorExplanation').exist? then
      error = ''
      1.upto(browser.div(:id,'errorExplanation').lis.length) do |index|
        error << (browser.div(:id,'errorExplanation').li(:index,index).text + ",")
      end
      browser.link(:text,'Continue shopping').click
      return false, error
    end
    assert_equal(browser.div(:id,'notice').text, 'Thank you for your order.',"Thank you for your order should appear.")
    return true,''
  end

  def Customer.empty_cart(browser)
    browser.goto(URL)
    browser.link(:text,"Show my cart").click
    if browser.div(:text,"Your cart is currently empty").exist? then
      assert('Cart was never empty')
    else
      browser.link(:text,'Empty cart').click
      assert_equal(browser.div(:id, 'notice').text,'Your cart is now empty')
    end
    return true,''
  end

  def Customer.check_cart_total(browser, exp_total)
    browser.goto(URL)
    browser.link(:text,'Show my cart').click
    if browser.div(:text,'Your cart is currently empty').exist? then
      return false,'Your cart is currently empty'
    end
    act_total = browser.cell(:id, 'totalcell').text[1..-1].to_f
    assert_equal(act_total,exp_total.to_f,"Check that cart total is as expected.")
    return true,act_total
  end
end

Admin Module admin.rb


require 'test/unit'
include Test::Unit::Assertions

module Admin
  TITLE = 'ADMINISTER Pragprog Books Online Store'
  URL = 'http://localhost:3000/admin/'

  def Admin.log_on(browser, username, password)
    browser.goto(URL)
    if browser.link(:text,'Log out').exist? then #if already logged in
      browser.link(:text,'Log out').click
    end
    browser.text_field(:id, 'user_name').set username
    browser.text_field(:id, 'user_password').set password
    browser.button(:value, ' LOGIN ').click
    if browser.div(:id, 'notice').exist? then
      return false,browser.div(:id, 'notice').text
    else
      return true,''
    end
  end

  def Admin.ship_items(browser, name)
    browser.goto(URL)
    browser.link(:text, 'Shipping').click
    num_orders = 0
    index = 0
    browser.form(:action,'/admin/ship').divs.each do |div|
      if div.class_name == "olname"
        index+=1
        if div.text == name then
          browser.form(:action,'/admin/ship').checkbox(:index, index).set
          num_orders+=1
        end
      end
    end

    browser.button(:value, ' SHIP CHECKED ITEMS ').click

    if num_orders == 1 then
      assert_equal(browser.div(:id,"notice").text, "One order marked as shipped","Correct notice")
    elsif num_orders > 1 then
      assert_equal(browser.div(:id,"notice").text, "#{num_orders} orders marked as shipped","Correct notice")
    end
    return true, num_orders.to_s
  end

end