Real vs Headless Browsers for Automated Acceptance Tests

When I was comparing/evaluating JavaScript browser automation tools for Automattic, I noticed that a lot of the tools were headless only: either they use PhantomJS/SlimerJS or have their own headless browser engine (like Zombie.js).

I must admit I am not a fan of headless browsers for a number of reasons:
Continue reading “Real vs Headless Browsers for Automated Acceptance Tests”

Watir-WebDriver with GhostDriver on OSX: headless browser testing

GhostDriver has been released which means it is now easy to run reliable headless WebDriver tests on Mac OSX.

Steps to get working on OSX

  1. First make sure you have homebrew installed
  2. Run
    brew update


    brew install phantomjs

    which should install PhantomJS 1.8.1 or newer

  3. Run irb and start using GhostDriver!
    require 'watir-webdriver'
    b = :phantomjs
    b.goto ""
    b.url #""
    b.title #"Google"

I’ve tested it on a large test suite (123 scenarios) and it behaves the same as other browsers with full JavaScript support. It took 8m13s in total: surprisingly it is slightly slower than ChromeDriver (7m30s) in my testing, but a little faster than the Firefox Driver (9m33s).

Well done to all involved in this project. It’s great to see a reliable, realistic headless browser with full JavaScript support for WebDriver finally released.

And yes, in case you’re wondering, it does screenshots!

Running Watir-WebDriver tests on Travis CI: a distributed build system

I recently came across Travis CI, a distributed build system that has close links to Github. I’ve seen quite a few projects use it as a CI system, but none that run headless browser tests.

Leveraging off the work I had done recently setting up my own Jenkins server in the cloud to run headless Watir-WebDriver tests, I thought I would have a go at running my WatirMelonCucumber and EtsyWatirWebDriver headless browser tests using Travis CI.

What I didn’t realize is how easy it’d be. The only things I had to do was make sure my Gemfile included rake, and also make sure there was some file existance checking happening for some log files, and it pretty much ran straight away. Wow!

Caveat Emptor

This is pretty new territory, so there’s a few things to watch out for:

  • Every now and, Travis complains about not having Firefox installed. I am not sure why this happens, maybe something to do with the different agents in use;
  • The locale of the build agents seems to be German, so when running my Google tests, the page content is in German, so it fails because it can’t find my expected (English) results; and
  • I can’t seem to capture my test results html file nor screenshot pngs, so it’s a red/green only affair at the moment.

But still, neat distributed free headless CI!

Running headless Watir-WebDriver tests using a real browser

I seem to get quite a few questions from people trying to use Watir-WebDriver in headless mode using a WebDriver server and HTMLUnit. It seems to me that it’s problematic, especially when your web app contains JavaScript (who’s app doesn’t?).

The problem stems from HTMLUnit’s JavaScript support. Whilst it does support JavaScript, myself and many others have found the JavsScript support to be pretty poor as it uses its own JavaScript engine (Rhino) that no other ‘real’ browser actually uses, which sorta raises a question: are you really testing your app’s JavaScript?

The other thing about HTMLUnit is the lack of screenshot capability, as it’s purely headless, there’s no way to capture a screenshot when something goes wrong, so you’ll find yourself re-running failed tests using a different browser, which may or may not reproduce your problem. Not ideal!

Finally, running WebDriver headless tests requires you to run a Selenium Server, which is additional overhead for your tests.

But why headless?

Whilst running automated browser tests in a headless form can speed things up, the main reason I see people wanting headless watir-webdriver support is so that tests can be run on headless Linux machines, for example, a Jenkins Server. I’ve found running tests in parallel speeds things up more than headless execution can, so it’s mainly headless server support most people are after.

Enter the headless gem.

The headless gem is a ruby wrapper for Xvfb that makes it super easy to run graphical applications (such as real web browsers) on a headless machine. This gem is perfect for running headless watir-webdriver tests using a real browser.

How? It’s simple.

require 'watir-webdriver'
require 'headless'
headless =
b = Watir::Browser.start ''
puts b.title

You’ll need Xvfb installed, which is as easy as:

sudo apt-get install xvfb

Or if you’re using cucumber like me, you just need the following code in your env.rb file:

  require 'headless'
  headless =
  at_exit do

This then means you can still launch and use Firefox (or Chrome) on a headless machine.

The best thing about it though is that Watir-WebDriver’s inbuilt screenshots still work perfectly, so it captures a screenshot of the actual page even though it has no display. How awesome is that!


The headless gem makes it super easy to run real browsers on headless servers. This way you get real JavaScript support without the need to run a Selenium server, and is perfect for running your tests as part of continous integration.

Stay tuned as I’ll be writing a post soon about how to use an EC2 instance to run your headless watir-webdriver tests in the cloud, for free!

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server ="/path/to/jar", :background => true)
# run your tests

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = :chrome
b.goto ''
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"

The only difference for Firefox:

b = :firefox

The only difference for IE:

b = :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b =, :url => '', :desired_capabilities => capabilities)
b = :firefox
b.goto ""
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using, but I haven’t tried this out yet.


Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading

Celerity: first impressions

Last week at the Test Automation Workshop (TAW) here in Australia, the topic of being able to run automated tests ‘headless’ came up, and I mentioned the Celerity project: a headless Watir port. I decided to have a play with Celerity tonight to see how easy it is to get up and running, and also look at it as a way to quickly run Watir scripts.


The installation was fairly straightforward. You need a Java 6 JDK, as well as the JRuby binaries. You’ll need to update two environment variables, namely add JRuby to your path, and update your JAVA_HOME variable. Installing Celerity involves a JRuby gem: "jruby -S gem install celerity". You can also install it from github, but I don’t know the difference between the two.

Availability of Gems

Ruby gems are supported in JRuby, but only if they don’t use C libraries. This means that Watir won’t work, nor will Roo. I use Roo to define test cases in spreadsheets, so it means I can’t do a comparison of execution times at the moment, at least until I get these test cases out of spreadsheets.

Using Celerity as a simple load testing tool instead

At TAW, Kelvin Ross thought Celerity sounded promising as a load testing tool, considering it was headless and lightweight. I thought I would give a simple google search script a try, running in both Celerity, and Watir, with 30 concurrent users.

Running under Watir

CPU peaked at 100% for the entire run, and each page varied considerable but took on average 10 seconds to load.

Running under Celerity

CPU use was normal, and each page took just over 1 second to load (minimal variance).


Celerity seems a promising way to execute basic load tests using a headless browser. The benefit of Celerity is support for javascript execution in the browser, the downside at the moment is lack of support for some ruby gems. If you could run Watir under JRuby, I could have used a single script.

Scripts Used
Celerity Script

require 'thread'
require "rubygems"
require "celerity"

def test_google
  browser =
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time =
  browser.button(:name, 'btnG').click
  end_time =
  puts end_time - start_time

threads = []
30.times do
  threads << {test_google}
threads.each {|x| x.join}

Watir Script

require 'thread'
require 'watir'
require 'watir/ie'

def test_google
  browser = Watir::IE.start('')
  browser.text_field(:name, 'q').value = 'Celerity'
  start_time =
  browser.button(:name, 'btnG').click
  end_time =
  puts end_time - start_time

threads = []
30.times do
  threads << {test_google}
threads.each {|x| x.join}