Lock down your browser versions if you run WebDriver tests

Recently our WebDriver tests that run in Chrome via a Windows service all suddenly stopped working even though we hadn’t made any changes to our tests. It seems Chrome had automatically updated itself on our WebDriver agents introducing a Chromium 38 bug meaning WebDriver won’t work at all (full details here and here). Getting these tests running again has been very painful, mainly due to Google not having standalone Chrome installers for any previous versions of Chrome publicly available.

If you run any WebDriver tests I highly recommend you lock down your browser versions to stop this happening to you in the future. Here’s how:

Firefox

Firefox is fantastic in this regard as they make every back version easily accessible as well as a simple way on all platforms to stop automatic upgrades. I tend to lock down to Firefox ESRs (Extended Support Releases) such as versions 24 and 31 which are listed on this comprehensive Wikipedia page.

To stop updates all you do is open preferences, advanced, update and select ‘never’.

Firefox Prevent Updates

Chrome

Chrome is a P.I.T.A. in both being able to install a previous version or lock down the currently installed one. Google prefer a Chrome web installer which always installs the latest version of Chrome, and if you want a specific version you need the alternate (offline) installer (for all users if you use a Windows service), but they only provide the latest installer. It’s hard if not impossible to find older alternate (offline) installers on the web, even oldapps.com can’t host them.

Once you have a version of Chrome on Windows that you want to keep, you need to download a group policy template, and disable automatic updates before running Chrome (so it doesn’t automatically update before you set the group policy). I won’t go into full details but you should be able to find all details here. Some sites mention using a plugin to stop updates but this doesn’t work so you’ll need to go down the group policy path.

Summary

Locking down browser versions avoids having to suddenly work out why your entire WebDriver test suite fails.

Setting text in Chromedriver is rather slow

A colleague contacted me today to see if there’s a way to increase the speed of a text_field.set operation in watir-webdriver when using Chrome on Mac OSX. I hadn’t noticed it before but Chrome seems to enter character by character into text fields which is noticeably slower than Firefox which seems to ‘set’ the field at once. Jari Bakken suggested I try Chrome native events which I didn’t even realize existed on Mac OSX.

So, I took Jari’s suggestion and I created a benchmark of a .set operation conducted ten times in a row on Firefox, and Chrome with and without native events. The results looked like:

                        user       system     total        real
set firefox             0.160000   0.020000   0.180000  ( 8.205473)
set chrome non-native   0.100000   0.010000   0.110000  (44.980334)
set chrome native       0.100000   0.010000   0.110000  (31.501647)

You can see that Chrome is drastically slower than Firefox in setting text fields, but native events does seem to provide a roughly 25% speed improvement. I think I will stick to running my watir-webdriver tests against Firefox unless specifically needed.

Sample Code for Benchmarks

Setting text using Chrome with native events enabled

require 'selenium-webdriver'
require 'watir-webdriver'
require 'bench'

d = Selenium::WebDriver.for :chrome, native_events: true
b = Watir::Browser.new d
b.goto 'bing.com'

benchmark 'set chrome native' do
  b.text_field(name: 'q').set 'ghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghgh'
end

run 10
b.close

Setting text using Chrome with native events disabled

require 'selenium-webdriver'
require 'watir-webdriver'
require 'bench'

b = Watir::Browser.new :chrome
b.goto 'bing.com'

benchmark 'set chrome non-native' do
  b.text_field(name: 'q').set 'ghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghgh'
end

run 10

b.close

Setting text using Firefox (native events aren’t available on OSX)

require 'selenium-webdriver'
require 'watir-webdriver'
require 'bench'

b = Watir::Browser.new :firefox
b.goto 'bing.com'

benchmark 'set firefox' do
  b.text_field(name: 'q').set 'ghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghghgh'
end

run 10

b.close

Configuring Chrome settings when running Watir-Webdriver

I recently found the need to ignore certificate errors when running Watir-Webdriver tests in Google Chrome. The way to do this is slightly different from doing it in Firefox. Unlike Firefox where you create capabilities and profiles, in Google Chrome you pass arguments to Chrome to enable/disable certain browser features.

For example: to ignore certificate errors, disable pop-ups, disable translate and specify a proxy you do:

b = Watir::Browser.new :chrome, :switches => %w[--ignore-certificate-errors --disable-popup-blocking --disable-translate --proxy-server=myproxy.com:8080]

NB: The %W[] converts a long string (separated by spaces) into an array of strings.

The full list of these options is available here. Thanks again to Jari Bakken for pointing me in the right direction.

Telling Watir-Webdriver to use a http and ssl proxy

I’ve been trying to get Watir-WebDriver to use a proxy for both http and https sites, but was having a bit of trouble.
There are a couple of key pointers: you need to specify the proxy address without the ‘http://’ prefix (or it’ll crash), and you need to also specify a :ssl parameter to Firefox profile.

Example: setting a http and https proxy for Firefox

profile = Selenium::WebDriver::Firefox::Profile.new
profile.proxy = Selenium::WebDriver::Proxy.new :http => 'my.proxy.com', :ssl => 'my.proxy.com'
browser = Watir::Browser.new :chrome, :profile => profile

Example: setting a http and https proxy for Chrome

switches = '--proxy-server=my.proxy.com'
browser = Watir::Browser.new :chrome, :switches => switches

Using the super-duper ChromeDriver with Watir-WebDriver (updated)

Anyone who has used Watir-WebDriver (or WebDriver for that matter) to test Google Chrome will know that it’s been pretty unreliable, and pretty much unusable in the past… until now.

The ChromeDriver executable was released recently which enables WebDriver to control Google Chrome natively using the Wire protocol. This means your Watir-WebDriver tests will run super fast, and super reliably.

For Mac OSX:

  1. Get the latest Watir-WebDriver gem
    gem update watir-webdriver
  2. Download the ChromeDriver binary from the Chromium site, and copy it to your path (such as /usr/local/bin/)
  3. This will work with any current version of Chrome
  4. Change your Watir-WebDriver script to use Watir::Browser.new :chrome and voila, super fast Chrome, with no weird first tab opened in the background. This is the key to tell if you’re doing it right, your app should appear in the first tab of Chrome, not the second.

I haven’t done this on Windows yet, so your mileage may vary, but I imagine it wouldn’t be too different.

Watir-WebDriver: A detailed introduction

Update: 22 August 2011: Please see: watirwebdriver.com for a detailed guide to Watir-WebDriver

Update: 22 July 2011: I have updated quite a number of things that have changed since the earlier releases

Watir-WebDriver is a really great tool; Jari Bakken‘s done a really good job of it. There’s not a huge amount on the web about it, how to get it up and running and use it. I’m aiming to fix that here.

For those who don’t know what Watir-WebDriver is, it’s basically a nice Watir (ruby) implementation on WebDriver, so it gives you four browsers (three real, one headless) using one neat API, out of the box. The thing I like about it is you don’t need to use JRuby (like Celerity), which means it plays nice with Cucumber (although Cucumber does work under JRuby).

I’ve written about how Watir, WebDriver and Selenium all fit together before, so this post aims to be a lot more hands-on.

Getting Watir-WebDriver Running

There are essentially two components you need: the Watir-WebDriver ruby gem, and the remote WebDriver server. The remote WebDriver server is only needed if you want to run your tests in headless mode without a real browser (or want to use Opera). You obviously need ruby first but I won’t detail that here. You can find info about ruby versions etc. at watir.com. If you’re on Mac or Linux, I strongly suggest using RVM and bundler.

The Watir-WebDriver ruby gem

It’s a simple matter of opening a command prompt and typing:

  • gem install watir-webdriver (windows); or
  • sudo gem install watir-webdriver (osx or linux – better to use RVM and bundler)

The remote WebDriver Server

This is the slightly tricky part. This is so that WebDriver can run headless without a real browser, and isn’t needed for real browser support (bar Opera). The quickest easiest way to get up and running is to download this java jar file, open a command prompt where you have saved it, and run:

java -jar selenium-server-standalone-2.0b1.jar

You can also specify the startup and max memory allocated, which is handy when running large test suites.

java -Xms1024M -Xmx2048M -jar selenium-server-standalone-2.0a7.jar

You need to have this java server running whenever using WebDriver, but it’s easy enough to bootstrap it.

Update: Jari has pointed out you can run the server programmatically which is even better:

require 'selenium/server'
server = Selenium::Server.new("/path/to/jar", :background => true)
server.start
# run your tests
server.stop

First Impressions

Waiting in Watir-WebDriver doesn’t seem as straightforward as with Watir, probably due to the underlying drivers, but fortunately there’s an inbuilt waiting library to use (also available in Watir 1.6.7+):

  • Watir::Wait.until { ... }: where you can wait for a block to be true
  • object.when_present.set: where you can do something when it’s present
  • object.wait_until_present:; where you just wait until something is present
  • object.wait_while_present:; where you just wait until something disappears

Hello Watir-WebDriver

It seems pertinent to start with a google search example.

Hello Watir-WebDriver in three browsers

These three browsers seem to work very similarly, but obviously Internet Explorer will only run on Microsoft Windows.

require 'rubygems'
require 'watir-webdriver'
b = Watir::Browser.new :chrome
b.goto 'www.google.com'
b.text_field(:name => 'q').set 'Watir-WebDriver'
b.button(:name => 'btnG').click
b.div(:id => 'resultStats').wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

The only difference for Firefox:

b = Watir::Browser.new :firefox

The only difference for IE:

b = Watir::Browser.new :ie

Hello Watir-WebDriver in Headless (HTML Unit)

I imagine this is the most anticipated feature for Watir users, as it means your tests run much faster, and you can still use ruby (unlike Celerity which uses JRuby). The script is fairly straightforward, you simply specify the hostname for your WebDriver server you started above. You need to specifically enable JavaScript by creating a capabilities profile.

require 'rubygems'
require 'watir-webdriver'
include Selenium
capabilities = WebDriver::Remote::Capabilities.htmlunit(:javascript_enabled => true)
b = Watir::Browser.new(:remote, :url => 'http://127.0.0.1:4444/wd/hub', :desired_capabilities => capabilities)
b = Watir::Browser.new :firefox
b.goto "www.google.com"
b.text_field(:name => "q").set "Watir-WebDriver"
b.button(:name => "btnG").click
b.div(:id => "resultStats").wait_until_present
puts "Displaying page: '#{b.title}' with results: '#{b.div(:id => "resultStats").text}'"
b.close

Sample Timings

As an experiment, I looped the Google Search script above 100 times to measure and compare the execution times.

  • Internet Explorer 8 on Windows 7: 400 seconds
  • Firefox 3.6 on Mac OSX: 277 seconds
  • Headless HTMLUnit on Mac OSX: 269 seconds

Very surprisingly, the headless run wasn’t much quicker at all. This may be due to running the test on Google over the Internet and not a local application.

Caveat Emptor

There are a number of differences between Watir-WebDriver and Watir. The main ones that are important to me are:

Zero based indexing as opposed to 1 based

For example,

In Watir/FireWatir: this finds the first table on a page:

puts b.table(:index => 1).text

But in Watir-WebDriver, it finds the second table on a page.

Attaching to windows
Attaching to new windows (for example pop-ups) has been fairly straightforward in Watir with its attach method, which is missing in Watir-WebDriver. It looks like you can iterate through a collection of windows in Watir-WebDriver using browser.windows, but I haven’t tried this out yet.

Summary

Watir-WebDriver is a very solid testing tool that uses a great browser automation engine (WebDriver) with a clean ruby API (Watir). I believe it will be a popular choice amongst testing teams, particually those using it for ATDD through Cucumber and running it as headless on a continuous integration server.

Further Reading