Upgrading WebdriverJs to Selenium 3

Yes, I know that Selenium 3 has been out for a while, but I’ve finally got around at looking at updating our end-to-end tests to use it. Newer versions of Firefox require Geckodriver which require Selenium 3.3+ so it’s a forced upgrade of sorts.

Continue reading “Upgrading WebdriverJs to Selenium 3”

Stop Firefox auto-updating (and breaking your CI build)

If you run regular automated WebDriver tests using Firefox then chances are you’ve encountered the problem where Firefox updates itself and your WebDriver bindings don’t support that version which causes all your tests to fail. It’s really annoying, so I suggest you set your Firefox install to never automatically update as below to avoid this happening.

Stop Firefox auto updating

Update:

As Alex points out below, you can do this programmatically for a server you don’t actually launch Firefox on, ie. a headless machine.

profile = Selenium::WebDriver::Firefox::Profile.new
 # disable autoupdate
 profile['app.update.auto'] = false
 profile['app.update.enabled'] = false

Automated local accessibility testing using WAVE and WebDriver

I wrote a while back about automated WCAG 2.0 accessibility testing in our build pipeline.

My solution involved capturing the HTML of each unique page in our application visited via WebDriver, then navigating to the online WAVE tool to check the accessibility of each chunk of HTML, all done during a dedicated accessibility stage of our build pipeline.

That was working pretty well until I noticed our IP address was starting to become blocked by WAVE due to a unreasonable number of checks… Doh!

Locally we’ve all been using the Firefox WAVE toolbar, as it’s very easy to run against a locally running instance of our application. Which made me think: how do I automate the Firefox WAVE toolbar to avoid having to use the WAVE web site alltogether?

Today I changed our accessibility tests to use the WAVE toolbar. It was a bit of a pain since we use Windows and the WAVE toolbar seems to be broken in configuring Firefox shortcut keys to trigger accessibility checks.

What I ended up doing is firing the keystrokes needed to run the check via the Tools menu in Firefox. I then check the DOM to ensure there are no WAVE errors and screenshot it and fail the test if there are. Quite simple really, and it works very reliably and quickly.

Makes me think I should have done this to start with, but a good outcome nonetheless.

Some C# WebDriver code to do this follows. This should be easy to adapt to another language.

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using System.Drawing.Imaging;

namespace Dominos.OLO.Html5.Tests.Acceptance
{
    [TestClass]
    public class AccessibilityTests
    {
        [TestMethod]
        public void AccessibilityTest()
        {
            var profile = new FirefoxProfile();
            profile.AddExtension("wavetoolbar1_1_8_en.xpi");
            var driver = new FirefoxDriver(profile);

            driver.Navigate().GoToUrl("http://apple.com");

            var b = driver.FindElement(By.TagName("body"));
            b.SendKeys(Keys.Alt + "T");
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowRight);
            b.SendKeys(Keys.Enter);

            var waveTips = driver.FindElements(By.ClassName("wave4tip"));
            if (waveTips.Count == 0) Assert.Fail("Could not locate any WAVE validations - please ensure that WAVE is installed correctly");
            foreach (var waveTip in waveTips)
            {
                if (!waveTip.GetAttribute("alt").StartsWith("ERROR:")) continue;
                var fileName = String.Format("{0}{1}{2}", "WAVE", DateTime.Now.ToString("HHmmss"), ".png");
                var screenShot = ((ITakesScreenshot)driver).GetScreenshot();
                screenShot.SaveAsFile(fileName, ImageFormat.Png);
                Assert.Fail("WAVE errors were found on the page. Please see screenshot for details");
            }
        }
    }
}

and finally the resulting screenshot:

Apple WAVE check results

Continuous delivery and ruining the user experience

“I was chatting with a random Canadian woman. She found out I worked for Mozilla and I got that sinking feeling, like “here we go again” because I knew exactly what was coming up. She proceeded to tell me the story of how she switched to Chrome because Firefox kept breaking her extensions and asking her to restart.

I’ve heard this story a lot in the last year.

I used to be proud to say I worked (or had worked) for Mozilla, but a careful listener might detect a certain sheepish quality that has crept into my voice lately when I name-check my former employer. And this is why. Even on the opposite side of the world, it’s always the same story: “I used to love Firefox”, but “I switched to Chrome because my extensions stopped working” or “I switched to chrome because Firefox kept asking me to restart”.

I’ve had this conversation with dozens of people across three continents. Not one person has had anything good to say about the rapid release process. Nearly 100% of my highly unscientific survey volunteered the information — unasked, unprompted — that the rapid release process had ruined Firefox for them.

Of course nobody says “rapid release process” because people don’t know that’s what it was called. They might start out complaining about version numbers, or some plugin that doesn’t work right, but when I ask enough questions to get to the root of the problem, it’s always the rapid release process.”

This article, by an ex-Mozilla developer all but confirms what I wrote about here nearly two years ago:

“The only downside I see to continuous delivery is when it’s used in an environment that needs to be actively upgraded by users; it’s no point pushing out new functionality daily if your users have to do an upgrade daily.”

I am at a point where after 8 years of using Firefox as my primary day to day web browser, I am seriously considering switching to using Google Chrome primarily. I find the Firefox rapid release process ridiculous: who releases a major version upgrade every few six weeks that appears the same as any previous version? And each version requires new WebDriver binaries, unlike Chrome.

I see the iOS update issue becoming more and more prevalent as we have more iDevices in our household that need constant updating. Other annoyances include Adobe Flash on the desktop (every second day it seems), Mac OSX installing updates that need restarts and paid OSX apps now require running App Store on Mac to check for updates. Madness!

Automatic Firefox authentication when using Selenium-WebDriver with AutoAuth

I came across a particularly challenging problem today automating a web app for a client that runs behind a corporate proxy on a different Windows domain. The corporate proxy used NTLM authentication, but since I was on a different domain, I couldn’t get Firefox to send this information automatically so an authentication dialog would always appear that looked similar to this (IE worked fine):

Normally with browser authentication it is fairly straightforward to embed the username and password into the URL and Firefox will pass this to the web application without any problems (it’ll even ignore the confirmation normally displayed to the user), but in this case it didn’t work as it was the proxy that was requesting the information, not the application.

require 'watir-webdriver'
b = Watir::Browser.new :firefox
b.goto 'http://admin:password@192.168.0.1'

I manually could get Firefox to store the credentials, but every time the WebDriver tests would run, this darn Authentication Required’ dialog would appear (without credentials if using the standard new WebDriver profile for each test run). I tried setting all sorts of Firefox about:config settings to do with NTLM but nothing would work. After lots of trial and error, and finding nothing useful on the Internet about this issue, a colleague pointed out a Firefox add-on called AutoAuth that automatically submits these dialogs using stored Firefox credentials. Voila!

Example using Watir-WebDriver (the quick way)

The easiest way is to the install the AutoAuth add-on on your default Firefox profile (the one that Firefox uses when launched manually), and store the credentials needed in the default Firefox password manager. All you then need to do is tell Watir-WebDriver to use the default profile:

require 'watir-webdriver'
b = Watir::Browser.new :firefox, :profile => 'default'
b.goto 'http://192.168.0.1'

The issue with the above code is that it’s not repeatable across machines, as the machine’s default profile must have AutoAuth installed, and the username and password in the password manager.

Example using Watir-WebDriver (the most repeatable way)

To make this more repeatable, first you need to create a Firefox profile by following the instructions here (we’ll call it WatirWebDriver).

Manually launch this profile and visit the site you need to authenticate to, enter the username and password and make sure you save the credentials in Firefox when prompted.

The script is then pretty simple: create a profile as a copy of the one you made, add the AutoAuth extension (download it and place the xpi file in your project directory), and visit the site:

profile = Selenium::WebDriver::Firefox::Profile.from_name 'WatirWebDriver'
profile.add_extension 'autoauth-2.1-fx+fn.xpi'
b = Watir::Browser.new :firefox, :profile => profile
b.goto 'http://192.168.0.1'

This script should visit the site and AutoAuth should kick in and automatically submit that pesky ‘Authentication Required’ dialog: take that!

Summary

Whilst this NTLM proxy authentication issue was a bit of an issue to begin with, we found a reasonable way to work around it. I don’t really like the dependency on an existing Firefox profile with the proxy credentials, but until I work out how to store credentials in a Firefox profile I create at runtime using Selenium (which I don’t believe is possible), I think that it’s necessary.

I’ve also updated the Watir-WebDriver Basic Browser Authentication page.

Waiting for watir-webdriver downloads (and determining the file name)

There was an interesting question on Stackoverflow recently about waiting for a file download whilst running a watir-webdriver test. Whilst it’s happy to download files, watir-webdriver won’t wait until they have downloaded, so essentially this is up to you to check.

The easiest way I have found to do this is, just before you download the file, check the contents of your download directory, then commence the download, and then wait until there’s an additional file in your downloads directory.

Once you have the additional file, you can read its file name to determine what was actually downloaded. This is useful for dynamic file names, or even static file names when you’re downloading multiples (as Firefox will add a (1), (2), (3) etc. to the file name).

The code to do this as follows:

require 'watir-webdriver'

file_name = nil
download_directory = "#{Dir.pwd}/downloads"
download_directory.gsub!("/", "\\") if Selenium::WebDriver::Platform.windows?
downloads_before = Dir.entries download_directory
 
profile = Selenium::WebDriver::Firefox::Profile.new
profile['browser.download.folderList'] = 2 # custom location
profile['browser.download.dir'] = download_directory
profile['browser.helperApps.neverAsk.saveToDisk'] = "text/csv,application/pdf"
 
b = Watir::Browser.new :firefox, :profile => profile

b.goto 'https://dl.dropbox.com/u/18859962/hello.csv'

30.times do
  difference = Dir.entries(download_directory) - downloads_before
  if difference.size == 1
    file_name = difference.first 
    break
  end  
  sleep 1
end
raise "Could not locate a new file in the directory '#{download_directory}' within 30 seconds" if not file_name
puts file_name

As usual, let me know if you know of a better way to do this.

Watir-WebDriver tests on Firefox 7: getting rid of the send data to Mozilla message

Update 6 October 2011: The send data to Mozilla question will be turned off by default in the next release (2.8.0) of the selenium-webdriver gem which watir-webdriver uses.

I’ve been running Watir-WebDriver tests against Firefox 7, which works superbly. The biggest change is Firefox 7 now supports performance metrics, so this means you can use the watir-webdriver-peformance gem: yay! It also means my EtsyWatirWebDriver project now collects page metrics using Firefox.

The only slight annoyance is the presence of the ‘send data to Mozilla?’ dialog bar. Never fear, it’s easily dismissed.

require 'watir-webdriver'
profile = Selenium::WebDriver::Firefox::Profile.new
profile['toolkit.telemetry.prompted'] = true
b = Watir::Browser.new :firefox, :profile => profile

Enjoy.

Send data to mozilla