Automated local accessibility testing using WAVE and WebDriver

I wrote a while back about automated WCAG 2.0 accessibility testing in our build pipeline.

My solution involved capturing the HTML of each unique page in our application visited via WebDriver, then navigating to the online WAVE tool to check the accessibility of each chunk of HTML, all done during a dedicated accessibility stage of our build pipeline.

That was working pretty well until I noticed our IP address was starting to become blocked by WAVE due to a unreasonable number of checks… Doh!

Locally we’ve all been using the Firefox WAVE toolbar, as it’s very easy to run against a locally running instance of our application. Which made me think: how do I automate the Firefox WAVE toolbar to avoid having to use the WAVE web site alltogether?

Today I changed our accessibility tests to use the WAVE toolbar. It was a bit of a pain since we use Windows and the WAVE toolbar seems to be broken in configuring Firefox shortcut keys to trigger accessibility checks.

What I ended up doing is firing the keystrokes needed to run the check via the Tools menu in Firefox. I then check the DOM to ensure there are no WAVE errors and screenshot it and fail the test if there are. Quite simple really, and it works very reliably and quickly.

Makes me think I should have done this to start with, but a good outcome nonetheless.

Some C# WebDriver code to do this follows. This should be easy to adapt to another language.

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using System.Drawing.Imaging;

namespace Dominos.OLO.Html5.Tests.Acceptance
{
    [TestClass]
    public class AccessibilityTests
    {
        [TestMethod]
        public void AccessibilityTest()
        {
            var profile = new FirefoxProfile();
            profile.AddExtension("wavetoolbar1_1_8_en.xpi");
            var driver = new FirefoxDriver(profile);

            driver.Navigate().GoToUrl("http://apple.com");

            var b = driver.FindElement(By.TagName("body"));
            b.SendKeys(Keys.Alt + "T");
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowDown);
            b.SendKeys(Keys.ArrowRight);
            b.SendKeys(Keys.Enter);

            var waveTips = driver.FindElements(By.ClassName("wave4tip"));
            if (waveTips.Count == 0) Assert.Fail("Could not locate any WAVE validations - please ensure that WAVE is installed correctly");
            foreach (var waveTip in waveTips)
            {
                if (!waveTip.GetAttribute("alt").StartsWith("ERROR:")) continue;
                var fileName = String.Format("{0}{1}{2}", "WAVE", DateTime.Now.ToString("HHmmss"), ".png");
                var screenShot = ((ITakesScreenshot)driver).GetScreenshot();
                screenShot.SaveAsFile(fileName, ImageFormat.Png);
                Assert.Fail("WAVE errors were found on the page. Please see screenshot for details");
            }
        }
    }
}

and finally the resulting screenshot:

Apple WAVE check results

Why I don’t like Jasmine JavaScript unit tests (for MVC web apps)

Update 11-Feb-2013: After some feedback, I would like to clarify what I am talking about here: I am talking from experience in applications not purely written in JavaScript: for example a MVC application written with say C# with JavaScript added to pages.

I’m really not a big fan of Jasmine JavaScript unit tests for applications not written purely in JavaScript. Here’s why:

jasmine logo

Most modern web applications use JavaScript as a way to provide client side validation and interaction. Most of this JavaScript manipulates the Document Object Model (DOM) of the browser, therefore the JavaScript is coupled to the DOM using ids or classes of elements.

As most web pages are dynamically rendered by the server, Jasmine, the JavaScript unit test framework, can’t run against the ‘real’ DOM as it appears in the application, so what it does is duplicates parts of the DOM to test the JavaScript is working properly based upon this duplication.

As there’s no reliance on the real application or its DOM, these tests run super fast, you can run thousands of JavaScript tests in seconds which is pretty amazing.

The problem becomes that over time the real DOM in the application is changed but the Jasmine tests and its own duplicated DOM is not! This means that often I find that all Jasmine tests pass (as they have to for a new version to be deployed to QA) but the JavaScript doesn’t work in the application because an id or class name has been changed and someone ‘forgot’ to update the Jasmine tests.

Tests that don’t pick up valid errors are worthless, and I continually find myself getting annoyed that all the Jasmine tests have passed, but JavaScript is broken and our application doesn’t work.

Another downside is that Jasmine tests are only as good as the JavaScript engine they are run against. We run our Jasmine tests in a headless PhantomJS browser, which is fast, but most of our JavaScript bugs are found in Internet Explorer, particularly earlier versions of it, so Jasmine tests again are worthless.

I don’t know of a way to overcome these downsides to Jasmine that I have mentioned. What we do have is a suite of end to end acceptance tests that run in 5 different ‘real’ browsers that check the JavaScript works correctly. Whilst these take 8 minutes to run per browser, they pick up a lot more JavaScript issues than Jasmine.

I am not still not sure whether it is worth investing the time and effort in Jasmine tests when they don’t provide sufficient confidence that our application ‘works’.

What have others experiences been with Jasmine JavaScript unit testing?

Automated WCAG 2.0 accessibility testing in the build pipeline

The web application I am currently working on must meet accessibility standards: WCAG 2.0 AA to be precise. WCAG 2.0 is a collection of guidelines that ensure your site is accessible for people with disabilities.

An example of poor accessibility design is missing an alt tag on an image, or not specifying a language of a document, eg:

<HTML lang="fr">

Building Accessibility In

Whilst later we’ll do doing accessibility assessments with people who are blind or have low vision, we need to make sure we build accessibility in. To do this, we need automated accessibility tests as part of our continuous integration build pipeline. My challenge this week was to do just this.

Automated Accessibility Tests

First I needed to find a tool to validate against the spec. We’re developing the web application locally so we’ll need to run it locally. There’s a tool called TotalValidator which offers a command line tool, the only downside is the licensing cost, as to use the command line version you need to buy at least 6 copies of the tool at £25 per copy, approximately US$240 in total. There’s no trial of command line tool unless you buy at least one copy of the pro tool at £25. I didn’t want to spend money on something that might not work so I kept looking for alternatives.

There are two sites I found that validate accessibility by URL or supplied HTML code: AChecker and WAVE.

AChecker: this tool which works really well. It even supplies an REST API, but I couldn’t find a way to call the API to validate HTML code (instead of by URL) which is what I would like to do. The software behind AChecker is open source (PHP) so you can actually install your own version behind your firewall if you wish.

WAVE: a new tool recently released by WebAim: Web Accessibility in Mind. Again this is an online checker that allows you to validate by URL or HTML code supplied, but unfortunately there’s no API (yet) and the results aren’t as easy to programatically read.

My Solution

The final solution I came up with is a specific tagged web accessibility feature in our acceptance tests project. This has scenarios that use WebDriver to navigate through our application capturing the HTML source code from each page visited. Finally, it visits the AChecker tool online and validates each piece of HTML source code it collected and fails the build if any accessibility problems are found.

AChecker Results

Build Light

We have a specific build light that runs all the automated acceptance tests and accessibility tests. If any of these fail, the light goes red.

Build Status Lights

It’s much better if it looks like this:

Build Lights All Green

Summary

It was fairly easy to use an existing free online accessibility checker to validate all HTML code in your local application, and make this status highly visible to the development team. By building accessibility in, we’re reducing the expected number of issues when we conduct more formal accessibility testing.

Bonus Points: faster accessibility feedback

Ideally, as a page is being developed, the developer/tester should be able to check accessibility (rather than waiting for the build to fail). The easiest way I have found behind a firewall is to use the WAVE Firefox extension, which displays errors as you use your site. Fantastic!

(Apple and Microsoft each have one known accessibility problem, Google has nine!)

Apple.com accessibility

Mobile apps still need automated tests

Jonathan Rasmusson recently wrote what I consider to be quite a contentious blog post about iOS application development titled “It’s not about the unit tests”.

“…imagine my surprise when I entered a community responsible for some of the worlds most loved mobile applications, only to discover they don’t unit test. Even more disturbing, they seem to be getting away with it!”

Whilst I agree with the general theme of the blog post which is change your mind, challenge assumptions:

“All I can say is to keep growing sometimes we need to challenge our most cherished assumptions. It doesn’t always feel good, but that’s how we grow, gain experience, and turn knowledge into wisdom.”

“The second you think you’ve got it all figured out you’ve stopped living.”

I don’t agree with the content.

Jonathan’s basic premise is that you can get away with little or no unit testing for your iOS application for a number of reasons including developing for a smaller screen size, no legacy, one language, visual development and developing on a mature platform. But the real reason that iOS get away with it is by caring.

“These people simply cared more about their craft, and what they were doing, than their contemporaries. They ‘out cared’ the competition. And that is what I see in the iOS community.”

But in writing this post, I believe he missed two critical factors when deciding whether to have automated tests for your iOS app.

iOS users are unforgiving

If you accidentally release an app with a bug, see how quickly you’ll start getting one star reviews and nasty comments in the App Store. See how quickly new users will uninstall your app and never use it again.

The App Store approval process is not capable of supporting quick bug fixes

Releasing a new version of your app that fixes a critical bug may take you 2 minutes (you don’t even need to fix a broken test or write a new test for it!) but it then takes Apple 5-10 business days to release it to your users. This doesn’t stop the one star reviews and comments destroying your reputation in the meantime.

Case in Point: Lasoo iPhone app

I love the Lasoo iPhone app, because it allows me to read store catalogs on my phone (I live in an apartment block and we don’t get them delivered). Recently I upgraded the app and then tried to use it but it wouldn’t even start. I tried the usual close/reopen, delete/reinstall but still nothing. I then checked the app store:

Lasoo iPhone app reviews
Lasoo iPhone app reviews

Oh boy, hundreds of one star reviews within a couple of days: the app is stuffed! I then checked twitter to make sure they knew it was broken, and to my surprise they’d fixed it immediately but were waiting for Apple to approve the fix.

I can’t speculate on whether Lasoo care or not about their app, but imagine for a second if they had just one automated test, one automated test that launched the app to make sure it worked, and it was run every time a change, no matter how small, was made. That one automated test would have saved them from hundreds of one star reviews and having to apologize to customers on twitter whilst they waited for Apple to approve the fix.

Which raises another point:

“[Apple] curate and block apps that don’t meet certain quality or standards.”

The Lasoo app was so broken it wouldn’t even start, so how did it get through Apple’s approval process for certain quality or standards?

Just caring isn’t enough to protect you from introducing bugs

We all make mistakes, even if we care. That’s why we have automated tests, to catch those mistakes.

Not having automated tests is a bit like having unprotected sex. You can probably get away with it forever if you’re careful, but the consequences of getting it wrong are pretty dramatic. And just because you can get away with it doesn’t mean that other people will be able to.

Roll your own page objects

There seems to be a lot of focus being put into page object ruby gems at the moment. Cheezy has done a fantastic job of the aptly named page-object that supports Watir-Webdriver and Selenium-Webdriver, and then there’s the more recent site_prism (also fantastic) by Nat Ritmeyer that works with Capybara. Before these two came along, I even wrote my own; the now retired watir-page-helper gem.

The premise of these gems is they make it super easy to create page objects for your ruby automated testing projects. But today I want to discuss another crazy idea with you: do you even need a gem at all to do page objects?

Background

I recently refactored some automated tests that Chris McMahon wrote as a potential framework for Wikimedia Foundation (creators of Wikipedia). Chris’s code used Cheezy’s excellent page-object gem so I happily went about my refactoring his code using that gem. Suddenly… I found instead of helping me it started to hinder me. I kept having to refer to page-object user guide I got from Cheezy in Austin to work out how things work. Namely:

  1. How to define elements: as they are different from watir-webdriver (eg. list_item vs li, cell vs td etc.)
  2. How to identify elements: as they are limited to certain supported attributes by element type, unlike watir-webdriver which supports every attribute for all elements
  3. What methods each element provides and what each does: as different elements create different methods with different behaviours, so calling a link element clicks it, whilst a span element returns its text.

The main problem I personally found was that Page-object has essentially created its own DSL to describe elements in page objects, and this DSL is subtly and not so subtly different from the Watir-Webdriver API, so the API I know and love doesn’t work in a lot of places.

An example. There’s a common menu bar on all the Wikimedia sites that displays the logged in user as a link (to the user’s page).

The link is only recognizable by its title attribute, and whilst this is supported by watir-webdriver (it supports any attribute), it is not supported by page-object. The source html looks like:

<a class="new" accesskey="." title="Your user page [ctrl-.]" href="/wiki/User:Alister.scott">Alister.scott</a>

What I would have liked to do was:

  link :logged_in_as, :title => 'Your user page [ctrl-.]'

But, instead, I had to do this (which isn’t very DRY):

  def logged_in_as
    @browser.link(:title => 'Your user page [ctrl-.]').text
  end

I believe essentially what has happened is the page-object, in its neutrality between selenium-webdriver and watir-webdriver, has created its own DSL that is somewhat of a halfway point between the two. This is probably fine for most people starting out, it’s just an API to learn, but for someone like me who has extensive experience with the watir-webdriver API (and loves the power of it), I find it limiting. This is particularly evident when I write a majority of my code using the watir-webdriver API under IRB.

So, I had to take a re-think. Why not roll my own page objects for Wikimedia Foundation?

Roll your own page objects

I recently had a discussion with a colleague/good friend about page objects which went along the lines of “I don’t understand those page object gems because you end up writing a custom page object pattern for each project anyway, as every project/application you work on is different in its own way”. It was one of those aha moments for me.

What I needed was to roll my own Wikimedia page objects.

Taking it back to basics, essentially there are three functions I see a page object pattern provides:

  1. Ability to easily instantiate pages in a consistent manner;
  2. Ability to concisely describe elements on a page, keep it DRY by avoiding repetition of element identifiers (using the underlying driver’s API); and
  3. Ability to provide higher level methods that use the elements to perform user oriented functions.

You can probably notice the helper methods – the magic – that gems like page-object and site_prism provide are missing from my list. This is on purpose, and is because, after lots of thought, I actually don’t find these useful, as they encourage specifications/steps to be too lower level. I would rather a high level method on the page (eg. login) than exposing my username and password fields and a login button.

A Wikimedia Page Model

Taking those things into consideration: this is the page model I came up with for Wikimedia.

Generic Base Page Class

The generic base page class is what everything else extends. It contains the instantiation code common to all pages, and the class methods needed to define elements and methods (more on these later).

Wikimedia Base Page Class

This page class contains elements and methods that are common to all Wikimedia pages. The ‘logged in user’ example above is a good example of something that is the same on every Wikimedia page, whether you’re on Wikipedia, or Wikimedia Commons etc.

Commons & Wikipedia Base Page Classes

These two classes are placeholders for elements and methods are common to a particular site. At the moment with my limited examples, these don’t contain content.

Commons & Wikipedia Page Classes

These are the actual pages that are representations of pages in Wikimedia. These are in separate modules so they are in different namespaces (you can have a Wikipedia::LogonPage and a Commons::LogonPage).

Some example pages:

class Wikipedia::LogoutPage < Wikipedia::BasePage
  page_url "#{Wikipedia::BASE_URL}/w/index.php?title=Special:UserLogout"
  expected_title "Log out - #{Wikipedia::TITLE}"

  element(:main_content_div) { |b| b.div(id: 'mw-content-text' ) }
  value(:main_content) { |p| p.main_content_div.text }
end

Here we can see we define a page_url and expected_title, which are used to instantiate the page.

Next we define an element passing in a block of watir-webdriver code for it, and a value by referencing the element we defined before it. Since these element and value methods execute blocks against self, and the class delegates missing methods to our browser, we can refer to either the browser (shown as b) or the page class (shown as p) in our blocks.

class Commons::LoginPage < Commons::BasePage
  page_url "#{Commons::BASE_URL}/w/index.php?title=Special:UserLogin"
  expected_title "Log in / create account - #{Commons::TITLE}"

  login_elements
  value(:logged_in?) { |p| p.logged_in.exists? }

  def login_with username, password
    username_field.set username
    password_field.set password
    login_button.click
  end
end

In this example, we again define the page_url and expected_title, but we have stored the login_elements with the WikimediaBasePage (as they are the same across all the sites) so we include them by specifying login_elements. We have also defined a login_with method that performs actions on our elements.

There are three available methods to define page elements, values and actions, and these all follow the same format of specifying the method name, and passing in a block of watir-webdriver code.

Calling the page objects from Cucumber step definitions

I chose to use Cucumber for the Wikimedia Foundation framework over Chris’s choice of RSpec as I find it easier to specify end-to-end tests in this way. I find the Cucumber step definitions encourage reuse of steps typically used to set up a test (that are often duplicated in RSpec).

I try to stick to calling the exposed methods, values or actions instead of the elements themselves from my Cucumber steps to ensure I am writing them at a high level. An example step using the page above looks like:

Given /^I am logged into Commons$/ do
  visit Commons::LoginPage do |page|
    page.login_with Commons::USERNAME, Commons::PASSWORD
    page.should be_logged_in
  end
end

The visit and on methods are defined in a Pages module that is mixed into the Cucumber World so these available on all step definitions. As named, the visit method instantiates and visits the page, whereas the on just instantiates it.

module Pages
  def visit page_class, &block
    on page_class, true, &block
  end

  def on page_class, visit=false, &block
    page = page_class.new @browser, visit
    block.call page if block
    page
  end
end

Summary

That’s all there really is the rolling your own page objects. I found this excercise useful as it gives me maximum flexibility and allows me to clearly define pages how I want to define them. I appreciate all the great work that Cheezy and Nat have done on their page object gems, if anything these contain great inspirations on how to roll your own custom page objects most suited to your environment and applications.

You can check out my full code here on Github.