Playing with Playwright

Playwright is a new browser automation library from Microsoft:

Playwright is a Node library to automate the Chromium, WebKit and Firefox browsers with a single API. It enables cross-browser web automation that is ever-green, capable, reliable and fast.

https://github.com/microsoft/playwright

I’m a big fan of Puppeteer, so this section in their FAQ stood out to me:

Puppeteer is a Node library which provides a high-level API to control Chrome or Chromium over the DevTools Protocol. Puppeteer project is active and is maintained by Google.

We are the same team that originally built Puppeteer at Google, but has since then moved on. Puppeteer proved that there is a lot of interest in the new generation of ever-green, capable and reliable automation drivers. With Playwright, we’d like to take it one step further and offer the same functionality for all the popular rendering engines. We’d like to see Playwright vendor-neutral and shared governed.

https://github.com/microsoft/playwright#q-how-does-playwright-relate-to-puppeteer

Playwright uses similar concepts to Puppeteer:

“Due to the similarity of the concepts and the APIs, migration between the two should be a mechanical task.”

https://github.com/microsoft/playwright#q-how-does-playwright-relate-to-puppeteer

Luckily I have a demo test suite written in Puppeteer which I have cloned and converted to use Playwright to see how it works, and compares.

Here are my thoughts:

I really, really like the BrowserContext concept

In Puppeteer, and WebDriverJs, you have Browsers and Pages. Each Page in a Browser share the state across the Browser, so to create isolated tests using the same Browser (to avoid the inefficiencies of spawning a Browser per test) you need custom code to delete all cookies and local storage between tests. Playwright solves this with the BrowserContext object which is a new incognito window where its pages are created: each test can use the same browser but a different BrowserContext. Super cool šŸ‘Œ

It automatically waits to click, and supports xpath expressions

Playwright automatically waits for elements to be available and visible before clicking, by default, and also has the same API for xpath expressions, which means this Puppeteer code:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.waitForXPath( '//span[contains(., "Scissors")]' );
const elements = await page.$x( '//span[contains(., "Scissors")]' );
await elements[0].click();
await page.waitForXPath( '//div[contains(., "Scissors clicked!")]' );

becomes a lot cleaner:

await page.goto( `${ config.get( 'baseURL' )}` );
await page.click( '//span[contains(., "Scissors")]' );
await page.waitFor( '//div[contains(., "Scissors clicked!")]' );

It supports three “browsers” but not as you know them

Q: Does Playwright support new Microsoft Edge?

The new Microsoft Edge browser is based on Chromium, so Playwright supports it.

https://github.com/microsoft/playwright#q-does-playwright-support-new-microsoft-edge

Playwright supports three “browsers” but not as you know them. I’d say it supports three rendering engines (Chromium, WebKit & Gecko) rather than Browsers as you can only use the (somewhat modified) browsers that come bundled with Playwright over using an already installed browser on your operating system (like Selenium does). This makes it easier to ensure consistency of test runs since the library is bundled with the browsers, but there are some risks your tests could pass on the bundled browsers but fail on “real” browsers. I would say that the claim it supports running on Microsoft Edge is a little misleading.

I’m unsure of CircleCI Support for WebKit and Firefox

I was able to get my tests running against Chromium on CircleCI using the same configuration as Puppeteer, however I couldn’t get the WebKit or Firefox tests to run on CircleCI even when having the default CircleCI browsers installed. I didn’t want to invest the time, but it is probably due to some headless Linux dependencies missing which could be solved in the project config.

Conclusion

If the only thing Playwright did better than Puppeteer was also supporting WebKit and Gecko then I wouldn’t suggest using it over Puppeteer, since Puppeteer is closely aligned with Chromium, and I’m going to run my tests solely in Chrome/Chromium anyway. I don’t believe in running the same e2e tests in multiple browsers: the maintenance overhead outweighs the benefits in my experience.

However, Playwright offers a much nicer BrowserContext concept, and the xpath support is much nicer (although I rarely use xpath expressions anyway).

If anything I am hoping Puppeteer adds support for BrowserContexts – I’ve raised a feature request here so feel free to comment on it if you think it would be a good idea.

All the sample code is available here: https://github.com/alisterscott/playwright-demo

npm ci

I recently discovered npm ci which you can use instead of npm install when running a node project on continuous integration (CI) system and want to install your npm dependencies. It does this in a more lightweight, more CI friendly way.

If you use npm test to run your tests, this can be shortened to npm t (much like npm i is npm install), and therefore you can run npm cit to install dependencies and run tests in CI.

Running Mocha e2e Tests in Parallel

I recently highlighted the importance of e2e test design. Once you have well designed e2e tests you can start running them in parallel.

There are a couple of approaches to scaling your tests out to be run in parallel:

  1. Running the tests in multiple machine processes;
  2. Running the tests across multiple (virtual) machines;

These aren’t mutually exclusive, you can run tests in parallel processes across multiple virtual machines – we do this at Automattic – each test run happens across two virtual machines, Docker containers on CircleCI, each of which runs six processes for either desktop or mobile responsive browser sizes depending on the container.

I have found running tests in multiple processes gives best bang for buck since you don’t need additional machines (most build systems charge based on container utilisation) and you’ll benefit from parallel runs on a local machine.

We write our e2e tests in WebDriverJs and use Mocha for our test framework. We currently use a tool called Magellan to run our e2e tests in separate processes (workers), however the tool is losing Mocha support and therefore we need to look at alternatives.

I have found that mocha-parallel-tests seems like the best replacement – it’s a drop in replacement runner for mocha tests which splits test specification files across processes available on your machine you’re executing your tests on – you can also specify a maximum limit of processes as the command line argumentĀ --max-parallel

There is another parallel test runner for mocha: mocha.parallel – but this requires updating all your specs to use a different API to allow the parallelisation to work. I like mocha-parallel-tests as it just works.

I’ve updated my webdriver-js-demo project to use mocha-parallel-tests – feel free to check it out here.

Using async/await with WebDriverJs

We’ve been using WebDriverJs for a number of years and the control flow promise manager that it offers to make writing WebDriverJs commands in a synchronous blocking way a bit easier, particularly when using promises.

The problem with the promise manager is that it is hard to understand its magic as sometimes it just works, and other times it was very confusing and not very predictable. It was also harder to develop and support by the Selenium project so it’s being deprecated later this year.

Fortunately recent versions of Node.js support asynchronous functions and use of the await command which makes writing WebDriverJs tests so much easier and understandable.

I’ve recently updated my WebDriverJs demo project to use async/await so I’ll use that project as examples to explain what is involved.

WebDriverJs would allow you to write consecutive statements like this without worrying about waiting for each statement to finish – note the use of test.it instead of the usual mocha it function:

test.it( 'can wait for an element to appear', function() {
	const page = new WebDriverJsDemoPage( driver, true );
	page.waitForChildElementToAppear();
	page.childElementPresent().then( ( present ) => {
		assert( present, 'The child element is not present' );
	} );
} );

When you were waiting on the return value from a promise you could use a .then function to wait for the value as shown above.

This is quite a simple example and this could get complicated pretty quickly.

Since the promise manager is being removed, we need to update our tests so they continue to execute in the correct order. We can make the test function asynchronous by adding the async prefix, remove the test. prefix on the it block, and add await statements every time we expect a statement to finish before continuing:

it( 'can wait for an element to appear', async function() {
	const page = new WebDriverJsDemoPage( driver, true );
	await page.waitForChildElementToAppear();
	assert( await page.childElementPresent(), 'The child element is not present' );
} );

I personally find this much easier to read and understand, less ‘magic’, but the one bit that stands out is visiting the page and creating the new page object. The code in the constructor for this page, and other pages, is asynchronous as well, however we can’t have an async constructor!

export default class BasePage {
	constructor( driver, expectedElementSelector, visit = false, url = null ) {
		this.explicitWaitMS = config.get( 'explicitWaitMS' );
		this.driver = driver;
		this.expectedElementSelector = expectedElementSelector;
		this.url = url;

		if ( visit ) this.driver.get( this.url );

		this.driver.wait( until.elementLocated( this.expectedElementSelector ), this.explicitWaitMS );
	}
}

How we can get around this is to define a static async function that acts as a constructor and returns our new page object for us.

So, our BasePage now looks like:

export default class BasePage {
	constructor( driver, expectedElementSelector, url = null ) {
		this.explicitWaitMS = config.get( 'explicitWaitMS' );
		this.driver = driver;
		this.expectedElementSelector = expectedElementSelector;
		this.url = url;
	}

	static async Expect( driver ) {
		const page = new this( driver );
		await page.driver.wait( until.elementLocated( page.expectedElementSelector ), page.explicitWaitMS );
		return page;
	}

	static async Visit( driver, url ) {
		const page = new this( driver, url );
		if ( ! page.url ) {
			throw new Error( `URL is required to visit the ${ page.name }` );
		}
		await page.driver.get( page.url );
		await page.driver.wait( until.elementLocated( page.expectedElementSelector ), page.explicitWaitMS );
		return page;
	}
}

In our Expect and Visit functions we call new this( driver ) which creates an instance of the child class which suits our purposes. So, this means our spec now looks like:

it( 'can wait for an element to appear', async function() {
	const page = await WebDriverJsDemoPage.Visit( driver );
	await page.waitForChildElementToAppear();
	assert( await page.childElementPresent(), 'The child element is not present' );
} );

which means we can await visiting and creating our page objects and we don’t have any asynchronous code in our constructors for our pages. Nice.

Once we’re ready to not use the promise manager we can set SELENIUM_PROMISE_MANAGER to 0 and it won’t use it any more.

Summary

The promise manager is being removed in WebDriverJs but using await in async functions is a much nicer solution anyway, so now is the time to make the move, what are you awaiting for? šŸ˜Š

Check out the full demo code atĀ https://github.com/alisterscott/webdriver-js-demo

Puppeteer for Automated e2e Chrome Testing in Node

I recently noticed the new Google Chrome project Puppeteer:

Puppeteer is a Node library which provides a high-level API to controlĀ headlessĀ Chrome or Chromium over theĀ DevTools Protocol. It can also be configured to use full (non-headless) Chrome or Chromium.

As someone who only runs WebDriver tests in Google Chrome anyway, this looks like a promising project that bypasses WebDriver to have full programmatic control of Google Chrome including for automated end-to-end (e2e) tests.

The thing I really love about this is no Chromedriver dependency and how installing the library installs Chromium by default which can be controlled headlessly with zero config or any other dependencies.

You can even develop scripts using this playground.

I set up a (very basic) demo project that uses Mocha + Puppeteer and it runs on CircleCI with zero config. Awesome.

AMA: JS vs Ruby

Butch Mayhew asks…

I have noticed you blogging more about JS frameworks. How do these compare to Watir/Ruby? Would you recommend one over the other?

My response…

I had a discussion recently with Chuck van der Linden about this same topic as he has a lot of experience with Watir and is now looking at JavaScript testing frameworks like I have done.

Some Background

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’tĀ make long term sense forĀ WordPress.comĀ where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

Whilst I see merit in both views: I still think having your automated acceptance tests in the same language as your application leads to better maintainability and adoptability.

I still think writing automated acceptance tests in Ruby is much cleaner and nicer than JavaScript Node tests, particularly as Ruby allows meta-programming which means page objects can be implemented really neatly.

The JavaScript/NodeJS landscape is still very immature where people are using various tools/frameworks/compilers and certain patterns or de facto standards haven’t really emerged yet. The whole ES6/ES2015/ES2016 thing is very confusing to newcomers like me, especially on NodeJS where some ES6+Ā featuresĀ are supported, but others require something like Babel to compile your code.

But generally with the direction ES is going, writing page objects as classes is much nicer than using functions for everything as in ES5.

Whilst there’s nothing I have found that is betterĀ (or even as good) in JavaScript/Mocha/WebDriverJS than Ruby/RSpec/Watir-WebDriver, I still think it’s a better long term decision for WordPress.com to use the JavaScript NodeJS stack for our e2e tests.