Real vs Headless Browsers for Automated Acceptance Tests

When I was comparing/evaluating JavaScript browser automation tools for Automattic, I noticed that a lot of the tools were headless only: either they use PhantomJS/SlimerJS or have their own headless browser engine (like Zombie.js).

I must admit I am not a fan of headless browsers for a number of reasons:

  • Headless browsers aren’t representative of real users: a bug found using a headless browser like PhantomJS may actually not be a bug. For example, running tests against in PhantomJS 2.0 throws an error on the login page – this doesn’t happen in a real browser like Chrome (see screenshot below). Our customers don’t use headless browsers.
  • Headless browsers aren’t faster than real browsers: contrary to popular belief, headless browsers aren’t faster than real browsers. Most of the time that it takes to load a page is the actual network content: html, JavaScript, images and CSS. This still has to be loaded by a headless browser, it just doesn’t have to render the pixels to a screen. Browser vendors like Google Chrome and Mozilla Firefox invest a lot of time/effort/money to make their browsers and JavaScript engines super fast which we can leverage by running tests in them.
  • Headless browsers make it harder to write/debug tests: since the browser is invisible, you can’t visually interact with a browser except by taking/saving screenshots [1]. This is a lot easier in a real browser which you can interact with via a REPL.
  • Real browsers can run headlessly anyway: Popular CI systems like CircleCI and TravisCI support running real browsers (Chrome, Firefox) on headless Linux slaves with zero configuration. The best of both worlds :)
The login page error when running via PhantomJS 2.0
The login page error when running via PhantomJS 2.0

At the end of the day, I much prefer browser automation tools that support driving real browsers for test and development, which are run headlessly through CI. This leads to representative test results and avoidance of ‘phantom’ bugs that aren’t reproducible in real browsers.

1. Some tools have some debug capability but it’s not the same as having a real browser with developer tools, a DOM, and a JavaScript console

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

2 thoughts on “Real vs Headless Browsers for Automated Acceptance Tests”

  1. Nice article.

    I hear from plenty of people who would disagree and want to run tests headless for the sake of speed. I always argued that I do not believe it is a ‘real world test’. Can you tell me what you think the advantage would be though? You must think there is one since you mention ‘The best of both worlds’ while running with CI.



    1. Thanks, I should have been clearer.
      Running headlessly on CI is good as you’ll be running on servers which are often Linux and don’t have displays, so you can’t really use a desktop browser.
      Saves having the buy physical machines with KVM switches which I have seen done in the past.


Comments are closed.