My Thoughts on Cypress.io

Run asks…

I’ve just started using Cypress.io. As someone who initially learned about testing conventions years ago through your blog, cypress seems to want to burn all old conventions to the ground in a way that immediately turned me off. After playing with it a bit and watching a talk by one of its founders, I’m a little more convinced now. It’s a great tool. Do you have an opinion on Cypress yet, and do you think old testing conventions are becoming obsolete thanks to much better reporting tools around testing?

There certainly seems to be a lot of hype and enthusiasm around Cypress.io. I recently saw another (rather evangelical) talk about Cypress.io here in Brisbane so I thought it was time to share my thoughts.

What exactly is Cypress.io?

Looking at Cypress.io it is described as a “JavaScript End to End Testing Framework” and “Fast, easy and reliable testing for anything that runs in a browser“. Some other descriptions on Cypress.io include “A complete end-to-end testing experience.” and “Cypress is the new standard in front-end testing that every developer and QA engineer needs.” 

Screen Shot 2019-07-06 at 2.54.48 pm.png

What is end-to-end testing?

I believe there are some specific traits that define what automated end-to-end (e2e) tests are:

  • They test a complete user flow through an application from start to finish (end-to-end)
  • They test how a real user would use a using a fully deployed system
  • They test the happy-path of the most commonly used scenarios, avoiding error validation or edge-cases

End-to-end tests are expensive to maintain and execute so the widely accepted view is to have as few of these as possible for your application, which means avoiding things like negative and error validation testing during end-to-end tests as these things can be tested much more easily and quickly in isolation as other types of automated tests (unit, component or integration).

Is Cypress.io a framework for writing end-to-end tests?

Despite its strongly worded marketing material, I don’t believe Cypress.io has been designed as an end-to-end (e2e) testing framework. I believe this was confirmed by Brian Mann at Assert(js) in the first half of the “I see your point, but…” presentation:

“You should always strive to test pages in total isolation – everything becomes faster, less coupled, and you won’t lose a single point of confidence that it’s all working together correctly. You don’t need to limit yourself trying to act and replicate everything a user would do.”

Screen Shot 2019-07-06 at 2.28.05 pm

I believe what Brian is referring to aren’t end-to-end tests but rather component tests. Brian showed using Cypress.io to test a login page where he wrote 6 isolated test specs to test login validation:

Screen Shot 2019-07-06 at 2.34.36 pm

What makes the question about whether Cypress.io is an e2e testing framework even more confusing is during the second half of the same presentation, Gleb Bahmutov, also from Cypress.io, states:

“Brian showed how we think about end-to-end testing. To us end-to-end should do the same things that a human would do to a fully deployed system. Right, that means real browser, real interactions, no shortcuts…”

Screen Shot 2019-07-06 at 2.45.19 pm

Also, confusingly, he stated that e2e test tools can do a pretty good job in unit testing:

Screen Shot 2019-07-06 at 2.45.38 pm

So what is Cypress.io then?

I now consider Cypress.io to be a strongly opinionated framework suited to writing isolated automated web component tests. I’m not sure why it is marketing as an e2e testing tool, and trying to compare itself to something like Selenium, which isn’t a component testing tool.

If you wanted to write isolated automated web component tests then Cypress.io would be worth a look at, since it offers many features to help you. However for true end-to-end purposes I think the limitations outweigh the benefits. The open source WordPress Gutenberg editor project tried Cypress.io for quite a while but ultimately found it too limiting and switched to Puppeteer.

Some things to considering when trying to use Cypress.io for true end-to-end testing

Even though Cypress.io is demonstrated as a way to write isolated web component tests, if you still want to use it to write true end-to-end tests then there’s some tradeoffs you need to consider.

Screen Shot 2019-07-06 at 2.55.49 pm

Despite the claims that “Cypress works on any front-end framework or website” and “Fast, easy and reliable testing for anything that runs in a browser” there are quite a few scenarios where you can’t use Cypress – for example if your front-end or website uses iFrames you can’t use Cypress.io.

iFrames

Cypress.io itself uses iFrames to inject itself into the browser so supporting iFrames whether on the same domain or cross domain aren’t supported with an open issue since 2016. At WordPress.com we used iFrames for the WordPress site customizer so it wouldn’t be possible to write an end-to-end test for WordPress.com using Cypress.io. In my current role I work on a web application which actually a series of React micro frontends which are rendered in iFrames within a web container, so we also can’t use Cypress.io for end-to-end testing.

Native browser events like file uploads and downloads

Things like uploading and downloading files that are trivial to do in WebDriver are either difficult or not supported in Cypress.io. Even things as simple as using the tab key isn’t supported.

Parallelism

Whilst Cypress is often promoted as a free and open source project there are certain features that are only available when running your tests in “record” mode with the Cypress Dashboard Service, which allows 500 tests (it blocks) to run in parallel per month before needing to pay for it. Parallel execution is one of these features, so you can’t even run tests in parallel locally without recording your results to the dashboard service.

The biggest issue I see with Cypress.io parallelization is that it is machine based, not process based:

parallelization-diagram.4c4cbac6

In this example, the CI container costs triple when each CI machine should be more than capable of running multiple Chromium browser sessions.

At WordPress.com we used CircleCI and we able to have up to 12 headless Chrome browsers using WebDriverJs in parallel on each CircleCI container, and across 3 containers this allowed 36 e2e tests running in parallel. To get the same result using Cypress.io would mean paying for 36 CircleCI containers.

Running e2e tests written in other e2e testing tools in parallel machine processes can be quite easy as I’ve explained previously on this blog.

There are times when running via the command line isn’t the same as running via the GUI

I noticed this when writing a demo cypress spec which would pass when running in the Cypress GUI runner but fail on the command line, which looking at these comments doesn’t seem uncommon.

    it( 'ignores alerts when leaving the page', function() {
        cy.visit('http://webdriverjsdemo.github.io/leave'); 
        cy.get('#homelink').click();
        cy.contains('WebDriverJs Demo Page').should('be.visible');
    } );

GUI Runner:

Screen Shot 2019-07-06 at 4.33.53 pm

Command Line:

Screen Shot 2019-07-06 at 4.37.17 pm

Logging in

One of the key messages of the first video was demonstrating you can log in without using the UI for subsequent tests which speeds things up. This is a good idea, but isn’t unique to Cypress.io: at WordPress.com we re-used a single login cookie across multiple e2e tests using WebDriverJS – the code is here.

Summary

From a distance Cypress looks like a polished tool for automated testing – I just think it’s incorrectly marketed as an end-to-end testing tool when it’s really only good for component testing. There are too many limitations in the tool in acting like a real user to use it to create true end-to-end automated tests.