AMA: integration tests to block pull requests?

ccy asks…

Do you think a small set of stable integration tests (e.g.: API tests that send real HTTP requests to the endpoint or selenium based web UI tests with or without mocked backend API calls) should be the part of the tests for developers PR (pull request) verification ?

I am thinking of where such integration test could make the biggest impact in the process. Usually, the only set of tests that will be taken seriously are the ones that block code merge. All the other tests would eventually end up obsolete or maintained by dedicated QA folks.

My response…

The only tests that stop a code merge for at the moment are the unit tests that the developers are responsible for writing. As explained previously, I am not a huge fan of integration/API tests since they tend to have blurred responsibilities and I am instead keen on lots of unit tests (using test doubles) and a handful of realistic end-to-end automated tests that use your system in the same way your users do (typically via a web browser).

We don’t currently run our e2e tests against individual pull requests, these are run on merge into master/deploy, as well as periodically against production. We do have a plan to run these against individual PRs in the future.

To answer your question, a small set of realistic e2e automated tests run against each pull request is a good idea to have confidence that the pull request isn’t fundamentally breaking your system in a way that isn’t detected via individual unit tests. It also shares the ownership/responsibility for keeping these e2e test up to date, as typically they cut across multiple product teams.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.