AMA: gaining confidence without integration tests

Lucy asks…

How do you gain confidence with the testing pyramid mentioned in this post? Unit testing is under the developers and you have only just enough e2e tests.

test pyramid watirmelon(2)

My response…

That is a great question Lucy. There’s a few different ways that we gain confidence from the way we do development and testing at

Developers are responsible for writing unit tests for Our ‘flow patrol’ testing squad doesn’t have a big involvement with these unit tests as they’re primarily a tool to speed development and give developers confidence in their own changes.

But developers at require at least one peer review on a pull-request they are working on, and they’re also fully responsible for the deploying their own changes into Production. This happens many times per day.

So the combination of developer-led unit tests, mandatory peer reviews and developers having full accountability for their own changes means that as a tester I have high confidence in things going well, which isn’t to say every single change goes well, but most do and we quickly revert those that don’t.

We supplement these developer-led tests with tester-led automated end-to-end (e2e) tests which we develop in collaboration with different developers to ensure our critical user flows are functioning all the time, particularly with such frequent releases.

Which leads me to my second point, we don’t have a tester test every change before it is released to Production, there are too many changes and our squad is too small. So our squad practices and advocates continuous dogfooding: everyone in Automattic testing our own software through continual use and exploration.

There are a couple of ways we achieve continuous dogfooding: most of our internal company communication is done using sites (called P2s), so by communicating internally we’re also using and testing the products and features we’re developing.

And most, if not all, staff also write at least one blogs; such as this one, and I, like many others, have some other non-work-related blogs I also write on So we’re encouraged to blog, and blogging on is also testing (in the real world).

Finally, to cover scenarios that we typically wouldn’t do on a day-to-day basis, or browsers/devices we wouldn’t use on a day-to-day basis, such as testing our New User eXperience (NUX), we built a Slack ‘testbot’ that we use to encourage whoever feels like testing something to easily be able to do that. For example:

alisterscott: I am looking for a flow to test
testbot: @alisterscott: try signing up for a new site for a hairdressing salon in Mozilla Firefox

Often during a dogfooding session we will take visual recordings (vizrecs) and share these internally for discussion and reference purposes.

So, to answer your original question, whilst there is a void of integration/API tests that fit somewhere between unit tests and full stack end-to-end tests, as testers we have a lot of confidence in what we release.

It recently occurred to me that I was missing the eye of providence from the previous hand-drawn pyramid, so this gives me a good opportunity to re-introduce the eye representing the practices we do that ensure we’re continually ‘looking over’ everything we do, including our automated tests, to make sure we have good quality products and a great user experience. Test Pyramid(2)

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

3 thoughts on “AMA: gaining confidence without integration tests”

  1. Thanks very much for your reply.
    Unit+end2end is not uncommon, but I think the most important part is developers own the changes and product.
    Dogfooding is not appliable in most companies.
    I don’t quite understand the testbot though. Would you mind explain or blog about it?
    And the idea of testing end-to-end user flows also inspires me. Thanks to that as well!

    Liked by 1 person

Comments are closed.