How do you gain confidence with the testing pyramid mentioned in this post? Unit testing is under the developers and you have only just enough e2e tests.
That is a great question Lucy. There’s a few different ways that we gain confidence from the way we do development and testing at WordPress.com.
Developers are responsible for writing unit tests for WordPress.com. Our ‘flow patrol’ testing squad doesn’t have a big involvement with these unit tests as they’re primarily a tool to speed development and give developers confidence in their own changes.
But developers at WordPress.com require at least one peer review on a pull-request they are working on, and they’re also fully responsible for the deploying their own changes into Production. This happens many times per day.
So the combination of developer-led unit tests, mandatory peer reviews and developers having full accountability for their own changes means that as a tester I have high confidence in things going well, which isn’t to say every single change goes well, but most do and we quickly revert those that don’t.
We supplement these developer-led tests with tester-led automated end-to-end (e2e) tests which we develop in collaboration with different developers to ensure our critical user flows are functioning all the time, particularly with such frequent releases.
Which leads me to my second point, we don’t have a tester test every change before it is released to Production, there are too many changes and our squad is too small. So our squad practices and advocates continuous dogfooding: everyone in Automattic testing our own software through continual use and exploration.
There are a couple of ways we achieve continuous dogfooding: most of our internal company communication is done using WordPress.com sites (called P2s), so by communicating internally we’re also using and testing the products and features we’re developing.
And most, if not all, staff also write at least one WordPress.com blogs; such as this one, and I, like many others, have some other non-work-related blogs I also write on WordPress.com. So we’re encouraged to blog, and blogging on WordPress.com is also testing WordPress.com (in the real world).
Finally, to cover scenarios that we typically wouldn’t do on a day-to-day basis, or browsers/devices we wouldn’t use on a day-to-day basis, such as testing our New User eXperience (NUX), we built a Slack ‘testbot’ that we use to encourage whoever feels like testing something to easily be able to do that. For example:
alisterscott: I am looking for a flow to test testbot: @alisterscott: try signing up for a new WordPress.com site for a hairdressing salon in Mozilla Firefox
Often during a dogfooding session we will take visual recordings (vizrecs) and share these internally for discussion and reference purposes.
So, to answer your original question, whilst there is a void of integration/API tests that fit somewhere between unit tests and full stack end-to-end tests, as testers we have a lot of confidence in what we release.
It recently occurred to me that I was missing the eye of providence from the previous hand-drawn pyramid, so this gives me a good opportunity to re-introduce the eye representing the practices we do that ensure we’re continually ‘looking over’ everything we do, including our automated tests, to make sure we have good quality products and a great user experience.