AMA: bdd frameworks, api test tools & unit testing responsibility

Swapnil Waghmare asks…

Are BDD frameworks like Specflow, Cucumber better for E2E tests?

My response…

There’s two ways I can interpret this question (my additions are in bold).

Firstly:

Are BDD frameworks that use ‘Given/When/Then’ feature files like Specflow, Cucumber  better than frameworks that use ‘describe/it’ blocks like RSpec and Mocha for e2e tests?

As I previously explained, Automattic’s unit tests are written in Mocha, so that was a logical choice for writing e2e tests as there is a lot of familiarity of it within Automattic, which will hopefully mean more developers are interested in the e2e tests we are writing using Mocha/WebDriverJs.

There are some challenges with writing end-to-end tests with Mocha (mainly that Mocha tests are all independent so will continue to run if a previous step in the scenarios fails) so I haven’t completely ruled out investigating a move to Cucumber at some point for the e2e tests.

Secondly:

Are BDD frameworks like Specflow, Cucumber better for E2E tests than using them for automated integration, component or unit tests?

I think you could write integration tests or even unit tests in Given/When/Then format as most unit tests follow the same arrange/act/assert pattern anyhow which is exactly what Given/When/Then is.

Keep in mind there is overhead in maintaining all the step definitions and feature files for Cucumber/Specflow that give you non technical readability so if you don’t require that readability it is probably overkill. But a personal preference nonetheless.

Swapnil Waghmare also asks…

Which tools have you used for API testing, which ones would you recommend?

My response…

At the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

I try to keep things simple, so in the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.

Swapnil Waghmare finally asks…

What is the reason behind writing E2E tests using JavaScript? Isn’t any Object oriented language a better choice for writing E2E tests? Should QA’s write Unit tests?

My response…

 

Firstly, JavaScript is actually object oriented, it’s just not class-based object oriented like Java, C# or Ruby. The newer versions of JavaScript, called ECMAScript, or ESScript are more class-based object oriented.

I’ve actually already answered why I chose JavaScript in a previous answer, so I’ll summarise that here and you can read the rest in that answer.

WordPress.com built an entirely new UI for managing sites using 100% JavaScript with React for the main UI components. I am responsible for e2e automated tests across this UI, and whilst I originally contemplated, and trialled even, using Ruby, this didn’t make long term sense for WordPress.com where the original WordPress developers are mostly PHP and the newer UI developers are all JavaScript.

As for whether QA’s should write unit tests? No, I don’t think so, as I believe unit tests should drive software development, and writing code by writing unit tests is much easier than trying to add unit tests after by someone else, as the original code will not likely be very testable as it wasn’t written with testability in mind. One benefit of code written with unit tests at the time is that it will mostly be better code as the tests are consuming your API that you are developing.

AMA: integration tests to block pull requests?

ccy asks…

Do you think a small set of stable integration tests (e.g.: API tests that send real HTTP requests to the endpoint or selenium based web UI tests with or without mocked backend API calls) should be the part of the tests for developers PR (pull request) verification ?

I am thinking of where such integration test could make the biggest impact in the process. Usually, the only set of tests that will be taken seriously are the ones that block code merge. All the other tests would eventually end up obsolete or maintained by dedicated QA folks.

My response…

The only tests that stop a code merge for WordPress.com at the moment are the unit tests that the developers are responsible for writing. As explained previously, I am not a huge fan of integration/API tests since they tend to have blurred responsibilities and I am instead keen on lots of unit tests (using test doubles) and a handful of realistic end-to-end automated tests that use your system in the same way your users do (typically via a web browser).

We don’t currently run our WordPress.com e2e tests against individual pull requests, these are run on merge into master/deploy, as well as periodically against production. We do have a plan to run these against individual PRs in the future.

To answer your question, a small set of realistic e2e automated tests run against each pull request is a good idea to have confidence that the pull request isn’t fundamentally breaking your system in a way that isn’t detected via individual unit tests. It also shares the ownership/responsibility for keeping these e2e test up to date, as typically they cut across multiple product teams.

AMA: RestAssured or similar tools for integration tests?

Jay asks…

Have you used RestAssured or similar tools for integration tests? Specifically, where a test URL is built with various parameters, sent, and the response data (XML/JSON) is finally verified.

If so, have any interesting patterns or anti-patterns emerged from this work?

I enjoy your blog a lot. It has really helped inform my work.

My response…

I haven’t used RestAssured or similar tools for integration tests. It looks like a nice DSL/syntactic sugar around consuming/validating REST services.

If I were writing a lot of integration tests in a language like Java I think this would be worthwhile, but at the moment my main focus is on automated unit test and automated e2e test coverage in JavaScript.

In the past when I’ve written REST integration tests I’ve just called them using in built libraries, and have used Postman quite a lot for manual testing and debugging.