Feature Toggles for Automated e2e Tests

feature-toggles-1

Feature toggles aren’t just for production code. Feature toggles are also a powerful technique to change the behaviour of your automated end-to-end tests without changing code.

Most modern web applications use feature toggles, also known as feature switches or feature flags, as a way to modify system behaviour without changing code. Using feature toggles is a safe way to progressively introduce new behaviour to a system by enabling new features for certain environments (eg. non-production), or for certain users, and are often combined with A/B Testing to test new features with users.

I’ve previously covered how to write automated e2e tests that handle A/B testing but how can you utilise feature toggles in your own automated e2e tests?

At Automattic we use feature toggles in our automated e2e tests for two main reasons:

  1. Some of our e2e tests have external dependencies and we use feature toggles to be able to quickly disable these tests should there an issue with a dependency. For example some of our tests depend on the Mailosaur API and if there’s a Mailosaur outage we use a feature toggle to disable these tests without making a code change; and
  2. Some of our e2e tests cover features only work when an application feature toggle is set. For example we recently released a mobile editor redesign as a feature toggle and we created a feature toggle in our automated e2e tests for this new production feature.

How do you actually do this?

We use Mocha/WebDriverJs and to implement feature toggles we simply add an entry to our config.json file and add conditional code

For example, if we add

"useNewMobileEditor": true

to our config file, we can easily override useNewMobileEditor as an environment variable in CircleCI to set it to whatever we desire.

The only other thing to do is make our test flows support the new behaviour. This is usually done by putting conditional statements in our code like:

if ( config.get( ‘useNewMobileEditor’ ) === true ) {
// do new thing
} else {
// do usual thing
}

Caveat Emptor

Whilst feature toggles are very powerful, they are also dangerous if overused. Too many feature toggles leads to unnecessary complexity of code and system behaviour which can lead to non deterministic test results.

Like feature toggles in production code, I suggest you limit these as much as possible and as soon as a feature is live and stable you should actively go and remove the feature toggle and the conditional code to make your codebase simpler and easier to understand.

******

Do you use feature toggles in your automated end-to-end tests? How have you found using these?

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

3 thoughts on “Feature Toggles for Automated e2e Tests”

  1. Nice post Al. We also use feature toggles in our tests. Website exposes an api call that has the feature toggle information. Automated tests use that api call to determine which features are turned on and off and run tests accordingly.

    Set up is done in such a way that, first step in a test is to check if the feature, the test is testing, is turned on/off and continue/exit accordingly. We have separate tests for the new thing and usual thing.

    Problem with this approach (something we have accepted) is that the test will show as passed if the feature is turned off. Ideally we would like to use an Ignore-If attribute on the test method to check the feature toggle status and ignore the tests if the feature is turned off, however ms unit framework doesn’t seem to have that feature and haven’t had much time to look into a custom ms test attribute to provide that ability.

    Liked by 1 person

  2. We use env vars (injected by Jenkins), and just abandon the test module before the tests are declared if turned off:

    if (process.env.SOMETHING === "false") {
        return;
    }
    
    describe('a feature', function() {
    ...
    

    The main problem is that people aren’t as keen on turning them back on, as they are at turning them off :(

    Liked by 1 person

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s