AMA: flakiness after deployment?

Butch Mayhew asks…

When it comes to Flaky tests, we are a .net shop and find that after a deployment pages have long load times the first time they are touched. Any tips/ideas on how to best approach this issue? (We do re-run our tests because of this). Great talk btw at GTAC, I was a remote realtime watcher!

My response…

At the last company I worked for we had the exact same problem, and we were a .NET shop also 🤔

How we solved this problem was to have a specific automated test that we called ‘warmup’ that hit our main page (with a large dynamically loaded menu) and it had a very long (1 or 2 minute) explicit wait for the content to appear. It also wouldn’t fail even if the content was still loading as we weren’t testing our system per say, just warming it up 🔥

We ran this sequentially before all our acceptance tests ran in parallel, and by having this single test to warm up our system it meant our actual acceptance tests were much more likely to be deterministic and not require any re-runs 😎

Author: Alister Scott

Alister is a Software Quality Engineer for Automattic.