Should you raise bugs you don’t know how to reproduce?

At Automattic we’re always dogfooding; we’re always testing as we develop and therefore we’re always finding bugs.

When I come across one of these bugs I try to capture as much as I can about what I was doing at the time when it happened, and also capture some evidence (a screenshot, visual recording, error messages, browser console) of it happening. But even then on occasion I can’t work out how to reproduce a bug. 

This doesn’t mean the bug isn’t reproducible; it’s just that because there’s so many factors in the highly-distributed, multi-level systems that we develop and test, this means I don’t know how to reproduce these exact circumstances, therefore I don’t know how to reproduce this bug.

The exact set of circumstances that may cause a bug are never entirely reproducible; just like you can never live the exact same moment of your life ever again.

So, when this bug occurred, perhaps the server was under a particular level of load at that exact moment in time. Or one of the load-balancers was restarting, or maybe even my home broadband speed was affected by my children streaming The Octonauts in the next room.

So it can seem almost like a miracle when you do know how to reproduce a bug.

But what do you do with all of those bugs that you haven’t been able to reproduce?

Like many things, I am of two minds about what to do with these bugs.

Bug reports without instructions on how to reproduce them are much more difficult to action since there’s no hypothesis listed to work against.

I recently came across a bug which bothered me as I couldn’t reproduce it. So I didn’t raise the bug since I couldn’t find what I thought was enough information to make the bug report actionable, and testable when it’s ‘fixed’.

A couple of weeks later I heard reports of this bug happening to customers. I quickly raised it without reproducible steps but with a hunch of what I thought might be causing it to happen.

This happened to be enough information for a developer to look through the code and find something that wasn’t quite right in particular circumstances: there was a race condition.

Trying to catch the race condition (via devopsreactions)


I was able to use Chrome’s inbuilt throttling tools to reproduce a similar issue and therefore I could verify the behaviour both before and after the fix created.

This exercise has taught me to raise every bug that I find (unless it already exists as a bug report, of course), with as much detail as I can at the time, even if that means I don’t know how to specifically reproduce it.

There is a good chance that there may be enough detail specified to allow someone familiar with the codebase to do some targeted investigation looking for potential ‘code smells’ that could be causing your issue, and fix it with details of what was wrong.

What’s your experience in raising bugs you don’t know how to reproduce?

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

12 thoughts on “Should you raise bugs you don’t know how to reproduce?”

  1. Interesting post! It’s not something I’ve thought about really since I always raise bugs even if they are not reproducible. I always note that I am unable to reproduce, and always go back and retest. I generally talk to the developers as well since they might have an idea of whats causing it. Basically it’s about gathering as much information about the issue as possible to give the devs the best chance of finding and fixing it.

    We had one bug, where you had to wait for a request to be made, count to 10 (not 10 seconds, literally 10) and then press a save button, and it would cause a furor of errors. Of course when first observing it, it looked like the issue just happened randomly, but having both test and dev investigate led to the root of the problem.

    Liked by 2 people

  2. Well timed article Al. I had few instances this week alone where I spotted a defect but couldn’t reproduce. I quickly logged the defects with information such as what I was doing at that time and logs from browser console. Devs looked at the defects (exceptions details) and the responses were like 1)yeah we’ve seen that one before but couldn’t consistently replicate and 2) Yup, that info is enough to fix the issue. For the one that couldn’t be replicated we worked together simulating production behavior (in terms of test data, concurrency, frequent data updates to the page) and found the root cause to be a race condition and a P1 issue.

    always raise the defects so it can be worked out as a team.

    Liked by 1 person

  3. Raise the nonreproducable bugs. As you noted, although you weren’t able to identify the reproduction pattern, that doesn’t mean that nobody can. More than once I’ve found an issue, been unable to pin down the exact scenario, but once I’ve talked with a dev they knew something, or had a hunch that was able to identify and correct the problem.

    Liked by 1 person

  4. Absolutely,we should be logging non-reproducible bugs !
    When I used to test embedded RF radio terminals, a significant test method we employed was field tests. Apart from being an excellent validation tool , field tests were also a major source of bugs/issues which were difficult to recreate back at base. There were several variables at play ,RF and physical conditions,state of the infrastructure at the time of problem, the way terminal was configured .
    However I always used record the “only happened once” issues because as I(and the Team) learnt how to
    1. Be aware of and capture various variables that influence creation of that problem
    2. Started finding ways to recreate those states (either back at base or in the field)
    the “only happened once ” turned into reproducible problems
    This took months of learning & rigor but was a fun and rewarding process :)
    ~ Sunjeet

    Liked by 1 person

  5. Hi Alister Scott , nice post .. one question , while testing , suddenly our machine crashes , how to send crashlogs to developers , ohh!, in my machine the software crashes , i need to report to developer , how to send reports for crashlogs , if suppose my machine is windows or Mac , could you give the answer for it ..


  6. I wonder how did this particular bug made it to triage and to sprint. For bugs with non-reproducible steps our triage process usually de-prioritizes them and they won’t get to sprint and hence developers wouln’t look at them (unless you pull in favors with one of the devs)


  7. Yeabsolutely, we should all be reporting every issue that we would be seeing.

    There was this skate game, which I was testing and ran into a major issue and could not repro it again. And, we could not let that go – both dev and testing team sat together and we tried all kinds of debug methods including testing a game in debug mode, had a look at the in-game coordinates, however could not zero in on the exact cause. After about four weeks’ or so, we got it ! Apparently, there was a challenge and it had about eight objectives, if i remember correctly, and we have to complete five of those objectives in a particular orderely fashion, then during the sixth objective, there it was….!! You can imagine the number of combinations that we had to do before finding the root – simple math permutations and combinations, isn’t it? at the end it was all fun.

    Liked by 1 person

Comments are closed.