Should you close old bugs?

Do you actively close bugs because they reach a certain age?

One of the (many) things I love about Automattic is the attention that is given to bug triage. Bug triage is the habit of continually grooming our bug lists to ensure they are constantly relevant, updated and reflective of the current state of our products. A benefit of this is that an up-to-date and prioritized bug list translates directly into a backlog of maintenance work items for a product development team.

The idea of maintaining a list of bugs as a credible source of truth was mentioned by Rands in Repose as a habit from his Apple days:

“My favorite internal application at Apple is a product called Radar. It’s a Cocoa application that served as our bug tracking system, and if you wanted to know what was going on regarding a specific application at Apple, you went to Radar.”

“Radar became a powerful tool because it became a credible source of truth regarding the product.”

One habit Automattic encourages but I personally don’t like doing in bug triage is actively closing older bugs because they reach a certain age.

The first reason that comes to mind on why I don’t like closing older bugs is because this means the bug log could no longer be considered a credible source of truth, as there are current production issues that are no longer represented as open bugs.

Over the course of my career I’ve seen many reasons why I personally don’t like closing old bugs:

Closed GitHub Issues are Second-Class Citizens

  • Searching GitHub Issues automatically defaults to searching only open issues, so if there’s a known active issue, but it was closed due to age, there’s a good chance it won’t be easily found and it may be raised as a new bug which duplicates the old closed one.
  • Even if you explicitly include closed bugs in search results, it’s not easy (without adding custom labels) to know at a glance which GitHub issues were closed due to being fixed versus being closed because they were too old. I’ve recently started working on janitorial pull requests to fix known production issues, and I’ve found I’m limited to fixing the production bugs that are still left open as the bugs closed due to age are too hard to find.

Closing Low Priority Bugs 

  • Often older low-priority bugs are closed early because when looking at them individually the impacts are low. But I’ve seen lots of smaller, low priority bugs snowball together to create a larger poor user experience.
  • Often bugs closed for inactivity are ‘edge cases’. Another way to look at edge cases is as stress cases for our users. As per the WordPress.org Flow Glossary/Design for Real Life Book (emphasis added):

But making digital products friendly isn’t enough to make them feel human.

Real life is complicated. It’s full of joy and excitement, sure, but also stress, anxiety, fear, shame, and crisis. We might experience harassment or abuse, lose a loved one, become chronically ill, get into an accident, have a financial emergency, or simply be vulnerable for not fitting into society’s expectations.

None of these circumstances is ideal, but all of them are part of life—and, odds are, your site or product has plenty of users in these moments, whether you’ve ever thought about them or not.

Our industry tends to call these edge cases—things that affect an insignificant number of users. But the term itself is telling, as information designer and programmer Evan Hensleigh puts it: “Edge cases define the boundaries of who [and] what you care about”. They demarcate the border between the people you’re willing to help and the ones you’re comfortable marginalizing.

That’s why we’ve chosen to look at these not as edge cases, but as stress cases: the moments that put our design and content choices to the test of real life.

It’s a test we haven’t passed yet. When faced with users in distress or crisis, too many of the experiences we build fall apart in ways large and small.

Instead of treating stress situations as fringe concerns, it’s time we move them to the center of our conversations—to start with our most vulnerable, distracted, and stressed-out users, and then work our way outward. The reasoning is simple: when we
make things for people at their worst, they’ll work that much better when people are at their best.

Won’t Fix Now is Different to Won’t Fix Forever

It’s easy to close a bug saying “won’t fix”, but won’t fix now is different to won’t fix forever. Priorities change. In my career, I’ve seen “won’t fix” bugs suddenly become fix right now as our CEO personally experienced the bug.

Sending the Wrong Message to Bug Reporters

Raising bugs that are never fixed can be demotivating for bug reporters, but having bugs closed just because they’re old increases this emotional impact on bug reporters. I recently saw this comment on a GitHub Issue for a product I work on:

if-still-broken

What I Propose Instead

Instead of closing old bugs, let’s use a label to identify them as old; this makes it easy to find old bugs and fix them. It would be great if our tools like GitHub Issues allowed us to automatically label older bugs, but in the meantime we could do something like this during bug triage.

Thanks

Thanks to my team lead and bug triage extraordinaire Lance Willett for reviewing a draft of this post.

Author: Alister Scott

Alister is an Excellence Wrangler for Automattic.

11 thoughts on “Should you close old bugs?”

  1. Great points on why closing (or deleting!!) old bugs is a bad idea without some sort of criteria to get them out of the current production backlog but not forgotten. I have been doing this for years across several companies; I split the resolutions up as “Will Not Fix” and “Future Fix” (or something similar) to make it clear what has collectively been decided as “Will not fix ever (without some compelling reason)” versus “Will not fix right now (but plan to come back to it)”.

    It’s trivial to add another status/label/whatever in pretty much every kind of tracker. All that is needed is to do it and let people know it exists.

    Of course, not everyone agrees: http://testobsessed.com/2012/08/bugs-spread-disease/

    In a perfect world I completely agree with Liz. We always run the risk of “out of sight, out of mind” and perpetually leaving something undone because we have new things to take care of, or the worst case of constantly going back to a triage meeting saying “yep, still not going to fix that right now”. I think most of us don’t have the luxury (or authority) to fix every bug and having an easy system to file it away but preserve its legacy is a nice middle ground.

    Liked by 1 person

  2. One case I’ve seen in very old products is the code has evolved to the point that the (old) bug is no longer a bug because that code no longer exists. That’s a case where I absolutely close old bugs.

    Liked by 1 person

  3. A bit of etiquette goes a long way to improve communication. I’ve been guilty of this, and my colleagues do a great job calling me on it. When you close something, you should clearly state why so everyone is on the same page. This helps with the demoralizing factor you brought up, “Having bugs closed just because they’re old increases this emotional impact on bug reporters.” The reason you give beyond just a date gives a richer context. Instead of just time, it’s making a decision based on priority, impact, focus, business goals, etc.

    Yet the timeframe given was helpful to make sure no one else wanted to vote, chime in, suggest a fix or workaround. Kind of like a “Call for comment” for a community group, or comments on public hearings, that would close after a given time, so everyone can move ahead with their lives.

    Liked by 1 person

  4. Here are my top reasons why we should keep closing older bugs and not just label them and keep open.

    – Everything needs a decision. Closing an issue as “maybe later” may be a valid decision. Saying you won’t ever get to something because it’s a minor edge case is a valid decision. What you don’t want is no clear decision; when that happens contributors to the project are confused on where they can best contribute.
    – Tools we use today for bug tracking aren’t good enough to sort the great from the good/bad. More labels is more noise and more overhead to get things done.
    – Not everything is worth fixing, even if it’s a current bug in production.
    – We can’t fix everything; there will never be enough time for everything. We have to decide based on priority.
    – If all bugs were as important as new work (and other bugs), they’d already be fixed by now.
    – Closing certain items allows efforts to focus on more urgent and impactful tasks.
    – It’s demoralizing to have hundreds or thousands of open items. The clutter and overhead makes it hard to find the right things to work on.
    – The mental load and stress of seeing a huge list of bugs every time you open the bug tracker could also be demotivating, and could also have an equal or worse emotional impact.

    Liked by 3 people

  5. A note on “Maybe Later” is better than “Won’t Fix”

    To address this in another bug tracker at Automattic (not GitHub), we introduced “Maybe Later” as a resolution to avoid the hard-and-fast closure of “Won’t Fix.” You can then sort by those at a later time and revisit the decisions.

    Liked by 1 person

  6. I go food shopping every week.
    Every week about 10% of my shopping is inedible.
    I bought a big chilled storage unit.
    I catalogue every inedible item with where I bought it, the receipt and how much I paid.
    Every week I review every item in the chilled storage unit.
    Sometimes I take things back, with the receipt, and get a refund.
    Am I mad?
    If I did do any of this I would be, so can someone explain to me why it would be a good idea. (No-one has been able to yet).
    Perhaps then, if it’s not a good idea, someone can explain to me why they would even contemplate doing the same, if not far worse, with defects (bugs)?
    A defect is a cost.
    Somewhere to store defects is a cost.
    Periodically reviewing defects is a cost.
    Every process step with a defect is a cost.
    Spending money to prevent defects is an investment with a potential return.

    Liked by 1 person

Comments are closed.