The typical bug tracking system has two, maybe three fields identifying bug cost benefit ratio:

1. Priority (assigned by business owner)
2. Severity (classification of bug – critical to low)
3. Estimated hours (a guess at how long it will take to do)

As you note, this does identify which bug is the important one to work on.

The system put forth in PEF/REV: A Multi-dimensional bug tracking metric adds more information on the business and developer components to bug cost benefit.

All values are on a 1 .. N scale (same top value for each).

PEF is identified by the business:

• Pain – How painful the bug is
• Effort – How much effort it takes to work around the bug
• Frequency – How often does the bug occur

REV comes from the developer:

• Risk – How risky is the fix to the system
• Effort – How much effort is there to fix the bug
• Verifiability – How easy is it to verify the bug fix

If, for example, you have a rarely happening crash that is easy to work around (turn on autosave), that could be a PEF of 7,1,1 (score 9). While fixing it may involve a change to a core module and have a REV of 9,3,2 (score 14).

On the other hand, you could have a annoyance that is always there (3,3,9 – score 15) that has a trivial fix (1,1,3 – score 5).

In this example, the annoyance looks to be a better cost/benefit thing to work on.

The problem you are describing is very common and the best answer might be “use your gut feeling”.

I usually divide bugs in three categories: Crashing, Annoying, Cosmetic (they might be called 1, 2, 3 – that doesn’t really matter) and then put down an overall estimate on the time needed to fix every bug (all bugs must have a rough updated estimate at all times).

When solving bugs Crashing > Annoying > Cosmetic and then I simply do a “shortest job first” to get as much initial throughput as possible.

I find it VERY difficult to calculate any kind of direct financial benefit from solving any bug – unless it is a very tightly scoped paid job.

One note you might need is that you really should live up to the fifth point on The Joel Test – this might be influenced by team size and distribution among various other “local” issues – but it is generally a sign of good practice.

Another consideration might be the type of test or testing organization that uncovers the bug, when trying to determine the impact and cost of the bug and fix. Unit or functional testing may point to something in the design that needs to be changed, and thus correcting early would be easier and less costly than later. System or integration testing may point to something that affects a wider variety of customers. And Standards testing, while often something non-critical to a large subset of customers, may lead to a loss of certification if not corrected and have a negative impact on the business.

That said, just because a particular test organization has a test fail shouldn’t automatically make a bug “critical.” For example, there might be a temptation to say something like “all system tests must pass before shipping, thus any system test that fails automatically results in a critical/high-severity bug.” Hopefully no organization would make that statement (good test exit criteria is a different discussion), but the point is that the “severity” of bug should be judged on its impact to the customer, product, or company image rather than where or when it is uncovered.

One possible way to address that is to make a distinction between “severity” and “urgency.” For example, as the release date nears there can be time pressure to determine if an apparently lower-severity bug might affect a large subset of customers, giving the bug a greater “urgency” and elevating work on that bug above (some) others at least until that determination can be made. A proper balance between the two, along with experience and good judgement, would help direct where time and effort is spent.

In addition to what others have said about the classification of bugs, etc, also consider looking at Afferent Coupling (Ca) for determining the level of criticalness a particular bug might represent. If you have a bug in a module with a high Ca count, I might be looking at fixing that one first as it could benefit other components in the system. Ca helps you determine the level of responsibility, and responsible modules with bugs in them hurt the whole application (read more about Ca here: http://www.ibm.com/developerworks/java/library/j-cq04256/index.html).

With that in mind, we tend to categorize bugs and place them in priority based on their impact to the customer. Your customer will vary, but having them involved in the discussion will ultimately drive what bugs should be fixed over others. It’s not real scientific, but as a whole team we tend to come to consensus on the priority and categorization of bugs, everyone has input on the “big ones” (all the stakeholders can give input), and we stack and rack from there.

You can’t determine the actual cost of all bugs. Some you can, for many it’s very hard to quantify. For example, say you have a bug that results in all of the text on buttons are slightly misaligned. It makes your 1.0 product look a little sloppy. This doesn’t cause your program to crash, or your users to lose data. Probably a pretty cheap bug, right?

Now, what if every customer of yours notices this problem, and every reviewer mentions it in reviews of your product. And what if this bug carries over from version 1.0 to 1.1 and 1.2. Your company may now have a reputation of being a bit sloppy in quality control. How expensive might this simple bug be in terms of potential lost sales for your company for future products? Or, how might this affect a review that your product gets — your product may perfectly meet the needs of your customers, but only gets an 9 out of 10 because it looks a little sloppy.

So, you need to not only look at the cost of a specific bug in terms of what it costs to fix and it’s direct impact on the user, but you have to consider it in the larger context of how it affects the perception of your product in the marketplace, and how that perception affects future sales.

This may seem far-fetched but it isn’t. At the time that I write this you can go to another site on the web that has an article about how Apple has moved the “1” on their calendar icon over by a pixel or two. If you do a search you’ll see several negative articles about this tiny little bug. Of course, this hasn’t directly affected Apple’s bottom line, but they promote themselves as the pinnacle of good design, so having a design bug affects that perception, even if only slightly.