How Buggy is your Product?

Want a really slick way to find out how buggy and nasty your product is? Buggy? Who wants to know that? We want good products, not buggy ones. Well, this method will help you get there. It’s a way to find out how many bugs there are still in your software when you release it.
Let’s face it, all products have bugs. Even fully released and supported ones. But managing the number of bugs at launch time is a good thing. Release too many, and you’re dead. Tech Support will hate you; your CFO will hate you, and worst of all, your customers will hate you. So clean things up, buster!

Here’s how you do it:

Let’s first start by supposing that you have a QA department testing your product prior to release. They test the documented and “undocumented” features of the product, record and report the bugs, then retest when engineers fix them. Hopefully, the product becomes more robust as the release date approaches. Less and less bugs are found. But we all know the product will ship with some known and unknown bugs. You just can find them all, and you can’t always fix the ones you find. But the process is good, and it’s working fine.

So, what if you intentionally slipped some known bugs into the product, hoping the QA department would find them? If you did, you’d know they were doing their jobs. After all, that’s what they are paid to do. So for instance, if you were receiving 1,000 bugs per week from the QA department, and you slipped 100 new bugs into it, you would hope to get 1,100 bugs the next week. You would know the Quality Assurance department was catching 100% of the bugs in the product.

But what if the QA department only found 70 of your artificially injected bugs? I.e. the ones you inserted just to test the percentage of bug found. If only 70 of the inserted bugs came out, that would tell you that their efficiency rate is about 70%. Given that they find 1,000 bugs per month, you might suspect there are actually 1,428 in the system. Why? Because they only found 70% of your artificial bugs, so what makes you think they are finding 100% of the others? Probably not, but that’s okay; nobody’s perfect. If they are 70% efficient, then 1,428 x 70% = 1,000.

Using this method, as your bug rate approaches zero, you’ll know about how many uncaught bugs remain.

Leave a Reply