Interview Questions

Describe methods to determine if you are testing an application too much?

Software QA/Testing Technical FAQs


(Continued from previous question...)

Describe methods to determine if you are testing an application too much?


Answer1:
While testing, you need to keep in mind following two things always:
-- Percentage of requirements coverage
-- Number of Bugs present + Rate of fall of bugs
-- Firstly, There may be a case where requirement is covered quite adequately but number of bugs do not fall. This indicates over testing.
--- Secondly, There may be a case where those parts of application are also being tested which are not affected by a CHANGE or BUG FIXTURE. This is again a case of over testing.
-- Third is the case as you have suggested, with slight modification, i.e bug has sufficiently dropped off but still testing is being at SAME levels as before.


Methods to determine if an application is being over-tested are--
1. Comparison of 'Rate of Drop in number of Bugs' & 'Effort Invested in Testing' (With all Requirements been met) That is, if bug rate is falling (as it generally happens in all applications), but effort invested in man hours does not fall, this implies Over testing.
2. Comparison of 'Achievment of bug rate threshold' & 'Effort Invested in Testing' (With all Requirements been met) That is, if bug rate has already achieved the agreed-upon value with business and still the testing efforts are being invested with no/little reduction.
3. Verifying if the 'Impact Analysis' for 'Change Requests' has been done properly and being implemented correctly. That is, to check and verify that the components of AUT which have got impacted by the new change are being tested only and no other unrequired component is being tested unneccessarily. If unaffected components are being tested, this implies Over testing.

Answer2:
If the bug find rate has dropped off considerably, the test group should shift its testing strategy. One of the key problems with heavy reliance on regression testing is that the bug find rate drops off even though there are plenty of bugs not yet found. To find new bugs, you have to run new tests.
Every test technique is stronger for some types of bugs and weaker for others. Many test groups use only a few techniques. In our consulting, James Bach and I repeatedly worked with companies that relied on only one or two main techniques.
When one technique, any one test technique, yields few bugs, shifting to new technique(s) is likely to expose new problems.
At some point, you can use a measure that is only partially statistical -- if your bug find rate is low AND you can't think of any new testing approaches that look promising, THEN you are at the limit of your effectiveness and you should ship the product. That still doesn't mean that the application is overtested. It just means that YOU'RE not going to find many new bugs.

Answer3:
Best way is to monitor the test defects over the period of time
Refer williams perry book, where he has mentioned the concept of 'under test' and 'over test', in fact the data can be plotted to see the criteria.
Yes one of the criteria is to monitor the defect rate and see if it is almost zero second method would be using test coverage when it reach 100% (or 100% requirement coverage)

(Continued on next question...)

Other Interview Questions