A few years ago I was explaining to a Managing Director of a software company about our testing process. He was an expert in software quality. I was enthusiastically explaining how we provide back ups for all possible failures in our testing methodology. He was intently listening and was appreciative of the many unique features in our process. When I told him about how we make up for deficiencies in test cases by spending certain percentage of time in ad hoc testing he immediately asked what would happen if we do not do ad hoc testing. I had the ready answer. I said if the test cases are not adequate it would help us give an added layer of test coverage if we do ad hoc testing. I was satisfied with my answer but he was not. He said if you are capable of finding defects by ad hoc testing, which requires more knowledge of the product than merely executing test cases, why not in the first place that knowledge be used to write good quality test cases. By his tone I knew he was posing an intellectual challenge. There was no sarcasm or criticism. There was only an eagerness to make me realize the importance of writing good test cases. He was merely asserting the fact that we needed to spend more time and resources to write test cases and not rely upon informal testing as an insurance against incomplete test cases.?
While there are many merits in this concept, like any idea if accepted on its literal meaning can lead to closed minds and lost opportunities .Can we completely do away with ad hoc testing? Even if we can write test cases that provide more than 90% coverage in the first round of testing where will the remaining coverage come from? Black box testing is an end user level testing. User environments, user’s skills and knowledge etc are varied across user spectrum. They play a significant role in their interaction with the software. Ad hoc testing provides such an opportunity for variability in testing environment and skill level.
When a test engineer executes test cases he observes behavioral changes in software. Similar changes also take place in his mind. He may have more ideas as to how a functionality can be used than what is written in the test case. By encouraging the test engineer to do ad hoc testing we not only test the software but also allow the engineer to test his ideas. Test execution therefore becomes a lively interaction between the engineer and the software than being a mere mechanical exercise.
In conclusion, ad hoc testing is not merely providing safeguards against inadequate test case but an integral part of testing ecosystem. It is not something we need to continue till we can write perfect test case. We need it as much to discover the imperfect testing world.