Duplicate the Problem

Our customer submitted a trouble ticket on one of the applications in our software suite. I immediately called the sys admin at the site where the problem was happening. And I was given a set of steps that were followed that result in the problem. So I used these steps to trace through the code in a development environment. I spent a couple days doing this. My conclusion was that there was no way for the specified problem to occur. It was just not possible.

Then I got a call from the sys admin. When the sys admin stood over watching the users follow the documented procedures for using the app, the problem never happened. That was the key that I needed to make some progress. I started thinking about other ways the users are not supposed to use the application. Then I was able to recreate the problem in development. The fix for this problem was not far behind.

So I got a build ready to be released to our customers with the fix. And I was out sick the day our test was supposed to test and release the software. But I left another developer responsible for this. When I got back, I was relieved that the software shipped on time. But I got a visit from a tester and from the project coordinator. Apparently the test team was unable to duplicate the problem. But they shipped the release out anyway. The tester said he followed all the steps from my unit test plan. However he could not make the problem happen with the old version of the code.

Now this scenario was wrong on multiple levels. First of all, a tester should not rely on a developer's unit test plan. Otherwise they are just repeating what a developer did and adding little to no value. Second it was troubling that they could not duplicate the problem. If you cannot make a problem happen, then you certainly cannot tell whether the problem is still happening with a new build. Third, the software should not have been shipped out if we were unsure whether it was right. There are a lot of things wrong in this world. But if your job is to test software fixes, you should at least make sure you know what you are doing. Otherwise you not only do not add value, you waste time in the process.

These are the last days on this particular project since another company won the contract away from us. However I at least went over to the tester, and helped him duplicate the problem. Then he was able to have a set of steps to execute against the new build with the fix. This still is not optimal, since I brought all my biases into the test situation by telling the tester what to do. However it was better than nothing. I hope we have a more thorough independent test team on my next project.