TEST DOCUMENT VIEW not usable/correct after tests rerun due to automatic overlay of 'NOT EXECUTED' status
The mechanism to set each test run just launch automatically to NOT EXECUTED result generates a not correct information in the TEST DOCUMENT VIEW.
This is true expecially when you are at second or next runs of same tests and its depends by the test execution dynamic.
In our experience the tests are rexecuted several times in same project, and a test run take days to be closed, so the test manager need to work in 'test result continuity', he need to know always the last consolidated result (set by its Tester, not by SCTM automatically) to be aware about the current 'real' status in the project.
For example: we have 100 tests in total with finished EP and this result: 60 PASSED, 30 FAILED and 10 NOT EXECUTED, as shown in Test Documentation view.
A new build is installed and now the test manager decides to re-run 50 tests, all the FAILED and 5 NOT EXECUTED and a portion of the PASSED (5 Test for regression scope) tests, so immediately the Test Document view shows that 40 test are PASSED and 60 are NOT EXECUTED.What's happen to the previous 'project status' in term of FAILED and PASSED ? disappear!
Question: is it a correct information about the current status of the project ? It is useful ? what this information says? Help me to understand.. please.
Our opinion (15 test manager opinions) is that this information is incorrect and not useful. The real information that the test manager needs is to understand where is the project 'cross runs and builds', minute by minute, for a angle view and which tests are to be confirmed (because are running again) in another angle views. The currect document view cannot help in any ways.
The test status, when a test is just launched, cannot be 'NOT EXECUTED', maybe we can consider a 'RUNNING' or 'PROGRESS' status, or better NOTHING.
Only when the Tester select the test in running and set its result, it MUST BE consider, never before, as all the other main competitors does (HP QC and IBM QM).
Thanks a lot for your attention.
We have introduced the manual test result viewer which will allow the view of in progress tests in a rich context which shows attachments and associated defects as well as showing the results for completed test runs.
This is online and you can print or PDF it.
We are looking to modify the business logic that will support result management across a wider scope.
I checked the news in SCTM v13 and the new manual testing interface, but this need is not adressed.
The tester need to have always evidence of a valid 'last test result', and we cannot consider valid a 'not executed' status automatically set by SCTM for a test rerun.
I can show you tons of cases where this condition created a great confusion and don't allow to understand the real state of progress of the project.
Comparing to other tool when a test is launched its status is 'neutral' or 'to be executed', preserving the last 'previous' status) and don't overlay a previous status until the tester decide to do it setting a new real status.
The impact is also very high on reporting, so we must create external table to save the results of SCTM setting the result 'valid' and 'not valid'.
Where a test is in a running EP (status=7) its 'not executed' status is considered valid only if a previous status doesn't exist. In this way we can offer a real test execution reporting.