Task Descriptor: Run Tests |
Run the appropriate collections of tests required to evaluate product quality. Capture test results that facilitate ongoing assessment of the product.
Based on Method Task: Run Tests
Purpose
To execute tests and evaluate the test results
Relationships
Roles | Main: Tester | Additional: | Assisting:
Inputs | Mandatory: Build Test Script | Optional: None | External: None
Outputs | Test Log Work Items List
Main Description
Run the system test, which addresses functional and system integration tests and, potentially, user acceptance tests.
Steps
Schedule test execution
Run your system tests as often as possible. Ideally, run your tests whenever new code is checked into your version control tool. For larger systems, this will be too expensive. Your tests may take several hours to run; therefore, you'll need to schedule tests less frequently. If possible, however, run the tests several times a day. As a minimum, run automated tests each night, but also run them during working hours, if possible.
Run the test
Run the test at the scheduled time based on the instructions in the Test Script. It is best that the script be automated. Good practices: Run the test in a separate test environment. Ensure that you run the test against the latest clean build. The first step of the test should be to set up the test environment (ensure that the network is available, that the test database is available and reset to a known state, and so on).
Close test run
Close the actual run as the last step of running the test. To do this, you must: Close the test logs. The appropriate test log files should be closed and placed in the appropriate folder or directory. Announce results. Send a notice to everyone involved in the project, informing them of the result of the test run and where they can find the test logs.
Examine the test log
Collect and compile information from test execution logs so you can: Capture the high-impact and risk issues discovered in running the tests. Identify errors in test creation, data inputs, or integrating applications and any architectural anomalies. Isolate the target of the test to determine failure points. Diagnose failure symptoms and characteristics. Assess and identify possible solutions. After completing these steps, verify that you have enough details to determine the impact of the results. In addition, make sure that enough information exists to assist individuals who are performing dependent tasks.
Identify failures and propose solutions
Identify whether or not the test has failed and propose a solution based on the type of test and category of failure. The test The approach to testing will determine the identified failures and candidates for solutions. Tests that are programmer supporting are used to help prepare and ensure confidence in the code. When identifying failures and proposing solutions for programmer supporting tests: Failures will be identified at an object or element level Solutions will be to help clarify the problem Test that are business supporting are used to uncover prior mistakes and omissions. Failures will identify omissions in requirements Solutions will help to clarify expectations of the system After you have this information and the steps proposed to resolve the failures, you can effectively categorize the type of failure and the appropriate type of solution. See [MAR03] for more information.
Communicate test results
Communicate the test results to the team. For failed tests this might involve adding bugs to the Work Items List. Communicating test results can affect the perception of the effectiveness of the tests. When communicating test results, it is important that you: Know the audience, so that appropriate information is communicated appropriately Run tests or scenarios that are likely to uncover the high-impact and risk issues or represent actual use of the system When preparing test result reports, answer the following questions: How many test cases exist, and what are their states (pass, fail, blocked, and so on)? How many bug reports have been filed, and what are their states (open, assigned, ready for testing, closed, deferred)? What trends and patterns do you see in test case and bug report states, especially opened and closed bug reports and passed and failed test cases? For test cases that were blocked or skipped, why are they in this state? Considering all test cases not yet run (and perhaps not even created yet), what key risks and areas of functionality remain untested? For failed test cases, what are the associated bug reports? For bug reports ready for confirmation testing, when can your team perform the test?
Properties
Multiple Occurrences |
Event-Driven |
Ongoing |
Optional |
Planned |
Repeatable |
Key Considerations
Run the test regularly. Ideally, that means whenever the code changes but, minimally, once a day. It should be possible for anyone on the test team to run the test at any time.
More Information
Guidelines | Maintaining Automated Test Suite Programming Automated Tests
This program and the accompanying materials are made available under the Eclipse Public License v1.0 which accompanies this distribution. View copyright information here: OpenUP Copyright