Validation Testing Framework

Components of the Validation Testing Framework

The major components of the Validation Testing Framework are the Test Agents, the Test Controller and the Test Manager. A Test Agent provides one or more testcases. The Test Controller manages the execution of tests provided by Test Agents. The Test Manager provides a user interface to select and execute a test script specifying configuration and test actions, and to view test progress and results.

A validation test can for example test the implementation of a device adapter to be tested for conformance to a profile or profiles that it implements. Validation tests are implemented as test agents in the validation test framework that is part of the device kit. The device kit provides a wizard that lets you create the projects for a new validation test. To run a validation test, a test manager servlet lets you select and run a test script, and review the test results.

Since the goal is to test adapters for physical devices, the framework supports manually assisted testing. As part of the test a tester can be prompted to assert certain conditions or perform certain actions.

The Test Agent and the Test Controller

The Test components will be instantiated and configured on the target system from a factory via the OSGi Configuration Admin service, using the mechanisms provided by SAT, or programmatically in a non-OSGi environment. TestAgents can be run stand-alone, or controlled by a TestController and a TestManager.

A TestAgent manages the configuration and running of a single JUnit Test, which can be either a TestSuite or a TestCase. Either a standard JUnit Test can be run, or a TestCase or TestSuite that extends base classes provided by the TestAgent framework.

The TestAgent framework provides the following base classes for implementing a TestCase and TestSuite:

ConfiguredTestSuite and ConfiguredTestCase – an abstract implementation of TestSuite and TestCase that receives configuration information and other context-specific information from the TestAgent.
NotificationClientTestCase – an abstract implementation of TestCase that can broadcast and receive messages using the NotificationService that is provided by the Device Kit. A NotificationClientTestCase is also a ConfiguredTestCase.

The TestAgent also optionally provides the TestSuite and TestCase with a TestSynchronizationService that allows the test to interact with a test user for the purpose of taking manual actions and performing verification external to the system under test.

A TestController, of which one instance is deployed on the target system, creates, updates, and destroys TestAgent instances, as well as other components like profiles, device adapters and agents via the OSGi Configuration Admin service. TestAgent instances will be run by the TestController and TestAgent instances report progress and results back to the TestController. Tests will synchronize during their run with the test environment via the Test Controller.

Figure 1 depicts the TestAgent and TestController in relation to JUnit and the Edge Configuration, which are prerequisites of the framework. The diagram also shows (in gray), a concrete example Test based on the framework.

Figure 1 - TestAgent and TestController

The Test Manager

As explained in the previous section, the TestController is responsible for running tests locally on a target. The TestController in turn, is managed by a TestManager, as is shown in the next diagram, Figure 2.

Figure 2 - TestController and TestManager

The TestManager is responsible for compiling and executing a TestScript, which is an XML specification of configuration actions and test actions to perform. The TestManager provides a user interface for interacting with the tester and presenting progress and results of running the test to the user. The TestManager sends commands to the TestController to process configuration action and to run individual tests. The TestController runs these commands asynchronously and calls back to the TestManager to report progress and result. The implementation of the user interface is decoupled from the TestManager so that different implementations of the user interface can be provided, like for example a web-based version and a version developed in the form of Eclipse tooling. A servlet-based implementation of the TestManager user interface is provided that runs on the target system itself.

The Test Script

The tester commands the Test Manager to run a test script. The test script is an xml document specifying:

The target system(s) under test.
A collection of test steps to run either sequentially or parallel. Test collections can be nested to allow test sequences to be run in parallel, or collections parallel tests to be run in sequence.

Each test step specifies either:

A set of configuration updates create and configure and possibly destroy tests, devices, adapters, profiles, transports and connections.
One or more TestAgent instances to run concurrently on a specific target system.

Running a Test Script

The following steps describe what happens when running a test script.

The TestController bundle is deployed on the target test system together with one or more TestAgent bundles.
The TestManager and its user interface are deployed on the target system (initial release) or a host system (future release),
The tester launches a test script in the TestManager,
Configuration updates from the test script are sent to the TestController and are run on the target system.
TestAgents are instantiated from their factory and configured by the OSGi Configuration Admin.
Commands from the test script to run tests are sent by the TestManager to the TestController.
The TestController locates the TestAgent service (the Test) matching the specification in the test commands and runs the Test.
Test results are collected and sent back to the TestManager.
The Test runs, and at any point during its execution it can broadcasts messages to the notification service via the NotificationProbe.
The Test can dynamically subscribe to broadcast topics via the NotificationProbe to receive and validate messages being published in the system under test.
The Test requests the TestManager to prompt the tester for creating or asserting external conditions.
The Test reports completion to the TestManager via the TestController, and, if applicable, the TestManager runs the next step, either configuration or testing, in the test script.