Automated Testing of Eclipse Builds

last updated: August 4, 2005

Description

This document describes how to run JUnit test plugins in Eclipse from the command line using Eclipse JUnit test plugins (see Appendix A for listing) and a testing framework. This method of running tests is intended for anyone who wishes to automate their testing process or reproduce the testing mechanism used in Eclipse builds by the Eclipse Release Engineering team.

Click here for a description of the testing framework, and how it can be used outside of the setup described here.

If you simply wish to write and run JUnit tests interactively from an Eclipse workbench, try one of the links below:

JUnit Support in Eclipse
Using JUnit

Requirements

1. All test machines will require a java runtime environment (version 1.4 or higher) installed and optionally added to the path environment variable (see -vm parameter below).
2. Info-ZIP UnZip version 5.41 or later installed and added to the path.
3. The Automated Test feature from an Eclipse build (http://www.eclipse.org/downloads/index.php). This contains the test framework plugin and the test plugins to run.
4. An Eclipse SDK zip file (http://www.eclipse.org/downloads/index.php) to test. Usually you will want to test a particular Eclipse build using a matching Automated Tests feature.

To run the Team CVS tests you will need extssh access to a CVS repository. The tests create a lot of junk modules and tags so you will want to keep the test repository seperate from your development repository. While it is not necessary to clean up the repository between each Team test run you will probably want to clean it up periodically.

Setup

1. Extract the eclipse-Automated-Tests-<buildid>.zip. On Window, extract this file close to the root of the drive to prevent some path names from exceeding the 255 character limit. This will create a top-level directory "eclipse-testing".
2. Place the Eclipse SDK zip file to test in the eclipse-testing directory. Do not unzip this file. (The test scripts will take care of this)
3. To run the Team CVS tests edit the test.xml script to provide connection information.

Replace the following strings in the "teamcvs" target of the test.xml file with the information described below:

Running Tests

1. cd to the eclipse-testing directory
2. Run the following script:

runtests [-os <operating system>] [-ws <windowing system>] [-arch <architecture>] [-noclean] [<testTarget>][-properties <path>][-vm <path to java executable>]

Please note that for 3.1 and higher stream builds, there are two targets are required to run all tests. The all test target runs tests we run with a 1.4 vm. The all5.0 target runs tests that require a 1.5 vm.

All parameters and targets are optional on windows systems. Just running "runtests " will run the all test target on Windows, installing a clean Eclipse between each test target. On *nix systems, the os, ws and arch parameters must be specified.

Test results are produced in an individual xml files for each component. These results are copied to the eclipse-testing\results\xml directory. These results can be converted into html by running the "genHtml" target as described in the table below. Please be aware that "Build Successful" printed to the console is NOT indicative of tests passing.

Parameters

Parameter Description
-os <operating system> The OS the tests are being run on. One of: aix, hpux, linux, qnx, solaris, win32. Default win32
-ws <windowing system> The windowing system the tests are being run on. One of: motif, gtk, photon, win32. Default win32
-arch <architecture> The architecture the tests are being run on. One of: ppc, PA_RISC, x86, sparc. Default x86
-noclean Run tests without installing a clean Eclipse between test targets. Default is to install a clean Eclipse between test targets.
-properties Used to reference a properties file containing additional Ant properties used in running tests.Can be used to pass additional vm arguments to the Java virtual machine running the tests by adding the entry "extraVMargs=<the args to pass to the vm>" to the specified properties file.
-vm The full path to the java executable with which to run the tests. Use this to specify a specific virtual machine with which to run the tests. For example, "-vm c:\jdk142\jre\bin\java". Default set to "java" (i.e. java executable on system path).

Test Targets

Specifies which ant target to run in the main test.xml. Typically this corresponds to a test project's tests. If you do not specify a test target then the all test target is run. The all5.0 test target must be run with a 1.5 vm to complete successfully.

all Runs all the targets below until all5.0
ant Runs the JUnit plugin tests for the Platform-Ant component.
ant-ui Runs the JUnit plugin tests for the Ant-UI component.
compare Runs the JUnit plugin tests for the Platform-Compare component
core Runs the JUnit plugin tests for the Platform-Core component.
coreexpressions Runs additional JUnit plugin tests for the Platform-Core component.
filebuffers Runs additional JUnit plugin tests for the Platform-Core component.
help Runs the JUnit plugin tests for the Platform-Help component.
jface Runs additional JUnit plugin tests for the Platform-UI component.
jdtdebug Runs the JUnit plugin tests for the JDT-Debug component.
jdttext Runs the JUnit plugin tests for the JDT-Text component
jdtui Runs the JUnit plugin tests for the JDT-UI component.
pdeui Runs additional JUnit plugin tests for the PDE-ui component
releng Runs the JUNit plugin tests for the releng component. Note, if you don't have the chkpii executable, this portion of the tests will fail.
jdtuirefactoring Runs additional JUnit plugin tests for the JDT-UI component.
ltkcorerefactoringtests Runs additional JUnit plugin tests for the JDT-UI component.
ltkuirefactoringtests Runs additional JUnit plugin tests for the JDT-UI component.
swt Runs the JUnit plugin tests for the Platform-SWT component.
teamcore Runs the JUnit plugin tests for the Platform-VCM component.
teamcvs Runs additional JUnit plugin tests for the Platform-Team component.
text Runs the JUnit plugin tests for the Platform-Text component.
ui Runs the JUnit plugin tests for the Platform-UI component.
uieditors Runs additional JUnit plugin tests for the Platform-UI component.
uiworkbenchtexteditor Runs additional JUnit plugin tests for the Platform-UI component.
uircp Runs additional JUnit plugin tests for the Platform-UI component.
update Runs the JUnit plugin tests for the Platform-Update component.
all5.0

Runs the following test suites which require a 1.5 vm to run sucessfully

jdtcorebuilder Runs the JUnit plugin tests for the JDT-Core component. This target must be run with a with a 1.5 vm.
jdtcorecompiler Runs the JUnit plugin tests for the JDT-Core component. This target must be run with a with a 1.5 vm.
jdtcoremodel Runs the JUnit plugin tests for the JDT-Core component. This target must be run with a with a 1.5 vm.


Appendix A - JUnit test plugins

The test plugins listed below are available from the dev.eclipse.org cvs repository and are available in binary distributable format from the Eclipse downloads page.

* indicates the plugin that contains the test.xml if component has more than one JUnit test plugin

Component JUnit plugin tests
Jdt-Core

org.eclipse.jdt.core.tests.builder *
org.eclipse.jdt.core.tests.compiler *
org.eclipse.jdt.core.tests.model *

Jdt-Debug org.eclipse.jdt.debug.tests
Jdt-UI

org.eclipse.jdt.ui.tests *
org.eclipse.jdt.ui.tests.refactoring *

Platform-Core

org.eclipse.core.tests.harness *
org.eclipse.core.tests.runtime
org.eclipse.core.tests.resources

Platform-Help org.eclipse.help.tests
Platform-SWT org.eclipse.swt.tests
Platform-UI org.eclipse.ui.tests
Platform-Update org.eclipse.update.tests.core
Platform-VCM(Team)

org.eclipse.team.tests.cvs.core
org.eclipse.team.tests.core

 

Appendix B- Supported os, ws, and arch value combinations

The following table contains the values that can be used with the -os, -ws, and -arch switches.

Operating System (os) Windowing System (ws) Architecture (arch)
aix motif ppc
hpux motif PA_RISC
linux gtk x86
linux gtk x86_64
linux gtk ppc
linux motif x86
macosx carbox ppc
solaris gtk sparc
solaris motif sparc
win32 win32 x86