): mark a successful result, which means this test has neither significant performance regression nor significant standard error
): mark a failing result, which means this test shows a significant performance regression (more than 10%)
): mark a failing result (see above) with a comment explaining this degradation.
or
): mark a failing or successful result with a significant standard error (more than 3%)
): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)(build_test_time - baseline_test_time) / baseline_test_time)sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)All 6 scenarios |
RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB) | RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB) | RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB) | Win XP Sun 1.4.2_10 (2 GHz 512 MB) | Win XP Sun 1.4.2_10 (3 GHz 2 GB) |
| StatePerformanceTest#testCreation() |
+0.0% [±3.3]
|
+0.5% [±2.4]
|
- 4.9% [±2.7]
|
- 3.4% [±3.3]
|
- 1.4% [±2.3]
|
| StatePerformanceTest#testResolution100() |
+1.7% [±2.6]
|
+2.7% [±1.9]
|
+4.2% [±2.3]
|
+1.4% [±1.8]
|
+3.0% [±1.3]
|
| StatePerformanceTest#testResolution1000() |
Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200706111606, jvm=sun}':
Elapsed Process: 3.42s is not within [0%, 110%] of 3.09s
- 10.4% [±0.9]
|
+1.7% [±0.4]
|
- 2.0% [±0.6]
|
+6.4% [±0.4]
|
+6.6% [±0.4]
|
| StatePerformanceTest#testResolution500() |
- 6.6% [±0.5]
|
+4.3% [±0.5]
|
+1.7% [±0.4]
|
+8.0% [±0.5]
|
+8.1% [±0.9]
|
| StatePerformanceTest#testResolution5000() |
Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200706111606, jvm=sun}':
Elapsed Process: 4.97s is not within [0%, 110%] of 4.14s
- 20.1% [±8.1]
|
- 6.7% [±7.8]
|
- 8.3% [±8.6]
|
+4.4% [±7.3]
|
- 6.1% [±13.2]
|
| StatePerformanceTest#testStoreAndRetrieve() |
- 3.0% [±1.3]
|
- 2.4% [±0.6]
|
- 1.4% [±0.8]
|
- 1.5% [±1.3]
|
- 2.2% [±0.6]
|