Performance of org.eclipse.core: I20080617-2000 relative to 3.3 (200806171530)

RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)

Win XP Sun 1.4.2_10 (3 GHz 2 GB)

RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)

Win XP Sun 1.4.2_10 (2 GHz 512 MB)

RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)

Scenario Status

• an icon showing whether the test fails or passes and whether it's reliable or not.
  The legend for this icon is:
  • Green (): mark a successful result, which means this test has neither significant performance regression nor significant standard error
  • Red (): mark a failing result, which means this test shows a significant performance regression (more than 10%)
  • Gray (): mark a failing result (see above) with a comment explaining this degradation.
  • Yellow ( or ): mark a failing or successful result with a significant standard error (more than 3%)
  • Black (): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)
  • "n/a": mark a test for with no performance results
• the value of the deviation from the baseline as a percentage (ie. formula is: (build_test_time - baseline_test_time) / baseline_test_time)
• the value of the standard error of this deviation as a percentage (ie. formula is: sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)
  When test only has one measure, the standard error cannot be computed and is replaced with a '[n/a]'.

• fly over image of failing tests to see the complete error message
• to look at the complete and detailed test results, click on its image

All 49 scenarios	RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)	RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)	RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)	Win XP Sun 1.4.2_10 (2 GHz 512 MB)	Win XP Sun 1.4.2_10 (3 GHz 2 GB)
BenchFileStore#testStoreExists()	+0.0% [±2.3]	- 1.0% [±2.6]	+0.0% [±1.6]	- 1.0% [±1.5]	- 0.6% [±1.2]
BenchFileStore#testStoreIsReadOnly()	- 5.2% [±1.0]	- 2.0% [±2.0]	+0.0% [±1.1]	- 2.0% [±1.2]	+0.2% [±1.1]
BenchFileStore#testStoreLastModified()	+0.9% [±0.7]	+0.0% [±1.7]	- 1.1% [±0.5]	- 1.7% [±0.9]	- 0.8% [±0.9]
BenchMiscWorkspace#testGetProject()	+2.7% [±1.0]	+0.5% [±1.0]	+1.4% [±0.7]	+0.6% [±2.1]	+0.0% [±1.5]
BenchMiscWorkspace#testNoOp()	- 8.2% [±4.2]	- 7.4% [±0.7]	- 2.4% [±1.7]	- 2.0% [±2.5]	- 4.7% [±1.8]
BenchWorkspace#testCountResources()	+6.8% [±0.8]	+1.3% [±1.4]	+65.6% [±0.7]	+7.1% [±4.2]	+8.3% [±2.1]
BenchWorkspace#testCountResourcesDuringOperation()	+6.0% [±3.3]	+0.9% [±1.2]	+1.7% [±0.8]	+0.9% [±1.5]	- 2.1% [±1.1]
BenchWorkspace#testFindMaxProblemSeverity()	+0.5% [±0.2]	- 1.0% [±0.2]	- 1.9% [±0.2]	- 2.4% [±1.2]	+0.0% [±0.3]
LocalHistoryPerformanceTest#testAddState()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 158ms is not within [0%, 110%] of 102ms - 54.9% [±79.2]	+1.9% [±20.0]	+0.0% [±17.9]	+0.0% [±14.9]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 111ms is not within [0%, 110%] of 99ms - 12.1% [±11.2]
LocalHistoryPerformanceTest#testBug28603()	+4.8% [±6.3]	+3.4% [±2.6]	+0.0% [±2.9]	+0.0% [±3.0]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 54ms is not within [0%, 110%] of 49ms - 10.2% [±8.2]
LocalHistoryPerformanceTest#testClearHistory100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 75ms is not within [0%, 110%] of 30ms - 150.0% [±94.0]	- 5.3% [±23.1]	+0.0% [±19.5]	Performance criteria not met when compared to '{config=eclipseperfwin1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 43ms is not within [0%, 110%] of 31ms - 38.7% [±43.8]	+0.0% [±30.4]
LocalHistoryPerformanceTest#testClearHistory20x20()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 16ms is not within [0%, 110%] of 11ms - 45.5% [±58.7]	+87.7% [±89.5]	+0.0% [±4.8]	+0.0% [±2.5]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 16ms is not within [0%, 110%] of 11ms - 45.5% [±35.7]
LocalHistoryPerformanceTest#testClearHistory4x100()	+44.4% [±36.2]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 308ms is not within [0%, 110%] of 11ms - 2,700.0% [±2,728.9]	+12.5% [±14.2]	+8.3% [±46.7]	+0.0% [±49.9]
LocalHistoryPerformanceTest#testCopyHistory100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.36s is not within [0%, 110%] of 1.03s - 31.7% [±44.1]	+48.4% [±36.9]	+0.9% [±4.6]	+0.3% [±2.0]	+6.3% [±6.0]
LocalHistoryPerformanceTest#testCopyHistory20x20()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 349ms is not within [0%, 110%] of 170ms - 105.3% [±100.0]	+4.7% [±3.9]	- 1.8% [±5.3]	- 2.3% [±2.4]	- 2.8% [±3.7]
LocalHistoryPerformanceTest#testCopyHistory4x100()	+15.9% [±12.5]	+2.2% [±8.6]	+6.8% [±9.2]	- 5.5% [±5.2]	- 8.3% [±5.8]
LocalHistoryPerformanceTest#testGetDeletedMembers100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 84ms is not within [0%, 110%] of 42ms - 100.0% [±109.9]	+4.5% [±8.9]	+2.9% [±9.1]	+13.0% [±13.9]	+34.3% [±33.6]
LocalHistoryPerformanceTest#testGetDeletedMembers20x20()	+3.4% [±55.0]	+27.3% [±20.5]	+33.3% [±36.0]	+0.0% [±0.0]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 31ms is not within [0%, 110%] of 23ms - 34.8% [±78.8]
LocalHistoryPerformanceTest#testGetDeletedMembers4x100()	+6.7% [±7.5]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 35ms is not within [0%, 110%] of 11ms - 218.2% [±0.0]	+0.0% [±7.1]	+0.0% [±49.3]	+6.2% [±3.1]
LocalHistoryPerformanceTest#testGetHistory()	- 2.4% [n/a]	+0.0% [n/a]	+40.9% [n/a]	+6.2% [n/a]	+6.2% [n/a]
LocalHistoryPerformanceTest#testHistoryCleanUp100x4()	- 1.3% [±5.7]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.45s is not within [0%, 110%] of 349ms - 315.8% [±246.9]	- 0.8% [±6.3]	- 2.1% [±8.5]	- 5.9% [±13.1]
LocalHistoryPerformanceTest#testHistoryCleanUp20x20()	+68.0% [±58.4]	+29.5% [±32.9]	- 0.4% [±2.0]	+52.2% [±52.7]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 918ms is not within [0%, 110%] of 624ms - 47.1% [±45.9]
MarkerPerformanceTest#testSetAttributes1()	- 0.3% [n/a]	+1.9% [n/a]	+0.0% [n/a]	- 1.0% [n/a]	+3.5% [n/a]
MarkerPerformanceTest#testSetAttributes2()	+0.7% [n/a]	+0.5% [n/a]	- 0.6% [n/a]	- 0.2% [n/a]	+1.5% [n/a]
PropertyManagerPerformanceTest#testGetProperty100x4()	- 1.0% [±1.3]	- 2.7% [±1.4]	- 0.2% [±0.7]	- 0.3% [±1.9]	- 0.4% [±1.1]
PropertyManagerPerformanceTest#testGetProperty20x20()	+12.6% [±9.5]	+8.4% [±4.1]	+0.3% [±0.1]	- 0.8% [±0.9]	+1.9% [±0.2]
PropertyManagerPerformanceTest#testGetProperty4x100()	+3.0% [±0.1]	+4.6% [±0.1]	+0.2% [±0.1]	- 1.0% [±0.2]	+1.6% [±0.0]
PropertyManagerPerformanceTest#testSetProperty100x4()	- 4.3% [±0.7]	+0.0% [±2.1]	- 0.6% [±0.4]	- 1.7% [±3.1]	- 0.3% [±2.1]
PropertyManagerPerformanceTest#testSetProperty20x20()	+0.0% [±1.3]	+0.9% [±2.4]	- 3.6% [±2.2]	- 4.2% [±5.0]	- 3.2% [±2.6]
PropertyManagerPerformanceTest#testSetProperty4x100()	+1.1% [±3.2]	- 0.6% [±4.5]	- 6.5% [±1.7]	- 4.5% [±8.4]	- 1.5% [±3.2]
WorkspacePerformanceTest#testCreateResources()	+9.9% [±24.5]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 9.97s is not within [0%, 110%] of 7s - 42.3% [±59.9]	+14.9% [±25.6]	- 1.0% [±7.9]	- 8.0% [±11.7]
WorkspacePerformanceTest#testDeleteProject()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 13.49s is not within [0%, 110%] of 8.62s - 56.6% [±40.6]	+0.2% [±18.4]	+28.9% [±24.9]	+12.1% [±4.6]	+13.4% [±2.3]
WorkspacePerformanceTest#testFolderCopy()	- 4.6% [±15.6]	- 0.2% [±5.9]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 11.81s is not within [0%, 110%] of 10.2s - 15.9% [±19.1]	- 6.6% [±1.6]	- 2.8% [±1.9]
WorkspacePerformanceTest#testFolderMove()	+0.3% [±77.6]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.97s is not within [0%, 110%] of 1.37s - 43.7% [±41.6]	+2.1% [±0.9]	- 0.1% [±2.6]	- 1.8% [±1.5]
WorkspacePerformanceTest#testRefreshProject()	- 0.4% [±5.5]	- 2.5% [±3.8]	+1.3% [±2.4]	- 3.7% [±4.6]	- 1.0% [±2.9]
BenchPath#testHash()	+0.0% [±2.9]	+2.0% [±0.8]	+0.8% [±0.2]	+0.9% [±1.0]	+0.0% [±0.9]
BenchPath#testPathCreation()	- 2.4% [±2.3]	+0.0% [±0.4]	+0.4% [±0.3]	- 0.7% [±0.9]	+0.9% [±0.8]
BenchPath#testToOSString()	+0.4% [±0.3]	- 2.9% [±0.5]	+0.9% [±0.3]	+28.9% [±1.3]	+22.4% [±1.1]
BenchPath#testToString()	- 3.4% [±0.4]	- 1.6% [±0.5]	+1.8% [±0.3]	+29.5% [±1.2]	+21.8% [±1.1]
ContentTypePerformanceTest#testLoadCatalog()	+44.6% [n/a]	+49.0% [n/a]	+48.8% [n/a]	+47.9% [n/a]	+42.0% [n/a]
ContentTypePerformanceTest#testNameMatching()	+0.8% [±0.7]	+5.6% [±0.6]	+2.5% [±0.7]	- 6.1% [±2.1]	+5.2% [±1.2]
PreferencePerformanceTest#testGetStringCommonPrefixKeys()	+3.0% [±0.8]	+3.0% [±0.8]	- 4.0% [±0.9]	Performance criteria not met when compared to '{config=eclipseperfwin1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.74s is not within [0%, 110%] of 1.57s - 10.8% [±1.3]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.08s is not within [0%, 110%] of 951ms - 13.1% [±0.9]
PreferencePerformanceTest#testGetStringMisses()	+1.1% [±1.4]	+5.4% [±1.6]	- 2.8% [±0.2]	Performance criteria not met when compared to '{config=eclipseperfwin1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.34s is not within [0%, 110%] of 1.13s - 19.1% [±2.7]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 896ms is not within [0%, 110%] of 740ms - 21.1% [±2.1]
PreferencePerformanceTest#testGetStringSequentialKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.74s is not within [0%, 110%] of 1.48s - 17.5% [±0.7]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.1s is not within [0%, 110%] of 957ms - 15.4% [±1.3]	+0.0% [±0.9]	- 5.9% [±1.0]	- 7.8% [±1.1]
PreferencePerformanceTest#testGetStringUniqueKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.58s is not within [0%, 110%] of 1.34s - 18.6% [±0.6]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1s is not within [0%, 110%] of 861ms - 16.3% [±1.1]	- 1.7% [±1.1]	- 7.3% [±1.8]	- 9.0% [±1.2]
PreferencePerformanceTest#testPutStringKeys()	+3.1% [±0.7]	+0.9% [±1.4]	+1.9% [±1.4]	+4.0% [±0.9]	+1.0% [±1.3]
PreferencePerformanceTest#testRemoveStringKeys()	+0.9% [±0.4]	+1.6% [±0.9]	+5.6% [±0.5]	+5.4% [±1.5]	+9.1% [±2.0]
StartupTest.testApplicationStartup	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 2.2s is not within [0%, 105%] of 2.1s - 5.1% [±2.4]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.86s is not within [0%, 105%] of 1.38s - 34.9% [±17.1]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.29s is not within [0%, 105%] of 1.21s - 7.2% [±0.6]	- 3.2% [±2.6]	- 3.9% [±2.1]
UIStartupTest.testUIApplicationStartup	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 20.23s is not within [50%, 105%] of 17.47s - 15.8% [±10.8]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 12.27s is not within [50%, 105%] of 11.22s - 9.4% [±5.5]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 10.18s is not within [50%, 105%] of 9.22s - 10.5% [±5.9]	Performance criteria not met when compared to '{config=eclipseperfwin1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 14.01s is not within [50%, 105%] of 12.46s - 12.4% [±12.8]	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 9.41s is not within [50%, 105%] of 7.78s - 20.9% [±11.2]