Performance of org.eclipse.core: M20090211-1700 relative to 3.3 (200806171530)

Select which kind of scale you want to use:

percentage time (linear) time (log)

RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)

Win XP Sun 1.4.2_10 (3 GHz 2 GB)

RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)

RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)

Scenario Status

• an icon showing whether the test fails or passes and whether it's reliable or not.
  The legend for this icon is:
  • Green (): mark a successful result, which means this test has neither significant performance regression nor significant standard error
  • Red (): mark a failing result, which means this test shows a significant performance regression (more than 10%)
  • Gray (): mark a failing result (see above) with a comment explaining this degradation.
  • Yellow ( or ): mark a failing or successful result with a significant standard error (more than 3%)
  • Black (): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)
  • "n/a": mark a test for with no performance results
• the value of the deviation from the baseline as a percentage (ie. formula is: (build_test_time - baseline_test_time) / baseline_test_time)
• the value of the standard error of this deviation as a percentage (ie. formula is: sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)
  When test only has one measure, the standard error cannot be computed and is replaced with a '[n/a]'.

• fly over image of failing tests to see the complete error message
• to look at the complete and detailed test results, click on its image

All 49 scenarios	RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)	RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)	RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)	Win XP Sun 1.4.2_10 (2 GHz 512 MB)	Win XP Sun 1.4.2_10 (3 GHz 2 GB)
BenchFileStore#testStoreExists()	+0.0% [±2.3]	+0.0% [±2.0]	- 1.1% [±1.7]	n/a	- 1.5% [±1.3]
BenchFileStore#testStoreIsReadOnly()	- 0.9% [±1.0]	+1.0% [±1.5]	+2.1% [±0.6]	n/a	- 0.6% [±1.3]
BenchFileStore#testStoreLastModified()	+0.9% [±0.6]	+0.0% [±2.0]	+1.1% [±0.5]	n/a	- 1.5% [±1.0]
BenchMiscWorkspace#testGetProject()	+2.4% [±1.0]	+1.0% [±1.1]	+0.0% [±0.7]	n/a	+1.8% [±2.8]
BenchMiscWorkspace#testNoOp()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 9.26s is not within [0%, 110%] of 8.33s - 11.2% [±4.9]	- 2.9% [±1.7]	+1.3% [±1.8]	n/a	- 4.0% [±0.8]
BenchWorkspace#testCountResources()	- 3.0% [±1.3]	- 5.9% [±1.2]	- 0.8% [±1.0]	n/a	+8.3% [±2.5]
BenchWorkspace#testCountResourcesDuringOperation()	+4.5% [±3.3]	- 0.6% [±0.9]	+0.8% [±0.8]	n/a	- 3.1% [±1.3]
BenchWorkspace#testFindMaxProblemSeverity()	+0.0% [±0.2]	- 1.0% [±0.1]	- 4.1% [±0.3]	n/a	- 0.7% [±0.2]
LocalHistoryPerformanceTest#testAddState()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 122ms is not within [0%, 110%] of 102ms - 19.6% [±44.5]	+1.9% [±19.7]	+0.0% [±17.8]	n/a	+3.0% [±7.4]
LocalHistoryPerformanceTest#testBug28603()	+4.8% [±6.4]	+0.0% [±3.0]	+0.0% [±2.9]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 62ms is not within [0%, 110%] of 49ms - 26.5% [±12.4]
LocalHistoryPerformanceTest#testClearHistory100x4()	- 3.3% [±41.4]	- 5.3% [±22.4]	- 5.6% [±23.6]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 27ms is not within [0%, 110%] of 19ms - 42.1% [±28.6]
LocalHistoryPerformanceTest#testClearHistory20x20()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 43ms is not within [0%, 110%] of 11ms - 290.9% [±226.0]	+88.9% [±89.5]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 9ms is not within [0%, 110%] of 8ms - 12.5% [±9.5]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 15ms is not within [0%, 110%] of 11ms - 36.4% [±35.8]
LocalHistoryPerformanceTest#testClearHistory4x100()	+44.4% [±36.2]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 127ms is not within [0%, 110%] of 11ms - 1,054.5% [±660.1]	+0.0% [±20.3]	n/a	+0.0% [±49.9]
LocalHistoryPerformanceTest#testCopyHistory100x4()	+11.0% [±9.9]	- 2.0% [±52.8]	- 4.9% [±6.0]	n/a	+5.1% [±6.0]
LocalHistoryPerformanceTest#testCopyHistory20x20()	+11.8% [±6.1]	- 1.6% [±7.8]	+3.7% [±3.5]	n/a	- 1.9% [±3.7]
LocalHistoryPerformanceTest#testCopyHistory4x100()	+17.4% [±12.0]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 65ms is not within [0%, 110%] of 46ms - 41.3% [±38.5]	+2.3% [±10.4]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 128ms is not within [0%, 110%] of 109ms - 17.4% [±13.3]
LocalHistoryPerformanceTest#testGetDeletedMembers100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 116ms is not within [0%, 110%] of 42ms - 176.2% [±186.0]	+4.5% [±8.9]	+0.0% [±8.1]	n/a	+34.3% [±33.6]
LocalHistoryPerformanceTest#testGetDeletedMembers20x20()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 83ms is not within [0%, 110%] of 29ms - 186.2% [±229.6]	+27.3% [±20.5]	+33.3% [±35.8]	n/a	+34.8% [±37.0]
LocalHistoryPerformanceTest#testGetDeletedMembers4x100()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 120ms is not within [0%, 110%] of 15ms - 700.0% [±710.0]	+0.0% [±4.5]	+0.0% [±7.1]	n/a	+0.0% [±0.0]
LocalHistoryPerformanceTest#testGetHistory()	+2.4% [n/a]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 32ms is not within [0%, 110%] of 28ms - 14.3% [n/a]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 27ms is not within [0%, 110%] of 22ms - 22.7% [n/a]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 31ms is not within [0%, 110%] of 16ms - 93.8% [n/a]
LocalHistoryPerformanceTest#testHistoryCleanUp100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.08s is not within [0%, 110%] of 451ms - 140.6% [±142.7]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 932ms is not within [0%, 110%] of 349ms - 167.0% [±165.9]	+0.0% [±6.3]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 809ms is not within [0%, 110%] of 643ms - 25.8% [±28.6]
LocalHistoryPerformanceTest#testHistoryCleanUp20x20()	+46.4% [±62.1]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.61s is not within [0%, 110%] of 549ms - 193.6% [±214.3]	+0.4% [±2.6]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.15s is not within [0%, 110%] of 624ms - 84.8% [±70.6]
MarkerPerformanceTest#testSetAttributes1()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 7.37s is not within [0%, 110%] of 6.54s - 12.7% [n/a]	+1.0% [n/a]	+0.0% [n/a]	n/a	+0.7% [n/a]
MarkerPerformanceTest#testSetAttributes2()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 7.69s is not within [0%, 110%] of 6.92s - 11.1% [n/a]	+0.2% [n/a]	- 0.9% [n/a]	n/a	- 2.5% [n/a]
PropertyManagerPerformanceTest#testGetProperty100x4()	- 1.3% [±1.2]	- 2.2% [±1.3]	+0.0% [±0.6]	n/a	- 0.2% [±0.9]
PropertyManagerPerformanceTest#testGetProperty20x20()	+12.1% [±9.5]	+10.4% [±3.7]	+0.4% [±0.1]	n/a	+1.7% [±0.2]
PropertyManagerPerformanceTest#testGetProperty4x100()	+3.7% [±0.1]	+4.8% [±0.1]	+0.2% [±0.1]	n/a	+1.5% [±0.0]
PropertyManagerPerformanceTest#testSetProperty100x4()	- 4.8% [±1.1]	- 3.6% [±2.1]	- 2.3% [±1.8]	n/a	- 3.1% [±2.2]
PropertyManagerPerformanceTest#testSetProperty20x20()	- 2.6% [±1.4]	- 6.6% [±3.3]	- 1.8% [±0.7]	n/a	- 7.4% [±4.0]
PropertyManagerPerformanceTest#testSetProperty4x100()	- 4.5% [±3.3]	+0.0% [±4.2]	- 1.4% [±1.3]	n/a	- 5.0% [±3.5]
WorkspacePerformanceTest#testCreateResources()	+34.9% [±25.7]	+9.0% [±17.9]	+6.8% [±24.7]	n/a	- 9.2% [±11.4]
WorkspacePerformanceTest#testDeleteProject()	+19.6% [±26.0]	+8.4% [±24.1]	+26.7% [±21.4]	n/a	+8.0% [±3.0]
WorkspacePerformanceTest#testFolderCopy()	+17.2% [±18.9]	+4.8% [±11.4]	+13.2% [±5.3]	n/a	- 4.5% [±1.9]
WorkspacePerformanceTest#testFolderMove()	+11.8% [±64.3]	+7.4% [±11.8]	+3.4% [±0.9]	n/a	+0.8% [±0.9]
WorkspacePerformanceTest#testRefreshProject()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 2.67s is not within [0%, 110%] of 720ms - 271.1% [±276.1]	- 9.9% [±13.4]	+1.9% [±2.3]	n/a	- 3.3% [±2.4]
BenchPath#testHash()	- 0.2% [±2.0]	+1.3% [±1.1]	- 3.4% [±0.2]	n/a	- 0.4% [±0.9]
BenchPath#testPathCreation()	+1.6% [±1.9]	+0.4% [±0.4]	- 0.2% [±0.3]	n/a	+0.0% [±0.8]
BenchPath#testToOSString()	+1.2% [±0.3]	- 0.8% [±0.5]	- 2.1% [±0.2]	n/a	+22.2% [±1.2]
BenchPath#testToString()	- 1.5% [±0.3]	- 0.7% [±0.6]	+1.2% [±0.3]	n/a	+23.4% [±0.9]
ContentTypePerformanceTest#testLoadCatalog()	+44.6% [n/a]	+49.0% [n/a]	+48.8% [n/a]	n/a	+42.0% [n/a]
ContentTypePerformanceTest#testNameMatching()	+5.1% [±0.7]	+10.9% [±0.5]	+2.8% [±0.6]	n/a	+7.2% [±1.3]
PreferencePerformanceTest#testGetStringCommonPrefixKeys()	+3.0% [±2.4]	- 0.9% [±2.2]	- 7.2% [±0.9]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.09s is not within [0%, 110%] of 951ms - 14.6% [±1.1]
PreferencePerformanceTest#testGetStringMisses()	- 5.0% [±4.2]	+3.4% [±1.6]	- 2.2% [±0.4]	n/a	Performance criteria not met when compared to '{config=eclipseperfwin2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 880ms is not within [0%, 110%] of 740ms - 18.9% [±1.9]
PreferencePerformanceTest#testGetStringSequentialKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.76s is not within [0%, 110%] of 1.48s - 19.0% [±1.0]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.09s is not within [0%, 110%] of 957ms - 13.9% [±0.7]	+1.0% [±0.9]	n/a	- 7.3% [±1.0]
PreferencePerformanceTest#testGetStringUniqueKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.64s is not within [0%, 110%] of 1.34s - 22.5% [±3.9]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1s is not within [0%, 110%] of 861ms - 16.4% [±1.1]	- 0.9% [±1.2]	n/a	- 9.9% [±1.4]
PreferencePerformanceTest#testPutStringKeys()	+2.1% [±0.7]	+0.9% [±1.4]	- 1.1% [±1.4]	n/a	+2.7% [±0.7]
PreferencePerformanceTest#testRemoveStringKeys()	+0.0% [±0.4]	+1.6% [±0.8]	+5.6% [±0.5]	n/a	+7.6% [±1.8]
StartupTest.testApplicationStartup	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 2.57s is not within [0%, 105%] of 2.1s - 22.8% [±18.0]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.48s is not within [0%, 105%] of 1.38s - 7.5% [±0.5]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 1.3s is not within [0%, 105%] of 1.21s - 7.3% [±0.1]	n/a	- 3.1% [±2.1]
UIStartupTest.testUIApplicationStartup	Performance criteria not met when compared to '{config=eclipseperflnx1, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 19.94s is not within [50%, 105%] of 17.47s - 14.2% [±10.4]	Performance criteria not met when compared to '{config=eclipseperflnx2, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 12.03s is not within [50%, 105%] of 11.22s - 7.3% [±5.0]	Performance criteria not met when compared to '{config=eclipseperflnx3, build=3.3_200706251500_200806171530, jvm=sun}': Elapsed Process: 9.99s is not within [50%, 105%] of 9.22s - 8.4% [±5.3]	n/a	- 3.8% [±10.1]