Performance of org.eclipse.core: M20070921-1145 relative to 3.2

RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)

Win XP Sun 1.4.2_10 (3 GHz 2 GB)

RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)

Win XP Sun 1.4.2_10 (2 GHz 512 MB)

RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)

Scenario Status

• an icon showing whether the test fails or passes and whether it's reliable or not.
  The legend for this icon is:
  • Green (): mark a successful result, which means this test has neither significant performance regression nor significant standard error
  • Red (): mark a failing result, which means this test shows a significant performance regression (more than 10%)
  • Gray (): mark a failing result (see above) with a comment explaining this degradation.
  • Yellow ( or ): mark a failing or successful result with a significant standard error (more than 3%)
  • Black (): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)
  • "n/a": mark a test for with no performance results
• the value of the deviation from the baseline as a percentage (ie. formula is: (build_test_time - baseline_test_time) / baseline_test_time)
• the value of the standard error of this deviation as a percentage (ie. formula is: sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)
  When test only has one measure, the standard error cannot be computed and is replaced with a '[n/a]'.

• fly over image of failing tests to see the complete error message
• to look at the complete and detailed test results, click on its image

All 51 scenarios	RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)	RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)	RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)	Win XP Sun 1.4.2_10 (2 GHz 512 MB)	Win XP Sun 1.4.2_10 (3 GHz 2 GB)
BenchFileStore#testStoreExists()	+0.0% [±2.2]	+0.0% [±2.1]	- 1.1% [±2.0]	+1.7% [±0.8]	+0.4% [±0.5]
BenchFileStore#testStoreIsReadOnly()	- 0.9% [±1.0]	- 1.0% [±1.5]	- 1.1% [±0.6]	+2.5% [±0.9]	+0.9% [±0.7]
BenchFileStore#testStoreLastModified()	+0.0% [±0.9]	+1.0% [±1.0]	+0.0% [±0.6]	+2.2% [±0.3]	+0.4% [±0.3]
*BenchMiscWorkspace#testGetProject() (vs. I20060801-0900)	+1.4% [±0.7]	+1.3% [±0.7]	+0.8% [±0.4]	- 5.4% [±0.6]	+5.2% [±0.4]
BenchMiscWorkspace#testNoOp()	+1.1% [±1.4]	+64.0% [±0.3]	+67.9% [±0.3]	+56.5% [±2.4]	+69.9% [±0.4]
BenchWorkspace#testCountResources()	- 4.8% [±1.4]	- 8.1% [±4.0]	+0.8% [±3.3]	+3.3% [±6.5]	- 2.7% [±2.5]
BenchWorkspace#testCountResourcesDuringOperation()	- 0.1% [±1.4]	- 1.9% [±0.9]	- 1.9% [±0.9]	- 0.9% [±0.4]	- 0.9% [±0.6]
*BenchWorkspace#testFindMaxProblemSeverity() (vs. N20070315-0010)	+1.7% [±0.6]	- 0.4% [±0.2]	+1.3% [±0.2]	+0.3% [±1.3]	- 1.1% [±0.1]
LocalHistoryPerformanceTest#testAddState()	+62.6% [±63.3]	+65.3% [±29.0]	+70.6% [±30.6]	+42.9% [±28.2]	+45.1% [±21.5]
LocalHistoryPerformanceTest#testBug28603()	+7.1% [±6.7]	+6.7% [±6.3]	+7.7% [±4.6]	+23.6% [±22.0]	+11.3% [±4.3]
LocalHistoryPerformanceTest#testClearHistory100x4()	+11.4% [±51.0]	Performance criteria not met when compared to '{config=eclipseperflnx2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 234ms is not within [0%, 110%] of 39ms - 500.0% [±550.5]	+13.6% [±31.9]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 31ms is not within [0%, 110%] of 23ms - 34.8% [±44.1]	+0.0% [±28.0]
LocalHistoryPerformanceTest#testClearHistory20x20()	+0.0% [±8.7]	+0.0% [±4.8]	+0.0% [±6.0]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 19ms is not within [0%, 110%] of 12ms - 58.3% [±46.7]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 15ms is not within [0%, 110%] of 4ms - 275.0% [±100.2]
LocalHistoryPerformanceTest#testClearHistory4x100()	+28.6% [±31.0]	Performance criteria not met when compared to '{config=eclipseperflnx2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 8ms is not within [0%, 110%] of 7ms - 14.3% [±3.6]	+14.3% [±5.1]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 23ms is not within [0%, 110%] of 8ms - 187.5% [±112.8]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 11ms is not within [0%, 110%] of 8ms - 37.5% [±75.1]
LocalHistoryPerformanceTest#testCopyHistory100x4()	+3.1% [±24.2]	+24.9% [±7.0]	Performance criteria not met when compared to '{config=eclipseperflnx3_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 775ms is not within [0%, 110%] of 532ms - 45.7% [±59.0]	- 3.0% [±12.2]	- 1.5% [±3.1]
LocalHistoryPerformanceTest#testCopyHistory20x20()	+52.3% [±36.7]	+64.0% [±48.9]	+24.3% [±20.2]	+9.4% [±8.6]	+1.1% [±7.5]
LocalHistoryPerformanceTest#testCopyHistory4x100()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 62ms is not within [0%, 110%] of 55ms - 12.7% [±15.4]	+35.7% [±38.1]	+34.4% [±40.7]	+15.2% [±17.4]	+12.1% [±19.3]
LocalHistoryPerformanceTest#testGetDeletedMembers100x4()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 74ms is not within [0%, 110%] of 51ms - 45.1% [±68.7]	+10.4% [±7.9]	+10.0% [±8.4]	+21.4% [±36.1]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 62ms is not within [0%, 110%] of 47ms - 31.9% [±33.0]
LocalHistoryPerformanceTest#testGetDeletedMembers20x20()	+25.0% [±29.2]	+0.0% [±3.1]	+7.1% [±5.1]	+0.0% [±2.3]	+25.8% [±27.5]
LocalHistoryPerformanceTest#testGetDeletedMembers4x100()	+48.0% [±46.0]	+58.6% [±63.8]	+37.5% [±9.9]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 31ms is not within [0%, 110%] of 23ms - 34.8% [±75.8]	+0.0% [±49.3]
LocalHistoryPerformanceTest#testGetHistory()	+10.0% [n/a]	- 6.7% [n/a]	+0.0% [n/a]	+0.0% [n/a]	+0.0% [n/a]
LocalHistoryPerformanceTest#testHistoryCleanUp100x4()	- 2.9% [±6.5]	Performance criteria not met when compared to '{config=eclipseperflnx2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 589ms is not within [0%, 110%] of 441ms - 33.6% [±58.9]	- 4.0% [±7.0]	+68.7% [±43.3]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 946ms is not within [0%, 110%] of 634ms - 49.2% [±44.0]
LocalHistoryPerformanceTest#testHistoryCleanUp20x20()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 578ms is not within [0%, 110%] of 441ms - 31.1% [±25.0]	+41.6% [±39.3]	- 6.0% [±1.9]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 2.06s is not within [0%, 110%] of 1.84s - 11.9% [±85.4]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 990ms is not within [0%, 110%] of 799ms - 23.9% [±31.5]
MarkerPerformanceTest#testSetAttributes1()	- 4.3% [n/a]	+5.0% [n/a]	+6.9% [n/a]	- 0.7% [n/a]	+5.1% [n/a]
MarkerPerformanceTest#testSetAttributes2()	- 7.7% [n/a]	+3.9% [n/a]	+5.4% [n/a]	+0.9% [n/a]	+3.6% [n/a]
PropertyManagerPerformanceTest#testGetProperty100x4()	+1.1% [±1.2]	+0.1% [±1.2]	- 0.5% [±0.7]	- 0.3% [±1.2]	- 0.3% [±0.7]
PropertyManagerPerformanceTest#testGetProperty20x20()	+1.1% [±0.1]	+12.4% [±14.4]	+0.0% [±0.1]	- 2.0% [±0.3]	- 4.1% [±0.1]
PropertyManagerPerformanceTest#testGetProperty4x100()	+0.3% [±0.0]	- 0.4% [±0.1]	+0.2% [±0.1]	- 3.4% [±0.1]	- 4.5% [±0.0]
PropertyManagerPerformanceTest#testSetProperty100x4()	- 2.8% [±0.6]	- 2.7% [±2.0]	+0.8% [±2.1]	- 3.1% [±1.5]	+0.3% [±1.0]
PropertyManagerPerformanceTest#testSetProperty20x20()	- 1.7% [±1.0]	+0.9% [±5.3]	+0.0% [±1.2]	- 3.0% [±4.5]	- 0.3% [±1.2]
PropertyManagerPerformanceTest#testSetProperty4x100()	+1.7% [±1.5]	+5.0% [±3.0]	+4.1% [±1.9]	- 1.5% [±3.3]	+0.0% [±1.9]
WorkspacePerformanceTest#testCreateResources()	+19.9% [±32.2]	+24.0% [±46.9]	+4.2% [±23.2]	- 5.1% [±12.0]	+11.9% [±17.4]
WorkspacePerformanceTest#testDeleteProject()	+14.9% [±24.7]	+43.3% [±21.0]	- 6.2% [±14.2]	- 5.3% [±4.1]	- 3.3% [±4.3]
WorkspacePerformanceTest#testFolderCopy()	+7.1% [±11.0]	+41.5% [±8.2]	+17.1% [±13.5]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.75m is not within [0%, 110%] of 1.48m - 18.0% [±4.8]	+1.6% [±11.7]
WorkspacePerformanceTest#testFolderMove()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 8.26s is not within [0%, 110%] of 3.23s - 155.7% [±86.8]	Performance criteria not met when compared to '{config=eclipseperflnx2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.42s is not within [0%, 110%] of 1.26s - 12.7% [±14.0]	Performance criteria not met when compared to '{config=eclipseperflnx3_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.11s is not within [0%, 110%] of 995ms - 11.9% [±10.1]	+1.8% [±1.1]	+4.1% [±12.5]
*WorkspacePerformanceTest#testRefreshProject() (vs. N20070215-0010)	+6.6% [±7.5]	- 8.3% [±3.1]	+6.8% [±9.1]	- 3.4% [±1.4]	- 2.0% [±0.4]
BenchPath#testHash()	- 2.0% [±0.2]	- 0.7% [±0.3]	- 2.1% [±0.2]	+3.3% [±0.2]	+1.2% [±0.3]
BenchPath#testPathCreation()	+1.2% [±0.8]	+2.4% [±0.5]	+0.3% [±0.4]	+0.9% [±0.5]	+1.5% [±0.4]
BenchPath#testToOSString()	+0.8% [±0.4]	- 1.9% [±0.5]	- 0.7% [±0.3]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.21s is not within [0%, 110%] of 868ms - 39.3% [±0.4]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 870ms is not within [0%, 110%] of 717ms - 21.3% [±0.4]
BenchPath#testToString()	+0.9% [±0.2]	- 1.0% [±0.4]	- 2.8% [±0.2]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.22s is not within [0%, 110%] of 853ms - 42.7% [±0.4]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 884ms is not within [0%, 110%] of 721ms - 22.6% [±0.5]
ContentTypePerformanceTest#testContentMatching()	- 4.2% [±2.0]	- 6.7% [±1.8]	- 2.0% [±1.4]	- 8.4% [±2.6]	- 6.8% [±2.0]
ContentTypePerformanceTest#testIsKindOf()	+0.5% [±0.9]	+0.0% [±0.7]	+0.5% [±0.8]	- 5.8% [±1.2]	- 1.7% [±0.6]
ContentTypePerformanceTest#testLoadCatalog()	+10.4% [±0.0]	+8.5% [±0.0]	+10.4% [±0.0]	+13.7% [±0.0]	+13.4% [±0.0]
ContentTypePerformanceTest#testNameMatching()	+1.5% [±0.6]	- 1.5% [±0.4]	- 6.1% [±0.5]	- 8.7% [±0.7]	- 5.2% [±0.6]
PreferencePerformanceTest#testGetStringCommonPrefixKeys()	+2.1% [±0.8]	+2.6% [±0.8]	- 0.5% [±1.0]	- 1.4% [±1.8]	- 2.7% [±2.1]
PreferencePerformanceTest#testGetStringMisses()	+18.5% [±0.3]	+16.9% [±0.4]	- 2.9% [±0.7]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.14s is not within [0%, 110%] of 1.01s - 12.8% [±3.2]	- 9.2% [±3.3]
PreferencePerformanceTest#testGetStringSequentialKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.57s is not within [0%, 110%] of 1.4s - 12.3% [±0.7]	- 7.6% [±1.0]	- 0.2% [±0.8]	+11.1% [±0.3]	+12.4% [±0.9]
PreferencePerformanceTest#testGetStringUniqueKeys()	Performance criteria not met when compared to '{config=eclipseperflnx1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 1.66s is not within [0%, 110%] of 1.22s - 36.2% [±21.3]	- 6.7% [±0.9]	- 0.7% [±0.8]	+10.1% [±2.9]	+15.5% [±1.2]
PreferencePerformanceTest#testPutStringKeys()	- 3.5% [±1.7]	- 1.8% [±0.7]	- 7.0% [±1.7]	- 8.4% [±0.2]	+3.6% [±0.4]
PreferencePerformanceTest#testRemoveStringKeys()	+0.0% [±0.4]	- 3.4% [±0.5]	+0.0% [±0.4]	- 9.0% [±1.3]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 67ms is not within [0%, 110%] of 59ms - 13.6% [±2.3]
StartupTest.testApplicationStartup	+6.6% [±0.0]	+8.3% [±0.0]	+15.7% [±0.0]	+0.0% [±0.0]	+0.9% [±0.0]
UIStartupTest.testUIApplicationStartup	- 2.8% [±0.0]	- 3.4% [±0.0]	Performance criteria not met when compared to '{config=eclipseperflnx3_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 9.12s is not within [50%, 105%] of 8.54s - 6.9% [±0.0]	Performance criteria not met when compared to '{config=eclipseperfwin1_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 11.87s is not within [50%, 105%] of 10.26s - 15.6% [±0.0]	Performance criteria not met when compared to '{config=eclipseperfwin2_R3.3, build=3.2_200606291905_200707261759, jvm=sun}': Elapsed Process: 7.6s is not within [50%, 105%] of 6.65s - 14.3% [±0.0]