Performance of org.eclipse.core: M20080911-1700 relative to 3.3 (200806171530)

RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)

Win XP Sun 1.4.2_10 (3 GHz 2 GB)

RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)

Win XP Sun 1.4.2_10 (2 GHz 512 MB)

RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)

Scenario Status

• an icon showing whether the test fails or passes and whether it's reliable or not.
  The legend for this icon is:
  • Green (): mark a successful result, which means this test has neither significant performance regression nor significant standard error
  • Red (): mark a failing result, which means this test shows a significant performance regression (more than 10%)
  • Gray (): mark a failing result (see above) with a comment explaining this degradation.
  • Yellow ( or ): mark a failing or successful result with a significant standard error (more than 3%)
  • Black (): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)
  • "n/a": mark a test for with no performance results
• the value of the deviation from the baseline as a percentage (ie. formula is: (build_test_time - baseline_test_time) / baseline_test_time)
• the value of the standard error of this deviation as a percentage (ie. formula is: sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)
  When test only has one measure, the standard error cannot be computed and is replaced with a '[n/a]'.

• fly over image of failing tests to see the complete error message
• to look at the complete and detailed test results, click on its image

All 49 scenarios	RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)	RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)	RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)	Win XP Sun 1.4.2_10 (2 GHz 512 MB)	Win XP Sun 1.4.2_10 (3 GHz 2 GB)
BenchFileStore#testStoreExists()	+1.8% [±2.5]	+1.0% [±1.9]	+2.1% [±1.5]	- 0.5% [±1.5]	- 2.5% [±1.4]
BenchFileStore#testStoreIsReadOnly()	+2.6% [±0.8]	+1.0% [±1.3]	+0.0% [±0.6]	+0.0% [±1.3]	- 1.1% [±1.3]
BenchFileStore#testStoreLastModified()	+2.6% [±0.7]	+1.0% [±1.8]	- 2.1% [±1.0]	+0.3% [±1.0]	- 1.5% [±1.1]
BenchMiscWorkspace#testGetProject()	+8.7% [±0.8]	- 1.1% [±0.9]	+0.4% [±0.7]	+1.8% [±2.1]	- 1.2% [±1.6]
BenchMiscWorkspace#testNoOp()	- 9.9% [±4.5]	+1.5% [±1.5]	- 0.8% [±1.8]	- 2.4% [±2.3]	- 5.0% [±2.2]
BenchWorkspace#testCountResources()	- 1.7% [±0.8]	- 4.6% [±1.3]	- 3.3% [±1.0]	+1.2% [±4.7]	+7.1% [±2.7]
BenchWorkspace#testCountResourcesDuringOperation()	+3.1% [±3.2]	- 1.7% [±1.0]	+1.7% [±0.8]	- 2.5% [±1.7]	- 4.5% [±1.2]
BenchWorkspace#testFindMaxProblemSeverity()	- 0.3% [±0.2]	- 0.3% [±0.1]	- 1.7% [±0.2]	+1.9% [±0.4]	+0.2% [±0.2]
LocalHistoryPerformanceTest#testAddState()	+2.9% [±37.4]	+0.0% [±19.8]	+0.0% [±17.9]	+9.6% [±12.7]	- 4.0% [±8.6]
LocalHistoryPerformanceTest#testBug28603()	+7.1% [±6.3]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 45ms is not within [0%, 110'%] of 29ms - 55.2% [±39.4]	+0.0% [±3.3]	- 3.8% [±3.9]	- 8.2% [±9.4]
LocalHistoryPerformanceTest#testClearHistory100x4()	+0.0% [±40.0]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 21ms is not within [0%, 110'%] of 19ms - 10.5% [±26.0]	- 5.6% [±23.6]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 38ms is not within [0%, 110'%] of 31ms - 22.6% [±48.9]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 27ms is not within [0%, 110'%] of 19ms - 42.1% [±28.6]
LocalHistoryPerformanceTest#testClearHistory20x20()	+0.0% [±6.3]	+90.1% [±89.5]	+12.5% [±4.8]	+0.0% [±2.7]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 15ms is not within [0%, 110'%] of 11ms - 36.4% [±69.3]
LocalHistoryPerformanceTest#testClearHistory4x100()	+44.4% [±36.1]	+36.4% [±29.6]	+25.0% [±14.2]	+8.3% [±46.2]	+0.0% [±49.9]
LocalHistoryPerformanceTest#testCopyHistory100x4()	+1.2% [±19.0]	+36.1% [±38.9]	+2.8% [±3.7]	+1.7% [±1.9]	+6.8% [±6.0]
LocalHistoryPerformanceTest#testCopyHistory20x20()	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 658ms is not within [0%, 110'%] of 170ms - 287.1% [±282.3]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 651ms is not within [0%, 110'%] of 127ms - 412.6% [±416.3]	- 2.8% [±2.9]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 487ms is not within [0%, 110'%] of 426ms - 14.3% [±7.8]	- 2.8% [±3.9]
LocalHistoryPerformanceTest#testCopyHistory4x100()	+17.4% [±12.1]	- 6.5% [±11.2]	+6.8% [±8.4]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 167ms is not within [0%, 110'%] of 146ms - 14.4% [±8.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 126ms is not within [0%, 110'%] of 109ms - 15.6% [±7.0]
LocalHistoryPerformanceTest#testGetDeletedMembers100x4()	+0.0% [±14.3]	+2.3% [±9.6]	+2.9% [±9.1]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 70ms is not within [0%, 110'%] of 54ms - 29.6% [±44.8]	+32.9% [±33.6]
LocalHistoryPerformanceTest#testGetDeletedMembers20x20()	+27.6% [±42.3]	+27.3% [±20.5]	+33.3% [±35.8]	+0.0% [±0.0]	+0.0% [±49.3]
LocalHistoryPerformanceTest#testGetDeletedMembers4x100()	+0.0% [±6.7]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 13ms is not within [0%, 110'%] of 11ms - 18.2% [±9.1]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 11ms is not within [0%, 110'%] of 10ms - 10.0% [±11.2]	+0.0% [±46.1]	+6.2% [±3.1]
LocalHistoryPerformanceTest#testGetHistory()	+57.1% [n/a]	+46.4% [n/a]	+40.9% [n/a]	+0.0% [n/a]	+0.0% [n/a]
LocalHistoryPerformanceTest#testHistoryCleanUp100x4()	+0.7% [±6.0]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 2.3s is not within [0%, 110'%] of 349ms - 581.1% [±422.8]	- 4.3% [±7.1]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 3.2s is not within [0%, 110'%] of 774ms - 318.6% [±307.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.0s is not within [0%, 110'%] of 643ms - 57.9% [±45.3]
LocalHistoryPerformanceTest#testHistoryCleanUp20x20()	- 5.4% [±93.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 697ms is not within [0%, 110'%] of 549ms - 27.0% [±68.5]	- 6.4% [±1.7]	+52.2% [±52.7]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.0s is not within [0%, 110'%] of 624ms - 66.2% [±38.9]
MarkerPerformanceTest#testSetAttributes1()	+2.7% [n/a]	+0.5% [n/a]	- 1.0% [n/a]	- 1.5% [n/a]	- 2.2% [n/a]
MarkerPerformanceTest#testSetAttributes2()	- 1.6% [n/a]	- 0.1% [n/a]	- 2.0% [n/a]	- 1.9% [n/a]	- 3.7% [n/a]
PropertyManagerPerformanceTest#testGetProperty100x4()	- 2.8% [±1.4]	- 2.4% [±1.4]	- 0.3% [±0.7]	- 0.9% [±2.0]	- 0.2% [±0.9]
PropertyManagerPerformanceTest#testGetProperty20x20()	+8.6% [±9.8]	+10.4% [±3.8]	+0.4% [±0.1]	- 1.0% [±0.7]	+1.9% [±0.2]
PropertyManagerPerformanceTest#testGetProperty4x100()	+1.3% [±0.0]	+4.8% [±0.1]	+0.3% [±0.1]	- 0.8% [±0.2]	+1.9% [±0.0]
PropertyManagerPerformanceTest#testSetProperty100x4()	- 5.2% [±1.0]	+0.6% [±1.8]	- 6.2% [±5.7]	+0.0% [±3.1]	- 1.7% [±2.2]
PropertyManagerPerformanceTest#testSetProperty20x20()	- 5.7% [±2.3]	- 1.8% [±3.0]	- 0.6% [±0.7]	- 8.9% [±10.6]	- 2.7% [±2.8]
PropertyManagerPerformanceTest#testSetProperty4x100()	- 0.6% [±2.4]	+0.6% [±3.9]	- 1.4% [±1.0]	- 3.0% [±8.1]	- 0.9% [±3.6]
WorkspacePerformanceTest#testCreateResources()	+11.7% [±32.5]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 9.2s is not within [0%, 110'%] of 7.0s - 32.0% [±41.0]	+11.4% [±27.6]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1mn 18.8s is not within [0%, 110'%] of 28.1s - 179.8% [±12.5]	- 2.6% [±10.5]
WorkspacePerformanceTest#testDeleteProject()	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 16.2s is not within [0%, 110'%] of 8.6s - 88.9% [±31.3]	+4.6% [±16.6]	+29.5% [±22.1]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 14.2s is not within [0%, 110'%] of 9.6s - 47.2% [±8.0]	+8.4% [±3.6]
WorkspacePerformanceTest#testFolderCopy()	- 7.9% [±20.9]	- 5.6% [±8.2]	- 6.3% [±19.8]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 3mn 19.8s is not within [0%, 110'%] of 1mn 33.9s - 112.7% [±4.3]	- 1.6% [±1.7]
WorkspacePerformanceTest#testFolderMove()	+1.0% [±76.9]	+11.3% [±11.8]	+1.1% [±1.6]	- 3.0% [±3.3]	- 0.6% [±0.7]
WorkspacePerformanceTest#testRefreshProject()	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 984ms is not within [0%, 110'%] of 720ms - 36.7% [±36.8]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 687ms is not within [0%, 110'%] of 607ms - 13.2% [±12.6]	+1.3% [±2.3]	- 1.1% [±4.1]	+0.5% [±2.3]
BenchPath#testHash()	+2.7% [±0.2]	+4.0% [±0.2]	- 2.5% [±0.2]	- 2.2% [±1.1]	- 1.7% [±0.9]
BenchPath#testPathCreation()	+0.0% [±1.9]	+0.0% [±0.4]	- 0.7% [±0.3]	+0.5% [±0.9]	+0.0% [±0.7]
BenchPath#testToOSString()	+0.6% [±0.2]	- 0.5% [±0.4]	+0.6% [±0.3]	+29.1% [±1.4]	+22.1% [±1.2]
BenchPath#testToString()	- 5.4% [±0.8]	- 0.5% [±0.5]	+0.7% [±0.3]	+29.9% [±1.2]	+23.0% [±1.1]
ContentTypePerformanceTest#testLoadCatalog()	+41.5% [n/a]	+49.0% [n/a]	+47.6% [n/a]	+62.0% [n/a]	+43.2% [n/a]
ContentTypePerformanceTest#testNameMatching()	+5.0% [±0.7]	+4.4% [±0.6]	+1.7% [±0.6]	+5.7% [±2.3]	+8.1% [±1.4]
PreferencePerformanceTest#testGetStringCommonPrefixKeys()	+3.5% [±0.8]	+0.7% [±1.1]	- 7.7% [±0.9]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.7s is not within [0%, 110'%] of 1.5s - 12.0% [±1.2]	- 7.7% [±1.2]
PreferencePerformanceTest#testGetStringMisses()	- 2.1% [±1.4]	+4.5% [±1.6]	- 3.5% [±0.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.3s is not within [0%, 110'%] of 1.1s - 19.5% [±2.1]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 879ms is not within [0%, 110'%] of 740ms - 18.8% [±2.3]
PreferencePerformanceTest#testGetStringSequentialKeys()	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.7s is not within [0%, 110'%] of 1.4s - 19.0% [±0.9]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.1s is not within [0%, 110'%] of 957ms - 15.2% [±0.5]	+0.4% [±0.9]	- 5.8% [±1.0]	- 7.2% [±1.1]
PreferencePerformanceTest#testGetStringUniqueKeys()	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.5s is not within [0%, 110'%] of 1.3s - 18.9% [±0.5]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.0s is not within [0%, 110'%] of 861ms - 20.3% [±3.4]	+0.0% [±1.1]	- 7.6% [±1.7]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 920ms is not within [0%, 110'%] of 829ms - 11.0% [±1.9]
PreferencePerformanceTest#testPutStringKeys()	- 1.6% [±0.8]	+3.3% [±1.4]	+1.7% [±1.4]	+3.8% [±1.0]	+3.6% [±0.6]
PreferencePerformanceTest#testRemoveStringKeys()	+0.0% [±0.4]	+3.1% [±0.4]	+1.9% [±0.5]	+5.4% [±1.5]	+9.1% [±1.8]
StartupTest.testApplicationStartup	- 5.0% [±1.8]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.7s is not within [0%, 110'%] of 1.3s - 25.8% [±17.8]	- 8.0% [±0.1]	- 2.7% [±2.2]	- 0.9% [±1.7]
UIStartupTest.testUIApplicationStartup	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 26.5s is not within [0%, 110'%] of 17.4s - 51.8% [±10.8]	- 5.9% [±5.2]	- 9.1% [±5.0]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 14.3s is not within [0%, 110'%] of 12.4s - 14.8% [±16.0]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 8.8s is not within [0%, 110'%] of 7.7s - 13.5% [±12.6]