Performance of org.eclipse.team: M20080911-1700 relative to 3.3 (200806171530)

Scenario Status

• an icon showing whether the test fails or passes and whether it's reliable or not.
  The legend for this icon is:
  • Green (): mark a successful result, which means this test has neither significant performance regression nor significant standard error
  • Red (): mark a failing result, which means this test shows a significant performance regression (more than 10%)
  • Gray (): mark a failing result (see above) with a comment explaining this degradation.
  • Yellow ( or ): mark a failing or successful result with a significant standard error (more than 3%)
  • Black (): mark an undefined result, which means that deviation on this test is not a number (NaN) or is infinite (happens when the reference value is equals to 0!)
  • "n/a": mark a test for with no performance results
• the value of the deviation from the baseline as a percentage (ie. formula is: (build_test_time - baseline_test_time) / baseline_test_time)
• the value of the standard error of this deviation as a percentage (ie. formula is: sqrt(build_test_stddev^2 / N + baseline_test_stddev^2 / N) / baseline_test_time)
  When test only has one measure, the standard error cannot be computed and is replaced with a '[n/a]'.

• fly over image of failing tests to see the complete error message
• to look at the complete and detailed test results, click on its image

All 17 scenarios	RHEL 3.0 Sun 1.4.2_10 (2 GHz 512 MB)	RHEL 3.0 Sun 1.4.2_10 (3 GHz 2 GB)	RHEL 4.0 Sun 1.4.2_10 (3 GHz 2.5 GB)	Win XP Sun 1.4.2_10 (2 GHz 512 MB)	Win XP Sun 1.4.2_10 (3 GHz 2 GB)
SyncTests#testSync100NoUI()AddedFiles	+7.1% [±4.2]	+10.7% [±5.0]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 3.0s is not within [0%, 110'%] of 2.6s - 15.0% [±5.3]	- 0.8% [±4.1]	+10.3% [±5.1]
SyncTests#testSync100NoUI()ModifiedFiles	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 2.9s is not within [0%, 110'%] of 2.6s - 11.8% [±8.8]	- 1.5% [±8.4]	+1.0% [±3.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 3.9s is not within [0%, 110'%] of 3.2s - 21.9% [±5.5]	+6.5% [±3.6]
SyncTests#testSync100NoUI()RemovedFiles	- 8.1% [±5.5]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 2.8s is not within [0%, 110'%] of 2.5s - 12.0% [±9.0]	+0.3% [±2.3]	+15.7% [±4.3]	- 5.1% [±3.5]
WorkflowTests#testBigWorkflowNoUI()Checkout	+4.3% [±5.0]	+2.9% [±4.0]	+6.3% [±2.7]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 9.1s is not within [0%, 110'%] of 6.5s - 38.5% [±4.3]	+2.6% [±5.4]
WorkflowTests#testBigWorkflowNoUI()Commit1	+12.9% [±9.1]	+10.7% [±10.8]	+1.7% [±12.5]	+12.7% [±7.3]	+4.5% [±7.6]
WorkflowTests#testBigWorkflowNoUI()Commit2	+4.8% [±2.3]	- 2.4% [±4.0]	+0.5% [±3.5]	+4.2% [±4.5]	+0.4% [±0.9]
WorkflowTests#testBigWorkflowNoUI()Commit3	+17.3% [±10.5]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.4s is not within [0%, 110'%] of 1.2s - 11.5% [±10.4]	+10.1% [±3.4]	+3.5% [±5.3]	- 6.4% [±7.2]
WorkflowTests#testBigWorkflowNoUI()Commit4	+13.1% [±7.3]	- 8.5% [±5.6]	+5.3% [±2.5]	+8.5% [±5.6]	+9.2% [±2.6]
WorkflowTests#testBigWorkflowNoUI()Replace1	+20.5% [±7.0]	+26.7% [±6.5]	+31.3% [±8.5]	+9.2% [±5.8]	+19.6% [±4.6]
WorkflowTests#testBigWorkflowNoUI()Replace2	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 2.8s is not within [0%, 110'%] of 2.1s - 34.9% [±7.5]	- 2.2% [±10.2]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.9s is not within [0%, 110'%] of 1.4s - 31.4% [±9.1]	+1.2% [±2.9]	- 0.6% [±3.2]
WorkflowTests#testBigWorkflowNoUI()Replace3	- 5.2% [±6.3]	+4.4% [±10.4]	Performance criteria not met when compared to '3.3_200706251500_200806171530': Elapsed Process= 1.4s is not within [0%, 110'%] of 1.3s - 12.1% [±8.0]	+7.4% [±8.1]	+2.0% [±2.1]
WorkflowTests#testBigWorkflowNoUI()Share	+3.1% [±2.4]	- 0.2% [±2.0]	+3.0% [±2.8]	- 3.0% [±1.0]	+5.9% [±2.0]
WorkflowTests#testBigWorkflowNoUI()Tag1	+8.6% [±4.3]	+16.5% [±8.7]	+17.3% [±9.0]	+11.7% [±2.6]	+14.9% [±3.1]
WorkflowTests#testBigWorkflowNoUI()Update1	- 0.4% [±2.9]	+3.8% [±9.8]	- 5.4% [±8.0]	+4.3% [±1.4]	+2.3% [±1.7]
WorkflowTests#testBigWorkflowNoUI()Update2	+7.8% [±1.8]	+7.5% [±2.3]	+5.2% [±2.4]	+6.0% [±1.3]	+3.7% [±1.4]
WorkflowTests#testBigWorkflowNoUI()Update3	+0.2% [±4.6]	+8.3% [±2.7]	+3.9% [±3.3]	+6.6% [±1.0]	+4.5% [±2.3]
WorkflowTests#testBigWorkflowNoUI()Update4	+9.1% [±5.6]	+13.8% [±6.2]	+9.9% [±4.9]	+9.3% [±2.8]	+5.8% [±1.8]