Performance Test Execution

What is Performance Test Execution?

Performance Test Execution word refers to running the tests specific to performance testing like load test, soak test, stress test, spike test etc. using a performance testing tool. Performance Test Plan contains detailed information on all these tests which need to be executed in the performance testing window. Basically, this phase has two sub-phases:

• Test Execution: To run the planned performance tests
• Result Analysis: To analyse the test result and prepare an interim test report

Purpose:

The Performance Test execution phase has the following activities to be performed:

• Execute the documented/agreed performance test
• Analyse the performance test result
• Verify the result against defined NFRs
• Prepare an Interim Performance Test Report
• Take the decision to complete or repeat the test cycle based on the interim test result

Accountability:

A performance test analyst or engineer executes the tests as per the testing timelines. A performance test lead or manager has a responsibility to analyse the test result and make a way forward plan on the basis of test results. If a performance test analyst or engineer has enough experience to understand the performance testing points then he can also analyse the result. In such a case, a performance test lead or manager has a responsibility to verify the report at his end before sending to the project team.

Approach for Test Execution:

Pre-execution activities:

Before starting the test execution there are a few pre-requisite that a performance tester should follow. He must pay attention to the below points:

1. Verify all the performance test scripts locally
2. Validate all the scenarios
3. Check all the external file paths in the test script. The file path should match the file location available at the load generator
4. Check whether the load generator and controller have sufficient disk space
5. Reboot the controllers and all the load generators (if feasible)
6. All the performance tests should run on the latest code of the application
7. The performance test environment should have the QA (functional test) passed version code only so that application is free from any functional bug
8. Verify the script on the load generator by running a smoke test before starting the actual load test
9. Verify all the test data (if feasible) so that there is no failure in the test due to the test data issue
10. Restart Web/Application/Database servers before starting a test
11. Perform server logs clean-up activity
12. Conduct a quick Healthcheck to verify the stability of the environment
13. Verify whether all the required monitors are up and running
14. Validate the run-time setting and parameter files
15. If the test is scheduled then check the system time must be synced with the testing tool time so that test can be started at the correct time.

After verifying all the checkpoints, a performance tester can press the ‘Run’ button to start the test.

Test Execution:

Once the test starts then check the graphs and stats in the live monitors of the testing tool. A performance tester needs to pay attention to some basic metrics like active users, transactions per second, hits per second, throughput, error count, error type etc. Also, he needs to check the behaviour of the users against the defined workload. At last, the test should stop properly and the result should be collated correctly at the given location.

Post-execution activities:

Once a performance test completes then a performance tester collects the result and starts the result analysis which acts as a post-execution activity. A performance tester should follow the below approach to conduct the performance test result analysis.

Approach for Test Result Analysis:

As mentioned at the beginning of this post, the second sub-phase of the performance test execution stage is Result Analysis. Performance Test Result Analysis is an important and more technical part of performance testing. Performance test result analysis requires expertise to determine the bottleneck and remediation options at the appropriate level of the software system – business, middleware, application, infrastructure, network etc.

Pre-result analysis activities:

Before starting the performance test result analysis, a performance tester should follow these important points:

1. The test should run for the defined duration
2. Filter out the ramp-up and ramp-down duration
3. Eliminate ‘Think Time’ from the graphs/stats
4. Eliminate ‘Pacing’ (if the tool counts) from the graphs/stats
5. No tool-specific error should occur like the failure of the load generator, memory issue etc.
6. No network-related issues should occur during the test like network failure, LGs becoming disconnected from the network etc.
7. The testing tool should collect the results from all the load generators and prepare a combined test report
8. CPU and Memory utilization percentage should be noted down for the pre-test (at least 1 hour), post-test (at least 1 hour) and during the test.
9. Use proper granularity to identify correct peaks and lows
10. Use the filter option and eliminate the unwanted transactions (if any)

Result Analysis:

1. Start the analysis with some basic metrics like
   1. Number of Users: The actual load during steady state should meet the user load NFR
   2. Response Time: The actual response during steady state should meet the response time NFR. The response time should be measured at two levels – Individual transaction response time and end-to-end response time. If NFRs are available for both levels then they should meet
   3. Transactions per second / Iterations per hour: If any of these metrics is defined then the actual result should match the defined figure
   4. Throughput: Throughput should match (not apple-to-apple) for the same set of tests
   5. Error: The error count should be less than the defined error tolerance limit
   6. Passed Transaction counts: Ideally, the passed count of the first transaction should match the passed count of the last transaction. If it is not the case then identify the failed transactions
2. Analyse the graphs:
   1. Set the proper granularity for all the graphs
   2. Read the graph carefully and note down the points
   3. Check the spikes and lows in the graph
   4. Merge the different graphs to identify the root cause of the issue
   5. If the performance testing tool and monitoring tool are not integrated then note the time when the error occurred and sync the graphs generated by both the tools
   6. Do not extrapolate the result on the basis of incomplete statistics
3. Analyse the other reports:
   1. Generate the heap dump and analyse the Java heap during the test
   2. Perform thread dump analyse to check the deadlock or stuck thread
   3. Analyse the Garbage Collector logs
   4. Analyse the AWR report to find a long time taking DB query

Post-result analysis activities:

A performance tester gathers all the results i.e. client-side and server-side stats and starts analysing the result. He verifies the results against the defined NFRs. After each test, the performance tester prepares an interim test report which is analysed by a Performance Test Lead or Manager.

Some key points for reporting:

1. It is good practice to generate an individual test report for all the tests
2. Define a template for the test report and use the same template to generate the report
3. Highlight the observations and defects in the test report
4. If the performance testing tool does not have reporting feature then prepare an interim test report (template link is available in the deliverable section of this post)
5. Attach all the relevant reports like heap dump analysis report, AWR report etc. with the interim test report
6. Provide defect description along with defect ID
7. Conclude the result as Pass or Fail status

Along with the result if a performance tester detects any performance bottleneck then he raises the defect and assigns it to the respective team for further investigation of the root cause. The root cause analysis activity is basically a team effort wherein the performance tester, system administrators, technical experts and DBA play a vital role. The test execution and bottleneck analysis activities are cyclic in nature.

After the tuning of the application, the same test is repeated again to verify the performance of the application. If the issue still persists then again application is pushed back for the tuning to meet the NFRs.

Deliverable:

Interim Performance Test Report is the only deliverable of this phase. Download the template of the Interim Performance Test Report

Example:

PerfMate has created the test scripts and scenarios in the last two phases and is now ready to start the test execution cycle. He checks all the required pre-requisite and starts the test. As per Performance Test Plan, he needs to conduct various tests on the application like load test, stress test, soak test and spike test.

Cycle#	Round#	Test Type	Test ID
Cycle 01	Round 01	Load Test	C1R1Load
	Round 01	Stress Test	C1R1Stress
	Round 01	Soak Test	C1R1Soak
	Round 01	Spike Test	C1R1Spike
	Round 02	Load Test	C1R2Load
	Round 02	Stress Test	C1R2Stress
	Round 02	Soak Test	C1R2Soak
	Round 02	Spike Test	C1R2Spike
Cycle 02	Round 01	Load Test	C2R1Load
	Round 01	Stress Test	C2R1Stress
	Round 01	Soak Test	C2R1Soak
	Round 01	Spike Test	C2R1Spike
	Round 02	Load Test	C2R2Load
	Round 02	Stress Test	C2R2Stress
	Round 02	Soak Test	C2R2Soak
	Round 02	Spike Test	C2R2Spike

After finishing each test, he analyses the application performance against the defined NFRs. He also checks the bottlenecks and raises all the identified defects in the defect management tool. 

At last, he prepares an interim test report and shares it with the project stakeholders. 

---