90th Percentile in Performance Testing

90th Percentile in Performance Testing

‘Percentile’ – a common but most important term, especially in performance testing. From the day first when we started analysing any performance testing report, we heard about 90th percentile. Even our mentor gave more stress on 90th percentile figures for response time. So, what is this 90th percentile exactly?

Let’s try to understand with an example. If you had 10 sheep and each sheep eat some KGs of grass on a daily basis. One day you weighted the grass and noted the figures of each sheep’s intake. Refer to the below table:

Sheep#S1S2S3S4S5S6S7S8S9S10
Grass(kg)33.244.83.62.93.433.83.9

Now, you need to find out what amount of grass has been consumed by 90% of sheep? So simply you need to sort the number with respect to consumed grass and ignore the last value.

1st2nd3rd4th5th6th7th8th9th10th
Sheep#S6S1S8S2S7S5S9S10S3S4
Grass(kg)2.9333.23.43.63.83.944.8

90th percentile value in 10 entries is a 9th value which is 4, so just ignore S4 with 4.8 (keep it hungry for some days, it eats so much).

The conclusion is 90% of total sheep either eat 4 KGs grass or below, so you got an upper limit of grass consumption. In terms of performance testing, you need to sort response time of a particular transaction or request in increasing order and then ignore 10% of the total count having high values. The last highest number in the remaining values will be 90th percentile.

Example:

A performance test script is executed for 25 iterations. The response time of the login transaction of each iteration is:

S. No.Iteration No.Login (Response Time (in sec))
111.5
221.6
331.1
440.9
552.1
661.9
771.4
881
990.8
10101.5
11111.8
12121.1
13131.6
14141.7
15151.3
16160.9
17171
18181.5
19192.3
20201.9
21211.8
22221.2
23231.4
24240.9
25251.5

Now, sort the list in increasing order with respect to response time.

S. No.Iteration No.Login (Response Time)
190.8
240.9
3160.9
4240.9
581
6171
731.1
8121.1
9221.2
10151.3
1171.4
12231.4
1311.5
14101.5
15181.5
16251.5
1721.6
18131.6
19141.7
20111.8
21211.8
2261.9
23201.9
2452.1
25192.3

Now, 22.5 is the 90% of the number of transactions i.e. 25.

=> 25 x (90/100) = 22.5

Round-off to 23. So the 23rd value will be 90th percentile which is 1.9 seconds. It means 90% of total iterations having response time 1.9 seconds or less than it. Similarly, you can calculate other percentile values like 70th, 80th or 95th percentile.

How 90th percentile calculated in MS Excel?

MS Excel uses below formula to calculate 90th percentile:

90th Percentile = 0.9 * (Number of Values – 1) + 1

The same formula was used by LoadRunner version 6.5. The method to find out 90th percentile has been changed in LoadRunner version 7 and above.

Why we need 90th percentile in Performance Testing?

  • Percentile is often considered as a performance goal. If the given SLA has 90th percentile NFR and it meets during the test then it shows that 90% of the users have an experience that matches your performance goals. It gives additional confidence to the client over his application.
  • Sometimes average response time appears extremely high and individual datasets seem normal. Even a couple of peaks in response times, skew the average response time numbers and impact the test. In such scenarios, 90th percentile (or other percentile values) eliminate the unusual spike data from the result.
  • In reality, most of the applications have very few high spikes in the graph; a statistician would say that the curve has a long tail. A long-tail does not imply many slow transactions, but few that are magnitudes slower than the norm. In that case, 90th Percentile helpful because it ignores 10% of the request having the spike (this can be ignored).
  • If the 50th percentile (median) of response time is 5 seconds that means that 50% of the transactions are either as fast or faster than 5 seconds. If the 90th percentile of the same transaction is at 8 seconds it means that 90% are as fast or faster and only 10% are slower. The average, in this case, could either be lower than 5 seconds or somewhere in between. A percentile gives a much better sense of real-world performance because it shows a slice of response time curve.
  • If we calculate the difference of the 90th percentile value and the average response time value and divide this difference with the average response time value then it gives an idea of the spread of different data points. If the ratio is extremely small, it means that average and 90th percentile values are very close to each other and will indicate good and constant performance of the application. However, if the ratio is large, it shows high deviation in response time and non-uniform performance of the application. This is one of the methods where 90th percentile is useful, although I would recommend to draw your conclusion using standard deviation only.

Percentiles are a really great and easy way of understanding the real performance characteristics of your application. They also provide a great basis for automatic base-lining, application behavioural learning and optimizing your application with a proper focus. However, averages are ineffective because they are too simplistic and one-dimensional.  In short, percentile (90th, 95th, 99th) is great in performance testing world!


You may be interested:

  1. Performance Testing Tutorial
  2. Performance Engineering Tutorial
  3. LoadRunner Tutorial
  4. Apache JMeter Tutorial

4 Responses

  1. Binh Thanh Nguyen says:

    Thanks, nice explanation

  2. alekh says:

    Hi Gagan,

    Can you elaborate following scenarios.

    If the 50th percentile (median) of response time is 5 seconds that means that 50% of the transactions are either as fast or faster than 5 seconds. If the 90th percentile of the same transaction is at 8 seconds it means that 90% are as fast or faster and only 10% are slower. The average, in this case, could either be lower than 5 seconds or somewhere in between. A percentile gives a much better sense of real-world performance because it shows a slice of response time curve.

    If we calculate the difference of the 90th percentile value and the average response time value and divide this difference with the average response time value then it gives an idea of the spread of different data points. If the ratio is extremely small, it means that average and 90th percentile values are very close to each other and will indicate good and constant performance of the application. However, if the ratio is large, it shows high deviation in response time and non-uniform performance of the application. This is one of the methods where 90th percentile is useful, although I would recommend to draw your conclusion using standard deviation only.

Leave a Reply

Your email address will not be published. Required fields are marked *