Non-Functional Requirement Gathering

Non Functional Requirement

What are the Non-Functional Requirements?

Non-functional requirements are the testing goals which are created especially for performance testing, security testing, usability testing, etc. It is a combined requirement for all types of non-functional test. Since PerfMatrix is a core performance testing site, so non-functional word will be specific to Performance Testing only. Some simple examples of Non-Functional Requirements are:

  • The number of users handled by the application
  • Page Response Time
  • The number of requests processed by the application per unit time
  • CPU Utilization
  • Memory Utilization
  • Error rate etc.

In non-functional requirement, some of the goals are time-bound like the response time of a page, request per second, resource utilization etc. whereas some of the goals are load bound like real-world user load, throughput etc.

Difference between functional and non-functional requirement:

The main difference in functional and non-functional requirement is that functional requirements are end-user result oriented and stressed over the correct output. For example: If a user uses a banking application and clicks ‘Account Balance’ link then the application must display the correct balance available in his account. On the other hand, the non-functional requirements are oriented to the performance of the system in terms of responsiveness and load-bearing capacity. For example: If 1000 users hit ‘Account Balance’ link at the same time then the application must respond to all the users within a defined time (say 3 seconds).

Purpose of NFR Gathering:

The project team or client sets the expectation for performance testing in the form of non-functional requirements. To collect the requirement, analyse them from performance testing perspective and finalise the quantitative NFRs; all these steps fall under the NFR gathering phase of PTLC (Performance Test Life Cycle). All the requirements are documented, categorized and concluded in the Non-Functional Requirement Document. The end result of this phase provides quantitative NFRs which helps to prepare a correct workload model during performance testing.

Accountability:

Performance Test Manager or Performance Test Lead has a responsibility to collect, discuss and finalise the Non-functional requirement.

Approach:

Once the performance testing scope is finalised in the risk assessment phase, then either of them (Performance Test Manager or Lead) has to schedule a meeting with the project team to understand the client’s expectation. He may get the requirement in layman term which may require thorough study and deep analyse to extract the testable NFRs. There is also a high probability not to get clear and reasonable requirements in one meeting so he needs to set-up multiple meetings with the project stakeholders. Once all the non-functional requirements are finalised and he gets the testable NFRs then these NFRs should be properly documented in the Non-Functional Requirement Document and get the approval from the project stakeholders.

How to proceed practically?

“How to get the performance requirements from a non-technical customer?” This is a valuable question that everyone has. In most of the cases, the non-functional requirements are either defined incompletely or they are more conceptually rather than quantitative. A Performance Test Manager/Lead needs to spend his time to understand the client’s expectation by asking the right set of questions and transforming the conceptual requirements into quantitative goals. He needs to play the role of a mediator who bridges the gap between the novice user language and performance testing terminologies.

It is the responsibility of Performance Test Manager/Lead to get the essence from the customer. One needs to quickly understand the customer and should start talking in their language. Don’t talk using performance testing jargons and make the customer feel that they are ignorant, which is not the case. Don’t expect that the customer will give the performance goals in a single shot. Rather, start the discussion with an overview and purpose of performance testing. The discussion can be started with questions like

  • What is the expected user load they are looking for?
  • How much the expected response time for a page?
  • What are the types of performance test can be included? Explain each type of performance tests and their purpose.

These simple questions make the client comfortable to explain what he wants.

Some additional Tips:

Also, try to build a good rapport with the customer. During the meeting/call, educate the customer by explaining some core performance testing terms with some good examples, so that he can explain exactly what his expectations are. The most important thing which a performance manager/lead should always avoid, not to scare the client by asking lots of question in one shot or talking too much technical. The way of asking the requirement should be very polite to impresses the customer and get his confidence.

Once the NFRs are collected, draft them in Non-Functional Requirement Document along with the date and points discussed in the meetings. The final Non-Functional Requirement Document must be signed-off from all the project stakeholders.

Challenges:

Non-Functional Requirement gathering is the toughest phase of performance testing. Many times a Performance Test Manager/Lead face challenges to collect the correct performance testing requirement and conclude them as a proper client’s expectation. In most of the cases, applications are new and hence a Performance Test Manager/Lead does not have any clue on the NFRs (non-functional requirements). Many times, the client pick some random numbers over the call and ask to perform NFT (non-functional testing) against those numbers and at the end of the test when such NFRs do not meet, client forced to modify the NFRs and pass the test at any cost.

Such performance tests are executed to showcase that performance testing had been done and the application performed very well in the test environment. But in reality, 82% of such applications are failed due to performance issue within 6 months or even in the less time period when user load increases in production. Hence key points are to analyse the NFR thoroughly, do not consider (agree) any random number as NFR and finalise the quantitative NFRs without any caveat.

Deliverable:

Non-functional requirement document has to draft once the scope is finalised. The outcome of each meeting may lead to multiple changes in the document. Non-functional requirement document should cover all the aspects along with major and minor changes in the requirement. Do not forget to mention the acceptable error percentage in the document because this is one of the criteria which will help to decide RAG status at the end of the testing.

Non-Functional Requirement Document Template:

Download the template of Non-Functional Requirement Document.

Example:

Let’s call PerfMate. In the previous phase (Risk Assessment) of the project, PerfMate has conducted the risk assessment and finalise the scope of performance testing. He got the approval from all the stakeholders on Risk Assessment document. Moving to the next phase, he starts to collect the non-functional requirement (specific to performance testing) and receives the following requirements:

  1. The application should be very fast.
  2. The response time of the application should be quick.
  3. The web server performance should be as high as possible.
  4. The application should support many users.
  5. The servers should not fail when a sudden load comes during sale and offer periods.
  6. The application should run without any failures for a long duration.

From a client point of view, he has given all the requirements and set his expectation, but from PerfMate perspective, he just got the conceptual requirements. Now, PerfMate explained to the client that the provided information is partial and cannot help him to define the performance testing goal. With a polite manner, he gained the customer’s confidence and convinced him to provide the quantitative requirements. He asked project stakeholders to answer some of his basic questions. At last, he managed to gather the following requirements in the next couple of days:

Question: How many types of users are using this application (GUI)?

Answer. 4 types of users

  1. Admin
  2. Seller
  3. End-user
  4. Call center employee

Question: What are the business scenarios of every user?

Answer: Business scenarios for each type of users are like:

Admin:

  1. To approve/reject a new seller
  2. To verify a newly added product and approve/reject it

Seller:

  1. To add a new product
  2. To delete the existing product

Buyer:

  1. To buy a product
  2. To cancel the order

Call Center Employee:

  1. To register the complain

Question: What is the AUT current and predicted peak user load for all its users’ actions over time?

Answer: Admin

  • Current: 4
  • Predicted: 10

Seller

  • Current: 25
  • Predicted: 100

Buyer

  • Current: 438
  • Predicted: 2500

Call Center

  • Current: 8
  • Predicted: 24

Question: What is the average of active user count (including all types of users) at a time?

Answer:

  • Total: 304
  • Admin: 3
  • Seller: 15
  • Buyer: 278
  • Call Center: 8

Question: What could be the active user count during peak hour?

Answer:

  • Total: 500
  • Admin: 4
  • Seller: 50
  • Buyer: 438
  • Call Center: 8

Question: Anytime during a day or month when average user count is suddenly increased?

Answer: 31st of every month there is a 1-minute sale from 09:00 to 09:01 and 10:00 to 10:01. During this period the active buyer count becomes three times i.e. 834 and active seller count becomes 23. None of the other (Admin/Customer Care) users count change.

  • Total: 868
  • Admin: 3
  • Seller: 23
  • Buyer: 834
  • Call Center: 8

Question: What is the request count receive at Admin end?

Answer: 200 requests per hour

Question: What would be the response time NFRs for all the scenarios?

Answer: None of the pages should breach 3 seconds average response time NFR. This NFR is applicable for all type of users, scenarios, and pages.

<Rest of the figures are displayed in the below NFR table>

Question: What would be the server-side NFRs?

Answer:

  • CPU utilization must not be more than 60%; except stress test.
  • Pre, post and steady-state memory utilization difference must not be more than 15%; except stress test.
  • Pre, post and steady-state disk utilization must not be more than 25%; except stress test.

Question: What is the size of the performance testing environment with respect to production?

Answer: 100% (Live like environment)

More or less PerfMate got the answers to all of his queries and noted down in the NFR document and share with the client for the approval. After getting approval on the NFR Document, he will start the preparation of the performance test strategy.

A typical NFR format prepared by PerfMate is:

Note: Do not consider the below table as a standard NFR format. It was made simple for understanding purpose; especially for beginners.

NFR ID Category Description Impact to
NFR01 Application The solution must be able to support 500 active users.
1. Admin: 4 (2 for seller and 2 for product approval)
2. Seller: 50
3. Buyer: 438
4. Call Center: 8
1. Admin
2. Seller
3. Buyer
4. Call Center
NFR02 Application The solution must be able to support the future volume of active users i.e. 2634
1. Admin: 10
2. Seller: 100
3. Buyer: 2500
4. Call Center: 24
1. Admin
2. Seller
3. Buyer
4. Call Center
NFR03 Application The solution must be performed well during the longer period of time with average volume. i.e. 304
1. Admin: 3
2. Seller: 15
3. Buyer: 278
4. Call Center: 8
1. Admin
2. Seller
3. Buyer
4. Call Center
NFR04 Application The solution must be able to support the spike load of the buyer and seller during the sale period.
1. Admin: 3
2. Seller: 23
3. Buyer: 834
4. Call Center: 8
1. Admin
2. Seller
3. Buyer
4. Call Center
NFR05 Application Admin gets an average of 200 requests per hour every time. 1. Admin
NFR06 Application The number of orders:
1. Peak Hour volume: 1340
2. Sale Hour Volume: 2830
3. Future Volume: 7500
4. Average Volume: 600
Note: 4% of the users cancel the order in every scenario.
1. Buyer
NFR07 Application Sellers add an average of 180 products per hour and delete 18 existing products every hour 1. Seller
NFR08 Application The call center employees get 40 complains per hour 1. Call Center
NFR09 Application The response time of any page must not exceed 3 seconds except stress test 1. Admin
2. Seller
3. Buyer
4. Call Center
NFR10 Application The error rate of transactions must not exceed 1% 1. Admin
2. Seller
3. Buyer
4. Call Center
NFR11 Server The CPU Utilization must not exceed 60% 1. Web
2. App
3. DB
NFR12 Server The Memory Utilization must not exceed 15% (Compare pre, post and steady state memory status) 1. Web
2. App
3. DB
NFR13 Server The disk Utilization must not exceed 15% (Compare pre, post and steady state memory status) 1. Web
2. App
3. DB
NFR14 Server There must not any memory leakage 1. Web
2. App
3. DB
NFR15 Application Buyers order at the average rate of

1. Peak Hour Rate: 3.06 products per hour

2. Sale Hour Rate: 3.39 products per hour

3. Future Volume: 3 products per hour

4. Average Volume Rate: 2.15 products per hour

1. Buyer

2 Responses

  1. Venky says:

    very good information.
    One question from me – If the application is completely new and Client has no knowledge on performance testing, how can we gather Non Functional Requirements in such situation?

Leave a Reply

Your email address will not be published. Required fields are marked *