Non-Functional Requirement Gathering

What are the Non-Functional Requirements?

Non-functional requirements are the testing goals which are created especially for performance testing, security testing, usability testing, etc. It is a combined requirement for all types of non-functional tests. Since PerfMatrix is a core performance testing site, here the non-functional word will be specific to Performance Testing only. Some simple examples of Non-Functional Requirements are:

• The number of users handled by the application
• Page Response Time
• The number of requests processed by the application per unit of time
• CPU Utilization
• Memory Utilization
• Error rate etc.

In non-functional requirement, some of the goals are time-bound like the response time of a page, request per second, resource utilization etc. whereas some of the goals are load-bound like real-world user load, throughput etc.

Difference between functional and non-functional requirements:

The main difference between functional and non-functional requirement is that functional requirements are end-user result oriented and stressed over the correct output. For example: If a user uses a banking application and clicks the ‘Account Balance’ link then the application must display the correct balance available in his account. On the other hand, the non-functional requirements are oriented to the performance of the system in terms of responsiveness and load-bearing capacity. For example: If 1000 users hit the ‘Account Balance’ link at the same time then the application must respond to all the users within a defined time (say 3 seconds).

Purpose of NFR Gathering:

The project team or client sets the expectation for performance testing in the form of non-functional requirements. To collect the requirement, analyse them from a performance testing perspective and finalise the quantitative NFRs; all these steps fall under the NFR gathering phase of PTLC (Performance Test Life Cycle). All the requirements are documented, categorized and concluded in the Non-Functional Requirement Document. The end result of this phase provides quantitative NFRs which help to prepare a correct workload model during performance testing.

Accountability:

The Performance Test Manager or Performance Test Lead has a responsibility to collect, discuss and finalise the Non-functional requirement.

Approach:

Once the performance testing scope is finalised in the risk assessment phase, then either of them (the Performance Test Manager or Lead) has to schedule a meeting with the project team to understand the client’s expectations. He may get the requirement in layman’s terms which may require thorough study and deep analyse to extract the testable NFRs. There is also a high probability of not getting clear and reasonable requirements in one meeting so he needs to set up multiple meetings with the project stakeholders. Once all the non-functional requirements are finalised and he gets the testable NFRs then these NFRs should be properly documented in the Non-Functional Requirement Document and get approval from the project stakeholders.

How to proceed practically?

“How to get the performance requirements from a non-technical customer?” This is a valuable question that everyone has. In most cases, the non-functional requirements are either defined incompletely or are more conceptual rather than quantitative. A Performance Test Manager/Lead needs to spend his time understanding the client’s expectations by asking the right set of questions and transforming the conceptual requirements into quantitative goals. He needs to play the role of a mediator who bridges the gap between the novice user language and performance testing terminologies.

It is the responsibility of the Performance Test Manager/Lead to get the essence from the customer. One needs to quickly understand the customer and should start talking in their language. Don’t talk using performance testing jargon and make the customer feel that they are ignorant, which is not the case. Don’t expect that the customer will give the performance goals in a single shot. Rather, start the discussion with an overview and purpose of performance testing. The discussion can be started with questions like

• What is the expected user load they are looking for?
• How much is the expected response time for a page?
• What types of performance tests can be included? Explain each type of performance test and its purpose.

These simple questions make the client comfortable to explain what he wants.

Some additional Tips:

Also, try to build a good rapport with the customer. During the meeting/call, educate the customer by explaining some core performance testing terms with some good examples, so that he can explain exactly what his expectations are. The most important thing that a performance manager/lead should always avoid, is not to scare the client by asking lots of questions in one shot or talking too much technical. The way of asking the requirement should be very polite to impress the customer and get his confidence.

Once the NFRs are collected, draft them in Non-Functional Requirement Document along with the date and points discussed in the meetings. The final Non-Functional Requirement Document must be signed off by all the project stakeholders.

Challenges:

Non-Functional Requirement gathering is the toughest phase of performance testing. Many times a Performance Test Manager/Lead face challenges to collect the correct performance testing requirement and conclude them as a proper client’s expectation. In most cases, applications are new and hence a Performance Test Manager/Lead does not have any clue about the NFRs (non-functional requirements). Many times, the client picks some random numbers over the call and asks to perform NFT (non-functional testing) against those numbers and at the end of the test when such NFRs do not meet, the client is forced to modify the NFRs and pass the test at any cost.

Such performance tests are executed to showcase that performance testing had been done and the application performed very well in the test environment. But in reality, 82% of such applications are failed due to performance issues within 6 months or even in the less time period when user load increases in production. Hence key points are to analyse the NFR thoroughly, not consider (agree) any random number as NFR and finalise the quantitative NFRs without any caveat.

Deliverable:

Non-functional requirement document has to draft once the scope is finalised. The outcome of each meeting may lead to multiple changes in the document. Non-functional requirement documents should cover all aspects along with major and minor changes in the requirement. Do not forget to mention the acceptable error percentage in the document because this is one of the criteria which will help to decide the RAG status at the end of the testing.

Non-Functional Requirement Document Template:

Download the template of the Non-Functional Requirement Document.

Example:

Let’s call PerfMate. In the previous phase (Risk Assessment) of the project, PerfMate has conducted the risk assessment and finalised the scope of performance testing. He got the approval from all the stakeholders on the Risk Assessment document. Moving to the next phase, he starts to collect the non-functional requirement (specific to performance testing) and receives the following requirements:

1. The application should be very fast.
2. The response time of the application should be quick.
3. The web server performance should be as high as possible.
4. The application should support many users.
5. The servers should not fail when a sudden load comes during sale and offer periods.
6. The application should run without any failures for a long duration.

From a client’s point of view, he has given all the requirements and set his expectation, but from PerfMate perspective, he just got the conceptual requirements. Now, PerfMate explained to the client that the provided information is partial and cannot help him to define the performance testing goal. In a polite manner, he gained the customer’s confidence and convinced him to provide the quantitative requirements. He asked project stakeholders to answer some of his basic questions. At last, he managed to gather the following requirements in the next couple of days:

Question: How many types of users are using this application (GUI)?

Answer. 4 types of users

1. Admin
2. Seller
3. End-user
4. Call center employee

Question: What are the business scenarios of every user?

Answer: Business scenarios for each type of user are like this:

Admin:

1. To approve/reject a new seller
2. To verify a newly added product and approve/reject it

Seller:

1. To add a new product
2. To delete the existing product

Buyer:

1. To buy a product
2. To cancel the order

Call Center Employee:

1. To register the complaint

Question: What are the AUT current and predicted peak user load for all its users’ actions over time?

Answer: Admin

• Current: 4
• Predicted: 10

Seller

• Current: 25
• Predicted: 100

Buyer

• Current: 438
• Predicted: 2500

Call Center

• Current: 8
• Predicted: 24

Question: What is the average active user count (including all types of users) at a time?

Answer:

• Total: 304
• Admin: 3
• Seller: 15
• Buyer: 278
• Call Center: 8

Question: What could be the active user count during peak hours?

Answer:

• Total: 500
• Admin: 4
• Seller: 50
• Buyer: 438
• Call Center: 8

Question: Anytime during a day or month when the average user count suddenly increased?

Answer: 31st of every month there is a 1-minute sale from 09:00 to 09:01 and 10:00 to 10:01. During this period the active buyer count becomes three times i.e. 834 and the active seller count becomes 23. None of the other (Admin/Customer Care) users’ counts changed.

• Total: 868
• Admin: 3
• Seller: 23
• Buyer: 834
• Call Center: 8

Question: What is the request count received at the Admin end?

Answer: 200 requests per hour

Question: What would be the response time NFRs for all the scenarios?

Answer: None of the pages should breach 3 seconds average response time NFR. This NFR is applicable to all types of users, scenarios, and pages.

<Rest of the figures are displayed in the below NFR table>

Question: What would be the server-side NFRs?

Answer:

• CPU utilization must not be more than 60%; except stress test.
• Pre, post and steady-state memory utilization differences must not be more than 15%; except for the stress test.
• Pre, post and steady-state disk utilization must not be more than 25%; except stress test.

Question: What is the size of the performance testing environment with respect to production?

Answer: 100% (Live like environment)

More or less PerfMate got the answers to all of his queries and noted them down in the NFR document and share them with the client for approval. After getting approval on the NFR Document, he will start the preparation of the performance test strategy.

A typical NFR format prepared by PerfMate is:

Note: Do not consider the below table as a standard NFR format. It was made simple for understanding purposes; especially for beginners.

NFR ID	Category	Description	Impact to
NFR01	Application	The solution must be able to support 500 active users. 1. Admin: 4 (2 for seller and 2 for product approval) 2. Seller: 50 3. Buyer: 438 4. Call Center: 8	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR02	Application	The solution must be able to support the future volume of active users i.e. 2634 1. Admin: 10 2. Seller: 100 3. Buyer: 2500 4. Call Center: 24	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR03	Application	The solution must be performed well during a long period of time with average volume. i.e. 304 1. Admin: 3 2. Seller: 15 3. Buyer: 278 4. Call Center: 8	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR04	Application	The solution must be able to support the spike load of the buyer and seller during the sale period. 1. Admin: 3 2. Seller: 23 3. Buyer: 834 4. Call Center: 8	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR05	Application	Admin gets an average of 200 requests per hour every time.	1. Admin
NFR06	Application	The number of orders: 1. Peak Hour Volume: 1340 2. Sale Hour Volume: 2830 3. Future Volume: 7500 4. Average Volume: 600 Note: 4% of the users cancel the order in every scenario.	1. Buyer
NFR07	Application	Sellers add an average of 180 products per hour and delete 18 existing products every hour	1. Seller
NFR08	Application	The call center employees get 40 complaints per hour	1. Call Center
NFR09	Application	The response time of any page must not exceed 3 seconds except stress test	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR10	Application	The error rate of transactions must not exceed 1%	1. Admin 2. Seller 3. Buyer 4. Call Center
NFR11	Server	The CPU Utilization must not exceed 60%	1. Web 2. App 3. DB
NFR12	Server	The Memory Utilization must not exceed 15% (Compare pre, post and steady-state memory status)	1. Web 2. App 3. DB
NFR13	Server	The disk Utilization must not exceed 15% (Compare pre, post and steady-state memory status)	1. Web 2. App 3. DB
NFR14	Server	There must not any memory leakage	1. Web 2. App 3. DB
NFR15	Application	Buyers order at the average rate of  1. Peak Hour Rate: 3.06 products per hour2. Sale Hour Rate: 3.39 products per hour3. Future Volume: 3 products per hour4. Average Volume Rate: 2.15 products per hour	1. Buyer

---