Insights on Cloud-Based Performance Testing

Introduction

Traditional load testing practices were being followed over a decade after the segregation of QA & Dev tasks which proved immense benefits. Although this practice is being followed long ago, there is always a question about the high cost spent on this activity. This new approach which will significantly reduce overall costs without making any compromise on key benefits (Quality, Brand Value etc) to customers is termed cloud-based performance testing.

Insight on Traditional (On-prem) Load Testing

In traditional methodology, Performance testing is usually conducted on-premises with a whole test infra setup built on a dedicated LAN environment. This testing activity is also performed during the end of the SDLC cycle which many times miss to cover real production issues due to scope sneak.

Challenges Faced in Traditional (On-prem) Load Testing

  1. Huge upfront cost: Traditional (On-prem) load testing involves procuring and setting up all test-related infrastructures during the start of the engagement. This led to spending more cost on infrastructure & maintenance tasks. Despite proper prior budget plans made, there is always a chance for an increase in cost or servers getting under-utilized or both.
  2. Laborious / Time-Consuming tasks: More time & effort will be involved in test build set-up activities for traditional load testing. Configuration management becomes complex if not managed properly and would also lead to error judgement & schedule slippages.
  3. Missing to Mimic Production Environment: Due to the non-availability of the specific configuration of hardware or other components, test infra setup will not be accurate sync-up with production in the case of traditional load testing. This would also lead to a risk of missing production defects.
  4. Lack of adoption of agile methodologies: On-prem testing has less flexibility in accommodating requirement (Application or infrastructure) changes based on customer needs. This makes load testing difficult for applications following agile life cycles as it would hinder supporting continuous testing activities which would make performance test results obsolete or outdated at the end

Key Core Benefits of Cloud-Based Load Testing

  1. Quick Scale of Infrastructure: Cloud-based infra has the efficiency to generate peak loads for enterprise performance testing in a quick span of time. Cloud data centres also have a good level of computing resources which aid in the quick provisioning of virtual machines for testing. For example, Cloud vendors have an auto-scaling feature which makes it easier to scale up or scale down on infrastructure based on real-time load demands. This will immensely help in overcoming performance issues faced due to test infrastructure availability.
  2. Global Availability: Cloud vendors have features of global infrastructure which means auto-provisioning of test machines across different geographies. This will help us to test for different network bandwidth variations and accurately mimic real users’ performance. For example: If test applications are being used by diverse geographies viz. US, UK, Australia etc., then test machines can be deployed on all specific regions which would aid in reflecting accurate test simulations.
  3. Pay-as-you-go model: This is a prime advantage of using the cloud. Usage-wise billing will help to bear the cost only for the utilized services. Scaling resources up and down when needed would help us to reduce the overall cost involved in testing. Cloud vendors have numerous utilities like cost management & cost explorer which give monthly reports on expenses based on consumptions made by individual account IDs. For example, Specific services can be procured & used for load testing in the case of shorter-duration test runs. This avoids us making long-term upfront costs spending.
  4. Quick time to build: Cloud vendors provide various tools & templates to make application deployment into the cloud in a short span of minutes instead of spending days or hours. Auto deployment feature available in the cloud which makes it easier to integrate various cloud-hosted applications with CI – CD pipeline tools.
  5. Multi-Device Support Testing: Cloud vendors have readymade solutions for not only testing web-based applications but also for mobile or other emulator device-based applications. This makes it easier to test applications with multiple devices. For example, Load generators required during load tests for supporting cross browsers & various mobile devices can be procured seamlessly in the cloud which aids inaccurate simulation of real-world tests.
  6. Environment Stability and Reliability: Most cloud vendors support 99.99% stability & reliability for their infrastructure. This makes us not to worry much about maintenance tasks or server failovers and focus only on performing load-testing activities on a continuous basis. For example, a Solid stable & reliable environment helps to test longer duration (Soak/Endurance) runs without any infra hiccups.

Cloud Load Testing Tools

Popular Cloud Load Testing Tools used widely in the market are as follows:

  1. BlazeMeter
  2. Micro Focus StromRunner Load
  3. SOASTA Cloud Test
  4. SmartBear Load Complete
  5. LoadImpact
  6. LoadStorm

Cloud Load test Life Cycle

  • Step 1 – Finalize Performance Test Scenarios: There is no difference in designing test scenarios. The usual traditional way will be followed which covers critical business flows & transactions as we do in on-prem load testing
  • Step 2 – Research for Cloud Vendors: There are a lot of cloud vendors available in today’s market. AWS, Azure & GCP are popular vendors in cloud servicing. Based on test infrastructure needs & customer goals, specific cloud vendors or multiple cloud vendors can be chosen for performance testing.
  • Step 3 – Product migration to the cloud: This step is considered as deploying test infra to the cloud. Application readiness should be thoroughly checked in terms of code version compatibility and software upgrades and ensure they work well without any flaws in the cloud.
  • Step 4 – Execute Performance Test Runs: Cloud also supports all types of load tests which we perform on-prem. The same methodology can be followed for test execution in the cloud. The provisioning of test machines globally is the key advantage of cloud testing. Various SaaS-based load testing tools are also available in the market which can be explored & utilized based on requirement feasibility. Automation build tools like Jenkins, Bamboo etc can also be readily & easily integrated with cloud-based load testing tools which helps in CI & CD test runs seamlessly
  • Step 5 – Monitor, Analyse and Share Results: Cloud vendors provide various monitoring tools to monitor the health of cloud resources viz. Cloud Watch in AWS. A proper check needs to be made to ensure the correct integration of performance monitoring tools in a cloud environment which monitors various test infrastructure components in the cloud. Deploying APM tools On-prem to monitor cloud machines can be adopted to mitigate any risks faced using tools inside the cloud. Results gathering & sharing follow the same principle as we do in on-prem performance testing.

Different Approaches for Deploying Load test in Cloud

  1. Target Machines (LAN) Vs Load Generators (Cloud)
  2. Full Infra Setup in Cloud

The above approaches describe deploying a test agent on the cloud and the target system can be either inside or outside the cloud. Testers can control test machines on the cloud through a console to manage all test-related activities.

Cloud Load Testing Entry Criteria

Below are the entry criteria that need to adhere for the application to be cloud-friendly and help in a smooth and seamless transition into the cloud environment:

  1. Software Concurrency: In Cloud, it is imperative that at all times to maintain a code base compatible with either N or N-1 of any supporting technology stack (i.e. RHEL, TLS, Node JS, WebLogic, JBoss, Apache, Oracle, PostGre SQL etc). There will be no or feeble support for ‘Deprecated’ and ‘End-of-Life’ versions of any supporting software in the cloud.
  2. Isolate Database Reads: In the cloud, it is imperative that any operational reporting needs are catered to only through a ‘read-only’ ODS database instance. All initiatives planned for cloud hosting would require factoring this and ensure to keep the OLTP system as BAU-friendly as possible which always helps to ensure optimal throughput and performance
  3. Limit PL/SQL code & Alter Performance Criteria for DB: In the cloud, it is important that all teams make sure to keep any-to-all business logic in the application tier and move their stored procedures, functions and triggers out of the DB tier. This would drastically reduce storage costs and improve network latency. For code retained in the DB, it is expected to make it run under 20ms and as tested against full production-sized data & transaction volumes. Exceptions can be applied only for batch jobs which get scheduled during non-business hours.

Conclusion

Cloud-based Load testing can become complex if things are not aligned in a proper way. The use of the right approach & strategy would aid in managing activities in a controlled way.


You may be interested:


1 thought on “Insights on Cloud-Based Performance Testing”

  1. Hi Mahesh, I attended an interview today and I was asked a question like this, have you run any Load Test where the Load Generators (LG) Machines are on Local Area Network ( LAN ). Can you please tell me what exactly it means ??

    Reply

Leave a Comment