Testing Cloud - It’s Time to Clear the Fog

A reputed company launches the beta version of its web chat and net meeting application which is hosted on the company’s own cloud infrastructure, let us assume it was called ‘CloChat’ . On deployment ‘CloChat’ fails miserably in meeting customer expectations resulting in the company’s cloud portfolio suffering irrefutable damages and also delaying the scheduled release of ‘CloChat’. 

The scenario explained above can be one of the worst nightmare of any cloud vendor. What exactly would have gone wrong in these cases? Considering that ‘CloChat’ was a product of a reputed company it must have gone through its regular testing cycles.

On investigation it was found that the testing team did not or rather could not emulate the actual scenarios that ‘CloChat’ faced once deployed in the “real” network. This “real” network is actually an assimilation of thousands or millions of users, who are unique in their actions and usage patterns.

All this actually points to the need of testing cloud or cloud based applications during pre-deployment by simulating realistic user scenarios. Generic applications testing can never ensure or predict application behavior post-deployment.

Fig. Simulating the faceless users is a key challenge in cloud testing

Testing Cloud or its applications in pre-deployment can be classified in a few broad categories:

Scalability Testing: How scalable is my cloud application is a million dollar question. Having an answer to this during pre-deployment can save a lot of heart burns later. It is a realistic indicator of whether the hardware and software infrastructures are under provisioned for the targeted number of customers. A few Cloud as a Service (CaaS) and Infrastructure as a Server(IaaS) vendors regularly skip apparently costlier scalability testing and choose to over provision their hardware resources instead, but this can never be a long term or a profitable business model. This apart any non-scalable network application risks the same fate as ‘CloChat’.

Security Testing: The security threats in cloud can be broadly classified into the ones arising due to known/unknown vulnerabilities and Distributed Denial of Service (DDoS) attacks. Understanding and closing some of these gaps at pre-deployment phase can actually prevent major data losses and system downtimes later. Moreover a robust and well secured Cloud provides the assurance to the skeptical customers who are not yet used to the idea of their data being stored in shared storage locations.

Data Integrity: If the Cloud Service providers are to be looked as banks then the costliest assets that they store is “data”. This data is susceptible to attacks, malformations, dirty writes and several other kinds of software and hardware distortions. It is very important to ensure that the underlying infrastructure of network and disks is “always” transferring and storing the same data as written by the user. Modern age user’s will have very low tolerance to any kind of data distortion and loss.

Quality of Experience (QoE): This is an umbrella term and all the above mentioned testing processes can come together to ensure the customers QoE. A test scenario where one thousand users are trying to login to their profiles at the same time can be a part of scalability testing but the average logging-in time of the thousand users actually form’s the users “Logging-In” QoE, similarly QoE should be calculated for separate user actions like chat, voice, video, mix of all etc. Apart from scalability, data integrity and security testing can also be related to QoE. A test scenario where customer gets integrity errors while downloading a file of enormous size can negatively impact Customer’s QoE. Likewise service continuity and seamless user operations during crisis situations like DDoS attack or multiple Server failures can contribute to a higher QoE.

Cloud is evolving and we will see more players operating in this space in coming years. As with any other services, developing user loyalty will be crucial to business continuity and growth. Keeping this in mind, the ones who manage to build a robust testing and development model that helps them adhere to their published SLA’s will enjoy maximum traction. And for the others, they might not see their user base increasing beyond the “free” ones, because with so many players in market, the customers who mean business will not hold on to a particular service just to be their beta testers.

This guest post was written by Amritam Putatunda - Technical Marketing Engineer at Ixia.