Data Analytics QA Strategy & Approach

Abstract

Big Data is a term encompassing the use of techniques to capture, process, analyses and visualize potentially large datasets in a reasonable timeframe not accessible to standard IT technologies. By extension, the platform, tools and software used for this purpose are collectively called “Big Data technologies. The current technology enables us to efficiently store and query large datasets, the focus is now on techniques that make use of the complete data set, instead of sampling. This has tremendous implications in areas like machine learning, pattern recognition and classification, to name a few. With the coming of Big data technologies like Hadoop, NoSQL, Messaging Queues etc. organization have got the tools to dive deep into the large amounts of data and come up with analytics and intelligence that can help them drive business growth and take right decisions in time. But, testing Big data is one of the biggest challenges that the organizations face because of the lack of knowledge on what to test and how to test. They have been facing challenges in defining appropriate test strategies, tools, working with NoSQL, setting up optimal test environments and so on. Different testing types like functional and non-functional testing are required to ensure that the data from varied sources is processed error free and is of good quality to perform analysis. Functional testing activities like validation of map reduce process, structured and unstructured data validation, data storage validation are important to ensure that the data is correct and is of good quality. Apart from functional validations other non-functional testing like performance and failover testing plays a key role to ensure the whole process is scalable and is happening within specified SLA.