Resource monitoring framework for big raw data processing

Abstract

Scientific experiments, simulations, and modern applications generate large amounts of data. Analysing resources required to process such big datasets is essential to identify application running costs for cloud or in-house deployments. Researchers have proposed keeping data in raw formats to avoid upfront utilisation of resources. However, it poses reparsing issues for frequently accessed data. The paper discusses detailed comparative analysis of resources required by in-situ engines and traditional database management systems to process a real-world scientific dataset. A resource monitoring framework has been developed and incorporated into the raw data query processing framework to achieve this goal. The work identified different query types best suited to a given data processing tool in terms of data to result time and resource requirements. The analysis of resource utilisation patterns has led to the development of query complexity aware (QCA) and resource utilisation aware (RUA) data partitioning techniques to process big raw data efficiently. Resource utilisation data have been analysed to estimate the data processing capacity of a given machine.