Light weight end to end pipeline for ingesting documents into UDR

Abstract

The work describes here is to create an end to end data pipeline with open source tools and technologies available, we have used Nifi which is ETL (Extract, Transform and load) tool to ingest the data and transform it using the inbuilt processors which Nifi provide. The main aim is to create the data flow without writing code, cause it requires lots of management and man power and testing and integration time instead using the inbuilt processors given in the Nifi tool and do the configuration in that, and create the data pipeline so it can be quickly built and requires the least human intervention.