Parallel Data Ingestion System for Hadoop Data Lakes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data ingestion and processing methods for large-scale data management, particularly in Hadoop-based Enterprise Data Lakes, are inefficient due to sequential processing and require manual intervention, leading to bottlenecks and high CPU and IO loads, especially when handling large datasets.
Innovation Solution
A method and system for parallel ingestion and processing of data into a data lake, involving splitting input data into balanced records, reading and transforming them in parallel, sorting key/value pairs, and writing them to an output file in the same order, utilizing a distributed system with modules for input, mapping, partitioning, and output processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If sequential single-threaded reading and transformation is used, then code simplicity is maintained, but processing speed and throughput are limited
Solution Approach 1:
The patent divides the input data into multiple input splits that can be processed independently and in parallel. Each split is processed by separate mapper tasks, enabling concurrent processing across multiple CPU cores while maintaining manageable complexity through modular task design
Solution Approach 2:
The patent transitions from single-threaded sequential processing to multi-threaded parallel processing by adding the dimension of concurrency. Multiple mapper tasks execute simultaneously on different input splits, leveraging multiple CPU cores to achieve speedup without proportionally increasing overall system complexity
2Productivity
If multiple transformation jobs are run simultaneously, then overall throughput increases, but CPU and IO bottlenecks intensify on individual processing nodes
Solution Approach 1:
The patent segments the data processing workload into multiple independent mapper tasks that operate on different input splits. This distribution of work across multiple processing nodes prevents any single node from becoming overwhelmed, balancing CPU load while maintaining high overall throughput
Solution Approach 2:
The patent distributes processing across multiple dimensions by utilizing multiple processing nodes and CPU cores simultaneously. Each node handles a portion of the total workload, spreading the CPU and IO burden across the distributed system rather than concentrating it on single nodes
3Loss of time
If files are split into blocks without sequential parsing, then processing time is reduced, but IO complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-splitting input files into manageable blocks before processing. These pre-defined splits can be processed independently and in parallel, eliminating the need for sequential parsing while reducing overall processing time through concurrent block processing
4Extent of automation
If manual ETL scripting is used for data ingestion, then data transformation flexibility is achieved, but processing automation and speed are reduced
Solution Approach 1:
The patent implements self-service automation where the system automatically performs data ingestion, splitting, transformation, and loading operations. The framework autonomously manages the ETL process without requiring manual scripting, achieving high automation levels while maintaining transformation capabilities through configured mappers and reducers
Data Source
AI summary
Embodiments of the present invention relate to systems and methods for ingesting input data containing a plurality of records into a data lake. In an embodiment, the method comprises splitting the input data into a plurality of input splits consisting of a balanced number of records; reading the records from the plurality of input splits in parallel, regardless of the format and encoding of the input source; converting the input data within the records into at least one key/value pair; transforming the values input data into a serializable format; sorting the key/value pairs of the transformed values such that the records are sorted in the same order as they were read; writing the transformed values to an output file; and storing the output file to the data lake.