Hadoop Node In-Memory Data Processing Engine
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face inefficiencies when handling data from multiple sources, such as Hadoop clusters and relational databases, due to the need for large data copies, slow processing times, and the risk of data staleness, which limits performance and efficiency.
Innovation Solution
A method where a Hadoop node processes data stored in volatile memory by executing a data processing engine that receives data from external sources, performs operations using both HDFS and external data, and utilizes dataflow graphs to manage data processing operations, allowing for parallel processing without relying on disk storage and custom query parsing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is copied from external sources to HDFS for processing, then data processing can be performed using Hadoop's distributed file system, but large data copies are required which increases storage requirements and processing time
Solution Approach 1:
The patent extracts only the necessary portion of external data into volatile memory at the Hadoop node rather than copying entire datasets to HDFS. The data processing engine receives and processes only the specific data portions needed for the current operation, eliminating unnecessary data copying while maintaining processing efficiency.
Solution Approach 2:
The system performs preliminary actions by pre-fetching and storing required external data in volatile memory before the main processing operation begins. This preliminary data preparation in memory eliminates the need for time-consuming data copies during the actual processing phase.
2Productivity
If data is stored in HDFS for processing, then data can be processed using distributed file system operations, but disk access slows down processing speed
Solution Approach 1:
The patent transitions from traditional disk-based HDFS storage to in-memory volatile storage for external data. By changing the storage dimension from persistent disk storage to volatile memory, the system achieves significantly faster data access speeds while maintaining the distributed processing architecture's throughput capabilities.
3Productivity
If data from multiple sources is processed using traditional methods, then comprehensive data analysis can be performed, but data staleness occurs due to slow processing times
Solution Approach 1:
The system enables continuous data processing by eliminating the batch copy-to-HDFS step. External data sources can continuously stream data into volatile memory for immediate processing, ensuring that the data being analyzed is always fresh and up-to-date while maintaining high throughput through parallel processing operations.
4Adaptability or versatility
If custom query parsing is implemented for external data sources, then data processing can be performed, but system complexity increases
Solution Approach 1:
The data processing engine implements a universal interface that can handle multiple external data source formats and protocols through a single unified mechanism. Rather than implementing separate custom parsers for each data source type, the engine provides multi-functional capability to process diverse data sources using common volatile memory-based operations, reducing system complexity while maintaining adaptability.
Data Source
AI summary
In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.