Distributed Data Processing System for High Volume Extraction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems are unable to efficiently and effectively process large volumes of data as they do not scale well with increasing data loads, making them brittle and incapable of handling high-volume data extraction efficiently.
Innovation Solution
A distributed data processing system that uses a map reduce algorithm and application framework to implement high volume data extraction and distribution over multiple channels, including receiving subscription requests, determining worker nodes, breaking data into chunks, processing, and distributing processed data to subscribers through various channels like REST/HTTP, MQ, or KAFKA.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If current systems process data using traditional methods, then processing is simple and straightforward, but the systems cannot scale with increasing data volumes and become brittle
Solution Approach 1:
The patent divides the data processing system into multiple independent worker nodes that can process data chunks in parallel. Each node operates independently, allowing the system to scale by adding more nodes without increasing overall system complexity significantly. The data is segmented into chunks that can be distributed across nodes.
Solution Approach 2:
The patent introduces a distributed architecture dimension to the processing system, moving from a single-node vertical scaling approach to a multi-node horizontal scaling approach. This adds a spatial dimension (number of nodes) to the processing capability, enabling scalability without proportionally increasing complexity.
2Productivity
If data is processed in large volumes using current methods, then complete data processing is achieved, but processing efficiency and effectiveness deteriorate
Solution Approach 1:
The patent implements continuous data processing by maintaining active worker nodes that continuously receive, process, and return data chunks. The system avoids idle periods by keeping processing pipelines full, with nodes continuously pulling new data chunks from the data source and pushing processed results back.
Solution Approach 2:
The patent incorporates feedback mechanisms where worker nodes report processing status, completion, and potential errors back to the coordinating node. This feedback loop ensures processing effectiveness by allowing the system to monitor and verify that data is being processed correctly at scale.
3Quantity of substance
If traditional data extraction methods are used, then implementation is straightforward, but the approach becomes brittle and cannot handle high-volume data
Solution Approach 1:
The patent implements dynamic worker node allocation where the number and capacity of worker nodes can be adjusted based on data volume requirements. Nodes can be added or removed dynamically, and data chunk sizes can be adjusted based on node performance and data characteristics, providing flexibility to handle varying data volumes.
Solution Approach 2:
The patent creates a universal data processing framework that can handle different types of data and processing requirements through the same worker node architecture. The system is designed to be multi-functional, accommodating various data extraction, transformation, and loading operations through configurable worker nodes rather than requiring separate systems for each function.
Data Source
AI summary
A method may include: receiving, at a computer program in a distributed data processing system, a subscription request from a subscriber to receive processed data from the distributed data processing system comprising a plurality of nodes; receiving, by a receiving node of the plurality of nodes, information about data to be processed from one or more data source; determining, by the receiving node, a number of worker nodes needed to process the data based on the information about the data; breaking, by the receiving node, the data into plurality of data chunks based on the number of worker nodes; distributing, by the receiving node, the data chunks to the worker nodes; processing, by the worker nodes, the data chunks; receiving, by a gathering node of the plurality of nodes, the processed data; and distributing, by the gathering node, the processed data to the subscriber.