Distributed Data Processing Task Allocation via Completion Time Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The sheer volume of generated data makes it infeasible to analyze or process it in a timely manner, as loading all data into memory or processing it all at once is impractical and time-consuming.
Innovation Solution
A distributed data storage and processing system that processes data in meaningful chunks, using a chunking attribute to determine importance and a data item completion time attribute to distribute tasks evenly across processing units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If all source data is loaded into memory at once for processing, then complete data analysis can be performed, but the system becomes infeasible due to enormous data volume and excessive processing time
Solution Approach 1:
The patent divides the enormous source data into manageable chunks based on a chunking attribute (such as time-based partitions or data category divisions). This segmentation allows the system to process data in smaller, memory-friendly portions rather than attempting to load all data simultaneously, thereby making the processing feasible and reducing memory requirements while maintaining overall processing capability.
Solution Approach 2:
The system performs preliminary sorting and prioritization of data chunks before processing based on a chunking attribute that reflects presumed importance to consumer clients. By pre-organizing data into priority-based chunks, the system can process high-value data first, delivering meaningful results sooner without waiting to process all data, thus reducing effective processing time for critical information.
2Loss of time
If data is processed in chunks based on presumed importance, then relevant data can be delivered to consumers sooner, but the processing order becomes complex requiring sophisticated task allocation
Solution Approach 1:
Each processing unit autonomously determines its own task allocation by evaluating the completion time attribute of available data items. Instead of requiring a centralized complex scheduling system, processing units independently select their next task based on current system state, simplifying the overall control architecture while achieving balanced load distribution and prioritized processing.
Solution Approach 2:
The task allocation system dynamically adjusts to changing conditions by continuously monitoring completion time attributes and reassigning tasks based on current processing unit availability and workload. This dynamic approach allows the system to adapt to varying data priorities and processing speeds without requiring static, pre-configured complex scheduling rules.
3Productivity
If data items are assigned to processing units using completion time as a proxy, then task distribution becomes more even across units, but requires additional computational overhead to estimate completion times
Solution Approach 1:
The system uses the completion time attribute as a simplified proxy metric to represent the expected processing duration of each data item. By transforming the complex multidimensional characteristics of data items into a single completion time parameter, the system achieves effective task distribution without requiring complex evaluation models, balancing processor utilization while minimizing computational overhead for task allocation decisions.
Data Source
AI summary
An organization's distributed data storage and processing system produces an enormous volume of source data (such as log files or other statistics). The organization uses a data item processing system to process the source data in prioritized chunks, and to further assign the data items within the chunks to different processing units based on estimates of processing completion time. In this way, it becomes feasible to process the source data for analysis by consumer clients within a reasonable amount of time, and the aggregate use of the processing units is made more efficient.