Task Allocation for Distributed Data Computing Reducing IO Redundancy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data initialization modes for resilient distributed datasets in Spark lead to IO redundancy and increased time consumption due to repeated data forwarding during distributed computing, particularly when data is transmitted over the network.
Innovation Solution
A task allocation method and apparatus that maps data slices to a resilient distributed dataset based on storage parameters, allowing computing tasks to be assigned to storage nodes corresponding to the data storage information, thereby reducing the need for data to be transmitted and only accessing data in local memories.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data are pulled via HBase clients, saved into RDDs after transformation, and distributed into multiple slices for distributed computing, then distributed computing can be performed with RDD operators, but data will be transmitted over the network repeatedly causing IO redundancy and increased time consumption
Solution Approach 1:
The patent applies preliminary action by pre-loading data from the distributed database into the memory of storage nodes before computing tasks are assigned. This allows the data to be readily available when computing tasks arrive, eliminating the need for repeated network transmissions during the computing process and thus reducing time consumption while maintaining distributed computing capability
2Productivity
If data are pulled via HBase clients, saved into RDDs after transformation, and distributed into multiple slices, then distributed computing can be performed, but IO redundancy is caused
Solution Approach 1:
The patent eliminates IO redundancy by pre-loading data into storage node memories before task assignment. This preliminary action ensures that when distributed computing tasks are executed, the data is already available locally without requiring repeated network I/O operations, thus reducing energy loss while preserving distributed computing functionality
Solution Approach 2:
The patent introduces storage node memory as an intermediary between the distributed database and the computing tasks. This intermediary layer caches data locally, allowing computing tasks to access data without repeated network transmissions to the distributed database, thereby reducing IO redundancy while maintaining the distributed computing architecture
Data Source
AI summary
The present application discloses a task allocation method and task allocation apparatus for distributed data computing. The task allocation method includes: receiving storage parameters for target data to be computed in distributed data; mapping data slices of the target data to a resilient distributed dataset based on the storage parameters, each data slice corresponding respectively to a partition in the resilient distributed dataset; assigning each partition to a storage node to generate a computing task and perform the computing tasks. By using data storage information in a distributed database to allocate computing tasks to storage nodes corresponding to the data, Only data in local memories need to be called during the computing process, IO redundancy and time consumed due to repeated data forwarding are reduced.