Dynamic File Chunking for Distributed Compute Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In big data analytics environments like Hadoop clusters, achieving data locality becomes increasingly difficult as the number of nodes and users increases, leading to inefficient use of compute resources due to fixed chunk sizes, which limits the ability to speed up processing jobs despite having more available nodes.
Innovation Solution
Implementing a method and apparatus for dynamic chunking of files based on available compute resources and Service Level Agreements (SLA), allowing for optimal resource allocation and pre-staging data in memory for faster access, while also recording job histories for optimization and job de-duplication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If fixed chunk sizes are used in HDFS, then data locality can be maintained with simple replication, but the ability to utilize available compute resources is limited and job processing speed cannot be increased
Solution Approach 1:
The patent applies dynamics by transitioning from fixed chunk sizes to dynamic chunking. The system dynamically determines the number of chunks based on available compute resources and job requirements. When more compute nodes are available, the file is divided into more chunks, allowing parallel processing across more nodes and increasing job processing speed while utilizing available resources flexibly.
Solution Approach 2:
The patent changes the parameter of chunk size from fixed to variable. Instead of using a predetermined fixed chunk size, the system adjusts the number of chunks as a variable parameter based on real-time availability of compute resources. This allows the system to optimize productivity by matching the number of processing chunks to the number of available compute nodes.
2Productivity
If more compute nodes are added to the cluster, then processing capacity increases, but achieving data locality becomes increasingly difficult
Solution Approach 1:
The system dynamically adjusts the number of chunks based on the number of available compute nodes. When more nodes are added to the cluster, the file is divided into more chunks, allowing data to be distributed across the expanded node set. This maintains data locality by ensuring that processing occurs on nodes where data resides, while accommodating increased cluster size.
Solution Approach 2:
The patent applies segmentation by dividing the file into a variable number of chunks that can be distributed across multiple compute nodes. Instead of treating the file as a single unit or using fixed chunking, the system segments the file into an optimal number of parts based on available resources, enabling efficient data distribution and processing across a larger cluster while maintaining locality.
3Reliability
If the number of replicas is increased to achieve high data locality, then data availability improves, but storage requirements increase
Solution Approach 1:
The system dynamically determines the optimal number of chunks based on available compute resources rather than using a fixed replication factor. This allows the system to achieve high data availability through dynamic chunking that matches the current cluster state, avoiding the need to permanently increase storage capacity while maintaining reliability.
Solution Approach 2:
The patent changes the approach from fixed replication to variable chunking. Instead of increasing the replication factor to improve data availability, the system uses dynamic chunking where the number of chunks adapts to available compute resources, achieving similar reliability goals without proportionally increasing storage requirements.
Data Source
AI summary
A method for achieving optimal use of compute, storage and network resources in a distributed computing system is provided. In one embodiment, a job request associated with a file is submitted. A Job Scheduler in conjunction with a workflow manager determines the best fit for dynamically chunking size of the file. The distributed storage subsystem provides a dynamic view to the compute resources where the job will be run. This knowledge is also used to pre-staging the data by the distributed storage subsystem. Furthermore, keeping a history of the job and their data allows for further improvements in resources and job execution times by being able to skip processing on data that has been done previously. Given the ability to chunk the file dynamically, it is possible to re-balance resources for running the job dynamically to meet its service level agreement.