Auto-scaling Framework for Big Data Package Rescaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Analytic systems face resource overconsumption and system failures due to large batch jobs that exceed processing thresholds, crowding out other jobs and leading to incomplete or erroneous processing.
Innovation Solution
An auto-scaling framework that rescales packages by splitting them into smaller subsets based on a package threshold size, allowing for efficient processing by ensuring that each package is within the threshold, thereby optimizing resource utilization and preventing system overload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If large batch jobs are processed to handle big data, then data processing capability is improved, but processing resources are overconsumed leading to system failures
Solution Approach 1:
The patent segments large batch jobs into smaller sub-jobs by dividing packages into smaller units. Each sub-job processes a smaller subset of data, ensuring that resource consumption remains within acceptable thresholds while collectively handling the complete data set. This segmentation prevents any single job from over-consuming resources and causing system failures.
Solution Approach 2:
The patent implements dynamic job scheduling where the system automatically adjusts the number and size of jobs based on real-time resource availability and package size. Jobs are created, scaled, or terminated dynamically to match current system capacity, allowing the system to handle variable data volumes without overwhelming fixed resource allocations.
2Productivity
If batch jobs are processed to complete data processing tasks, then processing throughput is improved, but other users' jobs are crowded out
Solution Approach 1:
By segmenting large batch jobs into smaller sub-jobs, the patent enables better multiplexing of processing resources. Multiple smaller jobs can run concurrently on shared resources without any single job monopolizing the system, thereby improving throughput while maintaining fair resource access for multiple users.
Solution Approach 2:
The system incorporates feedback mechanisms that monitor resource consumption and job completion status. Based on this feedback, the scheduler dynamically adjusts job allocation to ensure that processing throughput is optimized without unfairly prioritizing one user's jobs over others, thus maintaining multi-user resource sharing fairness.
3Speed
If packages are processed without size threshold checking, then processing speed is maintained, but timeouts and errors occur
Solution Approach 1:
The patent performs preliminary size threshold checking on packages before job creation. By pre-validating that packages meet size requirements and dividing them into appropriate sub-jobs beforehand, the system avoids timeouts and errors during execution while maintaining efficient processing speeds. This preliminary action ensures that only suitable packages are submitted for processing.
Data Source
AI summary
In some embodiments, there is provided a method including receiving a first package for processing as part of a job; unpacking the first package to include additional data linked to the first package, wherein the first package including the additional data forms a first unpacked package; in response to the first unpacked package being less than a package threshold size, processing the first unpacked package to form a first output; and in response to the first unpacked data being more than the package threshold size, rescaling the first package to satisfy the package threshold size before additional unpacking and processing is performed on the first package. Related manufactured articles, devices, systems, and methods are also disclosed.