Database Parallel Processing Architecture for Job Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing big data systems face resource constraints, leading to high latency and reduced throughput due to the need to prioritize and sequentially process numerous jobs, which complicates operations and consumes significant processing power and memory.
Innovation Solution
The system identifies common operations among jobs, stores their results, and leverages these results for subsequent jobs, enabling parallel processing and automatic resource freeing, thus reducing the number of operations needed and improving resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If jobs are processed sequentially due to resource constraints, then resource consumption is reduced, but system throughput and processing speed deteriorate
Solution Approach 1:
The patent merges common operations across multiple jobs by identifying and executing them once, then reusing the results for subsequent jobs. This combines the processing effort into a single execution rather than repeating it for each job, thereby reducing overall resource consumption while maintaining high throughput.
Solution Approach 2:
The system performs preliminary identification of common operations before processing jobs. By analyzing job requirements in advance and pre-executing shared operations, the system prepares results that can be reused across multiple jobs, reducing both resource consumption and processing time.
2Productivity
If jobs are processed in parallel, then system throughput is improved, but resource consumption and complexity increase
Solution Approach 1:
The patent segments jobs into independent units that can be processed in parallel, while further segmenting the operations within each job to identify commonalities. This segmentation allows the system to manage complexity by breaking down the processing into manageable, reusable components.
Solution Approach 2:
The system creates copies of operation results that can be reused across multiple jobs. Instead of re-executing common operations for each job, the system copies the results and distributes them to consuming jobs, thereby enabling parallel processing without proportionally increasing resource consumption or complexity.
3Productivity
If common operations are identified and results are stored for reuse, then the number of operations is reduced, but memory usage increases
Solution Approach 1:
The system implements a mechanism to discard operation results from memory once they have been consumed by all dependent jobs. This recovering of memory resources ensures that while results are stored during processing to enable reuse, the memory usage is automatically reduced after the results are no longer needed, balancing productivity gains with resource management.
4Loss of energy
If job prioritization is implemented due to resource limitations, then resource allocation is optimized, but latency is introduced
Solution Approach 1:
The patent enables continuous processing of multiple jobs in parallel by eliminating the need for sequential prioritization. Jobs are processed concurrently as resources become available, maintaining continuous useful action without the interruptions and delays inherent in prioritization-based sequential processing, thereby reducing latency while optimizing resource allocation.
Data Source
AI summary
A device that includes a parsing engine and an execution engine. The parsing engine is configured to identify micro operations corresponding with operations that are common among at least two jobs from a set of jobs and to store the identified micro operations in memory. The execution engine is configured to receive a job from the set of jobs, to identity micro operations and dependencies within the job, and to execute the identified micro operations for the job in accordance with the identified dependencies.