Data Processing Framework Change Evaluation via Job Subset Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data processing frameworks, evaluating the impact of changes on job processing becomes unfeasible as the number of jobs and runs increases, leading to potential negative impacts on performance due to unquantified effects.
Innovation Solution
A method and system for evaluating the rollout of a change to a data processing framework by selectively processing jobs with and without the change enabled, determining performance cost values, and iteratively adjusting the set of jobs to determine if the change is beneficial, using metrics like CPU-seconds, RAM byte-seconds, and elapsed time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of jobs and runs to be evaluated increases, then the comprehensiveness of the evaluation improves, but the feasibility and time required for evaluation deteriorates
Solution Approach 1:
The patent divides the large population of jobs into multiple subsets or cohorts. Instead of evaluating all jobs simultaneously, the system processes jobs in segmented batches, allowing the evaluation to be completed in manageable timeframes while still achieving comprehensive coverage across the entire job population over multiple evaluation cycles.
Solution Approach 2:
The system performs evaluations periodically or iteratively across different subsets of jobs. By cycling through different job subsets in repeated evaluation rounds, the system achieves comprehensive evaluation coverage without requiring all jobs to be evaluated at once, thus reducing the time loss for any single evaluation cycle.
2Measurement precision
If all jobs are evaluated with the change enabled, then the accuracy of impact quantification improves, but the processing resources and time required increases significantly
Solution Approach 1:
The patent applies different processing conditions to different subsets of jobs. Some jobs are processed with the change enabled while others are processed without the change, allowing for localized comparison and accurate impact quantification without requiring all jobs to be processed under the same conditions simultaneously.
Solution Approach 2:
Instead of processing all jobs with the change enabled, the system processes only a subset of jobs with the change while processing other jobs without the change. This partial action approach provides sufficient data for accurate impact quantification without the excessive resource consumption of evaluating every single job with the change enabled.
3Ease of operation
If changes are implemented without quantifying their impact, then the ease of implementation improves, but the risk of negative performance impact increases
Solution Approach 1:
The system performs preliminary evaluation and impact quantification before fully implementing changes across all jobs. By conducting automated evaluations on job subsets first, the system identifies potential negative performance impacts before widespread deployment, allowing for corrective actions to be taken before the change is fully operationalized.
Solution Approach 2:
The patent implements a feedback mechanism where evaluation results from job subsets are analyzed and used to determine whether to proceed with full change implementation. The automated evaluation system provides feedback on performance impact, and this feedback loops back to the change implementation process, ensuring that only changes with acceptable performance impacts are fully deployed.
Data Source
AI summary
Aspects of the disclosure relate to evaluating the rollout of a change to a data processing framework (DPF). A first set of jobs is selected out of a plurality of jobs. The first set of jobs is processed by the DPF with the proposed change enabled and the other jobs of the plurality of jobs by the DPF without the change enabled. A performance cost value is determined for each of the jobs in the first set of jobs processed by the DPF with the proposed change enabled and for each of the other jobs of the plurality of jobs processed by the DPF with the proposed change disabled. Based on the performance cost values of the jobs in the first set of jobs and the performance cost values of the other jobs of the plurality of jobs, a determination is made whether the proposed change is beneficial to the DPF.