Parallel Analytics Data Subdivision and Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The healthcare industry faces challenges in processing large volumes of healthcare claims due to missing or incorrect clinical or demographic information, leading to inefficient data processing and increased costs, which can be addressed by improving the subdivision and processing of datasets.
Innovation Solution
A system and method for efficient data processing that involves determining a control number of evaluation units within a processing system, allocating and releasing these units concurrently to process datasets, and using an XML framework to manage dependencies and intermediate results, allowing for dynamic subdivision and resource optimization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If completely subdividing input data into several map tasks before assignment to various processes, then data processing can be performed in parallel, but the initial subdivision is time consuming and resource intensive
Solution Approach 1:
The patent segments data processing into two phases: (1) dynamic task creation during processing, and (2) parallel execution of those tasks. This eliminates the need for complete pre-subdivision while maintaining parallel processing benefits.
Solution Approach 2:
The system performs preliminary actions by pre-defining the processing framework and data structures, but defers the actual task subdivision to occur dynamically during processing rather than beforehand, reducing the time and resources spent on initial subdivision.
2Reliability
If using historical solutions for data processing, then processing can be completed, but multiple rounds of processing are required and take longer than a month
Solution Approach 1:
The patent enables continuous processing by allowing evaluation units to be allocated, created, evaluated, and released concurrently. This eliminates the sequential rounds of processing required by historical solutions, completing the same accurate processing within a single continuous execution window.
Solution Approach 2:
The system dynamically adjusts the number and allocation of evaluation units during processing based on available resources and data volume. This dynamic adaptation allows the system to complete multiple processing rounds concurrently rather than sequentially, reducing total processing time while maintaining accuracy.
3Productivity
If allocating and releasing evaluation units concurrently, then processing efficiency improves, but system complexity increases
Solution Approach 1:
The evaluation unit framework serves multiple functions: data processing, task allocation, resource management, and result aggregation. By making the evaluation unit multi-functional, the patent reduces overall system complexity despite enabling concurrent operations.
Solution Approach 2:
The evaluation units autonomously allocate, create, evaluate, and release data subdivisions without requiring centralized coordination. This self-service capability simplifies management complexity while maintaining high processing efficiency through concurrent operations.
Data Source
AI summary
Methods, apparatus, and software packages for data processing are disclosed. In some embodiments, the method may include receiving a dataset. In some embodiments, the method may include determining a control number of a processing system. In some embodiments, the control number may include a number of evaluation units within the processing system. The method may include processing the dataset using a plurality of evaluation units. In some embodiments, processing the dataset may include allocating a free evaluation unit to form a busy evaluation unit. Processing the dataset may also include creating a data subdivision for the busy evaluation unit, the data subdivision including a part of the dataset. Processing the dataset may also include evaluating the data subdivision. Processing the dataset may also include releasing the busy evaluation unit. The allocating, creating, evaluating, and releasing may performed concurrently by the plurality of evaluation units.