Datapath Graph Dimensional Sub-Operations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Datapath graphs face challenges in processing big data efficiently and accurately, often requiring significant time for updates, and existing solutions that improve processing speed compromise accuracy.
Innovation Solution
The use of dimensions to divide operations into sub-operations, with each sub-operation associated with a subset of values, allowing for efficient processing and reducing the need to re-run operations on unchanged data subsets, thereby minimizing processing and data transfer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional datapath graphs process big data updates, then processing accuracy is maintained, but processing time becomes significantly long
Solution Approach 1:
The patent segments the big data processing task by dividing data into subsets based on dimension values. Instead of processing all data uniformly, the system partitions data according to specific dimension ranges (e.g., time periods, geographic regions) and processes only relevant subsets, thereby reducing processing time while maintaining accuracy for the required data scope.
Solution Approach 2:
The patent applies local quality by allowing different processing strategies for different data subsets based on their dimension characteristics. Each data subset is processed according to its specific dimension range and relevance, enabling optimized processing for each local segment rather than applying a single global processing approach to all data.
2Productivity
If other systems reduce processing time for big data updates, then speed improves, but processing accuracy deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-defining dimension schemas and data subset partitions before actual processing occurs. The system establishes dimension-based segmentation rules in advance, allowing it to quickly identify and process only the relevant data subsets needed for each query, thus improving efficiency without compromising accuracy through selective processing.
Solution Approach 2:
The patent changes processing parameters dynamically based on dimension values and query requirements. By adjusting which data subsets are processed based on dimension ranges and query-specific parameters, the system optimizes processing efficiency for each specific task while ensuring that the processed data maintains full accuracy for the requested scope.
3Reliability
If operations are performed on all data subsets, then complete accuracy is achieved, but data transfer and processing volume increases unnecessarily
Solution Approach 1:
The patent extracts only the necessary data subsets required for each specific operation based on dimension filtering. Instead of transferring and processing all available data, the system identifies and extracts only those data portions that match the query's dimension criteria, thereby reducing data volume and transfer requirements while maintaining complete accuracy for the extracted relevant data.
Data Source
AI summary
An operation associated with an artifact which includes a plurality of fragments is received. A dimension is obtained and a plurality of sub-operations is generated based at least in part on the operation and the dimension where each of the plurality of sub-operations is associated with a subset of values associated with the dimension. The plurality of sub-operations is performed on the plurality of fragments, including by generating, for each of the plurality of fragments, dimensional metadata which includes a subset of values associated with the dimension for that fragment.