Parallel Dynamic Aggregation System for Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently aggregating large volumes of data while maintaining a robust and stable data structure, often requiring a trade-off between processing efficiency and data structure stability, which can lead to interruptions in downstream applications.
Innovation Solution
Implementing a multi-stage parallel aggregation processing system that executes aggregation jobs and query sets in parallel, using a hybrid dynamic database schema that supports multiple dimension-dimension value pairs, allowing for simultaneous dividing and grouping of data within a single table, thereby increasing processing efficiency and maintaining data structure stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing business intelligence tools are used to aggregate data efficiently, then processing speed is improved, but data structure stability deteriorates due to rigid database design requiring changes when new aggregation needs arise
Solution Approach 1:
The patent segments the aggregation process into multiple independent stages (raw data aggregation, intermediate aggregation, final aggregation) that can be executed in parallel. Each stage produces results stored in separate tables, allowing the system to handle different aggregation needs without changing the core database schema, thus maintaining stability while improving processing efficiency through parallel execution.
Solution Approach 2:
The patent introduces a temporal dimension to data aggregation by creating multiple aggregation tables (e.g., daily, weekly, monthly aggregation tables) alongside the raw data table. This allows the system to maintain a stable raw data schema while providing efficient access to aggregated data at different time granularities, resolving the contradiction between processing speed and structure stability.
2Loss of time
If data is aggregated to meet time requirements of downstream applications, then processing efficiency is improved, but data structure complexity increases requiring multiple tables with different schemas
Solution Approach 1:
The patent segments aggregation operations into distinct parallel stages, each handling specific aggregation requirements. By dividing the monolithic aggregation process into smaller, independent units that can run simultaneously, the system reduces total processing time while keeping each segment's schema simple and manageable.
Solution Approach 2:
The patent creates a universal aggregation framework where a single multi-stage parallel processing system can handle various aggregation requirements (different time periods, different granularity levels, different data types) without requiring separate dedicated structures for each case, thus reducing overall complexity while maintaining versatility.
3Adaptability or versatility
If multiple tables with different schemas are used to store aggregation results, then processing flexibility is improved, but downstream application stability deteriorates due to increased impact of schema changes
Solution Approach 1:
The patent adds a temporal dimension to the data storage structure by creating time-based aggregation tables (daily, weekly, monthly) that complement the raw data table. This allows the system to provide flexible aggregation queries at different time granularities while maintaining a stable core schema, as the temporal tables follow a consistent naming and structure pattern that downstream applications can reliably interface with.
Data Source
AI summary
Aggregation of data and subsequent diving and/or grouping (i.e., slicing and dicing) of the aggregated data occurs by multi-stage parallel processing. Initially, multiple aggregation jobs are executed in parallel to aggregate data from a granular level to a higher level. Subsequently, query sets are executed in parallel for each of the aggregation jobs to divide and/or group the higher level data according to the queries and the segregated results are stored in a single table, which is partitioned for each aggregation job. Parallel processing of the query sets and storage in a single table is made possible by implementing a hybrid dynamic database schema, in which each table in the database includes one or more additional fields, each additional field comprising a set of dimension-dimension value pairs.