Aggregation Framework for Distributed Non-Relational Database Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-relational database systems face challenges in efficiently executing aggregation operations due to unpredictable results from traditional aggregation expressions, especially with data organized as document objects containing arrays and nested structures, leading to increased complexity and performance issues.
Innovation Solution
An aggregation framework is introduced that optimizes aggregation expressions by distributing pipeline steps to individual shard servers, using an unwind operation to flatten data structures, and reordering operations to reduce data processing and computational burden, enabling efficient execution of complex queries across multiple collections.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional aggregation expressions are used in non-relational database systems with document objects containing arrays and nested structures, then flexibility and scalability are improved, but execution consistency and predictability deteriorate
Solution Approach 1:
The patent segments the aggregation operation into distinct pipeline stages: data retrieval, unwinding of arrays and nested structures, flattening to row-oriented format, and aggregation computation. This segmentation isolates the unpredictable data structure handling from the aggregation logic, ensuring consistent execution by processing heterogeneous document structures through standardized transformation steps before aggregation.
2Ease of manufacture
If data is organized as document objects with arrays and nested structures in non-relational databases, then ease of data modeling is improved, but aggregation operation complexity increases
Solution Approach 1:
The patent introduces an intermediary transformation layer between the document-oriented data storage and aggregation operations. The unwind and flatten operations act as mediators that convert nested document structures into row-oriented intermediate representations, allowing standard aggregation algorithms to operate on complex data without increasing aggregation operation complexity. This intermediary layer preserves modeling flexibility while simplifying execution.
3Productivity
If aggregation operations are executed on distributed shard servers in non-relational databases, then scalability is improved, but computational overhead and data processing burden increase
Solution Approach 1:
The patent applies preliminary filtering and data reduction operations at each shard server before aggregation computation. By evaluating filter conditions and selecting only relevant documents at the shard level, the system reduces the volume of data that needs to be transmitted and processed during aggregation. This preliminary action minimizes computational overhead across the distributed system while maintaining scalability.
Data Source
AI summary
A system and computer implemented method for execution of aggregation expressions on a distributed non-relational database system is provided. According to an aspect, an operation is provided to perform the aggregation operation. An indication of at least one foreign collection of documents is accepted. An indication of at least one variable for at least one local field of at least one local collection of documents is accepted. An indication of a pipeline for execution on the at least one local collection of documents and the at least one foreign collection of documents is accepted. An operation comparing the local field of the at least one local collection of documents and the foreign field of the at least one foreign collection of documents is performed. Responsive to the operation, data from the at least one local collection of documents and the at least one foreign collection of documents is aggregated.