Aggregation Framework for Distributed Non-Relational Database Systems

Solution Overview

Problem

Non-relational database systems face challenges in executing aggregation expressions, such as sum, average, and minimum, due to the unpredictable nature of data organization, leading to inconsistent and complex query results, especially when dealing with nested arrays and documents.

Innovation Solution

An aggregation framework is introduced that optimizes the execution of aggregation operations by distributing pipeline steps across shard servers, using an unwind operation to flatten data structures, and reordering operations to reduce data processing and computational burden, allowing for efficient execution of complex queries across multiple collections.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If traditional aggregation expressions are executed on non-relational database systems with nested arrays and documents, then the system maintains flexibility and scalability, but the query results become unpredictable and complex

Engineering Contradiction:

Improveflexibility and scalability of non-relational database systemVSAvoidpredictability and consistency of aggregation query results

Core Design Contradiction:

Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The aggregation operation is divided into multiple pipeline stages, with each stage performing a specific transformation or aggregation function. This segmentation allows the system to process complex nested documents in a controlled, step-by-step manner, ensuring predictable results while maintaining the flexibility of non-relational data structures

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary processing of nested arrays and documents before executing aggregation expressions. By pre-flattening or pre-processing the hierarchical data structures into a standardized format, the system ensures that subsequent aggregation operations produce consistent and predictable results

Inventive Principle:

Principle #10Preliminary action

2Quantity of substance

If aggregation operations are executed on distributed non-relational database systems, then the system handles large volumes of data, but the computational burden and data processing complexity increase

Engineering Contradiction:

Improvevolume of data processedVSAvoidcomputational burden and processing complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The aggregation operation is divided into multiple pipeline stages, with each stage performing a specific transformation or aggregation function. This segmentation allows the system to process complex nested documents in a controlled, step-by-step manner, ensuring predictable results while maintaining the flexibility of non-relational data structures

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system performs preliminary processing of nested arrays and documents before executing aggregation expressions. By pre-flattening or pre-processing the hierarchical data structures into a standardized format, the system ensures that subsequent aggregation operations produce consistent and predictable results

Inventive Principle:

Principle #10Preliminary action

3Adaptability or versatility

If data is organized in hierarchical structures with nested arrays and documents, then the system maintains data flexibility, but aggregation expression execution becomes inconsistent

Engineering Contradiction:

Improvedata flexibility and hierarchical organizationVSAvoidconsistency of aggregation expression execution

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The aggregation operation is divided into multiple pipeline stages, with each stage performing a specific transformation or aggregation function. This segmentation allows the system to process complex nested documents in a controlled, step-by-step manner, ensuring predictable results while maintaining the flexibility of non-relational data structures

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system transforms the parameter representation of hierarchical data by converting nested arrays and documents into a flattened, standardized format suitable for aggregation operations. This parameter transformation maintains the semantic meaning of the hierarchical structure while enabling consistent aggregation expression execution

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10366100B2Aggregation framework system architecture and method

Publication Date: 2019.07.30 MONGODB INC

AI summary

A system and computer implemented method for execution of aggregation expressions on a distributed non-relational database system is provided. According to one aspect, an aggregation operation may be provided that permits more complex operations using separate collections. For instance, it may be desirable to create a report from one collection using information grouped according to information stored in another collection. Such a capability may be provided within other conventional database systems, however, in a non-relational database system such as NoSQL, the system is not capable of performing server-side joins, such a capability may not be performed without denormalizing the attributes into each object that references it, or by performing application-level joins which is not efficient and leads to unnecessarily complex code within the application that interfaces with the NoSQL database system.