Asynchronous Data Enrichment for Append-Only Stores

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Append-only data stores present challenges when information about events becomes available at different times, as they cannot be updated to include additional information that becomes available later, making it difficult to provide all relevant information in response to queries.

Innovation Solution

An asynchronous data enrichment process that combines event information from an append-only data store with additional information from separate data stores, which may be cached, to ensure all relevant information is provided in response to queries, even if the additional information becomes available after the initial event data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If information is written to an append-only data store, then data integrity and indexing efficiency are improved, but the ability to include subsequently available additional information is lost

Engineering Contradiction:

Improvedata integrityVSAvoidadditional information

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The system segments information storage into two separate components: the append-only data store for event information and the additional data store for supplementary information. This segmentation allows each component to serve its specific purpose - the append-only store maintains data integrity and indexing efficiency, while the additional store captures later-available information without compromising the original data structure.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The query processing system acts as an intermediary that retrieves information from both the append-only data store and the additional data store, then combines and returns the complete result set. This intermediary role enables the system to provide comprehensive information while preserving the append-only nature of the original data store.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If an append-only data store is used, then querying efficiency is improved, but completeness of information is worsened when data becomes available later

Engineering Contradiction:

Improvequerying efficiencyVSAvoidsubsequently available information

Core Design Contradiction:

ProductivityVSLoss of information

Solution Approach 1:

The system divides the data storage function into two segments: the append-only data store that maintains querying efficiency through its immutable structure, and the additional data store that captures subsequent information. This segmentation allows both efficiency and completeness to be achieved simultaneously.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The query processing system merges results from both the append-only data store and the additional data store into a single comprehensive response. This merging operation ensures that queries return complete information while the underlying append-only store maintains its performance advantages.

Inventive Principle:

Principle #5Merging (Combining)

3Device complexity

If records are made immutable in append-only store, then system simplicity and performance are improved, but adaptability to include new information is reduced

Engineering Contradiction:

Improvesystem simplicityVSAvoidinformation completeness

Core Design Contradiction:

Device complexityVSAdaptability or versatility

Solution Approach 1:

The system segments the information storage architecture into an immutable append-only data store for simplicity and performance, and a separate additional data store for capturing new information. This segmentation preserves the simplicity of the append-only store while adding the adaptability to include subsequently available information.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The query processing system performs multiple functions: it retrieves data from the append-only store, retrieves additional data from the additional store, combines the results, and returns comprehensive information. This multi-functionality enables the system to maintain simplicity while achieving information completeness.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11475017B2Asynchronous data enrichment for an append-only data store

Publication Date: 2022.10.18 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

A method for asynchronous data enrichment may include receiving a query that includes a request for information about an entity. The query may specify a particular time range. In response to receiving the query, event information that is associated with the entity and that corresponds to the specified time range may be combined with additional information that is relevant to the query and that corresponds to the specified time range. The event information may initially be written to a record in an append-only data store. The additional information may become available after the event information, such that the additional information is not included in the record in the append-only data store. Instead, the additional information may initially be written to one or more additional data stores. Both the event information and the additional information may be provided in response to the query.