Non-overlapping Collection Records for Data Stream Recovery

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In data systems that perform transactions in parallel, the overlap of transaction completions and interleaving of task entries in logs complicate recovery processes, making it difficult to ensure that only committed transactions are processed correctly, leading to inefficiencies in data stream processing and recovery operations.

Innovation Solution

The insertion of non-overlapping collection records into a data stream, each defined by a unique collection definition and including a data stream address range, allows for compact and ordered processing, enabling parallel serialization and speeding up recovery by ensuring that each data item belongs to only one collection and is processed accordingly.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If transactions are processed in parallel with interleaved task entries in logs, then processing throughput is improved, but recovery complexity and difficulty of ensuring correct transaction processing increase

Engineering Contradiction:

Improveprocessing throughputVSAvoidrecovery complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent segments the log into non-overlapping collection records, each representing a distinct transaction or set of tasks. Each collection record contains a collection definition that identifies which tasks belong to which transaction, allowing parallel processing while maintaining clear boundaries for recovery operations.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces collection records as intermediary structures between raw log entries and recovery processing. These collection records serve as mediators that organize interleaved task entries into coherent transaction groups, simplifying the recovery process by providing a structured intermediate representation.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Speed

If collection records are inserted to enable parallel processing, then recovery speed is improved, but data stream processing complexity increases

Engineering Contradiction:

Improverecovery speedVSAvoiddata stream processing complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent performs preliminary organization of log entries into collection records during the logging phase, before recovery is needed. This preliminary action groups tasks by transaction in advance, so that during recovery, preorganized collection records can be processed in parallel without requiring complex real-time analysis.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent changes the structural parameters of log organization by introducing collection records with specific formats including collection definitions and task entry identifiers. This parameter change transforms the raw log structure into a more recovery-friendly format that enables parallel processing while maintaining manageable complexity.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10592494B2Collection record for overlapping data stream collections

Publication Date: 2020.03.17 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The insertion into a data stream of collection records associated with multiple collections of data items in the data stream, for purposes of aiding in subsequent processing of the data stream by collections. Each collection record comprises a collection definition that does not overlap with the collection definition in any of the sequence of collection records including the collection definition of any neighboring collection record in the sequence of collection records. Each collection record also includes a data stream address range extending from the first data item to the last data item in the collection.