Streaming Consistency Checks for User Defined Type Fragments

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for consistency checking of user-defined type (UDT) fragments in databases require wholesale copying of data, leading to high memory and processing resource expenses, especially when fragments are split across multiple data pages or packets.

Innovation Solution

A system and method for performing streaming consistency checks that evaluate UDT fragments without materialization, using a data block processing component to identify fragment types and discard data blocks as processed, allowing for error detection without storing the entire fragment.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If conventional consistency checking methods are used to verify UDT fragments, then data consistency can be ensured, but memory and processing resources are excessively consumed due to wholesale copying of data

Engineering Contradiction:

Improvedata consistencyVSAvoidmemory and processing resources

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent segments the UDT fragment verification process into streaming operations that process data in chunks rather than loading entire fragments into memory. The verification is divided into manageable portions that can be processed sequentially as data streams through the system, reducing peak memory requirements while maintaining comprehensive consistency checking.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent extracts only the essential verification logic from the complete data fragment, separating the consistency checking function from the data itself. This allows the system to verify data integrity without retaining copies of the actual data in memory, removing only the necessary metadata and verification state rather than the entire fragment.

Inventive Principle:

Principle #2Taking out (Extraction)

2Adaptability or versatility

If UDT fragments are split across multiple data pages for storage flexibility, then storage efficiency is improved, but consistency checking becomes more complex and resource-intensive

Engineering Contradiction:

Improvestorage flexibilityVSAvoidconsistency checking complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary streaming verification mechanism that bridges multiple data pages containing fragmented UDT data. This intermediary layer tracks fragment boundaries and maintains verification state across page boundaries, simplifying the complexity of checking split fragments by providing a unified verification interface that handles the distributed nature of the data transparently.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If entire UDT fragments are materialized in memory for verification, then comprehensive consistency checking can be performed, but processing time and memory usage increase significantly

Engineering Contradiction:

Improveconsistency checking completenessVSAvoidprocessing time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent implements continuous streaming verification that operates throughout the data processing pipeline rather than as a separate batch operation. The consistency checking occurs continuously as data flows through the system, eliminating idle time between data loading and verification, and maintaining uninterrupted verification coverage without requiring complete materialization of fragments.

Inventive Principle:

Principle #20Continuity of useful action

Solution Approach 2:

The patent performs preliminary verification setup and initialization before data processing begins, preparing verification state machines and validation rules in advance. This preliminary action ensures that when data streams through the system, verification can commence immediately without delays associated with initializing checking mechanisms during data processing.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS7571153B2Systems and methods for performing streaming checks on data format for UDTs

Publication Date: 2009.08.04 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The subject invention relates to systems and methods that perform consistency checks of user defined type (UDT) fragments utilizing a stream of data without wholesale copying of the data. Streaming consistency checks facilitate discovering invalid fragments as soon as possible such that the entire UDT fragment does not have to be read, stored, analyzed, etc. Additionally, a UDT fragment can be evaluated without being completely materialized; thus, systems and/or methods employing the subject invention can yield improved performance as compared to conventional techniques by reducing required resources and enabling faster detection of invalid structures.