Application Data Expiration Views in Streaming Storage

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Contemporary data storage systems face challenges in managing data lifetime for multiple applications accessing the same data stream, as different organizations have varying data expiration requirements, leading to conflicts and incorrect data retrieval due to inconsistent expiration policies.

Innovation Solution

Implementing a system where each application has a specified data expiration time, with a controller determining the maximum or minimum expiration time among registered applications to manage stream cuts and views, ensuring data is truncated based on the most stringent policy, thereby maintaining different views of the data stream for each application.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a single data expiration policy is applied to a data stream, then storage resources are managed efficiently, but multiple applications with different expiration requirements cannot be served simultaneously

Engineering Contradiction:

Improveability to serve multiple applications with different expiration requirementsVSAvoidcomplexity of managing multiple expiration policies

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent segments the data stream into multiple virtual streams, each associated with a specific application and its expiration policy. The stream manager divides the physical data stream into logical segments that can be independently managed according to different expiration requirements, allowing multiple applications to access the same physical data with different expiration policies.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a virtualization dimension by creating virtual streams that map to a single physical stream. This additional layer of abstraction allows the system to manage multiple expiration policies simultaneously without increasing physical storage complexity, as each virtual stream can have its own expiration policy while sharing the underlying physical storage.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Reliability

If data is truncated based on the longest expiration period, then all applications can access data, but applications expecting recent data may retrieve outdated information

Engineering Contradiction:

Improveaccuracy of data retrieval for each applicationVSAvoiddata access efficiency

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The patent applies local quality by providing each application with a customized view of the data stream tailored to its specific expiration policy. Applications receive data with the appropriate expiration characteristics for their needs, rather than a uniform data view. This ensures that each application retrieves data with the correct temporal properties for its specific requirements.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The stream manager dynamically adjusts the effective expiration policy for each application based on its specific requirements. The system can modify which data events are visible to each application in real-time, allowing applications to access data with appropriate expiration characteristics without manual intervention or fixed policies.

Inventive Principle:

Principle #15Dynamics

3Ease of operation

If manual resolution of expiration conflicts is implemented, then data lifetime requirements can be addressed, but system operation becomes cumbersome and error-prone

Engineering Contradiction:

Improveautomation of expiration policy managementVSAvoidrisk of incorrect data retention

Core Design Contradiction:

Ease of operationVSLoss of information

Solution Approach 1:

The stream manager automatically resolves expiration conflicts by evaluating the expiration policies of all applications accessing a data stream and determining the appropriate effective expiration policy without human intervention. The system self-manages the complexity of coordinating multiple expiration requirements, eliminating manual configuration and reducing human error.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system continuously monitors the expiration policies of applications accessing each data stream and dynamically adjusts the effective expiration policy based on feedback from application requirements. This automated feedback mechanism ensures that data retention policies remain consistent with application needs without manual intervention.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS11323497B2Expiration of data streams for application programs in a streaming data storage platform

Publication Date: 2022.05.03 EMC IP HLDG CO LLC

AI summary

The described technology is generally directed towards a streaming data storage system that provides different views of a data stream to different applications based on a per-application data expiration policy. Applications that have different needs related to data lifetime management can be registered to read the same data stream, with each application individually assigned a data expiration time period. When one registered application reads the data stream, that application is presented with a view of the data stream that is subject to that application's expiration time period. When another registered application with a different expiration time period reads the same data stream, the other application is presented with a different view, according to that other application's expiration time period. The time periods for a registered application can be set between a minimum expiration period and a maximum expiration period, or set to a default value between the minimum and maximum.