Tiered Data Stream Management with Transparent Storage Interception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing streaming systems lack a transparent and efficient mechanism for managing tiered storage of data streams, leading to imbalanced data ingestion and inadequate data management once data is tiered, without considering throughput and storage device capabilities.
Innovation Solution
A framework is introduced as a transparent data management layer that intercepts storage requests and manages tiered data streams, providing storage functions across a heterogeneous environment, enabling content-aware data location and specialized data reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a transparent data management layer is introduced to manage tiered data streams, then data management efficiency and balance are improved, but system complexity increases
Solution Approach 1:
The patent introduces a transparent data management layer that acts as an intermediary between streaming applications and storage systems. This layer intercepts storage requests, manages tiered data streams, and coordinates data movement between different storage tiers, thereby improving data management efficiency while maintaining system transparency and avoiding direct complexity increases in the application layer.
2Stability of the object's composition
If storage requests are intercepted and managed transparently, then data tiering balance is improved, but processing overhead increases
Solution Approach 1:
The transparent data management layer performs preliminary actions by pre-processing storage requests before they reach the storage system. It intercepts requests, determines appropriate storage tiers based on data characteristics and access patterns, and prepares data movement operations in advance, thereby achieving balanced data tiering while minimizing processing overhead through efficient request routing.
3Adaptability or versatility
If a framework is introduced to provide storage functions across heterogeneous environments, then adaptability is improved, but device complexity increases
Solution Approach 1:
The patent implements a universal framework that provides standardized storage functions across heterogeneous storage environments. This framework maintains a unified interface layer that can adapt to different storage systems and configurations, enabling the system to work with various storage devices and protocols without requiring separate specialized components for each environment, thereby achieving adaptability while controlling framework complexity.
Data Source
AI summary
A method for managing a stream includes: intercepting a request originated from a streaming storage system that specifies a data tiering operation; analyzing the request based on a policy to obtain an analysis result; identifying, based on the analysis result and function metadata, a first function and a second function that need to process data chunks of the stream before performing the operation; making, based on the identifying, a determination that the first function and second function are available internally; processing, using the first function and based on the policy, the data chunks to obtain the processed data chunks; identifying, using the second function, that the processed data chunks need to be tiered to a first long-term storage; tiering the processed data chunks to the storage; and notifying, once the processed data chunks are tiered to the storage, the streaming storage system that the data tiering operation is completed.