Retention Policy Handling for Replicated Data Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in managing data retention policies and storage operations, particularly in flash storage systems, leading to unnecessary write operations and reduced reliability.
Innovation Solution
Implementing a direct-mapped flash storage system where higher-level processes manage data operations without lower-level storage controllers, utilizing non-volatile RAM for buffering and direct addressing of data blocks, and incorporating storage array controllers for failover and device management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If storage array controllers are used for device management, then device management capability is improved, but system complexity increases
Solution Approach 1:
The system is segmented into multiple storage array controllers (SACs), each managing specific storage devices independently. This allows distributed device management where each SAC handles a subset of devices, improving overall management capability while maintaining modular complexity that is easier to control than a monolithic controller.
Solution Approach 2:
Storage array controllers act as intermediary components between the host system and storage devices. The SACs manage the complexity of device operations, data movement, and retention policies, shielding higher-level processes from low-level storage management complexity while providing simplified interfaces for data operations.
2Productivity
If direct-mapped flash storage is implemented, then write operation efficiency is improved, but reliability may be reduced due to lack of lower-level controller management
Solution Approach 1:
The flash storage system implements self-service mechanisms where the storage system itself manages retention policies and data protection without requiring complex external controller intervention. The system automatically handles data movement, retention enforcement, and protection operations, improving write efficiency while maintaining reliability through built-in management capabilities.
Solution Approach 2:
The system performs preliminary actions by pre-configuring retention policies and data protection mechanisms at the time of data ingestion. Retention metadata is created upfront, and protection operations are prepared in advance, allowing efficient write operations while ensuring reliability through pre-established management frameworks.
3Reliability
If retention policies are enforced at lower-level controllers, then data retention compliance is improved, but write operation overhead increases
Solution Approach 1:
The storage array controller serves as an intermediary that receives retention policy requirements from higher-level processes and implements them efficiently. The SAC translates retention compliance requirements into optimized storage operations, ensuring data retention compliance while minimizing write overhead through intelligent policy enforcement at the array level rather than device level.
Solution Approach 2:
The system changes the parameter of policy enforcement location from lower-level device controllers to mid-level storage array controllers. This parameter change allows retention policies to be enforced at a more efficient level in the storage hierarchy, maintaining compliance while reducing the overhead impact on write operations through better resource management and batched operations.
Data Source
AI summary
An illustrative method includes determining that a first data instance stored within a first data store is replicated as a second data instance stored within a second data store, determining that a second retention policy associated with the second data instance stored within the second data store is at least as stringent as a first retention policy associated with the first data instance stored within the first data store, and modifying, based on the determining that the second retention policy is as least as stringent as the first retention policy, the first retention policy to allow for the first data instance to be modified or deleted without the first set of one or more conditions each being satisfied.