Data Management Platform for Efficient Backup Subset Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data management systems face challenges in efficiently accessing and managing large volumes of data across diverse platforms and locations, particularly in providing reliable backup, fast recovery, and data portability, due to complex data infrastructure and the need for precise data subsets for applications.
Innovation Solution
A data management platform that employs secondary data and push transmission techniques to identify and transmit relevant data subsets from a primary data source to targets like applications, NAS devices, or object stores, using file metadata and snapshot differences for filtered and scheduled data management operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all primary data is transmitted to targets, then data completeness is improved, but network bandwidth consumption and transmission time increase
Solution Approach 1:
The system extracts only the necessary subset of data from the primary data source based on target-specific criteria. The data management platform identifies and transmits only the relevant data portions that each target needs, rather than transmitting all primary data. This extraction approach maintains data completeness for each target's specific needs while significantly reducing overall network bandwidth consumption.
Solution Approach 2:
The system segments the data transmission process by dividing the primary data into different subsets tailored to different target requirements. Each target receives a customized data subset based on its specific needs, allowing the system to optimize transmission efficiency while ensuring each target gets the complete data set it requires for its function.
2Reliability
If data is transmitted on-demand from primary source, then data freshness is improved, but system complexity and access time increase
Solution Approach 1:
The system performs preliminary actions by pre-identifying and pre-processing data subsets for different targets. The data management platform proactively determines what data each target will need and prepares these subsets in advance, reducing the complexity of on-demand data retrieval and enabling faster access while maintaining data freshness.
Solution Approach 2:
The data management platform acts as an intermediary between the primary data source and multiple targets. It manages the complexity of data distribution by centralizing the logic for identifying and transmitting appropriate data subsets, simplifying the overall system architecture while ensuring data freshness through controlled access to the primary source.
3Productivity
If filtered data subsets are transmitted, then transmission efficiency is improved, but data identification complexity increases
Solution Approach 1:
The system uses feedback mechanisms where targets communicate their data requirements to the data management platform. This feedback loop enables the platform to efficiently identify and transmit the correct data subsets without complex manual configuration, as the targets themselves provide information about what data they need, simplifying the identification process while maintaining high transmission efficiency.
Data Source
AI summary
Some examples relate generally to a data management platform comprising: a storage device configured to store secondary data and one or more processors in communication with the storage device and configured to perform certain operations. The operations may include identifying an aspect of the secondary data stored in the storage device, the secondary data including a backup of respective primary data stored in a primary data source; identifying or receiving an indication of a target to receive data associated with the identified aspect of the secondary data; and transmitting the data associated with the aspect of the secondary data to the target.