Metadata-Based Storage De-duplication via Interceptor Application
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In cloud-based data storage solutions, redundant data is often stored unnecessarily, leading to inefficient use of storage space, networking resources, and increased computing costs, as well as a lack of data inventory management, which results in inefficient data governance and compliance with regulations.
Innovation Solution
A system comprising a management server and a storage server that uses a storage index to detect and prevent duplicate data from being stored by comparing metadata, prompting users to cancel redundant data storage requests and maintaining a data inventory for efficient resource allocation and compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If data is migrated to a centralized storage server, then storage overhead and maintenance costs are decreased, but redundant data is unnecessarily stored consuming storage space
Solution Approach 1:
The system performs preliminary de-duplication by comparing metadata of incoming data against existing storage index metadata before data is actually stored. The listener application detects data addition requests, the scanner application compares metadata proactively, and the interceptor application prevents duplicate storage before it occurs, resolving the contradiction by eliminating redundant storage space consumption while maintaining centralized storage benefits
Solution Approach 2:
The patent introduces a management server as an intermediary between the storage server and users. This intermediary contains the listener, scanner, and interceptor applications that work together to detect, analyze, and prevent duplicate data storage. The management server mediates data storage operations by comparing metadata against the storage index and blocking duplicate data, thus reducing storage space consumption while preserving centralized storage advantages
2Ease of manufacture
If data is migrated to a centralized storage server, then maintenance costs are decreased, but networking resources are consumed by transmitting duplicate data
Solution Approach 1:
The system performs preliminary de-duplication by comparing metadata of incoming data against existing storage index metadata before data is actually stored. The listener application detects data addition requests, the scanner application compares metadata proactively, and the interceptor application prevents duplicate storage before it occurs, resolving the contradiction by eliminating redundant storage space consumption while maintaining centralized storage benefits
Solution Approach 2:
The patent introduces a management server as an intermediary between the storage server and users. This intermediary contains the listener, scanner, and interceptor applications that work together to detect, analyze, and prevent duplicate data storage. The management server mediates data storage operations by comparing metadata against the storage index and blocking duplicate data, thus reducing storage space consumption while preserving centralized storage advantages
3Adaptability or versatility
If data is migrated to a centralized storage server, then access from any location is enabled, but computing resources are wasted on processing duplicate data
Solution Approach 1:
The system performs preliminary de-duplication by comparing metadata of incoming data against existing storage index metadata before data is actually stored. The listener application detects data addition requests, the scanner application compares metadata proactively, and the interceptor application prevents duplicate storage before it occurs, resolving the contradiction by eliminating redundant storage space consumption while maintaining centralized storage benefits
Solution Approach 2:
The patent introduces a management server as an intermediary between the storage server and users. This intermediary contains the listener, scanner, and interceptor applications that work together to detect, analyze, and prevent duplicate data storage. The management server mediates data storage operations by comparing metadata against the storage index and blocking duplicate data, thus reducing storage space consumption while preserving centralized storage advantages
4Device complexity
If data is stored without inventory management, then storage operations are simple, but data governance and compliance are inefficient
Solution Approach 1:
The system performs preliminary de-duplication by comparing metadata of incoming data against existing storage index metadata before data is actually stored. The listener application detects data addition requests, the scanner application compares metadata proactively, and the interceptor application prevents duplicate storage before it occurs, resolving the contradiction by eliminating redundant storage space consumption while maintaining centralized storage benefits
Solution Approach 2:
The system establishes a feedback mechanism where the storage index stores metadata about stored data, and this metadata is used by the listener, scanner, and interceptor applications to detect and prevent duplicate data. The feedback loop continuously monitors incoming data requests, compares them against the storage index, and provides feedback to block duplicates, thereby improving data governance and compliance while maintaining manageable system complexity
Data Source
AI summary
A management server is configured to detect that a request to add a new column of data to a storage server has been received, determine that the new column of data is a duplicate of a column of data already stored at the storage server by comparing metadata describing the new column and the metadata describing the tables stored at the storage server, and, in response to the new column of data being the duplicate of the column of data already stored in the storage server, generate, by an interceptor application of the management server, an indication that the new column of data is already stored at the storage server.