Data Archival Using Lossy Base Files and Patch Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data archival methods face challenges in efficiently storing and retrieving large volumes of data, particularly video content, due to the inability of traditional de-duplication algorithms to detect duplication between transcoded versions, leading to increased storage and compute costs, and hindering quick and efficient storage, retrieval, and transmission of data.
Innovation Solution
A data archiving system that compresses original data files using lossy compression techniques to generate base files and patch files for each level of lossy compression, allowing for efficient storage and retrieval of data at various levels of information loss, enabling users to reconstruct the original file with specified amounts of information loss without the need for real-time compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional de-duplication algorithms are used to store transcoded video versions, then storage capacity is increased, but storage efficiency deteriorates because duplication cannot be detected between versions
Solution Approach 1:
The system performs preliminary transcoding of the master video into multiple formats and resolutions before storage, creating a library of pre-processed versions. This eliminates the need for real-time transcoding when users request content, improving both storage efficiency through deduplication and delivery speed
Solution Approach 2:
The system varies parameters such as resolution, bitrate, and codec type when creating transcoded versions. By systematically changing these parameters across different video versions, the system enables effective deduplication while catering to diverse device requirements and network conditions
2Quantity of substance
If transcoding is performed on-the-fly when a user requests data, then storage space is reduced, but processing time and compute cost increase
Solution Approach 1:
The system performs all necessary transcoding operations in advance during a preprocessing phase, creating multiple versions of the video in different formats and resolutions. When users request content, the system can immediately serve the appropriate pre-transcoded version without performing real-time transcoding, thus eliminating processing delay
Solution Approach 2:
The video content is segmented into multiple transcoded versions with different parameters (resolution, bitrate, codec). This segmentation allows the system to store smaller, more manageable files that can be selectively served based on user device capabilities, reducing the storage burden of keeping all possible versions while enabling fast delivery
3Adaptability or versatility
If multiple transcoded versions of video content are stored, then user needs are better met, but storage requirements increase significantly
Solution Approach 1:
The system creates transcoded versions by systematically varying parameters such as resolution (e.g., 4K, 1080p, 720p), bitrate, and codec type. This parameter-based approach ensures coverage of diverse user device requirements while maintaining a manageable number of versions through intelligent selection of parameter combinations
Solution Approach 2:
The system designs the transcoded video library to serve multiple functions and device types simultaneously. By creating versions that cover common resolution and format requirements, a single transcoded library can cater to smartphones, tablets, desktops, and smart TVs, reducing the need for device-specific content while meeting diverse user needs
Data Source
Figure 1(a)
Figure 1(b)
Figure 2
AI summary
The present subject matter discloses a system and a method for data archival. A data archiving system (102) comprises a processor (108) and a compression module (120) coupled to the processor (108) and configured to compress an original data file to generate, for each level of lossy compression, a compressed base file and a patch file. The base file is generated using a lossy compression technique. The patch file for a particular level of lossy compression is generated using information lost at the particular level of lossy compression. The data archiving system (102) further comprises a data archiving module (122) coupled to the processor (108). The data archiving system (102) is configured to save a last base file and a plurality of patch files corresponding to each level of lossy compression, where the last base file is the base file corresponding to a last level of lossy compression.