CDN Video Segment Deduplication for Redundant Stream Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing content delivery networks (CDNs) face inefficiencies due to the propagation of redundant encoded video segments, leading to increased storage and traffic, as separate encoders produce perceptually identical but format- or path-differentiated transcoder outputs.
Innovation Solution
Implement a method to detect and discard redundant video segments by comparing hash values and quality metrics across segments encoded by multiple encoders, ensuring only one version is saved on the CDN, thereby reducing storage and traffic.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If redundant encoders send multiple versions of the same video segment to the CDN, then reliability and redundancy are improved, but storage usage and network traffic increase
Solution Approach 1:
The patent changes the parameter of video segment representation by computing hash values (digital fingerprints) of video segments and using these hash values for identification and deduplication. Instead of storing multiple redundant versions, the system stores a single version and uses hash-based metadata to represent redundant references, thereby reducing storage usage while maintaining redundancy capability
Solution Approach 2:
The patent creates a copy of the video segment's essential identifying information (hash value) rather than copying the actual video data. The hash value serves as a lightweight representation that enables redundancy management without duplicating the full video content, resolving the contradiction between maintaining redundancy and reducing storage
2Reliability
If redundant encoders send multiple versions of the same video segment to the CDN, then reliability and redundancy are improved, but network traffic increases
Solution Approach 1:
The patent extracts the essential identifying feature (hash value) from the video segment and separates it from the full video data. By transmitting and comparing only the hash values rather than full video segments, the system maintains redundancy capability while dramatically reducing network traffic and energy consumption
Solution Approach 2:
The patent changes the parameter being transmitted over the network from full video segment data to compressed hash value representations. This parameter transformation reduces the amount of data transmitted while preserving the ability to identify and manage redundant segments, thereby reducing network traffic while maintaining redundancy
3Reliability
If the CDN stores multiple versions of identical video segments, then encoder redundancy is maintained, but CDN efficiency decreases
Solution Approach 1:
The patent copies only the essential identifying information (hash value) rather than the full video segment data. This lightweight copy enables efficient redundancy detection and management in the CDN, allowing the system to maintain encoder redundancy while improving CDN efficiency by avoiding storage and transmission of duplicate full video segments
Solution Approach 2:
The patent transforms the redundancy representation from full video segment copies to hash value metadata. This parameter change enables the CDN to efficiently track and manage redundant segments through compact hash comparisons, thereby maintaining encoder redundancy capability while significantly improving CDN operational efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods are described herein for processing video data to cause a reduction in the number of versions of encoded content propagated via a content delivery network (CDN). Redundant encoders may send the versions of the same video segment for redundancy purposes. A server in the CDN may determine, based on information in a video segment, that the video segment may have been encoded based on source frames that are bitwise or perceptually identical to source frames of a previously received video segment. The server may then discard the identical video segment so that the redundant copy is not saved in the CDN.