Collaborative IoT Data Deduplication via Edge-Cloud Coordination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing amount of data generated by IoT devices poses challenges in data ingestion, storage management, and resource allocation, leading to high costs and resource scarcity, particularly in edge computing and cloud environments.
Innovation Solution
Implementing a collaborative data deduplication system where edge nodes and cloud layers work together to identify and eliminate duplicate data, using chunking and fingerprinting techniques to reduce data transmission and storage needs, thereby improving deduplication efficiency and reducing storage costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all data from IoT devices is stored in cloud infrastructure, then data availability and access are improved, but storage costs and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the essential unique information from data streams by identifying and retaining only first occurrences of data chunks while eliminating duplicates. This extraction approach maintains data availability for legitimate access while dramatically reducing storage requirements by removing redundant copies across the network
Solution Approach 2:
The system discards duplicate data chunks that have already been stored elsewhere in the network, using metadata references to recover access to the original unique copies. This allows the system to maintain data accessibility while eliminating redundant storage consumption
2Productivity
If data is transmitted from all edge devices to the cloud, then centralized data processing is improved, but network bandwidth utilization deteriorates
Solution Approach 1:
The patent extracts only unique data chunks and their metadata for transmission to the cloud, leaving duplicate copies at edge devices. This extraction strategy enables centralized processing of essential information while minimizing network bandwidth consumption by transmitting only the necessary unique portions of data
Solution Approach 2:
The system performs preliminary deduplication at edge devices before transmission, identifying and eliminating duplicate chunks locally. This preliminary action reduces the volume of data that needs to be transmitted over the network while ensuring that unique data reaches the cloud for centralized processing
3Quantity of substance
If data deduplication is implemented across the entire network, then storage efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the deduplication system into distributed components at edge devices and a centralized metadata management layer. Each edge device independently identifies and eliminates local duplicates using simple algorithms, while the cloud maintains a centralized index of unique chunks. This segmentation reduces individual device complexity while achieving network-wide storage efficiency
Solution Approach 2:
The system introduces a metadata intermediary layer that mediates between edge devices and cloud storage. This intermediary manages the centralized index of unique chunks and coordinates deduplication operations across the network, simplifying the complexity by centralizing coordination while keeping edge device operations simple and independent
Data Source
AI summary
One example method includes collaborative deduplication. A deduplication engine implemented at a cloud level collaborates or coordinates with an extension engine of the deduplication at an edge node. This allows data ingested at a node to be collaboratively deduplicated prior to transfer to the cloud and after transfer to the cloud.