Distributed Data Deduplication Using Edge Classification and Clustering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication techniques in distributed computing environments, such as edge computing, are inadequate for managing diverse data types and applications with different Quality of Service (QoS) requirements, leading to inefficiencies in data deduplication and bandwidth utilization.
Innovation Solution
Implementing a data deduplication system that classifies and clusters data based on previously uploaded data, using data classifiers and extension engines at edge nodes, along with a data deduplication engine in the cloud, to identify and remove duplicates collaboratively across edge nodes and cloud servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional deduplication techniques are used in distributed computing environments, then data storage space is reduced, but deduplication rates are low and bandwidth utilization is poor when dealing with diverse data types
Solution Approach 1:
The patent segments data into different data types (structured, unstructured, semi-structured) and applies type-specific deduplication strategies. Data classifiers categorize incoming data streams into distinct types, allowing the system to use appropriate deduplication methods for each type rather than applying a uniform approach, thereby improving overall deduplication rates while maintaining storage efficiency
Solution Approach 2:
The system changes the parameter of data representation by creating type-specific hash functions and comparison criteria for different data types. Structured data uses schema-based validation and column-level hashing, while unstructured data uses content-based hashing, allowing for more accurate duplicate detection across diverse data formats and improving deduplication effectiveness
2Ease of operation
If data is uploaded without prior deduplication at edge nodes, then upload process is simple, but bandwidth utilization is inefficient due to redundant data transmission
Solution Approach 1:
The system performs preliminary deduplication actions at edge nodes before data upload to the cloud. Data classifiers and extension engines execute local deduplication checks against previously uploaded data, removing duplicates upfront. This preliminary action reduces the volume of data transmitted over the network, improving bandwidth utilization while maintaining simple upload operations through automated client-side processing
Solution Approach 2:
The patent introduces extension engines as intermediary components between data sources and the upload process. These engines automatically perform deduplication operations without requiring manual intervention, bridging the gap between simple upload operations and complex deduplication requirements. The intermediary handles the complexity of deduplication internally while presenting a simple upload interface to users
3Device complexity
If a unified deduplication approach is used for all data types, then system complexity is low, but deduplication effectiveness is poor for diverse data types with different QoS requirements
Solution Approach 1:
The patent applies local quality by implementing data-type-specific deduplication strategies tailored to the characteristics of each data category. Structured data receives schema-based validation and column-level processing, while unstructured data gets content-based hashing and fingerprinting. Each data type receives customized treatment optimized for its specific properties and QoS requirements, improving deduplication effectiveness without requiring overly complex centralized control
Solution Approach 2:
The system achieves universality through a modular architecture where data classifiers, extension engines, and deduplication modules work together across all data types. The framework provides universal deduplication capabilities that adapt to different data types through configurable plugins and algorithms, maintaining system coherence while handling diverse data formats and QoS requirements through a unified multi-functional platform
4Use of energy by moving object
If deduplication is performed centrally in the cloud only, then edge node resources are conserved, but network bandwidth is wasted transmitting duplicate data
Solution Approach 1:
The system performs preliminary deduplication actions at edge nodes using extension engines that leverage local storage and computing resources. By executing deduplication checks locally before upload, the system prevents duplicate data from consuming network bandwidth in the first place. This preliminary action at the edge reduces upstream traffic while maintaining efficient resource utilization through distributed processing
Solution Approach 2:
The patent adds a spatial dimension to deduplication by distributing the deduplication function across multiple locations (edge nodes and cloud). Instead of single-point centralized deduplication, the system implements a multi-dimensional architecture where edge nodes perform local deduplication and the cloud performs global deduplication, optimizing both network bandwidth and distributed resource utilization through hierarchical processing
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for deduplicating data in distributed computing environments are provided herein. An example computer-implemented method includes obtaining, by a node of a distributed computing system, a set of data from an edge device, and identifying a data type corresponding to the set of data from among a plurality of data types. The method includes using information corresponding to the identified data type to determine whether one or more portions of data in the set of data were previously uploaded to at least one server of the distributed computing system. The method also includes, in response to determining that the one or more portions of the data were previously uploaded, updating the set of data by removing the one or more portions of the data from the set of data, and uploading the updated set of data to the at least one server.