Hierarchical Clustering Using Proxy Distance for File Deduplication

The patent introduces a proxy distance function as an intermediary between the actual file segments and the clustering algorithm. This proxy function approximates the true distance metric, allowing the algorithm to identify optimal clusters without performing computationally expensive direct comparisons of all file segments. The proxy distance serves as a mediator that preserves clustering quality while reducing computational complexity.

The patent transforms the clustering problem by changing the parameter representation from direct file segment comparisons to proxy distance calculations. By modifying how distances are computed (using the proxy function instead of exact segment comparison), the system achieves optimal cluster identification with reduced computational resources.

The patent implements a reference-based storage system where instead of storing multiple complete copies of identical file attachments, the system stores one master copy and creates lightweight references or pointers to it. Each file instance maintains a reference to the shared data, enabling restoration capability while dramatically reducing storage consumption. This is evident in the deduplication mechanism where duplicate segments are replaced with references.

The patent employs a nested structure where reference information is embedded within file metadata, and the actual data is nested in a shared storage location. The reference acts as a container pointing to the nested actual data, allowing multiple file instances to share the same underlying data storage while maintaining individual file integrity and restoration capability.