Hash-Based Duplicate Data Element Consolidation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large entities face inefficiencies and resource waste due to duplicate documents stored across sub-entities, leading to confusion and unnecessary storage consumption, as existing systems lack effective methods to identify and consolidate duplicate documents.
Innovation Solution
A method involving hashing algorithms to generate hash-values for documents, which are then used to identify and consolidate duplicates by storing only unique hash-values in a centralized data repository, replacing duplicate documents with reference pointers to conserve storage space and improve system efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If documents are stored in sub-entity data repositories independently, then each sub-entity can manage its documents autonomously, but duplicate documents are created across multiple repositories consuming unnecessary storage resources
Solution Approach 1:
The patent uses hash pointers as simplified copies that reference original documents rather than storing full document copies. Each sub-entity stores only a hash pointer (a compact cryptographic reference) instead of the entire document, allowing autonomous access while eliminating redundant storage of identical document content across multiple repositories
Solution Approach 2:
The patent implements a nested structure where hash pointers (smaller data structures) are stored within sub-entity repositories, and these hash pointers reference larger document objects stored in the centralized repository. This nested arrangement allows sub-entities to maintain autonomous document management capabilities while the centralized repository consolidates actual document storage, reducing overall duplication
2Ease of operation
If multiple versions of identical documents are stored across entity and sub-entity repositories, then document accessibility is maintained, but system complexity and confusion increase for users searching for documents
Solution Approach 1:
The patent introduces hash pointers as intermediary elements that mediate between sub-entity repositories and the centralized document repository. These hash pointers serve as unique identifiers and references that simplify user interaction - users search and access documents through the centralized repository using hash pointers, eliminating the need to navigate multiple separate repositories and reducing confusion about which version to access
Solution Approach 2:
The centralized repository with hash pointer indexing provides universal access functionality for all sub-entities. A single centralized repository serves multiple sub-entities, providing document storage, retrieval, and duplicate elimination services to all users through a unified interface, thereby reducing system complexity while maintaining accessibility
3Measurement precision
If hashing algorithms are implemented to identify duplicate documents, then duplicate detection accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The patent transforms the document identification problem by changing the parameter from storing and comparing full document content to storing and comparing compact hash values. This parameter transformation maintains high duplicate detection accuracy through cryptographic hash functions while dramatically reducing processing time and computational resources, as hashing a document once and storing its hash value is much faster than repeatedly comparing entire document contents
Data Source
AI summary
A method for reducing a storage of duplicated documents is provided. Methods may include hashing each document stored in the centralized data repository by executing a hashing algorithm on the document, outputting a hash-value and adding the hash-value and a hash pointer to a hash table. Methods may further include crawling the hash table to identify duplicate hash-values. For each hash-value recorded on the hash table two or more times, methods may include combining two or more duplicate hash-values into a cluster and for each cluster identifying, on the hash table, a unique hash-value. For the unique hash-value, methods may include maintaining the unique hash-value on the hash table and maintaining the document corresponding to the unique hash-value in the memory address. For each remaining duplicate hash-value stored in the cluster, deleting the corresponding document from the memory address and store the reference pointer at the memory address.