Image Dark Data Valuation via Similarity Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of dark data in storage systems, particularly unlabeled and unrelated image data, leads to significant storage costs and inefficiencies, as existing technologies lack effective methods for valuing and managing this 'dark data' based on semantic extraction and correlation analysis.
Innovation Solution
A method utilizing a deep self-taught hashing algorithm (DSTH) with similarity hashing and graph-structure ranking to assign significance scores to image data, constructing a hash map and calculating importance scores, allowing for the valuation and potential mining of image dark data by identifying semantically relevant images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dark data is stored without valuation, then storage capacity is maximized, but storage cost performance deteriorates
Solution Approach 1:
The system performs self-service by automatically evaluating dark data through hashing algorithms and significance scoring, enabling the storage system to autonomously identify valuable data without external intervention. This self-evaluation mechanism allows the system to prioritize retention of high-value dark data while managing storage resources efficiently.
Solution Approach 2:
The invention changes the parameter of data valuation by transforming unevaluated dark data into scored data entities with significance values. By applying hashing functions and significance scoring, the system converts static stored data into dynamically evaluable data structures that can be prioritized and managed based on their computed value.
2Measurement precision
If all dark data is evaluated and processed, then data valuation accuracy is improved, but processing time increases
Solution Approach 1:
The system extracts only the essential features needed for valuation by converting images to hash codes rather than processing complete image data. This extraction approach maintains valuation accuracy by preserving similarity information while dramatically reducing processing time through efficient hash-based comparison.
Solution Approach 2:
The invention creates simplified copies of dark data in the form of hash codes and significance scores, which serve as proxies for the original data. These copies enable rapid evaluation and prioritization without requiring processing of the full-resolution original images, thus maintaining accuracy while reducing time costs.
3Measurement precision
If deep self-taught hashing algorithm is used, then similarity detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The system performs preliminary action by pre-computing hash codes for all dark data during an initial processing phase. This preliminary hashing enables subsequent rapid similarity detection without repeatedly applying complex algorithms, as the hash representations are already available for quick comparison.
Solution Approach 2:
The invention substitutes complex mechanical image processing with a more efficient computational approach using hashing algorithms. By replacing direct image comparison with hash-based similarity detection, the system achieves high accuracy while reducing computational complexity through bitwise operations and mathematical hashing functions.
Data Source
AI summary
A method for valuation of image dark data based on similarity hashing, the method including: (1) acquiring a dark dataset, and inputting the dark dataset into a trained deep self-taught hashing algorithm model to obtain a hash code of each image in the dark dataset; (2) constructing a hash map according to the hash code of each image in the dark dataset obtained in (1), where the hash map includes a plurality of nodes, each node represents the hash code of each image; (3) acquiring a significance score of each node in the hash map obtained in (2); and (4) sorting significance scores of the plurality of nodes obtained in (3) in a descending order, and outputting images corresponding to top k significance scores of the plurality of nodes to users, where k is a natural number.