ML File-Print Identification for Corrupted Stream Data Types
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of data storage demand outpaces the capacity to store it, and existing methods like data compression and adding physical storage are inadequate, while file type identification based on file signatures is unreliable for modified or corrupted files.
Innovation Solution
A system and method using machine learning to identify file types by analyzing statistical similarities in binary distributions, generating a 'file-print' for files with missing or unreadable signatures, and applying a trained machine learning algorithm to predict file types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If file signature checking is used to identify file types, then the identification process is simple and fast, but the reliability deteriorates when files are modified or corrupted
Solution Approach 1:
The patent introduces file-prints as an intermediary representation that captures statistical characteristics of file contents. Instead of directly relying on file signatures that may be corrupted, the system extracts file-prints from the actual content and uses these as a reliable intermediary for type identification through machine learning comparison
Solution Approach 2:
The patent replaces the mechanical file signature checking method with a machine learning-based statistical analysis system. Instead of simple string matching against known signatures, the system uses computational algorithms to compare statistical distributions and identify file types even when signatures are modified or corrupted
2Quantity of substance
If traditional compression methods are used to increase storage capacity, then storage density improves slightly, but the effectiveness deteriorates with multi-media data types
Solution Approach 1:
The patent changes the fundamental parameter of data representation by encoding files based on their identified types using type-specific encoding algorithms. Instead of applying generic compression to all data, the system adapts encoding parameters and methods according to the statistical characteristics and type of each file, achieving superior compression ratios for multi-media data
3Reliability
If machine learning algorithms are implemented for file type identification, then identification accuracy improves for corrupted files, but system complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-training machine learning models with extensive file-print data from various file types before deployment. The models are pre-computed and stored, allowing the identification system to make accurate predictions without performing complex real-time training, thus reducing operational complexity while maintaining high accuracy
Solution Approach 2:
The patent creates simplified copies of file characteristics through file-prints, which are compact statistical representations that capture essential type information. These file-print copies are much smaller and simpler than the original files but retain sufficient information for accurate type identification through machine learning comparison
Data Source
AI summary
A system and method for file type identification involving extraction of a file-print of a file, the file-print being a unique or practically-unique representation of statistical characteristics associated with the distribution of bits in the binary contents of the file, similar to a fingerprint. The file-print is then passed to a machine learning algorithm that has been trained to recognize file types from their file-prints. The machine learning algorithm returns a predicted file type and, in some cases, a probability of correctness of the prediction. The file may then be encoded using an encoding algorithm chosen based on the predicted file type.