Data Identification via State Transition Matrix Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data identification methods are limited in scope and resource-intensive, requiring pre-trained models and labeled data, which restrict their ability to adapt to user demands and identify data types without corresponding predetermined rules.
Innovation Solution
A data identification method using a state transition matrix set generated from known data types, calculating sample state transition probabilities to determine the similarity between data samples, and identifying data types based on statistical information without pre-labeling or pre-trained models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-trained models and built-in rules are used for data identification, then identification accuracy for known data types is improved, but adaptability to unknown data types and user demand customization deteriorates
Solution Approach 1:
The system enables users to perform data identification through self-service by selecting data samples with known data types and automatically generating identification rules. Users provide sample data, and the system automatically extracts features and generates rules without requiring pre-trained models or manual rule configuration, thus achieving both accuracy for known types and adaptability for unknown types.
Solution Approach 2:
The system changes the parameters of data identification from using fixed pre-trained models to dynamically generating rules based on user-provided samples. By allowing users to input sample data and automatically extracting identification features, the system adapts its identification parameters to match user-specific needs and unknown data types while maintaining accuracy.
2Productivity
If pre-trained models and built-in rules are used for data identification, then identification speed for common data types is improved, but resource consumption for custom identification deteriorates
Solution Approach 1:
The system reduces resource consumption by enabling users to perform self-service data identification. Instead of requiring expensive pre-trained models for every possible data type, users provide samples and the system automatically generates rules, significantly reducing the need for human annotation and computer training resources while maintaining fast identification speeds.
Solution Approach 2:
The system performs preliminary action by allowing users to provide sample data with known data types before actual identification occurs. These samples are used to pre-generate identification rules, which then enable fast identification of new data without requiring real-time training or complex computational resources during the actual identification process.
3Device complexity
If built-in rules are used for data identification, then simplicity of the system is improved, but scope of application deteriorates
Solution Approach 1:
The system achieves universality by enabling the same simple framework to handle both known and unknown data types. Users can provide samples of any data type, and the system automatically generates appropriate identification rules, making the system universally applicable to various data types without requiring complex pre-trained models for each type.
Solution Approach 2:
The system maintains simplicity while expanding scope by enabling users to perform self-service identification. Instead of requiring a complex pre-trained model for every data type, the system provides a simple interface where users input samples and automatically receive generated rules, achieving both simplicity and broad applicability.
Data Source
AI summary
Implementations of the present specification disclose a data identification method, apparatus, device, and a computer-readable medium. A solution includes: obtaining a first data set, data samples in the first data set being at least a part of data of a to-be-identified field; obtaining a state transition matrix set generated based on statistics of data samples in a second data set, a data type of the data samples in the second data set being known; determining sample state transition probabilities corresponding to the data samples in the first data set based on the state transition matrix set; determining a ratio between a number of data samples in the first data set whose sample state transition probabilities are greater than a first threshold and a total number of the data samples in the first data set; and determining data corresponding to the to-be-identified field as being of a same data type as the data samples in the second data set in response to that the ratio is greater than a second threshold.