Private Data Identification Using Length and Character Distribution Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data identification methods rely on pre-trained models and regular expressions, which have limited application scope and require significant human and computational resources, failing to adapt to user-specific data types without pre-labeled training data.
Innovation Solution
A method that uses unsupervised learning to identify private data by calculating length and character distribution similarities between a reference data set and a target data set, allowing for active identification without pre-trained models or labeled data, thus reducing resource consumption and expanding application scope.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If pre-trained models and regular expressions are used for data identification, then identification accuracy for known data types is improved, but application scope is limited and human resource consumption increases
Solution Approach 1:
The system performs self-learning by automatically analyzing data patterns and generating identification rules without requiring manual annotation or pre-training. The model learns directly from the data itself, enabling adaptive identification of new data types as they emerge, thus expanding application scope while maintaining accuracy
Solution Approach 2:
The system dynamically adjusts identification parameters based on data characteristics rather than using fixed pre-trained parameters. By changing parameters adaptively according to the specific data being analyzed, the system can accurately identify diverse data types without requiring separate pre-trained models for each type
2Measurement precision
If pre-trained models are used for data identification, then identification capability for predetermined data types is improved, but computational resource consumption and human resource requirements increase
Solution Approach 1:
The system learns automatically from data patterns without requiring extensive pre-training computations or manual rule creation. This self-learning approach reduces computational overhead by performing lightweight pattern recognition directly on the data, eliminating the need for resource-intensive pre-training phases
Solution Approach 2:
The system uses lightweight, disposable identification rules generated on-demand rather than maintaining expensive, permanent pre-trained models. These generated rules are computationally inexpensive and can be discarded after use, reducing overall computational resource consumption
3Productivity
If pre-prepared embedded rules are used for data identification, then identification speed for covered data types is improved, but adaptability to user-specific data types without pre-labeled training data deteriorates
Solution Approach 1:
The system performs preliminary analysis of data patterns to generate identification rules on-demand before actual identification tasks. This preliminary pattern recognition enables rapid identification of user-specific data types without requiring pre-labeled training data, maintaining high speed while improving adaptability
Solution Approach 2:
The system dynamically generates and updates identification rules based on the specific data being analyzed rather than using static pre-prepared rules. This dynamic adaptation allows the system to maintain high identification speed for known types while simultaneously adapting to new user-specific data types
Data Source
AI summary
Implementations of the present specification disclose a method, apparatus, device, and a readable medium for identifying private data. A solution includes: obtaining a first length distribution vector and first character distribution statistical information of a first data set, the first data set being a set of private data samples; obtaining a second length distribution vector and second character distribution statistical information of a second data set, the second data set being a set of at least a part of data samples corresponding to a to-be-identified field; calculating a length distribution similarity and a character distribution similarity between the first data set and the second data set, and determining whether data corresponding to the to-be-identified field is private data of a same type as the first data set based on the length distribution similarity and the character distribution similarity.