Unique Entity Discovery in Structured Data Using Partitioned Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Discovering unique entities in large volumes of data is a costly and time-consuming process, especially when dealing with growing data volumes and increasing numbers of fields, requiring efficient methods to reduce analysis time while ensuring accuracy and consistency.
Innovation Solution
A system and method that utilize a unique column finding unit with six analysis techniques in a predefined sequential order, including light and heavy weight analyses, to efficiently identify unique entities by concatenating columns, creating vertical partitions, and validating combinations, with a time check mechanism to stop analysis within a predefined duration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data profiling is performed on huge volumes of data to discover all possible unique keys with high accuracy, then measurement precision is improved, but loss of time increases and productivity decreases
Solution Approach 1:
The patent divides the data into multiple partitions and processes each partition separately. The system creates vertical partitions of columns and analyzes them individually using the unique column finding unit, which applies six analysis techniques in sequence. This segmentation allows parallel processing and reduces the time required to analyze huge volumes of data while maintaining accuracy through comprehensive validation across all partitions.
Solution Approach 2:
The system performs preliminary actions by first concatenating columns to form entities and checking for uniqueness before proceeding to detailed partition analysis. The time check mechanism is established beforehand with a predefined time duration, and the system stops analysis if this limit is exceeded, ensuring time constraints are met while still performing necessary validation steps.
2Measurement precision
If data profiling is performed on huge volumes of data to discover all possible unique keys with high accuracy, then measurement precision is improved, but productivity decreases
Solution Approach 1:
The patent divides the data into multiple partitions and processes each partition separately. The system creates vertical partitions of columns and analyzes them individually using the unique column finding unit, which applies six analysis techniques in sequence. This segmentation allows parallel processing and reduces the time required to analyze huge volumes of data while maintaining accuracy through comprehensive validation across all partitions.
Solution Approach 2:
The system performs analysis on a percentage of rows (partial action) rather than requiring 100% of data to be processed, which increases productivity while still providing statistically significant results for unique key discovery. The time check mechanism also implements partial action by stopping analysis when a predefined time duration is reached, delivering results within acceptable timeframes.
3Adaptability or versatility
If the number of fields in data increases, then adaptability is improved, but device complexity increases and loss of time increases
Solution Approach 1:
The unique column finding unit is designed as a universal component that can handle any number and type of fields through a standardized six-technique analysis process. The system concatenates any combination of columns to form entities and applies the same validation logic regardless of field diversity, making the system adaptable to varying data structures without increasing complexity proportionally.
Solution Approach 2:
The patent divides the data into multiple partitions and processes each partition separately. The system creates vertical partitions of columns and analyzes them individually using the unique column finding unit, which applies six analysis techniques in sequence. This segmentation allows parallel processing and reduces the time required to analyze huge volumes of data while maintaining accuracy through comprehensive validation across all partitions.
4Adaptability or versatility
If the number of fields in data increases, then adaptability is improved, but loss of time increases
Solution Approach 1:
The unique column finding unit is designed as a universal component that can handle any number and type of fields through a standardized six-technique analysis process. The system concatenates any combination of columns to form entities and applies the same validation logic regardless of field diversity, making the system adaptable to varying data structures without increasing complexity proportionally.
Solution Approach 2:
The system performs preliminary actions by first concatenating columns to form entities and checking for uniqueness before proceeding to detailed partition analysis. The time check mechanism is established beforehand with a predefined time duration, and the system stops analysis if this limit is exceeded, ensuring time constraints are met while still performing necessary validation steps.
Data Source
AI summary
Unique key fields in structured data is a critical characteristic of data which plays significant role in data management. Profiling high volume of data for discovering all possible unique keys with high accuracy is a costly and time taking affair. A method and system for finding one or more unique entities in a data have been provided. The one or more unique entities obtained by this approach are complete and the response time is quick. The method is scalable to increasing volume of data and number of fields. The system is configured to perform the analysis process in multiple phases taking the less volume initially and increasing it gradually, thereby reducing the load on later phases as the unique results found with light volume phases. The method also comprises a time check mechanism after different stages if a user wants to do discovery for limited time.