Data Storage System Characterization via Summary Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face challenges in efficiently characterizing data sources across multiple databases, particularly in identifying relationships between fields and ensuring data quality, especially when dealing with large datasets or during data migration, where profiling all tables is time-consuming and costly, and consistency between replica databases needs to be ensured without re-profiling.
Innovation Solution
A method for characterizing data involves reading data from a storage system, processing summary data to identify relationships between fields from different data sources, and generating system information using a processor, which includes descriptive information and applying rules to aggregate data, allowing for data quality tracking and validation without requiring multiple profiling passes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all tables are profiled to ensure data quality and identify relationships, then data characterization accuracy is improved, but time consumption and processing cost increase significantly
Solution Approach 1:
The patent segments the data profiling process into two distinct phases: an initial comprehensive profiling phase that profiles all tables to establish baseline characteristics and relationships, and a subsequent targeted re-profiling phase that only profiles specific tables when changes are detected. This segmentation allows the system to achieve accurate data characterization without repeatedly profiling all tables, thereby reducing time consumption while maintaining accuracy.
Solution Approach 2:
The patent performs preliminary profiling of all tables during an initial phase to establish baseline data characteristics, relationships, and quality metrics before actual data migration or processing operations begin. This preliminary action creates a reference framework that enables efficient targeted re-profiling later, eliminating the need for complete re-profiling and significantly reducing the time required for subsequent operations.
2Reliability
If replica databases are re-profiled to ensure consistency, then data quality reliability is improved, but processing time and resource consumption increase
Solution Approach 1:
The patent performs preliminary profiling of replica databases during an initial synchronization phase to establish baseline characteristics and detect inconsistencies. This preliminary action creates a reference state that enables efficient change detection in subsequent phases, allowing the system to verify consistency without re-profiling all replica tables repeatedly.
Solution Approach 2:
The patent implements a feedback mechanism that monitors data changes in replica databases and triggers targeted re-profiling only when inconsistencies or significant changes are detected. The system compares current data characteristics against baseline profiles and activates re-profiling operations only when necessary, thereby maintaining data consistency reliability while minimizing processing time and resource consumption.
3Loss of information
If comprehensive data profiling is performed to identify field relationships, then data quality tracking capability is improved, but system resource consumption increases
Solution Approach 1:
The patent segments data quality tracking into two phases: an initial comprehensive profiling phase that profiles all tables to establish baseline relationships and quality metrics, consuming higher resources but only temporarily. The subsequent phase uses targeted re-profiling only for affected tables, significantly reducing ongoing resource consumption while maintaining the ability to track data quality through the segmented approach.
Solution Approach 2:
The patent performs preliminary profiling of all data sources during an initial phase to establish comprehensive baseline information about field relationships, data types, and quality characteristics. This preliminary action invests processing resources upfront to create a detailed reference framework, which then enables efficient targeted profiling and quality tracking with minimal ongoing resource consumption.
Data Source
AI summary
Characterizing data includes: reading data from an interface to a data storage system, and storing two or more sets of summary data summarizing data stored in different respective data sources in the data storage system; and processing the stored sets of summary data to generate system information characterizing data from multiple data sources in the data storage system. The processing includes: analyzing the stored sets of summary data to select two or more data sources that store data satisfying predetermined criteria, and generating the system information including information identifying a potential relationship between fields of records included in different data sources based at least in part on comparison between values from a stored set of summary data summarizing a first of the selected data sources and values from a stored set of summary data summarizing a second of the selected data sources.