Data Repository Profiling for Schema Efficiency and Metadata Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large commercial data repositories face challenges in evaluating the efficiency of their schema and ensuring the accuracy and validity of data and metadata, leading to significant operational and financial costs due to unreliable data.
Innovation Solution
A software tool that profiles large data repositories by analyzing table structures, field usage, data patterns, and business rules to identify inefficiencies and inaccuracies, allowing for the creation of a repository profile that can be used to correct and improve data quality and schema efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is imported from multiple sources with different quality control levels, then the quantity of data in the repository increases, but the accuracy and validity of the data deteriorates
Solution Approach 1:
The patent applies preliminary action by implementing data quality assessment and validation rules before data is imported into the repository. The system evaluates data against predefined criteria, checks for completeness and consistency, and validates metadata accuracy prior to ingestion, preventing poor quality data from entering the repository in the first place
Solution Approach 2:
The patent implements feedback mechanisms by continuously monitoring data quality metrics and providing information about data accuracy and validity. The system tracks quality indicators, compares them against standards, and uses this feedback to improve data import processes and identify sources of poor quality data for targeted improvement
2Adaptability or versatility
If the repository is maintained and revised over time with additional tables and fields, then the functionality and adaptability of the repository improves, but the efficiency of the schema deteriorates
Solution Approach 1:
The patent applies segmentation by organizing the repository schema into modular, hierarchical structures with clear separation of concerns. Data is divided into logical groups and tables are structured to minimize redundancy and optimize query performance, allowing the system to maintain efficiency while accommodating growing functionality
Solution Approach 2:
The patent implements parameter changes by dynamically adjusting schema parameters and optimization settings based on the current state and usage patterns of the repository. The system monitors schema efficiency metrics and automatically adjusts indexing, partitioning, and other parameters to maintain optimal performance as the repository evolves
3Reliability
If comprehensive data quality assessment is performed on large repositories, then the accuracy and validity of data is improved, but the time and resources required increases
Solution Approach 1:
The patent applies partial action by implementing selective data quality assessment that focuses on critical data elements and high-priority tables rather than uniformly assessing all data. The system identifies and prioritizes key quality indicators and concentrates assessment resources on areas with the greatest impact on data reliability
Solution Approach 2:
The patent replaces manual mechanical quality assessment with automated computational systems that continuously evaluate data quality. The system uses automated validation rules, statistical analysis, and machine learning algorithms to assess data accuracy and validity without requiring significant manual time investment
Data Source
AI summary
A method for metadata verification of a data repository of a computer system is described comprising accessing a data repository through a data profiler program, querying the data repository for metadata information about the data repository, presenting statistics regarding the metadata information using the profiler, analyzing the metadata information to identify suspect metadata information from the data repository and presenting the suspect metadata information for inspection using the profiler.