Data Table Association Measurement via Field Importance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for determining the degree of association between data tables in big data environments are not accurate enough to reflect the importance of data fields and suffer from exponential attenuation of association values, leading to inaccurate identification of important data tables.
Innovation Solution
A method that collects statistics on path length and path number of data tables and determines importance coefficients of fields based on their dependency relationships, using these metrics to calculate a degree of association that accounts for field usage and connectivity, thereby identifying data tables with high accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If current techniques calculate degree of association based on proportion of contributed fields, then calculation is simple, but measurement precision is insufficient and cannot reflect field-level importance differences
Solution Approach 1:
The patent segments the data table association measurement into field-level components. Instead of treating data tables as monolithic units, it breaks down the association calculation into individual field contributions, path segments, and dependency relationships. This segmentation enables precise measurement of which specific fields contribute to associations while maintaining manageable calculation complexity through modular computation of path lengths and field importance coefficients.
Solution Approach 2:
The patent introduces multiple parameters to enhance measurement precision: path length, path number, and field importance coefficients. These parameters transform the simple proportion-based calculation into a multi-dimensional measurement system that captures the nuanced relationships between data tables, accounting for both direct and indirect dependencies with varying weights.
2Reliability
If indirect dependency relationships are converted to direct relationships through intermediate tables, then association can be calculated, but exponential attenuation occurs causing rapid decrease in association values
Solution Approach 1:
The patent uses path length as an intermediary parameter to mediate the association calculation between data tables with indirect dependencies. Instead of directly converting indirect relationships to equivalent direct relationships (which causes exponential attenuation), it introduces path length as a linear scaling factor that preserves the true magnitude of associations while accounting for the number of intermediate steps in the dependency chain.
Solution Approach 2:
The patent performs preliminary analysis of the dependency graph structure before calculating associations. It pre-identifies all relevant paths between data tables, calculates their lengths, and determines field importance coefficients in advance. This preliminary action prevents information loss by ensuring all contribution paths are captured before the final association calculation, avoiding exponential attenuation of indirect relationships.
3Quantity of substance
If data warehouse stores thousands of data tables with dozens or hundreds of fields each, then data capacity is sufficient, but identifying important data tables becomes difficult and time-consuming
Solution Approach 1:
The patent replaces manual or simple heuristic identification methods with an automated computational system based on graph theory and dependency analysis. The system automatically traverses the dependency graph, calculates path lengths, determines field importance coefficients, and ranks data tables by their association degrees. This mechanical substitution of automated calculation eliminates time-consuming manual identification while handling thousands of data tables efficiently.
Solution Approach 2:
The patent transforms the identification problem into a parameter-based ranking system. By calculating quantitative parameters such as path length, path number, and field importance coefficients for each data table, the system enables automated sorting and identification of important tables. This parameter-driven approach replaces subjective or time-consuming manual assessment with objective, computable metrics that can be rapidly processed even for large-scale data warehouses.
Data Source
AI summary
Embodiments of the disclosure provide a method, apparatus, and system for identifying data tables. The method comprises acquiring a first dependency relationship between a plurality of data tables; collecting statistics on a path length and a path number of the data tables based on the first dependency relationship; acquiring a second dependency relationship between one or more fields in the data tables; determining importance coefficients of the one or more fields based on the second dependency relationship; determining a degree of association between the data tables by using the path length, the path number, and the importance coefficients; and identifying the data tables based on the degree of association.