Reusable Distance Measures in Data Column Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data analysis efficiency is hindered by the need for complex data preparation and comparison of unordered columns, which often require replacement and normalization, and existing methods lack reusable distance measures, leading to repetitive data preparation tasks across different analysis projects.
Innovation Solution
Storing distance measures as metadata within column definitions allows for reusable distance measures, enabling efficient data analysis by reusing defined measures across different analyses, delegating calculations, and precalculating metrics, thereby simplifying data comparison and display.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If distance measures are calculated during each data analysis project, then measurement precision is maintained, but loss of time increases due to repetitive calculations
Solution Approach 1:
The system performs preliminary calculation of distance measures during data loading and stores them in the database. When data is loaded, the system automatically computes distance measures for all column pairs and saves these pre-calculated values. During subsequent analysis projects, these pre-computed distance measures are directly reused without recalculation, eliminating repetitive computation while maintaining measurement precision.
2Measurement precision
If data preparation tasks are performed for each analysis project, then measurement precision is ensured, but productivity decreases due to repetitive manual work
Solution Approach 1:
The system implements self-service data preparation by automatically performing distance measure calculations during the data loading process. The database management system autonomously computes distance measures for all columns and stores them without requiring manual intervention. This automation eliminates repetitive manual data preparation tasks while ensuring consistent and precise distance measures are available for all analysis projects.
Solution Approach 2:
Data preparation including distance measure calculation is performed in advance during data loading. The system pre-computes all necessary distance measures and stores them in the database before any analysis projects begin. This preliminary preparation eliminates the need for repeated manual data preparation tasks in subsequent analyses, significantly improving productivity while maintaining precision.
3Ease of operation
If unordered columns are replaced with ordered columns from different data sources, then ease of operation improves for data comparison, but device complexity increases due to complex data preparation requirements
Solution Approach 1:
The system replaces the mechanical process of manually selecting and transforming columns with an automated computational approach. Instead of requiring users to manually replace unordered columns with ordered equivalents from different data sources, the system automatically calculates distance measures for any column pair using appropriate metrics. This substitution of manual mechanical operations with automated computational methods simplifies the user interface while handling the complexity internally.
Solution Approach 2:
The system changes the parameter representation from requiring explicit column transformations to using distance measure parameters. Rather than transforming unordered columns into ordered columns from different sources, the system accepts any column pair and computes their distance using appropriate metrics. This parameter change approach allows direct comparison of unordered columns while the system automatically selects and applies the correct distance metric, reducing data preparation complexity.
Data Source
AI summary
In one embodiment the present invention includes a computer-implemented method of analyzing data. The method includes storing, by a computer system, a column definition that includes metadata that defines a column. The method further includes generating, by the computer system, a distance measure for the column. The method further includes storing, by the computer system, the distance measure for the column as part of the metadata for the column in the column definition. In this manner, improvements may result in the areas of reuse, delegation, usability, and precalculation.