Schema Discovery Using Anchor Attributes for Heterogeneous Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Highly heterogeneous data sets lacking explicit schema pose significant usability and manageability challenges, as existing SQL tools require defined schema, and existing schema discovery approaches are inadequate for the level of heterogeneity found in unstructured data sets.
Innovation Solution
The method involves clustering conceptually similar records into groups to generate a reduced number of logical schemas, using user-provided seed anchor attributes to automatically infer additional anchor attributes and define schemas, thereby improving data analysis and application building capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If schema discovery is performed on highly heterogeneous data sets, then data usability and manageability improve, but the number of generated schemas becomes extremely large (thousands of schemas)
Solution Approach 1:
The patent segments the data set into multiple partitions based on anchor attributes, where each partition contains records with similar attribute combinations. This segmentation reduces the need to create thousands of separate schemas by grouping related records together, thereby improving data usability while controlling schema complexity.
Solution Approach 2:
The patent introduces a new dimension of organization by partitioning data based on anchor attributes rather than creating separate schemas for every possible attribute combination. This dimensional shift from schema-per-combination to partition-based organization resolves the contradiction between usability and complexity.
2Loss of information
If thousands of schemas are generated to represent all attribute combinations, then complete data coverage is achieved, but memory consumption and processor load increase significantly
Solution Approach 1:
The patent merges records with similar attribute patterns into the same partition, allowing a single schema to represent multiple records. This combining approach maintains complete data coverage while dramatically reducing memory consumption compared to storing separate schemas for every attribute combination.
Solution Approach 2:
The patent discards redundant schema definitions by identifying and grouping records with equivalent attribute structures into single partitions. The schema information is recovered and applied to all records in the partition, achieving complete coverage with minimal memory usage.
3Productivity
If existing SQL tools are used on heterogeneous data, then data analysis capability is maintained, but the tools require defined schema which is impractical for highly heterogeneous data
Solution Approach 1:
The patent performs preliminary partitioning of the heterogeneous data set into groups with similar attribute structures before applying SQL tools. This preliminary organization creates a manageable schema structure that enables existing SQL tools to function effectively on previously unmanageable heterogeneous data.
Solution Approach 2:
The patent introduces partitioning as an intermediary layer between the heterogeneous data and SQL tools. This intermediary structure translates highly heterogeneous data into a format that existing SQL tools can process, maintaining data analysis capability while bypassing the need for complex schema definitions.
Data Source
AI summary
Systems, methods, and computer-executable instructions for partitioning a data set include receiving anchor attributes of a data set. The data set includes records, with each record including attributes. A set of filter attributes that are not mutually exclusive with any of the anchor attributes is determined. A set of candidate attributes that include each unique attribute from the first data set, excluding the anchor attributes and the filter attributes, is determined. For each of the anchor attributes and the anchor attributes, an attribute context is determined. For each of the candidate attributes, a context similarity between each of the anchor attributes is determined. A new anchor attribute is selected from the set of candidate attributes based on the context similarity.