Dataset Relationship Visualization via Term Co-occurrence Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems lack effective mechanisms for dynamically surfacing relationships between datasets, which are typically static and not easily identifiable by users, limiting the ability to visualize and understand connections between them.
Innovation Solution
A computing system that constructs a data cube from source documents to extract count values of terms, identifying primary terms and their co-occurrences, and uses a visualizer to create a visualization that indicates relationships between datasets, allowing users to interactively explore and update these relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If datasets are organized with static classifications assigned by designers, then data organization is simple and straightforward, but relationships between datasets cannot be dynamically identified or visualized
Solution Approach 1:
The system pre-computes and stores term co-occurrence statistics in a data cube structure during data ingestion, so that relationship queries can be answered immediately without complex real-time analysis. This preliminary action enables dynamic relationship identification while maintaining fast query performance.
Solution Approach 2:
The patent introduces an intermediary visualization layer that sits between the raw datasets and the user. This visualization component translates complex dataset relationships into intuitive graphical representations, allowing users to discover connections without directly analyzing the underlying data structures.
2Loss of information
If comprehensive term co-occurrence analysis is performed across all source documents, then complete relationships between datasets can be identified, but computational complexity and processing time increase significantly
Solution Approach 1:
The system segments the large-scale text corpus into manageable source documents and further into term co-occurrence pairs. By processing and storing relationships at this granular level in a data cube, the system avoids the computational burden of analyzing entire documents or datasets as monolithic units.
Solution Approach 2:
The patent transforms the problem from analyzing relationships between entire datasets into analyzing co-occurrences of individual terms across documents. This dimensional transformation to the term level enables efficient storage and querying of relationships without requiring complex dataset-level comparisons.
3Adaptability or versatility
If static classifications are used to identify related datasets, then the system is simple to implement, but it cannot adapt to newly discovered relationships or changing data contexts
Solution Approach 1:
The system replaces static classification assignments with dynamic term co-occurrence analysis. Relationships are computed based on actual term patterns in the data rather than predetermined categories, allowing the system to automatically adapt to new relationships as data is added or modified without requiring manual reclassification.
Solution Approach 2:
The system performs self-service relationship discovery by automatically analyzing term co-occurrences in source documents and generating visualizations without requiring manual intervention. This autonomous operation enables continuous adaptation to new data while maintaining system simplicity through automated processes.
Data Source
AI summary
Technologies described herein relate to surfacing relationships between datasets. Datasets in a collection of datasets are assigned terms that are descriptive of content of the datasets. Co-occurrences of the terms in source documents that include unstructured text are identified, and relationships between datasets is identified based upon the co-occurrences of the terms in the source documents. The relationships between datasets are visualized in accordance with visualization instructions set forth by a user.