Automated Data View Creation via Catalog Linking

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users face difficulties in extracting meaningful data views from large datasets, requiring domain knowledge and manual updates, which are error-prone and obsolete when schema changes occur.

Innovation Solution

A system that accesses a data catalog to automatically infer and create data views by linking related data elements from multiple sources based on query terms, using classification and usage history to populate views and maintain relationships.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If users manually extract and create data views from large datasets, then they can obtain customized data projections, but they require extensive domain knowledge and manual effort which is error-prone and time-consuming

Engineering Contradiction:

Improveease of creating data viewsVSAvoidtime spent on manual view creation

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The system performs automatic view creation by having the data processing system itself infer and create views from raw data without requiring user intervention. The system analyzes data patterns, relationships, and semantics automatically to generate meaningful data views that users can then consume.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The system pre-processes and analyzes data in advance to build a data catalog with inferred relationships, semantics, and patterns before users need to query. This preliminary analysis enables rapid view creation when users submit queries, eliminating the need for manual exploration and view construction at query time.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If users manually update data views when schema changes occur, then views remain functional, but the process is error-prone and requires constant monitoring

Engineering Contradiction:

Improveaccuracy of data viewsVSAvoidcomplexity of maintaining data views

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system continuously monitors data sources and schema changes, and automatically detects when changes affect existing views. This feedback mechanism triggers automatic view updates that adapt to schema changes, data pattern changes, or relationship changes without user intervention, ensuring views remain accurate and functional.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system makes data views dynamic and adaptive rather than static. Views automatically adjust their structure, queries, and relationships based on current data patterns and schema changes. The system re-inferences view definitions from raw data whenever changes are detected, making the view maintenance process continuous and adaptive rather than manual and periodic.

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If users gain deep domain knowledge to understand data storage mechanisms, then they can create meaningful views, but this requires significant learning time and expertise

Engineering Contradiction:

Improveability to create meaningful viewsVSAvoidtime for learning domain knowledge

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

The system replaces the manual mechanical process of user analysis and view creation with an automated computational system. The system uses data analysis, pattern recognition, and inference algorithms to automatically understand data semantics and relationships, substituting user expertise with automated intelligence that can process and interpret data without human intervention.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Productivity

If the system stores relationships between data elements from multiple sources, then data views can be automatically populated, but the data catalog becomes more complex to maintain

Engineering Contradiction:

Improvespeed of view creationVSAvoidcomplexity of data catalog

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system segments the complex task of view creation into distinct components: data catalog building, relationship inference, view definition generation, and view population. The data catalog itself is segmented into separate data structures for storing data elements, relationships, semantics, and patterns. This segmentation allows each component to be maintained and processed independently, reducing overall system complexity.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS10635681B2Forming data responsive to a query

Publication Date: 2020.04.28 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

The present invention extends to methods, systems, and computer program products for forming data responsive to a query. Responding to a query can include selecting data views on top of data sources. A data catalog stores data elements from different data sources and stores links between data elements from the different data sources. A view can be selected for a query based on one or more of: classification of data included in the data catalog or prior usage of data included in the data catalog. A selected data view is populated with data elements from the data catalog using the links to determine that data elements spanning multiple data sources satisfy the query. The data catalog can be used to identify relationships between data elements from different data sources that would not otherwise be identified.