Hybrid Neural Network Encoder for Dynamic Schema Labeling

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional data analytics systems are rigid, inefficient, and inaccurate in extracting, transforming, and loading large digital data volumes due to their reliance on specific digital formats and labeling schemes, which limits flexibility and fails to adapt to new or modified labels without retraining.

Innovation Solution

The use of hybrid neural networks to dynamically determine schema labels for digital data repositories, employing a ranking loss and neural network encoders to generate vector embeddings for columns, allowing for accurate and efficient mapping to schema labels, including new or unseen labels without retraining.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If rule-based schema matching is used to align data to native labeling schemes, then data can be matched to existing schemas, but the system cannot flexibly adapt to modified or new labels without manual rule updates

Engineering Contradiction:

Improveadaptability to new labeling schemesVSAvoidmanual rule maintenance complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent replaces manual rule-based schema matching with a machine learning model that automatically learns and adapts to labeling schemes. The model is trained on historical matching records and uses this training to automatically match columns to schema labels, eliminating the need for manual rule creation and updates when labeling schemes change.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes from static rule-based matching to dynamic learned parameter matching. The machine learning model learns parameters from training data and can adapt to new labeling schemes by retraining on new historical matching records, allowing the system to flexibly accommodate schema modifications without manual intervention.

Inventive Principle:

Principle #35Parameter changes

2Extent of automation

If classifiers are used to match data to labeling schemes, then automated matching is achieved, but the system requires significant retraining resources when labels are added or modified

Engineering Contradiction:

Improveautomated schema matchingVSAvoidprocessing power for retraining

Core Design Contradiction:

Extent of automationVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary training using historical matching records before deployment. Once trained, the model can automatically match schemas without requiring retraining for each new label or modification, reducing the frequency and resource cost of retraining operations while maintaining automated matching capability.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If handcrafted rules are used for schema matching, then matching logic can be explicitly defined, but excessive effort and resources are required to build and maintain rules as schemas grow

Engineering Contradiction:

Improvematching accuracyVSAvoidtime to build and maintain rules

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces manual rule crafting with automated machine learning-based schema matching. The model learns matching patterns from historical records and automatically applies them to match columns with schema labels, eliminating the time-consuming process of manually creating and maintaining rules while maintaining or improving matching accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system enables self-service schema matching where the machine learning model automatically learns from historical matching records and performs matching without human intervention. This eliminates the need for data engineers to manually craft and maintain rules, significantly reducing the time and effort required to build and maintain schema matching as systems grow.

Inventive Principle:

Principle #25Self-service

4Ease of operation

If user input is required for schema matching, then customization is possible, but significant user interactions are needed resulting in inconsistent mappings across administrator devices

Engineering Contradiction:

Improveuser customization capabilityVSAvoidmapping consistency

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The patent replaces manual user-based schema matching with automated machine learning-based matching. The model processes data consistently across all administrator devices using the same learned patterns from historical records, eliminating human variability and ensuring consistent mappings while removing the need for user interactions.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11562234B2Dynamically determining schema labels using a hybrid neural network encoder

Publication Date: 2023.01.24 ADOBE INC

AI summary

Systems, methods, and non-transitory computer-readable media are disclosed for dynamically determining schema labels for columns regardless of information availability within the columns. For example, the disclosed systems can identify a column that contains an arbitrary amount of information (e.g., a header-only column, a cell-only column, or a whole column). Additionally, the disclosed systems can generate a vector embedding for an arbitrary input column by selectively using a header neural network and/or a cell neural network based on whether the column includes a header label and/or whether the column includes a populated column cell. Furthermore, the disclosed systems can compare the column vector embedding to schema vector embeddings of candidate schema labels in a d-dimensional space to determine a schema label for the column.