Vector Embedding Models for Relational Tables with Null Values

Solution Overview

Problem

Existing database technologies face challenges in effectively handling null values, as they improperly influence semantic vectors and hinder data imputation and similarity determination, leading to inaccurate record comparisons and vector embeddings.

Innovation Solution

The method involves generating semantic vectors using a neural network trained with database records that include null value tags, allowing for the identification of similar records to impute null values and reduce the impact of null values on vector embeddings by replacing them with actual values from similar records.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If null values are included in database records during vector embedding generation, then the complete data structure is preserved, but the semantic vectors become inaccurate and similarity determination is hindered

Engineering Contradiction:

Improveaccuracy of semantic vectorsVSAvoidmissing data in null values

Core Design Contradiction:

ReliabilityVSLoss of information

Solution Approach 1:

The system performs preliminary data imputation by identifying null values in the target record and replacing them with corresponding values from similar records before generating semantic vectors. This preliminary action ensures that the vector embedding process works with complete data, resolving the contradiction between preserving data structure and maintaining vector accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces similar records as intermediary sources to fill in null values. By querying the database for records with similar non-null attributes and using those as proxies for missing data, the system mediates between the incomplete target record and the requirement for complete input data in vector embedding.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of manufacture

If null values are used in database records, then data storage is simplified, but data imputation and similarity determination are improperly influenced

Engineering Contradiction:

Improvesimplicity of data storageVSAvoidprecision of similarity determination

Core Design Contradiction:

Ease of manufactureVSMeasurement precision

Solution Approach 1:

The system performs preliminary data imputation by identifying null values in the target record and replacing them with corresponding values from similar records before generating semantic vectors. This preliminary action ensures that the vector embedding process works with complete data, resolving the contradiction between preserving data structure and maintaining vector accuracy.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces similar records as intermediary sources to fill in null values. By querying the database for records with similar non-null attributes and using those as proxies for missing data, the system mediates between the incomplete target record and the requirement for complete input data in vector embedding.

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS11163761B2Vector embedding models for relational tables with null or equivalent values

Publication Date: 2021.11.02 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

Structured and semi-structured databases and files are processed using natural language processing techniques to impute data for null value tokens in database records from other records that have non-null values for the same attributes. Vector embedding techniques are used, including, in some cases, appropriately tagging null value tokens to reduce or eliminate their undue impact on semantic vectors generating using a neural network.