Distributed Representation Vectors for Text and Quantitative Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language processing systems struggle to effectively compare and determine similarity between diverse data elements, including natural language text and quantitative data, as they often rely on linguistic context and fail to integrate various data types within a common framework.
Innovation Solution
A method that generates distributed representation vectors for data elements by cleaning and parsing datasets, organizing them into records, compiling dictionaries, creating training pairs, and using an artificial neural network to compute embeddings for all types of data, including natural language text and quantitative data, allowing for vector-based comparison across different data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing natural language processing systems are used to compare diverse data elements, then linguistic context can be utilized, but the systems fail to integrate various data types (natural language text and quantitative data) within a common framework
Solution Approach 1:
The patent creates a universal embedding framework that handles multiple data types (natural language text, quantitative data, and other structured data) through a single common framework. The system generates distributed representation vectors for diverse data elements using unified processing steps including cleaning, parsing, organizing into records, compiling dictionaries, creating training pairs, and computing embeddings, thereby achieving multi-functionality without requiring separate systems for each data type
Solution Approach 2:
The patent transforms diverse data types into a common parameter space by converting all data elements into distributed representation vectors. This parameter transformation allows quantitative data, text, and other data types to be represented in the same vector space, enabling direct comparison and similarity determination across different data types through mathematical operations on their embeddings
2Adaptability or versatility
If traditional embedding methods are used for natural language text, then linguistic context is preserved, but the methods cannot effectively process quantitative data and other non-text data types
Solution Approach 1:
The system employs a universal processing pipeline that treats natural language text, quantitative data, and other data types uniformly. Each data element undergoes the same steps: cleaning, parsing to identify known classes, extraction, organizing into records, dictionary compilation, training pair creation, and embedding computation. This universal approach enables effective processing of all data types while maintaining measurement precision through consistent transformation into comparable vector representations
Solution Approach 2:
The patent introduces distributed representation vectors as an intermediary that bridges different data types. By converting quantitative data, text, and other data elements into a common vector space through neural network-based embedding, the system creates a mediator representation that enables accurate similarity determination across diverse data types while preserving the essential characteristics of each original data type
3Reliability
If separate processing methods are used for different data types, then each data type can be handled with specialized techniques, but computational complexity increases for multi-class scenarios
Solution Approach 1:
The patent merges the processing of different data types into a single unified framework. Instead of maintaining separate processing pipelines for natural language text, quantitative data, and other data types, the system combines them all into one common embedding generation process. The unified pipeline processes all data elements through identical steps (cleaning, parsing, organizing, dictionary compilation, training pair creation, and embedding computation), thereby reducing computational complexity while maintaining reliable processing quality for each data type
Data Source
AI summary
A method, computer program product and computer system is disclosed that generates a set of distributed representation vectors from a dataset of textual and non-text data. In one method, a computer system receives a dataset, cleans the received dataset, parses the cleaned dataset to identify known classes of data, extracts data elements from the dataset based on the known classes of data, organizes the extracted data elements into one or more records, compiles a dictionary of unique data elements and associated codes from the one or more records, creates a set of training pairs using permutations of the codes that correspond to data elements within each record, and computes a distributed representation vector for each of the data elements in the dictionary using the set of training pairs.