Tensor Query Language Vector Search Data Lake
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine-learning repositories, such as traditional relational databases, are not well-suited for applications like natural language processing, audio processing, and computer vision, leading to poor computational performance and challenges in efficient dataset vector searching, especially for large-scale, tensor-based data.
Innovation Solution
The system employs Tensor Query Language (TQL) to enable efficient vector searching by performing similarity comparisons between query embeddings and dataset embeddings, allowing for the generation of specialized output datasets for training machine-learning models, using processors to execute queries and provide output datasets based on Euclidean distance or cosine similarity calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional relational databases are used to store machine-learning datasets, then data infrastructure for analytical workloads is provided, but computational performance deteriorates and vector searching becomes inefficient for large-scale tensor-based data
Solution Approach 1:
The patent introduces a specialized data lake infrastructure as an intermediary between traditional relational databases and machine-learning workloads. This data lake uses a columnar storage format optimized for tensor operations, enabling efficient vector searching and embeddings operations while maintaining compatibility with existing data pipelines. The data lake acts as a mediator that translates traditional SQL queries into optimized tensor operations, resolving the contradiction between maintaining simple data infrastructure and achieving high computational performance for ML workloads.
2Measurement precision
If traditional relational databases are used, then data storage is achieved, but dataset vector searching efficiency deteriorates for embeddings and tensors
Solution Approach 1:
The patent transforms the storage parameters from traditional row-based relational database format to columnar storage format optimized for tensor operations. This parameter change enables efficient computation of Euclidean distances and cosine similarities for embeddings. The columnar storage allows for vectorized operations and better memory access patterns, dramatically improving both the accuracy and speed of vector searching for large-scale tensor-based machine-learning datasets.
3Productivity
If conventional machine-learning repositories are used, then data infrastructure is provided, but efficiency in generating specialized output datasets deteriorates
Solution Approach 1:
The patent implements preliminary action by pre-processing and organizing data into the optimized data lake format during the data ingestion phase. This preliminary organization into columnar storage with proper tensor formatting enables rapid generation of specialized output datasets later without requiring complex processing steps. The data is prepared in advance with appropriate indexing and metadata, making subsequent dataset generation operations simple and efficient.
Data Source
AI summary
Systems and methods for implementing tensor query-based vector search operations for multi-dimensional sample datasets of tensors are disclosed. The solution can utilize one or more processors coupled to memory to identify a query for a multi-dimensional sample dataset. The query can indicate an operation to search embeddings in the plurality of tensors of a plurality of samples of the dataset. Each sample can have a respective tensor of the plurality of tensors comprising one or more embeddings of the respective sample. The one or more processors can execute the query to generate an output dataset comprising a subset of samples of the plurality of samples. The subset of samples can be identified based on the operation and the respective one or more embeddings of each tensor of the subset of samples. The one or more processors can provide the output dataset.