Tensor Dataset Querying with TQL for Large-Scale Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional machine-learning repositories, such as traditional relational databases, are not well designed for applications like natural language processing, audio processing, and computer vision, leading to poor computational performance and impracticality for large-scale, tensor-based data.
Innovation Solution
The use of a Tensor Query Language (TQL) to efficiently store, access, and process large-scale machine-learning data in the form of tensors, allowing for rapid streaming and integration with deep learning frameworks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional relational databases are used to store machine-learning data, then data infrastructure for analytical workloads is provided, but computational performance deteriorates for tensor-based data
Solution Approach 1:
The patent changes the fundamental parameter of data representation from relational tables to tensor format. The system stores machine-learning data as tensors with explicit dimensionality information, allowing direct compatibility with deep learning frameworks. This parameter change enables the same infrastructure to handle both analytical workloads and tensor-based computations efficiently, resolving the performance deterioration issue.
Solution Approach 2:
The patent creates a universal data storage system that can handle multiple data types and workloads simultaneously. The tensor-based storage system serves both traditional analytical queries and deep learning operations, eliminating the need for separate systems. This multi-functionality allows the infrastructure to maintain reliability for analytical workloads while achieving high computational performance for tensor operations.
2Reliability
If relational databases are used for storing data, then ACID transactions and data consistency are ensured, but adaptability for non-tabular data deteriorates
Solution Approach 1:
The patent fundamentally changes the data representation parameter from tabular to tensor format, enabling native support for non-tabular data types such as images, audio, and video. The system maintains ACID transaction properties while storing data as tensors with preserved dimensional information, allowing the same storage infrastructure to handle diverse data formats without sacrificing consistency or adaptability.
Solution Approach 2:
The patent creates a composite data storage system that combines the reliability of ACID transactions with the versatility of tensor-based storage. The system integrates transaction management mechanisms with tensor data structures, enabling simultaneous support for both structured analytical queries and unstructured machine-learning data. This composite approach resolves the adaptability limitation of traditional relational databases.
3Quantity of substance
If conventional data lakes are used, then storage of large datasets is enabled, but processing efficiency for tensor data deteriorates
Solution Approach 1:
The patent changes the data organization parameter from conventional data lake structures to tensor-based storage. This allows the system to maintain the ability to store large datasets while dramatically improving processing efficiency. The tensor format with embedded dimensionality information enables direct consumption by deep learning frameworks, eliminating the need for complex data transformation pipelines and boosting processing throughput.
Solution Approach 2:
The patent performs preliminary organization of data into tensor format during the storage phase, rather than requiring transformation during processing. This preliminary action of structuring data according to its intended use case (machine learning) enables efficient processing from the outset, resolving the efficiency deterioration that occurs when conventional data lakes are used for tensor-based workloads.
Data Source
AI summary
Systems and methods for executing queries on tensor datasets are disclosed. A system can identify a query for a multi-dimensional sample dataset. Each sample of the multi-dimensional sample dataset can include one or more tensors. Each tensor of the one or more tensors can be associated with a respective identifier that is common to each sample of the multi-dimensional sample dataset. The query specifying a first identifier of a first tensor of the multi-dimensional sample dataset and a first range of a first dimension of the first tensor, or one or more operations such as sampling, grouping, ungrouping, or transformation operations, to perform on the first tensor of the multi-dimensional sample dataset. The system can parse the query, and execute the query to generate query results. The system can provide the query results as output.