Dual-Container Data Storage for Scalable Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data storage systems face challenges in balancing scalability and availability while being cumbersome to query, often requiring excessive resources and being inefficient in optimizing data retrieval processes.
Innovation Solution
A data storage system that utilizes two separate data storage containers, one optimized for sequentially changing data values and another for finite data values, allowing for efficient querying and load balancing across multiple data storage assets, with mechanisms for handling capacity overflow through 'spill' and 'split' operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional data storage systems use a single storage structure optimized for scalability, then the system can handle large data volumes, but query operations become cumbersome and inefficient
Solution Approach 1:
The patent divides the data storage system into two distinct containers: a sequential container for time-ordered data ingestion and a hashed container for efficient query operations. This segmentation allows each container to be optimized for its specific purpose, resolving the contradiction between scalable storage and efficient querying.
Solution Approach 2:
The sequential container acts as an intermediary between data sources and the hashed container. Data flows through the sequential container first, then gets redistributed to the hashed container for efficient querying. This intermediary structure enables both scalable ingestion and efficient query operations.
2Adaptability or versatility
If conventional systems configure storage to optimize scalability, then the system can grow to handle more data, but the system becomes overly complex
Solution Approach 1:
By separating the storage system into two specialized containers with clear division of labor, the patent achieves scalability without proportionally increasing complexity. Each container has a simple, well-defined structure optimized for its specific function.
Solution Approach 2:
Each container is designed with local quality optimized for its purpose: the sequential container has simple append operations for scalable ingestion, while the hashed container has optimized key-value structure for efficient querying. This localized optimization achieves overall system scalability without uniform complexity.
3Reliability
If conventional systems use a single storage container, then the structure is simple, but the system requires dramatically more storage assets to achieve high availability
Solution Approach 1:
The patent segments storage assets into two functional groups corresponding to the two containers. This segmentation allows for more efficient utilization of storage assets, achieving high availability without requiring dramatically more resources than a single-container system.
Solution Approach 2:
The dual-container structure provides multi-functionality: the sequential container handles data ingestion and time-based operations, while the hashed container handles efficient querying. This multi-functionality achieves high system availability without requiring excessive storage assets.
4Speed
If conventional systems optimize storage for query efficiency, then retrieval is faster, but the system becomes cumbersome to configure and operate
Solution Approach 1:
The hashed container is designed with local quality optimized for query efficiency, using hash-based indexing for fast retrieval. The sequential container maintains operational simplicity with append-only structure. This localized optimization achieves fast queries without making the entire system cumbersome to operate.
Solution Approach 2:
By segmenting the storage system into two containers with different optimization focuses, the patent achieves query efficiency in the hashed container while maintaining operational simplicity in the sequential container, avoiding the need to make the entire system cumbersome.
Data Source
AI summary
Various implementations of the invention store and retrieve data via a computing processor. The data is comprised of a plurality of data records with a sequentially changing data value and an other data value. For each data record in the plurality of data records, the computing processor causes the data record to be aggregated and stored with other data records in a first data storage container corresponding to a first data storage space based on the sequentially changing data value of the data record (i.e., sequential data space), and causes the data record to be aggregated and stored with other data records in a second data storage container corresponding to a second data storage space (i.e., finite data space). The data records in the second data storage container are stored based on the other data value. When a query is made against the data, the computing processor determines whether the query is best evaluated in the sequential data space or the finite data space. When the query is best evaluated in the sequential data space, the computing processor retrieves a first retrieved plurality of data records from the first data storage container. When the query is best evaluated in the finite data space, the computing processor retrieves a second retrieved plurality of data records from the second data storage container.