HDX File Format for Time-Series Data Storage and Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current cloud-based object storage services face inefficiencies in storing and retrieving large volumes of time-series data, particularly for OLAP applications, due to slower read times compared to local storage, necessitating a new strategy for cost-effective and performant data management.
Innovation Solution
The implementation of a unique database file format, referred to as the HDX file format, which separates compute infrastructure indexing and information retrieval from remote storage, enabling efficient storage and retrieval of time-series columnar-based data in a cloud-based object store by using a manifest file, data file, and index file, allowing for real-time processing and scalable query peers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If time-series data is stored in cloud-based object storage services, then storage scalability and reliability are improved, but data retrieval speed and access performance deteriorate
Solution Approach 1:
The patent segments time-series data into time-based partitions and further divides each partition into multiple segments (e.g., 100 segments per partition). This segmentation enables parallel processing during retrieval operations, where multiple query peers can simultaneously access different segments, significantly improving retrieval speed while maintaining cloud storage scalability.
Solution Approach 2:
The patent implements pre-computation of segment metadata and maintains segment trees that organize data segments in advance. Query peers can perform binary search on pre-built segment trees to quickly locate relevant segments without scanning entire partitions, enabling fast retrieval from cloud storage.
2Ease of manufacture
If data is stored in remote cloud storage, then storage cost-effectiveness is improved, but access performance compared to local storage deteriorates
Solution Approach 1:
The patent implements local caching of frequently accessed segment metadata and hot data segments in the compute infrastructure. Query peers first check local cache before accessing cloud storage, providing fast access for frequently queried data while maintaining cost-effective cloud storage for the complete dataset.
Solution Approach 2:
The patent dynamically adjusts the amount of data cached locally based on query patterns and available memory resources. The system adapts its caching strategy to balance between cloud storage cost savings and local access performance based on real-time workload characteristics.
3Ease of operation
If compute infrastructure is integrated with remote storage, then data access simplicity is improved, but scalability of both compute and storage deteriorates
Solution Approach 1:
The patent introduces a catalog service as an intermediary between compute infrastructure and cloud storage. The catalog service maintains metadata about data partitions and segments, enabling query peers to efficiently locate and access data without direct compute-storage coupling. This intermediary layer enables independent scaling of compute and storage resources.
Solution Approach 2:
The patent creates virtual copies of data segments distributed across multiple query peers. Each query peer maintains local references (segment trees) to data segments in cloud storage, enabling parallel query processing without centralized compute bottlenecks and allowing independent scaling of compute and storage capacities.
Data Source
AI summary
Time-series columnar-based information is received and indexed in a compute infrastructure for cost-effective cloud-based object storage. The approach leverages a file format that enables highly-performant search and retrieval of the data stored in the cloud. In operation, an indexer receives the time-series information, indexes that information according to the file format, and forwards the indexed information for storage to the object store, where it is stored as a set of time-based partitions. A partition comprises a set of files, namely, a manifest file, a data file, and an index file. These files are structured as a compact instance of a set of raw unstructured data that comprises the given partition. Highly-performant information retrieval is enabled in response to a time-bounded query, because operations at a query peer (with respect to one or more partitions) are carried out in real-time during query processing and without requiring retrieval of the data file as a whole.