Columnar Time-Series Database Engine Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing non-relational database formats, such as NoSQL databases, are not conducive for efficiently storing and processing time-series data due to their document-based structures, which lead to inefficiencies in storage and retrieval of large volumes of time-stamped data.
Innovation Solution
A database engine that stores time-series events as documents in a columnar format, using techniques like delta compression, ZigZag encoding, and Simple-8b encoding to efficiently compress and index data, allowing for efficient storage and querying of time-series data in a non-relational database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If timeseries data is stored in traditional non-relational database formats with document-based structures, then the database can maintain flexibility and ease of operation, but storage efficiency and query performance deteriorate when handling large volumes of time-stamped data
Solution Approach 1:
The patent segments timeseries data into discrete events organized in columnar format within buckets, separating the data structure from the document model to enable efficient storage and querying while maintaining operational flexibility
Solution Approach 2:
The patent introduces a columnar organization dimension to the traditional document-based storage, arranging data in columns rather than nested documents, which enables efficient range queries and temporal analysis without sacrificing the flexibility of the document model
2Ease of operation
If timeseries data is stored in traditional non-relational database formats with document-based structures, then the database can maintain flexibility and ease of operation, but storage size increases and storage efficiency deteriorates
Solution Approach 1:
The patent merges multiple document representations into a single columnar structure where identical fields across multiple documents are stored once, eliminating redundancy and reducing overall storage size while preserving data accessibility
Solution Approach 2:
The patent changes the fundamental parameter of data organization from document-oriented to columnar format, transforming how data is structured and stored to achieve compact representation without losing the flexibility of the original document model
3Measurement precision
If floating point values are stored with full precision, then measurement precision is maintained, but storage size increases and compression efficiency deteriorates
Solution Approach 1:
The patent changes the precision parameter of floating-point values by rounding to a specified number of decimal places, reducing storage requirements while maintaining sufficient measurement precision for the application
4Reliability
If data is stored in uncompressed format, then data integrity and accessibility are maintained, but storage size increases and retrieval efficiency deteriorates
Solution Approach 1:
The patent segments data into compressible units (columns and buckets) that can be efficiently compressed while maintaining integrity, enabling both compact storage and fast retrieval through selective decompression of only the needed portions
Data Source
AI summary
A system is provided for storing, in a database, a plurality of timeseries represented by a plurality of respective documents events in a columnar format. The system further is adapted compress at least one of the values within the plurality of documents. According to some embodiments, the system stores the compressed values as a Simple-8b block and calculates the optimal Simple-8b selector. According to some embodiments, the system is adapted to determine a secondary index based on values within the bucket.