Columnar Timeseries Storage with Simple-8b Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing non-relational database formats, such as MongoDB, are not conducive for storing timeseries data efficiently, as they are based on document structures rather than time-based formats, leading to inefficiencies in storage and retrieval of large volumes of time-series data.
Innovation Solution
A database engine is configured to store timeseries events as documents within buckets, using columnar formats, compression techniques like delta and ZigZag encoding, and indexing based on time and metadata, enabling efficient storage and retrieval of timeseries data in non-relational databases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If timeseries data is stored in traditional non-relational database formats (document-based), then the database can maintain its flexible document structure, but storage efficiency and retrieval performance deteriorate due to the mismatch between document structure and time-based data organization
Solution Approach 1:
The patent segments timeseries data into discrete events with specific time stamps, organizing them in a structured format that separates temporal information from document content. This segmentation allows the database to maintain document flexibility while enabling efficient time-based indexing and querying, resolving the contradiction between structural adaptability and retrieval efficiency.
Solution Approach 2:
The patent introduces a temporal dimension to the traditional document structure by organizing data as time-stamped events within documents. This dimensional addition enables time-based indexing and range queries while preserving the flexible document format, allowing the system to achieve both document versatility and time-efficient retrieval.
2Quantity of substance
If compression techniques are applied to reduce storage requirements, then storage efficiency improves, but data processing complexity increases
Solution Approach 1:
The patent applies parameter changes by transforming floating-point values to integers through scaling and rounding operations. This parameter transformation enables the use of Simple-8b compression techniques on numeric data, significantly reducing storage requirements while maintaining a relatively simple processing pipeline that can be efficiently implemented in database operations.
3Measurement precision
If floating-point data is stored with high precision, then data accuracy is maintained, but storage space consumption increases
Solution Approach 1:
The patent changes the parameter representation from floating-point to integer format through scaling and rounding. This transformation maintains measurement precision by preserving the relative differences between values while enabling more compact storage using Simple-8b encoding, effectively resolving the trade-off between accuracy and storage space.
Solution Approach 2:
The patent creates a transformed copy of the floating-point data in integer format for storage purposes. This copying approach allows the original precision to be preserved in the transformed representation while enabling efficient compression, separating the concerns of precision maintenance from storage optimization.
4Ease of manufacture
If negative numbers are stored using two's complement format, then the representation is standard and simple, but storage efficiency decreases due to sign extension requirements
Solution Approach 1:
The patent inverts the traditional approach to negative number representation by using ZigZag encoding instead of two's complement. This inversion maps signed integers to unsigned integers in a way that preserves small absolute values in the least significant bits, eliminating sign extension requirements and improving storage efficiency while maintaining ease of decoding through a simple inverse operation.
Data Source
AI summary
A system is provided for storing, in a database, a plurality of timeseries represented by a plurality of respective documents events in a columnar format. The system further is adapted compress at least one of the values within the plurality of documents. According to some embodiments, the system stores the compressed values as a Simple-8b block and calculates the optimal Simple-8b selector. According to some embodiments, the system is adapted to determine a secondary index based on values within the bucket.