Inverted Indexes for Analytics Query Acceleration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional software tools struggle to efficiently manage and analyze large, complex data sets known as 'big data' due to difficulties in storage, processing, and querying, often requiring massive parallel processing and resulting in performance vs. memory tradeoffs.
Innovation Solution
A data-analysis framework that generates and manages data segments using both real-time and offline processing, employing forward indexes for compression and inverted indexes to accelerate analytics queries, facilitating timely updates and efficient querying of large data sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compressed data is used to reduce storage space, then memory consumption is reduced, but query latency increases
Solution Approach 1:
The patent divides the data storage system into multiple data segments, each with its own inverted index. This segmentation allows the system to manage large datasets in manageable chunks, enabling efficient querying of specific segments without needing to decompress entire datasets, thus balancing storage efficiency with query performance.
Solution Approach 2:
The patent pre-computes and maintains inverted indexes for each data segment before queries are executed. These indexes are built in advance during data ingestion or ETL processes, allowing the query engine to quickly locate relevant data without performing full scans or decompression operations at query time, thereby reducing query latency while maintaining compressed storage.
2Ease of manufacture
If conventional software tools are used to manage big data, then implementation is simple, but processing efficiency and query performance deteriorate
Solution Approach 1:
The patent introduces an inverted index as an intermediary data structure between the compressed data segments and the query engine. This intermediary layer enables efficient data retrieval by providing direct access paths to relevant data without requiring full decompression or sequential scanning, thus significantly improving query performance while maintaining the simplicity of using standard query tools.
Solution Approach 2:
The patent changes the organizational parameters of data storage by implementing columnar storage formats and inverted indexes rather than traditional row-based storage. This parameter change enables more efficient filtering, aggregation, and querying operations on large datasets, dramatically improving processing efficiency while remaining compatible with existing analytics tools through standardized interfaces.
3Speed
If data is kept uncompressed for faster querying, then query speed improves, but memory consumption increases
Solution Approach 1:
The patent applies different quality characteristics to different parts of the data system: data segments remain compressed for storage efficiency, while inverted indexes maintain uncompressed or less-compressed structures for fast lookup. This local differentiation allows the system to optimize both storage and query performance simultaneously by applying the appropriate data format to each component based on its specific requirements.
Data Source
AI summary
The disclosed embodiments provide a system that processes data. During operation, the system obtains a set of records, wherein each of the records comprises one or more metrics and at least one dimension associated with the one or more metrics. Next, the system creates, in a data segment comprising the records, an inverted index for a column in the records based on a cardinality of the column. Finally, the system compresses the inverted index based on a jump value associated with record identifiers in the column.