Split Retrieval Structures for Fast APM Segment Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for storing and querying application performance monitoring (APM) data in segmented databases are inefficient, leading to long download times and high data transfer costs due to the large size of data segments and the need to load multiple indexes, especially when using object storage like AWS S3.
Innovation Solution
A computer-implemented method that organizes data records into batches with batch identifiers, tokenizes key-value pairs, and compresses data records, using a data structure with a retrieval section for tokens and a data section for compressed records, allowing fast and space-efficient storage and querying.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in a segmented database with multiple indexes for fast retrieval, then querying speed is improved, but data transfer time and costs increase due to downloading multiple indexes from object storage
Solution Approach 1:
The patent segments the database into multiple data segments, where each segment contains a subset of the data records. This segmentation allows the system to query only the relevant segments rather than loading all indexes from object storage, thereby reducing data transfer time while maintaining querying speed through targeted segment access.
Solution Approach 2:
The patent extracts only the necessary data segments and their corresponding indexes into memory based on the query requirements, rather than loading all indexes from object storage. This extraction approach minimizes data transfer time and costs while preserving fast querying capability for the relevant data subset.
2Productivity
If all indexes are loaded into memory for fast querying, then query performance is improved, but memory usage and data transfer costs increase
Solution Approach 1:
The patent divides the database into multiple segments and loads only the relevant segment indexes into memory during query operations. This segmentation strategy maintains high query performance for accessed data while significantly reducing memory usage compared to loading all indexes simultaneously.
Solution Approach 2:
The patent implements partial loading of indexes into memory based on query needs, rather than loading all indexes. This partial action approach achieves sufficient query performance for the required data subset while avoiding the excessive memory consumption and data transfer costs of loading complete indexes.
3Quantity of substance
If data is compressed to reduce storage size, then storage efficiency is improved, but decompression time during querying increases
Solution Approach 1:
The patent performs data compression during the data ingestion phase and stores the compressed segments. When querying, the system decompresses only the specific segments that are needed, rather than decompressing all data. This preliminary compression approach maintains storage efficiency while minimizing decompression time during query operations.
Data Source
AI summary
A computer-implemented method is presented for storing and querying records from a data segment. The objective of the disclosure is to find computer-implemented methods for storing and querying records from a data segment that are fast and space efficient storing and querying of APM data stored in a segmented database.