Result Caching Index for Object Storage Query Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Object storage systems face challenges in efficiently managing and analyzing disjoined, disparate, and malformed data, leading to performance issues and high costs due to the need for manual inspection and transformation processes, and existing caching techniques do not effectively scale to provide sub-second query responses for big data scenarios.
Innovation Solution
The implementation of a caching technique that generates an index based on source data objects in object storage, comprising a manifest, symbol file, and locality file, allowing for efficient data organization, compression, and query performance enhancement, enabling sub-second query responses and scalable performance for big data scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual inspection and transformation processes are used for data analysis, then data can be processed, but time consumption and costs increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-computing and caching query results before they are actually needed. The system identifies frequently accessed data patterns and pre-processes them into optimized formats, so when queries are executed, the results are already prepared and can be retrieved instantly, eliminating the need for time-consuming manual inspection and transformation at query time.
Solution Approach 2:
The patent uses copying by creating cached copies of query results and intermediate data processing states. Instead of repeatedly performing the same manual inspection and transformation operations, the system generates and stores optimized copies of the data in various formats, allowing rapid retrieval and reuse without re-executing the original time-consuming processing steps.
2Speed
If caching techniques are implemented to improve query performance, then response time decreases, but scalability to big data scenarios is limited
Solution Approach 1:
The patent applies segmentation by dividing large datasets into manageable chunks or partitions that can be independently cached and processed. The caching system is organized in a hierarchical or distributed manner, where data is segmented by query patterns, data types, or access frequency, allowing the system to scale to big data scenarios by processing and caching segments in parallel across multiple storage nodes.
Solution Approach 2:
The patent uses another dimension by introducing multiple dimensions to the caching strategy, such as caching at different levels of data granularity, organizing caches by multiple access patterns, or creating cached views from different perspectives of the same data. This multi-dimensional caching approach enables the system to handle diverse big data queries efficiently while maintaining sub-second response times for various data sizes and types.
3Ease of manufacture
If data is stored in disjoined and schema-less manner in object storage, then storage simplicity and scalability are improved, but data analysis complexity increases
Solution Approach 1:
The patent introduces an intermediary layer between the simple object storage and the complex data analysis requirements. This intermediary caching system automatically performs data validation, schema enforcement, and optimization transformations, translating the simple disjoined storage format into analysis-ready structured formats without requiring complex manual ETL processes. The intermediary handles the complexity internally while maintaining storage simplicity.
Solution Approach 2:
The patent applies parameter changes by dynamically transforming data parameters during the caching process. The system automatically adjusts data formats, schemas, and structures based on query requirements, converting disjoined object storage data into optimized analytical formats on-demand. This parameter transformation occurs automatically in the caching layer, reducing analysis complexity while preserving storage simplicity.
Data Source
AI summary
Apparatus, methods, and computer-readable media facilitating efficiently scaling result caching are disclosed herein. An example method includes generating an index based on a plurality of source data objects in an object storage system. The generated index comprises a manifest, at least one symbol file, and at least one locality file. The example method also includes receiving a search query for the plurality of source data objects stored in the object storage system, and querying the generated index based on the search query and a manifest root file of the manifest. Additionally, the example method includes generating a materialized view of a result set of the search query based on the querying of the generated index. The example method also includes storing a cached manifest file at the generated index, the cached manifest file mapping the search query to a segment of the generated index based on the result set.