Cloud Data Retrieval Using Flat N-Gram Indexes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data lakes face challenges in storing massive amounts of data inexpensively while ensuring quick data retrieval, particularly for interactive user interfaces and real-time analytics, due to high latencies and inefficient indexing methods like tree-based structures, which result in slow search times for substring matches.
Innovation Solution
The system stores electronic data as indexed records in a flat data structure, using n-grams for efficient substring matching, and leverages serverless technologies like AWS Lambda and S3 for cost-effective and fast data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Weight of stationary object
If traditional storage systems are used for cloud-based data lakes, then data can be stored at low cost with high capacity, but data retrieval time becomes excessively slow due to high latency
Solution Approach 1:
The patent segments the data retrieval process by separating hot data (frequently accessed) from cold data (infrequently accessed) and storing them in different locations with different access characteristics. This allows the system to maintain low-cost storage for cold data while providing fast access for hot data, resolving the contradiction between storage cost and retrieval speed.
Solution Approach 2:
The patent implements preliminary action by pre-processing and indexing data before it needs to be queried. Data is transformed into various formats (parquet, avro, json) and indexed in advance, so when a query arrives, the system can quickly locate and retrieve relevant data without performing heavy processing during the actual retrieval operation.
2Speed
If complex indexing structures are implemented to speed up substring searches, then search performance improves, but system complexity and resource consumption increase significantly
Solution Approach 1:
The patent creates simplified copies of the data in pre-processed formats (parquet, avro, json) and maintains inverted indexes that map search terms to data locations. These copies use straightforward data structures rather than complex hierarchical indexes, enabling fast substring searches while keeping the system manageable and resource-efficient.
Solution Approach 2:
The patent changes the parameters of data storage by transforming data into different formats optimized for specific query types. By converting data to columnar formats like parquet and maintaining inverted indexes with term-document mappings, the system achieves fast search performance without requiring complex multi-dimensional indexing structures.
3Weight of stationary object
If data is stored in raw format in cloud storage, then storage capacity is maximized and cost is minimized, but data retrieval and processing become extremely slow
Solution Approach 1:
The patent segments data into different storage formats based on access patterns and query requirements. Raw data is stored in cloud storage for cost-effective bulk storage, while processed data in parquet, avro, and json formats is maintained for fast querying. This segmentation allows the system to leverage both low-cost storage and high-performance processing simultaneously.
Solution Approach 2:
The system performs preliminary data processing and transformation when data is ingested, converting raw data into multiple formatted versions and creating indexes in advance. This pre-processing work is done during data ingestion rather than during query execution, enabling fast data processing throughput when queries are submitted without incurring additional processing costs at query time.
Data Source
AI summary
A system and method are provided that store electronic data describing events that have occurred in a computing system, index the electronic data to create indexed data records; and store the indexed data records in computer memory as part of a flat data structure.