Global Index Structure for Distributed Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The rapid growth of data worldwide, particularly in services like Microsoft's Azure Data Lake Store, poses challenges in real-time processing and querying due to the need to focus on finer details without losing original data, with high monetary costs associated with downtime, necessitating efficient data indexing and retrieval methods.
Innovation Solution
A global index structure is generated by agents executing on different data sources, which pre-process and index data as index nodes, merged by backend servers to create a hierarchical index structure, reducing the number of nodes searched and processing cycles required for query results, thereby decreasing read access time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a global index structure is created by merging index nodes from multiple data sources, then data retrieval speed is improved, but the complexity of maintaining and merging indexes increases
Solution Approach 1:
The patent divides the global index into distributed index nodes across multiple data sources. Each data source maintains its own index nodes independently, and the global index is formed by merging these segmented index nodes. This segmentation allows parallel maintenance and reduces the complexity of managing a single centralized index for all data.
Solution Approach 2:
Index nodes are pre-computed and maintained at each data source before queries are executed. This preliminary indexing action allows the system to quickly merge and traverse index nodes during query operations without performing complex indexing tasks in real-time, thereby improving retrieval speed while managing maintenance complexity through advance preparation.
2Reliability
If all index nodes are searched to ensure complete data retrieval, then data completeness is improved, but the processing time and computational resources increase
Solution Approach 1:
The patent extracts and maintains metadata information (such as min/max values, partitions, and index node characteristics) alongside the index nodes. This extracted metadata allows the query system to determine which index nodes are relevant to a specific query without searching all index nodes, ensuring data completeness while significantly reducing processing time by excluding irrelevant nodes from the search.
3Productivity
If data is pre-processed and indexed at data sources, then query performance is improved, but the computational resources required at data sources increase
Solution Approach 1:
The patent implements local indexing where each data source processes and indexes only its own data locally, creating index nodes specific to its data characteristics. This local quality approach allows data sources to optimize their indexing strategies according to their specific data types and access patterns, improving overall query performance while distributing computational resources efficiently across the system rather than concentrating all processing demands.
Data Source
AI summary
Methods, systems, apparatuses, and computer program products are directed to the generation of a global index structure. Agents executing on different data sources locally pre-process (e.g., format, filter, compress, encode, serialize etc.) data generated thereby and index such data. The agents also manage the resources thereof to perform the pre-processing and indexing operations. Each index generated by an agent is formatted as a plurality of index nodes. The index nodes and pre-processed data are provided to backend server(s) that maintain the global index structure and store the data in a globally distributed file system, which aid in unexpected disaster recovery. The backend server(s) generate the global index structure based on the index nodes. As new index nodes are received by the backend servers, the backend servers merge the newly-received index nodes with the global index structure. Global index structure traversal techniques for retrieving search keys are also described herein.