Distributed Index Query Mapping for Scalable Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As businesses generate large amounts of data, their indices grow rapidly, leading to capacity issues and fragmentation, requiring efficient management of distributed index data to support data retrieval, reorganization, and maintenance, especially during business changes like mergers or end-of-life scenarios.
Innovation Solution
A system and method for managing distributed index data by mapping queries to logically distributed indices using predefined rules, allowing for scalable indexing, reorganization, and archiving, while maintaining an audit trail and supporting Information Lifecycle Management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single index is used to store all data objects, then data retrieval can be performed using a generic index, but the index capacity is limited and cannot accommodate rapidly growing data volumes
Solution Approach 1:
The patent divides a single large index into multiple smaller distributed indices, each capable of being stored and managed independently. This segmentation allows the system to handle larger total data volumes by distributing index entries across multiple indices, while each individual index maintains manageable size and complexity.
2Adaptability or versatility
If data is reorganized due to business changes such as mergers or end-of-life scenarios, then the data structure can adapt to new requirements, but the complete dataset must be scanned to identify relevant subsets
Solution Approach 1:
The patent assigns different characteristics to different distributed indices based on their intended use and data type. Each index can be optimized for specific query patterns or business domains, allowing targeted reorganization of only relevant index subsets during business changes rather than scanning the entire dataset.
3Loss of substance
If index data is removed for end-of-life scenarios, then unnecessary data can be eliminated, but the index becomes fragmented
Solution Approach 1:
The patent extracts removed index entries from the active distributed indices and places them in a separate archival index. This extraction mechanism allows end-of-life data to be eliminated from the main index structure, preventing fragmentation while maintaining the ability to retrieve archived data when needed.
4Productivity
If multiple distributed indices are used to handle large data volumes, then index capacity increases and scalability improves, but query mapping and system complexity increase
Solution Approach 1:
The patent introduces a mapping table as an intermediary structure that stores the relationship between data objects and their corresponding distributed indices. This mapping table simplifies query processing by providing direct lookup paths to relevant indices, reducing the complexity of managing multiple distributed indices while maintaining high retrieval efficiency.
Data Source
AI summary
This disclosure provides various embodiments of systems, methods, and software for managing distributed index data. For example, in one embodiment, software for managing distributed index data may identify a query for one or more data objects from a data repository using a generic index, map the query to one of a plurality of logically distributed indices according to at least one rule, and execute the mapped query using the particular distributed index to retrieve the one or more data objects. Each distributed index may be in a disparate index repository that is on a disparate storage device. In some aspects, the one or more rules may include a plurality of business rules, as well as at least one technical rule, which may be stored in a rules repository. In further aspects, an administration view may be presented to a client such that one of the business rules is customized.