Storage System Index Management via Window Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage system index management solutions are inefficient and prone to errors, requiring users to develop special applications and failing to effectively manage indexes for streaming data storage systems, leading to performance issues.
Innovation Solution
A method for managing an index in a storage system by creating window index nodes based on events within a predetermined time window, adding them to the index when a threshold number of events are received, and using multiple computing resources to process events in parallel, thereby improving efficiency and reducing the need for custom index management applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If existing index management solutions are used, then users can store data in the storage system, but the index creation performance is poor and requires special applications to be developed
Solution Approach 1:
The storage system automatically creates and manages indexes for stored events without requiring users to develop special applications. The system self-services by integrating index creation functionality directly into the storage operations, eliminating the need for external index management tools while improving index creation performance through optimized internal processes
Solution Approach 2:
The patent combines data storage and index creation operations into a unified process. By merging these previously separate functions into a single integrated system, the storage system can simultaneously store events and create their corresponding indexes, reducing the need for separate special applications and improving overall efficiency
2Speed
If indexes are created for all events in streaming data storage system, then data access can be improved, but the system performance deteriorates due to inefficient index management
Solution Approach 1:
The patent applies different index management strategies to different portions of the data stream based on local characteristics. By analyzing event patterns and applying selective indexing to specific event types or time periods rather than uniformly indexing all events, the system improves data access speed for critical queries while maintaining overall system performance
Solution Approach 2:
The system dynamically adjusts index creation parameters such as indexing frequency, index granularity, and time window sizes based on data characteristics and query patterns. By changing these parameters adaptively, the system optimizes the balance between data access speed and overall system performance, avoiding the degradation caused by static inefficient index management
3Ease of operation
If index management is automated, then operation convenience is improved, but the system complexity increases
Solution Approach 1:
The storage system automatically performs index management tasks including index creation, maintenance, and optimization without user intervention. This self-service approach improves operation convenience by eliminating manual index management while the automated nature of these operations prevents significant increases in system complexity
Solution Approach 2:
The patent implements a universal index management mechanism that handles multiple event types and query patterns through a single integrated system. By creating a multi-functional index management subsystem that can adapt to different scenarios, the system improves ease of operation while avoiding the complexity that would result from multiple separate management mechanisms
Data Source
AI summary
An index of a storage system is managed. For example, events in a data stream to be stored are received. According to a predetermined length of a time window and occurrence times of the events, an event among the events that occurs within the time window is determined. Based on the event, a window index node is created including an index of the event. In response to determining that a current time point meets a threshold time point corresponding to the time window, the window index node is added to the index, and the threshold time point indicates that the number of received events that occur within the time window in the data stream reaches a threshold number. Thus, an index can be created in time for a large number of events entering the storage system. Further, the storage system can be queried and updated accurately and effectively.