Hierarchical Indexing for Big Data Queries and Responsive GUIs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing systems struggle to efficiently handle the complexity and volume of big data sets, leading to sluggish responses in graphical user interfaces and increased burden on databases due to inefficient data indexing and display of large, complex query results.
Innovation Solution
Implementing a database index with a hierarchy of interlinked data structures and a hash table that associates dates with data structures, allowing efficient querying and display of big data sets by optimizing search requests and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional data processing is used, then data storage capacity cost has decreased, but processing efficiency and query response speed deteriorate due to overwhelming data volume and complexity
Solution Approach 1:
The patent segments the data index into a hierarchy of interlinked data structures with multiple levels. The first level contains first data structures organized by date, the second level contains second data structures organized by entity, and the third level contains third data structures organized by attribute. This segmentation allows efficient navigation through large datasets by breaking down the search into manageable hierarchical steps, resolving the contradiction between storing large volumes of data and processing it efficiently.
Solution Approach 2:
The patent implements a nested hierarchical index structure where data structures are contained within one another across multiple levels. The first data structures are nested within the index, which contain pointers to second data structures, which in turn contain pointers to third data structures. This nesting enables compact storage of large datasets while maintaining efficient access patterns through the hierarchical relationships, addressing both storage capacity and processing efficiency requirements.
2Device complexity
If data is not indexed according to common query patterns, then data storage is simplified, but query response time increases due to inefficient search through large datasets
Solution Approach 1:
The patent performs preliminary organization of data into a hierarchical index structure before queries are executed. The data is pre-sorted and organized by date at the first level, by entity at the second level, and by attribute at the third level. This preliminary action enables queries to navigate the pre-organized structure efficiently rather than searching through unorganized data, significantly reducing query response time while maintaining a manageable indexing structure.
Solution Approach 2:
The patent creates a dynamic hierarchical index that can adapt to different query patterns. The hierarchy allows flexible navigation through multiple levels depending on the query requirements, enabling the system to optimize search paths based on common query patterns. This dynamic structure resolves the contradiction by providing a flexible indexing scheme that improves query performance without overly complicating the storage structure.
3Loss of time
If large query results are displayed immediately, then user feedback is provided quickly, but graphical user interface responsiveness deteriorates due to processing and display time
Solution Approach 1:
The patent implements periodic display of query results in batches rather than attempting to display all results at once. The system processes and displays results in periodic intervals, allowing the graphical user interface to remain responsive while gradually presenting large amounts of data. This periodic action resolves the contradiction by providing user feedback in manageable increments that maintain interface responsiveness even when dealing with large datasets.
Data Source
AI summary
Disclosed herein are an apparatus, non-transitory computer readable medium, and method for querying big data and displaying the results of the query. A multilevel hierarchy of data structures is associated with a particular date to reduce the linearity of the search. The client receiving the query results stores the results in a linked list with a record format consistent with the resulting records. The client also periodically refreshes a graphical user interface with the query results until all the results are received.