Sub-document Indexing for High Efficiency Data Querying
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database systems face challenges in efficiently storing and querying large amounts of data due to limited storage capacity and redundant hierarchies, leading to increased response times and inefficient resource usage.
Innovation Solution
A system that parses input documents into sub-documents, generates unique IDs and metadata for each sub-document, indexes them in an index database, and stores the input document in a document database, allowing for efficient querying by determining the relevant sub-documents based on query parameters and using block join queries to retrieve specific data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If data is stored in existing database systems with hierarchical structures, then data can be organized and managed, but querying specific data requires accessing higher level hierarchies resulting in retrieval of large amounts of extraneous information and longer response times
Solution Approach 1:
The patent segments the hierarchical data structure into flat, granular data elements that can be independently queried. Instead of requiring queries to traverse hierarchical levels, the system breaks down hierarchical data into discrete, queryable units with unique identifiers, allowing direct access to specific data elements without retrieving extraneous information from parent or sibling nodes.
Solution Approach 2:
The patent extracts the essential data elements from their hierarchical context and stores them as independent, flat records. By extracting data elements from the hierarchical structure and assigning them unique identifiers and pathways, the system enables direct querying of specific data without requiring access to the hierarchical structure, thereby eliminating retrieval of extraneous information.
2Quantity of substance
If database systems store large amounts of data, then comprehensive information is available, but storage capacity limitations prevent scaling to handle increasing data volumes
Solution Approach 1:
The patent transitions from a hierarchical data model to a flat, dimensionless data structure. By eliminating hierarchical levels and storing data as flat records with unique identifiers and pathways, the system removes the structural constraints that limit scaling capacity, allowing the database to horizontally scale to handle increasing data volumes without the overhead of hierarchical management.
3Productivity
If queries access data through hierarchical structures, then data organization is maintained, but computational resources are used inefficiently due to processing of unnecessary data
Solution Approach 1:
The patent performs preliminary actions by pre-processing hierarchical data into flat, query-optimized records during data ingestion. Unique identifiers and pathways are assigned to data elements in advance, and the flat structure is prepared for efficient querying. This preliminary transformation eliminates the need for computational traversal of hierarchical structures during query execution, significantly improving query processing efficiency and reducing computational resource usage.
Data Source
AI summary
The embodiments described herein enable the efficient storage and searching of large amounts of data to retrieve desired data with high accuracy and fast response time. The embodiments include for example parsing an input document containing input data into multiple sub-documents based on the structure of the input document and generating for each sub-document, a unique ID, pathway, and metadata. The embodiments also include storage of the ID and pathway for each sub-document in an index database and storage of the input document itself in a separate database. When a query for information from the input document is received, the ID and pathway of the sub document containing the queried information are identified and used to locate the queried information from the input document.