Intra-document Index for XQuery XML Stream Parsing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current XML parsing techniques for XQuery queries over streams of XML data consume excessive memory and latency due to processing all document pieces, regardless of relevance to the query, which is inefficient and hampers performance.
Innovation Solution
The method involves creating an intra-document index that allows selective skipping of irrelevant document portions based on query instructions, using the index to determine where to resume scanning, and saving relevant categories into a buffer, thereby reducing parsing time without altering the original document format.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If conventional parsing techniques process all document pieces, then complete document coverage is achieved, but parsing time and memory consumption increase significantly
Solution Approach 1:
The patent extracts only the relevant portions of the XML document that match the query criteria, using an index to identify and skip irrelevant sections. This extraction approach reduces parsing time by avoiding processing of unnecessary document pieces while maintaining complete coverage of relevant data.
Solution Approach 2:
The patent creates an index structure in advance that organizes document elements and their positions. This preliminary action enables the parser to quickly locate relevant sections without scanning the entire document, significantly reducing parsing time while preserving processing efficiency for relevant content.
2Ease of operation
If DOM interface is used for query processing, then flexible navigation is enabled, but memory consumption increases due to in-memory document representation
Solution Approach 1:
The patent segments the XML document into manageable index entries that record element positions and hierarchies without loading the entire document into memory. This segmentation allows flexible navigation through the document structure while maintaining low memory consumption by processing only relevant segments as needed.
Solution Approach 2:
The patent introduces an index structure as an intermediary between the query processor and the XML document. This intermediary provides navigation flexibility by storing element position information, enabling DOM-like traversal capabilities while avoiding the memory overhead of loading the complete document representation.
3Quantity of substance
If SAX interface is used for streamed processing, then memory consumption and latency are reduced, but parsing overhead increases due to event generation for all document pieces
Solution Approach 1:
The patent extracts only the relevant SAX events that correspond to query-matching elements by using the index to identify which document portions to process. This extraction eliminates the overhead of generating events for irrelevant document pieces while maintaining the low memory consumption and latency benefits of streamed processing.
Solution Approach 2:
The patent performs preliminary indexing of the XML document structure before streamed processing. This preliminary action enables the SAX parser to skip irrelevant sections and generate events only for relevant elements, reducing parsing overhead while maintaining the efficiency benefits of event-driven streamed processing.
4Loss of time
If index is created to skip irrelevant portions, then parsing time is reduced, but device complexity increases due to index structure and matching logic
Solution Approach 1:
The patent applies local quality by creating indexes only for specific elements and attributes that are relevant to typical queries, rather than indexing the entire document uniformly. This selective indexing reduces the complexity of the index structure while maintaining the ability to skip irrelevant portions effectively for the most common query patterns.
Solution Approach 2:
The patent implements partial indexing by creating index entries only for portions of the document that are likely to be queried, rather than indexing every element. This partial action reduces index structure complexity and matching logic overhead while still achieving significant parsing time reductions for relevant queries.
Data Source
AI summary
A system and method for parsing documents in query processing comprises producing at least one index of a document written in a mark-up language, corresponding the index to the document, scanning the document, and selectively skipping portions of the document based on instructions from the index. Furthermore, the mark-up language comprises any of HTML and XML; the skipped portions of the document comprise portions irrelevant to the query; the index comprises a plurality of elements representing textual categories of the query; and the instructions match the elements to the query. If the elements do not match the query, then the parser uses the index to skip the portions of the document corresponding to the unmatched elements. Moreover, each of the elements corresponds to a position in the document, wherein the position comprises an end position, which determines where to resume scanning the document upon skipping the portions of the document.