XML Document Caching via Delayed Node Instantiation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional XML document processing techniques, such as DOM implementations, face challenges with large documents due to high memory requirements, poor system performance, and difficulties in sharing parsed documents across processes, especially in Enterprise-class server systems, where scalability and predictability are essential.
Innovation Solution
The approach involves parsing XML documents into constituent nodes without instantiating objects, storing them in multiple independent cacheable partitions, and instantiating only the requested objects, allowing for efficient processing of large documents by managing memory usage and enabling sharing across processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the entire object model is loaded into memory for processing, then random access to any node is enabled, but memory consumption becomes unmanageable for large documents
Solution Approach 1:
The patent divides the XML document into multiple partitions, each representing a segment of the document tree. Only the partition containing the requested node is loaded into memory, while other partitions remain on disk. This segmentation allows random access to any node without loading the entire document into memory, resolving the contradiction between accessibility and memory consumption.
Solution Approach 2:
The patent performs preliminary parsing of the XML document to create a partition table that maps nodes to their respective partitions. This preliminary action enables the system to quickly locate and load only the necessary partition when a node is accessed, rather than loading the entire document. The partition table is stored in memory while the actual document content remains partitioned on disk.
2Quantity of substance
If ad-hoc partitioning is used to reduce memory usage, then memory consumption decreases, but application adaptation becomes very difficult
Solution Approach 1:
The patent creates a universal partitioning framework that works with any XML document regardless of its size or structure. The partition table format and access methods are standardized, allowing applications to access XML documents without needing to adapt to document-specific partitioning schemes. This universal approach maintains low memory consumption while eliminating the need for application-specific adaptation.
3Quantity of substance
If streaming approaches are used to process large documents, then memory efficiency improves, but non-document-order processing becomes difficult
Solution Approach 1:
By dividing the document into partitions that can be independently loaded, the system enables random access to any partition regardless of document order. Applications can request specific partitions by navigating the partition table, allowing processing in any order (depth-first, breadth-first, or custom orders) while maintaining memory efficiency by loading only the required partitions.
4Speed
If the entire document is parsed and stored in memory, then fast access to any node is achieved, but system performance degrades for very large documents
Solution Approach 1:
The partitioning approach maintains fast access speed by organizing partitions with efficient data structures and maintaining an index in the partition table. When a node is accessed, the system quickly locates the corresponding partition in the table and loads only that partition into memory. This segmentation prevents the performance degradation associated with loading entire large documents while maintaining O(1) or O(log n) access time through the partition index.
Data Source
AI summary
Certain example embodiments described herein relate to techniques for processing XML documents of potentially very large sizes. For instance, certain example embodiments parse a potentially large XML document, store the parsed data and some associated metadata in multiple independent blocks or partitions, and instantiate only the particular object model object requested by a program. By including logical references rather than physical memory addresses in such pre-parsed partitions, certain example embodiments make it possible to move the partitions through a caching storage hierarchy without necessarily having to adjust or encode memory references, thereby advantageously enabling dynamic usage of the created partitions and making it possible to cache an arbitrarily large document while consuming a limited amount of program memory.