Partitioned XML Parsing for Memory-Efficient Large Document Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current XML processing techniques, particularly those using the DOM model, face challenges with memory efficiency when handling large documents, leading to performance issues and limitations in scalability, especially in Enterprise-class servers, and lack effective methods for sharing parsed documents across processor sessions.
Innovation Solution
The approach involves parsing XML documents into multiple independent partitions without instantiating DOM objects, storing these partitions in memory or a non-transitory storage medium, and instantiating only requested objects, allowing for efficient processing and sharing of large XML documents by using a caching storage hierarchy and logical references among nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the entire XML document is loaded into memory using DOM model, then random access and manipulation of XML elements is enabled, but memory consumption becomes excessive for very large documents
Solution Approach 1:
The XML document is divided into multiple partitions, where only the required partition is loaded into memory at any given time. Each partition contains a subset of the document's nodes and elements, allowing random access within the partition while keeping memory consumption proportional to partition size rather than total document size.
Solution Approach 2:
The patent introduces a disk-based storage dimension to complement in-memory storage. Partitions that are not currently needed are stored on disk, and can be loaded into memory on demand. This adds a spatial dimension (disk vs. memory) to the storage hierarchy, enabling access to documents larger than available memory.
2Quantity of substance
If ad-hoc partitioning is used to process very large documents, then memory consumption is reduced, but application adaptation becomes very difficult
Solution Approach 1:
The patent creates a universal partitioning framework that works with standard DOM APIs, making it applicable to various XML processing scenarios without requiring application-specific modifications. The partitioning logic is encapsulated in the parser and storage layer, while the application layer continues to use familiar DOM interfaces.
Solution Approach 2:
The patent introduces an intermediary layer (the partitioned DOM implementation) that sits between the XML parser and the application. This intermediary handles the complexity of partitioning, storage, and retrieval, while presenting a simplified DOM interface to the application, thus shielding the application from partitioning complexity.
3Quantity of substance
If streaming approaches are used for processing large XML documents, then memory efficiency is improved, but the ability to perform non-document-order processing is limited
Solution Approach 1:
The patent performs preliminary parsing of the XML document into a structured partitioned format before the application processes it. During this preliminary phase, the document is parsed and organized into partitions with proper node relationships established, enabling both memory efficiency and subsequent flexible processing without requiring full in-memory loading.
Data Source
AI summary
Certain example embodiments described herein relate to techniques for processing XML documents of potentially very large sizes. For instance, certain example embodiments parse a potentially large XML document, store the parsed data and some associated metadata in multiple independent blocks or partitions, and instantiate only the particular object model object requested by a program. By including logical references rather than physical memory addresses in such pre-parsed partitions, certain example embodiments make it possible to move the partitions through a caching storage hierarchy without necessarily having to adjust or encode memory references, thereby advantageously enabling dynamic usage of the created partitions and making it possible to cache an arbitrarily large document while consuming a limited amount of program memory. Such techniques may be extended to enable atomic updates to be processed efficiently, e.g., by maintaining commit level information in a partition list and optionally implementing document shadowing.