Binary XML Storage With Inline Token Dictionaries for Data Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XML storage formats like Compact Schema-aware Binary XML (CSX) are limited by their coupling with XML schema, require a central token repository, and face challenges in data distribution, sharing, and scaling, leading to performance bottlenecks and inefficiencies.
Innovation Solution
The Transportable Binary XML (TBX) format encodes XML documents with an inline token dictionary, eliminating the need for a central repository and allowing schema-less storage, enabling efficient distribution, sharing, and scaling by incorporating optimizations such as single-byte token representation and datatype detection for leaf values, along with a token cache mechanism for rapid tag/attribute mapping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CSX format uses a central token repository for encoding/decoding, then encoding efficiency is improved, but data distribution and sharing become difficult
Solution Approach 1:
The patent divides the centralized token repository into distributed token stores at each node. Each node maintains its own token dictionary locally, eliminating the single central repository. This segmentation allows independent encoding/decoding at each node while maintaining consistency across the distributed system, resolving the contradiction between encoding efficiency and data distribution capability.
Solution Approach 2:
The patent extracts the token repository dependency from the encoding/decoding process. By using content-based hashing to generate tokens dynamically rather than relying on pre-stored token mappings, the system removes the requirement for a central token repository. This extraction enables seamless data distribution and sharing while maintaining encoding efficiency through deterministic token generation.
2Stability of the object's composition
If CSX format maintains a central token repository, then encoding consistency is improved, but scaling becomes problematic
Solution Approach 1:
The patent implements self-service encoding where each node independently generates tokens using content-based hashing on the XML content itself. This eliminates dependency on central token repository management, allowing each node to autonomously maintain encoding consistency. The system scales seamlessly because adding new nodes does not require coordination with or updates to a central repository, resolving the contradiction between encoding consistency and scaling capability.
3Adaptability or versatility
If TBX format uses inline token dictionary, then data portability is improved, but storage size increases
Solution Approach 1:
The patent changes the parameter of token representation from fixed multi-byte tokens to variable-length tokens generated by content-based hashing. This allows the system to use only the necessary number of bytes for each token based on the actual content, reducing overall storage requirements. The inline token dictionary achieves data portability while minimizing storage overhead through this parameter change.
4Productivity
If XML data is stored in distributed format, then scalability is improved, but query performance may deteriorate
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing content-based hash tokens within the XML data structure itself during the encoding phase. This preliminary computation of deterministic tokens allows for rapid query operations later without requiring complex distributed lookups or joins. The pre-computed tokens enable efficient filtering, joining, and querying across distributed nodes, maintaining query performance while achieving scalability.
Data Source
AI summary
A structured document storage format is provided in which tokens used for encoding and decoding are contained within the encoded document in an inline dictionary. Each document is encoded independently, and there is no central dependency or any dependency on other documents during DML and query execution. Each encoded document has all information with it on disk so that the document can be independently shared, decoded, or distributed. A mechanism is provided for quickly determining a mapping between tags and tokens without having to scan the entire inline token dictionary for each document. This allows the database system to process multiple documents, each having its own inline token dictionary, without having to fully scan the inline dictionary of every document for each tag that referenced in an operation.