Binary JSON Deduplication and Offset Navigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing approaches to converting JSON files into binary representations, such as BSON, face challenges including high storage requirements, inefficient navigation, and computationally expensive partial updates due to lack of metadata structures and reliance on general-purpose compression.
Innovation Solution
The proposed solution involves converting JSON files into Oracle binary JSON (OSON) files, which use relative node jump offsets and deduplicated leaf values to reduce file size and improve query performance. This is achieved through a process that determines whether the JSON file is mutable or immutable, applying stream or space optimized deduplication accordingly, and using relative offsets for compression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If JSON files are converted into binary representations like BSON, then storage efficiency is improved, but navigation and partial update efficiency deteriorate due to lack of metadata structures
Solution Approach 1:
The binary JSON format is segmented into distinct components: metadata structures containing navigation information are separated from the actual data payload. This segmentation allows the system to maintain efficient storage while enabling fast navigation through dedicated metadata pathways without parsing the entire data structure.
Solution Approach 2:
Metadata structures serve as intermediaries between the binary data and the navigation operations. These metadata structures contain pre-computed navigation information that mediates between the compact binary representation and the required navigation operations, eliminating the need to parse the entire data structure for navigation purposes.
2Quantity of substance
If general purpose compression algorithms are applied to BSON files, then storage footprint is reduced, but random access and update efficiency deteriorate due to required full decompression
Solution Approach 1:
The binary JSON format is segmented into distinct components: metadata structures containing navigation information are separated from the actual data payload. This segmentation allows the system to maintain efficient storage while enabling fast navigation through dedicated metadata pathways without parsing the entire data structure.
Solution Approach 2:
Instead of requiring full decompression of the entire BSON file for any operation, the format enables partial decompression or direct access to specific segments. The metadata structures allow the system to perform operations on only the necessary portions of the data, avoiding the excessive action of decompressing unrelated data.
3Ease of operation
If metadata structures are added to support navigation, then navigation efficiency is improved, but file size increases
Solution Approach 1:
The metadata structures use optimized parameter representations that minimize size while maintaining navigation functionality. By carefully selecting and encoding only the essential navigation parameters, the format achieves efficient navigation without proportionally increasing file size.
Solution Approach 2:
The metadata structures are designed with local quality optimization, containing navigation information only where needed rather than redundant global structures. This allows navigation efficiency to be improved in specific areas of the data without uniformly increasing the entire file size.
Data Source
AI summary
Data structures and methods are described for converting a text format data-interchange file into size efficient binary representations. A method comprises receiving a request to convert a data-interchange file, comprising a hierarchy of nodes, into a binary file. The method further comprises generating a tree representation of the nodes that reference a plurality of leaf values. The method further comprises, in response to determining that the binary file is to be compressed, embedding relative node jump offsets when generating the tree representation. The method further comprises, in response to determining that the data-interchange file is immutable, deduplicating the plurality of leaf values in a space optimized manner. The method further comprises, in response to determining that the data-interchange file is mutable, deduplicating the plurality of leaf values in a stream optimized manner. The method further comprises storing the deduplicated plurality of leaf values in the binary file.