Pipelined JSON Table Conversion With Parallel Ordinality Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for converting language-independent data formats like JSON to relational databases using JSON_TABLE commands are inefficient in terms of memory consumption and runtime cycles, particularly when parallel processing is involved, due to issues with error handling and top-level ordinality columns.
Innovation Solution
A pipelined approach is employed for parallel processing of JSON data without buffering, using a global atomic ordinality value counter and conditional execution based on the EMPTY ON ERROR option to handle errors and ensure consecutive ordinality values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parallel processing is used for converting JSON data to relational format, then processing speed is improved, but memory consumption increases due to buffering requirements
Solution Approach 1:
The patent divides the parallel processing task into segments where each thread processes a portion of JSON data and outputs results incrementally through a pipeline. This allows memory to be reused across processing stages rather than buffering entire datasets, reducing peak memory consumption while maintaining parallel processing throughput.
Solution Approach 2:
The patent introduces a pipeline intermediary that acts as a buffer between parallel processing threads and the final output. This intermediary manages memory efficiently by processing and forwarding data in chunks, allowing parallel threads to continue processing without requiring full output buffering, thus reducing overall memory requirements.
2Reliability
If parallel processing with buffering is used, then error handling becomes simpler, but runtime cycles increase due to data copying and memory management
Solution Approach 1:
The patent extracts error handling logic from the main parallel processing pipeline into separate error management mechanisms. This allows the primary processing threads to continue efficiently without being blocked by error checking overhead, while errors are captured and managed through dedicated pathways that minimize runtime impact.
Solution Approach 2:
The patent maintains continuous data flow through the processing pipeline by avoiding blocking operations. Error handling is implemented in a non-blocking manner that allows processing to continue uninterrupted, with errors logged or handled asynchronously, thus minimizing additional runtime cycles compared to traditional buffering approaches.
3Ease of manufacture
If traditional JSON_TABLE commands are used, then implementation is simpler, but conversion efficiency decreases due to sequential processing
Solution Approach 1:
The patent transforms the static sequential JSON_TABLE command into a dynamic parallel processing system. The implementation uses multiple processing threads that can be dynamically adjusted based on available resources, maintaining relative simplicity through standardized interfaces while achieving superior conversion efficiency through concurrent execution.
Solution Approach 2:
The patent replaces the mechanical sequential execution model of traditional JSON_TABLE commands with a parallel processing architecture. This substitution maintains implementation simplicity through high-level abstractions while dramatically improving conversion efficiency by utilizing multiple processing cores simultaneously.
4Ease of operation
If ordinality values are managed without a global counter, then thread independence is improved, but ordinality value accuracy decreases due to non-consecutive numbering
Solution Approach 1:
The patent merges the ordinality counter management into a global atomic counter that is shared across all processing threads. This single centralized counter ensures consecutive ordinality values are assigned accurately, while threads maintain independence by simply reading and incrementing the counter without direct interference, balancing both thread independence and ordinality accuracy.
Solution Approach 2:
The patent introduces a global atomic counter as an intermediary between parallel threads and ordinality value assignment. This intermediary coordinates value assignment atomically, ensuring consecutive numbering while allowing threads to operate independently without direct synchronization overhead, thus maintaining both thread independence and ordinality accuracy.
Data Source
AI summary
In an example embodiment, the execution of a command (such as JSON_TABLE) to convert language-independent data format data to a structured table is performed by parallel processing the data in parallel and without buffering all of the input and output. More specifically, a pipelined approach is used that is parallelizable and reduces memory requirements. Input rows are parallel processed within chunks, while produced rows are output directly.