Pipelined JSON Table Conversion With Parallel Ordinality Handling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Converting JSON data to relational format using the JSON_TABLE function in a database management system is inefficient in terms of memory consumption and runtime cycles when performed sequentially, and parallel processing introduces issues with error handling and top-level ordinality columns.
Innovation Solution
Implement a pipelined approach for parallel processing of JSON data without buffering, using a global atomic ordinality value counter to manage ordinality values and handle errors by ensuring all parallel operations complete without errors before outputting results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If sequential processing of JSON data conversion is used, then memory consumption is lower, but processing speed and runtime efficiency deteriorate
Solution Approach 1:
The patent divides the JSON data conversion process into multiple parallel segments or tasks that can be processed simultaneously. Each segment handles a portion of the conversion work, allowing the system to achieve higher throughput while maintaining manageable memory usage per segment rather than loading entire datasets into memory at once.
Solution Approach 2:
The patent transitions from sequential single-threaded processing to parallel multi-threaded processing, adding a temporal dimension to the execution model. Multiple processing threads operate concurrently on different portions of the data, effectively transforming the processing architecture from one-dimensional sequential execution to multi-dimensional parallel execution.
2Productivity
If parallel processing is implemented to improve processing speed, then productivity increases, but error handling complexity and memory requirements worsen
Solution Approach 1:
The patent introduces intermediary components such as thread-safe data structures, synchronization mechanisms, and error aggregation handlers that mediate between parallel processing threads and the final output. These intermediaries coordinate the parallel operations, collect results from multiple threads, and centralize error handling logic, thereby reducing the overall complexity of managing parallel execution.
Solution Approach 2:
The patent implements feedback mechanisms where processing results and error states from parallel threads are continuously monitored and aggregated. Error information flows back through the system in a structured manner, allowing the error handling logic to respond to issues from any thread in a unified way, simplifying the complexity of distributed error management.
3Loss of time
If parallel processing is used to reduce runtime cycles, then processing efficiency improves, but maintaining accurate ordinality values across threads becomes more difficult
Solution Approach 1:
The patent applies preliminary actions by pre-allocating and pre-synchronizing access to ordinality counters before parallel processing begins. Thread-safe counter mechanisms are established in advance, and synchronization protocols are put in place before the parallel threads start their work, ensuring that ordinality values can be accurately assigned without conflicts during the high-speed parallel execution.
Solution Approach 2:
The patent implements beforehand cushioning by introducing synchronization barriers and atomic operations that protect against ordinality value conflicts. These protective mechanisms are built into the processing architecture in advance, cushioning against potential race conditions and ensuring that even with parallel execution, the ordinality values remain accurate and consistent throughout the conversion process.
Data Source
AI summary
In an example embodiment, the execution of a command (such as JSON_TABLE) to convert language-independent data format data to a structured table is performed by parallel processing the data in parallel and without buffering all of the input and output. More specifically, a pipelined approach is used that is parallelizable and reduces memory requirements. Input rows are parallel processed within chunks, while produced rows are output directly.