Block-Based Data Transformer for Memory-Efficient JSON Conversion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Converting large text-based human-readable data formats, such as JSON, into a form suitable for processing by applications is resource-intensive in terms of time and processing power due to the inefficiencies of existing methods.
Innovation Solution
The use of transformer embodiments that operate on blocks of data, applying a sequence of transformer rules to convert data from one structured format to another, such as converting JSON-formatted documents to arrays of JAVA objects, reducing memory consumption and processing time by handling data in small fractions of the full document.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the entire text-based human-readable document is loaded into memory for conversion, then the conversion can be performed in a single pass, but the memory consumption becomes prohibitively large for large documents
Solution Approach 1:
The patent divides the large text-based document into smaller blocks or chunks that can be processed individually. Instead of loading the entire document into memory at once, the converter processes sequential blocks, converting each block from the text-based human-readable format to the target format separately. This segmentation approach maintains conversion productivity while dramatically reducing memory consumption to only the size of individual blocks rather than the full document.
2Reliability
If conventional conversion methods are used on large documents, then complete conversion is achieved, but the processing time and computational resources required become excessive
Solution Approach 1:
By segmenting the conversion process into independent block-level operations, the patent enables parallel processing capabilities and reduces the sequential processing bottleneck. Each block can be converted independently and written to output immediately, eliminating the need to wait for entire document processing before producing results. This maintains complete conversion reliability while significantly reducing total conversion time.
Solution Approach 2:
The patent performs preliminary setup by establishing the conversion configuration and rules before processing begins. The converter is pre-configured with the source format specifications, target format requirements, and conversion rules, allowing each subsequent data block to be processed immediately upon arrival without additional setup overhead. This preliminary action ensures complete and accurate conversion while minimizing per-block processing time.
Data Source
AI summary
A system includes a processor configured to obtain a sequence of transformer rules. The transformer rules specify a set of data elements arranged according to a first structured data format, and structural changes to be performed on the data elements that convert the data elements into a second structured data format. The processor receives a block of data from a file arranged according to the first structured data format. The processor executes the sequence of transformer rules to perform the structural changes to the block of data. When executing the particular transformer rule, the processor applies an adapter associated with the transformer rule to modify values in the block of data specified by the particular transformer. The processor then provides for display or storage the block of data as converted into the second structured data format by the sequence of transformer rules.