Streaming Parser for Structured Data Files
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large human-readable data-interchange files, such as those in XML, JSON, and YAML formats, require significant memory and processing resources, making them inefficient for use in computing devices with limited capacity, particularly in IoT devices, and result in high latency during download and parsing.
Innovation Solution
A streaming parser that processes files in discrete blocks in a pipelined fashion, allowing for parallel reception and parsing, thereby reducing memory requirements and overall latency, and enabling operation on devices with limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large human-readable data-interchange files are downloaded and parsed in full before processing, then complete data availability is achieved, but memory storage requirements and processing time increase significantly
Solution Approach 1:
The patent divides the data-interchange file into discrete blocks that are processed sequentially. Instead of loading the entire file into memory, the system downloads and parses one block at a time, allowing large files to be handled with limited memory resources while maintaining complete data processing capability
Solution Approach 2:
The patent performs preliminary parsing actions on data blocks as they are being downloaded, rather than waiting for complete file download. This overlapping of download and parse operations reduces total processing time and memory requirements by processing data incrementally as it becomes available
2Reliability
If large human-readable data-interchange files are downloaded and parsed in full before processing, then complete data availability is achieved, but processing time increases significantly
Solution Approach 1:
The patent performs preliminary parsing actions on data blocks as they are being downloaded, rather than waiting for complete file download. This overlapping of download and parse operations reduces total processing time and memory requirements by processing data incrementally as it becomes available
Solution Approach 2:
The patent maintains continuous processing by immediately parsing downloaded data blocks without waiting for the entire file to be downloaded. This continuous action eliminates idle time between download and parse operations, significantly reducing total processing time while ensuring complete data is eventually processed
3Ease of operation
If human-readable data-interchange formats are used for communication, then simplicity and readability are improved, but file size and memory requirements increase
Solution Approach 1:
The patent divides the large human-readable data-interchange file into discrete blocks that are processed sequentially. Instead of loading the entire file into memory, the system downloads and parses one block at a time, allowing large files to be handled with limited memory resources while maintaining the benefits of human-readable formats
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An embodiment may involve a computing system that includes a processor and memory. The memory may contain program instructions executable by the processor to repeatedly perform, for each block of a textual data-interchange file, operations including: obtaining a block of the file, where the block contains one or more records each containing one or more elements; identifying any pre-defined elements contained in records that are completed within the block, where the pre-defined elements are specified by a set of paths, the paths each hierarchically defining a location of an element within a record; storing, and into one or more files or one or more database tables, the pre-defined elements contained in records that are completed within the block; and determining whether the block ends with a partial record, and maintaining any such partial record for later storage in conjunction with processing of a subsequent block of the file.