Binary Record Containers for Schemaless Data Without Deserialization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems face challenges in efficiently handling large volumes of schemaless data due to high CPU and memory resource requirements for serialization and deserialization, especially in environments with automated heap memory management, leading to performance issues and serialization overhead during modification processing steps.
Innovation Solution
A unified, binary data format that supports sequential, stream-oriented processing of data records, allowing for network transfer, selective read access, and modification without deserialization, using a serialized record container with offset values and meta data for efficient handling of schemaless data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If automated serialization frameworks like Protocol Buffers are used for network transfer of evidence records, then development effort is reduced and network transfer is simplified, but CPU and memory resources are excessively consumed during serialization and deserialization operations
Solution Approach 1:
The patent applies preliminary action by pre-serializing data records into a compact binary format at the point of generation, before network transfer. This eliminates the need for repeated serialization/deserialization operations during processing, as the data is already in an efficient serialized form that can be directly transmitted and processed without CPU-intensive framework operations.
2Ease of operation
If serialization/deserialization frameworks are used in environments with automated heap memory management, then memory management is simplified, but garbage collection burden increases due to short-lived objects created during serialization operations
Solution Approach 1:
The patent extracts the serialization operation from the regular processing flow by implementing a dedicated serialization step that produces compact binary records. These serialized records are then transmitted and processed without requiring deserialization, effectively removing the serialization/deserialization object creation burden from the main processing loop and reducing garbage collection pressure on the automated memory management system.
3Adaptability or versatility
If data records are transmitted between processing nodes via interconnecting networks, then distributed scalable architecture is achieved, but network transfer overhead and latency increase
Solution Approach 1:
The patent applies parameter changes by transforming the data format into a compact binary representation with optimized field encoding. This parameter change in data structure reduces the size of transmitted records and eliminates the need for complex serialization/deserialization protocols, thereby reducing network transfer time and overhead while maintaining distributed architecture capabilities.
4Adaptability or versatility
If schemaless or semi-structured data is ingested without pre-defined structure, then data variety and flexibility are increased, but processing complexity increases due to need for structure analysis and semantic identification
Solution Approach 1:
The patent applies segmentation by dividing the processing pipeline into distinct stages: ingestion of schemaless data, structure analysis phase, semantic identification phase, and serialization phase. Each stage handles a specific aspect of processing, allowing the system to accommodate diverse data formats while managing complexity through modular, sequential processing steps rather than attempting to handle all variations simultaneously.
Data Source
Figure 1a~1b
Figure 2
Figure 3
AI summary
A serialization data format for the transfer, analysis, and modification of schemaless mass data is proposed. The data format supports stream oriented, pipeline-based processing, and it enables read access of contained data without deserialization, and modification access that only requires the deserialization of portions of structure and meta data. Incoming, semi-structured data records may be transformed into records of the proposed serialization data format, compressed and stored in processing buffers containing multiple of those records. During processing, only individual records are decompressed, and processed records are then compressed and stored in output processing buffers for efficient memory usage. Manipulations of data records are performed by appending new values to data records, invalidating old ones and updating access data structures to refer to new values instead of old ones, to enable various modification activities by only requiring append or not-size-changing operations of serialized data records.