DynField Format for High-Performance Stream Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing technologies face challenges in efficiently handling high-performance, low-latency stream data processing, particularly with complex, hierarchical data formats that require variable field sizes and structures, leading to issues like data bloat and increased processing time.
Innovation Solution
The dynamic field (DynField) format is introduced, which is self-describing and flexible, allowing for variable length fields and complex nested data structures, enabling efficient processing without the need for state-tracking or backtracking, and is optimized for record-oriented streaming data processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If fixed field format is used for stream processing, then processing speed is improved through direct address calculation, but data bloat occurs when field sizes must accommodate all legal values
Solution Approach 1:
The patent applies dynamics by transitioning from fixed field formats to dynamic field formats where field lengths can vary. The system uses length prefixes and type indicators that allow fields to be compact when values are small but expand to accommodate larger values, eliminating the need to allocate maximum possible size for all fields uniformly.
Solution Approach 2:
The patent changes the parameter of field length from fixed to variable. By introducing length metadata and using encoding schemes that adapt to actual data values, the system optimizes the balance between processing efficiency and data compactness, allowing direct address calculation to remain effective while reducing overall data size.
2Productivity
If fixed field format is used, then processing efficiency is improved, but the format cannot represent hierarchical data or variable repetition
Solution Approach 1:
The patent uses dynamic field formats with type indicators and length metadata that enable the representation of hierarchical structures and variable repetitions. The system can dynamically interpret field relationships and nesting levels while maintaining efficient processing through structured metadata.
Solution Approach 2:
The patent creates a universal data format that can represent multiple data structures including hierarchical data, variable repetitions, and flat structures within the same framework. The format uses type indicators and length fields to universally handle diverse data organization patterns while maintaining processing efficiency.
3Quantity of substance
If variable length fields are used to reduce data bloat, then data size is reduced, but processing complexity increases due to metadata management
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing field lengths and types in metadata structures before processing. This allows the processing engine to efficiently navigate variable-length fields using pre-computed address information and length data, reducing the complexity of runtime calculations.
Solution Approach 2:
The patent introduces metadata as an intermediary layer between the variable-length data fields and the processing engine. This metadata layer handles the complexity of length tracking and field identification, allowing the main processing logic to work with simplified, standardized field references rather than directly managing variable boundaries.
4Adaptability or versatility
If simple delimited formats like CSV are used to allow variable length fields, then data flexibility is improved, but processing speed decreases due to byte-by-byte crawling
Solution Approach 1:
The patent uses preliminary action by pre-calculating field boundaries and storing length information in metadata during data generation or initial parsing. This eliminates the need for byte-by-byte crawling during processing, as the system can directly jump to field boundaries using pre-computed position information.
Solution Approach 2:
The patent replaces the mechanical byte-by-byte crawling approach with a more efficient system that uses metadata-driven direct access. Instead of sequentially examining each byte to identify field boundaries, the system uses stored length and position metadata to directly navigate to fields, substituting computational overhead with efficient data structure access.
Data Source
AI summary
Improved computer technology is disclosed for enabling high performance stream processing on data such as complex, hierarchical data. In an example embodiment, a dynamic field schema specifies a dynamic field format for expressing the incoming data. An incoming data stream is then translated according to the dynamic field schema into an outgoing data stream in the dynamic field format. Stream processing, including field-specific stream processing, can then be performed on the outgoing data stream.