Data Fragment Aggregation via Unified Structure Model
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current network traffic processing systems process data streams as independent events, preventing linkage between data fragments and requiring high memory for aggregation, leading to increased costs and potential information loss due to limited memory capacities.
Innovation Solution
A system comprising an extraction module and an aggregation module that outputs and processes data fragments with structure data, allowing for modular and adaptable aggregation across different protocols and decoder units, enabling efficient data processing and conversion into a suitable format for further analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data streams are processed as independent events without linkage, then processing simplicity is maintained, but the ability to link and analyze related data fragments is lost
Solution Approach 1:
The system segments data processing into distinct functional modules: extraction module for acquiring data fragments, structure data module for organizing data with metadata, and analysis module for processing. This segmentation allows each module to handle specific tasks independently while maintaining overall data linkage through structured interfaces, resolving the contradiction between processing simplicity and information preservation.
Solution Approach 2:
Structure data acts as an intermediary between raw data fragments and analysis operations. It provides a standardized format with metadata that enables linkage between related data fragments while maintaining processing simplicity through uniform interfaces. The structure data module serves as a mediator that organizes data without requiring complex processing logic in other modules.
2Measurement precision
If a decoder-specific data model is used for each decoder type, then decoding accuracy is improved, but the ability to link and aggregate data from different decoder types is prevented
Solution Approach 1:
The structure data model provides a universal framework that can accommodate data from multiple decoder types. It defines standardized fields and metadata structures that work across different protocols and decoder implementations, enabling aggregation and linkage of data from heterogeneous sources while maintaining the decoding accuracy benefits of specialized decoders through appropriate data representation.
3Quantity of substance
If all data is aggregated in the decoder with limited memory, then memory usage is constrained, but information may be lost due to insufficient memory capacity
Solution Approach 1:
The extraction module separates data acquisition from aggregation functions. It extracts relevant data fragments and structure data from the data stream and passes them to the analysis module, removing the burden of complete data aggregation from the decoder's limited memory. This extraction approach ensures data completeness by allowing comprehensive data collection in the analysis module while keeping decoder memory usage constrained to essential processing data only.
4Loss of information
If the decoder keeps all information until end of data flow, then data completeness is ensured, but memory requirements and system costs increase
Solution Approach 1:
The system performs preliminary organization of data into structure data format during the extraction phase, before complete data collection is necessary. This preliminary structuring allows for efficient data management and enables the analysis module to process data in organized chunks rather than requiring all raw data to be held in memory simultaneously, ensuring data completeness while reducing peak memory requirements and system costs.
Data Source
Figure 1
Figure 2~4
Figure 5~6
AI summary
A system (10) for analyzing and interpreting at least one data stream comprises at least one extraction module (28) and at least one aggregation module (30). The extraction module (28) is configured to output data fragments extracted from the data stream including structure data. The aggregation module (30) is configured to receive the outputted data fragments from the extraction module (28). The aggregation module (28) is configured to process the data fragments including the structure data. The aggregation module (30) is configured to aggregate data fragments according to one predefined structure data for different protocols, data streams and/or decoder units (34) assigned to the at least one extraction module (30). Further, a method of aggregating data fragments is described.