Data Pre-processor Partitioning Netflow Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional data processing systems are inadequate for handling large volumes of netflow records and other big data applications due to scalability issues and skewed data distributions, making it impractical to analyze them effectively.
Innovation Solution
A data pre-processor partitions and transforms scalar data into directional data using bins with quantiles, generating workloads suitable for parallel processing, and employs index keys and partition key tuples to facilitate efficient analysis across multiple servers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional data processing systems are used to handle large volumes of netflow records, then data analysis can be performed, but the systems become inadequate due to scalability issues and skewed data distributions
Solution Approach 1:
The patent segments the large collection of netflow data into multiple partitions, each handled by separate processing components. This segmentation allows the system to scale by adding more processing units without overwhelming a single system, directly addressing the scalability limitation of traditional data processing systems.
Solution Approach 2:
The patent introduces a new dimensional approach by transforming the data processing architecture from a single-system model to a distributed multi-partition model. This dimensional change enables the system to handle skewed data distributions more effectively by distributing the load across multiple dimensions of processing resources.
2Quantity of substance
If large volumes of netflow records are processed, then comprehensive analysis is achieved, but storage needs and processing time increase significantly
Solution Approach 1:
The patent performs preliminary actions by pre-processing and partitioning the netflow data before the main analysis phase. This preliminary partitioning organizes the data into manageable segments with skewed distributions handled in advance, which significantly reduces the processing time required for the subsequent analysis of large data volumes.
Solution Approach 2:
By segmenting the large data volume into smaller partitions, the system can process each partition independently and in parallel, reducing the overall processing time while maintaining comprehensive analysis of the complete data set.
3Device complexity
If traditional data systems are used for netflow analysis, then simple processing is maintained, but the systems become impractical for enterprise-scale data
Solution Approach 1:
The patent introduces dynamic partitioning capabilities that allow the system to adapt its complexity based on the data characteristics. For enterprise-scale data with skewed distributions, the system dynamically creates appropriate partition strategies, maintaining simplicity for smaller data sets while enabling sophisticated processing when needed.
Solution Approach 2:
The system changes key parameters such as partition count, partitioning strategy, and processing depth based on the scale and characteristics of the input data. This parameter adaptation allows the system to maintain simplicity for small-scale processing while automatically scaling to handle enterprise-level data volumes effectively.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatuses, methods and storage medium associated with processing data are disclosed herewith. In embodiments, an apparatus may include a data pre-processor to partition a collection of scalar data associated with unidirectional relationships of a plurality of two endpoint pairs into a plurality of workloads of directional data associated with the plurality of two endpoint pairs. In embodiment, the partition operation may include partition of the two endpoint pairs into a plurality of bins, where each bin includes a plurality of quantiles of the two endpoint pairs. Other embodiments may be described and/or claimed.