Reporting Data Volume Reduction via Frequency-Based Dataset Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of reporting data in data networks due to high granularity and growing network throughput leads to redundant data processing, requiring scalable resources and inefficient data transfer.
Innovation Solution
A system comprising a data collecting node that classifies data fields into low frequency change and high frequency change data, combining consecutive datasets into a smaller combination dataset for processing, reducing data transfer volume by aggregating high frequency change data and sending only the low frequency change data and analytical function parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high granularity reporting data is collected from growing network devices, then network visibility and analysis precision are improved, but data volume and processing complexity increase significantly
Solution Approach 1:
The patent segments reporting data into multiple datasets based on different network devices and time periods. Each dataset is processed independently, allowing the system to handle large volumes of high-granularity data through division into manageable units that can be aggregated efficiently
Solution Approach 2:
The patent combines multiple consecutive datasets into a single consolidated dataset. This merging process consolidates redundant information while preserving necessary high-granularity details, reducing the overall data volume that needs to be processed by centralized systems
2Loss of information
If multiple consecutive datasets are transmitted to centralized systems, then complete network behavior analysis is achieved, but data transfer volume and network bandwidth consumption increase
Solution Approach 1:
The patent extracts and removes duplicate information from multiple consecutive datasets before transmission. By identifying and eliminating redundant data elements while retaining unique network behavior information, the system reduces data transfer volume without compromising analysis completeness
Solution Approach 2:
The patent performs preliminary data consolidation and deduplication at the edge network devices before data is transmitted to centralized systems. This advance processing reduces the burden on network bandwidth and centralized processing resources while ensuring complete analysis capability
3Loss of information
If centralized analytics systems process all collected reporting data, then comprehensive network insights are provided, but system resource requirements and processing time increase
Solution Approach 1:
The patent segments the data processing task into distributed operations at edge devices and centralized systems. Edge devices perform initial consolidation and filtering, while centralized systems focus on higher-level analysis, dividing the processing workload to reduce overall processing time
Solution Approach 2:
The patent performs preliminary data consolidation, deduplication, and filtering at edge network devices before data reaches centralized analytics systems. This advance preparation reduces the processing burden on centralized systems, enabling faster generation of network insights
Data Source
AI summary
Decreasing a volume of data transfer over a network may commence with collecting a plurality of datasets having subscriber data. The method may continue with classifying data fields of each dataset of the plurality of datasets into low frequency change data and high frequency change data based on predetermined criteria. The method may further include combining a plurality of consecutive datasets of the plurality of datasets into a combination dataset. The combination dataset may include the low frequency change data and aggregated high frequency change data from the plurality of consecutive datasets. The method may continue with providing the combination dataset to a data processing node.