Log Data Aggregation via Node Segmentation and Preliminary Action
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for processing log data are inefficient in aggregating and analyzing user access logs, leading to high bandwidth usage and potential performance bottlenecks during data transmission and calculation.
Innovation Solution
A method and system for processing log data that involves acquiring log data within a preset time period, determining attribute value groups, calculating indicator statistic values for each group, and generating a log data aggregation table, which reduces bandwidth usage and distributes calculation across multiple nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If user access logs from all information nodes are sent to a log aggregation server through a message queue for aggregation calculation, then centralized log analysis can be achieved, but bandwidth resources are excessively occupied and performance bottlenecks occur during data transmission and calculation
Solution Approach 1:
The patent divides the centralized aggregation process into multiple segments: each information node performs local aggregation on its own logs, then only the aggregated results are sent to the log aggregation server. This segmentation reduces the volume of data transmitted across the network, thereby reducing bandwidth consumption while still achieving comprehensive log analysis through the combination of results from multiple nodes.
2Loss of information
If all user access logs are transmitted to a central log aggregation server for processing, then complete log data can be analyzed, but performance bottlenecks occur during data transmission and calculation
Solution Approach 1:
The patent applies preliminary action by having each information node perform aggregation calculations on its local logs before transmitting the results to the central server. This pre-processing step reduces the amount of data that needs to be transmitted and processed centrally, thereby improving overall processing efficiency while maintaining the completeness of log analysis through the combination of results from all nodes.
3Reliability
If log aggregation is performed centrally on all information nodes' logs, then comprehensive analysis is achieved, but the system experiences performance bottlenecks during calculation
Solution Approach 1:
The patent segments the aggregation calculation process by distributing it across multiple information nodes rather than performing it centrally. Each node independently aggregates its own logs, which reduces the computational load on any single system and eliminates the performance bottleneck that would occur with centralized processing of all logs. The comprehensive analysis is still achieved by combining the aggregated results from all nodes.
Solution Approach 2:
The patent performs aggregation calculations in advance at each information node before the data needs to be analyzed centrally. This preliminary action reduces the amount of work required during the final analysis phase, thereby reducing the time required for comprehensive log analysis while maintaining the completeness of the results.
Data Source
AI summary
Embodiments of the present disclosure disclose a method, apparatus and system for processing log data. A specific embodiment of the method includes: acquiring multiple pieces of to-be-processed log data generated based on a log within a preset time period, the to-be-processed log data including multiple attributes and preset indicators; determining multiple attribute value groups; determining, for an attribute value group, based on indicator values of preset indicators respectively indicated by multiple pieces of to-be-processed log data having each attribute value in the attribute value group, an indicator statistic value of the preset indicators of a log having each attribute value in the attribute value group; and generating a log data aggregation table based on indicator statistic values of preset indicators respectively corresponding to the multiple attribute value groups.