Trace Log Compression Using Function Call Dictionaries and Grammars
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data compression methods for trace logs generated during MPI program execution have low compression rates and long compression times, particularly in high-concurrency scenarios.
Innovation Solution
Constructing a dictionary and grammar set based on function call records in trace logs to improve compression efficiency, considering data features and semantic similarities among function calls, and combining dictionaries and grammar sets across processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general compression methods are used on trace logs, then compression can be performed without considering data features, but the compression rate is low and compression time is long
Solution Approach 1:
The patent applies preliminary action by constructing dictionaries and grammar sets before compression based on function call records in the trace log. The dictionary is built by analyzing function call patterns and semantics in advance, and the grammar set is constructed to represent the structural rules of function calls. This preliminary analysis enables the compression algorithm to efficiently encode the trace log by referencing the pre-built dictionary and grammar, thereby achieving high compression rates without sacrificing compression speed.
2Productivity
If trace logs are compressed without considering data features, then the compression process is simple, but the compression rate is low
Solution Approach 1:
The patent applies segmentation by dividing the compression process into distinct stages: function call record analysis, dictionary construction, grammar set construction, and compression encoding. The dictionary is segmented into function call patterns and their semantics, while the grammar set is segmented into structural rules. This segmentation allows each component to be optimized independently, achieving high compression rates while maintaining a manageable and systematic compression process.
3Productivity
If dictionaries and grammar sets are constructed for each process individually, then process-specific compression can be achieved, but the overall compression efficiency is reduced
Solution Approach 1:
The patent applies merging by combining dictionaries and grammar sets across multiple processes. Instead of maintaining separate dictionaries and grammar sets for each process, the patent merges them into unified structures that capture common function call patterns and semantics across all processes. This merging reduces redundancy and improves overall compression efficiency while simplifying the management of compression resources.
Data Source
AI summary
This application discloses a data compression method and apparatus, and relates to the field of data processing technologies. The method includes: obtaining a first trace log generated in a running process of a first process, where each log record in the first trace log includes a function call record of calling a communication function by the first process; and constructing a first dictionary and a first grammar set based on the function call record in the first trace log, to compress the function call record in the first trace log, where for a first function call record included in any log record in the first trace log, a first symbol string in the first dictionary indicates the first function call record. According to the method, a compression rate of compressing the trace log can be improved, and compression duration can be shortened.