Cloud Log Record Compression via Counter ID Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based systems face challenges in efficiently storing and compressing large volumes of statistical data logs due to their size and bandwidth requirements, despite existing compression and memory management techniques, as the number of users and features expands.
Innovation Solution
The method involves rearranging counter IDs in cstat records to place frequently used counters at lower IDs, optimizing storage by using a possible offset only when necessary, and employing machine learning to update the arrangement based on historical data, thereby reducing the space required for counter IDs and improving decompression rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compression techniques are applied to log data, then the size of log data is reduced, but the decompression time and processing complexity increase
Solution Approach 1:
The patent segments the log data into fixed-width records with a standardized format including record type, counter ID, and value fields. This segmentation allows for efficient parsing and processing during decompression, as each record can be independently and quickly interpreted without complex nested structures
Solution Approach 2:
The patent changes the parameter representation by using fixed-width fields and standardized data types for each record component. This parameter standardization enables faster decompression processing while maintaining compact storage, resolving the contradiction between compression efficiency and decompression speed
2Adaptability or versatility
If the number of counters in cstat records increases to accommodate more features and users, then the monitoring capability is improved, but the storage space and bandwidth requirements increase
Solution Approach 1:
The patent extracts only the essential components of counter data into a standardized record format (record type, counter ID, value). By taking out only the necessary information and eliminating redundant data structures, the system can accommodate more counters without proportionally increasing storage requirements
Solution Approach 2:
The patent transitions from storing complete counter objects to storing compact numeric records with fixed widths. This dimensional change from complex data structures to simple numeric fields allows the system to scale to tens of thousands of counters while maintaining efficient storage density
3Measurement precision
If the sampling rate is increased to capture more detailed statistics, then the measurement precision is improved, but the data volume and processing load increase
Solution Approach 1:
The patent applies parameter changes by using fixed-width numeric fields for all counter values regardless of their magnitude. This standardization allows high-precision sampling at increased rates without proportionally increasing data volume, as each sample occupies a predictable, compact space
Solution Approach 2:
The patent implements partial action by recording only the essential counter state (ID and value) at each sampling interval rather than complete counter objects. This partial recording approach enables high sampling rates while keeping data volume manageable through selective data capture
Data Source
AI summary
Systems and methods include obtaining statistics based on monitoring in a cloud-based system for a given time period; and, responsive to determining an arrangement of counters for N counters, storing each of M counters for the given time period as a plurality of records with each record including a record type, a possible offset to a next record in terms of a counter identifier (ID), and a counter value, wherein N and M are integers and M<<N, and wherein the arrangement is determined such that most frequently used counters occupy lower counter IDs. The systems and methods can further include updating the arrangement of the counters for the N counters, to perform an optimization such that the most frequently used counters occupy lower counter IDs.