Selective Telemetry Sampling for Exposed-Secret Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log telemetry data often exposes sensitive information, posing security and legal risks, and traditional methods like encryption or purging data impede troubleshooting and are inefficient.
Innovation Solution
Selective and dynamic sampling of data records during generation, triggered by predetermined conditions, identifies a subset for analysis using various models to detect sensitive data and applies security processes only when necessary.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If comprehensive scanning of all log telemetry data is performed to detect sensitive information, then security detection capability is improved, but processing time and computational resources are excessively consumed
Solution Approach 1:
The patent segments the log telemetry data into distinct data records and further divides them into batches or groups. By processing data in segments rather than as a monolithic whole, the system can apply sampling techniques to select representative subsets for analysis, reducing overall processing time while maintaining detection effectiveness.
Solution Approach 2:
The patent implements partial action by scanning only a sampled subset of data records rather than performing comprehensive scanning on all data. The sampling mechanism selects representative records that are sufficient for detecting sensitive information patterns, avoiding the excessive computational burden of processing every single record while maintaining adequate security detection capability.
2Object-affected harmful factors
If encryption or purging of log telemetry data is applied to protect sensitive information, then security risk is reduced, but data integrity and troubleshooting capability are compromised
Solution Approach 1:
The patent applies preliminary action by detecting and flagging sensitive information patterns in sampled data records before the data is widely distributed or stored. By identifying sensitive data early in the pipeline through pattern matching in samples, the system can apply targeted security measures only where needed, rather than encrypting or purging all data uniformly, thus preserving data integrity for troubleshooting while mitigating security risks.
Solution Approach 2:
The patent applies local quality by selectively applying security measures only to specific data records that contain sensitive information, rather than uniformly encrypting or purging all log data. The sampling and pattern detection mechanisms identify which local portions of the data require protection, allowing the system to maintain high data integrity for debugging while providing targeted security protection where sensitive information is present.
3Productivity
If dynamic sampling with conditional triggering is implemented, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent implements dynamics by making the sampling rate and triggering conditions adjustable and adaptive. The system can dynamically modify sampling parameters based on data characteristics, security requirements, and resource availability. This dynamic approach allows the system to optimize processing efficiency for different scenarios while managing complexity through configurable parameters rather than hard-coded complex logic.
Solution Approach 2:
The patent applies parameter changes by allowing the sampling rate, batch size, and triggering conditions to be modified as configurable parameters. This enables the system to adapt processing efficiency to different operational contexts without fundamentally changing the system architecture. By exposing these as adjustable parameters, the system manages complexity while maintaining high productivity through flexible configuration rather than complex internal logic.
Data Source
AI summary
Disclosed systems and methods identify a data record set and determine whether one or more predetermined conditions exist for triggering analysis of one or more records in the data record set. Disclosed embodiments trigger the analysis only in response to determining that the predetermined conditions have been met. Upon triggering the analysis of the data record set, disclosed embodiments identify a subset of the data record set to undergo the analysis while refraining from performing the analysis on the remaining records in the data record set. Further, embodiments identify an analysis model based on a level of analysis to be performed and apply the analysis model to the subset of the data record set to identify any presence of sensitive data. Lastly, disclosed embodiments selectively perform a security process to the data record set in response to detecting the presence of the sensitive data.