Telemetry Sampling for Sensitive Data Scanning in Log Records
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing log telemetry data management techniques expose sensitive information, leading to security and legal risks, and traditional methods like encryption or purging data impede troubleshooting and are inefficient.
Innovation Solution
Selective and dynamic sampling of data records during generation, triggered by predetermined conditions, to identify and analyze only a subset of records using various analysis models, applying security processes only when sensitive data is detected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional encryption or purging methods are applied to log telemetry data, then security risks are reduced, but troubleshooting efficiency and data integrity are impaired
Solution Approach 1:
The patent applies partial action by selectively scanning only a subset of data records rather than all records. The system dynamically determines a scan ratio and selects specific records for security scanning, thereby reducing the impact on troubleshooting efficiency while still maintaining security for critical data.
Solution Approach 2:
The patent implements local quality by applying different security scanning strategies to different portions of the data. High-priority data fields such as credentials, personally identifiable information, and financial data are scanned with higher intensity, while less sensitive data is scanned with lower intensity or excluded from scanning.
2Reliability
If comprehensive security scanning is performed on all log telemetry data, then security coverage is improved, but resource consumption and processing time increase
Solution Approach 1:
The system performs security scanning on only a partial subset of data records by dynamically determining a scan ratio between 0 and 1. This reduces resource consumption while maintaining adequate security coverage for the most critical data.
Solution Approach 2:
The patent segments the data scanning process into priority levels. Data fields are categorized by sensitivity and priority, and the scanning process is divided into multiple passes or stages, with high-priority fields scanned first using available resources.
Solution Approach 3:
The scanning process is made dynamic through adaptive adjustment of the scan ratio based on available system resources, data priority levels, and security policies. The system can dynamically increase or decrease scanning intensity depending on current operational conditions.
3Productivity
If dynamic sampling and selective scanning are implemented, then resource efficiency is improved, but system complexity increases
Solution Approach 1:
The system performs preliminary actions by pre-classifying data fields into priority categories and pre-determining scan ratios based on security policies and resource availability. This preliminary setup reduces the complexity of real-time decision-making during the scanning process.
Solution Approach 2:
The patent implements self-service mechanisms where the system automatically adjusts scanning parameters based on monitored resource conditions and data characteristics, reducing the need for complex external configuration and management.
4Speed
If selective scanning of data subsets is performed, then processing speed is improved, but detection precision may be reduced
Solution Approach 1:
The patent applies local quality by concentrating scanning resources on high-priority data fields with higher detection precision, while using faster, less precise methods for lower-priority fields. This ensures that critical sensitive data is detected accurately while maintaining overall processing speed.
Solution Approach 2:
The system applies partial action by performing comprehensive scanning on a selected subset of high-priority records, ensuring high detection precision for the most critical data while accepting that not all records are scanned with the same level of scrutiny.
Data Source
AI summary
Disclosed systems and methods identify a data record set and determine whether one or more predetermined conditions exist for triggering analysis of one or more records in the data record set. Disclosed embodiments trigger the analysis only in response to determining that the predetermined conditions have been met. Upon triggering the analysis of the data record set, disclosed embodiments identify a subset of the data record set to undergo the analysis while refraining from performing the analysis on the remaining records in the data record set. Further, embodiments identify an analysis model based on a level of analysis to be performed and apply the analysis model to the subset of the data record set to identify any presence of sensitive data. Lastly, disclosed embodiments selectively perform a security process to the data record set in response to detecting the presence of the sensitive data.