Multithreaded Telemetry Enrichment for Reduced Lock Contention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Processing large volumes of telemetry data for cybersecurity analysis poses challenges, particularly in enriching telemetry messages to support time-critical detection, and existing methods struggle with lock contention and inefficient distribution of enrichment data across threads.
Innovation Solution
A multithreaded enrichment processing architecture uses multiple threads to extract and distribute enrichment data across shared memory resources, employing a mapping function for uniform distribution and reducing lock contention through independently lockable storage buckets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple threads share the same storage bucket for enrichment data, then data consistency is improved, but lock contention increases and processing speed decreases
Solution Approach 1:
The patent divides the single shared storage bucket into multiple separate storage buckets, with each bucket dedicated to a specific thread. This segmentation eliminates lock contention between threads while maintaining data consistency within each thread's dedicated bucket, thereby resolving the contradiction between reliability and productivity
Solution Approach 2:
The patent introduces a load balancer as an intermediary component that receives incoming telemetry messages and distributes them to appropriate threads based on load balancing algorithms. This intermediary manages the distribution of work across threads without requiring them to contend for shared resources, improving processing speed while maintaining consistency
2Reliability
If a single thread processes all telemetry messages sequentially, then data consistency is maintained, but processing speed and throughput decrease
Solution Approach 1:
The patent segments the single-threaded processing model into multiple parallel threads, each capable of independently processing telemetry messages. By assigning dedicated storage buckets to each thread, the system maintains data consistency while achieving parallel processing throughput
Solution Approach 2:
The patent implements dynamic thread pool management where threads can be created, activated, or deactivated based on system load and processing requirements. This dynamic approach allows the system to scale processing capacity while maintaining consistency through controlled access patterns
3Speed
If enrichment data is stored in shared memory accessible by all threads, then access efficiency is improved, but lock contention and synchronization overhead increase
Solution Approach 1:
The patent segments shared memory into multiple thread-specific storage buckets, allowing each thread to access its dedicated bucket without synchronization. This eliminates lock contention and reduces synchronization overhead while maintaining efficient access speeds through localized memory access
Solution Approach 2:
The patent assigns different storage characteristics to different threads through their dedicated buckets. Each thread has exclusive access to its own storage bucket, providing local quality that eliminates the need for global synchronization mechanisms and reduces overhead
Data Source
AI summary
Enriched telemetry data is generated via multithreaded enrichment processing. A first processing thread extracts from a first telemetry message a piece of enrichment data and a first network address pair, and selects a storage bucket in shared memory using the first network address pair. The first thread temporarily locks the storage bucket to store the piece of enrichment data. A second processing thread extracts from a second telemetry message a matching network address pair, identifies the storage bucket using the the second network address pair, determines whether the bucket is locked, and if not, retrieves the piece of enrichment data from the storage.