Shared Memory Queue for Inter-Process Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Surveillance server systems face challenges in managing high volumes of data from multiple sources, leading to potential security compromises due to instability and lack of scalability and flexibility, which can result in data loss and compromised security.
Innovation Solution
A surveillance system design that separates command and control modules from archiving modules, utilizing a data queue in shared memory for inter-process communication, allowing high data throughput and predictive data degradation to prioritize critical data, while using a multiple process synchronization system to maintain system performance and prevent data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a robust server system is designed to manage large quantities of surveillance data, then reliability is improved, but scalability and flexibility deteriorate
Solution Approach 1:
The server system is divided into multiple independent processes (input processes and reader processes) that communicate through a shared memory queue. Each process handles specific tasks independently, allowing the system to scale by adding more processes without compromising the reliability of individual components. The command and control module is separated from the archiving module, enabling independent development, deployment, and scaling of different functional components.
2Productivity
If multiple input processes write simultaneously to a shared data queue, then productivity is improved, but data integrity may deteriorate due to potential simultaneous writes to the same location
Solution Approach 1:
A synchronization mechanism acts as an intermediary between multiple input processes and the shared data queue. This intermediary coordinates access to the queue, ensuring that while multiple processes can write simultaneously (maintaining high productivity), conflicts are resolved through synchronization primitives that prevent data corruption and maintain integrity.
3Loss of information
If the reader process reads all data from the queue, then completeness is improved, but system responsiveness deteriorates when the queue is full and input processes must wait
Solution Approach 1:
The system changes the parameter of data priority classification, dividing data into different priority levels. When the queue is full, the system can selectively drop or defer lower-priority data while maintaining high-priority data, allowing input processes to continue operating without blocking while still ensuring critical data is preserved.
4Reliability
If critical data is prioritized over less important data, then reliability is improved, but data loss increases for non-critical data
Solution Approach 1:
The system applies different quality treatments to different portions of data based on their priority classification. Critical data receives guaranteed processing and preservation, while non-critical data is subject to selective dropping when resources are constrained. This local differentiation of data treatment ensures reliability for important data while accepting controlled loss for less important data.
Data Source
AI summary
Described is a novel method of inter-process communication used in one example in a surveillance system whereby multiple input processes communicate surveillance data to a reader process that consumes the data from the input processes. A locking mechanism is provided to reserve a reservable portion of queue metadata which comprises queue pointer(s) such that only one process may move the queue pointer(s) at a time. Reservation is provided with little or no kernel operations such that reservation costs are negligible. Arbitrary size queue slots may be reserved by moving the points. Writing and reading into the queue is done outside of the locking mechanism allowing multiple processes to access and work in the queue simultaneously leading to a rapid queue synchronization mechanism that requires little or no resort to expensive kernel operations.