Disk Based Queue With Checksums And Sub-Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current queue processing systems face challenges in scalability and data integrity, particularly in automated and distributed industrial and commercial processes, where data tampering due to power or hardware failures can occur, and efficient data delivery with prioritization is complex.
Innovation Solution
A scalable disk-based queue system is implemented with a delay factor, using checksums for data integrity, mapping the queue head and tail with primary storage for faster computations, and dividing the queue into sub-queues based on a time period for network file system scalability, enabling efficient data processing and tampering detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a traditional queue system is used for automated processes, then data processing can be performed, but data integrity is compromised due to potential tampering from power or hardware failures
Solution Approach 1:
The patent applies preliminary action by computing and storing checksums for each data element before it is enqueued. This pre-computed integrity verification data is stored alongside the queue data, enabling detection of any tampering or corruption that may occur during storage or transmission. When data is dequeued, the checksum is automatically verified to ensure integrity, preventing propagation of corrupted data through the system.
2Productivity
If queue size is increased to handle larger data volumes, then processing capacity improves, but scalability is limited by memory constraints
Solution Approach 1:
The patent implements segmentation by dividing the queue into multiple segments or partitions that can be independently managed. Each segment can be stored in separate memory locations or even across different storage devices, allowing the queue to scale beyond single memory constraints. This segmentation enables the queue to grow horizontally by adding more segments while maintaining efficient access patterns through proper indexing and pointer management.
3Duration of action of moving object
If data is stored in queue for longer periods to achieve delivery goals, then service delivery is improved, but data tampering risk increases
Solution Approach 1:
The patent applies feedback by implementing continuous integrity monitoring through checksum verification. Each time data is accessed, modified, or transferred within the queue system, the checksum is automatically recalculated and compared against the stored checksum value. This feedback mechanism provides immediate detection of any tampering or corruption, alerting the system to potential issues and preventing further propagation of compromised data throughout the retention period.
4Reliability
If checksum verification is performed on every data element, then data integrity is ensured, but computational overhead increases
Solution Approach 1:
The patent applies partial action by implementing selective checksum verification based on risk assessment and data criticality. Not every data element undergoes full checksum verification at each access point - instead, the system performs verification at strategically chosen checkpoints and for high-priority data elements. This approach maintains adequate integrity assurance while reducing the overall computational overhead compared to verifying every single data element at every access point.
Data Source
AI summary
The present disclosure provides a system for performing a method for implementation of a scalable queue with a delay factor. The method includes a first step of creating a disk based queue. The method includes another step of mapping head of the disk based queue and tail of the disk based queue with a primary storage. The method includes yet another step of adding checksum in between data inside the disk based queue. The method includes yet another step of dividing the disk based queue in to a plurality of sub-queues with the delay factor. Furthermore, the checksum is added for identifying tampering in data, maintaining persistence and throughput in the disk based queue. Moreover, the delay factor is achieved by dividing the disk based queue in to the plurality of sub-queues based on a time period. The disk based queue is highly scalable using a network file system.