Persistent Memory for Consistent Batch and Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In database systems that support both streaming and batch processing, there is a lack of consistency between the data used for streaming processing and batch processing due to separate data paths, leading to inconsistencies caused by potential failures in writing data to the database for batch processing.
Innovation Solution
Implementing a system that uses an intermediary non-volatile memory to validate and store incoming data, ensuring it is consistent for both streaming and batch processing, and then sending this validated data to a remote disk for batch processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If separate data paths are used for streaming and batch processing to enable faster access for streaming, then streaming processing speed is improved, but data consistency between streaming and batch processing deteriorates
Solution Approach 1:
The patent introduces an intermediary component called a data consistency manager that mediates between the streaming data path and batch processing data path. This manager validates data integrity and ensures that the same data is used in both processing paths, resolving the consistency issue while maintaining separate fast paths for streaming and batch processing
2Adaptability or versatility
If data is written to database for batch processing, then batch processing functionality is enabled, but data consistency with streaming processing deteriorates due to potential write failures
Solution Approach 1:
The patent implements preliminary validation of data integrity before data is written to the database for batch processing. The data consistency manager checks data validity in advance and only allows consistent data to proceed to the database, preventing write failures from causing consistency issues between streaming and batch processing
3Manufacturing precision
If redundancy check is performed on incoming data, then data quality for batch processing is improved, but data writing to database may fail causing inconsistency with streaming processing
Solution Approach 1:
The patent implements a feedback mechanism where the data consistency manager monitors both the streaming processing path and batch processing path, and uses this feedback to ensure that only data that successfully passes redundancy checks in streaming processing is also used in batch processing. This feedback loop maintains consistency even when redundancy checks cause data writing failures
Data Source
AI summary
Systems, methods, and computer-readable media are provided for consistent data to be used for streaming and batch processing. The system includes one or more devices; a processor coupled to the one or more devices; and a non-volatile memory coupled to the processor and the one or more devices, wherein the non-volatile memory stores instructions that are configured to cause the processor to perform operations including receiving data from the one or more devices; validating the data to yield validated data; storing the validated data in a database on the non-volatile memory, the validated data being used for streaming processing and batch processing; and sending the validated data to a remote disk for batch processing.