Stream Data Migration via Input Server Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing stream data processing systems face challenges in migrating data without halting input of new stream data, leading to impaired real-time properties and potential data loss, with data coincidence issues between migration sources and destinations causing prolonged migration times.
Innovation Solution
A stream data processing system with an input server that accumulates and transfers data to a migration destination server, stops data transfer to the source server upon completion, and resumes data transfer after ensuring data coincidence, allowing intermediate values and query results to be migrated efficiently, thereby reducing migration time and preventing data loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the copying method is used to migrate stream data, then data can be transferred to the migration destination, but the input of new stream data must be temporarily halted, impairing real-time properties
Solution Approach 1:
The patent applies preliminary action by accumulating stream data in a buffer before migration begins. The input server collects and stores incoming stream data during the migration process, allowing the migration to proceed without halting the input of new data. This buffer acts as a temporary storage that decouples the input process from the migration process, maintaining real-time data flow while enabling data transfer to the migration destination.
2Reliability
If the copying method is used to migrate stream data, then data can be transferred, but unless the halted stream data is stored, it may be lost
Solution Approach 1:
The patent uses preliminary action by pre-accumulating stream data in a buffer before migration starts. This buffer stores incoming data during the migration process, ensuring that no data is lost while the migration proceeds. The buffer serves as a temporary holding area that prevents data loss without requiring the system to wait for migration completion before accepting new data.
3Reliability
If the warm-up method is used, then stream data can be transmitted to both migration source and destination, but data according to execution of queries may not coincide for hours
Solution Approach 1:
The patent applies the intermediary principle by introducing a buffer as a mediating component between the migration source and migration destination. The buffer receives stream data from the source and releases it to the destination in a controlled manner, ensuring that data coincidence is achieved much faster than the warm-up method. This intermediary buffer coordinates the data flow between source and destination, preventing the hours-long discrepancy experienced in the warm-up approach.
Data Source
AI summary
The stream data processing system has an input server which accumulates input stream data to be migrated, instructs a migration source and a migration destination to perform data migration, transfers accumulated data to the migration destination in response to an accumulated data transfer instruction, then stops transferring data to the migration source, and transfers subsequent data to the migration destination subsequently to the transfer of the accumulated data. The migration source migrates both a query and an intermediate value that has been produced as a result of completion of execution of the query on input data. A migration destination server reads the accumulated data and the subsequent data that are transferred from the input server, skips reading the portions of these transferred data corresponding to and preceding the migrated intermediate value, and discards the output that is transferred from the migration source and precedes the query execution results.