Server Data Collection Wagon Error Recovery Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current big data collection systems face issues with data loss and redundancy, particularly when system errors occur, as solutions like Flume and Nifi use memory buffers for speed but risk data loss and buffer files for stability, leading to instability and redundancy.
Innovation Solution
A server architecture that includes a gatherer, wagon, historian file, and destination folder, allowing the wagon to enter a ROLLBACK state and re-collect data upon errors, with the historian file deleting collection histories and the temporary folder moving data to the destination folder to prevent redundancy and ensure data integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If memory buffer is used for increasing processing speed, then processing speed is improved, but data loss risk increases
Solution Approach 1:
The patent applies preliminary action by recording file collection histories in the historian file before actual data collection occurs. This allows the system to restore collection status after errors without losing track of what needs to be collected, thus maintaining both speed and reliability.
Solution Approach 2:
The patent uses copying by maintaining a historian file that stores copies of file collection histories. This copy mechanism allows the system to recover from errors by referencing the stored history, preventing data loss while maintaining processing speed through quick restoration.
2Reliability
If buffer file is used for securing stability, then data stability is improved, but data redundancy increases
Solution Approach 1:
The patent implements feedback by checking the historian file to determine whether files have been successfully collected before proceeding with collection. This feedback mechanism prevents redundant data collection while maintaining stability, as the system only collects data that hasn't been successfully stored yet.
Solution Approach 2:
The system performs preliminary checking of the historian file before data collection to determine if collection is needed. This preliminary action prevents redundant collection operations, reducing data redundancy while maintaining stable data collection processes.
3Reliability
If data collection is restarted after system error, then data recovery is improved, but time consumption increases
Solution Approach 1:
The patent uses preliminary action by maintaining pre-recorded file collection histories in the historian file. When errors occur, the system can quickly restore collection status by referencing this pre-existing history rather than重新开始, thus improving recovery speed while maintaining data integrity.
Solution Approach 2:
The historian file serves as a copy of the collection status that can be quickly restored after errors. This copying mechanism enables rapid data recovery without time-consuming re-collection of already processed files, reducing overall time consumption while ensuring complete data recovery.
4Reliability
If file collection history is maintained, then data integrity is improved, but storage overhead increases
Solution Approach 1:
The patent extracts only the essential file collection history information needed for recovery purposes into a separate historian file. This extraction approach maintains data integrity by preserving necessary collection status information while minimizing storage overhead by storing only critical metadata rather than full data copies.
Data Source
AI summary
Disclosed is a server. A server according to an embodiment of the present disclosure includes a gatherer configured to receive a file collection list from an external server, at least one wagon configured to collect data from the external server based on the file collection list received from the gatherer, a historian file configured to store a transmission history or a file collection history, and a destination folder configured to store the data collected by the wagon, and the wagon is further configured to, when an error occurs during data collection, collect data for a file again in which the error occurs. Accordingly, it is possible to quickly collect data again when an error occurs.