Database Archiving via Dual-Thread Queue for Monitoring Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Integrated supervisory systems in public transportation, such as subways and urban rails, face inefficiencies in data storage due to varying numbers of monitoring points and mismatched database server configurations, leading to suboptimal data archiving and storage efficiency.
Innovation Solution
A data archiving method involving the acquisition of monitoring data, storage as multiple data objects, and a two-threaded process where one thread reads data into a queue and another writes it to the database, with a mechanism to monitor and adjust storage load rates to prevent data accumulation and loss.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If relational databases (Oracle, MySql, MSSQL) are used for data storage with fixed configurations, then data storage is achieved, but data storage efficiency and creating efficiency of monitoring data become mismatched due to varying numbers of monitoring points across different routes
Solution Approach 1:
The patent implements dynamic database configuration by allowing database parameters to be automatically adjusted based on the number of monitoring points. The system dynamically selects appropriate database configurations and parameters according to the specific route's monitoring scale, enabling both high creating efficiency for routes with many monitoring points and optimized storage efficiency for routes with fewer monitoring points.
Solution Approach 2:
The patent changes database parameters dynamically based on the number of monitoring points. Different database configurations (e.g., buffer sizes, connection pool settings, archiving intervals) are selected according to the scale of monitoring data, allowing the system to optimize both data creation speed and storage efficiency for each specific route scenario.
2Productivity
If database server configuration and parameters are customized for each route, then storage efficiency may be optimized, but system complexity increases due to varying configurations across multiple routes
Solution Approach 1:
The patent implements automatic database configuration where the system self-adjusts database parameters based on the number of monitoring points. The database management system automatically selects appropriate configurations and parameters without requiring manual intervention for each route, thereby maintaining storage efficiency optimization while reducing system complexity and configuration burden.
3Reliability
If monitoring data is stored and processed in real-time, then data availability is improved, but data loss may occur due to data accumulation when write speed cannot keep up with collection speed
Solution Approach 1:
The patent implements preliminary data archiving by pre-processing and archiving monitoring data before it accumulates to problematic levels. The system proactively archives data in the background according to predefined rules and thresholds, ensuring data is stored safely before the write speed becomes a bottleneck, thereby preventing data loss while maintaining real-time data availability for monitoring purposes.
Solution Approach 2:
The patent implements a feedback mechanism that monitors the relationship between data collection speed and write speed. When the system detects that write speed cannot keep up with collection speed, it triggers archiving operations to prevent data loss. This feedback loop ensures data reliability by dynamically adjusting archiving behavior based on real-time performance metrics.
Data Source
AI summary
A data archiving method, an electronic device, and a readable storage medium relate to the field of the database technology. The method includes: acquiring monitoring data collected by at least one monitoring subsystem; storing, according to a preset rule, the monitoring data as multiple data objects corresponding to the monitoring subsystem; controlling a first thread corresponding to the monitoring subsystem to read the data objects into a corresponding data queue according to a preset order; and controlling a second thread corresponding to the monitoring subsystem to write the data objects from the data queue into a database in response to the first thread reading the data objects into the data queue each time.