Central-Cluster Indicator Data Processing for Long-Term Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage methods in edge clusters for applications like video on demand and cloud gaming result in poor performance and ineffective data transmission due to large data volumes and limited data life cycles, hindering efficient data management and storage.
Innovation Solution
Implementing a central cluster with transceiver, coordinated write, and database components to pre-aggregate and convert indicator data into a target storage format, optimizing data transmission and storage by reducing data amounts and extending data life cycles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If indicator data is periodically pulled and stored by service endpoint in edge cluster, then data can be collected for monitoring service performance, but data storage performance deteriorates and data transmission becomes ineffective due to large data volumes
Solution Approach 1:
The patent segments the data storage system into multiple components: edge nodes that generate data, a data collection service that aggregates data from multiple edge nodes, and a database service that stores the aggregated data. This segmentation allows data to be collected and processed in a distributed manner, improving storage performance while maintaining monitoring reliability.
Solution Approach 2:
The patent introduces a data collection service as an intermediary between edge nodes and the database service. This intermediary aggregates indicator data from multiple edge nodes before storing it in the database, reducing the overall data volume and improving storage efficiency while maintaining complete monitoring coverage.
2Reliability
If large quantity of indicator data is generated and stored in service endpoint, then comprehensive monitoring data is available, but data transmission efficiency deteriorates and data life cycle is limited
Solution Approach 1:
The patent merges data from multiple edge nodes at the data collection service before transmission to the database. By combining data at an intermediate point rather than having each edge node transmit separately, the system reduces redundant transmission and improves overall data transmission efficiency while maintaining complete monitoring data.
3Ease of operation
If indicator data is deleted after a period of time to ensure normal service endpoint running, then service stability is maintained, but data life cycle is limited and long-term data retention cannot be achieved
Solution Approach 1:
The patent extracts the data storage function from the edge node service endpoint and places it in a dedicated database service. This allows the service endpoint to maintain stability by not storing large volumes of data locally, while the database service handles long-term data retention separately, extending the data life cycle without affecting service endpoint operation.
Data Source
AI summary
A data processing method, apparatus, and system, a computer device, a readable storage medium, and a computer program product relate to the field of cloud technologies and a blockchain technology, and the method includes: receiving, by using a transceiver component, collection indicator data sent by an edge cluster; performing pre-aggregation processing on the collection indicator data to obtain pre-aggregated indicator data, and sending the pre-aggregated indicator data to a coordinated write component; converting, by the coordinated write component, the pre-aggregated indicator data into conversion indicator data that has a target storage format, and performing merging processing on the conversion indicator data to obtain storage indicator data; writing the storage indicator data into a database component; and writing, by the database component, the storage indicator data into a storage disk.