Database Upscaling Using Incremental Data Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database upscaling and downscaling operations are manually performed by operations personnel without visibility into data distribution patterns, leading to significant resource consumption and potential online service disruptions due to inaccurate time estimation.
Innovation Solution
A database management method and apparatus that includes acquiring incremental data and metadata, determining upscaling conditions, executing data validation, and performing data migration to new databases, ensuring accurate and rapid scaling without downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If manual upscaling and downscaling operations are performed by operations personnel, then database scaling can be performed, but resource consumption increases and service disruptions may occur due to inaccurate time estimation
Solution Approach 1:
The system performs preliminary actions by automatically acquiring incremental data and metadata before scaling operations, determining upscaling conditions, and executing data validation in advance. This preparation work is done automatically without manual intervention, ensuring accurate time estimation and preventing service disruptions during the actual scaling operation.
Solution Approach 2:
The database management system performs self-service through automated incremental data acquisition, automatic determination of upscaling conditions based on metadata lists, and self-executed data validation. This eliminates reliance on manual operations personnel, reducing resource consumption while maintaining service continuity through precise automated timing and execution.
2Measurement precision
If data validation is executed on incremental data according to metadata list, then data accuracy is improved, but processing time increases
Solution Approach 1:
The system applies partial action by executing data validation selectively based on determined upscaling conditions rather than validating all data unconditionally. The validation is performed on incremental data according to metadata lists only when upscaling conditions are met, achieving necessary data accuracy while minimizing unnecessary processing time.
3Manufacturing precision
If data migration is performed on validated incremental data to complete database upscaling, then scaling accuracy is improved, but system complexity increases
Solution Approach 1:
The system segments the database scaling process into distinct phases: acquiring incremental data and metadata, determining upscaling conditions, executing data validation, and performing data migration. This segmentation allows each step to be handled independently with specific rules, improving scaling accuracy while managing system complexity through modular processing.
Solution Approach 2:
The metadata list serves as an intermediary between the incremental data and the validation/migration processes. It provides structured information that guides the validation and migration operations, ensuring accurate scaling while simplifying the overall system architecture by mediating between data sources and processing operations.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Provided in the embodiments of the present disclosure are a database management method and apparatus. The database management method comprises: first, acquiring, based on a current time, first incremental data of a database and a metadata list of the first incremental data; then, determining, based on the first incremental data, whether the database meets upscaling conditions; next, in response to determining that the database meets the upscaling conditions, executing a data validation on the first incremental data according to the metadata list to obtain validated incremental data; and finally, executing a data migration on the validated incremental data based on a new database and the database to complete database upscaling management.