Data repairing method and related apparatus

Abstract

A data repairing method and a related apparatus, applied to the field of data management and maintenance. In the present application, when a data management system is activated, a data repair apparatus obtains repair configuration information, the repair configuration information comprising indication information of a first scan mode; the data repair apparatus scans, at a scanning precision corresponding to the first scan mode, data blocks storing data of the data management system, so as to detect an abnormal data block; and the data repair apparatus cleans the detected abnormal data block, thereby repairing an anomaly of the data management system. In this way, the present application enables prompt repair of the data management system. Furthermore, the scanning precision level corresponding to the first scan mode is pre-configured, such that data repair work can be performed in a targeted manner, thereby improving data repair efficiency and helping maintain the reliability and availability of the data management system.

Description

A data repair method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510138112.1, filed on February 7, 2025, entitled "A Data Repair Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of data management technology, and in particular to a data repair method and related apparatus. Background Technology

[0003] With the rapid development of internet and IoT technologies, data management systems have become an indispensable core component of modern business systems, widely used in e-commerce, finance, healthcare, transportation, and other fields. Data in a data management system is stored in data blocks. During operation, data management systems may experience anomalies such as power outages, hardware failures, and network failures. These anomalies can easily generate abnormal data blocks, which can affect the functionality of the data management system, necessitating its repair.

[0004] Due to the sheer volume of data and the complexity of data management systems, detecting anomalies is quite challenging. In some solutions, users trigger system checks and cleanup of abnormal data blocks only after discovering the system malfunctions. This often results in delayed data repair efforts, severely impacting the reliability and availability of the data management system.

[0005] Therefore, improving the efficiency of data repair and maintaining the reliability and availability of data management systems are urgent problems to be solved in this field. Summary of the Invention

[0006] This application provides a data repair method and related apparatus, which can scan data blocks storing data from a data management system at a scanning precision corresponding to a first scanning mode to detect abnormal data blocks, and clean up the detected abnormal data blocks, thereby repairing the anomalies in the data management system. Furthermore, the scanning precision corresponding to the first scanning mode is pre-configured, which allows the data repair work to be performed more specifically, thereby improving the efficiency of data repair and contributing to maintaining the reliability and availability of the data management system.

[0007] Firstly, this application provides a data repair method applied to a data repair device. The data repair device is used to repair a data management system, which stores data in a file system. The data management system and the file system belong to a data storage system. The data repair device can be a standalone device, such as a computing device, or a software and / or hardware module within a standalone device, such as a chip, circuit, or computer program. For ease of description, the following description focuses on the data repair device as the implementing entity.

[0008] The data repair method includes: upon starting the data management system, the data repair device acquires repair configuration information, including indication information for a first scan mode. This first scan mode indicates the precision of scanning data blocks in the distributed file system. The first scan mode belongs to one of multiple scan precision levels. The data management system includes at least one data table for storing data. Each data table forms at least one logical storage unit, and each logical storage unit corresponds to at least one data block in the distributed file system. The data repair device scans the data blocks in the distributed file system using the first scan mode to obtain bad block scan results. These bad block scan results include indication information for at least one abnormal data block, indicating abnormal data reading and / or data writing. The data repair device cleans up at least one abnormal data block.

[0009] In the above embodiments, the data management system is the core of data management. It includes at least one data table, and the data in each data table can be divided into at least one logical storage unit. The data in these logical storage units is stored in the data blocks of the file system. Those skilled in the art will understand that when the data management system experiences an anomaly, such as hardware failure, network failure, or abnormal power outage, data reading and writing operations will be interrupted, resulting in abnormal data blocks in the file system. For example, when an abnormal power outage occurs, data may be lost before it can be persisted to the data blocks of the distributed file system. This results in abnormal data blocks in the file system that cannot be read and / or written normally, which will significantly affect the reliability of the data management system service.

[0010] In the above solution, the data repair device can automatically detect whether there are any anomalies in the file system's data blocks when the data management system starts up, and clean up the abnormal data blocks, ensuring the normal operation of the data management system. In this way, data repair can be performed more promptly, which helps improve the reliability and availability of the data management system. Furthermore, automated scanning and repair are highly repeatable and require no manual intervention, thus improving the efficiency of data repair.

[0011] Furthermore, the above solution supports defining the precision of scanned data blocks and includes information indicating the scan precision in the repair configuration information, making the precision of data block scanning controllable. This allows different storage systems and business systems to use reasonable scan precision, achieving a balance between scan efficiency and anomaly location accuracy, and significantly improving the user experience of data repair functions.

[0012] This application scans data blocks in the file system according to the first scan mode instruction information in the repair configuration information to detect abnormal data blocks. The method for detecting abnormal data blocks in this application is automated and requires no manual intervention. Compared with the traditional method of manually detecting bad blocks (i.e., abnormal data blocks), the solution provided in this application has high repeatability, strong adaptability, and can save labor costs.

[0013] For example, the file system can be a distributed file system, such as HDFS. In this case, the data management system can be a data management system based on the distributed file system, such as HBase. Taking HBase as an example of a data management system, the logical storage unit can be a region.

[0014] In one possible implementation of the first aspect, the multiple scanning accuracy levels include: a first level, checking if header information of the data block exists; a second level, checking if a portion of the data in the data table within the data block can be read; and a third level, checking if all data in the data table within the data block can be read.

[0015] In the above implementation, different scanning precision levels correspond to different levels of scanning accuracy. Higher scanning accuracy leads to a higher accuracy rate in detecting abnormal data blocks, but also consumes more resources and takes longer to repair. This approach allows for flexible scanning of data blocks at different precision levels based on different situations and needs, finding a balance between scanning efficiency and accuracy, thereby improving the flexibility of data repair. For example, if only a macroscopic understanding of the distribution and approximate number of abnormal data blocks in the data management system is needed, Level 1 can quickly perform preliminary location and save system resources. Alternatively, if a balance between efficiency and accuracy is required, Level 2 can be used for data block scanning. Furthermore, for situations requiring a comprehensive and detailed examination of data blocks, such as repairing data in financial transaction systems or medical data systems, high-precision scanning is crucial for ensuring data quality, and Level 3 can be employed.

[0016] In another possible implementation of the first aspect, the data repair method further includes: after the data management system is started, the data repair device updates a first identifier so that the first identifier indicates that the data management system has started normally. When the data management system is started, if the first identifier indicates that the data management system has started normally, the first identifier is also used to indicate that the data management system previously stopped abnormally. When the data management system enters a normal shutdown process, the data repair device updates the first identifier so that the first identifier indicates that the data management system has stopped normally.

[0017] The above embodiments provide a first identifier update mechanism. Using this mechanism, a clear indication can appear on the first identifier when the system abnormally stops. Therefore, the data repair device can quickly and accurately determine whether the system experienced an abnormal stop in the past using the first identifier. Timely repair of the data management system ensures its reliability and availability.

[0018] In some solutions, anomalies in the data management system can only be detected after the system has been running and issues such as slow response or errors occur, which can disrupt the normal operation of business systems. This application, however, enables rapid anomaly detection, facilitating timely remediation and improving the reliability of business systems.

[0019] In another possible implementation of the first aspect, the data repair method further includes: when the first identifier indicates that the data management system has stopped abnormally, the data repair device writes an exception record to a distributed coordination service component, which is used to record the abnormal state information of the data management system.

[0020] In the above implementation, the distributed coordination service component records abnormal states of the data management system. In this way, operations and maintenance personnel can check the data management system based on the abnormal state information recorded in the distributed coordination service component, and optimize the data management system based on the inspection results. This helps maintain the stability and reliability of the data management system services. Furthermore, in actual implementation, the distributed coordination service component is also used to manage the components of the data management system. When the distributed coordination service component records an abnormal state of the data management system, it can take timely adjustment measures to help maintain the stability of the data management system.

[0021] In another possible implementation of the first aspect, the repair configuration information further includes repair mode indication information, and the data repair method further includes: the data repair device determining, at least based on the repair mode indication information, whether to repair the data management system. If the data management system is to be repaired, the operation of "scanning data blocks in the distributed file system in a first scan mode to obtain bad block scan results" is performed.

[0022] In the above implementation, the repair mode includes scenarios for repairing the data management system. For example, data repair can be performed on the data management system in all scenarios, or only when the data management system stops abnormally, and not at all. In this way, data repair can be performed according to specific circumstances or needs, which improves the flexibility of data repair.

[0023] In another possible implementation of the first aspect, the data repair device determines whether to repair the data management system based at least on the indication information of the repair mode, including the following operations: the data repair device determines whether to repair the data management system based on the indication information of the repair mode and a first identifier, wherein the first identifier is used to indicate whether the data management system stopped abnormally during its last run.

[0024] In the above embodiments, data repair of the data management system needs to consider the repair mode and the abnormal conditions of the data management system. For example, in some cases, the impact of an anomaly on the data management system's services is minimal; for instance, the core business functions of the data management system can still operate normally after an anomaly, or the risks caused by the anomaly are potentially controllable. In such cases, data repair might waste unnecessary resources and costs, so choosing not to repair the data management system is more appropriate. As another example, in some situations, the stability requirements of the data management system are high, requiring all services to operate normally. Therefore, the repair mode can be set to perform repair in all situations to maintain system stability. In this way, data repair can be performed according to the needs of the data management system's services, which improves the efficiency and flexibility of data repair.

[0025] In another possible implementation of the first aspect, the repair mode is either all scenarios or abnormal stop. The data repair device determines whether to repair the data management system based on the repair mode indication information and the first identifier, including: when the first repair mode is all scenarios, the data repair device determines to repair the data management system; or, when the first repair mode is abnormal stop and the first identifier indicates that the data management system abnormally stopped during its last run, the data repair device determines to repair the data management system.

[0026] In the above implementation, data repair is performed in all scenarios or when the data management system malfunctions. For data management systems with high security and stability requirements, performing data repair in all scenarios can better maintain the accuracy, integrity, and consistency of data, ensuring the smooth operation of the data management system services.

[0027] In some cases, an abnormal shutdown of a data management system can lead to data anomalies, significantly reducing service reliability. Configuring data repair in abnormal situations allows the data management system to quickly return to a stable state, maintaining normal service and minimizing data loss. Furthermore, if no anomalies are detected, the data management system can skip the repair process, allowing it to quickly return to operational status.

[0028] In another possible implementation of the first aspect, the data repair device determines whether to repair the data management system based on the repair mode indication information and the first identifier, further comprising: determining not to repair the data management system when the first repair mode is closed repair; or, determining not to repair the data management system when the first repair mode is abnormal stop and the first identifier indicates that the data management system did not stop abnormally during its last run.

[0029] In the above implementation, disabling data repair means not performing data repair under any circumstances. For systems with low requirements for the data management system, or systems where the core functions of the data management system can still operate normally in the event of an anomaly, performing data repair would lead to greater resource consumption, time consumption, and cost consumption. In these scenarios, choosing not to perform data repair is a more appropriate choice. For some systems, an abnormal shutdown will affect the reliability of the data management system service, while other situations will not affect the data management system. Therefore, for these systems, it is possible to configure data repair to only be performed in the event of an abnormal shutdown, and not perform data repair in other situations.

[0030] In yet another possible implementation of the first aspect, the data repair method further includes: operating the data management system normally if it is determined that no repair will be performed on the data management system.

[0031] In the above implementation, for data management systems that do not require data repair, the data management system can operate normally. This can reduce the resource consumption, time consumption and cost consumption of data repair, and also enable business systems to provide services as soon as possible, reducing the impact on services.

[0032] In another possible implementation of the first aspect, the repair configuration information further includes indication information on the repair timing, and the data repair method further includes: repairing the data management system at the repair timing.

[0033] In the above implementation, the timing of data repair must be considered when performing data repair on the data management system. Repairing the data management system at the appropriate time helps ensure data quality, maintain the reliability of the data management system, and enhance its stability.

[0034] In another possible implementation of the first aspect, the timing of the repair includes: before the management service in the data management system is started, or during the operation of the management service in the data management system.

[0035] In the above implementation, the data management system's management service is responsible for managing the data management system's nodes, such as logical storage unit servers, and plays a crucial role in providing services. The timing of data management system repair includes before and during the startup of the management service. For some complex data management systems or those with large amounts of data, performing data repair before the management service starts can prevent crashes or startup failures, ensuring a smooth startup and operation of the management service, thereby guaranteeing the stability and reliability of the data management system's services. For some simple data management systems or those with small amounts of data, repairing the data management system during the operation of the management service ensures smooth service flow.

[0036] In another possible implementation of the first aspect, the data management system further includes a metadata table, which includes logical storage unit information of at least one data table recorded by the data management system. Each logical storage unit is managed by a corresponding logical storage unit server. The data management system also includes at least one logical storage unit server, each of which is used to deploy logical storage units.

[0037] The data repair method further includes: a data repair device acquiring logical storage unit information, which describes the attribute values ​​of logical storage units in the data management system. Based on information from the metadata table and the logical storage unit information, the data repair device obtains indication information for at least one abnormal logical storage unit, where the attribute values ​​of the abnormal logical storage unit differ between the metadata table and the logical storage unit information, or the abnormal logical storage unit is not deployed in the logical storage unit server. The data repair device repairs at least one data table, where the at least one data table belongs to the data table associated with at least one abnormal logical storage unit.

[0038] Metadata describes data management and is stored in metadata tables. It's used by the data management system to manage logical storage units. Metadata includes, but is not limited to, one or more of the following: the logical storage unit server where the logical storage unit is deployed, the data information of the data tables included in the logical storage unit, and the data blocks associated with the logical storage unit. In actual implementation, inconsistencies can occur between logical storage units in the data management system. Specifically, the information about the logical storage unit in the metadata table may differ from the actual situation of the logical storage unit in the distributed file system, or the logical storage unit may not be deployed on the logical storage unit server. These problematic logical storage units are called abnormal logical storage units. When abnormal logical storage units exist, data problems arise, and the data management system may be unable to provide normal services, such as data being unable to be read or written correctly. In this case, it is necessary to repair the data tables to make the logical storage units consistent, thus ensuring the normal operation of the data management system.

[0039] In another possible implementation of the first aspect, the repair configuration information further includes a first instruction for instructing the repair of logical storage unit inconsistencies.

[0040] In the above implementation, the data repair device repairs the inconsistency of logical storage units according to the first instruction. This allows the repair of inconsistencies in logical storage units to be done without manual intervention, reducing the workload of maintenance personnel and improving the timeliness of the repair.

[0041] Furthermore, in some cases, logical storage unit inconsistency repair may not be necessary. This is because, in actual implementation, temporary logical storage unit inconsistencies may occur, such as during large-scale data migrations or system upgrades. However, these situations may resolve themselves after the operation is completed. In such cases, not repairing logical storage unit inconsistencies can avoid excessive intervention that could impact the reliability and stability of the data management system service. Therefore, in some situations, not repairing logical storage unit inconsistencies helps maintain the reliability and stability of the data management system service.

[0042] In another possible implementation of the first aspect, at least one data table includes a metadata table, and repairing at least one data table includes: repairing the metadata table before the management service in the data management system is started.

[0043] In the above implementation, when an anomaly exists in the metadata table, data repair needs to be performed before the data management system's management service starts. This is because the metadata table is a crucial index and directory of data within the data management system. The accuracy and completeness of the metadata directly impact the reliability of the data management system's services. The management service, responsible for managing the services provided by the data management system, plays a central role in the service provision process. Therefore, when an anomaly occurs in the metadata table, it needs to be repaired before the data management system's management service starts to maintain the reliability of the data management system's services and ensure their normal operation.

[0044] In another possible implementation of the first aspect, the repair configuration information further includes at least one business table to be repaired, and at least one data table includes a target data table, the type of which is a business table. The target data table belongs to at least one business table to be repaired, and the target data table belongs to a data table associated with at least one abnormal logical storage unit.

[0045] In the above implementation, at least one business table to be repaired is pre-configured in the repair configuration information. When a business table to be repaired is abnormal, the data repair device only needs to repair the abnormal business table. This is because in some cases of data anomalies, only anomalies in business tables related to the core functions of the data management system will affect the reliability and stability of the data management system. In these cases, prioritizing the repair of the abnormal core business tables can save data repair time, reduce resource consumption, and lower repair costs.

[0046] In another possible implementation of the first aspect, obtaining logical storage unit information includes: obtaining logical storage unit information from a distributed file system, wherein the logical storage unit information in the distributed file system is backed up from the logical storage unit server to the distributed file system by a coprocessor.

[0047] In the above implementation, the distributed file system contains backup data of logical storage unit information. This helps maintain the stability of the data management system and ensures the reliability of its services. For example, in the event of a hardware failure, the backup logical storage unit information in the distributed file system helps to restore data as quickly as possible after the data management system returns to normal. Furthermore, when logical storage unit inconsistencies occur, the backup logical storage unit information in the distributed file system can serve as a reference to help repair these inconsistencies.

[0048] In another possible implementation of the first aspect, the repair configuration information is user-defined, or the repair configuration information is part of the default configuration.

[0049] In the above implementation, the repair configuration information can be user-defined. This allows users to set the repair configuration information according to actual conditions, such as business characteristics, user needs, and system status, making data repair more aligned with business characteristics and improving the flexibility and autonomy of data repair. Furthermore, the repair configuration information can also be a default configuration that meets basic data repair needs. Users do not need to spend a lot of time and effort researching and configuring repair configurations; they can quickly start data repair work simply by using the default configuration, lowering the barrier to entry.

[0050] In another possible implementation of the first aspect, backup data of the structure information of at least one data table is stored in a distributed coordination service component, which manages the server nodes of the data management system, and the server nodes of the data management system include at least one logical storage unit server.

[0051] In the above implementation, the distributed coordination service component stores backup data of the data table structure information. In this way, when data anomalies occur in the data management system, the data repair device can also repair the data based on the backup data of the data table structure information in the distributed coordination service component, improving the efficiency of data repair and further contributing to ensuring the reliability of the data management system service and maintaining its stability.

[0052] In another possible implementation of the first aspect, the data repair method further includes: when the structural information of the first data table changes during the provision of business services by the data management system, updating the backup data of the structural information of the first data table in the distributed coordination service component, wherein the first data table belongs to at least one data table.

[0053] In the above implementation, the distributed coordination service component can respond promptly to changes in the data management system's data and update the backup data of the data table structure information in a timely manner. In this way, the backup data of the data table structure information in the distributed coordination service component has better timeliness, thereby helping to maintain the stability of the data management system.

[0054] Secondly, this application provides a data repair apparatus, which includes units or modules for performing the method of the first aspect.

[0055] Thirdly, this application provides a data repair apparatus, which includes a processor and a memory. The memory is used to store computer instructions, and the processor is used to call the computer instructions stored in the memory to implement the method of the first aspect.

[0056] Fourthly, this application provides a chip including a processor and an interface circuit. The interface circuit is used to receive signals from other communication devices and transmit them to the processor or to send signals from the processor to other communication devices. The processor is used to implement the method of the first aspect through logic circuits or by executing code instructions.

[0057] Fifthly, this application provides a readable storage medium for storing a computer program that, when executed by a processor, causes a communication device including the processor to perform the method of the first aspect.

[0058] In a sixth aspect, this application provides a computer program product that, when executed by a processor, causes a communication device including the processor to perform the method of the first aspect.

[0059] The beneficial effects of the technical solutions provided in the second to sixth aspects of this application can be referred to the beneficial effects of the technical solutions in the first aspect. Attached Figure Description

[0060] The accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0061] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0062] Figure 2A is a flowchart illustrating a data repair method provided in an embodiment of this application;

[0063] Figure 2B illustrates an example of an abnormal logical storage unit;

[0064] Figure 3 is a flowchart illustrating another data repair method provided in an embodiment of this application;

[0065] Figure 4 is a detailed schematic diagram of the data repair method provided in the embodiment of Figure 3;

[0066] Figure 5 is a schematic diagram of a data repair device provided in an embodiment of this application;

[0067] Figure 6 is a schematic diagram of another data repair device provided in an embodiment of this application. Detailed Implementation

[0068] Before introducing the embodiments of this application, the terminology that may be used in the embodiments of this application will be introduced first.

[0069] 1. Data Management System

[0070] A data management system is a software system used to organize, store, retrieve, manage, and maintain data. For example, users can define the structure of the data management system, including but not limited to creating tables, defining field data types, and setting constraints, or one or more of these. Based on type, data management systems can be divided into relational data management systems and non-relational data management systems. Relational data management systems are suitable for scenarios requiring high data consistency, while non-relational data management systems are suitable for scenarios requiring high data flexibility and scalability. For example, the Hadoop database (HBase) is a non-relational distributed database designed for big data, aiming to handle large-scale datasets and providing a highly reliable, high-performance, highly scalable, and strongly consistent solution for storing massive amounts of data and facilitating random read / write access.

[0071] 2. File System

[0072] A file system is a mechanism and architecture in an operating system used to store and manage computer files and data. Data management systems store data as files within the file system. A distributed file system is a file system architecture that distributes files across multiple nodes and allows access and management via a network. For example, HBase stores data as HFiles (a binary file format) in the Hadoop Distributed File System (HDFS). HDFS is a highly distributed file system used to store and manage extremely large datasets.

[0073] 3. Data Blocks

[0074] A data block is the basic unit for storing and processing data in a computer. A file system divides the storage space of a storage device (such as a hard drive or solid-state drive) into fixed-size data blocks. It manages these data blocks to store and access files. When a user creates or saves a file, the file system allocates several contiguous or non-contiguous data blocks to store the file content. For example, HDFS divides files into fixed-size data blocks, with a default size of 128 megabytes (MB). Of course, the size of the data block is configurable and can be adjusted according to different application scenarios. For example, for storing large files, a larger data block size can reduce metadata overhead and improve data transfer throughput. For storing small files, a smaller data block size may be more conducive to data management and processing, but it will increase the burden of metadata management.

[0075] 4. Data Table

[0076] A data table, also known as a table, is the basic structure used to organize and store data in a data management system. For example, HBase organizes data in the logical form of data tables. A row key is a unique identifier for each row in a data table, and all rows in the table are stored in lexicographical order according to their row keys. For instance, for a table storing log data, the row key can be designed in the form of [timestamp]_[log_type]_[log_id] to facilitate fast retrieval based on a timestamp range.

[0077] 5. Logical storage unit

[0078] Logical storage units are the core units for managing data in a data management system. For example, the core unit of HBase data management is the region. HBase divides a data table into multiple regions based on the range of row keys, and each region contains a range of row keys. For instance, a data table containing user information is divided into regions according to the range of user IDs (as row keys). Region001 stores user information with user IDs from 1 to 10000, region002 stores user information with user IDs from 10001 to 20000, and so on.

[0079] Logical storage units are assigned to logical storage unit servers for management, with each logical storage unit server managing at least one logical storage unit server. For example, a logical storage unit server managing a region is a regionserver, and each regionserver manages at least one region. During a data write operation, the client sends a write request to the logical storage unit server. This write request includes the data to be written and information about the target logical storage unit. Upon receiving the write request, the logical storage unit locates the logical storage unit based on the write request and performs the data write operation. During a data read operation, the logical storage unit receives a read request from the client. This read request includes the data table information (such as row key information) of the data to be read. The logical storage unit locates the corresponding logical storage unit based on the read request and performs the data read operation.

[0080] Furthermore, as data is continuously written, the amount of data in the logical storage unit will continue to grow. When the size of the logical storage unit reaches a certain threshold, the logical storage unit server will trigger a logical storage unit server split operation, splitting the logical storage unit into two smaller logical storage units. The new logical storage units are allocated to the same or different logical storage unit servers to achieve a balanced distribution of data. When the amount of data in some logical storage units is small, or when logical storage units become sparse due to data deletion, the logical storage unit server will perform a logical storage unit merging operation to reduce the number of logical storage units and improve data storage efficiency and query performance.

[0081] 6. Metadata

[0082] Metadata is used by data management systems for data management and maintenance, and is stored in metadata tables (or meta tables). For example, the HBase system uses metadata to describe and manage information related to HBase data tables and regions.

[0083] For example, please refer to Table 1, which lists an example of an HBase metadata table.

[0084] Table 1. Examples of metadata representation

[0085] As can be seen from Table 1, both table001-region001 and table001-region002 include the data of table001, table002-region003 includes the data of table002, and regionserver001 deploys the data of table001-region001, table001-region002, and table002-region003.

[0086] In some solutions, the format of metadata is manually defined, for example, when developing a data management system, developers define it based on data field names, data types, length limits, etc. In other solutions, metadata is generated by the system by default. Based on type, HBase metadata includes, but is not limited to, at least one of the following: table-level metadata, region metadata, and column family metadata.

[0087] Table-level metadata includes, but is not limited to, at least one of the following: table name, creation time, and table attributes (such as whether compression is enabled, replication factor, etc.). Region metadata includes, but is not limited to, the row key range of each region and the regionserver (the component that deploys the region, described below) where the region resides. Column family metadata includes, but is not limited to, the column family name and storage attributes (such as maximum version number, compression algorithm, etc.).

[0088] In addition, HDFS also contains backup information of region metadata, such as file name, size, permissions, owner, creation time, and modification time.

[0089] 7. Inconsistent logical storage units

[0090] Inconsistency in logical storage units refers to differences in the attributes of logical storage units across the data management system and file system. For example, a region is a logical storage unit; region inconsistency means that the region's state information is inconsistent across the HBase meta table, HDFS, and regionserver. For instance, the attribute values ​​of a region in the meta table differ from those in HDFS. Another example is that a region exists in HDFS but is not deployed on the regionserver.

[0091] 8. Data Management System Cluster

[0092] A data management system cluster is a distributed system consisting of at least one management service node, at least one logical storage unit server node, and one distributed coordination service component node. These nodes cooperate to achieve the storage, management, and processing of large-scale data. The management service node is responsible for the management and coordination of the data management system cluster, including but not limited to one or more of the following: load balancing of the logical storage unit servers, allocation of logical storage units, creation and deletion of tables, and modification of table structures. The distributed coordination service component node is responsible for monitoring and managing all nodes of the data management server cluster.

[0093] For example, an HBase cluster is a distributed system consisting of at least one master node, at least one regionserver node, and one zookeeper node.

[0094] The master node is the management service node, responsible for the management and coordination of the HBase cluster, including but not limited to managing one or more of the following: regionserver load balancing, region allocation, table creation and deletion, and table structure changes. For example, when a user wants to create a new HBase table, the master node allocates the necessary metadata for the table, such as the initial region allocation for the new HBase table. Furthermore, the master node is also responsible for HBase cluster region management; for instance, it decides which regionserver to allocate a new region to based on the load and storage conditions of each regionserver in HBase. Another example is when a regionserver fails; the master node reassigns regions from the failed regionserver to other healthy regionservers to ensure data availability and normal cluster operation. Moreover, the master node is also responsible for managing and monitoring HBase load balancing. For example, when there is an imbalance in the load of HBase regionservers, the master node migrates some regions from overloaded regionservers to less overloaded regionservers, thereby ensuring that resources within the cluster are used efficiently.

[0095] The regionserver node is a distributed coordination service component node responsible for storing and processing HBase data. Each regionserver manages multiple regions. For example, a regionserver node receives data read and write requests from clients. For read requests, the regionserver reads data from the corresponding region and provides it to the client. For write requests, the regionserver stores the data in the appropriate region.

[0096] The ZooKeeper node is responsible for monitoring and managing the various nodes in the HBase cluster. For example, the ZooKeeper node is responsible for master election (electing a new master node to take over cluster management when the primary master node fails or becomes unavailable) and failover to ensure high availability of the cluster. For instance, when the master node fails, the ZooKeeper node can coordinate with other nodes in the cluster to elect a new master node, ensuring that the cluster does not stop service due to the master node failure. Furthermore, the ZooKeeper node also stores HBase metadata, such as regionserver information, region deployment information, and the structure of tables in HBase.

[0097] The system and business scenarios of the embodiments of this application are described below. It should be noted that the system architecture and business scenarios described in this application are for the purpose of making the technical solutions of this application clearer, and do not constitute a limitation on the technical solutions provided in this application. Those skilled in the art will know that with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0098] Please refer to Figure 1. Figure 1 is a schematic diagram of the architecture of a data storage system provided in an embodiment of this application. The data storage system 10 includes a data management system 101 and a file system 102. Optionally, it also includes a coordinator 103 and a configuration device 104.

[0099] The data management system 101 has data storage capabilities. Specifically, the data management system 101 includes at least one data table, and each data table contains at least one piece of data. Optionally, in actual implementation, the data management system 101 is a non-relational data management system, such as HBase.

[0100] The data management system 101 also includes at least one logical storage unit, which is the basic unit of data management. Specifically, each data table forms at least one logical storage unit, meaning that the data of each data table is distributed across at least one logical storage unit. For example, a data table containing user information may have its logical storage units divided according to the range of user IDs. User information with IDs from 1 to 10000 is stored in the first logical storage unit, and user information with IDs from 10001 to 20000 is stored in the second logical storage unit.

[0101] The data management system 101 includes at least one logical storage unit server, which is responsible for storing and processing data of the data management system 101. Each logical storage unit server manages at least one logical storage unit. For example, the logical storage unit server receives read and write requests from clients. For a read request, the logical storage unit server reads data from the corresponding logical storage unit and provides it to the client. For a write request, the logical storage unit server stores the data in the corresponding logical storage unit.

[0102] In this embodiment, the data management system 101 includes a data repair device 1011. The data repair device 1011 is a computing device capable of scanning at least one data block in the file system 102 to detect abnormal data blocks. For example, the data repair device 1011 can scan the data blocks of the file system 102 according to different scanning modes, each corresponding to a different level of scanning accuracy. Furthermore, the data repair device 1011 is used for data repair; for abnormal data blocks, the data repair device 1011 can clean them up to maintain the stability of the data management system 101 service. In some cases, the data repair device 1011 is also used to repair inconsistencies in logical storage units within the data management system.

[0103] Optionally, the data stored in the data management system 101 includes business data, which is provided by the corresponding business system and stored in the business table. Further optional, the data stored in the data management system 101 also includes metadata, which is used to describe and manage the data in the data management system 101. For example, a piece of metadata may include, but is not limited to, one or more of the following: the identifier of the logical storage unit, the status of the logical storage unit, the name of the data table managed by the logical storage unit, and the data range of the data table managed by the logical storage unit.

[0104] Optionally, the data management system 101 is also used to provide data-related services to other business systems. For example, the data management system 101 can store business data from business systems and can receive at least one operation from other business systems, including but not limited to add, delete, query, and modify operations. For instance, a business system sends a request to the data management system 101 to read and / or write data through the data management system 101's interface, and the data management system 101 provides corresponding services to the business system based on the business system's operation request.

[0105] Optionally, the data management system 101 includes a management service component responsible for managing the services provided by the data management system. For example, the first management service component monitors the status of the first logical storage unit server, the second logical storage unit server, and the first logical storage unit. The first logical storage unit server deploys the data of the storage unit, and the second logical storage unit server deploys the data of the second logical storage unit. When the first logical storage unit fails, the first management service component can replicate the first logical storage unit to the second logical storage unit server to ensure the normal operation of the data management system. As another example, the first management service component manages the status of the first and second logical storage units. When the data volume of the first logical storage unit is overloaded, the first management service component can split the data managed by the first logical storage unit and allocate the split data to the second logical storage unit. As yet another example, the first logical storage unit handles user requests for creating, deleting, and modifying tables. When a user initiates a table creation operation, the first logical storage unit records relevant information about the new table in the metadata, such as the table name and the index of the data in the table, and allocates an initial logical storage unit for the table.

[0106] The file system 102 has data storage capabilities and is used to store data from the data management system 101. Specifically, the file system 102 includes at least one data block, and each data block stores data from at least one logical storage unit in the data management system 101.

[0107] Optionally, the file system 102 also stores information about each logical storage unit in the data management system 101, including but not limited to the name, deployment information, status, stored table name, and one or more of the start and end row keys of the stored data in at least one logical storage unit in the data management system 101. Optionally, the file system 102 is a distributed file system, such as HDFS.

[0108] The coordinator 103 manages the server nodes of the data management system 101. For example, the coordinator 103 is responsible for managing at least one logical storage unit server of the data management system 101. Optionally, the coordinator 103 includes the status information of the data management system 101. For example, the coordinator 103 can record the status information of the data management system 101 in the event of an abnormal power outage. Further optionally, the coordinator 103 also includes the structure information of the data tables of the data management system 101. For example, when the structure of a table changes, the coordinator 103 can update the structure information of the data tables of the data management system 101 recorded by the coordinator 103 according to the changes. Optionally, the coordinator 103 is a distributed coordination service component, such as ZooKeeper.

[0109] The configuration device 104 provides configuration information, such as repair configuration information, to the data repair device 1011. Optionally, the configuration device 104 has the ability to interact with a user and can receive configuration information input from the management user 105. For example, the configuration device 104 receives repair configuration information input from the management user 105.

[0110] Optionally, the capabilities of the aforementioned data management system 101, file system 102, coordinator 103, and configuration device 104 are implemented by the same physical device. For example, the data management system 101, file system 102, coordinator 103, and configuration device 104 are deployed on the same server. Alternatively, the capabilities of the aforementioned data management system 101, file system 102, coordinator 103, and configuration device 104 are implemented by different physical devices. For example, the data management system 101, file system 102, coordinator 103, and configuration device 104 are deployed on four different servers.

[0111] With the rapid development of internet and IoT technologies, data management systems have become an indispensable core component of modern business systems, widely used in e-commerce, finance, healthcare, transportation, and other fields. However, data management systems can experience anomalies during operation, such as hardware failures, network failures, and abnormal power outages. These anomalies can easily generate abnormal data blocks. Abnormal data blocks can impair the functionality of the data management system, potentially affecting the normal operation of the business system.

[0112] Due to the sheer volume of data and the complexity of data management systems, existing data remediation methods are often inefficient. For example, some data remediation solutions require manual intervention. When an anomaly occurs in the data management system, the customer needs to contact relevant operations and maintenance personnel to detect and clean up the abnormal data blocks based on the detection results. When dealing with complex data management systems or situations involving multiple abnormal data blocks, this data remediation approach becomes extremely cumbersome and time-consuming. This not only increases system maintenance costs but also severely impacts the reliability and availability of the data management system, thereby affecting the operation of related business systems.

[0113] In view of this, this application provides a data repair method that can detect abnormal data blocks according to the detection precision corresponding to a first scanning mode, and then repair the data based on the results of the abnormal data blocks. Furthermore, the scanning precision corresponding to the first scanning mode is pre-configured, which allows the data repair work to be more targeted, thereby improving the efficiency of data repair and contributing to maintaining the reliability and availability of the data management system.

[0114] The system architecture and application background provided by the embodiments of this application have been described above. The method provided by the embodiments of this application will be described below.

[0115] Please refer to Figure 2A, which is a schematic flowchart of a data repair method provided in an embodiment of this application. Optionally, this method is applied to a data storage system, such as the data storage system shown in Figure 1, and is implemented by the data repair device 1011 shown in Figure 1. In some schemes, the data repair method may also be performed by a device with other names. For ease of understanding, the following description uses the data repair device as the executing entity.

[0116] The data repair method shown in Figure 2A may include multiple steps from S201 to S203. It should be understood that, for ease of description, this embodiment uses the sequence of steps S201 to S203, and is not intended to limit the execution to this specific order. This embodiment does not limit the order, execution time, or number of executions of the one or more steps described above. Steps S201 to S203 are as follows:

[0117] Step S201: When the data management system is started, the data repair device obtains repair configuration information.

[0118] The data management system is used to manage the data in the data storage system. For example, the data management system 101 shown in Figure 1 includes, but is not limited to, functions for data storage, data organization, data query and retrieval, and data updating and maintenance. For instance, the data management system can receive data read requests from users and output corresponding data based on those requests.

[0119] A distributed file system has data storage capabilities and can store data in the form of at least one data block. Optionally, the distributed file system is the file system 102 shown in Figure 1.

[0120] A data repair device is a computing device, such as a computing machine, used to repair data in a data management system. For example, when data in the data management system cannot be read, the data repair device repairs the data based on backup information in a distributed file system to ensure the normal operation of the data management system.

[0121] The repair configuration information describes the repair strategy. As one possible design, the repair configuration information includes one or more of the following: indication information for the first scan mode, indication information for the repair mode, indication information for the repair timing, a first indication, or at least one business table to be repaired. This information is used by the data repair device to perform data repair. These pieces of information are described below:

[0122] (1) The indication information of the first scan mode is used to indicate the scanning accuracy of data blocks in the distributed file system. The first scan mode belongs to one of multiple scanning accuracy levels, and different scanning accuracy levels correspond to different scanning accuracy. For example, the data repair device scans the data blocks in the file system according to the scanning accuracy corresponding to the first scan mode to detect abnormal data blocks.

[0123] For ease of understanding, three different scanning accuracy levels are provided as examples below.

[0124] The first level checks whether the header information of the data block exists. This header information includes, but is not limited to, one or more of the following: the data block identifier, size, creation time, verification information, and timestamp. For example, if the data block identifier is valid or corrupted, the data management system will have difficulty accurately locating the data block, and the data block may be abnormal.

[0125] The second level checks whether a portion of the data in the data table within the data block can be read. For example, a specified number of bytes can be set to read a portion of the data in each data block's data table. For instance, a preset read size of 256 bytes (B) can be used. For data tables exceeding 256B, read from the first data entry until 256B is reached; for data tables not exceeding 256B, read all data. If data cannot be read during the process, it indicates an error in the data block containing that data table.

[0126] The third level checks whether all data in the data tables within the data block can be read. This level of precision is suitable for situations requiring the stability and integrity of the data management system. For example, if the data management system manages data from a banking or financial system, which has extremely high requirements for data accuracy and integrity, then the third level of precision is necessary to scan the data management system.

[0127] (2) The repair mode indication information is used to indicate the scenarios in which the data management system is repaired. For example, the data management system may be repaired in all scenarios, or only when the data management system stops abnormally, or no data repair may be performed under any circumstances. The repair mode can be set according to specific situations or needs. For example, for some data management systems with high requirements for data accuracy and integrity, such as data management systems for managing financial data, it can be set to perform data repair in all scenarios.

[0128] For cases where data repair is only performed when the data management system experiences an abnormal shutdown, it is necessary to determine whether data repair is required by combining the first identifier, which indicates whether the data management system previously experienced an abnormal shutdown. For example, the repair mode configured in the repair configuration information indicates that data repair should be performed when the data management system experiences an abnormal shutdown, but the first identifier indicates that the data management system did not previously experience an abnormal shutdown; therefore, data repair is not required in this case. As another example, the repair mode configured in the repair configuration information indicates that data repair should be performed when the data management system experiences an abnormal shutdown, and the first identifier indicates that the data management system previously experienced an abnormal shutdown, such as a power outage; in this case, data repair is not required.

[0129] (3) The indication information for the timing of data repair is used to indicate when the data management system should perform data repair. For example, data repair can be performed before the management service of the data management system is started. Or, data repair can be performed during the operation of the management service of the data management system.

[0130] In some possible implementations, when an anomaly occurs in the metadata table of the data management system, data repair needs to be performed before the management service starts. For example, in the HBase system, the master component provides management services for the HBase system. When an anomaly occurs in the HBase metadata table, it can cause errors during the master's startup process, and in severe cases, it can lead to the collapse of the HBase cluster. Therefore, when an anomaly occurs in the metadata of HBase, data repair needs to be performed before the master component starts to ensure that the master can start normally, thereby ensuring the stability and reliability of HBase.

[0131] In other possible implementations, when a business data table in the data management system experiences anomalies, the timing of the repair can be set based on the importance of the business. For example, for business data tables involving high data privacy and security requirements, such as financial data or logistics management data, these tables need to be repaired before the data management system's management service starts after anomalies occur to ensure the normal operation of the data management system. As another example, for business data tables involving less real-time requirements, such as historical cached data or business log data, these tables can be repaired during the operation of the data management system's management service after anomalies occur.

[0132] (4) The first instruction is used to instruct the repair of logical storage unit inconsistencies, where the logical storage unit can be the aforementioned region. For example, the first instruction is a specific parameter field with a value of true. For another example, the first instruction can be a constraint condition, such as when the number of inconsistent logical storage units reaches 10, the data management system should perform logical storage unit inconsistency repair.

[0133] Inconsistency in logical storage units refers to a discrepancy between the information about logical storage units included in the metadata table and the actual situation of these logical storage units in the distributed file system, or the logical storage unit not being deployed on a logical storage unit server. When logical storage unit inconsistency occurs, data service anomalies can arise. For example, data in a first logical storage unit is deployed on a first logical storage unit server and a second logical storage unit server. Due to a network failure, the first and second logical storage unit servers cannot communicate normally. When a client initiates a data write request to the first logical storage unit and successfully writes a new data record, the inability of the first logical storage unit server to communicate with the second logical storage unit server leads to inconsistency in the data stored in the first and second logical storage unit servers. This can affect the normal operation of business. For instance, in an e-commerce system, it may result in inconsistent product information displayed to users. In such cases, it is necessary to repair logical storage unit inconsistencies to ensure the normal operation of the data management system.

[0134] In other possible implementations, the data management system does not need to repair logical storage unit inconsistencies. For example, in some cases, logical storage unit inconsistencies do not affect the system's functionality or performance; in such cases, it is possible to choose not to repair the logical storage unit inconsistencies. Yet another example is that in some cases, the data management system automatically performs load balancing and reallocates logical storage units, because there is no need to repair logical storage unit inconsistencies.

[0135] (5) The indication information of the business tables that need to be repaired is used to guide the data repair device to repair the scope of the data. For example, the indication information of the business tables that need to be repaired includes the indication information of the first business table, the second business table, and the third business table. When the first business table, the second business table, and the fourth business table are abnormal, the data repair device can repair the data of the first business table and the second business table according to the indication information of the business tables that need to be repaired.

[0136] In some other possible implementations, the repair configuration information is user-defined. For example, a user could define "enable data repair in abnormal shutdown scenarios, perform data block scanning at the first-level scan precision, and do not repair logical storage unit inconsistencies." Even more exemplarily, a user could define "enable data repair in all scenarios, perform data block scanning at the second-level scan precision, perform data repair before the data management system's management service starts, perform logical storage unit inconsistencies, and prioritize repairing the first business table."

[0137] In other possible implementations, the repair configuration information is set to the system default settings. For example, the scan accuracy corresponding to the first scan mode is preset to level one, and no repair is performed for inconsistencies in logical storage units.

[0138] In other possible implementations, the repair configuration information is stored in a configuration center, such as the configuration device 104 shown in Figure 1. Users can set the repair configuration information through the configuration center, and the data management system retrieves the repair configuration information from the configuration center upon startup. Optionally, the configuration center and the data management system are deployed on the same server. Even more alternatively, the configuration center and the data management system are deployed on different servers.

[0139] In some possible implementations, during the startup process of the data management system, the data repair device acquires a first identifier and determines whether the data management system previously experienced an abnormal shutdown based on the information indicated by the first identifier. In practice, the first identifier can be referred to as the start / stop identifier. When the data management system shuts down normally, it updates the first identifier to indicate a normal shutdown before stopping. The next time the data management system starts, the data repair device acquires the first identifier indicating a normal shutdown, indicating that the data management system previously shut down normally. After the data management system starts and the data repair device determines whether the previous shutdown was abnormal, the data management system updates the first identifier to indicate a normal start. If an abnormal shutdown occurs during the operation of the data management system, such as an abnormal shutdown due to a power outage, the data management system may not have time to update the first identifier to indicate a normal shutdown before stopping. The next time the data management system starts, the acquired first identifier indicates a normal start (the information updated during the previous startup), indicating that the data management system previously shut down abnormally.

[0140] In some cases, the indication information for the first identifier can be a field, such as the `normal_start_stop.flag` field. For example, "0" represents a normal stop, and "1" represents a normal start, with the initial value set to "1". Before the data management system stops normally, it updates the `normal_start_stop.flag` flag to "0". After the data management system starts, it updates the `normal_start_stop.flag` flag to "1". If an anomaly occurs and the data management system cannot update the `normal_start_stop.flag` flag in time, during the next startup of the data management system, if the data repair device obtains that the `normal_start_stop.flag` flag is "1", the data repair device can determine that the previous stop of the data management system was an abnormal stop.

[0141] In some possible implementations, when the last shutdown of the data management system was an abnormal shutdown and determined to be a power outage, the data repair device writes a power outage identifier to the distributed coordination service component. For example, when HBase experiences an abnormal power outage, it records the power outage identifier in ZooKeeper. Exemplarily, after determining that the last shutdown of the data management system was abnormal, the data repair device performs various exclusions, including but not limited to one or more of hardware failures, network failures, and system crashes, to determine whether the last shutdown of the data management system was due to an abnormal power outage. Exemplarily, the distributed coordination service component has a power outage module for recording the power outage status of the data management system. When it is determined that the last shutdown of the data management system was due to an abnormal power outage, the data repair device writes a power outage identifier to the power outage module in the distributed coordination service component, recording the power outage status of the data management system. Exemplarily, the power outage identifier can be a simple text file, including but not limited to one or more of the timestamp of the power outage, system status, etc.

[0142] Optionally, the power failure flag can also be recorded in other storage systems, such as the computer's local storage or file system. For example, when HBase experiences an abnormal power failure, the power failure flag can be recorded in HDFS.

[0143] Step S202: The data repair device scans the data blocks in the distributed file system in the first scan mode to obtain the bad block scan results.

[0144] The bad block scan result includes indication information for at least one abnormal data block, indicating that the data read and / or data write of the abnormal data block is abnormal. For example, the indication information for at least one abnormal data block includes, but is not limited to, one or more of the following: the identifier of the abnormal data block, the identifier of the abnormal data table, and information about corrupted data in the abnormal data table.

[0145] For example, please refer to Table 2, which lists an example of indication information for at least one abnormal data block.

[0146] Table 2 Examples of indication information for at least one abnormal data block

[0147] It should be noted that the table is only for visually representing the contents of the indication information of at least one abnormal data block. In actual implementation, the indication information of at least one abnormal data block may be represented in other forms, such as text data, linked lists, or sets of corresponding relationships.

[0148] As described in step S201 above, the first scan mode is used to indicate the accuracy of scanning data blocks in the distributed file system. For example, when the first scan mode corresponds to the first level of scan accuracy, the data repair device checks for bad blocks by reading the identifier, verification information, and timestamp of each data block in the distributed file system. If the identifier, verification information, and timestamp of a data block cannot be read, it indicates that the data block is an abnormal data block. As another example, when the first scan mode corresponds to the second level of scan accuracy, the data repair device checks for bad blocks by checking whether a specified number of bytes of data in the data table of each data block can be read. For example, checking 256 bytes of data in each data table: For data tables with a data size exceeding 256 bytes, reading 256 bytes of data starting from the first data in the table; for data tables with a data size not exceeding 256 bytes, reading all data in the table. If data cannot be read during the reading process, it indicates that the data block containing that data table is abnormal.

[0149] In some implementations, step S202 is executed when the data repair device determines that the data management system needs repair. The following section describes how the data repair device determines whether to repair the data management system.

[0150] As one possible implementation, the data repair device determines whether to repair the data management system based on the repair mode indication information. As mentioned above, the repair mode indication information is used to indicate the scenario for repairing the data management system. For example, data repair can be performed on the data management system in all scenarios, or only when the data management system stops abnormally, or no data repair can be performed under any circumstances. The repair mode can be set according to specific circumstances or needs.

[0151] In some cases, the data repair device needs to determine whether to perform data repair based on the repair mode indication information and the first identifier. For example, if the repair mode indication information indicates that the data management system should be repaired in the event of an abnormal stop, the data repair device needs to determine whether to repair the data management system based on the indication of the first identifier. If the first identifier indicates that the last stop of the data management system was an abnormal stop, the data repair device will perform data repair; if the first identifier indicates that the last stop of the data management system was a normal stop, the data repair device will not perform data repair.

[0152] In other solutions, when the data repair device determines that it will not repair the data management system, the data repair device skips the repair operation. For example, steps S202 and / or S203 are not executed. At this time, the data management system enters normal operation.

[0153] Step S203: The data repair device cleans up at least one abnormal data block.

[0154] For at least one detected anomalous data block, the cleanup operation performed by the data repair device includes one or more of the following: deleting corrupted files, deleting the anomalous data block, and marking the anomalous data block as unusable. For example, for data blocks that cannot be directly deleted, the data repair device marks them as unusable. As another example, for anomalous data blocks where only part of the data is corrupted, the data management system can simply delete the relevant data files.

[0155] In some solutions, the data repair device can also repair data tables in the data management system, such as metadata tables and business data tables in HDFS. As one possible implementation, the data repair device can utilize logical storage unit information and metadata table information to detect and repair abnormal data tables. For example, abnormal data tables include those with region inconsistency issues. The metadata table includes logical storage unit information for at least one data table recorded by the data management system. Each logical storage unit is managed by a corresponding logical storage unit server, and the data management system also includes at least one logical storage unit server, such as a regionserver in HDFS, used to deploy logical storage units.

[0156] As one possible implementation example, the data repair device acquires logical storage unit information and, based on the information in the metadata table and the logical storage unit information, obtains indication information for at least one abnormal logical storage unit. The logical storage unit information describes the attribute values ​​of logical storage units in the data management system. The abnormal logical storage unit may have attribute values ​​that differ between the metadata table and the logical storage unit information, or the abnormal logical storage unit may not be deployed in the logical storage unit server.

[0157] For example, please refer to Figure 2B, which illustrates an example of an abnormal logical storage unit. In Figure 2B, `regioninfo` is the attribute value of the deployment information of a region in HDFS recorded by HDFS, the `meta` table is the metadata table in HBase, and `regionserver001` is a logical storage unit server in HBase. As can be seen from Figure 2B, there is a difference between the deployment information of `table002-region004` recorded in `regioninfo` and the information in the `meta` table. Specifically, the row key ending value of the data for `table002` stored in `table002-region004` in `regioninfo` is 2999, but the row key ending value of the data for `table002` stored in `table002-region004` in the `meta` table is 1999. Furthermore, `regioninfo` also shows that `table002-region004` is deployed in `regionserver001`, but `table002-region004` is not deployed in `regionserver001` in HBase. In this case, `table002-region004` is an abnormal logical storage unit.

[0158] In some solutions, the file system contains logical storage unit information, which the data repair device retrieves from the file system. For example, HDFS stores the attribute values ​​of regions.

[0159] As mentioned above, the logical storage unit is the core unit for managing data in a data management system. Data for each data table is distributed across at least one logical storage unit. Therefore, when an inconsistency occurs between logical storage units, the data table associated with the corresponding abnormal logical storage unit will exhibit abnormal data. For example, if table001-region001 deploys data from table001, and table001-region001 is detected as an abnormal logical storage unit, then table001 will become an abnormal data table. Optionally, the data table with an abnormal logical storage unit may be a metadata table or a business data table.

[0160] Furthermore, the data repair device repairs data tables, where at least one data table belongs to a data table associated with at least one abnormal logical storage unit. For example, when the data tables associated with at least one abnormal logical storage unit include table001, table002, table003, and table004, the data repair device may repair table001, table002, table003, and table004, or it may repair only some of the tables in table001, table002, table003, and table004, or it may not repair them at all. These implementation methods are selected according to the specific circumstances.

[0161] For example, in some solutions, administrators can customize the scope of repair. For instance, the indication information for the business tables requiring repair in the configuration information includes a first table, a second table, and a third table. When the detection result indicates that the first, second, and third tables need repair, the data repair device repairs them. When the detection result indicates that the second and fourth tables need repair, the data repair device only needs to repair the second table. When the detection result indicates that the fourth table needs repair, the data repair device does not perform any data repair.

[0162] In other possible implementations, the administrator can pre-set the timing for repairing the business data tables. For example, the first and second business data tables can be repaired before the data management system's management service is started, while the second and third business data tables can be repaired when the second and / or third business data tables are needed.

[0163] In other possible implementations, when an anomaly is detected in the metadata of the data management system, the metadata needs to be repaired before the management service of the data management system is started. For example, the repair configuration information may also include instruction information for repairing metadata. This instruction information instructs that when an anomaly occurs in the metadata of the data management system, the metadata should be repaired before the management service of the data management system is started. When an anomaly occurs in the metadata, the data repair device repairs the metadata before the management service of the data management system is started, according to the instruction information for repairing metadata included in the repair configuration information.

[0164] In other possible implementations, the data repair device can choose whether or not to repair logical storage unit inconsistencies. For example, the repair configuration information includes an indication not to repair logical storage unit inconsistencies. For instance, the repair configuration information includes a parameter with the field `Hbase.region.inconsistent.restore.enable`, where the data repair device does not repair logical storage unit inconsistencies when the value of this field is false. As another example, the data repair device repairs logical storage unit inconsistencies when the value of this field is true.

[0165] In other possible implementations, during the data management system startup process, a custom coprocessor backs up the deployment information of the logical storage units in the file system. This deployment information includes, but is not limited to, one or more of the following: the logical storage unit's identifier, name, status, table name it manages, and the data range of the table it manages. In this way, the data repair device can also rely on the backup data of the logical storage unit deployment information stored in the file system when repairing data in the data management system. For example, when repairing inconsistencies in logical storage units, it can refer to the backup data of the logical storage unit deployment information stored in the file system for repair.

[0166] In other possible implementations, during the startup of the data management system, a custom coprocessor backs up the structure information of the stored tables in a distributed service coordination component. This distributed coordination service component manages the server nodes in the data system, including but not limited to at least one management service component managing the data management system and at least one logical storage unit server component. In this way, the data management system can repair data tables based on the structure information of the stored tables in the distributed service coordination component when repairing data.

[0167] The embodiment shown in Figure 3 illustrates a data repair method. This method scans data blocks in a distributed file system according to different levels of data scanning accuracy, cleans up data blocks based on the bad block scan results, and thus repairs the data in the data management system. This approach improves the efficiency of data repair, reduces labor costs, and helps maintain the stability and reliability of the data management system's services.

[0168] The foregoing, in conjunction with Figure 2A, introduced several possible implementations of the data repair method. The following, in conjunction with Figure 3, introduces a specific implementation.

[0169] Please refer to Figure 3, which is a flowchart illustrating a data repair method provided in an embodiment of this application. Optionally, this method is applied to a data storage system, such as the data storage system shown in Figure 1, and may be implemented by the data repair device 1011 shown in Figure 1. In some schemes, the data repair method may also be performed by a device with other names. For ease of understanding, the following description uses the data repair device as the executing entity.

[0170] The data repair method shown in Figure 3 may include multiple steps in steps S301 to S303. It should be understood that, for ease of description, this embodiment uses the sequence of steps S301 to S303, and is not intended to limit the execution to this specific order. This embodiment does not limit the order, execution time, or number of executions of the one or more steps described above. Steps S301 to S303 are as follows:

[0171] Step S301: During the startup of the data management system, the data repair device acquires the first identifier.

[0172] The data management system is HBase, and an introduction to HBase can be found above. The first identifier, also known as the start / stop identifier or startup / stop identifier, indicates whether HBase's last shutdown was abnormal, such as an abnormal power outage. This first identifier is stored in an identifier table. The following describes a scheme for proactively identifying abnormal power outages.

[0173] During HBase startup, the first flag is checked. If the first flag exists and its content matches the information written during a normal shutdown, it indicates that HBase's last shutdown was normal. The old flag is then cleared, a new flag is written, and the relevant information for a normal HBase startup is filled in. However, if HBase experiences an abnormal power outage, there isn't enough time to modify the first flag, leaving only the information from when HBase started. Therefore, during the normal startup process after power restoration, the data repair device obtains a first flag whose content is not the information from when the shutdown was normal, indicating that HBase's last shutdown was due to a power outage. For example, if `normal_start_stop.flag` is the first flag, and during HBase startup, the data repair device obtains that `normal_start_stop.flag` contains information written during a normal shutdown, it updates `normal_start_stop.flag` to reflect normal startup. However, if HBase experiences a power outage, there isn't enough time to modify `normal_start_stop.flag`. Therefore, during the normal startup process after power restoration, the data repair device obtains a `normal_start_stop.flag` whose content is not the information from when the shutdown was normal, indicating that HBase's last shutdown was abnormal.

[0174] Furthermore, during HBase startup, a custom coprocessor backs up the deployment information of storage regions to HDFS and the structure information of storage tables to ZooKeeper. When HBase provides services, the custom coprocessor can monitor changes in the table structure and incrementally refresh the backed-up metadata. Here, "region" refers to a logical storage unit as mentioned in the preceding embodiments, HDFS is a distributed file system as mentioned in the preceding embodiments, and ZooKeeper is a distributed coordination service component as mentioned in the preceding embodiments.

[0175] Backing up the deployment information of storage regions to HDFS helps resolve region inconsistencies. For example, in a scenario recovering from an abnormal power outage, after cleaning up bad blocks in HDFS, some region data may be cleaned up as well, potentially causing abnormal region attributes within HDFS data blocks. Therefore, when resolving region inconsistencies, the backup deployment information of the regions in HDFS can be used as a reference. Similarly, backing up the structure information of storage tables to ZooKeeper also helps resolve region inconsistencies. For example, the structure information of storage tables in ZooKeeper can be used as a reference when resolving region inconsistencies. For instance, region deployment information is stored in the .regioninfo file.

[0176] Step S302: When the first indicator indicates that the data management system has been shut down due to a power failure, the data repair device obtains the repair configuration information.

[0177] The repair configuration information includes, but is not limited to, the HBase repair mode, the HBase bad block scanning mode, the HBase repair timing, indications for region inconsistency repair, and one or more items from the list of business table names to be repaired. Optionally, the repair configuration information is defined by the administrator. Further optionally, the repair configuration information is a default configuration pre-set by the system. Optionally, when the repair configuration information is defined by the administrator, the data repair device uses the administrator-defined repair configuration information to perform data repair on HBase.

[0178] HBase repair mode indicates the scenario for repairing HBase, including but not limited to one or more of the following scan modes: enabling data repair in all scenarios, enabling data repair when HBase stops abnormally, and disabling HBase data repair.

[0179] HBase bad block scanning mode indicates the scanning mode used to scan HDFS data blocks, including but not limited to one or more of the following scan precision levels: 1-Basic fsck, 2-Intermediate read, and 3-Advanced readAll. Higher precision levels result in higher scan accuracy and greater accuracy in bad block detection, but also consume more resources and take longer. Specifically: 1-Basic fsck detects bad blocks by checking the block header; its accuracy is moderate, and some bad blocks may go undetected; it consumes almost no resources and has a short detection time. 2-Intermediate read checks if some data in the data table within the block can be read; its accuracy is moderate, but some intermediate blocks may be corrupted and cannot be detected; it consumes a small amount of resources and has a moderate detection time. 3-Advanced readAll checks if all data in the data table within the block can be read; it has high accuracy, consumes a large amount of resources, and has a long detection time.

[0180] HBase repair timing refers to the timing of data repair. Optionally, data repair timing includes, but is not limited to, one or more of the following times: before or during the startup of the HBase master service. When HBase experiences an abnormal shutdown, data corruption can occur. If the corruption involves metadata, the master service will malfunction, leading to HBase service failure. Therefore, when metadata is lost, repair should be performed by default before the master service starts. If the corruption involves business data, repair can be selected based on configuration. For example, the configuration can repair the first and second types of business data before the master service starts, and repair the third and fourth types of business data while the master service is running.

[0181] In some implementations, region inconsistency issues may arise when HBase experiences an abnormal shutdown. For example, the attribute values ​​of regions in the meta table may differ from those of regions in HDFS. Another example is regions existing in HDFS but not deployed on the region server; see the aforementioned technical terminology for details. The indication information for whether to repair region inconsistencies is used to indicate whether to repair them. In other words, when HBase experiences region inconsistencies, one can choose to repair them or not. For instance, in some cases, region inconsistencies may have a very limited impact on HBase services. For example, business systems may not have high real-time requirements for data, or may only access partial data, and the inconsistent regions may not involve that data. In this case, to avoid affecting normal business operations, one can choose not to repair the region inconsistencies, or wait until HBase is idle to repair them. Yet another example, in some cases, many business systems that rely on HBase have high requirements for data real-time performance and accuracy. Region inconsistencies may cause anomalies in these business systems, such as inaccurate query results or write failures. Therefore, when region inconsistencies occur, they need to be repaired to restore the business systems to normal operation as quickly as possible and ensure business continuity.

[0182] The list of business table names requiring repair includes the name of at least one business table that needs repair, indicating which business table the data repair device should prioritize for repair during data repair. For example, when anomalies are detected in the first, second, and third business tables, the list of business table names requiring repair only includes the first business table; therefore, the data repair device only needs to repair the first business table. As another example, when anomalies are detected in the first, second, and third business tables, the list of business table names requiring repair includes the fourth and fifth business tables; in this case, data repair of these business tables is not required.

[0183] Optionally, the repair configuration information is stored in the configuration center in the form of a table. For example, please refer to Table 3, which lists one case of repair configuration information.

[0184] Table 3 Example of Repair Configuration Information

[0185] In Table 3, the HBase repair configuration information is stored in the hbase-site.xml file. The parameter field of the HBase repair configuration information is Hbase.repair.mode, with a value of Abnormal_stop, which indicates that data repair should be performed on HBase in the event of an abnormal stop.

[0186] The parameter field for HBase bad block scanning mode is `Hbase.corrupt.block.scan.code`, with a value of 1 - primary fsck, indicating that data blocks in HDFS are scanned according to the scan precision corresponding to 1 - primary fsck. The parameter field for HBase repair timing is `Hbase.repair.timing`, with a value of `PreStart`, indicating that data repair should be performed before the HBase master service starts. The parameter field for indicating whether region inconsistency repair is needed is `Hbase.region.inconsistent.restore.enable`, with a value of `True`, indicating that data affecting HBase services will be automatically repaired. The parameter field for the list of business table names to be repaired is `Hbase.region.repair.tablename.list`, with a value of `test1`, `test2`, `test3`, indicating that when data is abnormal, the data tables in `test1`, `test2`, and `test3` will be repaired.

[0187] Furthermore, when the data repair device determines that the last stop of HBase was a power failure by using the first identifier, the data repair device writes a power failure identifier to the ZooKeeper component to record the power failure status of HBase.

[0188] Step S303: When the repair configuration information indicates that the data management system should be repaired, the data repair device repairs the data management system.

[0189] For example, when the repair mode indicates that data repair should be performed in all scenarios, the data repair device performs data repair on HBase. As another example, when the repair mode indicates that data repair should be initiated in the event of an abnormal stop and the data repair device detects that HBase's last stop was abnormal, the data repair device performs data repair on HBase.

[0190] Optionally, when the repair configuration information includes an instruction to disable detection and repair, the data repair device skips detection and repair, and HBase operates normally. For example, if HBase's last stop was an abnormal stop, and the system's reliance on HBase is low due to its low sensitivity to data, or if the HBase data is not core data required by the business system, then the repair configuration information can be configured to disable detection and repair, preventing data repair from being performed on HBase, and HBase can be started directly.

[0191] Please refer to Figure 4, which is a flowchart illustrating the data repair method provided in the embodiment of Figure 3. Figure 4 illustrates different scenarios when the data repair device performs data repair. For example, if the administrator writes a custom repair configuration to the configuration center, the data repair device will use the custom configuration for repair. If the administrator has not written a configuration, the data repair device will use the pre-set default configuration for data repair. Another example is during HBase startup. If the first identifier indicates a normal stop, the data repair device will clear the identifier's content and update it to normal startup. If the first identifier indicates a power outage scenario, the data repair operation under the power outage scenario will be performed. Furthermore, in the event of an abnormal power outage, if the repair configuration information indicates data repair under this scenario, the data repair device will perform data repair based on other content in the repair configuration information (such as the repair timing indication). If the configuration information indicates no data repair under this scenario, the data repair device will skip detection and repair, and HBase will start normal service.

[0192] The embodiment shown in Figure 3 provides an automated data repair solution. The data repair device can repair HBase data according to the repair configuration information, solving the problem of high technical requirements for operation and maintenance personnel and reducing labor costs. In addition, the embodiment shown in Figure 3 also provides an active identification solution for abnormal stop scenarios, which can solve the problem of slow response speed after HBase abnormal stop (such as abnormal stop due to power failure).

[0193] The methods of the embodiments of this application have been described in detail above. The apparatus of the embodiments of this application is provided below. It should be understood that the module division of the apparatus (which may include a system) provided in this application is merely an exemplary illustration of one way to divide the structure of the apparatus. In practical applications, the structure of the apparatus may have other division methods. This application is equally applicable to apparatuses with other division methods but the same function. Furthermore, the apparatus name provided in this application is an exemplary name and can be replaced in specific implementations.

[0194] Please refer to Figure 5, which is a schematic diagram of a data repair device provided in an embodiment of this application. The data repair device 50 can be a standalone device (e.g., the data repair device 1011 shown in Figure 1) or a module within a standalone device, including hardware and / or software modules. This data repair device 50 is used to implement the aforementioned data repair method, such as the data repair method shown in Figures 2A and 3.

[0195] The data repair device 50 includes an acquisition unit 501 and a processing unit 502. The acquisition unit 501 is used to perform operations such as acquisition, for example, executing steps S201 and S301. The processing unit 502 is used to perform operations such as scanning, obtaining, cleaning, updating, writing, confirming, executing, based on, running, repairing, and based on, for example, executing steps S202 and S303.

[0196] In one possible implementation, the acquisition unit 501 is used to acquire repair configuration information when the data management system is started, and the processing unit 502 is used to scan the data blocks in the distributed file system in a first scanning mode to obtain bad block scan results, and to clean up at least one abnormal data block.

[0197] In another possible implementation, the acquisition unit 501 is further configured to acquire a first identifier when the data management system is started.

[0198] In another possible implementation, the processing unit 502 is further configured to update the first identifier after the data management system is started so that the first identifier indicates that the data management system is started normally, and to update the first identifier after the data management system is started so that the first identifier indicates that the data management system is stopped normally.

[0199] In another possible implementation, the processing unit 502 is further configured to write an exception record to the distributed coordination service component when the first identifier indicates that the data management system has stopped abnormally.

[0200] In another possible implementation, the processing unit 502 is further configured to determine, at least based on the indication information of the repair mode, whether to repair the data management system, and, if the data management system is repaired, to perform the operation of "scanning data blocks in the distributed file system in the first scan mode to obtain bad block scan results".

[0201] In another possible implementation, the processing unit 502 is further configured to determine whether to repair the data management system based on the repair mode indication information and the first identifier.

[0202] In another possible implementation, the processing unit 502 is further configured to determine to repair the data management system when the first repair mode is all scenarios, or to determine to repair the data management system when the first repair mode is abnormal stop and the first identifier indicates that the data management system was abnormally stopped during the last run.

[0203] In another possible implementation, the processing unit 502 is further configured to determine that the data management system should not be repaired when the first repair mode is closed repair, or to determine that the data management system should not be repaired when the first repair mode is abnormal stop and the first identifier indicates that the data management system did not stop abnormally during the last run.

[0204] In another possible implementation, the processing unit 502 is also configured to operate the data management system normally if it is determined that no repairs will be performed on the data management system.

[0205] In another possible implementation, the processing unit 502 is also used to repair the data management system during the repair process.

[0206] In another possible implementation, the acquisition unit 501 is further configured to acquire logical storage unit information, and the processing unit 502 is configured to obtain indication information of at least one abnormal logical storage unit based on the information in the metadata table and the logical storage unit information, and to repair at least one data table.

[0207] In another possible implementation, the processing unit 502 is also used to repair the metadata table before the management service in the data management system is started.

[0208] In another possible implementation, the acquisition unit 501 is also used to acquire logical storage unit information from the distributed file system.

[0209] In another possible implementation, the processing unit 502 is further configured to update the backup data of the structure information of the first data table in the distributed coordination service component when the structure information of the first data table changes during the provision of business services by the data management system.

[0210] For details of the aforementioned possible implementation methods, please refer to the specific descriptions in the embodiments of the data repair method above.

[0211] This application also provides a computing device. Please refer to Figure 6, which is a schematic diagram of a data repair device provided in this application embodiment. The data repair device 60 includes: a processor 601, a communication interface 602, a memory 603, and a connection line 604. The processor 601, communication interface 602, and memory 603 communicate with each other via the connection line 604. The data repair device 60 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the data repair device 60.

[0212] Processor 601 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0213] The communication interface 602 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the data repair device 60 and other devices or communication networks.

[0214] The memory 603 may include volatile memory, such as random access memory (RAM). The processor 601 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0215] The memory 603 stores executable program code, which the processor 601 executes to implement the functions of the data repair device, thereby implementing the data repair method described above. That is, the memory 603 stores instructions for executing the data repair method.

[0216] Connection line 604 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 6, but this does not imply that there is only one bus or one type of bus. Connection line 604 can include pathways for transmitting information between various components of the data repair device 60 (e.g., processor 601, memory 603, and communication interface 602).

[0217] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions that can run on a computing device or be stored on any available medium. When the computer program product is run on a computing device, it causes the computing device to perform the data repair method described above.

[0218] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives). The computer-readable storage medium includes instructions that instruct the computing device to perform the data repair method described above.

[0219] In addition, a few additional points need to be made regarding this application:

[0220] I. The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

[0221] 2. Unless otherwise stated, “multiple” means two or more.

[0222] 3. Unless otherwise specified or in case of logical conflict, the terms and / or descriptions in different embodiments of this application are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0223] IV. The various numerical designations used in this application are merely for descriptive convenience and are not intended to limit the scope of protection of this application. The magnitude of the serial numbers used in this application does not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic. For example, the terms "first," "second," "third," "fourth," and other various terminology (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0224] Furthermore, any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0225] V. The terms “comprising” and “having” and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are expressly listed, but may include other steps or modules that are not expressly listed or that are inherent to such process, method, product or device.

[0226] VI. In this application, "for indicating" can be understood as "enabling". "Enabling" can include direct enabling and indirect enabling. When describing information for enabling A, it can include whether the information directly enables A or indirectly enables A, but does not necessarily mean that the information carries A.

[0227] The information that enables the information is called the information to be enabled. In the specific implementation process, there are many ways to enable the information to be enabled, such as, but not limited to, directly enabling the information to be enabled, such as the information to be enabled itself or its index. It can also be indirectly enabled by enabling other information, where there is a relationship between the other information and the information to be enabled. It can also enable only a part of the information to be enabled, while the other parts are known or pre-agreed upon. For example, enabling specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing enabling overhead to some extent. Simultaneously, common parts of various pieces of information can be identified and enabled uniformly to reduce the enabling overhead caused by individually enabling the same information.

[0228] VII. In this application, "predefined" may include preconfiguration. For example, predefining certain information means that the information is calculated or received in advance before performing an action that uses the information. The "predefined" can be implemented by pre-storing corresponding codes, tables, or other means that can be used to indicate relevant information in the device (e.g., controller or vehicle). This application does not limit the specific implementation method.

[0229] 8. The term "storage" or "preservation" in this application can refer to storage in one or more memory devices. These memory devices can be separately configured or integrated into an encoder, decoder, processor, or communication device. Alternatively, some memory devices can be separately configured, while others can be integrated into a decoder, processor, or communication device. The type of memory can be any form of storage medium, and this is not limited.

Claims

1. A data repair method, characterized in that, The method includes: When the data management system is started, repair configuration information is obtained. The repair configuration information includes indication information of a first scan mode. The indication information of the first scan mode is used to indicate the scanning accuracy of the data blocks in the distributed file system. The first scan mode belongs to one of multiple scanning accuracy levels. The data management system includes at least one data table for storing data. Each data table forms at least one logical storage unit, and each logical storage unit corresponds to at least one data block in the distributed file system. The bad block scan results are obtained by scanning data blocks in the distributed file system using the first scanning mode. The bad block scan results include indication information of at least one abnormal data block, and the abnormal data block is abnormal in data reading and / or data writing. Clean up the at least one abnormal data block.

2. The method according to claim 1, characterized in that, The multiple scanning accuracy levels include: The first level checks whether the header information of the data block exists; The second level checks whether some data in the data table within the data block can be read. The third level checks whether all data in the data table within the data block can be read.

3. The method according to claim 1 or 2, characterized in that, The method further includes: When the data management system is started, a first identifier is obtained, which is used to indicate whether the data management system stopped abnormally during its last run.

4. The method according to claim 3, characterized in that, The method further includes: After the data management system starts, the first identifier is updated so that the first identifier indicates that the data management system has started normally. When the data management system is started, if the first identifier indicates that the data management system is started normally, the first identifier is also used to indicate that the data management system stopped abnormally during its previous operation. After the data management system is started, the first identifier is updated so that the first identifier indicates that the data management system has stopped operating normally.

5. The method according to claim 3 or 4, characterized in that, The method further includes: When the first identifier indicates that the data management system has stopped abnormally, an exception record is written to the distributed coordination service component, which is used to record the abnormal status information of the data management system.

6. The method according to any one of claims 1-5, characterized in that, The repair configuration information also includes indication information for the repair mode, and the method further includes: At least based on the indication information of the repair mode, determine whether to repair the data management system; In the event that the data management system is being repaired, the operation of "scanning data blocks in the distributed file system in the first scanning mode to obtain bad block scan results" is performed.

7. The method according to claim 6, characterized in that, The step of determining whether to repair the data management system, at least based on the indication information of the repair mode, includes: Based on the indication information of the repair mode and the first identifier, it is determined whether to repair the data management system. The first identifier is used to indicate whether the data management system stopped abnormally during its last run.

8. The method according to any one of claims 7, characterized in that, The repair mode is either all scenarios or abnormal stop; The step of determining whether to repair the data management system based on the indication information of the repair mode and the first identifier includes: When the first repair mode is all scenarios, it is determined that the data management system needs to be repaired; or, When the first repair mode is abnormal stop and the first identifier indicates that the data management system stopped abnormally during its last run, it is determined that the data management system should be repaired.

9. The method according to any one of claims 7, wherein determining whether to repair the data management system based on the indication information of the repair mode and the first identifier further comprises: When the first repair mode is off repair, it is determined that the data management system will not be repaired; or, If the first repair mode is abnormal stop and the first identifier indicates that the data management system did not stop abnormally during its last run, it is determined that the data management system will not be repaired.

10. The method according to claim 7 or 9, characterized in that, The method further includes: If it is determined that the data management system will not be repaired, the data management system shall be operated normally.

11. The method according to any one of claims 1-10, characterized in that, The repair configuration information also includes indication information on when to repair, and the method further includes: At the specified repair time, the data management system is repaired.

12. The method according to claim 11, characterized in that, The timing of the repair includes: Before the management service in the data management system is started; During the operation of the management services in the data management system.

13. The method according to any one of claims 1-12, characterized in that, The data management system further includes a metadata table, which includes logical storage unit information of the at least one data table recorded by the data management system. Each logical storage unit is managed by a corresponding logical storage unit server. The data management system further includes at least one logical storage unit server, and each of the at least one logical storage unit servers is used to deploy the logical storage unit. The method further includes: Obtain the logical storage unit information, which is used to describe the attribute values ​​of the logical storage units in the data management system; Based on the information in the metadata table and the logical storage unit information, indication information for at least one abnormal logical storage unit is obtained. The attribute values ​​of the abnormal logical storage unit differ between the metadata table and the logical storage unit information, or the abnormal logical storage unit is not deployed in the logical storage unit server. Repair at least one data table, wherein the at least one data table is a data table associated with the at least one abnormal logical storage unit.

14. The method according to claim 13, characterized in that, The repair configuration information also includes a first indication, which is used to indicate that the logical storage unit is inconsistent.

15. The method according to claim 13 or 14, characterized in that, The at least one data table includes a metadata table. The repair of at least one data table includes: Before starting the management service in the data management system, repair the metadata table.

16. The method according to any one of claims 13-15, characterized in that, The repair configuration information also includes at least one business table that needs to be repaired, and the at least one data table includes a target data table, the type of which is a business table; The target data table belongs to the at least one business table that needs to be repaired, and the target data table belongs to the data table associated with the at least one abnormal logical storage unit.

17. The method according to any one of claims 13-16, characterized in that, The step of obtaining the logical storage unit information includes: The logical storage unit information is obtained from the distributed file system, and the logical storage unit information in the distributed file system is backed up from the logical storage unit server to the distributed file system by the coprocessor.

18. The method according to any one of claims 1-17, characterized in that, The repair configuration information is defined by the user, or the repair configuration information is a default configuration.

19. The method according to any one of claims 1-18, characterized in that, The backup data of the structure information of at least one data table is stored in a distributed coordination service component. The distributed coordination service component is used to manage the server nodes of the data management system. The server nodes of the data management system include at least one logical storage unit server.

20. The method according to claim 19, characterized in that, The method further includes: When the structure information of the first data table changes during the provision of business services by the data management system, the backup data of the structure information of the first data table in the distributed coordination service component is updated, and the first data table belongs to the at least one data table.

21. A data repair device, characterized in that, The data repair apparatus includes a unit or module for performing the method according to any one of claims 1-18.

22. A data repair device, characterized in that, The data repair device includes a processor and a memory, the memory being used to store computer instructions, and the processor being used to invoke the computer instructions stored in the memory to implement the method according to any one of claims 1-18.

23. A chip, characterized in that, It includes a processor and an interface circuit, wherein the interface circuit is used to receive signals from other communication devices and transmit them to the processor, or to send signals from the processor to other communication devices. The processor is used to implement the method of any one of claims 1 to 18 through logic circuits or by executing code instructions.

24. A readable storage medium, characterized in that, The readable storage medium is used to store a computer program that, when executed by a processor, causes a communication device including the processor to perform the method of any one of claims 1 to 18.

25. A computer program product, characterized in that, When the computer program product is executed by a processor, it causes a communication device including the processor to perform the method of any one of claims 1 to 18.

Images