Method and apparatus for managing data items, and device and medium

Abstract

Provided are a method and apparatus for managing data items, a device and a medium. The method comprises: acquiring an update message of a storage system, the update message comprising an identifier of an updated data item in the storage system; detecting the identifier in a data cache of the storage system; and, in response to detecting the identifier in the data cache, deleting from the data cache a cached data item corresponding to the identifier. When data items are updated, using the example implementation of the present disclosure can automatically delete dirty data from data caches. Such proactive elimination mechanism does not rely on shorter validity period settings of the data caches, but can eliminate the dirty data only when the data is updated, thereby ensuring data consistency.

Description

Methods, apparatus, devices, and media for managing data items Technical Field

[0001] Exemplary implementations of this disclosure generally relate to data storage management, and more particularly to methods, apparatus, devices, and computer-readable storage media for managing data items in a data storage system. Background Technology

[0002] Various data storage technologies have been developed, and data storage systems can include a large number of data items. Especially in internet environments, many scenarios involve high-concurrency access to massive amounts of data. Multi-level caching can be implemented to cache individual data items, thereby improving data access efficiency. To ensure data consistency, a short expiration time is typically set for cached data items; when a data item's storage time in the cache exceeds this expiration time, it is deleted. This results in a low cache hit rate, making it difficult to fully utilize cache storage space and requiring data access requests to be passed to downstream storage devices, thus increasing the workload of downstream storage devices. Therefore, it is desirable to utilize cache storage space more effectively and improve data access efficiency.

[0003] Summary of the Invention

[0004] In a first aspect of this disclosure, a method for managing data items is provided. In this method, an update message from a storage system is obtained, the update message including an identifier of a data item being updated in the storage system. The identifier is detected in a data cache of the storage system. In response to detecting the identifier in the data cache, the cached data item corresponding to the identifier is deleted from the data cache.

[0005] In a second aspect of this disclosure, an apparatus for managing data items is provided. The apparatus includes: an acquisition module configured to acquire an update message of a storage system, the update message including an identifier of a data item being updated in the storage system; a detection module configured to detect the identifier in a data cache of the storage system; and a deletion module configured to delete a cached data item corresponding to the identifier from the data cache in response to detecting the identifier in the data cache.

[0006] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0008] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] In the following detailed description, the above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent, taken in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 shows a block diagram of an application environment according to an exemplary implementation of the present disclosure;

[0012] Figure 2 shows a block diagram for managing data items according to some implementations of this disclosure;

[0013] Figure 3 shows a block diagram of several services for managing data items according to some implementations of this disclosure;

[0014] Figure 4 shows a block diagram of a hierarchical structure for data caching according to some implementations of this disclosure;

[0015] Figure 5 shows a block diagram illustrating the different states that the data cache enters according to some implementations of this disclosure;

[0016] Figure 6 shows a block diagram of the data rows in an update message according to some implementations of this disclosure;

[0017] Figure 7 shows a flowchart of a method for managing data items according to some implementations of this disclosure;

[0018] Figure 8 shows a block diagram of an apparatus for managing data items according to some implementations of the present disclosure; and

[0019] Figure 9 shows a block diagram of a device capable of implementing various implementations of the present disclosure. Detailed Implementation

[0020] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0027] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0028] Example Environment

[0029] In the internet environment, many scenarios involve high-concurrency access to massive amounts of data. Currently, multi-level caching has been proposed to cache data items in data storage systems, thereby improving data access efficiency. Referring to Figure 1, which describes an application environment according to some implementations of this disclosure, Figure 1 shows a block diagram 100 of an application environment according to an exemplary implementation of this disclosure. As shown in Figure 1, a user 112 can access data provided at a server 120 via an application running at a client 110. The server 120 may include a storage system 126, which may include a database system implemented based on various architectures.

[0030] It should be understood that server 120 may include one or more devices. To improve data access speed, a multi-level caching architecture can be provided. Specifically, a local cache 122 and a distributed cache 124 can be provided to cache various data items from storage system 126. Local cache 122 may reside in the memory of the service instance (e.g., a microservice instance) providing the data access service. In response to receiving an access request, if the target data item specified in the access request cannot be obtained from local cache 122, the access request is forwarded to the downstream distributed cache 124. If distributed cache 124 contains the target data item, the target data item can be retrieved from distributed cache 124; otherwise, the access request is forwarded to downstream storage system 126 to retrieve the target data item from storage system 126.

[0031] To ensure data consistency, cached data items typically have short expiration times. Data items are deleted when their storage time exceeds this expiration period. Local caches reside in the memory of each microservice instance. Currently, mainstream local caches usually automatically evict expired data in memory by setting short expiration times. This ensures that cached data in memory is invalidated promptly, preventing dirty data from appearing in the cache (i.e., the original data in storage system 126 is updated to a new version, but the data in the cache is not updated and remains the old version, leading to inconsistency between the data in the cache and the data in storage system 126). Dirty data causes a degradation in data access consistency (the new version should be returned, but the old version is actually returned).

[0032] While the automatic expiration and eviction process is relatively simple, its short expiration period (typically set to a few seconds, such as 5 seconds or other values, to meet data consistency requirements) means that only a limited number of data items can be cached locally, resulting in a low cache hit rate. In high-concurrency scenarios, a large number of data items may be accessed in a short period of time, and the low cache hit rate makes it difficult to fully utilize the cache's storage space. Furthermore, data access requests need to be passed to downstream storage devices, increasing the workload of those devices. Therefore, it is desirable to utilize the cache's storage space more effectively and improve the cache hit rate.

[0033] Overview of Data Item Management

[0034] To at least partially address the shortcomings of the prior art, a method for managing data items is proposed according to an exemplary implementation of this disclosure. In summary, improving the hit rate of a data cache (e.g., local cache 122) requires increasing its validity period, thereby caching more valid data items. To address the problem of data consistency degradation due to the lack of a mechanism for timely removal of dirty data, recently updated data items can be collected and pushed to the service instance handling data access requests to proactively delete dirty data from the data cache.

[0035] Referring to Figure 2, which describes an outline of an exemplary implementation of the present disclosure, Figure 2 illustrates a block diagram 200 for managing data items according to some implementations of the present disclosure. As shown in Figure 2, the storage system 126 may include a large number of data items, such as data item 220, etc. An update message 210 may be obtained, which includes an identifier 212 of the data item (e.g., data item 220) that has been updated in the storage system 126. The identifier 212 may be detected in the data cache 230 of the storage system 126. In response to the detection of the identifier 212 in the data cache 230 of the storage system 126, the cached data item 220' corresponding to the identifier 212 may be deleted from the data cache 230.

[0036] Using some implementation methods disclosed herein, dirty data (i.e., data item 220') in the data cache can be automatically deleted when a data item is updated. In this way, the proposed proactive eviction mechanism does not rely on a short validity period setting for the data cache 230, but can evict dirty data only when data is updated, thereby ensuring data consistency. Therefore, a longer validity period can be set, and more data items can be cached, significantly improving the hit rate of the local cache. Furthermore, long-term data caching can be disabled through latency tolerance and error tolerance to prevent long-term data caching from failing to retrieve the most recently updated data items in abnormal scenarios, thereby mitigating the problem of data consistency degradation.

[0037] Detailed process of data item management

[0038] Having described an overview of some implementations according to this disclosure, further details of data item management will be described below. Figure 3 shows a block diagram 300 of multiple services for managing data items according to some implementations of this disclosure. As shown in Figure 3, a data collection service 310, a data push service 320, and an online query service 330 may be provided. In summary, the data collection service 310 may proactively push the identifier of the most recently updated data item. Only the most recently updated data item will cause the data in the data cache of the query service to become dirty data, thereby returning an outdated data version and causing degradation of data consistency. The data push service 320 may receive data input from the data collection service 310 and access (including write and read operations) the data storage 324 accordingly. The online query service 330 may provide online query functionality to client devices.

[0039] According to some implementations of this disclosure, the various services described above can be provided by any computing device with computing capabilities. Here, the various services can communicate via a network to collaboratively complete the technical solutions of this disclosure. According to some implementations of this disclosure, update messages can be determined by the data collection service 310. Specifically, the data collection service 310 can collect updates from the data storage 310.

[0040] Specifically, various data storage technologies provide the capability to notify the storage system of data change records. Taking a conventional relational database as an example, one can subscribe to the database's master-slave synchronization to write these change records to a message queue. Specifically, log data (e.g., "Binlog") can be provided, and a database log data collection service can be initiated. Here, log data can be used to record database data modification operations, primarily for data replication, data recovery, and auditing data changes. When a data item in the database is updated, the database can automatically create a log record and add it to the log data.

[0041] During the generation of an update message, in response to the detection that log data in the storage system has been updated, an identifier is determined based on the identifier of the data item corresponding to the log record; and a timestamp can be determined based on the update time of the updated log record in the log data. The log record may include the identifier of the updated data item (e.g., a key) and the timestamp of the update (i.e., creation). The identifier can then be extracted from the log record to generate the update message. Alternatively and / or additionally, the update message may further include a timestamp, thereby allowing the extracted timestamp to be added to the update message.

[0042] According to some implementations of this disclosure, the data push service 320 may include three main nodes: a write service 322, a storage system 324, and a read service 326. Specifically, the write service 322 is responsible for data push write operations, that is, receiving data pushed from upstream, performing batch aggregation in memory, and then writing it into the internal storage system 324. The purpose of performing batch aggregation here is mainly to improve the write performance of the storage system and reduce the number of calls to the storage system.

[0043] Storage system 324 is primarily responsible for storing the pushed data and generating auto-incrementing IDs according to the order in which the data is written (specifically, the ID assigned to each row of data to be pushed is a strictly monotonically increasing positive integer). These IDs serve as the key for data queries, preventing data omissions and duplications. Read service 326 is responsible for the data push read service, providing the function of querying pushed data. It can periodically load the latest data from storage system 324 into memory to update the data cache. Furthermore, read service 326 can provide a query interface to query data in storage system 324. The auto-incrementing IDs serve as a unified key during data loading and querying, ensuring the orderliness and integrity of the data.

[0044] In this way, data items can be cached for a longer period in the data cache, thereby achieving a high cache hit rate. At this point, the online query service 330 can provide the function of querying the cached data. Furthermore, the SDK (Software Development Kit) in the online query service 330 can integrate interfaces for corresponding functions, thereby supporting the access party to quickly implement the desired functionality. Specifically, the data push read service 320 can be polled at predetermined time intervals to retrieve relevant information of the most recently updated data items.

[0045] Specifically, after retrieving the identifier of a data item, the system can check if the identifier is present in the data cache and then proactively delete any dirty data associated with that identifier from the cache. In this way, when a client accesses the data item again, since the cache does not contain a cached version of that data item, the access request needs to be forwarded to other downstream storage devices to retrieve the latest version of the data item, and then the latest version of the data item is stored in the data cache. This method reuses the cache miss operation process, thereby ensuring data consistency.

[0046] According to some implementations of this disclosure, the data cache may include a first-level cache and a second-level cache, with the first-level cache always enabled. See Figure 4 for further details, which shows a block diagram 400 of the hierarchical structure of the data cache according to some implementations of this disclosure. As shown in Figure 4, the data cache 230 may include an L0 cache 410 and an L1 cache 420. Here, the capacity, refresh frequency, and access speed of the two cache levels may differ. The L0 cache 410 may always be enabled and used to cache data items.

[0047] According to some implementations of this disclosure, the first capacity of the first-level cache can be smaller than the second capacity of the second-level cache, the first refresh frequency of the first-level cache can be higher than the second refresh frequency of the second-level cache, and the first access speed of the first-level cache can be higher than the second access speed of the second-level cache. For example, L0 cache 410 can be a short-lived cache (e.g., with a validity period of several seconds, and expired data items are refreshed and cleared when the validity period expires). L1 cache 420 can be a long-lived cache (e.g., with a validity period of tens of seconds or longer, and expired data items are refreshed and cleared when the validity period expires). In this way, the two-level caches can clear expired data according to their own validity periods, thereby supporting the caching of data items according to different validity periods.

[0048] Since the L1 cache is updated based on push operations, it can be referred to as the push cache for ease of description. According to some implementation methods of this disclosure, a data push fault tolerance mechanism can be provided. Specifically, the push cache may be automatically disabled if an exception occurs when the SDK fetches data. To ensure that the original short-lived cache remains effective after the push cache is disabled, the data cache can adopt a multi-level caching design. The data cache is divided into two levels, L0 and L1, where L0 is a regular short-lived cache and L1 is a long-lived cache. Access follows a multi-level caching approach: first accessing the L0 cache; if the L0 cache misses, then accessing the L1 cache continues. The L1 cache may be disabled, but the L0 cache is always enabled. Even if the L1 cache is disabled, the L0 cache can still continue to function normally. Compared to restarting a new short-lived local cache after the L1 cache is disabled, the L0 cache of this disclosure does not require preheating, ensuring that the cache hit rate does not instantly drop to zero, thus ensuring a smoother degradation process.

[0049] According to some implementations of this disclosure, the update message further includes: a timestamp of the data item being updated. At this point, the time difference between the timestamp and the current timestamp can be determined; and the state of the data cache can be set based on the time difference. It should be understood that a larger time difference indicates a greater delay in the arrival of the update notification, and a higher probability that the data item will be updated again during that delay. By using the time difference to set different states for the data cache, different processes can be used to perform subsequent operations, thereby improving the consistency and efficiency of data access.

[0050] According to some implementation methods of this disclosure, setting the data cache state based on time differences includes setting the second-level cache state based on time differences. Specifically, the push cache can be managed by SDK 332, in which case the SDK can periodically send query requests to the read service 326. Because the query requests are made via network requests, fault tolerance strategies must be considered when network requests fail. Here, three different states can be provided, and each state can correspond to a dedicated operation process.

[0051] According to some implementation methods of this disclosure, the normal (OK) state can mean that update notifications can be pulled normally from the data push service 320. At this time, the pulled data can be traversed normally, and then the corresponding dirty data in the data cache can be deleted.

[0052] According to some implementations of this disclosure, an "Error" state can indicate a failure to retrieve update notifications, such as encountering consecutive errors or continuously retrieving empty data while polling the data push service 320 to retrieve data. In this state, because the identifier of the updated data item cannot be obtained, the online query service 330 cannot promptly delete the corresponding dirty data in the data cache. To prevent further degradation of data consistency, the push cache can be automatically disabled.

[0053] According to some implementations of this disclosure, the "Chasing" state can represent the following: data can be retrieved, but the data latency (the time it takes for pushed data to be retrieved by the online query service node) exceeds a predetermined threshold. In this case, the system enters the chasing state because the high latency of the retrieved data prevents the timely deletion of dirty data in the cache, which also leads to data consistency degradation. At this point, the push cache can be automatically disabled, and the relevant dirty data can continue to be deleted until the data latency returns to a normal level, at which point the push cache can be re-enabled.

[0054] According to some implementations of this disclosure, setting the state of the second-level cache based on time difference can include at least one of the following: enabling the second-level cache in response to determining that the difference meets a predetermined threshold time; or disabling the second-level cache in response to determining that the difference does not meet a predetermined threshold time. Alternatively and / or additionally, disabling the second-level cache in response to determining that obtaining an update message has failed; and resetting the second-level cache. Specifically, the state of the second-level cache can be set using the enable / disable module 422 shown in Figure 4.

[0055] See Figure 5 for further details regarding the state settings. Figure 5 shows block diagrams 500 illustrating the different states of the data cache according to some implementations of this disclosure. As shown in Figure 5, the push cache 540 can operate under the management of the data push SDK 550. Specifically, at block 510, in response to receiving an update message within a predetermined threshold time (i.e., the delay is less than the predetermined threshold time), the system enters a normal state. At this point, the push cache can be enabled at block 512, and subsequently, dirty data can be cleaned up at block 514. Specifically, data items corresponding to the identifiers in the update messages can be removed from the push cache.

[0056] According to some implementations of this disclosure, in order to delete the cached data item corresponding to the identifier from the data cache, the cached data item corresponding to the identifier can be deleted from both the first-level cache and the second-level cache. At this point, the entire data cache will no longer contain dirty data corresponding to that identifier, and data consistency can be guaranteed based on the cache miss handling process. Under normal conditions, both the L0 cache and the L1 cache are effective, and can respectively provide cached data items valid for short periods and cached data items valid for long periods. In this way, the probability of cache hits can be increased, thereby improving data access performance.

[0057] Alternatively and / or additionally, at box 530, in response to not receiving an update message within a predetermined threshold time (i.e., the delay exceeds the predetermined threshold time), a catch-up state is entered. At this time, the push cache can be disabled at box 532, and subsequently, dirty data can be cleaned up at box 534. Specifically, data items corresponding to the identifiers in the update messages can be removed from the push cache. Using some implementations of this disclosure, the push cache is disabled, and the entire data cache is downgraded to include only the L0 cache. At this time, the L0 cache can continue to serve, providing cached data items that are valid for a short period.

[0058] Alternatively and / or additionally, at box 520, in response to a failure during the reception of an update message (i.e., no update message is received for an extended period, or the update message is empty), an error state is entered. In this case, push caching can be disabled at box 522, and then reset at box 524, for example, by clearing the push cache. Since push caching is disabled, the entire data cache will be downgraded to include only the L0 cache. The L0 cache can continue to serve, providing cached data items valid for a short period. Push caching no longer includes any data items and will be re-enabled after the next successful reception of an update message.

[0059] It should be understood that although the process of managing the data cache was described above using the example of an update message including only the identifier and timestamp of a single data item, alternatively and / or additionally, an update message may include multiple data rows, and each data row may include the identifier and timestamp of a single data item. See Figure 6 for further details, which shows a block diagram 600 of a data row in an update message according to some implementations of this disclosure. As shown in Figure 6, a data row 610 may include the identifier 212 of the updated data item and the timestamp at which the data item was updated.

[0060] According to some implementations of this disclosure, the data push service 320 can be polled at predetermined time intervals to determine the most recently updated data items. For example, in each poll, the identifiers and timestamps of all updated data items can be obtained, and the identifiers and timestamps of each data item can be processed one by one to execute the process described in Figure 5. In this way, the process of pulling update notifications can be simplified, and the relevant data of all updated data items can be transmitted at once. It should be understood that although the number of updated data items may be large, since each data row does not occupy a large amount of storage, excessive data transmission load will not be generated.

[0061] Alternatively and / or additionally, batch size thresholds can be set. For example, update messages can be configured to include only N data rows, allowing multiple updated data items to be processed in multiple batches when more data items are updated. This reduces the amount of data transferred during a single fetch process when a large number of data items are updated, thereby reducing peak workload. Alternatively and / or additionally, for a specific service instance, only update notifications related to data items previously accessed by that service instance can be transmitted. This allows for more granular processing of individual service instances, further reducing the workload of each service instance.

[0062] According to some implementation methods disclosed herein, the data push SDK can determine whether the data retrieval latency is too high during the polling process. Specifically, the current timestamp can be subtracted from the timestamp parsed from the data row of the update message; the difference between the two is the end-to-end latency of the update message. Here, the timestamp in the update message should meet reasonable values. If the timestamp generated in real time when the collection service 316 consumes each log record is selected, it may lead to accumulation and distortion of the timestamp, thus preventing the online query service 330 from perceiving the latency when retrieval data. Therefore, the generation time of the Binlog message itself (BornTime, i.e., the generation time of the Binlog message itself) should be used here, which will not be affected by abnormal situations such as message accumulation in the collection service when consuming Binlog.

[0063] According to some implementations of this disclosure, a data cache managed according to the methods described above can be used to serve query requests. Specifically, in response to receiving an access request from a client for accessing a target data item in the storage system, it is determined whether the data cache includes the target data item; and in response to determining that the data cache includes the target data item, the target data item in the data cache is returned to the client. Using some implementations of this disclosure, if the target data item is found in the data cache (whether it is the first-level cache L0 or the second-level cache L1), the target data item can be returned directly. In this way, it is not necessary to perform queries on other downstream data stores, thereby improving data response speed.

[0064] According to some implementations of this disclosure, in response to receiving an access request from a client for accessing a target data item in the storage system, it can be determined whether the data cache includes the target data item; in response to determining that the data cache does not include the target data item, the target data item is requested from the downstream data storage of the data cache; the requested target data item is written to the data cache, and the target data item is returned to the client. Using some implementations of this disclosure, if the target data item is not found in the data cache (first-level cache L0 and / or second-level cache L1), it indicates a cache miss. At this time, the downstream data storage can be queried to request the target data item. In this way, the cache miss handling process can be reused to solve data consistency problems, thereby ensuring that the client device can obtain the latest version of the target data item.

[0065] It should be understood that a single-level and / or multi-level caching structure can exist between the data cache and the storage system. In this case, the data storage includes at least one of the following: the storage system, or another data cache between the data cache and the storage system (e.g., the distributed cache 124 shown in Figure 1). In this way, faster data storage can be accessed preferentially through a step-by-step pass-through method.

[0066] According to some implementations of this disclosure, in determining whether the data cache includes the target data item, the first-level cache can be accessed firstly based on access speed. In response to determining that the first-level cache does not include the target data item, it can be determined whether the second-level cache includes the target data item. In this way, if the target data item is found in the faster first-level cache, the expected data can be returned directly. This improves data access speed.

[0067] Using some implementation methods disclosed herein, dirty data in the data cache can be automatically deleted when data items are updated. Furthermore, latency tolerance and error tolerance can be set to automatically degrade and disable long-lived local caches, preventing them from failing to retrieve the most recently updated data items in abnormal scenarios, thus avoiding potential data consistency issues. The data cache is implemented using a multi-level caching approach, with the push cache located in the L1 cache and flexibly enabled and disabled. The L0 cache can be used as a short-lived cache and can smoothly degrade to the short-lived L0 cache when the L1 cache is disabled. In the event of network anomalies, the data cache will not suffer from data consistency degradation due to the inability to fetch update messages in a timely manner. Instead, the enabling / disabling of the push cache can be flexibly determined based on the fetch results.

[0068] Example process

[0069] Figure 7 illustrates a flowchart of a method 700 for managing data items according to some implementations of this disclosure. At block 710, an update message from the storage system is obtained, the update message including an identifier of the data item being updated in the storage system. At block 720, the identifier is detected in the data cache of the storage system. At block 730, in response to the detection of the identifier in the data cache, the cached data item corresponding to the identifier is deleted from the data cache.

[0070] According to some implementations of this disclosure, the update message further includes: the timestamp of the data item being updated, and method 700 further includes: determining the time difference between the timestamp and the current timestamp; and setting the state of the data cache based on the time difference.

[0071] According to some implementations of this disclosure, the data cache includes a first-level cache and a second-level cache. The first-level cache is enabled, and the state of the data cache is set based on time differences, including setting the state of the second-level cache based on time differences.

[0072] According to some implementations of this disclosure, the state of the second-level cache based on time difference includes at least one of the following: enabling the second-level cache in response to determining that the difference meets a predetermined threshold time; or disabling the second-level cache in response to determining that the difference does not meet a predetermined threshold time.

[0073] According to some implementations of this disclosure, deleting cached data items corresponding to an identifier from the data cache includes deleting cached data items corresponding to an identifier from both the first-level cache and the second-level cache.

[0074] According to some implementations of this disclosure, the method 700 further includes: disabling the second-level cache in response to determining that obtaining the update message has failed; and resetting the second-level cache.

[0075] According to some implementations of this disclosure, the method 700 further includes: in response to receiving an access request from a client for accessing a target data item in a storage system, determining whether a data cache includes the target data item; and in response to determining that the data cache includes the target data item, returning the target data item in the data cache to the client.

[0076] According to some implementations of this disclosure, the method 700 further includes: in response to receiving an access request from a client for accessing a target data item in a storage system, determining whether a data cache includes the target data item; in response to determining that the data cache does not include the target data item, requesting the target data item from a data storage downstream of the data cache, the data storage including at least one of the following: a storage system, or another data cache between the data cache and the storage system; writing the requested target data item to the data cache; and returning the target data item to the client.

[0077] According to some implementations of this disclosure, determining whether the data cache includes the target data item includes: in response to determining that the first-level cache does not include the target data item, determining whether the second-level cache includes the target data item.

[0078] According to some implementations of this disclosure, the update message is determined based on the following: in response to detecting that the log data of the storage system has been updated, a timestamp is determined based on the update time of the updated log record in the log data; and an identifier is determined based on the identifier of the data item corresponding to the log record.

[0079] According to some implementations of this disclosure, the first capacity of the first-level cache is smaller than the second capacity of the second-level cache, the first refresh frequency of the first-level cache is higher than the second refresh frequency of the second-level cache, and the first access speed of the first-level cache is higher than the second access speed of the second-level cache.

[0080] Example devices and equipment

[0081] Figure 8 shows a block diagram of an apparatus 800 for managing data items according to some implementations of the present disclosure. The apparatus 800 includes: an acquisition module 810 configured to acquire an update message from a storage system, the update message including an identifier of a data item being updated in the storage system; and a deletion module 820 configured to delete a cached data item corresponding to an identifier from the data cache in response to detecting the identifier in the data cache of the storage system.

[0082] According to some implementations of this disclosure, the update message further includes: a timestamp of the data item being updated, and the device further includes a management module configured to: determine the time difference between the timestamp and the current timestamp; and set the state of the data cache based on the time difference.

[0083] According to some implementations of this disclosure, the data cache includes a first-level cache and a second-level cache, the first-level cache is enabled, and the management module is configured to: set the status of the second-level cache based on time differences.

[0084] According to some implementations of this disclosure, the management module is further configured to: enable the second-level cache in response to determining that the difference meets a predetermined threshold time; or disable the second-level cache in response to determining that the difference does not meet a predetermined threshold time.

[0085] According to some implementations of this disclosure, the deletion module is further configured to delete cached data items corresponding to the identifier from the first-level cache and the second-level cache.

[0086] According to some implementations of this disclosure, the management module is further configured to: disable the second-level cache in response to determining that obtaining an update message has failed; and reset the second-level cache.

[0087] According to some implementations of this disclosure, the management module is further configured to: in response to receiving an access request from a client for accessing a target data item in the storage system, determine whether the data cache includes the target data item; and in response to determining that the data cache includes the target data item, return the target data item in the data cache to the client.

[0088] According to some implementations of this disclosure, the management module is further configured to: in response to receiving an access request from a client for accessing a target data item in the storage system, determine whether the data cache includes the target data item; in response to determining that the data cache does not include the target data item, request the target data item from a data storage downstream of the data cache, the data storage including at least one of the following: the storage system, or another data cache between the data cache and the storage system; write the requested target data item to the data cache; and return the target data item to the client.

[0089] According to some implementations of this disclosure, the management module is further configured to: determine whether the second-level cache includes the target data item in response to determining that the first-level cache does not include the target data item.

[0090] According to some implementations of this disclosure, the update message is determined based on the following: in response to detecting that the log data of the storage system has been updated, a timestamp is determined based on the update time of the updated log record in the log data; and an identifier is determined based on the identifier of the data item corresponding to the log record.

[0091] According to some implementations of this disclosure, the first capacity of the first-level cache is smaller than the second capacity of the second-level cache, the first refresh frequency of the first-level cache is higher than the second refresh frequency of the second-level cache, and the first access speed of the first-level cache is higher than the second access speed of the second-level cache.

[0092] Figure 9 shows a block diagram of a device 900 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 900 shown in Figure 9 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. The computing device 900 shown in Figure 9 can be used to implement the methods described above.

[0093] As shown in Figure 9, the computing device 900 is in the form of a general-purpose computing device. Components of the computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 900.

[0094] Computing device 900 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 900.

[0095] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 9, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. The memory 920 may include a computer program product 925 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0096] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 900 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0097] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 900 can also communicate as needed with one or more external devices (not shown) via communication unit 940. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 900, or with any device (e.g., network card, modem, etc.) that enables computing device 900 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0098] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0099] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0100] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0101] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0103] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for managing data items, comprising: Obtain an update message from the storage system, the update message including an identifier of the data item being updated in the storage system; The identifier is detected in the data cache of the storage system; as well as In response to detecting the identifier in the data cache, the cached data item corresponding to the identifier is deleted from the data cache.

2. The method of claim 1, wherein the update message further comprises: The timestamp at which the data item is updated, and the method further include: Determine the time difference between the timestamp and the current timestamp; and The state of the data cache is set based on the time difference.

3. The method of claim 2, wherein the data cache includes a first-level cache and a second-level cache, the first-level cache is enabled, and setting the state of the data cache based on the time difference includes: The state of the secondary cache is set based on the time difference.

4. The method of claim 3, wherein setting the state of the second-level cache based on the time difference includes at least one of the following: In response to determining that the difference meets a predetermined threshold time, the second-level cache is enabled; or In response to determining that the difference does not meet a predetermined threshold time, the second-level cache is disabled.

5. The method of claim 3, wherein deleting the cached data item corresponding to the identifier from the data cache comprises: Delete the data item in the cache corresponding to the identifier from the first-level cache and the second-level cache.

6. The method of claim 3, further comprising: In response to the determination that obtaining the update message has failed, the second-level cache is disabled; as well as Reset the second-level cache.

7. The method of claim 2, further comprising: In response to receiving an access request from a client for accessing a target data item in the storage system, determine whether the data cache includes the target data item; as well as In response to determining that the data cache includes the target data item, the target data item in the data cache is returned to the client.

8. The method of claim 3, further comprising: In response to receiving an access request from a client for accessing a target data item in the storage system, determine whether the data cache includes the target data item; In response to determining that the data cache does not include the target data item, the target data item is requested from a data storage downstream of the data cache, wherein the data storage includes at least one of the following: the storage system, or another data cache between the data cache and the storage system; Write the requested target data item to the data cache; as well as The target data item is returned to the client.

9. The method of claim 7, wherein determining whether the data cache includes the target data item comprises: In response to determining that the first-level cache does not include the target data item, determine whether the second-level cache includes the target data item.

10. The method of claim 2, wherein the update message is determined based on the following: In response to detecting an update to the log data of the storage system, the timestamp is determined based on the update time of the updated log record in the log data; and The identifier is determined based on the identifier of the data item corresponding to the log record.

11. The method according to claim 3, wherein the first capacity of the first-level cache is smaller than the second capacity of the second-level cache, the first refresh frequency of the first-level cache is higher than the second refresh frequency of the second-level cache, and the first access speed of the first-level cache is higher than the second access speed of the second-level cache.

12. An apparatus for managing data items, comprising: The acquisition module is configured to acquire update messages from the storage system, the update messages including identifiers of data items that have been updated in the storage system; A detection module is configured to detect the identifier in the data cache of the storage system; as well as A deletion module is configured to delete a cached data item corresponding to the identifier from the data cache in response to detecting the identifier in the data cache.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement the method according to any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1 to 11.

Images