Method and apparatus for managing data processing system, and device and medium
By obtaining the message queue delay time in the data processing system and starting the query, the problem of increased system complexity in the prior art is solved, and real-time monitoring of the data item processing status and optimization of system performance are achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- BEIJING ZITIAO NETWORK TECH CO LTD
- Filing Date
- 2025-01-27
- Publication Date
- 2026-07-30
AI Technical Summary
Existing technologies require code modifications to determine the processing status of data items when managing data processing systems, which increases system complexity and lacks a simple and effective way to query the processing status of data items at each processing node.
By retrieving a set of messages from the message queue of the data processing system, determining the corresponding delay time, and launching a queryer based on the delay time to query the status of data items in the processing nodes, the data processing system can be managed.
It enables real-time monitoring of the data processing system's operational status, optimization of system performance, identification of bottlenecks and workloads, and improvement of system management efficiency without modifying existing code.
Smart Images
Figure CN2025075498_30072026_PF_FP_ABST
Abstract
Description
Methods, apparatus, devices, and media for managing data processing systems Technical Field
[0001] Implementations of this disclosure generally relate to computer technology, and in particular to methods, apparatus, devices, and computer-readable storage media for managing data processing systems. Background Technology
[0002] Data processing systems can handle various types of data items and may include multiple processing nodes. Data items can pass through multiple processing nodes to generate a final processing result, resulting in a relatively long data processing chain. During data processing, it is desirable to query the processing status of data items at each processing node. Currently, checkpoint-based methods have been proposed to determine the status of each data item; however, this approach requires code modification and increases the complexity of the data processing system. Therefore, a simpler and more efficient method is desired to determine the processing status of data items, thereby managing the data processing system. Summary of the Invention
[0003] In a first aspect of this disclosure, a method for managing a data processing system is provided. In this method, a set of messages is retrieved from a message queue of the data processing system, each message corresponding to a set of data items in the data processing system. A delay time corresponding to the set of messages is determined, the delay time indicating the query time for querying the processing status of a set of data items in at least one processing node within the data processing system. Based on the delay time, at least one queryer in the data processing system is initiated, the set of data items being processed by at least one processing node, and the at least one queryer is used to query the processing status of the set of data items in at least one processing node. The data processing system is managed based on the processing status.
[0004] In a second aspect of this disclosure, an apparatus for managing a data processing system is provided. The apparatus includes: an acquisition module configured to acquire a set of messages from a message queue of the data processing system, each set of messages corresponding to a set of data items in the data processing system; a determination module configured to determine a delay time corresponding to the set of messages, the delay time indicating a query time for querying the processing status of a set of data items in at least one processing node within the data processing system; a startup module configured to start at least one queryer in the data processing system based on the delay time, wherein the set of data items is processed by at least one processing node, and the at least one queryer is used to query the processing status of the set of data items in at least one processing node; and a management module configured to manage the data processing system based on the processing status.
[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.
[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.
[0007] In a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to a first aspect of this disclosure.
[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0009] The above and other features, advantages, and aspects of the various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:
[0010] Figure 1 shows a block diagram of an application environment according to one implementation of the present disclosure;
[0011] Figure 2 shows a block diagram of a data processing management system according to some implementations of this disclosure;
[0012] Figure 3 shows a block diagram of obtaining a message queue according to some implementations of this disclosure;
[0013] Figure 4 shows a block diagram of a data processing system managed by producers and consumers based on message queues according to some implementations of this disclosure;
[0014] Figure 5 shows a block diagram of the processing status of query data items according to some implementations of this disclosure;
[0015] Figure 6 shows a block diagram of the state of a data processing system according to some implementations of this disclosure;
[0016] Figure 7 shows a block diagram of the state of a data processing system according to some implementations of this disclosure;
[0017] Figure 8 shows a flowchart of a method for managing a data processing system according to some implementations of the present disclosure;
[0018] Figure 9 shows a block diagram of an apparatus for managing a data processing system according to some implementations of the present disclosure; and
[0019] Figure 10 shows a block diagram of a device capable of implementing various implementations of the present disclosure. Detailed Implementation
[0020] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.
[0021] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.
[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.
[0023] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.
[0024] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.
[0025] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.
[0026] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.
[0027] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.
[0028] Example Environment
[0029] The data processing system can process various types of data items. The system may include multiple processing nodes, and each data item can pass through multiple processing nodes to generate a final processing result. In this case, the data processing chain is relatively long, and it is desirable to query the processing status of data items at each processing node. Referring to Figure 1, which illustrates a block diagram 100 of an application environment according to one implementation of this disclosure, a user can submit data items to the data processing system 140 via application 110. Application 110 can present media items 120, which may include various types, such as, but not limited to, videos, short videos, music, text, images, games, or rich media data including combinations of the above types. Application 110 can provide controls 130, which users can click to provide comments about the media items. Here, the media items and comments provided in application 110 can be collectively referred to as data items.
[0030] Data processing system 140 can receive data items and process each data item using multiple processing nodes 142, 144, ..., 146. In the data processing system, data items will pass through multiple processing nodes. To improve the operating efficiency of the data processing system, it is desirable to determine the state of data items within the system. Currently, a checkpoint-based approach has been proposed to determine the state of each data item; however, this approach requires code modification and increases the complexity of the data processing system. Therefore, a simpler and more efficient method is desired to determine the processing state of data items, thereby managing the data processing system.
[0031] System Management Overview
[0032] To at least partially address the shortcomings of the prior art, a method for managing a data processing system is proposed according to one implementation of this disclosure. Referring to Figure 2, which describes an overview of one implementation of this disclosure, Figure 2 shows a block diagram 200 for managing a data processing system according to some implementations of this disclosure. As shown in Figure 2, a set of messages 220 can be retrieved from a message queue 210 of the data processing system. This set of messages may include one or more messages, and each set of messages corresponds to a set of data items in the data processing system. In the context of this disclosure, a set of "XX" may include one or more "XX". For example, a set of messages may include one or more messages, that is, at least one message. Assuming that message queue 210 includes N messages, the first set of messages may include N1 messages, the second set of messages may include N2 messages, and so on.
[0033] For ease of description, a media sharing system will be used as an example of a data processing system to describe the specific management process. In this case, data items may include: media items published by objects (e.g., users) in the media sharing system, and / or comments on media items published by objects in the media sharing system. Using some implementations of this disclosure, various different types of data items can be processed in a similar manner, thereby managing the data processing system in a simpler and more efficient way.
[0034] Message queue 210 may include at least one set of messages, and each set of messages may have its own delay time. The delay time corresponding to a set of messages can be determined, and the delay time indicates the query time for querying the processing status of a set of data items in at least one processing node in the data processing system. Here, the data processing system may include a set of processing nodes, and each processing node may be invoked in a predetermined order. For example, the delay time of the first set of messages may be represented as ΔT1, the delay time of the second set of messages may be represented as ΔT2, and so on.
[0035] The data processing system 140 may include at least one queryer 240, which can query the processing status of a data item within a processing node. Assuming the data processing system includes M processing nodes, there can be M queries corresponding to each of the M processing nodes. Furthermore, at least one queryer 240 in the data processing system can be activated based on a delay time 230. A set of data items is processed by at least one processing node, and at least one queryer is used to query the processing status 250 of a set of data items within at least one processing node. The data processing system can be managed based on the processing status 250. For example, bottlenecks in the data processing system, the workload of each processing node, etc., can be determined based on the processing status, thereby optimizing the data processing system.
[0036] Using some implementation methods disclosed herein, the delay time can be specified as a point in time during the system's operation, and the processing status of query data items can be initiated at the specified point in time. In this way, the operating status of the data processing system can be monitored in real time.
[0037] Detailed process of system management
[0038] Having described an outline of some implementations according to this disclosure, further details regarding methods for managing the data processing system will be described below. According to some implementations of this disclosure, during the operation of the data processing system 140, messages can be continuously received from a message queue. Messages are generated based on data items within the data processing system. Specifically, a user can submit media items to the data system 140 via an application, and various processing nodes within the data processing system 140 can process the received media items. For example, using comments as an example of data items, the data processing system 140 can receive submitted comments, a processing node can verify whether the user has the right to post a comment, a processing node can write the comment to a database, a processing node can verify whether the comment content meets predetermined requirements (e.g., whether the comment's word count meets a predetermined range, etc.), a processing node can post the comment, and so on.
[0039] According to some implementations of this disclosure, a message queue can be determined in the following manner. Specifically, data items can be received from a client device connected to the data processing system; a message can be generated based on the data item, the message including the identifier of the data item and attribute information associated with the data item, and the message can be added to the message queue. See Figure 3 for further details regarding the message queue, which shows a block diagram 300 of obtaining a message queue according to some implementations of this disclosure. As shown in Figure 3, a message can correspond to a data item; for example, messages 312, ..., can be generated based on data item 316, and message 314 can be generated based on data item 318.
[0040] According to some implementations of this disclosure, a message may include multiple dimensions: an identifier for the data item, and relevant information about the user who published the data item (e.g., the user's identifier, the identifier of the client device used by the user, etc.). It should be understood that the data processing system may have a distributed architecture, and the received data items may come from different regions; therefore, it is necessary to determine a globally unique identifier for each data item in order to accurately distinguish each data item.
[0041] According to some implementations of this disclosure, configuration information can be used to describe the grouping method of the message queue. For example, it can be specified that N1 messages in the message queue are divided into a first group of messages, and N2 messages in the message queue are divided into a second group of messages, and so on. Alternatively and / or additionally, the configuration information can specify that each group includes the same number (e.g., M) of messages.
[0042] According to some implementations of this disclosure, a corresponding message can be generated for each received data item, in which case the message queue includes multiple messages corresponding to all data items. Alternatively and / or additionally, a subset of data items can be randomly selected from the received data items, and corresponding messages can be generated for the selected subset. For example, a subset of data items can be selected according to a predetermined time interval or a predetermined proportion. In this way, when the data volume is large, it is not necessary to process each data item individually; instead, representative data items can be selected to generate messages.
[0043] According to some implementations of this disclosure, configuration information describing the grouping method of a message queue can be obtained, and a group of messages can be retrieved from the message queue based on the configuration information. The configuration information can be stored in various ways; for example, it can be stored independently of the message queue. Specifically, the number of messages included in each group can be specified in a separate file. Alternatively and / or additionally, configuration information can be added to the message queue. Specifically, a group identifier can be set for each message in the message queue. For example, the first to M messages in the message queue can have group identifier 1, the (M+1)th to 2Mth messages in the message queue can have group identifier 2, and so on. Using some implementations of this disclosure, a desired number of messages can be retrieved from the message queue in a predetermined manner, thereby determining the state of the data processing system.
[0044] According to some implementations of this disclosure, the number of messages in a group can be set based on the performance of the data processing system. The higher the performance of the data processing system, the larger the number of messages. Alternatively and / or additionally, the number of messages in a group can be set based on the number of data items in the data processing system. The larger the number of data items, the larger the number of messages. In this way, the corresponding status can be detected in a manner more suited to the performance of the data processing system.
[0045] According to some implementations of this disclosure, a group of messages can have a corresponding delay time. As shown in Figure 3, a group of messages may include messages 312, ..., and 314, and the delay time can be 1 minute. According to some implementations of this disclosure, the message queue may include multiple groups of messages, and each group of messages corresponds to multiple different delay times. Continuing to refer to Figure 3, a group of messages 320 may include messages 322, ..., and 324 generated based on other data items, and the delay time can be 2 minutes, and so on. Using some implementations of this disclosure, the status of data items corresponding to different message groups can be queried at different time points, thereby avoiding resource shortages caused by executing a large number of queries simultaneously.
[0046] According to some implementations of this disclosure, the delay time for each message group can be specified in the configuration information. For example, the configuration information can specify that the delay time for the first group of messages is ΔT1, the delay time for the second group of messages is ΔT2, and so on. Alternatively and / or additionally, the configuration information can specify that the interval between the delay times of two consecutive groups is ΔT. For example, ΔT = 1 minute can be specified, in which case the delay times for each group are 1 minute, 2 minutes, 3 minutes, and so on. Alternatively and / or additionally, information about the delay time can be added to the message queue.
[0047] According to some implementations of this disclosure, the data processing system can operate asynchronously. Specifically, updates to the message queue can be independent of the processing of individual data items, as described in more detail in Figure 4. Figure 4 shows a block diagram 400 of a data processing system managed by producers and consumers based on message queues according to some implementations of this disclosure. According to some implementations of this disclosure, producers 410 and consumers 420 can be implemented using applications and can be deployed to any computing-capable device within the data processing system.
[0048] As shown in Figure 4, a producer 410 can add messages to the tail of the message queue, and a consumer 420 can retrieve messages from the head of the message queue. In response to determining that at least one queryer has been initiated, a set of messages is removed from the message queue. As the data processing system operates, the messages in message queue 210 can continuously change, thereby allowing for real-time determination of the processing status of each latest data item, thus enabling real-time management of the data processing system's operation.
[0049] According to some implementations of this disclosure, the producer 410 can be implemented by an API service in the data processing system, and the consumer 420 can be implemented by a script. See Figure 5 for further details, which shows a block diagram 500 of the processing status of a queried data item according to some implementations of this disclosure. As shown in Figure 5, the API service 510 can receive data items 511 from the application. At block 512, a globally unique identifier can be generated for the data item, and a message can be generated. At block 513, preprocessing can be performed, for example, determining whether a user has the right to post a comment, etc. At block 514, the data item can be loaded into the database. At block 515, the data item can be verified to meet predetermined conditions. At block 516, parameters required for posting a comment can be set. At block 519, the data item can be posted; and at block 518, the corresponding response data can be processed. According to some implementations of this disclosure, the above processes can be performed by different processing nodes in the data processing system.
[0050] Alternatively and / or additionally, generated messages can be sent asynchronously to message queue 210 at box 517. The data processing system can continuously receive data items from client devices of various users and continuously add messages to be processed to the tail of message queue 210.
[0051] Script 520 can act as a consumer to consume messages in message queue 210, and event handling can be performed at box 526. Specifically, the event handling process begins at box 521. At box 522, messages can be parsed according to configuration information; for example, a group of messages can be extracted from message queue 210 according to a predetermined grouping method, and the delay time corresponding to the group of messages can be obtained. At box 523, a queryer can be set based on the obtained delay time. At box 524, a query can be executed at a predetermined time. At box 525, the query results can be obtained, such as the processing status of data items in various subsystems. Subsequently, the obtained results can be uploaded at box 526.
[0052] According to some implementations of this disclosure, the technical solutions described above can be executed in a media sharing system. Specifically, users can post comments, and the API service can generate a unique comment ID and perform preprocessing, loading, and inspection processes on the comment before posting it. In parallel with the above process, the API service can generate a corresponding message based on the unique comment ID and send the message to a message queue. The message may include the comment ID and client parameters for posting the comment (such as user parameters, device parameters, etc.).
[0053] According to some implementations of this disclosure, during the process of sending messages to the message queue, configuration information can be obtained, and multiple messages can be divided into one or more groups based on the configuration information, and the delay time for messages in each group can be set. Alternatively and / or additionally, groups can be formed and the delay time for messages in each group can be set during the process of reading messages from the message queue. Specifically, the first group of messages can be delayed by 1 minute, the second group of messages by 2 minutes, and the third group of messages by 3 minutes, and so on. After obtaining the messages, the script can query various processing nodes, such as checking whether the comment exists in the comment database, whether the comment has been verified, whether the comment has been sent successfully, and so on.
[0054] According to some implementations of this disclosure, query results can be presented through a visualization panel. Since the published comments are divided into multiple groups, each with a different delay time, the status of each comment at different time points (e.g., 1 minute, 2 minutes, 3 minutes, etc.) can be seen: for example, how many comments have been stored in the database, how many comments have been verified, how many comments have been successfully published, etc. See Figure 6 for more information on presenting query results; Figure 6 shows a block diagram 600 of the state of the data processing system according to some implementations of this disclosure.
[0055] As shown in Figure 6, in managing a data processing system based on processing status, the distribution of multiple data items in the data processing system at at least one processing node can be determined based on the processing status. Different diagrams represent different processing nodes. Assuming the data processing system includes four processing nodes, distribution diagram 610 can represent the proportion of data items at the four processing nodes. For example, approximately 50% of the data items are located at processing node 610, approximately 25% at processing node 620, approximately 12.5% at processing node 630, and approximately 12.5% at processing node 640. Using some implementations of this disclosure, the distribution of data items at each processing node can be clearly understood, thereby facilitating the identification of processing nodes that may cause processing bottlenecks.
[0056] According to some implementations of this disclosure, in managing a data processing system based on processing status, the processing status of multiple data items in the data processing system at different points in time can be determined based on the processing status. See Figure 7 for further details, which shows a block diagram 700 of the status of a data processing system according to some implementations of this disclosure. As shown in Figure 7, the horizontal axis represents time, and the vertical axis represents the success rate in the processing nodes, for example, the proportion of items successfully written to the database. Curves 710 and 720 can, for example, represent the processing status of different types of data items; curve 710 can correspond to comments, and curve 720 can correspond to media items. Using some implementations of this disclosure, multiple queryers can be used to query the processing status (e.g., processed / unprocessed) of data items in multiple processing nodes respectively. In this way, the performance of each processing node can be monitored to determine whether each processing node meets the expected performance.
[0057] According to some implementations of this disclosure, in the process of managing a data processing system based on processing status, the node status of at least one processing node can be determined based on the processing status. Specifically, if the proportion of successfully processed data items at a certain processing node is significantly lower than the proportion of successfully processed data items at other processing nodes, then that processing node can be considered a bottleneck affecting the overall performance of the data processing system. Alternatively and / or additionally, it can be considered that the processing node may be malfunctioning, and the functionality of the processing node can be further tested. Using some implementations of this disclosure, it is not necessary to run a dedicated diagnostic program; instead, anomalies in processing nodes can be detected through the processing window of data items.
[0058] Using some implementation methods disclosed herein, the delay time can be specified as a point in time during the system's operation, and the processing status of data items can be queried at the specified point in time. In this way, the operating status of the data processing system can be monitored in real time.
[0059] Example process
[0060] Figure 8 illustrates a flowchart of a method 800 for managing a data processing system according to some implementations of this disclosure. At block 810, a set of messages is retrieved from a message queue of the data processing system, each message corresponding to a set of data items in the data processing system. At block 820, a delay time corresponding to the set of messages is determined, the delay time indicating the query time for querying the processing status of a set of data items in at least one processing node within the data processing system. At block 830, based on the delay time, at least one queryer in the data processing system is initiated, the set of data items being processed by at least one processing node, and the at least one queryer is used to query the processing status of a set of data items in at least one processing node. At block 840, the data processing system is managed based on the processing status.
[0061] According to some implementations of this disclosure, the method 800 further includes: in response to determining that at least one queryer has been initiated, removing a set of messages from the message queue.
[0062] According to some implementations of this disclosure, obtaining a set of messages includes: obtaining configuration information describing the grouping method of the message queue; and obtaining a set of messages from the message queue based on the configuration information.
[0063] According to some implementations of this disclosure, the message queue includes multiple sets of messages, and each set of messages corresponds to a different delay time.
[0064] According to some implementations of this disclosure, the message queue is determined based on: receiving data items from a client device connected to the data processing system; generating a message based on the data items, the message including an identifier of the data items and attribute information associated with the data items; and adding the message to the message queue.
[0065] According to some implementations of this disclosure, managing a data processing system based on processing status includes: determining the node status of at least one processing node based on the processing status.
[0066] According to some implementations of this disclosure, managing a data processing system based on processing status includes: determining the distribution of multiple data items in the data processing system at at least one processing node based on the processing status.
[0067] According to some implementations of this disclosure, managing a data processing system based on processing status includes: determining the processing status of multiple data items in the data processing system at different points in time based on the processing status.
[0068] According to some implementations of this disclosure, the data processing system is a media sharing system, and the data items include: media items published by objects in the media sharing system, and / or comments on media items published by objects in the media sharing system.
[0069] Example devices and equipment
[0070] Figure 9 shows a block diagram of an apparatus 900 for managing a data processing system according to some implementations of the present disclosure. The apparatus 900 includes: an acquisition module 910 configured to acquire a set of messages from a message queue of the data processing system, each set of messages corresponding to a set of data items in the data processing system; a determination module 920 configured to determine a delay time corresponding to the set of messages, the delay time indicating the query time for querying the processing status of a set of data items in at least one processing node in the data processing system; a startup module 930 configured to start at least one queryer in the data processing system based on the delay time, wherein the set of data items is processed by at least one processing node, and the at least one queryer is used to query the processing status of the set of data items in at least one processing node; and a management module 940 configured to manage the data processing system based on the processing status.
[0071] According to some implementations of this disclosure, the apparatus 900 further includes a processing module configured to: remove a set of messages from a message queue in response to determining that at least one queryer has been activated.
[0072] According to some implementations of this disclosure, the acquisition module is further configured to: acquire configuration information describing the grouping method of the message queue; and acquire a set of messages from the message queue based on the configuration information.
[0073] According to some implementations of this disclosure, the message queue includes multiple sets of messages, and each set of messages corresponds to a different delay time.
[0074] According to some implementations of this disclosure, the message queue is determined based on: receiving data items from a client device connected to the data processing system; generating a message based on the data items, the message including an identifier of the data items and attribute information associated with the data items; and adding the message to the message queue.
[0075] According to some implementations of this disclosure, the management module is further configured to: determine the node status of at least one processing node based on the processing status.
[0076] According to some implementations of this disclosure, the management module is further configured to: determine the distribution of multiple data items in the data processing system at at least one processing node based on the processing status.
[0077] According to some implementations of this disclosure, the management module is further configured to: determine the processing status of multiple data items in the data processing system at different points in time based on the processing status.
[0078] According to some implementations of this disclosure, the data processing system is a media sharing system, and the data items include: media items published by objects in the media sharing system, and / or comments on media items published by objects in the media sharing system.
[0079] Figure 10 shows a block diagram of a device 1000 capable of implementing various implementations of the present disclosure. It should be understood that the computing device 1000 shown in Figure 10 is merely exemplary and should not constitute any limitation on the functionality and scope of the implementations described herein. The computing device 1000 shown in Figure 10 can be used to implement the methods described above.
[0080] As shown in Figure 10, the computing device 1000 is in the form of a general-purpose computing device. Components of the computing device 1000 may include, but are not limited to, one or more processors 1010, memory 1020, storage devices 1030, one or more communication units 1040, one or more input devices 1050, and one or more output devices 1060. The processor 1010 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1020. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 1000.
[0081] Computing device 1000 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1000, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1020 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 1030 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 1000.
[0082] The computing device 1000 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 10, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. The memory 1020 may include a computer program product 1025 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.
[0083] The communication unit 1040 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1000 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1000 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.
[0084] Input device 1050 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1060 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1000 can also communicate with one or more external devices (not shown) via communication unit 1040 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1000, or with any device (e.g., network card, modem, etc.) that enables computing device 1000 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interface (not shown).
[0085] According to an implementation of this disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the method described above. According to an implementation of this disclosure, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the method described above.
[0086] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0087] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0088] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0090] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.
Claims
1. A method for managing a data processing system, comprising: A set of messages is obtained from the message queue of the data processing system, and the set of messages corresponds to a set of data items in the data processing system. Determine the delay time corresponding to the set of messages, the delay time indicating the query time for querying the processing status of the set of data items in at least one processing node in the data processing system; Based on the delay time, at least one queryer in the data processing system is activated, the set of data items is processed by the at least one processing node, and the at least one queryer is used to query the processing status of the set of data items in the at least one processing node; as well as The data processing system is managed based on the processing status.
2. The method according to claim 1, further comprising: In response to determining that at least one queryer has been activated, the set of messages is removed from the message queue.
3. The method according to claim 1, wherein obtaining the set of messages includes: Obtain configuration information describing the grouping method of the message queue; as well as The set of messages is retrieved from the message queue based on the configuration information.
4. The method according to claim 1, wherein the message queue comprises multiple sets of messages, and the multiple sets of messages correspond to multiple different delay times.
5. The method of claim 1, wherein the message queue is determined based on the following: Receive data items from client devices connected to the data processing system; A message is generated based on the data item, the message including the identifier of the data item and attribute information associated with the data item; and Add the message to the message queue.
6. The method of claim 1, wherein managing the data processing system based on the processing state comprises: Based on the processing status, the node status of the at least one processing node is determined.
7. The method of claim 1, wherein managing the data processing system based on the processing state comprises: Based on the processing status, the distribution of multiple data items in the data processing system at at least one processing node is determined.
8. The method of claim 1, wherein managing the data processing system based on the processing state comprises: Based on the processing status, the processing status of multiple data items in the data processing system at different points in time is determined.
9. The method of claim 1, wherein the data processing system is a media sharing system, and the data item comprises: Media items published by objects in the media sharing system, and / or comments on media items published by objects in the media sharing system.
10. An apparatus for managing a data processing system, comprising: The acquisition module is configured to acquire a set of messages from the message queue of the data processing system, the set of messages corresponding to a set of data items in the data processing system. A determination module is configured to determine a delay time corresponding to the set of messages, the delay time indicating the query time for querying the processing status of the set of data items in at least one processing node in the data processing system; A startup module is configured to start at least one queryer in the data processing system based on the delay time, wherein the set of data items is processed by the at least one processing node, and the at least one queryer is used to query the processing status of the set of data items in the at least one processing node; as well as The management module is configured to manage the data processing system based on the processing status.
11. An electronic device, comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.
12. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9.
13. A computer instruction product comprising computer instructions, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.