Data processing method and apparatus, and computer device and storage medium
By establishing time stamps and relationships in the disk data table, the problem of data loss caused by write-lock preemption was solved, achieving efficient data processing and management, ensuring data consistency and integrity, and improving system stability and performance.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-20
- Publication Date
- 2026-06-11
Smart Images
Figure CN2025136241_11062026_PF_FP_ABST
Abstract
Description
Data processing methods, apparatus, computer equipment and storage media
[0001] Related applications
[0002] This application claims priority to Chinese patent application filed on December 5, 2024, application number 2024117819362, entitled "Data Processing Method, Apparatus, Computer Equipment and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, storage medium, and computer-readable instruction product. Background Technology
[0004] In the field of network technology, business scenarios such as resource refresh and configuration distribution often require task distribution, especially when facing performance demands on the order of hundreds of millions per day. Related technologies typically copy disk data into memory to leverage memory's efficient read / write capabilities to handle task requests from upstream and requests from downstream.
[0005] In this process, the system first modifies the data in memory through upstream task requests, and then transmits the modified data to the corresponding IP (Internet Protocol) address through downstream requests. However, when writing data from the memory data table to the disk data table, write lock contention often occurs. This phenomenon may lead to data loss, thus failing to meet the needs of actual business operations. Summary of the Invention
[0006] According to various embodiments disclosed in this application, a data processing method, apparatus, computer device, computer storage medium, and computer-readable instruction product are provided.
[0007] Firstly, this application provides a data processing method. The method includes:
[0008] In response to a data addition request to a disk data table, the request time of the data addition request is obtained, and the time identifier corresponding to the data addition request is determined based on the request time;
[0009] If the time identifier does not exist in the disk data table, the disk data table is locked, and the time identifier is created in the disk data table; and
[0010] Establish a relationship between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier.
[0011] In one embodiment, the method further includes:
[0012] In response to an interaction request for target data, retrieve the timestamp of each data item in the disk data table;
[0013] Using the time identifier in the disk data table, a preset number of data items are selected from the disk data table and stored in the memory data table; wherein, the preset number is the maximum data storage capacity of the memory data table;
[0014] The interaction request is used to perform corresponding interactive operations on the memory data table.
[0015] In one embodiment, the disk data includes one or more data files; the step of selecting a preset number of data items from the disk and storing them in a memory data table using the timestamps in the disk data table includes:
[0016] The number of target data files is determined by using the maximum data storage capacity of the memory data table and the data storage capacity of the data file.
[0017] The target data files are determined using the number of target data files and the timestamps of each data file; and
[0018] Using the timestamp of the target data file and the timestamp of each data item in the data file, target data is selected and stored sequentially in the memory data table.
[0019] In one embodiment, the method further includes:
[0020] In response to a request to acquire target data, determine whether the acquisition request includes a time identifier;
[0021] If the acquisition request includes a time identifier, the data corresponding to the time identifier is retrieved from the memory data table, and the data corresponding to the next time identifier is retrieved sequentially; and
[0022] If the time identifier is not included in the acquisition request, the data corresponding to the first time identifier is obtained from the memory data table, and the data corresponding to the next time identifier is obtained in sequence.
[0023] In one embodiment, the memory data table further includes an association between accounts and data identifiers; the step of determining whether the acquisition request includes a timestamp in response to a request to acquire target data includes:
[0024] In response to an account's request to retrieve target data, a data identifier is obtained from the interaction request, and an in-memory data table is used to determine whether there is an association between the account and the data identifier; and
[0025] If the account and the data identifier are associated, determine whether the acquisition request includes a time identifier.
[0026] In one embodiment, the method further includes:
[0027] In response to an interaction request for target data, the target data in the disk data table is added to the memory data table; and
[0028] After a preset time has elapsed, the interactive operation corresponding to the interactive request is performed using the memory data table.
[0029] Secondly, this application also provides a data processing apparatus. The apparatus includes:
[0030] The data acquisition module is used to respond to a data addition request to a disk data table, acquire the request time of the data addition request, and determine the time identifier corresponding to the data addition request based on the request time;
[0031] An identifier creation module is configured to, when the time identifier does not exist in the disk data table, lock the disk data table and create the time identifier in the disk data table; and
[0032] The data interaction module is used to establish the association between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and to add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier.
[0033] In one embodiment, the device further includes:
[0034] The identifier acquisition module is used to obtain the time identifier of each data in the disk data table in response to the interaction request for target data;
[0035] The data storage module is used to select a preset number of data items from the disk data table and store them in a memory data table using the timestamps in the disk data table; wherein the preset number is the maximum data storage capacity of the memory data table; and
[0036] The data interaction module is also used to perform corresponding interactive operations on the memory data table using the interaction request.
[0037] In one embodiment, the disk data includes one or more data files; the data storage module includes:
[0038] The quantity determination submodule is used to determine the number of target data files by utilizing the maximum data storage capacity of the memory data table and the data storage capacity of the data files;
[0039] The file determination submodule is used to determine the target data file using the number of target data files and the timestamp of each data file; and
[0040] The data storage submodule is used to select target data from the target data file and store it in the memory data table in sequence using the time identifier of the target data file and the time identifier of each data in the data file.
[0041] In one embodiment, the apparatus further includes a request determination submodule, a data acquisition submodule, and a data acquisition submodule;
[0042] The request judgment submodule is used to respond to a request to obtain target data and determine whether the request includes a time identifier.
[0043] The data acquisition submodule is configured to, when the acquisition request includes a time identifier, retrieve the data corresponding to the time identifier from the memory data table, and sequentially retrieve the data corresponding to the next time identifier; and
[0044] The data acquisition submodule is used to acquire the data corresponding to the first time identifier from the memory data table when the acquisition request does not include a time identifier, and then acquire the data corresponding to the next time identifier in sequence.
[0045] In one embodiment, the memory data table further includes the association between accounts and data identifiers; the request judgment submodule includes an association determination unit and a request judgment unit;
[0046] The association determination unit is configured to, in response to an account's request to obtain target data, obtain a data identifier from the interaction request and, using a memory data table, determine whether an association exists between the account and the data identifier; and
[0047] The request determination unit is used to determine whether the acquisition request includes a time identifier when there is an association between the account and the data identifier.
[0048] In one embodiment, the device further includes a data adding module and a data interaction module;
[0049] The data adding module is used to add target data from the disk data table to the memory data table in response to an interaction request for target data; and
[0050] The data interaction module is used to perform the interaction operation corresponding to the interaction request using the memory data table after a preset time has elapsed.
[0051] Thirdly, this application also provides a computer device. The computer device includes a memory and one or more processors, the memory storing computer-readable instructions, and the one or more processors executing the computer-readable instructions to implement the data processing method as described in any of the embodiments of this disclosure.
[0052] Fourthly, this application also provides a computer storage medium. The computer storage medium stores computer-readable instructions thereon, which, when executed by one or more processors, implement the data processing method as described in any one of the embodiments of this disclosure.
[0053] Fifthly, this application also provides a computer-readable instruction product. The computer-readable instruction product includes computer-readable instructions that, when executed by one or more processors, implement the data processing method as described in any of the embodiments of this disclosure.
[0054] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description
[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the disclosed drawings without creative effort.
[0056] Figure 1 is an application environment diagram of the data processing method according to one or more embodiments;
[0057] Figure 2 is a flowchart illustrating the data processing method according to one or more embodiments;
[0058] Figure 3 is a schematic diagram of the data interaction process according to one or more embodiments;
[0059] Figure 4 is a schematic diagram of the data storage to memory process according to one or more embodiments;
[0060] Figure 5 is a schematic diagram of the data acquisition process according to one or more embodiments;
[0061] Figure 6 is a flowchart illustrating the determination of association relationships based on one or more embodiments;
[0062] Figure 7 is a schematic diagram of the data storage and interaction process according to one or more embodiments;
[0063] Figure 8 is a schematic diagram of a data table according to one or more embodiments;
[0064] Figure 9 is a flowchart illustrating the initial unit according to one or more embodiments;
[0065] Figure 10 is a schematic diagram of the data landing process according to one or more embodiments;
[0066] Figure 11 is a schematic diagram of the data removal process according to one or more embodiments;
[0067] Figure 12 is a schematic diagram of the downstream task acquisition process according to one or more embodiments;
[0068] Figure 13 is a flowchart illustrating the response to an upstream request according to one or more embodiments;
[0069] Figure 14 is a flowchart illustrating the response to a downstream request according to one or more embodiments;
[0070] Figure 15 is a flowchart of the handling unit according to one or more embodiments;
[0071] Figure 16 is a structural block diagram of a data processing apparatus according to one or more embodiments;
[0072] Figure 17 is an internal structural diagram of a computer device according to one or more embodiments. Detailed Implementation
[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0074] The data processing method provided in this application embodiment can be applied to the application environment shown in Figure 1. This application environment includes a server and a terminal. The server communicates with the terminal via a network. The data storage system can store the data that the server needs to process. The data storage system can be integrated on the server or placed on the cloud or other network servers. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle devices, projection devices, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.
[0075] In one embodiment, as shown in Figure 2, a data processing method is provided. This embodiment illustrates the application of this method to the terminal in Figure 1. It is understood that this method can also be applied to the server in Figure 1, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0076] Step S100: In response to a data addition request to the disk data table, obtain the request time of the data addition request, and determine the time identifier corresponding to the data addition request based on the request time.
[0077] In one embodiment, the disk data table may include time stamps and data, etc.; for example, it may include multiple data groups corresponding to each time stamp, and each data group includes multiple data items; in actual use, the data group corresponding to each time stamp may be obtained from the same task, for example, the data group includes data group A and data group B; data group A includes data from task a and data from task b; then, different time stamps may all include the stamps of data group A and data group B, that is, they may all include data from task a and data from task b; however, the data of the corresponding task will also be different depending on the time stamp used.
[0078] In one embodiment, adding data to a disk data table may include adding data from a memory data table to the disk data table; wherein, the memory data table can be used for IP-data interaction; the request time may include the time when data is requested to be added to the memory data table; it is understood that the data in the memory data table can eventually be stored in the disk data table, therefore adding data to the memory data table includes adding data to the disk data table.
[0079] In one embodiment, determining a time identifier using the request time may include setting a preset time period, quoting the request time and the time period to obtain the corresponding time identifier. It is understood that quoting the request time and the time period yields the number of periods corresponding to the request; the number of periods can represent a time identifier, etc. Data within one period is stored in a storage area corresponding to a time identifier, etc.
[0080] Step S200: If the time identifier does not exist in the disk data table, lock the disk data table and create a time identifier in the disk data table.
[0081] In one embodiment, the disk data table does not contain a timestamp, meaning the disk data table does not store data with that timestamp. In this case, to avoid read / write conflicts and write preemption, the disk data is locked.
[0082] In one embodiment, the time identifier does not exist in the disk data table, so it is necessary to create the time identifier in the disk data table, etc.
[0083] Step S300: Establish the association between the time stamp corresponding to the data addition request and the time stamp in the disk data table, and add the data corresponding to the data addition request to the storage area corresponding to the newly created time stamp.
[0084] In one embodiment, there are relationships between time stamps in the disk data table; for example, the relationships between time stamps in the disk data table can include sorting the time stamps by request time, with each time stamp pointing to the next time stamp, i.e., the previous request time points to the next request time. It is understood that no time stamp points to the first time stamp (i.e., the earliest request time), and the last time stamp does not point to any time stamp (i.e., the latest request time), etc. In another embodiment, establishing a relationship between the time stamp corresponding to the data addition request and the time stamps in the disk data table can include using the last time stamp in the disk data table (i.e., the time stamp corresponding to the latest request time) to point to the newly created time stamp, etc.
[0085] In one embodiment, after establishing the association between the newly created time stamps, the data addition request can be added to the storage area of the disk data table corresponding to the newly created time stamp, etc.
[0086] In the aforementioned data processing method, the request time of a data addition request is used to determine the corresponding timestamp. If this timestamp does not exist in the disk data table, it is created in the disk data table, and its association with other timestamps in the disk data table is established. The data corresponding to the data addition request is then added to the storage area corresponding to the timestamp. By obtaining the request time and generating a timestamp when responding to a data addition request, data updates in the disk data table can be effectively managed. In the absence of a timestamp, locking the disk data table and creating a new timestamp ensures data consistency and integrity. Simultaneously, this mechanism reduces lock contention during write operations, allowing the data transfer unit to quickly deliver new data to the management unit, ensuring timely storage of valid data and preventing data loss. This method ensures stable system operation in high-concurrency environments, improving data processing efficiency and guaranteeing the integrity of valid data.
[0087] In one embodiment, as shown in FIG3, the method further includes:
[0088] Step S400: In response to the interaction request for the target data, obtain the timestamp of each data in the disk data table.
[0089] In one embodiment, the interaction request may include listening to the target data to see if there is a possibility of interaction with the target data. In actual use, the disk data table may be electrically connected to a program to interact with the disk data table. In this case, the interaction request may include opening the program, i.e., listening to whether the program is opened, and if the program is opened, proceeding with subsequent steps, etc.
[0090] Step S500: Using the time identifier in the disk data table, select a preset number of data items from the disk data table and store them in the memory data table; wherein, the preset number is the maximum data storage capacity of the memory data table.
[0091] In one embodiment, storing data in a memory data table using time identifiers may include selecting the data with the latest time identifier (i.e., the data corresponding to the time identifier closest to the current time) from a disk data table using time identifiers, and storing the selected data in the memory data table.
[0092] Step S600: Use the interaction request to perform corresponding interactive operations on the memory data table.
[0093] In one embodiment, after storing the data in an in-memory data table, the corresponding interactive operations can be performed using the in-memory data table.
[0094] In this embodiment, data from the disk data table is stored in a memory data table, and the memory data table is used for corresponding interactive operations. By storing data from the disk data table in the memory data table to complete data interaction, the data access speed of memory is much faster than that of disk, thus significantly improving the efficiency of data processing and interactive operations. At the same time, using the memory data table greatly reduces the interaction burden with the disk data table, improving the efficiency of the memory data table in retrieving data from the disk data table. Furthermore, by setting a preset number and time markers, the use of the memory data table can be effectively managed, not only avoiding memory overflow problems but also ensuring the timeliness and relevance of the memory in the memory data table, thus improving the accuracy of data processing.
[0095] In one embodiment, as shown in Figure 4, the disk data includes one or more data files; using the timestamps in the disk data table, a preset number of data items are selected from the disk and stored in a memory data table, including:
[0096] Step S501: Determine the number of target data files by using the maximum data storage capacity of the memory data table and the data storage capacity of the data files.
[0097] In one embodiment, the maximum data storage capacity of the memory data table may include the memory size of the memory data table; the data storage capacity of the data files, i.e., the number of data items stored in each data file; it is understood that the memory size of the memory data table is usually larger than the number of data items in the data files, so it is usually necessary to store the data of multiple data files in memory. For example, the number of target data files can be obtained by quotienting the maximum memory data storage capacity with the data storage capacity of the data files.
[0098] Step S502: Determine the target data file using the number of target data files and the time identifier of each data file.
[0099] In one embodiment, the data file may include a timestamp, etc.; the timestamp of the data file can be determined by the timestamps of the data stored in the data file. For example, the timestamp of the data file can be the earliest timestamp or the latest timestamp of the data stored in the data file, etc.
[0100] In one embodiment, the target data file with the earliest time identifier can be selected from the data files.
[0101] Step S503: Using the time identifier of the target data file and the time identifier of each data in the data file, select the target data from the target data file in sequence and store it in the memory data table.
[0102] In one embodiment, the data file with the latest timestamp can be selected from the target data files, and all data from the other target data files can be stored in the memory data table. Then, the data with the latest timestamp can be selected from the target data file with the latest timestamp and added to the memory data table in turn, until the memory of the memory data table is full. It can be understood that the number of target data files indicates that the memory data table cannot store enough data if one target data file is missing. Therefore, one target data file can be selected, and all other target data files can be stored in the memory data table. Target data can be selected from the selected target data file and stored in the memory data table, etc.
[0103] In this embodiment, by utilizing the maximum storage capacity of the in-memory data table and the storage capacity of the data files, the required number of target data files can be effectively determined. This process not only optimizes data storage management but also ensures efficient memory utilization. Next, by using the timestamps of the target data files, the required data files can be accurately selected, thereby improving data retrieval efficiency. Finally, storing the target data in the in-memory data table facilitates rapid data access and processing, providing a solid foundation for subsequent data analysis and decision-making. This method enhances the system's flexibility and responsiveness, contributing to improved overall performance.
[0104] In one embodiment, as shown in FIG5, the method further includes:
[0105] Step S401: In response to the request to obtain the target data, determine whether the request includes a time identifier.
[0106] In one embodiment, the target data acquisition request may include a data identifier and a time identifier, etc.; for example, the data corresponding to the data identifier may include identifiers at different time points, that is, target data corresponding to different time identifiers. Therefore, it is necessary to determine whether the acquisition request includes the time identifier corresponding to the target data identifier, etc.
[0107] Step S402: If the request includes a time identifier, retrieve the data corresponding to the time identifier from the memory data table, and then retrieve the data corresponding to the next time identifier in sequence.
[0108] In one embodiment, the request may include a time identifier, then the data corresponding to the time identifier under the corresponding data identifier is retrieved; and the data corresponding to subsequent time identifiers under the data identifier is retrieved sequentially, and so on. It is understood that one time identifier in the memory data table will point to the next time identifier, therefore data corresponding to different time identifiers can be retrieved sequentially.
[0109] Step S403: If the request does not include a time identifier, retrieve the data corresponding to the first time identifier from the memory data table, and then retrieve the data corresponding to the next time identifier in sequence.
[0110] In one embodiment, if the request does not include a time identifier, the data corresponding to the first time identifier in the memory data table is obtained; wherein, the first time identifier may include the earliest time identifier, i.e., the time identifier furthest from the current time, etc.; and according to the time identifier pointed to by the first time identifier, the data corresponding to the next time identifier is obtained in sequence, etc.
[0111] In this embodiment, by determining whether the retrieval request contains a timestamp, the system flexibly adapts to different data access needs. This avoids sequential retrieval from the first time point in the in-memory data table each time, improving data retrieval efficiency. When the request contains a timestamp, the system can directly retrieve data at that specific time point, ensuring data accuracy and relevance; while without a timestamp, the system can retrieve data sequentially from the first time point, guaranteeing data integrity and system stability. This design not only improves data retrieval efficiency but also optimizes the data processing flow.
[0112] In one embodiment, as shown in Figure 6, the memory data table also includes the association between account and data identifier; in response to a request to obtain target data, it is determined whether the request includes a timestamp, including:
[0113] Step S411: In response to the account's request to obtain target data, obtain the data identifier from the interaction request, and use the in-memory data table to determine whether there is an association between the account and the data identifier.
[0114] Step S412: If there is a correlation between the account and the data identifier, determine whether the retrieval request includes a time identifier.
[0115] In one embodiment, the association between an account and a data identifier can characterize which target data the account follows, i.e., what data the account can obtain.
[0116] In one embodiment, if there is an association between an account and a data identifier in the memory data table, it means that the account can obtain the data of the data identifier. If there is no association between an account and a data identifier in the memory data table, it means that the account does not care about the data of the data identifier and therefore cannot obtain the data.
[0117] In this embodiment, by confirming the association between accounts and data identifiers, the system ensures that only authorized users can access specific data, thus improving system security and data protection. Once the association is confirmed, the system then checks whether the retrieval request contains a timestamp. This process not only enhances the flexibility of data access but also provides precise data retrieval methods based on actual needs. This approach makes data management more efficient and reliable while reducing the risk of accidental operations.
[0118] In one embodiment, as shown in FIG7, the method further includes:
[0119] In step S700, in response to the interaction request for target data, the target data in the disk data table is added to the memory data table.
[0120] In step S800, after a preset time has elapsed, the interactive operation corresponding to the interactive request is performed using the memory data table.
[0121] In one embodiment, interaction with the memory data table is performed after a preset time, which avoids interacting with the memory data table before all data in the memory data table has been written, thus avoiding read-write conflicts.
[0122] In this embodiment, by adding the target data from the disk data table to the memory data table, the data in memory is ensured to be up-to-date and relevant, thereby improving the efficiency of interactive operations. Performing interactive operations only after a preset time effectively avoids accessing data before it is fully written to memory, reducing the risk of read-write conflicts. This not only improves system stability and data consistency but also optimizes resource utilization, enabling faster and more reliable responses during interactions. This mechanism enhances the overall performance of the system.
[0123] In one embodiment, the disk data table and the memory data table can be as shown in Figure 8. The data table can include a `tasks` format and an `iptaskGroups` format. The `tasks` format stores timestamps (T1, T2, ..., Tn); each timestamp corresponds to multiple task data groups (taskGroupA, taskGroupB, etc.) and a pointer to the next timestamp, i.e., `_nextT`. Each task data group stores multiple task data (task1, task2, etc.). The `iptaskGroups` format includes multiple accounts (ipA, ipB, etc.). Each IP address has an associated task group. ipA is associated with taskGroupA and taskGroupB, meaning ipA monitors the data in taskGroupA and taskGroupB. In another embodiment, the data on the disk can be stored as multiple data files, with each data file containing multiple timestamps. This reduces the number of data files (avoiding storing one timestamp per data file).
[0124] In one embodiment, the data processing method can be applied to four units, including an initialization unit, a management unit, a transceiver unit, and a transfer unit; wherein, the initialization unit can be implemented as shown in the schematic diagram in FIG9, including:
[0125] Step S901: Determine whether a data file exists on the disk; if a data file exists on the disk, proceed to step S902; if a data file does not exist on the disk, proceed to step S906.
[0126] Step S902: Read the latest data file from the disk data table and load the accumulated preset number of files into the memory data table. It can be understood that the data files can be named using timestamps, and the data is read line by line starting from the largest file until the preset number is reached. The preset number is the amount of data stored in the memory data table. For example, data3, data2, and data1 represent the data closest to the current time to the furthest time, and each contains ten data entries. Since the memory data table can store fifteen data entries, ten data entries need to be retrieved from data3, and five data entries need to be retrieved from data2 and stored in the memory data table, etc.
[0127] Step S903: Update the oldest time identifier and the newest time identifier in the memory data; where the oldest time identifier is the time identifier furthest from the current time, and the newest time identifier is the time identifier furthest from the current time, etc.
[0128] Step S904: Determine whether the iptaskGroups file exists in the disk data table. If the iptaskGroups file does not exist in the disk data table, proceed to step S906. If the iptaskGroups file exists in the disk data table, proceed to step S905.
[0129] Step S905: Load the memory in iptaskGroups into the memory data table as well;
[0130] Step S906, sleep preset time.
[0131] The initial unit completes the transfer of data from the disk data table to the memory data table using the above methods. The management unit can include data persistence and data extraction, where the data logic can be implemented as shown in the flowchart in Figure 10, including:
[0132] Step S911: A preset signal is received;
[0133] Step S912: Determine if there is an exit signal; if there is an exit signal, perform the grounding logic and continue to step S912 to determine if the received signal is an exit signal; if the received signal is not an exit signal, proceed to step S913.
[0134] Step S913: Determine whether there is a dropped signal in the signal. If there is a dropped signal in the signal, perform the dropped logic and continue to determine whether there is a dropped signal in the signal. If there is no dropped signal in the signal, proceed to step S914.
[0135] Step S914: Retrieve data from the temporary variable in the memory data table and perform the persistence logic; the data in the temporary variable is the data stored in the memory data table due to the interaction between the account and the memory data table. The persistence logic may include storing the data from the memory data table into a disk data table, or obtaining the time identifier and task data, calculating the data file identifier using the time identifier, and storing the data in the file corresponding to the identifier. Data removal can be implemented using the flowchart shown in Figure 11, including:
[0136] Step S921: Determine whether the preset time for execution is met; if the preset time for execution is met, proceed to step S922; if the preset time for execution is not met, return to continue waiting for the preset time to be met; wherein, the preset time may include sequentially removing data after the preset time has elapsed;
[0137] Step S922: Calculate the number of items in the memory data table;
[0138] Step S923: Determine whether the number of items in the memory data table is greater than the memory of the memory data table; if the number of items in the memory data table is greater than the memory of the memory data table, proceed to step S924; if the number of items in the memory data table is less than or equal to the memory of the memory data table, proceed to step S927.
[0139] Step S924: Determine the number of redundant data items in the memory data table, and select the data item whose time identifier is furthest from the current time;
[0140] Step S925: Release all selected data;
[0141] Step S926: Determine the time identifier in the memory data table that is furthest from the present as the furthest time identifier, etc.
[0142] Step S927: Calculate the number of data files;
[0143] Step S928: Determine whether the number of data files is greater than the maximum capacity of the disk data table. If the number of data files is greater than the maximum capacity of the disk data table, proceed to step S929. If the number of data files is less than or equal to the maximum capacity of the disk data table, then end the process.
[0144] Step S929: Delete the data file corresponding to the time identifier furthest from the current time from the disk data table until the number of data in the disk data table meets the maximum capacity, etc.
[0145] The transceiver unit may include downstream request acquisition, response to upstream request, and response to downstream request. The downstream request acquisition flowchart is shown in Figure 12, including:
[0146] Step S931: Does the memory data table include the account corresponding to the retrieval request and the task data group followed by the account? If the memory data table includes the account corresponding to the retrieval request and the task data group followed by the account, proceed to step S932; otherwise, proceed to step S923.
[0147] Step S932: Set the preset variable to 0 and end the acquisition request; whereby a preset variable of 0 indicates that the account does not exist in the memory data table.
[0148] Step S933 returns information such as the task groups followed by the account.
[0149] The response to an upstream request can be illustrated in the flowchart shown in Figure 13, including:
[0150] Step S941: Store the upstream data in the memory data table;
[0151] Step S942: Determine the time identifier of the data using the output storage time;
[0152] Step S943, wait to acquire the write lock;
[0153] Step S944: Determine whether the timestamp of the added data is stored in the memory data table; if the timestamp of the added data is stored in the memory data table, proceed to step S945; if the timestamp of the added data is not stored in the memory data table, proceed to step S948.
[0154] Step S945: Determine whether a task data group identifier exists in the memory data table. If a task data group identifier exists in the memory data table, proceed to step S946; if a task data group identifier does not exist in the memory data table, proceed to step S950.
[0155] Step S946: Add the data from the memory data table to the disk data table;
[0156] Step S947: Release the write lock;
[0157] Step S948: Send the added data, time stamp, and the time stamp closest to the current time to the transport unit;
[0158] Step S949: Replace the time identifier in the disk data table that is closest to the current time with the time identifier of the added data; and initialize the temporary memory in the memory data table (wherein, the temporary memory is used to store the added data, etc.), and proceed to step S945;
[0159] Step S950: Create the corresponding task data group identifier in the disk data table.
[0160] The response to downstream requests can be illustrated in the flowchart shown in Figure 14, including:
[0161] Step S951: Determine whether there is a timestamp in the retrieval request. If there is no timestamp in the retrieval request, determine the timestamp of the retrieval request as the timestamp furthest from the current time in the memory data table.
[0162] Step S952: Determine whether the time identifier in the retrieval request exists in the memory data table. If the time identifier in the retrieval request does not exist in the memory data table, determine the time identifier of the retrieval request as the time identifier in the memory data table that is furthest from the current time.
[0163] Step S953, wait to acquire the read lock;
[0164] Step S954: Determine whether the time stamp of the request points to the next time stamp; if the time stamp of the request points to the next time stamp, proceed to step S955; if the time stamp of the request does not point to the next time stamp, proceed to step S957.
[0165] Step S955: Obtain all data corresponding to the time identifier of the acquisition request from the memory data table, and determine the time identifier of the acquisition request as the next time identifier; and determine the data status as "not finished reading";
[0166] Step S956: Release the read lock; and return all task data, data status, and time stamp to the requester.
[0167] Step S957: Determine whether the data status is "read completely". If the data status is "read completely", proceed to step S958; if the data status is not "read completely", proceed to step S959.
[0168] Step S958: Obtain all data corresponding to the time identifier of the acquisition request from the memory data table, and determine the data status as "read completed"; then proceed to step S956;
[0169] In step S959, the data status "read completed" is obtained, and step S959 is performed.
[0170] The transport unit can be illustrated in the flowchart shown in Figure 15, including:
[0171] Step S961: Obtain the time identifier closest to the current time, the data to be added, and the corresponding time identifier from the disk data table;
[0172] Step S962, wait to acquire the write lock;
[0173] Step S963: Add the data to be added and the corresponding time identifier to the disk data table;
[0174] Step S964: Set the time identifier in the disk data table that is closest to the current time to the newly added time identifier;
[0175] Step S965: Release the read lock, execute the landing logic, and return to step S962.
[0176] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0177] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.
[0178] In one embodiment, as shown in FIG16, a data processing device 100 is provided, including: a data acquisition module 101, an identifier establishment module 102, and a data interaction module 103, wherein:
[0179] The data acquisition module is used to respond to data addition requests to the disk data table, obtain the request time of the data addition request, and determine the time identifier corresponding to the data addition request based on the request time.
[0180] The identifier creation module is used to lock the disk data table and create a time identifier in the disk data table when the time identifier does not exist.
[0181] The data interaction module is used to establish the association between the timestamp corresponding to the data addition request and the timestamp in the disk data table, and to add the data corresponding to the data addition request to the storage area corresponding to the newly created timestamp.
[0182] In one embodiment, the device further includes:
[0183] The identifier acquisition module is used to obtain the time identifier of each data in the disk data table in response to the interaction request for target data;
[0184] The data storage module is used to select a preset number of data items from the disk data table and store them in the memory data table using the timestamps in the disk data table; where the preset number is the maximum data storage capacity of the memory data table.
[0185] The data interaction module is also used to perform corresponding interactive operations on the in-memory data table using interaction requests.
[0186] In one embodiment, the disk data includes one or more data files; the data storage module includes:
[0187] The quantity determination submodule is used to determine the number of target data files by utilizing the maximum data storage capacity of the memory data table and the data storage capacity of the data files;
[0188] The file determination submodule is used to determine the target data file by using the number of target data files and the timestamp of each data file;
[0189] The data storage submodule is used to select target data from the target data file and store it into an in-memory data table by using the timestamp of the target data file and the timestamp of each data item in the data file.
[0190] In one embodiment, the device further includes:
[0191] The request judgment submodule is used to respond to a request to obtain target data and determine whether the request includes a timestamp.
[0192] The data acquisition submodule is used to retrieve the data corresponding to the time identifier from the memory data table when the acquisition request includes a time identifier, and then retrieve the data corresponding to the next time identifier in sequence;
[0193] The data acquisition submodule is also used to retrieve the data corresponding to the first time identifier from the memory data table when the acquisition request does not include a time identifier, and then retrieve the data corresponding to the next time identifier in sequence.
[0194] In one embodiment, the memory data table further includes the association between accounts and data identifiers; the request determination submodule includes:
[0195] The association determination unit is used to respond to an account's request to obtain target data, obtain a data identifier from the interaction request, and use an in-memory data table to determine whether there is an association relationship between the account and the data identifier;
[0196] The request judgment unit is used to determine whether the retrieval request includes a timestamp when there is a relationship between the account and the data identifier.
[0197] In one embodiment, the device further includes:
[0198] The data addition module is used to add target data from the disk data table to the memory data table in response to an interactive request for target data.
[0199] The data interaction module is used to perform interactive operations corresponding to the interaction requests using an in-memory data table after a preset time has elapsed.
[0200] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of one or more processors in a computer device, or stored in software in the memory of the computer device, so that one or more processors can invoke and execute the operations corresponding to each module.
[0201] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 17. The computer device includes one or more processors, memory, input / output interfaces (I / O), and a communication interface. The one or more processors, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The one or more processors of the computer device provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer-readable instructions, and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the non-volatile storage media. The database of the computer device stores task data. The input / output interfaces of the computer device are used for exchanging information between the one or more processors and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection. When the computer-readable instructions are executed by the one or more processors, a data processing method is implemented.
[0202] Those skilled in the art will understand that the structure shown in Figure 17 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.
[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a non-volatile computer-readable storage medium. When executed, these computer-readable instructions can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. One or more processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0206] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A data processing method, wherein, The method includes: In response to a data addition request to a disk data table, the request time of the data addition request is obtained, and the time identifier corresponding to the data addition request is determined based on the request time; If the time identifier does not exist in the disk data table, the disk data table is locked, and the time identifier is created in the disk data table; and Establish a relationship between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier.
2. The method according to claim 1, wherein, The method further includes: In response to an interaction request for target data, retrieve the timestamp of each data item in the disk data table; Using the time identifier in the disk data table, a preset number of data items are selected from the disk data table and stored in the memory data table; wherein, the preset number is the maximum data storage capacity of the memory data table; and The interaction request is used to perform corresponding interactive operations on the memory data table.
3. The method according to claim 2, wherein, The disk data includes one or more data files; the step of selecting a preset number of data files from the disk and storing them in a memory data table using the time identifier in the disk data table includes: The number of target data files is determined by using the maximum data storage capacity of the memory data table and the data storage capacity of the data file. The target data files are determined using the number of target data files and the timestamps of each data file; and Using the timestamp of the target data file and the timestamp of each data item in the data file, target data is selected and stored sequentially in the memory data table.
4. The method according to claim 2, wherein, The method further includes: In response to a request to acquire target data, determine whether the acquisition request includes a time identifier; If the acquisition request includes a time identifier, the data corresponding to the time identifier is retrieved from the memory data table, and the data corresponding to the next time identifier is retrieved sequentially; and If the time identifier is not included in the acquisition request, the data corresponding to the first time identifier is obtained from the memory data table, and the data corresponding to the next time identifier is obtained in sequence.
5. The method according to claim 4, wherein, The memory data table also includes the association between accounts and data identifiers; the step of responding to a request to obtain target data, determining whether the request includes a timestamp, includes: In response to an account's request to retrieve target data, a data identifier is obtained from the interaction request, and an in-memory data table is used to determine whether there is an association between the account and the data identifier; and If the account and the data identifier are associated, determine whether the acquisition request includes a time identifier.
6. The method according to claim 2, wherein, The method further includes: In response to an interaction request for target data, the target data in the disk data table is added to the memory data table; and After a preset time has elapsed, the interactive operation corresponding to the interactive request is performed using the memory data table.
7. A data processing apparatus, wherein, The device includes: The data acquisition module is used to respond to a data addition request to a disk data table, acquire the request time of the data addition request, and determine the time identifier corresponding to the data addition request based on the request time; An identifier creation module is configured to, when the time identifier does not exist in the disk data table, lock the disk data table and create the time identifier in the disk data table; and The data interaction module is used to establish the association between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and to add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier.
8. The apparatus according to claim 7, wherein, The device further includes an identification acquisition module and a data storage module. The identifier acquisition module is used to acquire the time identifier of each data in the disk data table in response to the interaction request for target data; The data storage module is used to select a preset number of data items from the disk data table and store them in a memory data table using the timestamps in the disk data table; wherein the preset number is the maximum data storage capacity of the memory data table; and The data interaction module is also used to perform corresponding interactive operations on the memory data table using the interaction request.
9. The apparatus according to claim 8, wherein, The disk data includes one or more data files; the data storage module includes a quantity determination submodule, a file determination submodule, and a data storage submodule; The quantity determination submodule is used to determine the number of target data files by utilizing the maximum data storage capacity of the memory data table and the data storage capacity of the data file; The file determination submodule is used to determine the target data file using the number of target data files and the time identifier of each data file; and The data storage submodule is used to select target data from the target data file and store it in the memory data table in sequence using the time identifier of the target data file and the time identifier of each data in the data file.
10. The apparatus according to claim 8, wherein, The device further includes a request determination submodule, a data acquisition submodule, and a data acquisition module; The request judgment submodule is used to respond to a request to obtain target data and determine whether the request includes a time identifier. The data acquisition submodule is configured to, when the acquisition request includes a time identifier, retrieve the data corresponding to the time identifier from the memory data table, and sequentially retrieve the data corresponding to the next time identifier; and The data acquisition submodule is used to acquire the data corresponding to the first time identifier from the memory data table when the acquisition request does not include a time identifier, and then acquire the data corresponding to the next time identifier in sequence.
11. The apparatus according to claim 10, wherein, The memory data table also includes the association between accounts and data identifiers; the request judgment submodule includes an association determination unit and a request judgment unit; The association determination unit is used to respond to an account's request to obtain target data, obtain a data identifier from the interaction request, and use an in-memory data table to determine whether there is an association between the account and the data identifier; and The request determination unit is used to determine whether the acquisition request includes a time identifier when there is an association between the account and the data identifier.
12. The apparatus according to claim 8, wherein, The device also includes a data addition module and a data interaction module; The data adding module is used to add the target data from the disk data table to the memory data table in response to an interaction request for the target data. and The data interaction module is used to perform the interaction operation corresponding to the interaction request using the memory data table after a preset time has elapsed.
13. A computer device comprising a memory and one or more processors, the memory storing computer-readable instructions, wherein... When the one or more processors execute the computer-readable instructions, they implement the steps of the method according to any one of claims 1 to 6.
14. A computer storage medium having computer-readable instructions stored thereon, wherein, When the computer-readable instructions are executed by one or more processors, they implement the steps of the method according to any one of claims 1 to 6.
15. A computer-readable instruction product comprising computer-readable instructions, wherein, When executed by one or more processors, the computer-readable instructions implement the steps of the method according to any one of claims 1 to 6.