Data stream processing method, apparatus and system

Abstract

The present disclosure relates to the technical field of artificial intelligence, and provides a data stream processing method, apparatus and system. The processing method comprises: acquiring a token data stream, the token data stream comprising one or more token units, and each token unit comprising a token unit header and a token unit body; parsing metadata information in the one or more token units; selecting token data from the one or more token units on the basis of the metadata information; and inputting the selected token data into a model to obtain a processing result.

Description

Data stream processing methods, devices and systems

[0001] Cross-references

[0002] This disclosure claims priority to Chinese Patent Application No. 202411599614.6, filed on November 11, 2024, entitled "Data Stream Processing Method, Apparatus and System", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of artificial intelligence (AI) technology, and in particular to a method, apparatus and system for processing data streams. Background Technology

[0004] Token data is the basic data unit processed by the model. In the field of AIGC (Artificial Intelligence Generated Content), user input is processed by a large model to generate corresponding output, which is then presented to the user. In this process, the user's input first needs to be processed into a token stream. By parsing the token stream, token data can be obtained. The token data is then processed by the large model to obtain the processing result.

[0005] In large-scale model scenarios, token data streams may be transmitted between different nodes. Currently, there is no definition for token data streams in related technologies. Summary of the Invention

[0006] According to a first aspect of the present disclosure, a data stream processing method is provided, comprising: acquiring a token data stream, the token data stream including one or more token units, each token unit including a token unit header and a token unit body; parsing metadata information in the one or more token units; selecting token data from the one or more token units according to the metadata information; and inputting the selected token data into a model to obtain a processing result.

[0007] According to some embodiments of this disclosure, the one or more token units include a metadata token unit, and the metadata information includes one or more of first metadata information and second metadata information, wherein: the first metadata information includes information located in the token unit header of the token unit other than the metadata token unit, and the second metadata information includes information located in the token unit body of the metadata token unit.

[0008] According to some embodiments of this disclosure, the first metadata information includes one or more of first information related to the token unit and second information related to the token frame in the token unit body.

[0009] According to some embodiments of this disclosure, the first information includes one or more of the following: token unit type, token unit start code, token unit length, and token unit level.

[0010] According to some embodiments of this disclosure, the token unit type indicates whether the token unit is a single token frame unit or a hybrid token frame unit.

[0011] According to some embodiments of this disclosure, the second information includes one or more of the following: the number of token frames, the identifier of the token frames, the length of the token frames, the type of the token frames, and the grouping information of the token frames.

[0012] According to some embodiments of this disclosure, the type of the token frame indicates that the token data contained in the token frame is text features, image features, audio features, or multimodal features.

[0013] According to some embodiments of this disclosure, selecting token data from the one or more token units based on the metadata information includes: selecting a token unit of a specific level based on the token unit hierarchy and the token unit length; and selecting token data contained in the token frame within the token unit of the specific level.

[0014] According to some embodiments of this disclosure, the specific level includes multiple levels, and the model includes multiple functional modules; the step of inputting the selected token data into the model to obtain the processing result includes: inputting the token data in the token units of the multiple levels into the functional modules corresponding to each of the multiple levels to obtain the processing result.

[0015] According to some embodiments of this disclosure, the step of selecting token data from the one or more token units based on the metadata information includes one or more of the following: selecting token data contained in a token frame of a specific type based on the type and length of the token frame; selecting token data contained in a token frame of a specific group based on the grouping information and length of the token frame; selecting token data contained in a token frame with a specific identifier based on the identifier and length of the token frame, and skipping redundant token frames.

[0016] According to some embodiments of this disclosure, the second metadata information includes one or more of encoding / decoding information and prompt word information.

[0017] According to a second aspect of the present disclosure, a data stream processing method is provided, comprising: acquiring token data; determining metadata information based on the characteristics of the token data; and generating a token data stream based on the token data and the metadata information, wherein the token data stream includes one or more token units, and each token unit includes a token unit header and a token unit body.

[0018] According to some embodiments of this disclosure, the one or more token units include a metadata token unit, and the metadata information includes one or more of first metadata information and second metadata information, wherein: the first metadata information includes information located in the token unit header of the token unit other than the metadata token unit, and the second metadata information includes information located in the token unit body of the metadata token unit.

[0019] According to some embodiments of this disclosure, the first metadata information includes one or more of first information related to the token unit and second information related to the token frame in the token unit body.

[0020] According to some embodiments of this disclosure, the first information includes one or more of the following: token unit type, token unit start code, token unit length, and token unit level.

[0021] According to some embodiments of this disclosure, the token unit type indicates whether the token unit is a single token frame unit or a hybrid token frame unit.

[0022] According to some embodiments of this disclosure, the second information includes one or more of the following: the number of token frames, the identifier of the token frames, the length of the token frames, the type of the token frames, and the grouping information of the token frames.

[0023] According to some embodiments of this disclosure, the type of the token frame indicates that the token data contained in the token frame is text features, image features, audio features, or multimodal features.

[0024] According to some embodiments of this disclosure, the second metadata information includes one or more of encoding / decoding information and prompt word information.

[0025] According to some embodiments of this disclosure, the metadata information further includes third metadata information, which includes information in the token unit header of the metadata token unit.

[0026] According to a third aspect of the present disclosure, a data stream processing apparatus is provided, comprising: a module configured as the processing method described in any of the above embodiments.

[0027] According to a fourth aspect of the present disclosure, a data stream processing apparatus is provided, comprising: a memory; and a processor coupled to the memory, configured to execute the processing method described in any of the above embodiments based on instructions stored in the memory.

[0028] According to a fifth aspect of the present disclosure, a data stream processing system is provided, comprising: a first node configured to generate a token data stream according to the processing method of a second aspect of the present disclosure; and a second node configured to process the token data stream according to the processing method of a first aspect of the present disclosure to obtain the processing result, and to transmit the processing result to the first node.

[0029] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, including computer program instructions, wherein the computer program instructions, when executed by a processor, implement the processing methods described in any of the above embodiments.

[0030] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, it implements the processing methods described in any of the above embodiments. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] The accompanying drawings form part of this specification, illustrating exemplary embodiments of the present disclosure, and together with the specification serve to explain the principles of the present disclosure.

[0033] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, in which:

[0034] Figure 1 is a schematic flowchart of a data stream processing method according to some embodiments of the present disclosure.

[0035] Figure 2 is a schematic diagram of the structure of a token data stream according to some embodiments of this disclosure.

[0036] Figure 3 is a schematic flowchart of a data stream processing method according to some other embodiments of this disclosure.

[0037] Figure 4 is an application example of the data stream processing method of some embodiments of this disclosure.

[0038] Figure 5 is a schematic diagram of the structure of a data stream processing apparatus according to some embodiments of the present disclosure.

[0039] Figure 6 is a schematic diagram of the structure of a data stream processing apparatus according to other embodiments of the present disclosure.

[0040] Figure 7 is a schematic diagram of the structure of a data stream processing apparatus according to some embodiments of the present disclosure.

[0041] Figure 8 is a schematic diagram of the structure of a data stream processing system according to some embodiments of the present disclosure. Detailed Implementation

[0042] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0043] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0044] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0045] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0046] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0047] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0048] In large AI model applications, token data streams may be transmitted between different nodes, such as between servers and clients, or between nodes at different levels of the server.

[0049] The token data stream processing method proposed in this disclosure can be applied to any node that needs to process the token data stream. For example, the node can be a server, a client, a node in a multi-level server, or a transmission node between a server and a client.

[0050] Figure 1 is a schematic flowchart of a data stream processing method according to some embodiments of the present disclosure.

[0051] In some embodiments, the data stream processing method includes one or more steps in steps 102-108 as shown in FIG1.

[0052] In step 102, obtain the token data stream.

[0053] In some embodiments, a node obtains a token data stream from another node; for example, a server obtains a token data stream from a client. The token data stream can be generated by the client; for example, the client maps user input to token data and then generates a token data stream based on the token data.

[0054] Figure 2 is a schematic diagram of the structure of a token data stream according to some embodiments of this disclosure.

[0055] As shown in Figure 2, the token data stream includes one or more token units 10, each token unit 10 including a token unit header 11 and a token unit payload 12. It can be understood that the portion of the token unit 10 excluding the token unit header 11 constitutes the token unit payload 12. The token unit payload 12 can also be referred to as the token unit payload.

[0056] In some embodiments, the token unit 10 in the token data stream includes a metadata token unit 10a and other token units 10b besides the metadata token unit 10a, i.e., non-metadata token units. The number of metadata token units 10a can be one or more, and the number of other token units 10b can also be one or more.

[0057] Figure 2 schematically illustrates the contents of the token unit body 12 in the token unit 10.

[0058] For example, for metadata token unit 10a, the token unit body 12 includes metadata. In some embodiments, the type of metadata includes multiple types, such as type A or type B. For example, type A indicates that the metadata is token information metadata, and type B indicates that the metadata is prompt word metadata.

[0059] For example, for other token units 10b, the token unit body 12 includes a token frame containing token data, such as token data 1, token data 2, ..., token data N. In some embodiments, the token frame type includes multiple types, such as type A, type B, etc. For example, type A indicates that the token data contained in the token frame is text-based; type B indicates that the token data contained in the token frame is image-based.

[0060] In step 104, the metadata information in one or more token units is parsed.

[0061] In some embodiments, the metadata information in the token data stream may include information located in the token unit header 11. In other embodiments, where the token data stream includes a metadata token unit 10a, the metadata information in the token data stream may also include information located in the token unit body 12 of the metadata token unit 10a.

[0062] In step 106, token data is selected from one or more token units based on metadata information.

[0063] In some embodiments, the selected token data may be token data from the token data stream that meets preset conditions. These preset conditions may be determined, for example, based on metadata information. This will be described below in conjunction with different embodiments.

[0064] It should be understood that when the token unit 10 includes the metadata token unit 10a, token data is selected from the other token units 10b besides the metadata token unit 10a.

[0065] In some implementations, the selected token data includes token data contained in at least a portion of the token frames within a portion of the token unit 10b. In other implementations, the selected token data includes token data contained in at least a portion of the token frames within each token unit 10b.

[0066] In step 108, the selected token data is input into the model to obtain the processing result.

[0067] In some embodiments, the model is a large model or one of multiple functional modules of a large model. For example, the large model may be one of multiple candidate large models. As some implementations, the large model includes a multi-layered structure, and each functional module of the large model may include one or at least two layers of the large model's structure.

[0068] In some embodiments, the processing result may include a token data stream (e.g., obtained after processing a large text model) or a media data stream. Media data streams may include, for example, image data, video data, audio data, etc.

[0069] In some embodiments, the processing result may be further parsed and presented to the user. For example, the parsing may be performed by the node executing the method shown in Figure 1 itself; or, for example, the node executing the processing method shown in Figure 1 may transmit the processing result to another node (e.g., a client) for parsing. Parsing may be, for example, demapping (e.g., decoding) the token data stream, or decoding the media data stream.

[0070] In this way, the user's input can be processed to obtain the final result.

[0071] In the above embodiments, the token data in the token data stream is organized in the form of token units, each token unit containing a token unit header and a token unit body excluding the header. By defining the structure of the token data stream and dividing it into token units, token data can be randomly accessed on a token unit basis when selecting token data based on metadata information; that is, any token unit that needs to be accessed can be accessed. Thus, it is possible to select any appropriate token data input model to obtain a processing result that meets the user's needs.

[0072] In some embodiments, the metadata information parsed in step 104 includes one or more of first metadata information and second metadata information. Here, the first metadata information includes information located in the token unit header 11 of the token unit 10b (excluding metadata token unit 10a), and the second metadata information includes information located in the token unit body 12 of the metadata token unit 10a.

[0073] The first metadata information and the second metadata information are introduced below.

[0074] First, let's introduce the first piece of metadata.

[0075] In some embodiments, the first metadata information includes one or more of the following: first information related to the token unit 10b and second information related to the token frames in the token unit body 12. It is understood that the first information reflects information about the token unit 10b as a whole, while the second information reflects information about the token frames within the token unit 10b.

[0076] In some embodiments, the first information includes one or more of the following: token unit type, token unit start code, token unit length, and token unit level.

[0077] As some implementations, the token unit type indicates whether the token unit 10b is a single token frame unit or a mixed token frame unit.

[0078] If token unit 10b is a single token frame unit, then token unit 10b includes only one token frame. For example, the token data contained in the unique token frame in token unit 10b is one of text features, image features, audio features, and multimodal features. Here, the multimodal features include at least two features, for example, at least two of text features, image features, and audio features.

[0079] If token unit 10b is a hybrid token frame unit, then token unit 10b includes multiple token frames. In some implementations, the multiple token frames may be of the same type; for example, the token data contained in the multiple token frames may all be of the same type among text features, image features, audio features, and multimodal features. In other implementations, at least some of the token frames may be of different types. For example, one token frame in token unit 10b may contain token data of text features, while another token frame may contain token data of audio features.

[0080] As some implementations, the token unit length indicates the overall length of the token unit 10b or the length of the token unit body 12 within the token unit 10b.

[0081] In some implementations, the token unit hierarchy is determined based on the priority or weight of the token units. For example, the priority or weight of a token unit is determined based on the importance of the token data within that token unit.

[0082] In some embodiments, the second information includes one or more of the following: the number of token frames, the token frame identifier, the token frame length, the token frame type, and the token frame grouping information.

[0083] In some implementations, the length of the token frame can be an absolute length, such as the total number of bytes contained in the token frame. In other implementations, the length of the token frame can also be a relative length, that is, an offset value relative to the starting byte of the token unit in which the token frame is located.

[0084] As some implementations, the type of the token frame indicates whether the token data contained in the token frame is text features, image features, audio features, or multimodal features.

[0085] For different types of token units, the information of the token frames in the token units included in the second information may not be completely different. For example, the second information may include one or more of the following three items in a single token frame unit: the token frame identifier, the token frame type, and the token frame grouping information. As another example, the second information may include one or more of the following five items in a mixed token frame unit: the number of token frames, the token frame identifier, the token frame length, the token frame type, and the token frame grouping information.

[0086] Next, we will introduce the second metadata information.

[0087] In some embodiments, the second metadata information includes one or more of encoding / decoding information and prompt information. For example, the encoding / decoding information indicates the encoding / decoding method of the token data, such as the encoding / decoding method of text, image, and audio, respectively. When the second metadata information includes prompt information, the node can more accurately identify the user's needs based on the prompt information, thereby helping to obtain a more satisfactory processing result for the user.

[0088] In addition to the first and second metadata information described above, the metadata information in the token data stream may also include other metadata information, such as third metadata information, which includes information located in the token unit header 11 of the metadata token unit 10a.

[0089] In some embodiments, the third metadata information includes one or more of the following: token unit type, token unit length, and update flag.

[0090] The token unit type indicates that the token unit 10 is a metadata token unit 10a, and the token unit length indicates the overall length of the metadata token unit 10a or the length of the token unit body 12 in the metadata token unit 10a.

[0091] An update flag indicates whether the content of the token body 12 of a certain metadata token unit 10a has been updated relative to the previous metadata token unit 10a. For example, if the token data stream includes multiple metadata token units 10a, when parsing metadata information, the update flag in a certain metadata token unit 10a can determine whether it is necessary to parse the content of the token body 12 of that metadata token unit 10a. If the content of the token body 12 of that metadata token unit 10a has been updated relative to the previous metadata token unit 10a, then the content of the token body 12 of that metadata token unit 10a needs to be parsed; if the content of the token body 12 of that metadata token unit 10a has not been updated relative to the previous metadata token unit 10a, then the content of the token body 12 of that metadata token unit 10a does not need to be parsed.

[0092] In some embodiments, the metadata information includes one or more of first metadata information, second metadata information, and third metadata information. By extending the metadata information, various token data streams can be parsed as needed.

[0093] In some embodiments, token data can be selected based on one or more of the first metadata information and the second metadata information.

[0094] The following describes some methods for selecting token data based on the first metadata information. When selecting token data, one or more of the following methods can be used. It is understood that when selecting token data, it is chosen from other token units 10b besides the metadata token unit 10a.

[0095] In some methods, token units at a specific level are selected based on the token unit hierarchy and token unit length. In this case, the token data contained in the token frame within the token unit at a specific level can be selected.

[0096] For example, a specific level may include multiple levels, and the model may include multiple functional modules. In this case, token data from token units at multiple levels can be input into the corresponding functional modules at each level to obtain the processing results. For example, token data from token units at different levels can be input into different functional modules within the model.

[0097] In other methods, the token data contained in a specific type of token frame is selected based on the token frame type and the token frame length. The type of token frame allows selection of a specific type, while the length of the token frame allows precise determination of the token data contained within that specific type. For example, the token data contained in a specific type of token frame might be text features.

[0098] In other methods, the token data contained in a specific group of token frames is selected based on the token frame grouping information and the token frame length. The grouping information allows selection of a specific group of token frames, while the length of each token frame accurately determines the token data contained within that group. For example, if a group of token frames comprises four frames, the grouping information identifies these four frames as belonging to the same group, and the length of each of these four frames determines the token data contained in each frame, thus achieving the goal of selecting the token data contained in a specific group of token frames.

[0099] Among the above methods, by dividing the token unit into levels and classifying or grouping the token frames, the corresponding token data can be selectively processed when selecting token data, thereby improving data processing efficiency.

[0100] In other methods, based on the token frame's identifier and length, the token data contained in a token frame with a specific identifier is selected, and redundant token frames are skipped. For example, a backup token frame of a certain token frame can be added to a mixed token frame unit. If that token frame has already been parsed during parsing, the backup token frame, i.e., the redundant token frame, can be skipped. In this way, the token data can be made more resistant to packet loss during transmission, reducing stuttering during token data parsing.

[0101] The following describes some ways to select token data based on second-level metadata information.

[0102] In some embodiments, token data with the same encoding / decoding method is selected based on the encoding / decoding information, and the token data with the same decoding method is input into a model with corresponding decoding capabilities. For example, token data with different decoding methods are input into different sub-models within the model. This improves the reliability of data processing.

[0103] Figure 3 is a schematic flowchart of a data stream processing method according to some other embodiments of this disclosure.

[0104] In some embodiments, the data stream processing method includes one or more steps in steps 302-306 as shown in FIG3.

[0105] In step 302, obtain the token data.

[0106] Here, the acquired token data may include one or more of the following: token data generated by a large model, token data obtained after parsing a token data stream, and token data generated based on user prompts.

[0107] In step 304, metadata information is determined based on the characteristics of the token data.

[0108] The characteristics of token data include, for example, the type of token data, the quantity of token data, and the size of token data.

[0109] Metadata information includes one or more of the first, second, and third metadata information described above. The specific content of the first, second, and third metadata information can be found in the description above and will not be repeated here.

[0110] In step 306, a token data stream is generated based on the token data and metadata information.

[0111] Referring to Figure 2, the generated token data stream includes one or more token units 10, each token unit 10 including a token unit header 11 and a token unit body 12.

[0112] In some embodiments, metadata information may be determined based on one or more of transmission requirements, user needs, and model characteristics, and the token data and metadata information may be organized into one or more token units.

[0113] For example, transmission requirements include network conditions. In cases of poor network quality, redundant token frames can be added to the token data stream; in cases of good network quality, redundant token frames are not necessary.

[0114] For example, users might focus on text-based token data. In this case, the token data can be categorized (e.g., by adding a token frame type) to distinguish text-based token data from other types. Alternatively, token units can be organized by filtering only text-based token data.

[0115] In some embodiments, the token unit 10 in the generated token data stream includes one or more metadata token units 10a, and one or more other token units 10b other than metadata token units 10a, i.e., non-metadata token units.

[0116] In some embodiments, the token data stream is transmitted after it is generated. For example, a node generates a token data stream and transmits it to another node, which then parses the token data stream and performs subsequent processing.

[0117] In the above embodiments, the token data in the generated token data stream is organized in the form of token units, each token unit containing a token unit header and a token unit body excluding the header. By defining the structure of the token data stream and dividing it into token units, the token data can be flexibly organized according to requirements when generating the token data stream. Furthermore, when selecting token data to input into the model, random access to the token data can be performed on a token unit basis, which helps to meet various needs.

[0118] This disclosure provides some embodiments that offer examples of token unit definitions, which are illustrated below with reference to Tables 1-3. The syntax and semantic information described below can correspond to the token unit described in any of the above embodiments.

[0119] definition:

[0120] The TokenUnit() function contains a TokenUnitHeader() header and a TokenUnitPayload() body. The TokenUnitPayload() body contains all data except for the TokenUnitHeader.

[0121] grammar:

[0122] Table 1 shows the syntax of TokenUnit().

[0123] Table 1

[0124] Table 2 shows the syntax of the TokenUnitHeader() function.

[0125] Table 2

[0126] Table 3 shows the syntax of the token unit body TokenUnitPayload().

[0127] Table 3

[0128] Semantics:

[0129] The `unitType` parameter indicates the token unit type. For example, a value of 0 indicates a single token frame unit; a value of 1 indicates a mixed token frame unit; and a value of 2 indicates a metadata token unit.

[0130] unitSize indicates the length of the entire token unit (i.e., the token unit header TokenUniHeader and the token unit payload body), for example, in bytes.

[0131] unitPayloadSize indicates the length of the token unit body TokenUnitPayload, for example, in bytes.

[0132] The `unitLevelFlag` indicates whether to indicate the token unit's hierarchy information. For example, a value of 0 indicates no token unit hierarchy information, while a value of 1 indicates the token unit hierarchy information.

[0133] `unitLevel` indicates the level of the token unit. For example, token units can be categorized based on their priority. A smaller value indicates a higher priority token unit.

[0134] The frameId indicates the identifier of the token frame. The token frame identifier is used to distinguish the order of token frames.

[0135] The tokenType indicates the data type of the token data contained in the token frame. For example, a value of 1 indicates that the token data has text features; a value of 2 indicates that the token data has image features; a value of 3 indicates that the token data has audio features; and a value of 4 indicates that the token data has multimodal features.

[0136] frameNum indicates the number of token frames contained in a hybrid token frame unit.

[0137] frameId[i] indicates the identifier of the i-th token frame in the hybrid token frame unit.

[0138] tokenType[i] indicates the data type of the token data contained in the i-th token frame in the hybrid token frame unit. The meaning of the field value is the same as that of tokenType.

[0139] The tokenGroupFlag indicates whether to indicate the grouping information of the token data. For example, a value of 0 indicates no grouping information of the token data, while a value of 1 indicates the grouping information of the token data.

[0140] tokenGroupType indicates the group identifier of the token data; tokens with the same group identifier belong to the same group.

[0141] tokenGroupType[i] indicates the group identifier of the token data contained in the i-th token frame in the mixed token frame unit. The meaning of the field value is the same as that of tokenGroupType.

[0142] frameOffset[i] indicates the length of the i-th token frame in the mixed token frame unit, for example, in bytes.

[0143] The updateFlag indicates the update flag for the metadata token unit. For example, a value of 0 indicates that the content of the token unit body TokenUnitPayload of the metadata token unit has not been updated compared to the previous metadata token unit; a value of 1 indicates that the content of the token unit body TokenUnitPayload of the metadata token unit has been updated compared to the previous metadata token unit.

[0144] reserved fields are fields that are kept in reserve.

[0145] codecType indicates the decoding method for token data. For example, it indicates the respective decoding method for different types of token data.

[0146] The promptFlag indicates whether to indicate prompt-related information. For example, a value of 0 indicates no prompt-related information, while a value of 1 indicates prompt-related information.

[0147] promptLen indicates the length of the prompt word, for example, in bytes.

[0148] promptByte[i] indicates one byte of the prompt word. It should be noted that the prompt word information is not limited to text information, but can also be other media information, such as images, audio, etc.

[0149] payloadByte[i] indicates a single byte within the token frame.

[0150] Other embodiments of this disclosure provide another example of token unit definition, which is described below with reference to Tables 4-7. This example differs from the examples shown in Tables 1-3 in two ways: a) the start code unitStartCode is used instead of unitType and unitSize; b) the metadata token unit is subdivided into two types: token information metadata token unit and prompt word metadata token unit.

[0151] One of the differences a and b mentioned above can individually replace the corresponding part in Tables 1-3, or the differences a and b mentioned above can be combined to replace the corresponding part in Tables 1-3, thereby obtaining more implementation methods.

[0152] Table 4 shows the syntax of the TokenUnitHeader() function.

[0153] Table 4

[0154] Table 5 shows the syntax of the token unit body TokenUnitPayload().

[0155] Table 5

[0156] Table 6 shows the syntax of the metadata information MetadataTokenInfo() in the token unit body TokenUnitPayload() of the token information metadata token unit.

[0157] Table 6

[0158] Table 7 shows the syntax of the metadata information MetadataTokenInfo() in the token unit body TokenUnitPayload() of the prompt word metadata token unit.

[0159] Table 7

[0160] The following only introduces the semantics of the different fields in the examples shown in Tables 4-7 and Tables 1-3. The semantics of the other fields are described above and will not be repeated here.

[0161] The `unitStartCode` indicates the starting code of the token unit, used to distinguish different token unit types. Each token unit begins with the token unit starting code.

[0162] `nextEndCode()` is a conditional function that returns a boolean value. It returns 1 if the last 32 bits are the token frame end code 0x000001C0, and 0 otherwise.

[0163] Single token frame units and mixed token frame units end with the token frame end code.

[0164] In addition, data types are illustrated in the notes in Tables 1-7. It should be understood that data types are not restrictive.

[0165] Next, referring to Figure 4, an application example of this disclosure embodiment will be given, taking a large-scale model application as an example.

[0166] Figure 4 is an application example of a data stream processing method according to some embodiments of this disclosure. In the example shown in Figure 4, the data stream processing method is executed by at least two nodes, including a client and a server.

[0167] In step S1, the client receives user input. For example, the user describes the task in words and attaches a video to be processed.

[0168] The user-described task serves as the prompt information. For example, the user-described task could be to extend a video of a first duration into a second duration, where the second duration is longer than the first.

[0169] In step S2, the client maps the user's input to token data and generates a token data stream A based on the token data.

[0170] The client can generate token data stream A according to the processing method described in the related embodiments of Figure 3. The structure of token data stream A conforms to the structure of token data stream proposed in the embodiments of this disclosure.

[0171] Token data stream A comprises multiple token units, in which some token frames contain token data with image features, and some token frames contain token data with audio features. For example, a token unit may contain only token data with image features; another example may contain only token data with audio features; yet another example may contain some token frames with audio features and some token frames with image features.

[0172] In step S3, the server parses the metadata information in the token unit of the token data stream A, and extracts different types of token frames according to the token unit type, token unit length, token frame type and token frame length in the metadata information.

[0173] If the token unit is a single token frame unit, then the TokenUnitPayload() portion of that token unit is a token frame of the type indicated in TokenUnitHeader().

[0174] If the token unit is a mixed frame token unit, the corresponding type of token frame can be extracted according to the token frame type and token frame length indicated in TokenUnitHeader().

[0175] The extracted token frames are reorganized into two sub-streams based on their type: Token Sub-Stream A1 and Token Sub-Stream A2. In other words, based on the extracted token frames, token units are re-divided and the corresponding fields in the TokenUnitHeader() function of each token unit are populated.

[0176] The token frames in token sub-data stream A1 contain token data with image features, and the token frames in token sub-data stream A2 contain token data with audio features. Both token sub-data streams A1 and A2 conform to the token data stream structure proposed in the embodiments of this disclosure. Token sub-data stream A1 corresponds to the video to be processed, and token sub-data stream A2 corresponds to the voice describing the user's task.

[0177] In step S4, the server inputs the token sub-data stream A1 into the video big model to obtain the corresponding processing result (i.e., token data stream B), and inputs the token sub-data stream A2 into the audio big model to obtain the corresponding processing result (i.e., token data stream C).

[0178] Both token data stream B and token data stream C conform to the token data stream structure proposed in the embodiments of this disclosure.

[0179] For example, a video big model can perform preliminary processing on the video to be processed (e.g., understanding the video content) to obtain a token data stream B, and an audio big model can perform preliminary processing on the user-described task (e.g., understanding the task) to obtain a token data stream C.

[0180] In step S5, the server inputs token data stream B and token data stream C into the multimodal large model to obtain the processing result (i.e., media data stream).

[0181] The multimodal large model can process token data stream C and token data stream B according to the processing method proposed in the embodiments of this disclosure to obtain the user-described task and the video to be processed. Then, the multimodal large model can expand the video to be processed into a video of a second duration according to the task requirements.

[0182] In step S6, the server transmits the media data stream to the client, which processes the media data stream (e.g., performs audio and video decoding and rendering) to present the corresponding content.

[0183] In the above examples, the client and server can organize the token data to generate a token data stream according to the processing method of the embodiments of this disclosure, or, after parsing the token data stream, select the corresponding token data input model to generate the processing result and finally present it to the user.

[0184] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they largely correspond to the method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0185] This disclosure also provides a data stream processing apparatus, including a module configured to execute the processing method of any of the foregoing embodiments.

[0186] Figure 5 is a schematic diagram of the structure of a data stream processing apparatus according to some embodiments of the present disclosure.

[0187] As shown in Figure 5, the data stream processing device includes an acquisition module 501, a parsing module 502, a selection module 503, and an input module 504.

[0188] The acquisition module 501 is configured to acquire a token data stream, which includes one or more token units, each token unit including a token unit header and a token unit body.

[0189] The parsing module 502 is configured to parse metadata information in one or more token units.

[0190] Selection module 503 is configured to select token data from one or more token units based on metadata information.

[0191] Input module 504 is configured to input selected token data into the model to obtain the processing result.

[0192] Figure 6 is a schematic diagram of the structure of a data stream processing apparatus according to some embodiments of the present disclosure.

[0193] As shown in Figure 6, the data stream processing device includes an acquisition module 601, a determination module 602, and a generation module 603.

[0194] The acquisition module 601 is configured to acquire token data.

[0195] The determination module 601 is configured to determine metadata information based on the characteristics of the token data.

[0196] The generation module 603 is configured to generate a token data stream based on token data and metadata information. The token data stream includes one or more token units, each token unit including a token unit header and a token unit body.

[0197] Figure 7 is a schematic diagram of the structure of a data stream processing apparatus according to some embodiments of the present disclosure.

[0198] As shown in FIG7, the data stream processing apparatus 700 includes a memory 701 and a processor 702 coupled to the memory 701. The processor 702 is configured to execute the method of any of the foregoing embodiments based on instructions stored in the memory 701.

[0199] In some embodiments, memory 701 may include, for example, system memory, fixed non-volatile storage medium, etc. System memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0200] In some embodiments, the data stream processing apparatus 700 may further include an input / output interface 703, a network interface 704, a storage interface 705, etc. These interfaces 703, 704, and 705, as well as the memory 701 and processor 702, can be connected, for example, via a bus 706. The input / output interface 703 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 704 provides a connection interface for various networked devices. The storage interface 705 provides a connection interface for external storage devices such as SD cards (Secure Digital Memory Cards) and USB flash drives (Universal Serial Bus).

[0201] Figure 8 is a schematic diagram of the structure of a data stream processing system according to some embodiments of the present disclosure.

[0202] As shown in Figure 8, the data stream processing system includes a first node 801 and a second node 802. For example, the first node 801 is a client, and the second node 802 is a server. Alternatively, the first node 801 may be a node within the server, and the second node 802 may be another node within the server.

[0203] The first node 801 is configured to generate a token data stream according to the data stream processing method provided in the embodiment shown in FIG3 and any embodiment related to FIG3, and transmit the generated token data stream to the second node 802.

[0204] The second node 802 is configured to process the token data stream generated by the first node 801 to obtain a processing result according to the data stream processing method provided in the embodiment shown in FIG1 and any embodiment related to FIG1; and to transmit the processing result to the first node 801.

[0205] In some embodiments, the second node 802 can directly transmit the processing result to the first node 801. In other embodiments, the second node 802 can transmit the processing result to the first node 801 via other nodes.

[0206] For example, the first node 801 can parse the processing result to present the corresponding content to the user. Alternatively, the first node 801 can further process the processing result to obtain another processing result, which can then be transmitted to other nodes for parsing to present the corresponding content to the user.

[0207] This disclosure also provides a computer-readable storage medium including computer program instructions that, when executed by a processor, implement the processing method of any of the above embodiments.

[0208] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the processing method of any of the above embodiments.

[0209] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0210] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0211] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that the functions specified in one or more flowchart illustrations and / or one or more block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0212] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0214] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for processing a data stream, comprising: Acquire a token data stream, the token data stream comprising one or more token units, each token unit comprising a token unit header and a token unit body; Parse the metadata information in the one or more token units; Select token data from one or more token units based on the metadata information; as well as Input the selected token data into the model to obtain the processing result.

2. The processing method according to claim 1, wherein, The one or more token units include metadata token units, and the metadata information includes one or more of first metadata information and second metadata information, wherein: The first metadata information includes information located in the token unit header of the token unit other than the metadata token unit, and The second metadata information includes information located in the token unit body of the metadata token unit.

3. The processing method according to claim 2, wherein, The first metadata information includes one or more of the following: first information related to the token unit and second information related to the token frame in the token unit body.

4. The processing method according to claim 3, wherein, The first information includes one or more of the following: Token unit type, token unit start code, token unit length, and token unit level.

5. The processing method according to claim 4, wherein, The token unit type indicates whether the token unit is a single token frame unit or a mixed token frame unit.

6. The processing method according to any one of claims 3-5, wherein, The second information includes one or more of the following: The number of token frames, the identifier of the token frame, the length of the token frame, the type of the token frame, and the grouping information of the token frame.

7. The processing method according to claim 6, wherein, The type of the token frame indicates whether the token data contained in the token frame is text features, image features, audio features, or multimodal features.

8. The processing method according to any one of claims 4-5, wherein, The step of selecting token data from the one or more token units based on the metadata information includes: Select a token unit of a specific level based on the token unit level and the token unit length; and Select the token data contained in the token frame within the token unit of the specific level.

9. The processing method according to claim 8, wherein, The specific level includes multiple levels, and the model includes multiple functional modules; The step of inputting the selected token data into the model to obtain the processing result includes: The token data from the token units at the multiple levels is input into the corresponding functional modules at each level to obtain the processing result.

10. The processing method according to claim 6, wherein, The step of selecting token data from the one or more token units based on the metadata information includes one or more of the following: Based on the type and length of the token frame, select the token data contained in a specific type of token frame; Based on the grouping information and length of the token frames, select the token data contained in a specific group of token frames; Based on the identifier and length of the token frame, select the token data contained in the token frame with the specific identifier, and skip redundant token frames.

11. The processing method according to any one of claims 2-5, wherein, The second metadata information includes one or more of the encoding / decoding information and prompt word information.

12. A method for processing a data stream, comprising: Retrieve token data; Based on the characteristics of the token data, determine the metadata information; as well as A token data stream is generated based on the token data and the metadata information. The token data stream includes one or more token units, and each token unit includes a token unit header and a token unit body.

13. The processing method according to claim 12, wherein, The one or more token units include metadata token units, and the metadata information includes one or more of first metadata information and second metadata information, wherein: The first metadata information includes information located in the token unit header of the token unit other than the metadata token unit, and The second metadata information includes information located in the token unit body of the metadata token unit.

14. The processing method according to claim 13, wherein, The first metadata information includes one or more of the following: first information related to the token unit and second information related to the token frame in the token unit body.

15. The processing method according to claim 14, wherein, The first information includes one or more of the following: Token unit type, token unit start code, token unit length, and token unit level.

16. The processing method according to claim 15, wherein, The token unit type indicates whether the token unit is a single token frame unit or a mixed token frame unit.

17. The processing method according to any one of claims 14-16, wherein, The second information includes one or more of the following: The number of token frames, the identifier of the token frame, the length of the token frame, the type of the token frame, and the grouping information of the token frame.

18. The processing method according to claim 17, wherein, The type of the token frame indicates whether the token data contained in the token frame is text features, image features, audio features, or multimodal features.

19. The processing method according to any one of claims 13-16, wherein, The second metadata information includes one or more of the encoding / decoding information and prompt word information.

20. The processing method according to any one of claims 12-16, wherein, The metadata information also includes third metadata information, which includes information in the token unit header of the metadata token unit.

21. A data stream processing apparatus, comprising: A module configured to perform the processing method according to any one of claims 1-20.

22. A data stream processing apparatus, comprising: Memory; as well as A processor coupled to the memory is configured to execute the processing method of any one of claims 1-20 based on instructions stored in the memory.

23. A data stream processing system, comprising: The first node is configured to generate a token data stream according to the processing method of any one of claims 12-20; as well as The second node is configured to process the token data stream according to any one of claims 1-11 to obtain a processing result, and to transmit the processing result to the first node.

24. A computer-readable storage medium comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the processing method according to any one of claims 1-20.

25. A computer program product comprising a computer program, wherein, When the computer program is executed by the processor, it implements the processing method according to any one of claims 1-20.

Images