Method for determining reading order of data set, apparatus, electronic device and storage medium

Abstract

Provided in the embodiments of the present application are a method for determining the reading order of a data set, an apparatus, an electronic device and a computer-readable storage medium. On the basis of data load characteristics of a data set, the present method may allocate data having different load characteristics to different groups, thereby ensuring that data load characteristics corresponding to data in each group are similar. In addition, on the basis of the sizes of batches, the method may also divide the data within each group into a plurality of batches. By means of random selection among the plurality of batches in the plurality of groups, the reading order of the data set can be determined. In this way, the order of the plurality of batches in the data set is randomly shuffled, and data in a same batch is derived from a same group. In this way, when data of each batch is processed, since the data has the similar load characteristics and is more balanced, time and resources required for processing a plurality of pieces of data in the batch are more consistent, thereby accelerating a training process or an inference process of the data set, and avoiding waste of resources.

Description

Methods, devices, electronic equipment, and storage media for determining the order in which datasets are read.

[0001] This application claims priority to Chinese Patent Application No. 202510331009.9, filed on March 19, 2025, entitled “Method, Apparatus, Electronic Device, Storage Medium for Determining Dataset Reading Order”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application relate to the field of computer technology. The embodiments of this application relate to methods for determining the reading order of datasets, apparatus for determining the reading order of datasets, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0003] Artificial Intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. Among these, Data Parallelism (DP) is a widely used technique in distributed AI systems. It improves the computational efficiency and speed of AI models by dividing the dataset into multiple subsets and processing these subsets in parallel across different computing nodes. In data parallelism, each computing node receives a complete copy of the trained model to process different subsets of data. This method distributes the training task across multiple nodes, significantly improving processing speed and efficiency. Summary of the Invention

[0004] This application provides a scheme for determining the reading order of a dataset. This scheme allocates data with different load characteristics to different groups based on the dataset's data load characteristics, thereby ensuring that the data load characteristics corresponding to the data within a group are similar. Furthermore, this scheme can also divide the data within a group into multiple batches based on batch size. By randomly selecting data from multiple batches within multiple groups, the reading order of the dataset can be determined. In this way, the order of multiple batches in the dataset is randomly shuffled, and the data within the same batch originates from the same group. Through this method, when processing a batch of data, because the data has more balanced similar load characteristics, the time and resources required to process multiple data points in a batch are more consistent, thereby accelerating the training or inference process of the dataset and avoiding resource waste.

[0005] In some implementations of the first aspect, the dataset is a video dataset, and based on the data load characteristics of the dataset, multiple data points are divided into multiple groups, including: dividing multiple videos in the video dataset into multiple groups based on the video duration and / or semantic content corresponding to multiple videos in the video dataset. In this way, multiple videos can be divided according to the load characteristics of the video dataset, thereby achieving load balancing within each batch. This ensures that the resources and time required to process multiple videos in a subsequent batch are similar, thus avoiding resource waste during video processing.

[0006] In some implementations of the first aspect, the dataset is an image dataset, and the division of multiple data into multiple groups based on the data load characteristics of the dataset includes: dividing multiple images in the image dataset into multiple groups based on the size features corresponding to multiple images in the image dataset, where the size features include one or more of resolution, aspect ratio, and number of pixels. Since the images can be divided into blocks to determine the corresponding image embeddings (vectors) during the extraction of image features for downstream image processing tasks, if the size features within the same group are the same, the time required to process the same batch of images and the resulting vector length will be similar, thereby fully utilizing computational resources and reducing unnecessary computational overhead.

[0007] In some implementations of the first aspect, dividing multiple images into multiple groups based on the size features corresponding to multiple images in the image dataset includes: determining multiple preset resolutions corresponding to each group; and assigning each image to a corresponding group based on the matching results between each image in the image dataset and the multiple preset resolutions, wherein the resolution of the images within a group is within a preset range of the preset resolutions of the group. By matching the resolutions of the groups with the resolutions of the images, images can be directly and accurately assigned to the corresponding groups, thereby improving the efficiency and reliability of grouping.

[0008] In some implementations of the first aspect, determining the dataset reading order by randomly selecting multiple batches from multiple groups based on a preset batch size includes: determining the multiple batches contained in each group by segmenting the data in each group based on the preset batch size; and determining the dataset reading order by randomly selecting multiple batches from multiple groups. This method avoids the occurrence of specific distribution patterns in the target dataset reconstructed from multiple batches, thus preventing negative impacts on subsequent data processing tasks.

[0009] In some implementations of the first aspect, multiple groups include target groups, and the determination of multiple batches contained in each group by segmenting the data in each group includes: randomly shuffling the order of multiple target data contained in the target group to determine the shuffled target data sequence; and determining multiple target batches contained in the target group by randomly dividing the target data sequence based on a preset batch size, wherein each target batch contains the same number of target data. By randomly shuffling the order of the data in this way, potential order dependencies and data distribution biases between data within a group can be eliminated, thereby preventing the model from learning specific patterns and improving the model's generalization ability and robustness.

[0010] In some implementations of the first aspect, determining the reading order of the dataset by randomly selecting multiple batches from multiple groups includes: after determining the multiple batches contained in each of the multiple groups, sequentially adding corresponding identification information to each batch in each group; determining the adjusted order of each batch by randomly adjusting the order of multiple identification information; and determining the reading order of the dataset based on the adjusted order of each batch. This method can break the grouping characteristics of the dataset globally, thus ensuring load balancing during data processing and ensuring that multiple data sets can be repeatedly distributed, thereby guaranteeing the accuracy of data processing.

[0011] In some implementations of the first aspect, the dataset includes a first data subset and a second data subset. Based on the data load characteristics of the dataset, the multiple data sets are divided into multiple groups, including: dividing the multiple first data sets into multiple groups based on the data load characteristics of the multiple first data sets contained in the first data subset; and dividing the multiple second data sets into multiple groups based on the data load characteristics of the multiple second data sets contained in the second data subset. Each of the multiple groups contains both first and second data sets. This approach ensures the comprehensiveness and integrity of the data contained in the batch, without affecting the accuracy of the data processing task.

[0012] In some implementations of the first aspect, dividing multiple datasets into multiple groups based on their data load characteristics includes: determining multiple clustering results by clustering the dataset based on the data load characteristics corresponding to multiple data points; and determining multiple groups and the multiple data points contained within each group based on the multiple clustering results. In this way, clustering algorithms can better understand the distribution and characteristics of the dataset, enabling the rapid and accurate division of multiple data points into corresponding groups, thus providing useful and accurate information for subsequent data processing tasks.

[0013] In some implementations of the first aspect, the dataset includes multiple first-modal data and multiple second-modal data. Dividing the multiple data into multiple groups based on the data load characteristics of the dataset includes: determining the first data load characteristics corresponding to the multiple first-modal data and the second data load characteristics corresponding to the multiple second-modal data; determining the fused data load characteristics based on the first and second data load characteristics; and dividing the multiple first-modal data and multiple second-modal data into multiple groups based on the fused data load characteristics. In this way, when the dataset is a multimodal dataset, the data can be allocated to corresponding groups by combining the characteristics of multiple modalities, thereby ensuring the accuracy and efficiency of data processing in multimodal scenarios.

[0014] In some implementations of the first aspect, dividing multiple first modal data and multiple second modal data into multiple groups based on fusion modal characteristics includes: combining multiple first modal data and multiple second modal data to determine multiple combined data; and dividing multiple combined data into multiple groups based on fusion data load characteristics.

[0015] In some implementations of the first aspect, the dataset is used to train the target model, and the method further includes: training the target model using each batch in the dataset based on the reading order, until all batches of data have been used to train the target model. By using the dataset to train the target model in this way, processing efficiency can be improved because the data load characteristics contained in the batches used to train the target model are similar. Furthermore, in determining the reading order, the order of multiple data points is randomly shuffled through random selection, which prevents the target model from learning specific patterns, thereby improving the generalization ability and robustness of the trained target model.

[0016] In some implementations of the first aspect, the method further includes: inputting multiple batches of the target dataset into multiple target models in the order of reading; obtaining inference results corresponding to multiple batches after processing by the multiple target models; and using one of the multiple target models to process one batch of data. In this way, during the batch-based data processing task using multiple target models, since the load characteristics of the multiple data points contained in each batch are similar, the time required for any target model to process each data point in that batch is similar, eliminating processing time differences and thus reducing processing time and saving processing resources. Furthermore, using multiple models to process data in batches can improve data processing efficiency and shorten processing time.

[0017] In some implementations of the first aspect, determining the reading order of the dataset includes: randomly shuffling the order of multiple data points contained in each of the multiple groups and randomly shuffling multiple batches of the multiple groups to determine the reading order of the dataset. Based on this method, by randomly shuffling the order of the images and randomly dividing the batches, potential order dependencies and data distribution biases can be eliminated, improving the generalization ability and robustness of the model.

[0018] According to a second aspect of this application, an electronic device is provided, comprising: a processing unit and a memory, wherein the processing unit executes instructions in the memory to cause the electronic device to perform a method, the method comprising: acquiring a dataset containing multiple data, the data including one or more selected from image data, video data, text data, and audio data; dividing the multiple data into multiple groups based on data load characteristics of the dataset; and determining the reading order of the dataset by randomly selecting multiple batches from the multiple groups based on a preset batch size, wherein multiple data contained in the same batch originate from the same group.

[0019] According to a third aspect of this application, an apparatus for determining the reading order of a dataset is provided, comprising: a dataset acquisition unit, a grouping unit, and a reading order determination unit. The dataset acquisition unit is configured to acquire a dataset containing multiple data points. The grouping unit is configured to divide the multiple data points into multiple groups based on the data load characteristics of the dataset. The reading order determination unit is configured to determine the reading order of the dataset by randomly selecting multiple batches from multiple groups based on a preset batch size, wherein multiple data points contained in the same batch originate from the same group.

[0020] According to a fourth aspect of this application, a computer-readable storage medium is provided that stores one or more computer instructions thereon, wherein one or more computer instructions are executed by a processor to cause the processor to perform the method according to a first aspect of this application.

[0021] According to a fifth aspect of this application, a computer program product is provided, including machine-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this application. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0023] Figure 1 shows a schematic diagram of an example environment in which several embodiments of this application may be implemented;

[0024] Figure 2 shows a flowchart of a method for determining the reading order of a dataset according to an embodiment of this application;

[0025] Figure 3 illustrates a schematic diagram of determining a token sequence based on an image block sequence according to an embodiment of this application;

[0026] Figure 4 illustrates a schematic diagram of the process for determining the reading order of a dataset according to an embodiment of this application;

[0027] Figure 5 illustrates a schematic diagram of a process for determining the reading order of a dataset according to some other embodiments of this application;

[0028] Figure 6 illustrates a schematic diagram of determining the order of data reading during model training, provided by some embodiments of this application.

[0029] Figure 7 illustrates a schematic diagram of an application scenario provided by an embodiment of this application; and

[0030] Figure 8 shows a schematic block diagram of an apparatus for determining the reading order of a dataset according to an embodiment of this application. Detailed Implementation

[0031] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.

[0032] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " indicates "and / or," for example, A / B can mean A or B, or A and B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0034] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0035] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0036] The numbers or values ​​used in this specification are illustrative and are intended only to facilitate understanding of the technology of the embodiments of this application, and are by no means intended to limit the scope of this application.

[0037] As mentioned above, the core idea of ​​data parallelism is to divide a large dataset into several smaller subsets and distribute these subsets to different dynamic programming (DP) nodes. Each node runs the same copy of the model but processes a different subset. After each training round, each node summarizes the calculated gradients and updates the model parameters. This way, each node can use the updated model parameters in the next training round, ensuring the model remains consistent across all nodes. However, this approach presents a problem. Taking an image dataset and a ViT (Vision Transformer) model as an example, when using the ViT model to determine the one-dimensional vector (also called a token) corresponding to an image, the processing time will vary between different DP nodes when processing different image subsets due to differences in the number, type, or resolution of the images. Furthermore, the length of the output tokens will also differ. A token can refer to the smallest meaningful unit in the dataset, such as a word, a number, or a Chinese character. This can cause some DP nodes to wait idly; for example, a DP node that has completed its processing task may have to wait for other nodes that have not yet finished. On the other hand, because DP nodes need to be synchronized, different data will be padded to the same length. The greater the difference in sample data length, the more padding is required, resulting in wasted computing power and longer training time. In other words, multiple data processing needs to be synchronized in the same batch. If the data load is uneven, it will lead to a waste of computing resources.

[0038] Therefore, embodiments of this application provide a scheme for determining the reading order of a dataset. This scheme allocates data with different load characteristics to different groups based on the dataset's data load characteristics, ensuring that the data within each group has similar data load characteristics. Furthermore, this scheme can also divide the data within a group into multiple batches based on batch size. By randomly selecting data from multiple batches within multiple groups, the reading order of the dataset can be determined. This results in a randomized order of multiple batches within the dataset, with data within the same batch originating from the same group. Thus, when processing a batch of data, because the data has more balanced similar load characteristics, the time and resources required to process the data in that batch are more consistent, thereby accelerating the training or inference process of the dataset.

[0039] Figure 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of this application may be implemented. The example environment 100 may include electronic devices 102 (e.g., electronic devices 102-1, 102-2, and 102-3) and servers 108 (e.g., servers 108-1, 108-2, and 108-3) communicating with electronic devices 102 via a network. Electronic devices 102 may include, but are not limited to, mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. Servers 108 may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server may also be a server in a distributed system, or a server incorporating blockchain technology. A server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; or it can be an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0040] As shown in Figure 1, electronic device 102 can acquire dataset 104, which may include multiple data sets (e.g., data 1, data 2, data 3… data i). It is understood that the multiple data sets may originate from the same electronic device or from different electronic devices. Data set 104 can be divided into multiple data subsets, which can be input into different target models for training or for inference by multiple identical target models to obtain the output results corresponding to the data subsets. In some embodiments, the data may be image data, video data, audio data, text data, etc. Data set 104 may include only one modality of data, such as only image data, or it may include multiple modalities of data, such as both image data and text data. The text data may be the text corresponding to the image data, for example, the text in the image or text used to describe the content of the image.

[0041] In some embodiments, servers 108 (e.g., servers 108-1, 108-2, and 108-3) may each deploy a target model. For example, each server may run a copy of the same target model, but process different subsets of data or different batches of data. The target model can be a model that needs to be trained or a model that has already been trained. It is understood that the dataset can be a sample dataset used to train the target model or the actual dataset used for inference to obtain results using the target model. The target model is the model corresponding to the data processing task; for example, for an image detection task, the target model can be an image detection model; for an image segmentation task, the target model can be an image segmentation model; for an image classification task, the target model can be an image classification model. During inference, the target model deployed on multiple servers can be the same model, and the target models deployed on different servers can obtain inference results for a batch of images. By distributing the inference tasks of multiple data sets in dataset 104 to multiple servers for execution, the efficiency of inference can be improved and computation time saved.

[0042] In some embodiments, taking dataset 104 as an image dataset as an example, the target model can extract image features to perform more downstream tasks, such as image detection and image classification. During the feature extraction process, the target model can divide the image into blocks to determine the corresponding image block sequence and the token sequence corresponding to the image block sequence. In this process, since the number of image blocks obtained after dividing images of different resolutions or sizes is different, the length of the token sequence is also different. Because synchronization between servers is required, different images (different tokens) will be padded to the same length. The greater the difference in token length between images, the more padding is required, resulting in wasted computing power and longer training time.

[0043] Based on this, by performing real-time reordering on dataset 104, the problem of unbalanced data load during the training or inference process of the target model can be solved. For example, different data can be assigned to different groups based on the data load characteristics corresponding to the data in dataset 104 (e.g., data 1, data 2, data 3... data i). For example, data can be divided into different groups based on the similarity of data load characteristics (e.g., using clustering algorithms such as K-means or DBSCAN). Data with similar load characteristics are assigned to the same group, so the data within the same group are more similar in load. On this basis, based on the batch size, the data within the group can be divided into multiple batches to ensure a balanced distribution of data in each batch. By randomly selecting multiple batches, the reading order 106 of the dataset can be determined (e.g., the reading order of the dataset is batch A, batch B, and batch C). It can be understood that the data load characteristics of multiple data in batch A (e.g., data 1, data 3, data 8, etc.) can be the same, for example, the resolution of data 1, data 3, data 8, etc. can be the same, or the video duration of data 1, data 3, data 8, etc. can be the same. Based on this, batch A can be input to server 108-1 to train the target model deployed in server 108-1, or the target model can output the inference results corresponding to batch A. Similarly, batch B can be input to server 108-2 to train the target model deployed in server 108-2, or the target model can output the inference results corresponding to batch A. Batch C can be input to server 108-3 to train the target model deployed in server 108-3, or the target model can output the inference results corresponding to batch A.

[0044] This approach ensures that the data in batches input to the target model on the server has similar data load characteristics, reducing wasted computational power caused by padding token sequences of varying lengths. Because the data load within batches is more balanced, the time and resources required by the target model to process each batch are more consistent, thus accelerating the entire training or inference process. Furthermore, by optimizing data batching and grouping strategies, computational resources and memory can be utilized more effectively, improving overall resource utilization.

[0045] It should be noted that the electronic device 102 and the server 108 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this. The number of servers and terminal devices is also not limited. The solution provided in this application can be implemented independently by the electronic device 102, independently by the server 108, or jointly by the electronic device 102 and the server 108; this application does not impose any specific limitations on these aspects.

[0046] The following is a flowchart of a method 200 for determining the reading order of a dataset according to an embodiment of this application, with reference to FIG2. It should be noted that the method 200 according to an embodiment of this application can be implemented, for example, at the server 108 shown in FIG2 or at the electronic device 102. It should be understood that the method 200 may also include additional actions not shown, and the scope of this application is not limited in this respect.

[0047] As shown in Figure 2, at step 202, a dataset containing multiple data points is obtained. The data includes one or more of image data, video data, text data, and audio data. The dataset can be a sample dataset used to train the model or an actual dataset to be used for inference. The data can have multiple modalities; the dataset can include data of one modality or two or more modalities. Taking a sample dataset as an example, it can include multiple samples, each with category labels (e.g., labels with unique semantic meaning). Category labels can be set by the user according to actual training needs. For example, in the case of an autonomous driving model and an image sample set, category labels could include "traffic signs," "vehicles," "pedestrians," etc. In one example, the dataset could include 1000 images (e.g., images 1-1000), 1000 videos, or 1000 texts. In some embodiments, each data point in the dataset can contain identification information, which can be information used to uniquely represent the data, such as strings, numbers, letters, etc. For example, the identification information for data in the dataset could be "abcc" or "123," etc.

[0048] In section 204, based on the data load characteristics of the dataset, multiple datasets are divided into multiple groups. Data load characteristics refer to the load on the data being loaded, or the characteristics inherent in the data itself during training. These include data size, distribution, and quality. In some embodiments, data load characteristics can also refer to the processing method of the data when inputting it into the model, such as batch size and data augmentation parameters. Of course, it could also refer to the computational resource requirements of the data during training, such as memory usage and I / O load. Taking data application in model training as an example, data load characteristics can refer to the impact of data on the model and computational resources during training, such as the impact of data size, distribution, quality, and data loading methods on training efficiency and model performance. Data size can include the quantity and dimensionality of the data. Data dimensionality can include feature dimensions (e.g., number of pixels, number of fields) and complexity (e.g., resolution, length of text sequences). Data distribution can refer to whether the proportion of samples from different categories in the dataset is balanced. In some examples, taking an image dataset as an example, data load characteristics can include, but are not limited to, image resolution, category distribution, and batch size. Taking a video dataset as an example, data payload features may include, but are not limited to, video length (duration, number of frames, etc.), resolution, video content, and category. Taking an audio dataset as an example, data payload features may include, but are not limited to, audio sampling rate, audio duration, number of channels, bit depth, and audio category.

[0049] It's understandable that data load characteristics essentially affect the number of tokens corresponding to a given data point. For example, image resolution affects the number of tokens associated with an image, and similarly, image aspect ratio also affects the number of tokens associated with an image. In other words, any factor that can influence the number of tokens associated with a data point can be used as a data load characteristic and as a basis for grouping. This ensures that the number of tokens associated with multiple data points within a group is consistent or within a certain margin of error.

[0050] In some embodiments, the number of groups and the grouping strategy can be set by the user according to actual training or inference needs. Taking image feature parameters as resolution as an example, the user can set three groups, such as Group 1, Group 2, and Group 3. The resolution corresponding to Group 1 is 1920×1080, the resolution of Group 2 can be 2048×1080, and the resolution of Group 3 can be 1280×1706. Based on this, multiple images can be divided into multiple groups according to the matching result between the image resolution and the group resolution. For example, image 1 (resolution of 1280×1706) can be assigned to Group 3, and image 2 (resolution of 1920×1080) can be assigned to Group 1.

[0051] In section 206, based on a preset batch size, the reading order of the dataset is determined by randomly selecting multiple batches from multiple groups. Multiple data points within the same batch originate from the same group. The batch size can be the amount of data used each time the model is processed or trained. The batch size can be set by the user based on hardware resources (e.g., processor memory size), training requirements, or inference requirements. For example, during training, the processor space should be fully utilized, and a larger batch size should be chosen within the upper limit of the processor load. In some embodiments, multiple data points within different groups can be divided based on the batch size. For example, within each group, multiple segments of size batch_size are slidably obtained from the multiple data points contained in that group, using a window size of batch_size. That is, each group is actually a collection of multiple batches, each batch containing a batch_size number of data points. It can be understood that in the process of determining multiple batches, the data within a group can be shuffled to determine the disordered data, so that the data within a group has no specific arrangement order, and the grouping has no specific basis. In this way, different batches within different groups can be obtained. By randomly selecting multiple batches (randomness is both group-wise and batch-wise; for example, the first random selection could be batch 3 from group 1, the second could be batch 16 from group 4, and so on), the reading order of the dataset can be determined, thus ensuring the diversity and randomness of the dataset. For example, the batch order can be determined based on the randomly selected batches. The reading order of the dataset can be determined based on the batch order. The reading order of the dataset can be the order in which multiple batches within the dataset are input to different models. For example, the reading order of the dataset could be batch 3, batch 16, batch 20, batch 1, batch 8… batch 25, etc. Since the load characteristics corresponding to the data within a batch are similar, the processing time required by the target model to process the data within that batch is similar. This reduces waiting time and resource consumption during processing, thereby improving data processing efficiency.

[0052] Based on this approach, by optimizing data batching and grouping strategies, the load characteristics of data within the same batch are made similar. This ensures that the time required by the model to process the same batch of data and the resulting vector length are similar, thereby making full use of computing resources and reducing unnecessary computational overhead. In other words, because the data load within a batch is more balanced, the time and resources required by the target model to process each batch are more consistent, thus accelerating the entire training or inference process. Furthermore, random selection avoids the model's dependence on specific data distributions or orders, thereby enhancing the model's generalization ability.

[0053] Figure 3 illustrates a schematic diagram of determining a token sequence based on an image patch sequence according to an embodiment of this application. As shown in Figure 3, taking the ViT model as an example, the ViT model can segment an image into small patches and use them as an input sequence when processing an image. Each image patch can be converted into a vector through a series of transformations. For example, the input image 302 can be 224×224 in size, and the image 302 can be divided into fixed-size image patches, with each patch being 16×16. Each image can generate (224×224) / (16×16) = 196 patches, meaning the input sequence length is 196. Each patch has a dimension of 16×16×3 = 768. Each image patch is input to a linear projection 304, which converts the image patch into a vector form that the model can process. For example, after projection through the linear projection 304 layer, 196 vectors of 768 dimensions can be obtained, which constitute the model's input sequence tokens. The length of this sequence is 196, corresponding to the number of patches in the image. In this process, tokens can be positionally encoded to provide location information for each image patch in the sequence. For example, sine and cosine functions can be used to encode positions, thus introducing temporal and spatial structure into the sequence. It can be understood that the introduction of positional encoding helps the model understand the local and global contextual relationships within the image.

[0054] As shown in Figure 3, in the ViT model, the image size or resolution affects the token length. The token length (i.e., the length of the input sequence) directly depends on the image size and the patch size. If the image size increases or the patch size decreases, the number of generated patches increases, leading to an increase in the length of the input sequence. Conversely, if the image size decreases or the patch size increases, the number of generated patches decreases, and the length of the input sequence decreases accordingly. In data-parallel systems, to improve processing efficiency, the dataset is typically divided into multiple batches, and each batch is assigned to different computing nodes or GPUs for parallel processing. However, since different images may have different resolutions, the lengths of the generated token sequences will also be inconsistent. This can introduce synchronization problems in parallel processing, as different computing nodes may need to wait for each other to complete the processing of token sequences of different lengths. For example, to ensure synchronization between images, tokens of different lengths need to be padded to the same length. Based on this, the dataset can be divided into multiple groups based on image resolution. Specifically, the dataset can be divided into multiple groups according to the image resolution, with images in each group having similar resolutions. Then, these groups are divided into different batches. Because images in the same batch have similar resolutions, the generated token sequences are also of similar length. This can reduce synchronization issues in parallel processing to some extent.

[0055] In some embodiments, taking an image dataset as an example, multiple images in the image dataset can be grouped based on the resolution of the image dataset. Before grouping, a minimum and / or maximum size can be set according to the characteristics of the image dataset. For example, the maximum resolution of the image can be set (e.g., setting max(H,W) = 512) or the maximum width (width less than 512) and maximum height (height less than 512) of the image can be set. Alternatively, the minimum width and minimum height of the image can be set (width, height not less than 28). Based on this, the image can be scaled according to preset rules to control the aspect ratio of the image. During the adjustment of the image aspect ratio, it is necessary to maintain the visual quality and information integrity of the image. For example, if the width / height of the image is greater than 200, the width and height of the image can be determined by enlarging the width of the image (width,height = height × 180,height); if the height / width of the image is greater than 200, the width and height of the image can be determined by enlarging the height of the image (width,height = width,width × 180).

[0056] In some embodiments, to ensure the accuracy and efficiency of segmentation, the pixel values ​​of an image (e.g., the width and height) can be adjusted to integer multiples of a set value. For example, the adjusted height can be determined using `Resize_h = round(height / factor) × factor`, where `factor` is a value set by the user according to actual needs. The `round()` function can be used to round numbers to a specified precision. Similarly, the adjusted width can be determined using `Resize_w = round(width / factor) × factor`. Based on this, the number of tokens corresponding to the image can be determined according to the adjusted width and height. For example, the number of tokens in the adjusted image can be determined using `token = (H_resized / 14) * (W_resized / 14)`. Based on the calculated number of tokens, the images can be assigned to different groups. For example, a series of token number intervals can be preset, and then each image can be assigned to the corresponding interval group according to its token number. In some examples, multiple buckets can be created, each representing a range of tokens, such as [0-100], [101-200], [201-300], etc., and then images can be assigned to the corresponding buckets.

[0057] In other embodiments, the image dataset can also be divided based on the aspect ratio of the images. For example, the number of segments corresponding to an image can be determined based on its aspect ratio. After determining the number of segments, the image can be segmented. The number of segments can, to some extent, represent the number of tokens, so images can be assigned to corresponding groups or buckets based on the number of segments. It is understood that, to ensure all images (including original images and segmented images) have the same size for subsequent processing, all images and image blocks can be resized to a fixed target size (e.g., 224×224, 448×448, etc.). For example, the resize function in an image processing library can be used to resize all images to the target size.

[0058] In some embodiments, after dividing the dataset into multiple subsets based on image resolution, to prevent the model from learning specific patterns, the images within each group can be randomly shuffled, making the image order unpredictable. Specifically, randomly shuffling images within a group can break any potential order dependencies, preventing the model from using these relationships to improve performance. Furthermore, random shuffling ensures that the model can learn more general feature representations when processing images at different resolutions, rather than relying solely on features specific to a particular resolution. In other words, by randomly shuffling the image order and randomly partitioning batches, potential order dependencies and data distribution biases can be eliminated, improving the model's generalization ability and robustness. In some embodiments, random shuffling can be applied to the data within a group or to all groups.

[0059] In some embodiments, clustering algorithms can be used to divide the dataset into several clusters based on size features (e.g., resolution), meaning that images belonging to the same cluster are more similar in size. During the clustering process, the clustering algorithm iteratively updates the cluster centers until a convergence condition is met (e.g., the cluster centers stop updating or a preset number of iterations is reached). Each cluster has one cluster center. Based on this, the subset of data corresponding to the same cluster can be grouped together, with the data in the group having the same size (e.g., the same resolution). The clustering algorithm can include, but is not limited to, k-means clustering, density-based spatial clustering of applications with noise (DBSCAN), ordering points to identify the clustering structure (OPTICS), hierarchical clustering, etc., and is not limited here.

[0060] In this way, clustering algorithms can be used to better understand the distribution and characteristics of a dataset, thereby quickly and accurately dividing multiple data into corresponding groups, providing useful and accurate information for subsequent data processing tasks.

[0061] In some embodiments, the dataset input to the target model can also be a video dataset. A video can be viewed as a collection of consecutive frames, each frame essentially being a still image. By decoding the video frame by frame, image sequences and corresponding audio sequences can be obtained. These image sequences can be used to train various computer vision tasks such as image classification, object detection, and action recognition. Furthermore, the video dataset can be split into an image dataset and a corresponding audio dataset. In other words, the video dataset can be divided into an image dataset and an audio dataset, and then grouped separately for each.

[0062] In other embodiments, video datasets can also be grouped based on their own data load characteristics. These data load characteristics may include video resolution, video frame rate, total video duration, etc. In other embodiments, the data load characteristics may also include the video resolution and the adjusted sampling frame rate according to preset rules, and of course, the semantic content of the video, such as the video category (dynamic, static, etc.). In some embodiments, a video dataset may include multiple videos (e.g., Video 1, Video 2, Video 3…Video 100, etc.), and based on the data load characteristics corresponding to the multiple videos, these videos can be assigned to different groups. It can be understood that Video 1 can be a video obtained by sampling the acquired video, for example, it can be composed of 10 consecutive or non-consecutive video frames selected from the acquired video. The number of samples can be set by the user according to actual needs, for example, it can be determined based on the semantic content of the video. For example, if the acquired video is dynamic, the number of samples can be larger, for example, greater than a set value; if the acquired video is partially static, the number of samples can be smaller, for example, less than a set value.

[0063] In some embodiments, when the dataset is an audio dataset, to reduce the computational waste caused by padding or truncation during model training, different audio data can be divided into multiple groups based on the data load characteristics of the audio dataset. The data load characteristics of the audio data may include, but are not limited to, audio sampling rate (e.g., the audio sampling rate can refer to the number of times an analog audio signal is sampled per second), duration, number of channels, bit depth, etc. The bit depth of the audio data refers to the number of digital bits used to quantize each sample point, usually measured in bits. Bit depth determines the dynamic range and fidelity of the audio signal. For example, bit depth can be 16-bit, 24-bit, and 32-bit. The number of audio channels refers to the number of independent audio signals collected or played back at different spatial locations during recording or playback. The number of channels determines the number of sound sources during recording or the number of speakers required for playback. For example, the number of channels can be mono, stereo, surround sound, etc.

[0064] In some embodiments, audio data can be binned according to duration (e.g., 0-5s, 5-10s, 10-20s), with audio data within the same bin having similar durations. Based on this, audio data from different bins can be input into audio models on multiple nodes for training. Alternatively, multiple nodes can process audio data within different groups, executing audio processing tasks in parallel. In this way, because the durations of audio data within the same group are relatively similar, the processing time required by the node's audio model when processing multiple audio data within that group is similar, thus avoiding wasted computational power due to truncation or padding and ensuring balanced data load. In other embodiments, semantic parsing of the audio data can determine its category (e.g., music, human voice, ambient sound, background sound, etc.). Based on this, multiple audio data can be assigned to multiple groups according to their category. For example, audio data containing human voice can be assigned to the "human voice" bin.

[0065] In some embodiments, the dataset can also be a text dataset. A text dataset can include multiple texts, and the lengths of the different texts can be the same or different. To ensure that the model requires similar processing time for each batch of text in the text dataset, guaranteeing a balanced data load, multiple texts in the text dataset can be grouped based on the data load characteristics of the text dataset (e.g., text length, text type, etc.). For example, multiple texts with similar lengths can be assigned to the same group. In some embodiments, the text can be converted into a token sequence, and the text length corresponding to the token sequence can be determined. For example, in the case of the text "hello,world!this is a test.", the corresponding token sequence can be "['hello',',','world','!','this','is','a','test','.']". It can be seen that the text length corresponding to this text is token 9. The text lengths corresponding to other texts can be determined in the same way. In some embodiments, the text lengths corresponding to multiple text buckets can be set by the user; for example, the text length corresponding to bucket 1 can be 0-10, the text length corresponding to bucket 2 can be 11-20, the text length corresponding to bucket 3 can be 21-30, etc. Based on this, the text can be assigned to the corresponding bucket according to its length. For example, the text "hello,world!this is a test." can be assigned to bucket 1.

[0066] In some embodiments, the text dataset can be preprocessed before grouping to improve grouping efficiency and accuracy. Preprocessing may include data cleaning (e.g., removing useless characters, HTML tags, etc.), duplicate text removal, and word segmentation. For example, cleaning the text dataset can remove unnecessary whitespace characters such as spaces, tabs, and line breaks, or remove punctuation marks and special characters. In some embodiments, hash algorithms or text similarity calculations can be used to identify and remove duplicate text. This helps reduce data redundancy and improve grouping efficiency.

[0067] In practical applications, text lengths vary. Excessively long texts can lead to inefficient processing and even data overflow; conversely, excessively short texts may lack sufficient information to support subsequent analysis or model training. To avoid these problems, reasonable upper and lower limits for text length can be set based on the specific application requirements and the characteristics of the dataset. For example, texts exceeding a certain length or falling short of another set length can be filtered from the dataset. In some examples, a shorter lower limit (e.g., 5 characters) can be set to avoid overly short texts, while an upper limit (e.g., 500 characters) can be set to avoid the computational burden of processing excessively long texts.

[0068] Exemplary, with reference to FIG4, a process 400 for determining the reading order of a dataset provided in an embodiment of this application will be described. FIG4 shows a schematic diagram of a process 400 for determining the reading order of a dataset provided in an embodiment of this application. The example process 400 may be an example implementation of method 200, and is executed, for example, by server 108 as shown in FIG1. ​​It should be understood that the example process 400 may also include additional actions not shown, or some of the actions may be omitted. Furthermore, the order of actions shown in the example process 400 is only an example, and in some other embodiments, the execution order of actions may be changed without departing from the scope of this application.

[0069] Figure 4 illustrates the process using image dataset 402 as an example to train a target model (which can be a visual encoding model). In determining the reading order of the image dataset, images of different resolutions can be allocated to different buckets (groups) based on their resolution. Image resolution refers to the number or density of pixels stored in an image, typically used to measure image sharpness and detail. Image resolution is directly related to the image's height and width. Both height and width values ​​can be expressed in pixels. For example, an image with a resolution of 7000×5000 means it has a width of 7000 pixels, a height of 5000 pixels, and a total of 35,000,000 pixels (35 megapixels).

[0070] In some embodiments, different buckets or different groups correspond to different resolutions. For example, the resolution of bucket 1 (B1) can be 1×1, the resolution of bucket 2 (B2) can be 1×2, the resolution of bucket 3 (B3) can be 1×3, the resolution of bucket 4 (B4) can be 1×4, the resolution of bucket 5 (B5) can be 2×2, the resolution of bucket 6 (B6) can be 1×5, the resolution of bucket 7 (B7) can be 1×6, and the resolution of bucket 8 (B8) can be 2×3. It should be noted that the resolutions of different buckets here are only examples and do not constitute a limitation of this application. It can be understood that the resolution here refers to the blocks contained in the image, and the size of a block can be 448×448. For example, the resolution of bucket 8 can be containing 2×3 image blocks (the size of the image blocks is 448×448).

[0071] In practical applications, images can have various resolutions. To ensure that images of different resolutions can be assigned to different preset buckets, an image can be assigned to a preset bucket if its resolution falls within a certain range of that bucket's resolution. For example, different resolution ranges can be set for different groups, such as a "low resolution" bucket (e.g., 320x240 to 640x480), a "medium resolution" bucket (e.g., 641x481 to 1280x960), and a "high resolution" bucket (e.g., 1281x961 and higher). It's understood that the resolution boundaries of the buckets can be adjusted according to actual needs. During this adjustment, it's crucial to ensure that the resolution ranges between buckets do not overlap to avoid assignment conflicts. Based on this, if image 1 has a resolution of 800x600, this resolution falls within the range of the "medium resolution" bucket (641x481 to 1280x960), and image 1 can be assigned to the "medium resolution" bucket. In this way, images of different resolutions can be assigned to corresponding buckets, thus flexibly adapting to different application scenarios and resolution requirements.

[0072] In some embodiments, image dataset 402 may contain an index sequence. The index sequence can be a sequence of multiple index values, where different index values ​​can be used to uniquely identify different images, and the index values ​​can indicate which image dataset an image originates from. The index sequence allows for quick access, retrieval, and processing of images, facilitating rapid location of specific images during subsequent processing or analysis. In some embodiments, the index sequence can be a numerical index, for example, using consecutive integers or specific numerical identifiers to identify each image. In some examples, the index value of image 1 in image dataset 402 can be index1, the index value of image 2 can be index2, and the index value of image k can be index k. It is understood that, to further characterize the source of an image, a composite index can be added to the image; for example, the index value of image 1 can be index1 + sample1, where sample1 can be the identification information of image dataset 402, used to indicate that image 1 originates from image dataset 402.

[0073] As shown in Figure 4, based on the resolution of each image in image dataset 402, each image can be assigned to a corresponding bucket. For example, the image assigned to bucket 1 can be a subset of images 404. It can be understood that the resolution of each image in subset 404 is the same as or within a certain range of the resolution of bucket 1. For example, subset 404 can contain 100 images with a resolution of 1×1 or close to 1×1. Since the images are input to the target model in batches, the images within a bucket (e.g., subset 404 in bucket 1) can be divided into multiple batches (also called batch-pkg) based on the batch size. In machine learning or deep learning model training, batch size is an important hyperparameter, determining the number of images processed simultaneously during each training session. For example, images within a bucket can be grouped into batch-pkgs according to their batch size using a sliding window. The window size can be determined based on the batch size. For example, the sliding window can start from the first image in bucket 1 and move forward one image at a time until all images in the bucket have been grouped. Each time you slide, the images within the window are treated as a batch (batch-pkg). This process is repeated until all images in the bucket have been divided into different batches. For example, the images in bucket 1 can be divided into three batches: batch 406 (batch 1), batch 408 (batch 2), and batch 410 (batch 4).

[0074] In some embodiments, different index values ​​can be assigned to batches. For example, the index value of batch 406 can be batch_index1, the index value of batch 408 can be batch_index2, and the index value of batch 410 can be batch_index4. It is understood that the remaining buckets can also be divided into multiple batches by dividing the images they contain in the manner described above. To facilitate the determination of the subsequent image dataset reading order, batches within each bucket can be identified using consecutive integers or specific numerical identifiers. For example, bucket 2 can contain batch 4 (corresponding index value of batch_index4), batch 5 (corresponding index value of batch_index5), and batch 6 (corresponding index value of batch_index6). Similarly, bucket 8 can contain batch 21 (corresponding index value of batch_index21), batch 22 (corresponding index value of batch_index22), batch 23 (corresponding index value of batch_index23), and batch 24 (corresponding index value of batch_index24). It should be noted that the naming method of the index values ​​can be flexibly adjusted to adapt to different application scenarios and needs. For example, a simpler naming convention (such as b1, b2, etc.) can be used to reduce storage space usage.

[0075] It's important to note that while the batch size is fixed, the number of images contained in image subset 404 (i.e., the number of images in bucket 1) is random. The number of images in a bucket and the batch size are often not perfectly integer multiples of each other. This can lead to situations where the number of images in bucket 1 is not divisible by the batch size (e.g., some images are not assigned to their corresponding batches, or the last batch contains fewer images than the batch size). In such cases, the remaining images can be used as training data for the next iteration and can be directly input into the target model for training without affecting the accuracy of the target model's training.

[0076] It's understandable that during the training of machine learning or deep learning models, if the data exhibits a specific order or pattern, the model might learn this pattern rather than the true features, thus affecting its generalization ability. Therefore, to prevent the target model from learning specific patterns during training and impacting its generalization ability, the images within the bucket can be randomly shuffled, making the image order unpredictable. For example, a shuffle algorithm (such as a hash-based shuffle algorithm or a sorting-based shuffle algorithm) can be used to randomly shuffle the images within the bucket. The shuffle algorithm randomly shuffles a set of elements (e.g., shuffling an ordered (or unordered) set of elements), similar to the random arrangement of cards during shuffling. This ensures the image order is random, eliminating any potential order-related influences. In some examples, shuffling functions in programming languages ​​(such as `random.shuffle()` in Python) can be used to randomly sort the list of images within the bucket.

[0077] It should be noted that the images within the bucket can be divided into different batches first, and then the images within the bucket as a whole can be shuffled. Alternatively, the images within the bucket can be shuffled first, and then the shuffled images can be divided into different batches; this application does not limit this approach.

[0078] As shown in Figure 4, after determining multiple batches, the reading order of the dataset is determined by randomly selecting multiple batches (e.g., by shuffling multiple batches), resulting in the image dataset reading order 412 (e.g., the batches included in the image dataset reading order 412 correspond to the order of batch 14, batch 1, batch 21, batch 15... batch 4, etc.). For example, the order of multiple batches can be determined using a random function (such as random.shuffle() in Python). It can be understood that images from different batches may originate from different buckets, but data from the same batch originates from the same bucket. This ensures that the data processed by the models on the nodes is similar, and the generated token lengths are also similar. In some embodiments, based on the image dataset reading order 412, these batches can be assigned to different DP nodes (e.g., batch 14 can be assigned to DP node 1, batch 1 to DP node 2, batch 21 to DP node 3, and batch 15 to DP node 4, etc.). Each node runs the same copy of the target model but processes different subsets of data. After each round of training, each node summarizes the calculated gradients and updates the model parameters. In this way, each node can use the updated model parameters in the next training round (ensuring that the entire model remains consistent across all nodes) until the predetermined number of training rounds is reached or other stopping conditions are met. This approach leverages data parallelism to improve the training speed of the target model and reduce training time.

[0079] In other words, the purpose of the shuffle operation is to ensure that the model does not see the same data order repeatedly during training. This helps the model learn the overall distribution of the data, rather than simply memorizing patterns in specific data sequences. There are two shuffle processes during dataset reconstruction: one shuffle within each group's batches, which helps break the continuity of data within each group and increases data diversity; and another shuffle across the buckets (groups), which helps break the grouping characteristics of the data globally, ensuring the model can learn the differences between different groups. This approach guarantees both balanced data load within each image patch and repeated data shuffling, thus ensuring training accuracy.

[0080] In some embodiments, the images in the image dataset may originate from multiple devices; that is, the image dataset may contain multiple subsets of data from different devices. Alternatively, when training a target model using the image dataset, training time can be reduced by sharding the dataset and processing it in parallel across multiple computing resources. Sharding refers to dividing the dataset into multiple subsets or segments. Given data sharding, during the reconstruction of the dataset, it is necessary to ensure that a batch contains images from each subset, or images from all devices. This guarantees the comprehensiveness and integrity of the training data and does not affect the training accuracy of the model.

[0081] For example, FIG5 illustrates a schematic diagram of a process 500 for determining the reading order of an image dataset according to other embodiments of this application. Example process 500 may be an example implementation of method 200 and is executed, for example, by server 108 as shown in FIG1. ​​It should be understood that example process 500 may also include additional actions not shown, or some of these actions may be omitted. Furthermore, the order of actions shown in example process 500 is merely illustrative; in other embodiments, the execution order of actions may be changed without departing from the scope of this application.

[0082] As shown in Figure 5, the image dataset 502 can be divided into multiple slices, namely slice A, slice B, and slice C. Slices A, B, and C each contain multiple images. Based on this, the images in slices A, B, and C can be further divided into multiple buckets (e.g., bucket 1, bucket 2, bucket 3, bucket 4, bucket 5, bucket 6, bucket 7, bucket 8) according to their resolution. For example, bucket 1 can include images from slices A, B, and C. Furthermore, based on the batch size, the images within each bucket can be divided into multiple batches (e.g., batch 504, batch 506, and batch 508). It is understood that a batch must contain data from all slices. For example, batch 504 can include images from slice A, slice B, and slice C. It should be noted that within each batch, images from different slices should be randomly selected to ensure data randomness and diversity. After determining multiple batches, the reading order of the image dataset can be determined by the two Shuffle operations described in the above embodiments 510, which will not be elaborated here.

[0083] Figure 6 illustrates a schematic diagram of determining the input order of the dataset during model training, provided by some embodiments of this application. As shown in Figure 6, at 602, the dataset (Dataset) can be initialized. The dataset can be an abstract class representing a collection of data. It is understood that the model is data-driven, and its success depends on the support of massive amounts of data. The essence of machine learning models lies in their ability to continuously summarize patterns and extract experience from data, thereby adjusting their parameters. Therefore, the dataset is crucial for model training. The initialization of the dataset can include defining a data list, dictionary, or other data structures. At 604, after initializing an empty list to store data samples, samples can be added to the dataset by reading a JSON file or a JSONL file. A JSON file has a tree-like nested structure, where the entire file is a complete JSON object. A JSONL file is a lightweight text file format where each line is a valid JSON object, with no nesting relationships between them. For example, a JSONL file can be opened and read line by line, parsing the content of each line to determine the JSON object and adding the parsed sample to the dataset.

[0084] In version 606, after acquiring the dataset, a sample list (SampleList) can be constructed. The sample list can be a list containing multiple data samples. Each sample in the sample list is a dictionary or tuple, and the sample can contain the input data (image data) and its corresponding label. For example, a sample is a dictionary with some keys. In SampleList, the values ​​of these keys are cleverly grouped according to whether it is a tensor or a list and assigned back to the dictionary. In version 608, to prevent the model from learning specific patterns during training, the sample list can be reordered after construction. For example, the order of SampleList can be shuffled by random sorting, sorting by label, or sorting in a specific order. For instance, Python's `random.shuffle` function can be used to randomly shuffle the order of SampleList.

[0085] In version 610, a data loader can be created. The data loader is used to process data during training. Essentially, a data loader is a tool for batch loading data; it divides the dataset into batches and provides data loading functionality. Using a data loader, the entire dataset can be easily iterated over, and data can be loaded in batches during training. It's worth noting that the data loader supports batch processing and data shuffling; the size of each batch can be specified by setting the batch size during data loader creation.

[0086] In section 612, relevant parameters in the target training model can be set, such as setting the model's network parameters, determining the dataset required for training, and setting training-related information (the name of the algorithm used, the number of training iterations, the training time, and the model's performance metrics). For example, in section 614, since the quality of the model's initialization parameters can affect the speed and effectiveness of network convergence, the model's parameters can be initialized to avoid problems such as vanishing or exploding gradients, thereby improving the efficiency of model training. Parameter initialization refers to the process of setting initial values ​​for the weights and biases in the model before training begins. For example, the model's parameters can be initialized using methods such as all-zero initialization (e.g., initializing the model's weight parameters to 0 and the bias terms to 0), random initialization (randomly initializing the weights to small random numbers), fixed-value initialization (e.g., using fixed values ​​as the initial values ​​for the model's parameters), and gradient-based initialization (e.g., He initialization, Xavier initialization, etc.). In section 616, the dataset reconstruction scheme provided in this application embodiment can be used to perform real-time rearrangement of the above dataset to solve the problem of unbalanced data load during model training.

[0087] In section 618, iterative steps (Do-steps) can be set during the training process. In each Do-step, the model processes a batch of data, calculates the loss, and updates the parameters. It can be understood that before each training iteration (epoch) begins, the data in the dataset can be randomly shuffled using the scheme provided in this application embodiment until the predetermined number of training iterations is reached.

[0088] Figure 7 illustrates an application scenario provided by an embodiment of this application. As shown in Figure 7, in a multimodal scenario, the input to the language model can be data of various modalities, such as image data, video data, text data, and audio data. Different data correspond to different tokens; that is, the tokens input to the language model are diverse, such as text tokens, image tokens, speech tokens, and video tokens. Before using a feature extraction module (e.g., the ViT model) to determine the token corresponding to an image, the method for determining the reading order of the dataset provided in this application embodiment can be used to determine the batch for input to the feature extraction module. This ensures that the images contained within a batch have the same or similar resolution. Since the ViT model divides the image into fixed-size blocks when determining the features corresponding to the image, when the image resolution is consistent, the number of blocks divided into each image will be the same. This means that the token length converted from each image will also be the same. In other words, the length of the image tokens input to the language model is consistent. In a multimodal scenario, maintaining consistent image token lengths makes the fusion of multimodal data smoother. Furthermore, fixed-length inputs can more effectively utilize computing resources, such as the parallel computing capabilities of GPUs. When the input sequence length is consistent, batch processing can be more easily implemented, thereby improving computational efficiency.

[0089] Figure 8 shows a schematic block diagram of an apparatus 800 for determining the reading order of a dataset according to an embodiment of this application. The apparatus 800 for determining the reading order of a dataset can be implemented in an electronic device 102 or a server 108. The apparatus 800 for determining the reading order of a dataset may include a dataset acquisition unit 810, a grouping unit 820, and a reading order determination unit 830. The dataset acquisition unit 810 is configured to acquire a dataset containing multiple data sets. The grouping unit is configured to divide the multiple data sets into multiple groups based on the data load characteristics of the dataset. The reading order determination unit is configured to determine the reading order of the dataset by randomly selecting multiple batches from multiple groups based on a preset batch size, wherein multiple images contained in the same batch originate from the same group.

[0090] In some embodiments, where the dataset is a video dataset, the grouping unit 820 is further configured to divide the multiple videos in the video dataset into multiple groups based on the video duration and / or semantic content corresponding to the multiple videos in the video dataset.

[0091] In some embodiments, where the dataset is an image dataset, the grouping unit 820 is further configured to: divide the multiple images in the image dataset into multiple groups based on the size features corresponding to the multiple images in the image dataset, wherein the size features include one or more of resolution, aspect ratio, and number of pixels.

[0092] In some embodiments, the grouping unit 820 is further configured to: determine multiple preset resolutions corresponding to multiple groups respectively; and assign each image to a corresponding group based on the matching result of each image in the image dataset with the multiple preset resolutions, wherein the resolution of the image in the group is within a preset range of the preset resolution of the group.

[0093] In some embodiments, the read order determination unit 830 is further configured to: determine multiple batches contained in each group by segmenting the data in each group based on a preset batch size; and determine the read order of the dataset by randomly selecting multiple batches from multiple groups.

[0094] In some embodiments, where the plurality of groups include target groups, the reading order determination unit 830 is further configured to: determine a shuffled target data sequence by randomly shuffling the order of the plurality of target data contained in the target group; and determine a plurality of target batches contained in the target group by randomly dividing the target data sequence based on a preset batch size, wherein each target batch in the plurality of target batches contains the same amount of target data.

[0095] In some embodiments, the read order determination unit 830 is further configured to: after determining the multiple batches contained in each of the multiple groups, sequentially add corresponding identification information to each batch in each group; determine the order of each batch after adjustment by randomly adjusting the order of the multiple identification information; and determine the read order of the dataset based on the order of each batch after adjustment.

[0096] In some embodiments, wherein the dataset includes a first data subset and a second data subset, the grouping unit 820 is further configured to: divide the multiple first data into multiple groups based on the data load characteristics of the multiple first data contained in the first data subset; and divide the multiple second data into multiple groups based on the data load characteristics of the multiple second data contained in the second data subset, wherein each of the multiple groups contains the first data and the second data.

[0097] In some embodiments, the grouping unit 820 is further configured to: determine multiple clustering results by clustering the dataset based on the data load characteristics corresponding to multiple data in the dataset; and determine multiple groups and multiple data contained in the multiple groups based on the multiple clustering results.

[0098] In some embodiments, the dataset includes multiple first modal data and multiple second modal data, and the grouping unit 820 is further configured to: determine a first data load feature corresponding to the multiple first modal data and a second data load feature corresponding to the multiple second modal data; determine a fused data load feature based on the first data load feature and the second data load feature; and divide the multiple first modal data and the multiple second modal data into multiple groups based on the fused data load feature.

[0099] In some embodiments, the grouping unit 820 is further configured to: combine multiple first modal data and multiple second modal data to determine multiple combined data; and divide the multiple combined data into multiple groups based on the characteristics of the fused data load.

[0100] In some embodiments, the dataset is used to train the target model, and the apparatus 800 further includes: training the target model using each batch in the dataset based on the reading order, until all batches of data have been used to train the target model.

[0101] In some embodiments, the apparatus 800 further includes: inputting multiple batches of the target dataset into multiple target models in the order of reading, obtaining inference results corresponding to multiple batches after processing by the multiple target models, wherein one of the multiple target models is used to process the data of one batch.

[0102] In some embodiments, the read order determination unit 830 is further configured to determine the read order of the dataset by randomly shuffling the order of multiple data contained in each of the multiple groups and randomly shuffling multiple batches of the multiple groups.

[0103] This application may be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this application.

[0104] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0105] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0106] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0107] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0111] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining the reading order of a dataset, comprising: Obtain a dataset containing multiple data types, including one or more of image data, video data, text data, and audio data; Based on the data load characteristics of the dataset, the multiple data points are divided into multiple groups; as well as Based on a preset batch size, the reading order of the dataset is determined by randomly selecting multiple batches from the multiple groups, and multiple data in the same batch come from the same group.

2. The method according to claim 1, wherein the dataset is a video dataset, and dividing the plurality of data into multiple groups based on the data load characteristics of the dataset includes: Based on the video duration and / or semantic content of multiple videos in the video dataset, the multiple videos in the video dataset are divided into multiple groups.

3. The method according to claim 1, wherein the dataset is an image dataset, and dividing the plurality of data into multiple groups based on the data load characteristics of the dataset includes: Based on the size features corresponding to multiple images in the image dataset, the multiple images in the image dataset are divided into multiple groups. The size features include one or more of the following: resolution, aspect ratio, and number of pixels.

4. The method according to claim 3, wherein dividing the plurality of images into a plurality of groups based on the resolutions corresponding to the plurality of images in the image dataset includes: Determine the multiple preset resolutions corresponding to the multiple groups respectively; as well as Based on the matching results of each image in the image dataset with the multiple preset resolutions, each image is assigned to a corresponding group, and the resolution of the images in the group is within a preset range of the preset resolution of the group.

5. The method according to claim 1, wherein determining the reading order of the dataset by randomly selecting multiple batches from the plurality of groups based on a preset batch size includes: Based on a preset batch size, the multiple batches contained in each group are determined by randomly segmenting the data in each of the multiple groups; as well as The reading order of the dataset is determined by randomly selecting multiple batches from the multiple groups.

6. The method of claim 5, wherein the plurality of groups includes target groups, and determining the plurality of batches contained in each group by segmenting the data in each of the plurality of groups comprises: By randomly shuffling the order of multiple target data contained in the target group, a shuffled target data sequence is determined; Based on the preset batch size, multiple target batches are determined by randomly dividing the target data sequence, and each of the multiple target batches contains the same amount of data.

7. The method according to claim 5, wherein determining the reading order of the dataset by randomly selecting multiple batches from the plurality of groups comprises: After determining the multiple batches contained in each of the multiple groups, corresponding identification information is added sequentially for each batch in each group; The order of each batch after adjustment is determined by randomly rearranging the order of multiple identification information items; as well as The order in which the dataset is read is determined based on the adjusted order of each batch.

8. The method of claim 1, wherein the dataset comprises a first data subset and a second data subset, and dividing the plurality of data into multiple groups based on the data load characteristics of the dataset includes: Based on the data load characteristics of the multiple first data contained in the first data subset, the multiple first data are divided into multiple groups; as well as Based on the data load characteristics of the multiple second data contained in the second data subset, the multiple second data are divided into multiple groups, and each of the multiple groups contains first data and second data.

9. The method of claim 1, wherein dividing the plurality into multiple groups based on the data load characteristics of the dataset comprises: Based on the data load characteristics corresponding to multiple data in the dataset, multiple clustering results are determined by clustering the dataset. as well as Based on the multiple clustering results, the multiple groups and the multiple data contained in the multiple groups are determined.

10. The method of claim 1, wherein the dataset comprises a plurality of first modality data and a plurality of second modality data, and dividing the plurality of data into a plurality of groups based on the data load characteristics of the dataset comprises: Determine the first data load characteristics corresponding to the plurality of first modal data and the second data load characteristics corresponding to the plurality of second modal data; Based on the first data load characteristics and the second data load characteristics, the fused data load characteristics are determined; as well as Based on the fused data load characteristics, the multiple first modal data and the multiple second modal data are divided into the multiple groups.

11. The method of claim 10, wherein dividing the plurality of first modal data and the plurality of second modal data into the plurality of groups based on the fused modal features comprises: The plurality of first modal data and the plurality of second modal data are combined to determine a plurality of combined data; as well as Based on the characteristics of the fused data load, the multiple combined data are divided into multiple groups.

12. The method of claim 1, wherein the dataset is used to train the target model, and the method further comprises: Based on the reading order, the target model is trained using each batch in the target dataset until all batches of data have been used to train the target model.

13. The method according to claim 1, further comprising: Multiple batches in the target dataset are input into multiple target models in the order of reading. After processing by the multiple target models, the inference results corresponding to the multiple batches are obtained. One of the multiple target models is used to process the data of one batch.

14. The method of claim 1, wherein determining the reading order of the dataset further comprises: The reading order of the dataset is determined by randomly shuffling the order of multiple data items contained in each of the multiple groups and by randomly shuffling multiple batches of the multiple groups.

15. An electronic device comprising: Processing unit and memory, The processing unit executes instructions in the memory, causing the electronic device to perform a method, the method comprising: Obtain a dataset containing multiple data types, including one or more of image data, video data, text data, and audio data; Based on the data load characteristics of the dataset, the multiple data points are divided into multiple groups; and Based on a preset batch size, the reading order of the dataset is determined by randomly selecting multiple batches from the multiple groups, and multiple data in the same batch come from the same group.

16. An apparatus for determining the reading order of a dataset, comprising: The dataset acquisition unit is configured to acquire multiple datasets, the data including one or more of image data, video data, text data, and audio data; A grouping unit is configured to divide the multiple data into multiple groups based on the data load characteristics of the dataset. as well as The reading order determination unit is configured to determine the reading order of the dataset by randomly selecting multiple batches from the multiple groups based on a preset batch size, wherein multiple data in the same batch come from the same group.

17. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to cause the processor to perform the method according to any one of claims 1 to 14.

18. A computer program product comprising machine-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 14.

Images