Cloud management platform-based data acquisition method and cloud management platform

Abstract

The present application discloses a cloud management platform-based data acquisition method and a cloud management platform, which can improve user experience. The method of the present application comprises: a storage node created by a cloud management platform for a user stores a tensor of the user, the tensor may comprise a plurality of chunks, and each first chunk may comprise data belonging to at least one semantic category. When the user needs to acquire target data, the user can send a data acquisition request of the user to a data acquisition interface provided by the cloud management platform. Since the data acquisition request is used for indicating the target data that the user needs to acquire, the cloud management platform can first perform semantic recognition on the data acquisition request to obtain a first semantic category of the target data. Then, the cloud management platform can notify the storage node to determine, from among the plurality of first chunks, a target first chunk matching the first semantic category, and to provide the target first chunk to the cloud management platform. Then, the cloud management platform can acquire the target data from the target first chunk, and provide the target data to the user.

Description

A data acquisition method based on a cloud management platform and the cloud management platform

[0001] This application claims priority to Chinese patent application filed on January 23, 2025, with application number 202510113871.2 and title "A Data Acquisition Method Based on a Cloud Management Platform and a Cloud Management Platform", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud technology, and in particular to a data acquisition method based on a cloud management platform and a cloud management platform. Background Technology

[0003] With the rapid development of cloud technology, more and more users are choosing to use cloud storage services provided by cloud providers to store their large amounts of data, thereby reducing data storage costs and ensuring data security.

[0004] In related technologies, cloud vendors' cloud management platforms can store multiple pieces of user data in the form of tensors within a storage cluster. Within this tensor, the user's data is typically presented in units of data chunks; that is, the tensor consists of multiple chunks, and any single chunk can contain several pieces of the user's data. When a user needs to retrieve certain data, the cloud management platform can instruct the storage cluster to read these multiple chunks one by one from the tensor, then select the data the user needs from these chunks and return it to the user.

[0005] In the process described above, since the user's multiple data are scattered across various chunks, when the user needs some of the data, the cloud management platform often needs to have the storage cluster acquire a large number of chunks in order to read the data required by the user from these chunks. This data acquisition method is often inefficient, resulting in high time costs for data acquisition. Summary of the Invention

[0006] This application provides a data acquisition method and a cloud management platform based on a cloud management platform, which can improve the efficiency of the cloud management platform in completing data acquisition, reduce the time cost required for data acquisition, and thus improve the user experience.

[0007] A first aspect of this application provides a data acquisition method based on a cloud management platform. The cloud management platform implementing this method manages infrastructure that provides cloud services to users. This infrastructure may include storage nodes created by the cloud management platform for users. These storage nodes store tensors belonging to the users, and each tensor may include multiple first chunks, each first chunk including data belonging to at least one semantic category. The method includes:

[0008] When a user needs to obtain target data, the user can send a data retrieval request to the data retrieval interface provided by the cloud management platform. In this way, the cloud management platform can receive the data retrieval request sent by the user through the data retrieval interface, and the data retrieval request can be used to indicate the target data that the user needs to obtain.

[0009] Upon receiving a data acquisition request, the cloud management platform can use certain algorithms or models to perform semantic recognition on the data acquisition request, thereby obtaining the first semantic category to which the target data belongs.

[0010] After obtaining the first semantic category, the cloud management platform can instruct the storage node to traverse the tensors of the users it stores, so as to read the target first chunk that matches the first semantic category from the multiple first chunks contained in the tensor, and return the target first chunk to the cloud management platform.

[0011] After obtaining the first target chunk, the cloud management platform can retrieve the target data from the first target chunk and return the target data to the user through the data acquisition interface for the user's use, thereby meeting the user's data acquisition needs.

[0012] As can be seen from the above method, since the cloud management platform can identify the first semantic category to which the target data required by the user belongs, and the multiple first chunks contained in the user's tensor correspond to multiple semantic categories, the cloud management platform can notify the storage node to select the target first chunk corresponding to the first semantic category from these multiple first chunks. This allows the cloud management platform to directly obtain the target data from the target first chunk and return it to the user. Therefore, the storage node can accurately return the target first chunk containing the target data to the cloud management platform, and the number of target first chunks is often small, which can improve the efficiency of the cloud management platform in completing data acquisition, reduce the time cost required for data acquisition, and thus improve the user experience.

[0013] In one possible implementation, the tensor also includes multiple second chunks, each corresponding to at least one first chunk. Each second chunk includes the semantic category to which the data in its corresponding first chunk belongs. The cloud management platform instructs the storage node to determine the first chunk matching the first semantic category from the multiple first chunks. This includes: the cloud management platform instructing the storage node to determine the target second chunk matching the first semantic category from the multiple second chunks, and to determine the target first chunk corresponding to the target second chunk from the multiple first chunks. In the aforementioned implementation, since the user's tensor also includes multiple second chunks, each corresponding to at least one first chunk, and each second chunk including the semantic category to which the data in its corresponding first chunk belongs, the cloud management platform can instruct the storage node to traverse the user's tensor it stores to read the target second chunk matching the first semantic category from the multiple second chunks contained in the tensor. Based on this, the storage node can also traverse these multiple first chunks to read the target first chunk corresponding to the target second chunk from these multiple first chunks. Since the user's tensor contains not only multiple first chunks but also multiple second chunks, and the second chunks contain the semantic categories of the data contained in the first chunks, the second chunks can serve as indexes to the first chunks. Once the storage node determines the target second chunk that matches the target semantic category from among the multiple second chunks, it can determine the target first chunk corresponding to the target second chunk. The determined target first chunk contains the target data required by the user. Therefore, the storage node can accurately and quickly locate the target first chunk containing the target data required by the user.

[0014] In one possible implementation, the tensor also includes multiple third chunks, each corresponding to at least one first chunk. Each third chunk includes metadata of the data included in its corresponding first chunk. The data acquisition request is also used to indicate the metadata of the target data. The method further includes: the cloud management platform notifying the storage node to determine the target third chunk corresponding to the target first chunk from the multiple third chunks and sending the target third chunk to the cloud management platform; the cloud management platform obtaining the metadata of the target data from the target third chunk and sending the metadata of the target data to the user through the data acquisition interface. In the aforementioned implementation, since the user's tensor also includes multiple third chunks, each corresponding to at least one first chunk, and each third chunk including metadata of the data included in its corresponding first chunk, if the data acquisition request is also used to indicate the metadata of the target data, it means that the user also needs to obtain the metadata of the target data. Therefore, the cloud management platform can also allow the storage node to continue to traverse the user's tensor stored therein to read the target third chunk corresponding to the target first chunk from the multiple third chunks contained in the tensor and send the target third chunk to the cloud management platform as well. Furthermore, the cloud management platform can also read the target data's metadata from the target third chunk and send the metadata to the user through the data acquisition interface. Therefore, the cloud management platform not only meets the user's need to obtain the target data but also their need to obtain the target data's metadata, thereby improving the user experience.

[0015] In one possible implementation, the metadata of the target data includes at least one of the following: the target data's tag, the target data's generation time, the target data's size, and the target data's identifier. In the aforementioned implementation, the target data's metadata may include one or more of the following: the target data's tag, the target data's generation time, the target data's size, and the target data's identifier. This metadata is available to the user and can also be used by the cloud management platform to determine whether the target data meets the conditions set by the user.

[0016] In one possible implementation, the data acquisition request also indicates conditions set by the user. The cloud management platform sends the target data to the user through the data acquisition interface, including: the cloud management platform selecting data that meets the conditions from the target data based on the target data's metadata, and then sending the data meeting the conditions to the user through the data acquisition interface. In the aforementioned implementation, if the data acquisition request also indicates conditions set by the user, after reading the target data, the cloud management platform can select data that meets the conditions from the target data based on the target data's metadata and remove data that does not meet the conditions. Then, the cloud management platform can send the data meeting the conditions to the user through the data acquisition interface, thereby accurately meeting the user's data acquisition needs and also satisfying the user's specific customization needs for data, thus further improving the user experience.

[0017] In one possible implementation, the data acquisition request is further used to indicate the semantic category to which the target data belongs. The method also includes: the cloud management platform notifying the storage node to send the target second chunk to the cloud management platform; the cloud management platform retrieving the second semantic category containing the target data from the target second chunk, and sending the second semantic category to the user through the data acquisition interface, wherein the second semantic category includes the first semantic category. In the aforementioned implementation, if the data acquisition request is also used to indicate the semantic category to which the target data belongs, it indicates that the user also wants to obtain the semantic category to which the target data belongs. Therefore, the cloud management platform can also send the target second chunk to the cloud management platform. Then, the cloud management platform can also read the second semantic category of the target data (the second semantic category includes the first semantic category) from the target second chunk and send the second semantic category to the user through the data acquisition interface. Thus, it can be seen that the cloud management platform, in addition to satisfying the user's needs for obtaining the target data and its metadata, can also satisfy the user's needs for obtaining the semantic category of the target data, thereby further improving the user experience.

[0018] In one possible implementation, the method further includes: a cloud management platform receiving a data upload request from a user via a data upload interface, wherein the data upload request includes multiple pieces of user data; the cloud management platform performing semantic recognition on the multiple pieces of data included in the data upload request to obtain semantic categories of the multiple pieces of data; the cloud management platform dividing the multiple pieces of data into multiple first chunks based on the semantic categories of the multiple pieces of data, and generating tensors based on the multiple first chunks; and the cloud management platform notifying storage nodes to store the tensors. In the aforementioned implementation, when a user needs to store multiple pieces of data in the cloud, the user can input a data upload request into the data upload interface provided by the cloud management platform. In this way, the cloud management platform can receive the data upload request sent by the user through the data upload interface, and the data upload request may include multiple pieces of user data. Then, the cloud management platform can perform semantic recognition on the multiple pieces of data included in the data upload request using a certain algorithm or model to obtain the semantic categories of these multiple pieces of data. Then, the cloud management platform can refer to the semantic categories of these multiple pieces of data to divide these multiple pieces of data into multiple first chunks, and generate tensors of the user based on these multiple first chunks. Therefore, cloud management platforms can meet users' data storage needs, reduce the pressure on users' local data storage, lower users' data storage costs, and ensure users' data security.

[0019] In one possible implementation, the data upload request also includes metadata for multiple data sets. The method further includes: the cloud management platform dividing the semantic categories of the multiple data sets into multiple second chunks, dividing the metadata of the multiple data sets into multiple third chunks, and constructing a correspondence between the multiple first chunks, multiple second chunks, and multiple third chunks; the cloud management platform generating a tensor based on the multiple first chunks includes: the cloud management platform generating a tensor based on the multiple first chunks, multiple second chunks, multiple third chunks, and the correspondence. In the aforementioned implementation, the user's data upload request may also include metadata for multiple data sets. Therefore, the cloud management platform may further divide the semantic categories of these multiple data sets into multiple second chunks, divide the metadata of these multiple data sets into multiple third chunks, and construct a correspondence between these multiple first chunks, multiple second chunks, and multiple third chunks. Subsequently, the cloud management platform can use these multiple first chunks, multiple second chunks, multiple third chunks, and the correspondence to generate the user's tensor and send the tensor to the storage node so that the storage node stores the tensor. Therefore, the user tensors generated by the cloud management platform contain user data, scalars of this data, and semantic categories of this data. This allows the data, scalars, and semantic categories to be centrally stored in the storage nodes as tensors, eliminating the need for separate storage. This reduces data storage costs and index building costs.

[0020] In one possible implementation, storage nodes include physical servers, virtual machines, containers, microvirtual machines, or bare metal servers.

[0021] A second aspect of this application provides a cloud management platform for managing infrastructure providing cloud services. The infrastructure includes storage nodes storing user tensors. Each storage node stores a plurality of first data chunks, and each first chunk includes data belonging to at least one semantic category. The cloud management platform includes: a first receiving module for receiving a data acquisition request sent by a user through a data acquisition interface, wherein the data acquisition request indicates target data to be acquired; a first identification module for performing semantic recognition on the data acquisition request to obtain a first semantic category to which the target data belongs; a first notification module for notifying the storage nodes to determine a target first chunk matching the first semantic category from the plurality of first chunks and sending the target first chunk to the cloud management platform; and a first acquisition module for acquiring target data from the target first chunk and sending the target data to the user through the data acquisition interface.

[0022] In one possible implementation, the tensor also includes multiple second chunks, each second chunk corresponding to at least one first chunk, each second chunk including the semantic category to which the data in its corresponding first chunk belongs, and a first notification module for notifying the storage node to determine a target second chunk that matches the first semantic category from the multiple second chunks, and to determine a target first chunk that corresponds to the target second chunk from the multiple first chunks.

[0023] In one possible implementation, the tensor also includes multiple third chunks, each third chunk corresponding to at least one first chunk, each third chunk including metadata of the data included in its corresponding first chunk, and the data acquisition request is also used to indicate the metadata of the target data. The cloud management platform also includes: a second notification module, used to notify the storage node to determine the target third chunk corresponding to the target first chunk from the multiple third chunks, and send the target third chunk to the cloud management platform; and a second acquisition module, used to acquire the metadata of the target data from the target third chunk, and send the metadata of the target data to the user through the data acquisition interface.

[0024] In one possible implementation, the metadata of the target data includes at least one of the following: the tag of the target data, the generation time of the target data, the size of the target data, and the identifier of the target data.

[0025] In one possible implementation, the data acquisition request is also used to indicate the conditions set by the user. The first acquisition module is used to select data that meets the conditions from the target data based on the metadata of the target data, and send the data that meets the conditions to the user through the data acquisition interface.

[0026] In one possible implementation, the data acquisition request is also used to indicate the semantic category to which the target data belongs. The cloud management platform also includes: a third notification module for notifying the storage node to send the target second chunk to the cloud management platform; and a third acquisition module for acquiring the second semantic category to which the target data belongs from the target second chunk and sending the second semantic category to the user through the data acquisition interface, wherein the second semantic category includes the first semantic category.

[0027] In one possible implementation, the cloud management platform further includes: a second receiving module, used to receive a data upload request sent by a user through a data upload interface, wherein the data upload request includes multiple data from the user; a second identification module, used to perform semantic recognition on the multiple data included in the data upload request to obtain the semantic categories of the multiple data; a generation module, used to divide the multiple data into multiple first chunks based on the semantic categories of the multiple data, and generate tensors based on the multiple first chunks; and a storage module, used to notify storage nodes to store the tensors.

[0028] In one possible implementation, the data upload request also includes metadata for multiple data items. The cloud management platform further includes: a construction module, used to divide the semantic categories of multiple data items into multiple second chunks, divide the metadata of multiple data items into multiple third chunks, and construct the correspondence between multiple first chunks, multiple second chunks, and multiple third chunks; and a generation module, used to generate tensors based on multiple first chunks, multiple second chunks, multiple third chunks, and the correspondence.

[0029] In one possible implementation, storage nodes include physical servers, virtual machines, containers, microvirtual machines, or bare metal servers.

[0030] A third aspect of this application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to cause the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect according to the instructions.

[0031] A fourth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in the first aspect or any possible implementation of the first aspect.

[0032] A fifth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0033] In this embodiment, the cloud management platform creates storage nodes for users to store user tensors. These tensors can contain multiple chunks, and each first chunk can contain data belonging to at least one semantic category. When a user needs to obtain target data, the user can send a data acquisition request to the data acquisition interface provided by the cloud management platform. Since the data acquisition request indicates the target data the user needs, the cloud management platform can first perform semantic recognition on the data acquisition request to obtain the first semantic category of the target data. Then, the cloud management platform can instruct the storage node to determine the target first chunk matching the first semantic category from the multiple first chunks contained in the tensor and provide the target first chunk to the cloud management platform. Finally, the cloud management platform can obtain the target data from the target first chunk and provide the target data to the user for use. In the aforementioned process, since the cloud management platform can identify the first semantic category to which the target data required by the user belongs, and the multiple first chunks contained in the user's tensor correspond to multiple semantic categories, the cloud management platform can notify the storage node to select the target first chunk corresponding to the first semantic category from these multiple first chunks. This allows the cloud management platform to directly obtain the target data from the target first chunk and return it to the user. Therefore, the storage node can accurately return the target first chunk containing the target data to the cloud management platform, and the number of target first chunks is often small, which improves the efficiency of the cloud management platform in completing data acquisition, reduces the time cost required for data acquisition, and thus improves the user experience. Attached Figure Description

[0034] Figure 1 is a schematic diagram of a cloud service system provided in an embodiment of this application;

[0035] Figure 2a is another structural schematic diagram of the cloud service system provided in the embodiment of this application;

[0036] Figure 2b is another structural schematic diagram of the cloud service system provided in the embodiment of this application;

[0037] Figure 3 is a schematic diagram of an application example of the cloud service system provided in the embodiments of this application;

[0038] Figure 4 is a flowchart illustrating a data acquisition method based on a cloud management platform provided in an embodiment of this application.

[0039] Figure 5 is a schematic diagram of data acquisition provided in an embodiment of this application;

[0040] Figure 6 is a schematic diagram of data uploading provided in an embodiment of this application;

[0041] Figure 7 is a schematic diagram of the structure of a cloud management platform provided in an embodiment of this application;

[0042] Figure 8 is a schematic diagram of a computing device provided in an embodiment of this application;

[0043] Figure 9 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0044] Figure 10 is a schematic diagram of computer devices in a computer cluster connected via a network according to an embodiment of this application. Detailed Implementation

[0045] This application provides a data acquisition method and a cloud management platform based on a cloud management platform, which can improve the efficiency of the cloud management platform in completing data acquisition, reduce the time cost required for data acquisition, and thus improve the user experience.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0047] With the rapid development of cloud technology, more and more users are choosing to use cloud storage services provided by cloud providers to store their large amounts of data, thereby reducing data storage costs and ensuring data security. For example, users can store their training data for model training in a storage cluster created by the cloud provider. When users need to retrieve this training data, the cloud provider can retrieve it from the storage cluster and return it to the user for use.

[0048] In related technologies, cloud vendors' cloud management platforms can store multiple user data points in the form of tensors within a storage cluster. Within this tensor, the user's data is typically presented in units of chunks; that is, the tensor consists of multiple chunks, and any single chunk can contain several pieces of the user's data. When a user needs to retrieve certain data, the cloud management platform can instruct the storage cluster to read these multiple chunks one by one from the tensor, then select the data the user needs from these chunks and return it to the user.

[0049] In the process described above, since the user's multiple data are scattered across various chunks, when the user needs some of the data, the cloud management platform often needs to have the storage cluster acquire a large number of chunks in order to read the data required by the user from these chunks. This data acquisition method is often inefficient, resulting in high time costs for data acquisition.

[0050] Furthermore, since the cloud management platform needs to provide a large number of chunks from the storage cluster when reading data for users, and these chunks contain a lot of data that users do not need, the storage cluster often consumes a lot of communication resources (such as bandwidth) when transmitting these chunks to the cloud management platform. This results in low effective utilization of communication resources and a certain amount of resource waste.

[0051] To address the aforementioned problems, this application provides a data acquisition method based on a cloud management platform. This method can be implemented through a cloud service system (i.e., a distributed storage system). Figure 1 is a schematic diagram of the structure of a cloud service system provided in this application embodiment. As shown in Figure 1, the cloud service system includes infrastructure that can provide cloud services and a cloud management platform that manages this infrastructure. The cloud management platform and the infrastructure are described in detail below:

[0052] A cloud management platform can centrally manage the infrastructure of the entire cloud service system (for example, creating multiple storage nodes to meet a user's needs according to their instructions; these storage nodes can store user data using tensors, etc.). The cloud management platform can also be accessible to users outside the cloud service system and respond to their requests. For instance, the cloud management platform can provide various interfaces such as login, data upload / retrieval, and data retrieval interfaces for user clients (e.g., the user's terminal device or browser on that device) to access. The login interface allows the cloud management platform to authenticate a user's client, granting access upon successful authentication. Similarly, the data upload interface allows user clients to send data upload requests to the cloud management platform. Since these requests contain multiple data items to be stored, along with their metadata, the cloud management platform can perform semantic recognition on the data (or the data items and their metadata) to determine their semantic categories. Therefore, the cloud management platform can divide the multiple data sets and their metadata into multiple chunks (each chunk typically contains data belonging to at least one semantic category) based on the semantic categories of these data sets, generating a tensor containing these chunks, and storing this tensor in the storage nodes. Alternatively, the cloud management platform can allow user clients to send data retrieval requests to the platform via a data retrieval interface. Since the data retrieval request indicates the target data the user needs, the cloud management platform can first perform semantic recognition on the request to determine the target semantic category to which the target data belongs. Next, the cloud management platform can instruct the storage nodes to identify the chunks matching the target semantic category from the multiple chunks contained in the tensor and provide these chunks to the cloud management platform. Then, the cloud management platform can retrieve the target data from these chunks and provide it to the user for use.

[0053] The infrastructure comprises multiple storage clusters serving users, including clusters for caching data and clusters for persistent data storage. Generally, the clusters for caching data are located closer to the cloud management platform, while those for persistent data are located further away. Each cluster, whether for caching or persistent data storage, can contain multiple storage nodes. During data upload, the cloud management platform can generate a user's tensor and store it (containing multiple chunks) on one or more storage nodes within the persistent data storage cluster. During data retrieval, the cloud management platform can instruct the persistent data storage cluster to return one or more chunks. Typically, the persistent data storage cluster will first send these chunks to the caching data storage cluster, which then forwards them to the cloud management platform and caches them. Therefore, if the cloud management platform needs to retrieve these chunks again later, it can quickly obtain them directly from the caching data storage cluster, reducing data retrieval time.

[0054] Further, as shown in Figure 2a (Figure 2a is another structural schematic diagram of the cloud service system provided in this application embodiment), the cloud management platform may include multiple interfaces (including the aforementioned login interface, data upload interface, and data acquisition interface, etc.), a tensor module, a vector module, and an embedding module. As shown in Figure 2b (Figure 2b is another structural schematic diagram of the cloud service system provided in this application embodiment), after the cloud management platform's interface receives a user's data upload request, the tensor module can transmit multiple data items (or, these multiple data items and their metadata) contained in the data upload request to the embedding module. This allows the embedding module to perform semantic recognition on these multiple data items, thereby obtaining the semantic categories of these multiple data items (also referred to as the semantic representations of these multiple data items or the vectors of these multiple data items, etc.), and returning them to the tensor module. The tensor module can also provide the semantic categories of these multiple data items to the vector module, allowing the vector module to calculate the similarity between the semantic categories of these multiple data items. Based on the similarity, the vector module clusters these multiple data items according to their semantic categories, thereby obtaining clustering results and returning them to the tensor module. These clustering results are used to indicate the data belonging to each semantic category obtained from the division of these multiple data items. The tensor module can also determine (at least one) data belonging to each semantic category based on the clustering results, and preferentially group data belonging to the same semantic category into the same chunk. This allows the multiple data sets to be divided into multiple chunks, each of which can contain data from at least one semantic category. The tensor module can also generate tensors based on these multiple chunks and store the tensors in the storage nodes of the storage cluster used for persistent data storage.

[0055] Furthermore, after the cloud management platform's interface receives a user's data retrieval request, the tensor module can also transmit the request to the mapping module. This allows the mapping module to perform semantic recognition on the request, thereby determining the target semantic category of the target data indicated by the request, and returning this result to the tensor module. The tensor module can then instruct the storage nodes in the storage cluster used for persistent data storage to identify the chunks from the multiple chunks contained in the user's tensor that match the target semantic category, and provide these chunks to the cloud management platform.

[0056] Furthermore, for the storage nodes in each storage node serving the user, they can be cloud instances in the infrastructure. These cloud instances can be presented in various ways. For example, these cloud instances can be physical servers selected by the cloud management platform, bare metal servers selected by the cloud management platform, virtual machines (VMs) created by the cloud management platform on physical servers using virtualization technology, containers (Docker) created by the cloud management platform on physical servers using virtualization technology, micro VMs created by the cloud management platform on physical servers using virtualization technology, and so on.

[0057] Furthermore, regardless of how the storage nodes serving users are presented, they all possess a certain amount of physical storage resources (for example, when the storage node is a physical server, these physical storage resources can be the memory, hard disk, block devices, and encapsulated persistent objects that can be read and written, etc.). Therefore, each storage node can use its storage resources to store a certain amount of data, thereby providing remote data storage services to users. It is worth noting that the storage resources of a storage node can be divided into multiple storage areas. Each storage area serves as a medium for storing data. These media have the following characteristics: data can only be appended to these media, and cannot be modified while writing to them; that is, they do not support write-in-place, only append.

[0058] Furthermore, for a user's multiple storage nodes, these multiple storage nodes can be deployed in the same site or different sites. The site can be presented in various forms. For example, the site can be a region in the infrastructure, an availability zone in the infrastructure, a data center (DC) in the infrastructure, a room in the infrastructure, a rack in the infrastructure, and so on.

[0059] It is worth noting that in different application scenarios, the infrastructure of the aforementioned cloud service system can provide different cloud services, such as object storage service (OBS). For ease of introduction, the following description uses the application scenario of the cloud service system providing OBS as an example. As shown in Figure 3 (Figure 3 is a schematic diagram of an application example of the cloud service system provided in this application embodiment), in this application scenario, the cloud service system can be presented as an object storage system, the cloud management platform can be presented as a platform containing an object storage service layer, the physical server where the storage node is located can be presented as an object storage device (OSD), the storage node can be presented as a bucket or object storage bucket, the physical storage resources occupied by the storage node can be presented as physical hard disks, and the storage area in the physical storage resources can be presented as a persistent storage unit (PLOG). The following is a brief introduction to the basic concepts involved:

[0060] (1) OBS: OBS is an object-based storage service with advantages such as massive capacity, security, high reliability, and low cost. OBS is an internet-accessible service. Users can establish a connection with the object storage service layer through a client, create buckets in the storage nodes managed by the object storage service layer, and then access and manage the objects in the buckets. The object storage service layer uses a sequential distribution method to store objects in buckets by default. The sequential distribution method can also be called lexicographical distribution or range distribution.

[0061] (2) Object: An object is the basic unit of data storage in an object storage system. An object is actually a collection of a file's data and its related attribute information (metadata). Data uploaded by users to OBS is stored in buckets in the form of objects. An object consists of three parts: key, metadata, and data. The key, i.e., the object name, is, for example, a character sequence with a length greater than 0 and not exceeding 1024 characters after UTF-8 encoding. Each object in a bucket has a unique object key.

[0062] (3) Bucket: A bucket is a container for storing objects in OBS. Object storage provides a flat storage method based on buckets and objects. All objects in a bucket are at the same logical level, eliminating the multi-level tree directory structure in the file system. Each bucket has its own storage category, access permissions, region, and other attributes. Users can create buckets with different storage categories and access permissions and configure more advanced attributes to meet the storage needs of different scenarios.

[0063] (4) OSD: OSD is the basic storage unit of the object storage system. It is set on the physical disk and is a fixed-size storage space of the physical disk. The object storage system manages the physical disks of multiple physical servers in the form of OSD.

[0064] In the object storage system shown in Figure 3, users can log in to the object storage service layer through their client, select and purchase object storage services, such as creating buckets and configuring bucket names. After detecting the user's operation (such as creating a bucket), the service layer can issue a creation command to the object storage device. This creation command includes information such as the bucket name and domain name, and is used to notify the storage device to create the bucket and save the bucket name, bucket domain name, and other information. After the bucket is created, users can access the bucket domain name through the object storage client, locate the bucket on the storage device, and upload, download, and delete objects from the bucket.

[0065] The service layer includes a control unit and an operating system. The control unit runs on the operating system, which includes disk drivers and physical network card drivers. The control unit controls the disk controller through the disk drivers to set up the physical disk as multiple persistent storage units (PLOG).

[0066] The infrastructure for running object storage services can be set up across multiple data centers in different regions. For example, object storage devices are set up in multiple data centers in different regions, and each data center includes multiple object storage devices. Each object storage device includes at least one physical disk. For example, the disk controller sets physical disk 11 and physical disk 12 in Region 1 as 4 PLOGs.

[0067] After receiving the creation instruction for bucket 1, the service layer notifies the control unit to create bucket 1. The control unit creates bucket 1 on physical disk 1 and physical disk 2 in the object storage device through the operating system. The PLOG of bucket 1 is distributed on physical disk 1 and physical disk 2, and information such as bucket name and bucket domain name of bucket 1 is stored.

[0068] When a user needs to upload multiple objects, such as multiple data items, to bucket 1 via the client, the user can call the data upload interface provided by the object storage service layer through the client. This will cause the object storage service layer to trigger the object storage device to write the multiple data items into bucket 1 as tensors. For example, the control unit in the object storage service layer can write the multiple data items into bucket 1 as tensors.

[0069] When a user needs to retrieve (i.e. download) a specific piece of data from multiple data sets in bucket 1 through a client, the user can call the data retrieval interface provided by the object storage service layer through the client. This will cause the object storage service layer to trigger the object storage device to read the data from the user's tensor stored in bucket 1 and return it to the user's client for the user to use.

[0070] In addition, the object storage service layer also provides users with data processing interfaces such as data deletion, data merging, and data modification interfaces, which will not be elaborated on here.

[0071] It should be understood that the example shown in Figure 3 above is only a schematic illustration of the application scenario where the cloud service system provides OBS externally, and does not limit the types of cloud services that the cloud service system provides externally. Correspondingly, in other application scenarios, the cloud service system can provide other types of cloud services externally. In this case, the form presented by the cloud management platform and storage nodes will be different. For example, in other application scenarios, storage nodes can be presented as virtual machines, physical servers, etc. Based on the cloud service system described above, it is known that the storage node stores a user's tensor, which can contain multiple chunks, and each chunk can contain data belonging to at least one semantic category. When a user needs to obtain target data, the user can send a data acquisition request to the data acquisition interface provided by the cloud management platform. Since the data acquisition request is used to indicate the target data that the user needs to obtain, the cloud management platform can first perform semantic recognition on the data acquisition request to obtain the target semantic category of the target data. Then, the cloud management platform can instruct the storage node to determine the chunks that match the target semantic category from the multiple chunks contained in the tensor and provide these chunks to the cloud management platform. Then, the cloud management platform can retrieve the target data from these chunks and provide it to the user. In this process, because the cloud management platform can identify the target semantic category of the target data the user needs, and because the user's tensor contains multiple chunks corresponding to multiple semantic categories, the cloud management platform can instruct the storage nodes to select one or more chunks corresponding to the target semantic category. This allows the cloud management platform to directly retrieve the target data from these chunks and return it to the user. Therefore, the storage nodes can accurately return chunks containing the target data to the cloud management platform, and the number of these chunks is often small, improving the efficiency of data retrieval, reducing the time cost of data acquisition, and thus improving the user experience. To further understand the working process of the cloud management platform, the following description, in conjunction with Figure 4, illustrates this process. Figure 4 is a flowchart illustrating a data acquisition method based on a cloud management platform provided in this embodiment. As shown in Figure 4, this method can be implemented through a cloud service system as shown in Figure 1. The cloud service system includes infrastructure providing cloud services to users and a cloud management platform managing this infrastructure. This infrastructure includes storage nodes created by the cloud management platform for users. The storage nodes store tensors of the users, which may contain multiple chunks. Any one of these chunks may contain data belonging to at least one semantic category. The method includes:

[0072] 401. The cloud management platform receives data acquisition requests sent by users through the data acquisition interface. The data acquisition request is used to indicate the target data to be acquired.

[0073] In this embodiment, when a user needs to obtain target data, the cloud management platform can provide a data acquisition interface to the user's client (e.g., a data acquisition bar in the user interface). Therefore, the user can input a data acquisition request set by the user into the data acquisition interface through their client. In this way, the cloud management platform can receive the data acquisition request sent by the user's client through the data acquisition interface. The data acquisition request can be used to indicate the target data that the user needs to obtain.

[0074] Specifically, data retrieval requests can be presented in the following ways:

[0075] A data retrieval request can include not only parameters describing the content of the target data (e.g., instructions, text, images, videos, etc.) but also parameters describing the target data, its metadata, and its semantic category. For example, this parameter could be the data type; if it's raw data, the user only needs the target data; if it's a scalar or raw data, the user needs both the target data and its metadata; and if it's a scalar, raw data, or vector, the user needs the target data, its metadata, and its semantic category. Furthermore, a data retrieval request can include parameters describing certain conditions. For instance, a time period might indicate that the user needs to filter the target data, and the processed data will meet the conditions—data generated within that time period.

[0076] Therefore, a data acquisition request typically includes one or more of the above parameters. Thus, a data acquisition request can be used not only to indicate the target data, but also to indicate the metadata of the target data, the semantic category to which the target data belongs, and the conditions set by the user, etc.

[0077] For example, as shown in Figure 5 (Figure 5 is a schematic diagram of data acquisition provided in an embodiment of this application), when the target data that the user needs to acquire is an image of a white dog, the user can log in to the cloud management platform. The cloud management platform provides a user interface, which includes a data acquisition field. The user can then enter the following data acquisition request into the data acquisition field, and the data acquisition request may include the following parameters:

[0078] Parameter 1: Data content: "white dog";

[0079] Parameter 2: Data type: scalar, raw data, vector, etc.;

[0080] Parameter 3: Condition: Generation time is less than August;

[0081] ...

[0082] Based on the data retrieval request, parameter 1 is the text "white dog", indicating that the target data that the user needs to obtain is about a white dog. Parameter 2 is the data type "scalar, raw data, and vector", indicating that the user needs to obtain the target data, the metadata of the target data, and the semantic category to which the target data belongs. Parameter 3 is the condition "generate data generated before August", indicating that the user needs to filter the initially obtained target data. The processed data is the data that finally meets the condition, that is, the data generated before August.

[0083] Therefore, the cloud management platform can obtain the data acquisition request sent by the user through the data acquisition section.

[0084] 402. The cloud management platform performs semantic recognition on the data acquisition request to obtain the first semantic category to which the target data belongs.

[0085] Upon receiving a data acquisition request, since the data acquisition request can be information describing the target data, the cloud management platform can use certain algorithms (e.g., semantic recognition algorithms, etc.) or models (e.g., semantic recognition models, etc.) to perform semantic recognition (also known as semantic transformation) on the data acquisition request, thereby obtaining the first semantic category to which the target data belongs.

[0086] As in the example above, after receiving the data acquisition request, since the data acquisition request contains the text "white dog", the cloud management platform can perform semantic recognition on the data acquisition request to obtain the target semantic category to which the target data belongs (i.e., the first semantic category mentioned above). The target semantic category can be represented as vector0, which is used to represent the target semantic category as white dog.

[0087] 403. The cloud management platform notifies the storage node to determine the target first chunk that matches the first semantic category from multiple first chunks, and sends the target first chunk to the cloud management platform.

[0088] 404. The cloud management platform obtains the target data from the first target chunk and sends the target data to the user through the data acquisition interface.

[0089] Since the storage nodes store user tensors, and these tensors contain multiple first chunks, and each first chunk contains data from one or more users belonging to at least one semantic category—meaning each first chunk corresponds to (matches) at least one semantic category—the cloud management platform can send a chunk retrieval request to the storage nodes. This request may contain the first semantic category. The storage nodes can then parse the chunk retrieval request to obtain the first semantic category. Finally, the storage nodes can iterate through these stored first chunks to read the target first chunk that matches the first semantic category and return it to the cloud management platform.

[0090] After obtaining the first target chunk, the cloud management platform can retrieve the target data from it and return it to the user's client via a data acquisition interface for user use. This satisfies the user's data acquisition needs.

[0091] Specifically, the cloud management platform can construct user tensors in the following ways:

[0092] When a user needs to store multiple data items in the cloud, the cloud management platform can provide a data upload interface to the user's client (e.g., a data upload bar in the user interface). The user can then input their configured data upload request into this interface. The cloud management platform can then receive the data upload request from the user's client through this interface. This data upload request can include multiple data items and their metadata. It should be noted that the metadata for any single data item may include its label, date, size, and identity document (ID), among other things.

[0093] Upon receiving a data upload request, the cloud management platform can parse it to obtain multiple data points from the user. Next, the cloud management platform can perform semantic recognition on these data points using a specific algorithm or model to determine their semantic categories. Then, based on these semantic categories, the cloud management platform can divide the data into multiple first chunks. Any one of these first chunks can contain data belonging to at least one semantic category. For example, all data in one first chunk might belong to the same semantic category, while some data in another first chunk might belong to one semantic category and others to a different semantic category, and so on.

[0094] Furthermore, the cloud management platform can divide the semantic categories of this data into multiple second chunks, divide the metadata of this data into multiple third chunks, and construct the correspondence between these first chunks, second chunks, and third chunks. In this correspondence, each second chunk typically corresponds to at least one first chunk, meaning that each second chunk can contain the semantic category of the data contained in its corresponding first chunk. Similarly, each third chunk typically corresponds to at least one first chunk, meaning that each third chunk can contain the metadata of the data contained in its corresponding first chunk.

[0095] Subsequently, the cloud management platform can use these multiple first chunks, multiple second chunks, multiple third chunks, and the corresponding relationships to generate a tensor for the user, and send this tensor to the storage node so that the storage node can store the tensor. This fulfills the user's data storage requirements.

[0096] Continuing with the example above, as shown in Figure 6 (Figure 6 is a schematic diagram of data upload provided in an embodiment of this application), suppose the user has images i1, i2, i3, ..., i12, where i1 contains images of cats and dogs playing in the snow, i2 contains images of dogs playing in the grass of a park, i3 contains images of a white cat, ..., and i12 contains images of dogs running in the snow. When the user needs to store these images in the cloud, the user can log in to the cloud management platform. The cloud management platform provides a user interface, which includes a data upload section. Users can then enter data upload requests in the data upload field. The data upload request includes i1 to i12, the tags of i1 to i12, and the generation tags of i1 to i12. Among them, the tag of i1 is cats and dogs playing in the snow, the tag of i2 is dogs playing in the grass in the park, the tag of i3 is a white cat, ..., the tag of i12 is a dog running in the snow, the generation time of i1 is January, the generation time of i2 is February, the generation time of i3 is August, ..., the generation time of i12 is July.

[0097] Upon receiving a data upload request, the cloud management platform can parse the request to obtain i1 to i12, the tags of i1 to i12, and the generation times of i1 to i12. Next, the cloud management platform can perform semantic recognition on i1 to i12, resulting in semantic category vector1 for i1 (indicating i1's semantic category is cats and dogs playing in the snow), semantic category vector2 for i2 (indicating i2's semantic category is dogs playing in the grass in a park), and semantic category vector12 for i12 (indicating i12's semantic category is dogs running in the snow).

[0098] Then, since vector1, vector2, vector4, vector5, vector7, vector8, and vector12 have similar or identical semantic categories, and vector3, vector6, vector9, vector10, and vector11 are similar or identical, the cloud management platform can divide i1, i2, and i4 into chunk0, i5, i7, and i8 into chunk1, i12, i3, and i6 into chunk2, and i9, i10, and i11 into chunk3. Chunk0, chunk1, chunk2, and chunk3 are the aforementioned multiple first chunks.

[0099] Next, the cloud management platform can further divide vector1, vector2, vector4, vector5, vector7, and vector8 into chunk6, and vector12, vector3, vector6, vector9, vector10, and vector11 into chunk7. Chunk6 and chunk7 are the aforementioned multiple second chunks, with chunk6 corresponding to chunk0 and chunk1, and chunk7 corresponding to chunk2 and chunk3.

[0100] Subsequently, the cloud management platform can further divide the tags from i1 to i12 into chunk4, and the generation time from i1 to i12 into chunk5. Chunk4 and chunk5 are the aforementioned multiple third chunks, with chunk4 corresponding to chunk0, chunk1, chunk2, and chunk3, and chunk7 corresponding to chunk0, chunk1, chunk2, and chunk3.

[0101] Then, the cloud management platform can use chunk1 to chunk7 to generate the user's tensor and store the tensor in the storage node.

[0102] More specifically, the cloud management platform and storage nodes can determine the target first chunk in the following ways:

[0103] Since the user's tensor also includes multiple second chunks, each second chunk corresponding to at least one first chunk, and each second chunk including the semantic category of the data in its corresponding first chunk, the cloud management platform can send a chunk retrieval request to the storage node. This chunk retrieval request may contain the first semantic category. Next, the storage node can parse the chunk retrieval request to obtain the first semantic category. Then, the storage node can traverse these stored second chunks to read the target second chunk that matches the first semantic category (that is, some or all of the semantic categories in the target second chunk contain the first semantic category). Based on this, the storage node can also traverse these first chunks to read the target first chunk corresponding to the target second chunk.

[0104] After obtaining the first target chunk, the cloud management platform can retrieve the target data from the first target chunk and return the target data to the user's client through the data retrieval interface for the user to use.

[0105] More specifically, the cloud management platform can also perform the following operations:

[0106] Since a user's tensor also includes multiple third chunks, each corresponding to at least one first chunk, and each third chunk includes metadata of the data contained in its corresponding first chunk, if a data retrieval request is used not only to indicate the target data but also to indicate the target data's metadata, it indicates that the user wants to retrieve not only the target data but also its metadata. Based on this, the cloud management platform can also allow storage nodes, after retrieving the target first chunk, to continue traversing its stored multiple third chunks to read the target third chunk corresponding to the target first chunk. Simultaneously with sending the target first chunk to the cloud management platform, the cloud management platform can also send the target third chunk to the cloud management platform.

[0107] After reading the target data from the first target chunk, the cloud management platform can also read the target data's metadata from the third target chunk, and send the target data and its metadata to the user through the data acquisition interface.

[0108] More specifically, the cloud management platform can also perform the following operations:

[0109] If a data retrieval request indicates not only the target data and its metadata, but also the semantic category to which the target data belongs, it means the user wants to obtain not only the target data and its metadata, but also the semantic category to which the target data belongs. Based on this, while sending the first and third target chunks to the cloud management platform, the cloud management platform can also send the second target chunk to the cloud management platform simultaneously.

[0110] After reading the target data from the first target chunk and the target data's metadata from the third target chunk, the cloud management platform can also read the second semantic category of the target data from the second target chunk and send the target data, its metadata, and the second semantic category to the user through the data acquisition interface. It should be noted that some or all of the semantic categories in the second target chunk include the first semantic category; therefore, these semantic categories in the second target chunk are the second semantic categories of the target data.

[0111] More specifically, the cloud management platform can also perform the following operations:

[0112] If the data retrieval request not only indicates the target data, the target data's metadata, and the semantic category to which the target data belongs, but also indicates the conditions set by the user, after reading the target data, the cloud management platform can select data that meets the conditions from the target data based on the target data's metadata. Then, the cloud management platform can send the data that meets the conditions, the metadata of the data that meets the conditions (obtained from the target data's metadata), and the semantic category to which the data that meets the conditions belongs (obtained from the second semantic category) to the user through the data retrieval interface.

[0113] Continuing with the example above, after obtaining vector0, the cloud management platform can send a chunk retrieval request to the storage node, which contains vector0. Upon receiving this request, the storage node can parse it to obtain vector0. Then, the storage node can iterate through the user's tensors. In chunks 6 and 7, vector5 in chunk 6 indicates that the semantic category of i5 is "a white dog next to a beautiful woman," vector5 indicates that the semantic category of i5 is "a white dog next to a beautiful woman," vector5 indicates that the semantic category of i5 is "a white dog next to a beautiful woman," vector7 indicates that the semantic category of i5 is "a white dog next to a little girl," and vector8 indicates that the semantic category of i5 is "a white dog running." All three vectors contain vector0, indicating that chunk 6 matches vector0. Similarly, chunk 7 does not match vector0. Therefore, the storage node can read chunk 6 (i.e., the aforementioned target second chunk).

[0114] Next, since chunk6 corresponds to chunk0 and chunk1, the storage node can read chunk0 and chunk1 (i.e., the aforementioned target first chunk). Then, since chunk0 and chunk1 correspond to chunk4 and chunk5, the storage node can also read chunk4 and chunk5 (i.e., the aforementioned target third chunk). In this way, the storage node can send chunk0, chunk1, chunk4, chunk5, and chunk6 to the cloud management platform.

[0115] Subsequently, the cloud management platform can read i5, i7, and i8 (these three data are the aforementioned target data) from chunk0 and chunk1, read the labels of i5, i7, and i8 (these three labels are the aforementioned target data metadata) from chunk4, read the generation time of i5, i7, and i8 (these three generation times are the aforementioned target data metadata) from chunk5, and read vector5, vector7, and vector8 (these three vectors are the aforementioned target data second semantic category) from chunk6.

[0116] Since the user-defined condition is that the generation time is less than 8 months, the cloud management platform can remove i8, the tag of i8, the generation time of i8, and vector8. In this way, the cloud management platform can return i5, i7 (these two data points meet the aforementioned conditions), the tag of i5, the tag of i7, the generation time of i5, the generation time of i7, vector5, and vector7 to the user.

[0117] In this embodiment, the cloud management platform creates storage nodes for users to store user tensors. These tensors can contain multiple chunks, and each first chunk can contain data belonging to at least one semantic category. When a user needs to obtain target data, the user can send a data acquisition request to the data acquisition interface provided by the cloud management platform. Since the data acquisition request indicates the target data the user needs, the cloud management platform can first perform semantic recognition on the data acquisition request to obtain the first semantic category of the target data. Then, the cloud management platform can instruct the storage node to determine the target first chunk matching the first semantic category from the multiple first chunks contained in the tensor and provide the target first chunk to the cloud management platform. Finally, the cloud management platform can obtain the target data from the target first chunk and provide the target data to the user for use. In the aforementioned process, since the cloud management platform can identify the first semantic category to which the target data required by the user belongs, and the multiple first chunks contained in the user's tensor correspond to multiple semantic categories, the cloud management platform can notify the storage node to select the target first chunk corresponding to the first semantic category from these multiple first chunks. This allows the cloud management platform to directly obtain the target data from the target first chunk and return it to the user. Therefore, the storage node can accurately return the target first chunk containing the target data to the cloud management platform, and the number of target first chunks is often small, which improves the efficiency of the cloud management platform in completing data acquisition, reduces the time cost required for data acquisition, and thus improves the user experience.

[0118] Furthermore, in this embodiment of the application, since the cloud management platform can accurately return the target first chunk containing the target data to the cloud management platform, and the target data in the target first chunk is the data needed by the user, even if the storage node occupies a lot of communication resources when transmitting the target first chunk to the cloud management platform, since most of the transmitted data is the data needed by the user, the utilization rate of communication resources can be improved, thereby saving resources.

[0119] Furthermore, in this embodiment of the application, the cloud management platform can provide users with a unified data acquisition interface (which can also be understood as the same protocol). This interface allows users to obtain the original quantity (user's data), scalar (data metadata), and vector (semantic category of data) in the tensor. Users do not need to obtain these different types of data separately through multiple operations, which helps to reduce user operations and improve user experience.

[0120] Furthermore, in this embodiment, the user tensor generated by the cloud management platform includes user data, scalars of this data, and semantic categories of this data. This allows the data, scalars of this data, and semantic categories of this data to be centrally stored in the storage node in the form of tensors, without the need for separate storage. This can reduce data storage costs and index building costs.

[0121] The above is a detailed description of the data acquisition method based on a cloud management platform provided in the embodiments of this application. The cloud management platform provided in the embodiments of this application will be described below. Figure 7 is a structural schematic diagram of the cloud management platform provided in the embodiments of this application. As shown in Figure 7, the cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes storage nodes, which store user tensors. Each storage node stores a plurality of first data chunks, and each first chunk includes data belonging to at least one semantic category. The cloud management platform includes:

[0122] The first receiving module 701 is used to receive a data acquisition request sent by a user through a data acquisition interface, wherein the data acquisition request is used to indicate the target data to be acquired;

[0123] The first identification module 702 is used to perform semantic recognition on the data acquisition request to obtain the first semantic category to which the target data belongs;

[0124] The first notification module 703 is used to notify the storage node to determine the target first chunk that matches the first semantic category from multiple first chunks, and send the target first chunk to the cloud management platform;

[0125] The first acquisition module 704 is used to acquire target data from the first target chunk and send the target data to the user through the data acquisition interface.

[0126] In one possible implementation, the tensor also includes multiple second chunks, each second chunk corresponding to at least one first chunk, each second chunk including the semantic category to which the data in its corresponding first chunk belongs, and the first notification module 703 is used to notify the storage node to determine the target second chunk that matches the first semantic category from the multiple second chunks, and to determine the target first chunk that corresponds to the target second chunk from the multiple first chunks.

[0127] In one possible implementation, the tensor also includes multiple third chunks, each third chunk corresponding to at least one first chunk, each third chunk including metadata of the data included in its corresponding first chunk, and the data acquisition request is also used to indicate the metadata of the target data. The cloud management platform also includes: a second notification module, used to notify the storage node to determine the target third chunk corresponding to the target first chunk from the multiple third chunks, and send the target third chunk to the cloud management platform; and a second acquisition module, used to acquire the metadata of the target data from the target third chunk, and send the metadata of the target data to the user through the data acquisition interface.

[0128] In one possible implementation, the metadata of the target data includes at least one of the following: the tag of the target data, the generation time of the target data, the size of the target data, and the identifier of the target data.

[0129] In one possible implementation, the data acquisition request is also used to indicate conditions set by the user. The first acquisition module 704 is used to select data that meets the conditions from the target data based on the metadata of the target data, and send the data that meets the conditions to the user through the data acquisition interface.

[0130] In one possible implementation, the data acquisition request is also used to indicate the semantic category to which the target data belongs. The cloud management platform also includes: a third notification module for notifying the storage node to send the target second chunk to the cloud management platform; and a third acquisition module for acquiring the second semantic category to which the target data belongs from the target second chunk and sending the second semantic category to the user through the data acquisition interface, wherein the second semantic category includes the first semantic category.

[0131] In one possible implementation, the cloud management platform further includes: a second receiving module, used to receive a data upload request sent by a user through a data upload interface, wherein the data upload request includes multiple data from the user; a second identification module, used to perform semantic recognition on the multiple data included in the data upload request to obtain the semantic categories of the multiple data; a generation module, used to divide the multiple data into multiple first chunks based on the semantic categories of the multiple data, and generate tensors based on the multiple first chunks; and a storage module, used to notify storage nodes to store the tensors.

[0132] In one possible implementation, the data upload request also includes metadata for multiple data items. The cloud management platform further includes: a construction module, used to divide the semantic categories of multiple data items into multiple second chunks, divide the metadata of multiple data items into multiple third chunks, and construct the correspondence between multiple first chunks, multiple second chunks, and multiple third chunks; and a generation module, used to generate tensors based on multiple first chunks, multiple second chunks, multiple third chunks, and the correspondence.

[0133] In one possible implementation, storage nodes include physical servers, virtual machines, containers, microvirtual machines, or bare metal servers.

[0134] It should be noted that the information interaction and implementation process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in the embodiments of this application, and will not be repeated here.

[0135] Please refer to Figure 8, which is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 8, the computing device 800 (which can be used to present the aforementioned cloud management platform) includes: a processor 801, a memory 802, a communication interface 803, and a bus 804. The processor 801, the memory 802, and the communication interface 803 are coupled through the bus (not labeled in the figure). The memory 802 stores instructions. When the execution instructions in the memory 802 are executed, the computing device 800 executes the method performed by the cloud management platform in the above method embodiment.

[0136] The computing device 800 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together to implement a system-on-a-chip (SOC).

[0137] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0138] The memory 802 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0139] The memory 802 stores executable program code, and the processor 801 executes this executable program code to implement the functions of the aforementioned first receiving module, first identification module, first notification module, and first acquisition module, thereby realizing the aforementioned data acquisition method based on the cloud management platform. That is, the memory 802 stores instructions for executing the aforementioned data acquisition method based on the cloud management platform.

[0140] The communication interface 803 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 800 and other devices or communication networks.

[0141] In addition to the data bus, the 804 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be divided into address bus, data bus, and control bus.

[0142] Please refer to Figure 9, which is a schematic diagram of a computing device cluster provided in an embodiment of this application. As shown in Figure 9, the computing device cluster 900 includes at least one computing device 800.

[0143] As shown in Figure 9, the computing device cluster 900 includes at least one computing device 800. The memory 802 of one or more computing devices 800 in the computing device cluster 900 may store the same instructions for executing the aforementioned data acquisition method based on the cloud management platform.

[0144] In some possible implementations, the memory 802 of one or more computing devices 800 in the computing device cluster 900 may also store partial instructions for executing the data acquisition method based on the cloud management platform described above. In other words, a combination of one or more computing devices 800 can jointly execute the data acquisition method based on the cloud management platform described above.

[0145] It should be noted that the memory 802 in different computing devices 800 within the computing device cluster 900 can store different instructions, which are used to execute certain functions of the aforementioned cloud management platform. That is, the instructions stored in the memory 802 of different computing devices 800 can implement the functions of one or more modules, such as the first receiving module, the first identification module, the first notification module, and the first acquisition module.

[0146] In some possible implementations, one or more computing devices 800 in the computing device cluster 900 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0147] Please refer to Figure 10, which is a schematic diagram of computer devices in a computer cluster provided in this application embodiment being connected via a network. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0148] In one possible implementation, the memory in computing device 800A stores instructions for performing the functions of modules such as the first receiving module. Meanwhile, the memory in computing device 800B stores instructions for performing the functions of modules such as the first identification module, the first notification module, and the first acquisition module.

[0149] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices. Similarly, the functions of computing device 800B can also be performed by multiple computing devices.

[0150] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform the steps executed by the cloud management platform in the embodiment shown in FIG4.

[0151] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the steps performed by the cloud management platform in the embodiment shown in FIG4.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data acquisition method based on a cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes storage nodes that store user tensors. Each tensor includes multiple first data chunks, and each first chunk includes data belonging to at least one semantic category. The method includes: The cloud management platform receives a data acquisition request sent by the user through a data acquisition interface, wherein the data acquisition request is used to indicate the target data to be acquired; The cloud management platform performs semantic recognition on the data acquisition request to obtain the first semantic category to which the target data belongs; The cloud management platform notifies the storage node to determine the target first chunk that matches the first semantic category from the plurality of first chunks, and sends the target first chunk to the cloud management platform; The cloud management platform obtains the target data from the target first chunk and sends the target data to the user through the data acquisition interface.

2. The method according to claim 1, characterized in that, The tensor further includes multiple second chunks, each second chunk corresponding to at least one first chunk. Each second chunk includes the semantic category to which the data in its corresponding first chunk belongs. The cloud management platform notifies the storage node to determine, from the multiple first chunks, the first chunk that matches the first semantic category, including: The cloud management platform notifies the storage node to determine a target second chunk that matches the first semantic category from a plurality of second chunks, and to determine a target first chunk that corresponds to the target second chunk from a plurality of first chunks.

3. The method according to claim 2, characterized in that, The tensor further includes multiple third chunks, each third chunk corresponding to at least one first chunk, each third chunk including metadata of the data included in its corresponding first chunk, the data acquisition request further being used to indicate the metadata of the target data, and the method further includes: The cloud management platform notifies the storage node to determine the target third chunk corresponding to the target first chunk from the plurality of third chunks, and sends the target third chunk to the cloud management platform; The cloud management platform obtains the metadata of the target data from the target third chunk, and sends the metadata of the target data to the user through the data acquisition interface.

4. The method according to claim 3, characterized in that, The metadata of the target data includes at least one of the following: the tag of the target data, the generation time of the target data, the size of the target data, and the identifier of the target data.

5. The method according to claim 3 or 4, characterized in that, The data acquisition request is also used to indicate the conditions set by the user, and the cloud management platform sends the target data to the user through the data acquisition interface, including: The cloud management platform selects data that meets the conditions from the target data based on the target data's metadata, and sends the data that meets the conditions to the user through the data acquisition interface.

6. The method according to any one of claims 2 to 5, characterized in that, The data acquisition request is further used to indicate the semantic category to which the target data belongs, and the method further includes: The cloud management platform notifies the storage node to send the target second chunk to the cloud management platform; The cloud management platform obtains a second semantic category containing the target data from the target second chunk, and sends the data containing the second semantic category to the user through the data acquisition interface, wherein the second semantic category contains the first semantic category.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The cloud management platform receives a data upload request sent by the user through a data upload interface, wherein the data upload request includes multiple data items of the user; The cloud management platform performs semantic recognition on the multiple data included in the data upload request to obtain the semantic categories of the multiple data. The cloud management platform divides the multiple data into multiple first chunks based on the semantic categories of the multiple data, and generates the tensor based on the multiple first chunks; The cloud management platform notifies the storage node to store the tensor.

8. The method according to claim 7, characterized in that, The data upload request also includes metadata of the plurality of data, and the method further includes: The cloud management platform divides the semantic categories of the multiple data into multiple second chunks, divides the metadata of the multiple data into multiple third chunks, and constructs the correspondence between the multiple first chunks, the multiple second chunks, and the multiple third chunks; The cloud management platform generates the tensor based on the multiple first chunks, including: The cloud management platform generates the tensor based on the plurality of first chunks, the plurality of second chunks, the plurality of third chunks, and the corresponding relationships.

9. The method according to any one of claims 1 to 8, characterized in that, The storage nodes include physical servers, virtual machines, containers, microvirtual machines, or bare metal servers.

10. A cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure includes storage nodes that store user tensors. Each tensor includes multiple first data chunks, and each first chunk includes data belonging to at least one semantic category. The cloud management platform includes: The first receiving module is configured to receive a data acquisition request sent by the user through a data acquisition interface, wherein the data acquisition request is used to indicate the target data to be acquired; The first identification module is used to perform semantic recognition on the data acquisition request to obtain the first semantic category to which the target data belongs; The first notification module is used to notify the storage node to determine the target first chunk that matches the first semantic category from the plurality of first chunks, and send the target first chunk to the cloud management platform; The first acquisition module is used to acquire the target data from the target first chunk and send the target data to the user through the data acquisition interface.

11. The cloud management platform according to claim 10, characterized in that, The tensor also includes multiple second chunks, each second chunk corresponding to at least one first chunk. Each second chunk includes the semantic category to which the data in its corresponding first chunk belongs. The first notification module is used to notify the storage node to determine a target second chunk that matches the first semantic category from the multiple second chunks, and to determine a target first chunk that corresponds to the target second chunk from the multiple first chunks.

12. The cloud management platform according to claim 11, characterized in that, The tensor further includes multiple third chunks, each third chunk corresponding to at least one first chunk. Each third chunk includes metadata of the data included in its corresponding first chunk. The data acquisition request is also used to indicate the metadata of the target data. The cloud management platform further includes: The second notification module is used to notify the storage node to determine the target third chunk corresponding to the target first chunk from the plurality of third chunks, and to send the target third chunk to the cloud management platform; The second acquisition module is used to acquire the metadata of the target data from the target third chunk, and send the metadata of the target data to the user through the data acquisition interface.

13. The cloud management platform according to claim 11, characterized in that, The metadata of the target data includes at least one of the following: the tag of the target data, the generation time of the target data, the size of the target data, and the identifier of the target data.

14. The cloud management platform according to claim 12 or 13, characterized in that, The data acquisition request is also used to indicate the conditions set by the user. The first acquisition module is used to select data that meets the conditions from the target data based on the metadata of the target data, and send the data that meets the conditions to the user through the data acquisition interface.

15. The cloud management platform according to any one of claims 11 to 14, characterized in that, The data acquisition request is also used to indicate the semantic category to which the target data belongs, and the cloud management platform further includes: The third notification module is used to notify the storage node to send the target second chunk to the cloud management platform; The third acquisition module is used to acquire a second semantic category containing the target data from the target second chunk, and send the data containing the second semantic category to the user through the data acquisition interface, wherein the second semantic category contains the first semantic category.

16. The cloud management platform according to any one of claims 10 to 15, characterized in that, The cloud management platform also includes: The second receiving module is used to receive a data upload request sent by the user through a data upload interface, wherein the data upload request includes multiple data of the user; The second identification module is used to perform semantic recognition on the plurality of data included in the data upload request to obtain the semantic category of the plurality of data; A generation module is used to divide the multiple data into multiple first chunks based on the semantic categories of the multiple data, and generate the tensor based on the multiple first chunks; A storage module is used to notify the storage node to store the tensor.

17. The cloud management platform according to claim 16, characterized in that, The data upload request also includes metadata of the multiple data sets, and the cloud management platform also includes: A construction module is used to divide the semantic categories of the multiple data into multiple second chunks, divide the metadata of the multiple data into multiple third chunks, and construct the correspondence between the multiple first chunks, the multiple second chunks, and the multiple third chunks; The generation module is used to generate the tensor based on the plurality of first chunks, the plurality of second chunks, the plurality of third chunks, and the corresponding relationship.

18. The cloud management platform according to any one of claims 10 to 17, characterized in that, The storage nodes include physical servers, virtual machines, containers, microvirtual machines, or bare metal servers.

19. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 9.

20. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 9.

Images