Data compression method and related device

Abstract

Provided in the present application are a data compression method and a related device, which are used for improving compression gain. The data compression method comprises: acquiring M binary sequences, the M binary sequences comprising a first data set, the first data set comprising M first data segments, the bit positions of the M first data segments across the M binary sequences being the same, and M being an integer greater than 1, wherein in the first data set, the number of first data segments having a value of v1 is n1, and the value of each first data segment is determined by means of bits comprised in the first data segment; and compressing the first data segments having the value of v1 into second data segments having a value of m1, so as to obtain n1 second data segments, with m1 indicating the ranking of the number of first data segments having the value of v1 in the first data set.

Description

Data compression methods and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411925057.2, filed on December 23, 2024, entitled “Data Compression Method and Related Equipment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and in particular to data compression methods and related equipment. Background Technology

[0003] With the development of computer technology, large-scale data may emerge in various scenarios, such as the training and inference of large models, the widespread adoption of databases, and cloud service applications. However, the update speed of hardware storage devices is relatively slow and cannot keep up with the rapid growth of large-scale data. Therefore, how to reduce the storage space occupied by large-scale data has become an urgent problem to be solved.

[0004] One data compression method employs entropy distribution for encoding compression. Specifically, it involves statistically analyzing the probability or frequency of different data values ​​in a batch of data and constructing a binary tree. In the binary tree, the leaf nodes represent the probability of each data value, and the parent node represents the sum of the probabilities of its child nodes. Each hop from the root node to a leaf node is either 0 or 1, thus obtaining a unique encoded value representing the data corresponding to each leaf node. This unique encoded value is then used to replace the data, resulting in the compressed data.

[0005] In this method, if the path between the leaf node and the root node is long, the length of the encoded value may exceed the length of the data it represents. The storage space occupied by the encoded value is greater than the storage space of the original data, resulting in reduced compression benefits. Summary of the Invention

[0006] This application provides a data compression method and related equipment for improving the benefits of data compression.

[0007] Firstly, this application provides a data compression method, including:

[0008] In this application, the compression object is a data segment with the same bit position in multiple binary sequences. The compression result indicates the quantity order of the data segment in the dataset, and the quantity order is determined based on the value of the data segment. The value of the data segment is determined by the number of bits included in the data segment; for example, the value of 100 is represented as 4 in decimal. For example, M binary sequences are obtained, which include a first dataset. The first dataset includes M first data segments, and these M first data segments have the same bit position in the M binary sequences. That is, one first data segment is selected from each binary sequence, and the bit positions of these M first data segments are the same; in other words, the number of bits in these M first data segments is the same. Here, M is a positive integer greater than 1. In the first dataset, there are n1 first data segments with the value v1. These n1 first data segments are compressed into n1 second data segments, each with the value m1. m1 represents the quantity order of the first data segment with the value v1 in the first dataset, which can be understood as the order value. The value of each of the n1 second data segments obtained by compressing the first data segments is m1, which also means that the values ​​of the second data segments obtained by compressing the first data segments with the same value are also the same.

[0009] In this application, the compression target is data segments within a binary sequence, reducing the amount of data processed and thus improving compression efficiency. Furthermore, the compression operation is based on the order of the data segments' frequency within the corresponding dataset. Since the data segments in the dataset share the same bit positions across multiple binary sequences, they typically exhibit similar characteristics, such as similar values ​​or concentrated value distribution. This limits the number of values ​​that can be taken from the data segments in the dataset. Since the compressed data segments represent the order of frequency of data segments with the same value within the dataset, the compressed data segments occupy fewer bits than the original data segments, thereby reducing storage space usage and maximizing compression benefits.

[0010] In some optional implementations of the first aspect, the datasets included in the multiple binary sequences may include data segments with multiple values. For example, the first dataset includes a first data segment with value v1 and n2 first data segments with value v2. These n2 first data segments are compressed into n2 second data segments, each with value m2, where m2 represents the order of the number of first data segments with value v2 in the first dataset. The quantity relationships of first data segments with different values ​​in the first dataset are different, resulting in different values ​​for the compressed second data segments. In schemes where n2 is greater than n1, m2 is less than m1. That is, in the first dataset, the more numerous the first data segments with the same value, the smaller the value of the compressed second data segment. If n2 equals n1, the value of the second data segment is determined by combining the values ​​of the first data segments. For example, a smaller first data segment is defined as having a smaller order value; that is, in schemes where n2 equals n1 and v2 equals v1, m2 is less than m1. In summary, when compressing the first data segment into the second data segment, priority is given to the first data segment with the most identical values ​​in the first dataset. The more identical values ​​there are, the higher the ranking of the corresponding values, and the smaller the ranking value. For first data segments with different values ​​but the same number of values, the ranking value is defined based on the values ​​of the first data segment. This ensures a one-to-one correspondence between the values ​​of the first data segment and the values ​​of the second data segment; first data segments with the same values ​​correspond to second data segments with the same values, and first data segments with different values ​​correspond to second data segments with different values.

[0011] In this application, the values ​​of the compressed second data segments are determined by sorting the first data segments with the same value from most to least numerous. This ensures that the more numerous the first data segments with the same value, the smaller the value of the resulting second data segment. Setting a smaller sorting value for data segments with a large number of the same value is beneficial for further reducing storage space and improving compression efficiency. Furthermore, for first data segments with the same number but different values, they are compressed into second data segments with different values, ensuring a one-to-one correspondence between the values ​​of the first and second data segments, thus guaranteeing the accuracy of data compression.

[0012] In some optional implementations of the first aspect, the entropy of the dataset containing the data segment to be compressed (i.e., the dataset to be compressed) is less than the entropy threshold. For example, the entropy of the first dataset is less than the entropy threshold. Entropy reflects the amount of data; lower entropy indicates a smaller data volume, resulting in a larger compressed space and greater compression benefits. The entropy threshold can be set based on the needs of the actual application, and is not specifically limited here.

[0013] In this application, the dataset to be compressed can be determined by calculating its entropy. Since entropy reflects the amount of data in the dataset, it further indicates the potential for compression. Compressing data segments in the dataset with entropy less than the entropy threshold ensures that the compression is beneficial. Furthermore, the entropy threshold can be flexibly set, further enriching the implementation methods of the technical solution in this application.

[0014] In some optional implementations of the first aspect, all data segments in the compressed dataset are to be compressed. These data segments can be compressed either all at once or in batches. For example, in the latter approach, the aforementioned first dataset includes a first data subset, which contains N first data segments, where N is an integer greater than 1 and less than M. Within the first data subset, there also exists a first data segment with the value v1, and the number of first data segments with the value v1 among these N first data segments is the smallest in the first dataset. This means that among the N first data segments, the first data segment with the value v1 has the largest sort value. The N first data segments are compressed to obtain N second data segments, and the length of each second data segment is m1 times the number of effective bits. In other words, the length of each of the N second data segments corresponding to the first data subset is the same, and this length is the number of effective bits corresponding to the largest sort value among the N first data segments included in the first data subset. Furthermore, the length of the second data segment is less than the length of the first data segment. The number of effective bits in m1 can be understood as the minimum number of bits required to express m1 in binary form. For example, if m1 is 8, the number of effective bits is 1000, and the number of bits is 3.

[0015] In this application, the first data segment is compressed into the corresponding sorted value, reducing the storage space occupied. Compressing N first data segments comprising a data subset into N second data segments of the same length, i.e., using a single standard for compression, accelerates the compression time.

[0016] In some alternative implementations of the first aspect, metadata corresponding to the data subset is also stored, indicating the length of each second data segment corresponding to the data subset. This allows the length of each second data segment to be determined based on the metadata during decompression, thereby enabling accurate decompression.

[0017] In some optional implementations of the first aspect, a mapping set is also stored, indicating a one-to-one correspondence between the value of the first data segment in the first dataset and the sort value of the first data segment. The sort value of the first data segment indicates the order of the number of first data segments with the same value in the first dataset, that is, it indicates the value of the second data segment corresponding to that first data segment.

[0018] In this application, the mapping set corresponds to the dataset. Therefore, the overhead of storing one mapping set is sufficient to compress the first data segment included in the corresponding dataset. During decompression, the decompression value corresponding to the second data segment can also be determined based on this mapping set, reducing the consumption of storage resources. Furthermore, the one-to-one correspondence between the values ​​of the first and second data segments indicated by the mapping set ensures the accuracy of data compression.

[0019] In some optional implementations of the first aspect, the content of the mapping set can have several possibilities: Optionally, the mapping set includes the value of the first data segment in the first dataset and the sorting value of the first data segment for each value. Optionally, the mapping set includes the value of the first data segment in the first dataset and the number of first data segments for each value in the first dataset. Optionally, the mapping set includes the value of the first data segment in the first dataset, the number of first data segments for each value in the first dataset, and the sorting value of the first data segment.

[0020] In this application, the mapping relationship set includes a variety of possibilities, which enriches the implementation methods and application scenarios of the technical solution and enhances the flexibility of the technical solution.

[0021] In some optional implementations of the first aspect, after obtaining the M binary sequences, the numerical distribution of the first dataset is also obtained. The numerical distribution indicates the number of first data segments for each value in the first dataset. The method for obtaining the numerical distribution of the first dataset varies depending on the scenario. If the current computational conditions satisfy the truth value calculation conditions, then the number of first data segments for each value in the first dataset is calculated, which is equivalent to obtaining the truth value of the numerical distribution of the first dataset. If the current computational conditions do not satisfy the truth value calculation conditions, then the number of first data segments for each value in the first dataset is estimated based on historical numerical distribution information. Here, historical numerical distribution information indicates the number of data segments for each value in Z historical data segments, where the bit positions of the Z historical data segments are the same as the bit positions of the M first data segments, and Z is an integer greater than 1. In other words, the estimation of the numerical distribution of the first dataset is based on the numerical distribution of multiple historical data segments with the same bit positions as the first dataset. Since data segments with the same bit positions often have the same or similar characteristics, the error range is within an acceptable range when estimating based on these Z historical data segments.

[0022] In this application, there are multiple possible ways to obtain the numerical distribution of the first dataset, enriching the implementation methods of the technical solution. Furthermore, in solutions that do not meet the true value calculation conditions, the numerical distribution of the first dataset can be predicted based on historical numerical distribution information, avoiding the computational overhead caused by frequently calculating the true value of the numerical distribution and saving computational resources.

[0023] In some optional implementations of the first aspect, the current calculation conditions satisfy the truth value calculation conditions, including: the current time is at the beginning of the next calculation cycle, or the interval between the current calculation step and the calculation step corresponding to the previous truth value is a preset interval.

[0024] In this application, there are multiple possibilities for truth value calculation conditions, which can be flexibly applied to different application scenarios, thus enriching the implementation methods of the technical solution of this application.

[0025] In some optional implementations of the first aspect, the number of data segments included in each data subset can also be determined based on instructions. For example, a target instruction is obtained, indicating the number of N. Based on the target instruction, the first dataset is divided into multiple data subsets. The target instruction can be triggered by the user, such as the user setting the number of N; it can also be set by operations personnel referring to the computing power settings of the hardware; in addition, it can be set based on other methods, such as combining historical parameters and setting the range of N values ​​in system parameters, etc., which are not specifically limited here.

[0026] In some alternative implementations of the first aspect, multiple data segments of the binary sequence can be compressed separately. The compression principle for each data segment is the same. For example, the aforementioned M binary sequences also include a second dataset, which includes M third data segments. The M third data segments have the same bit position in the M binary sequences and do not overlap with the bit positions of the first data segments. In the second dataset, the number of third data segments with the value x1 is n3. Compressing the third data segments with the value x1 into fourth data segments with the value y1 results in n3 fourth data segments, where y1 indicates the order of the number of third data segments with the value x1 in the second dataset.

[0027] In this application, different data segments of a binary sequence can be compressed separately, further reducing the storage space occupied and increasing the compression benefits.

[0028] Secondly, this application provides a data decompression method, including:

[0029] Obtain the compressed sequence, which includes N second data segments. The value of each second data segment indicates the order of the corresponding first data segment in the first dataset; in other words, the value of the second data segment is the sorting value of the first data segment. The first dataset includes M first data segments with the same bit position in M ​​binary sequences, where M is an integer greater than 1 and N is an integer greater than 1 and less than or equal to M. The first data segments can also be understood as the data segments before compression. Obtain metadata, which indicates the length of each second data segment in the compressed sequence. Based on the metadata, determine the value of each second data segment. Obtain a mapping set, which indicates a one-to-one correspondence between the values ​​of the first and second data segments, that is, a one-to-one correspondence between the values ​​of the first and second data segments, i.e., a one-to-one correspondence between the values ​​of the first and second data segments. Then, based on the value of each second data segment and the mapping set, determine the value of the first data segment corresponding to each second data segment to complete the decompression.

[0030] In this application, the sorting value of the original data segment indicated by the mapping relationship set corresponds one-to-one with the value of the original data segment, so that one sorting value uniquely corresponds to one original data segment, ensuring that the decompression result is error-free and realizing lossless decompression.

[0031] In some optional implementations of the second aspect, the content of the mapping set can have several possibilities: Optionally, the mapping set includes the value of the first data segment in the first dataset and the sorting value of the first data segment for each value. Optionally, the mapping set includes the value of the first data segment in the first dataset and the number of first data segments for each value in the first dataset. Optionally, the mapping set includes the value of the first data segment in the first dataset, the number of first data segments for each value in the first dataset, and the sorting value of the first data segment.

[0032] In this application, the mapping relationship set includes a variety of possibilities, which enriches the implementation methods and application scenarios of the technical solution and improves the flexibility of the solution.

[0033] Thirdly, this application provides a data compression apparatus, comprising:

[0034] The acquisition unit is used to acquire M binary sequences. The M binary sequences include a first dataset, which in turn includes M first data segments. These M first data segments have the same bit position in the M binary sequences, where M is an integer greater than 1. In the first dataset, the number of first data segments with the value v1 is n1, and the value of each first data segment is determined by the number of bits it contains.

[0035] The processing unit is used to compress a first data segment with a value of v1 into a second data segment with a value of m1, resulting in n1 second data segments, where m1 indicates the number of the first data segments with a value of v1 in the first dataset.

[0036] The data compression device is used to implement the method shown in the first aspect or any possible implementation of the first aspect, and its beneficial effects are as described above, and will not be repeated here.

[0037] Fourthly, this application provides a data decompression apparatus, comprising:

[0038] The acquisition unit is used to acquire a compressed sequence, which includes N second data segments. The value of each second data segment indicates the order of the corresponding first data segment in the first dataset. The first dataset includes M first data segments, where M is an integer greater than 1 and N is an integer greater than 1 and less than or equal to M. Meta-information is also acquired, indicating the length of each second data segment.

[0039] The processing unit is used to determine the value of each second data segment based on the metadata.

[0040] The acquisition unit is also used to acquire a mapping relationship set, which indicates that the values ​​of the first data segment and the values ​​of the second data segment correspond one-to-one.

[0041] The processing unit is also used to determine the value of the first data segment corresponding to each second data segment based on the value and mapping relationship set of each second data segment.

[0042] The data decompression apparatus is used to implement the method shown in the second aspect above, or any possible implementation of the second aspect, and its beneficial effects are as described above, and will not be repeated here.

[0043] Fifthly, this application provides a computing device including a processor and a memory. The processor of the computing device is used to execute instructions stored in the memory, causing the computing device to implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0044] Sixthly, this application provides a computing device cluster including at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster implements the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0045] In a seventh aspect, this application provides a computer program product containing instructions that, when executed on a processor, implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect; or, when the instructions are run by a cluster of computer devices, cause the cluster of computer devices to implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0046] Eighthly, this application provides a computer-readable storage medium storing computer program instructions that, when executed on a processor, implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect; or, when executed by a cluster of computer devices, cause the cluster of computer devices to implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0047] The beneficial effects shown in any of the fifth to eighth aspects are similar to those of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0048] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0049] Figure 2 is a schematic diagram of another system architecture provided in an embodiment of this application;

[0050] Figure 3 is a flowchart illustrating a data compression method provided in an embodiment of this application;

[0051] Figure 4 is a schematic diagram provided in an embodiment of this application;

[0052] Figure 5 is another flowchart illustrating the data compression method provided in an embodiment of this application;

[0053] Figure 6 is another flowchart illustrating the data compression method provided in an embodiment of this application;

[0054] Figure 7 is a schematic diagram provided in an embodiment of this application;

[0055] Figure 8 is a schematic diagram of the compression result provided in an embodiment of this application;

[0056] Figure 9 is another schematic diagram of the compression result provided in the embodiment of this application;

[0057] Figure 10 is a flowchart illustrating a data decompression method provided in an embodiment of this application.

[0058] Figure 11 is a flowchart provided in an embodiment of this application;

[0059] Figure 12 is a structural schematic diagram of a data compression device provided in an embodiment of this application;

[0060] Figure 13 is a schematic flowchart of numerical distribution estimation provided in an embodiment of this application;

[0061] Figure 14 is a schematic diagram of a data compression device provided in an embodiment of this application;

[0062] Figure 15 is a schematic diagram of a data decompression apparatus provided in an embodiment of this application;

[0063] Figure 16 is another structural schematic diagram of the computing device provided in an embodiment of this application;

[0064] Figure 17 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0065] Figure 18 is another schematic diagram of the computing device cluster provided in the embodiment of this application. Detailed Implementation

[0066] This application provides a data compression method and related equipment to improve the benefits of data compression.

[0067] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0068] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0069] First, please refer to Figures 1 and 2, which are schematic diagrams of the system architecture provided in the embodiments of this application.

[0070] As shown in Figure 1, a communication connection is established between the terminal device and the server. Based on this connection, the terminal device accesses data stored on the server or uses services provided by the server. In the data compression method provided in this embodiment, the user can trigger the server to perform a data compression operation through the terminal device. The server responds to the instruction from the terminal device and, based on the instruction, compresses the data stored on the server or the data managed by the server. For example, in a data writing scenario, the server compresses the data written by the user through the terminal device.

[0071] In the data decompression method provided in this application embodiment, a user can trigger a server to perform a data decompression operation through a terminal device. The server responds to the instruction from the terminal device and, based on the instruction, decompresses the data stored on the server or the data managed by the server. For example, in a data reading scenario, the server decompresses the compressed data and returns it to the terminal device.

[0072] Optionally, the server can also implement data compression and / or decompression itself. For example, in the training or inference tasks of artificial intelligence (AI) models, the server can compress and decompress the parameters generated by model iteration. The specific implementation process of data compression and decompression will be explained in detail later and will not be elaborated here.

[0073] Furthermore, the server is equipped with a computing card that performs data compression or decompression methods. The computing card includes one or more processors, which can be homogeneous or heterogeneous. For example, a computing card may include multiple central processing units (CPUs), which can have the same or different specifications. Another example is that a computing card may include a graphics processing unit (GPU) and a neural network processing unit (NPU). In addition, the computing card may also include other types of processors, such as tensor processing units (TPUs), etc., which are not limited here.

[0074] In AI model training or inference tasks, the computing card compresses and decompresses the data generated during model iteration or inference. The compressed data can be stored in the server's storage space or on other devices connected to the server; the specific storage method is not limited here.

[0075] In some optional implementations, the technical solutions provided in this application can also be applied in cloud computing scenarios. In other words, the technical solutions provided in this application can also be implemented based on cloud services. As shown in Figure 2, a tenant logs into the cloud service platform 30 via terminal 10 through the Internet 20 using an account and password registered on the cloud service platform 30. The cloud service platform 30 manages infrastructure, which includes multiple data centers located in different regions. For example, region 1 in Figure 2 includes cloud data center 1 and cloud data center 2, and region 2 includes cloud data center 3 and cloud data center 4. Each cloud data center has multiple servers, and the servers run business instances (including at least one of virtual machines, containers, and dedicated hosts).

[0076] In this embodiment, the business instance deploys data compression and / or data decompression services. The tenant purchases cloud services from the cloud service platform 30 via terminal 10 and sends a request to the cloud service platform 30. This request is used to request cloud services from the cloud service platform 30. The specific content of the cloud services includes providing the tenant with data compression and / or data decompression methods, as described below.

[0077] It should be noted that the terminal device mentioned in the embodiments of this application can be a device with wireless transceiver function, or a terminal. Specifically, it can refer to user equipment (UE), access terminal, subscriber unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal devices can also be satellite phones, cellular phones, smartphones, wireless data cards, wireless modems, machine-type communication devices, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, communication devices mounted on high-altitude aircraft, wearable devices, drones, robots, terminals in device-to-device (D2D) communication, terminals in vehicle-to-everything (V2X) communication, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, mixed reality (MR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, and wireless terminals in smart cities. This application does not limit the scope of wireless terminals, such as those in a city, smart homes, or future communication networks.

[0078] In addition, in the embodiments of this application, there may be more or fewer terminal devices in the system. The number and type of terminal devices are determined according to actual needs, and are not limited here.

[0079] In summary, the data compression and decompression methods provided in this application can be applied to computing cards, accelerators, servers, virtual machines, cloud service platforms, etc. In the following description, a server is used as the execution subject as an example. The implementation process of other types of execution subjects is similar.

[0080] Please refer to Figure 3 below, which is a flowchart illustrating the data compression method provided in the embodiments of this application.

[0081] 301. Obtain M binary sequences, including a first dataset. The first dataset includes M first data segments with the same bit position in the M binary sequences. The number of first data segments with the value v1 in the first dataset is n1, where M is a positive integer greater than 1.

[0082] M binary sequences can be understood as data to be compressed. The data type can be various: it can be integer data or floating-point data; it can be vector or tensor, and no specific limitation is made here.

[0083] For example, in the training or inference of an AI model, the binary sequence can be activation values ​​generated during forward and backward computation, first or second-order momentum stored during the optimizer update phase, or data used during the gradient calculation phase, etc., without being limited here. For example, the binary sequence can also be data in a database, such as data representing banking transaction records.

[0084] It is understandable that the M binary sequences have the same length, which is also the number of bits used to store the binary sequence. It is possible that some data may have fewer than the required number of bits, meaning the higher-order bits may be 0. However, in data storage, storing binary sequences based on the same length, or the same physical unit, facilitates unified management of the binary sequences. Here, the physical unit is the data type of binary data identified by specific rules in computing devices, such as: 32-bit floating-point numbers (FP32), FP16, BF16, FP8, unsigned integers (unsigned int, UNIT), INT8, INT4, INT2, etc., without further specific restrictions here.

[0085] In this embodiment, after obtaining M binary sequences, the dataset to be compressed is determined from these M binary sequences. The dataset to be compressed includes M data segments, and these M data segments have the same bit position in the M binary sequences. This means that each binary sequence includes one data segment. In summary, this process can be understood as decomposing the M binary sequences into multiple datasets according to their bit positions, and then determining the dataset to be compressed from them. The process is explained in detail below with reference to the schematic diagram:

[0086] Please refer to Figure 4, which is a schematic diagram provided in an embodiment of this application. In the embodiment shown in Figure 4, there are M binary sequences, and it is assumed that the physical unit of the binary sequence is float32. Using the physical unit of UNIT8, each binary sequence is divided into 4 data segments, and the length of each data segment is 8 bits.

[0087] M data segments at the same bit position are considered as a dataset. In the embodiment shown in Figure 4, this includes four datasets: dataset 1 to dataset 4. That is, the M data segments in a dataset correspond one-to-one with M binary sequences, and each data segment is determined from one binary sequence. For example, dataset 1 shown in Figure 4 includes the data segment of the highest byte from the M binary sequences.

[0088] In this application, the dataset to be compressed can be determined from multiple datasets included in M ​​binary sequences based on the numerical distribution of the dataset or the sampled numerical distribution.

[0089] The numerical distribution of the dataset, i.e., the true numerical distribution of the current dataset, represents the number of data segments with each value in the current dataset. The numerical distribution of the dataset is calculated by statistically analyzing the values ​​of all data segments. The value of a data segment is determined by the number of bits it contains, and refers to the numerical value represented by the data segment. For example, the value of the data segment 10000000 is represented as 128 in decimal.

[0090] Sampling numerical distribution can involve sampling the historical numerical distribution of a dataset, or sampling the numerical distribution of a subset of the original data. The former approach can be understood as estimating the current numerical distribution of the dataset based on the sampled historical numerical distribution. This estimation method is possible because numerical distributions in batched data exhibit certain patterns. By summarizing historical information and drawing experience, the current numerical distribution of the dataset can be predicted, avoiding frequent calculations of the true value and reducing computational resource consumption. In the latter approach, a subset of the original data is sampled, and the numerical distribution of this subset is calculated. Then, the numerical distribution of the entire dataset containing this subset of original data is estimated based on its numerical distribution.

[0091] Furthermore, there are many possible parameters for selecting datasets to be compressed, including entropy, concentration ratio, Herfindahl-Hirschman index, maximum frequency percentage, number of valid data segments (i.e., data segments with a frequency > 0) in the dataset, variance, kurtosis, relative entropy, KL divergence, etc., which are not limited here.

[0092] For example, entropy can be information entropy, and an information entropy threshold can be set. By comparing the information entropy of a dataset with the information entropy threshold, datasets with information entropy less than the threshold are selected for compression. Information entropy reflects the amount of information; the lower the information entropy, the less information the data carries, and theoretically, the less storage space is required, resulting in greater compression gains. The information entropy threshold can be set based on the needs of practical applications; it's understood that the lower the information entropy, the higher the compression gains.

[0093] In this embodiment, the dataset to be compressed can be determined by calculating its entropy. Since entropy reflects the amount of data in a dataset, it further indicates the potential for compression. Compressing data segments within a dataset whose entropy is less than the entropy threshold ensures that the compression is beneficial. Furthermore, the entropy threshold can be flexibly set, further enriching the implementation methods of the technical solution in this application.

[0094] For example, let's take KL (Kullback-Leibler) divergence as an example. KL divergence, also known as relative entropy, represents the difference between the current numerical distribution and a uniform numerical distribution. The larger the KL divergence value, the larger the difference. Since the uniform numerical distribution has the highest information entropy, a larger difference indicates a lower information entropy for the current numerical distribution. A KL divergence threshold can be set. By comparing the KL divergence of the dataset with the KL divergence threshold, datasets with KL divergence greater than the threshold are selected as the datasets to be compressed.

[0095] For example, consider the number of valid data segments appearing in the dataset. The bit positions of the data segments in the dataset are the same. If the total number of valid data segments appearing in the dataset is less than or equal to half the number of values ​​that the bits in the data segments can represent, then the length of the maximum sorted value of all valid data segments appearing in the dataset is less than the length of the data segment. For example, suppose the length of a data segment in the dataset is 8 bits, and the range of values ​​that can be represented is [0, 255], including 256 values. If there are 128 possible values ​​for all valid data segments appearing in the dataset, then the maximum sorted value of these data segments is 128, which can be represented by only 7 bits, less than 8 bits. The sorted value indicates the order of the number of data segments with the same value in the dataset, which will be explained in detail in step 302 below.

[0096] Optionally, in practical applications, the quantity threshold can be set as the number of values ​​that the bits in the data segment of the dataset can represent. You can also set it to other values, as long as it is less than 1 / 2.

[0097] It should be noted that the method of selecting the dataset to be compressed from the multiple datasets included in the M binary sequences based on other parameters is similar to the implementation method in the previous example, and will not be repeated here.

[0098] In the embodiments of this application, the data segments included in the dataset to be compressed have a concentrated numerical distribution and can be presented in various ways, which will be described below.

[0099] In some optional implementations, the dataset to be compressed includes data segments with the same value. This means that multiple data segments in the dataset to be compressed have the same sort value. Therefore, the maximum sort value of the dataset to be compressed is less than the number of data segments included in the dataset. The more data segments with the same value, the smaller the maximum sort value, and the higher the compression benefit. There are several possibilities for the dataset to be compressed to include data segments with the same value:

[0100] Optionally, the values ​​of the data segments in the dataset to be compressed can be concentrated within a specific value range. For example, if the dataset to be compressed has a bit range of 8, the values ​​of the data segments included can all be in the range [110, 220]. It is understandable that the smaller the value range, the smaller the maximum sort value, and the higher the compression benefit.

[0101] Optionally, the number of values ​​in the data segments included in the dataset to be compressed is less than or equal to a preset number threshold. For example, if the dataset to be compressed has 8 bits and includes 125 values ​​in the data segments, this is less than or equal to the preset number threshold of 128. It is understandable that the smaller the preset number threshold, the smaller the maximum sorted value, and the higher the compression benefit.

[0102] In some optional implementations, the dataset to be compressed includes data segments with invalid bit fields. An invalid bit field refers to a high-order bit that is 0; the information represented by an invalid bit field is invalid or redundant. For example, the value 15 can be represented by binary 1111, but each data segment in the dataset is 8 bits long, so the value 15 is represented as 00001111. The highest 4 bits in this representation are invalid bits. As another example, the value 17 can be represented by binary 10001, but each data segment in the dataset is 8 bits long, so the value 17 is represented as 00010001. The highest 3 bits in this representation are invalid bits.

[0103] Understandably, the more data segments with invalid bit fields in the dataset to be compressed, the higher the compression benefit. The longer the invalid bit fields, the higher the compression benefit.

[0104] It should be noted that the embodiment shown in Figure 4 above is merely an example of dividing each of the M binary sequences into multiple data segments. In practical applications, there are many possible ways to decompose binary sequences.

[0105] Optionally, the decomposition method of the binary sequence can be selected based on the data type of the binary sequence and prior experience. For example, assuming the physical unit of the binary sequence is FP32, prior experience shows that decomposing it based on the physical unit of UNIT8 results in the lowest information entropy of the dataset. Therefore, the server decomposes the binary sequence based on the physical unit of UNIT8 based on this existing experience.

[0106] Optionally, the server can also iterate through multiple physical units and select the physical unit for decomposing the binary sequence. For example, suppose the physical unit of the binary sequence is FP32. The server can iterate through the binary sequence using various methods such as UNIT8, FP16, and UNIT4, and decompose the binary sequence accordingly. The server analyzes the datasets obtained from decomposition using different physical units to determine the final decomposition method. Furthermore, the server calculates the parameters of the datasets obtained from each physical unit decomposition and selects the decomposition method with the greatest compression benefit based on these parameters. These parameters include those mentioned earlier for filtering the datasets to be compressed; specific details are not limited here.

[0107] Optionally, when decomposing a binary sequence based on a physical unit, the server can decompose the binary sequence in other ways besides dividing each original data into multiple data segments as shown in Figure 4. For example, assuming the binary sequence is FP32, by traversing the binary sequence using UNI8, it can first decompose the first 1 to 8 bits of each binary sequence as a data segment, obtaining a first dataset containing M data segments. Then, it can decompose the second 2 to 9 bits of each binary sequence as a data segment, obtaining a second dataset containing M data segments. This process continues until the 25 to 32 bits of each binary sequence are decomposed as a data segment, obtaining a final dataset containing M data segments. Finally, based on the parameters of each dataset, the datasets to be compressed are selected.

[0108] The server determines the dataset to be compressed from M binary sequences, and further determines some or all of the data segments in the dataset to be compressed as a group of data segments, and compresses each group of data segments separately. For example, taking the first dataset as the dataset to be compressed, the first dataset includes M first data segments. Among these M first data segments, the number of first data segments with the value v1 is n1.

[0109] 302. Compress the first data segment with value v1 into a second data segment with value m1, resulting in n1 second data segments, where m1 indicates the order of the number of the first data segments with value v1 in the first dataset.

[0110] The server compresses a first data segment with a value of v1 into a second data segment with a value of m1, indicating that the second data segment obtained by compressing a first data segment with the same value also has the same value. Furthermore, the value m1 of the second data segment indicates the quantity ranking of the first data segment with the value v1 in the first dataset. In other words, in this embodiment, compressing a data segment essentially compresses the data segment to its ranking value within the corresponding dataset.

[0111] In this embodiment, the object of compression is a data segment within a binary sequence, reducing the amount of data processed and thus improving compression efficiency. Furthermore, the data segment compression operation is based on the order of the data segment's quantity within the corresponding dataset. Since the data segments in the dataset share the same bit positions across multiple binary sequences, they typically possess similar characteristics, such as similar values ​​or concentrated value distribution. This limits the number of values ​​in the data segments included in the dataset, and since the compressed data segment's value represents the order of the number of data segments with the same value within the dataset, the compressed data segment occupies fewer bits than the original data segment, thereby reducing storage space usage and maximizing compression benefits.

[0112] Furthermore, the sorting value indicates the order of the number of first data segments with the same value in the first dataset. The first dataset may include first data segments with multiple values, and the number of data segments with different values ​​may be the same or different. To ensure compression benefits while maintaining compression accuracy, this application embodiment defines prioritizing sorting based on the number of first data segments with the same value from most to least. For first data segments with the same number but different values, sorting is then performed in conjunction with the value of the first data segment.

[0113] For example, the first dataset processing includes not only a first data segment with a value of v1, but also n2 first data segments with a value of v2. The server compresses these n2 first data segments into n2 second data segments, each with a value of m2, where m2 represents the order of the number of first data segments with the value v2 in the first dataset. Specifically, in the case where n2 is greater than n1, m2 is less than m1. In the case where n2 equals n1 and v2 equals v1, m2 is less than m1. This ensures a one-to-one correspondence between the values ​​of the first and second data segments; first data segments with the same value have the same value in their corresponding second data segments, and first data segments with different values ​​have different values ​​in their corresponding second data segments.

[0114] In this application, the values ​​of the compressed second data segments are determined by sorting the first data segments with the same value from most to least numerous. This ensures that the more numerous the first data segments with the same value, the smaller the value of the resulting second data segment. Setting a smaller sorting value for data segments with a large number of the same value is beneficial for further reducing storage space and improving compression efficiency. Furthermore, for first data segments with the same number but different values, they are compressed into second data segments with different values, ensuring a one-to-one correspondence between the values ​​of the first and second data segments, thus guaranteeing the accuracy of data compression.

[0115] The preceding example provides a simple overview of the compression principle for a data segment with a specific value in the dataset to be compressed. The following section details the compression process for the data segments within the dataset to be compressed. In summary, the server determines the dataset to be compressed from M binary sequences, further identifies some or all of the data segments within the dataset as a group, and compresses each group of data segments separately.

[0116] In some optional implementations, all data segments in the dataset to be compressed are treated as a group of data segments for compression. For example, if the dataset to be compressed is a first dataset, and the number of first data segments with the value v1 in the first dataset is the smallest, then the M second data segments obtained by compressing the M first data segments in the first dataset will all have the same length, which is m1 bits of effective bits. m1 is the value of the second data segment obtained by compressing the first data segment with the value v1. Since the number of first data segments with the value v1 in the first dataset is the smallest, the sorting value of the first data segment with the value v1 in the M first data segments in the first dataset is the largest, and m1 has the longest effective bits.

[0117] In some optional implementations, a subset of data segments in the dataset to be compressed is considered as a group of data segments. In other words, the dataset to be compressed comprises multiple data subsets, each of which includes a group of data segments. For example, a first dataset comprises multiple data subsets, each containing N first data segments. Taking the first data subset as an example, there exists a first data segment with the value v1 within the first data subset, and the number of first data segments with the value v1 among these N first data segments is the smallest. Therefore, in the N second data segments obtained by compressing the N first data segments included in the first data subset, the length of each second data segment is m1 bits of effective bits, and the length of the second data segment is less than the length of the first data segment.

[0118] For example, please refer to Figure 5, which is a flowchart illustrating the data compression method provided in an embodiment of this application. In the embodiment shown in Figure 5, it is assumed that N is 16, meaning that the first data subset includes 16 first data segments. The server obtains the numerical distribution of the first dataset containing these 16 first data segments and constructs the dictionary M shown in Figure 5. The numerical distribution of the first dataset indicates the values ​​of the first data segments included in the first dataset and the number of each value of the first data segment in the first dataset. In dictionary M, the values ​​of the first data segments correspond one-to-one with the sorting values ​​of the first data segments. The sorting value of the first data segments indicates the order of the number of first data segments with the same value in the first dataset. The principle of quantity sorting has been explained above and will not be repeated here.

[0119] The server determines the sort value for each first data segment based on the dictionary M and the values ​​of the first data segments. For example, the sort value of a first data segment with a value of 1 is 1, and the sort value of a first data segment with a value of 5 is 4. Among these 16 first data segments, the first data segment with a value of 5 is the least common in the first dataset, and its corresponding sort value is 4. That is to say, the maximum sort value corresponding to the first data subset is 4, and the binary form of this maximum sort value is 100, with a length of 3 bits. Therefore, the server stores the sort value corresponding to each first data segment in 3-bit format. For example, a sort value of 1 is stored as 001, and a sort value of 0 is stored as 000.

[0120] It should be noted that in the embodiment shown in Figure 5, dictionary M is used to reflect the number of data segments in the dataset. In practical applications, it can also reflect the frequency of data segments in the dataset. For example, replacing the second column in dictionary M shown in Figure 5 with frequency works similarly to the previous explanation, and will not be elaborated further here.

[0121] In this application, the first data segment is compressed into the corresponding sorted value, reducing the storage space occupied. Compressing N first data segments comprising a data subset into N second data segments of the same length not only ensures that each sorted value can be stored completely, but also uses a standard for compression, accelerating the compression time.

[0122] The value of N can be specified by the user, configured by system parameters, or determined based on prior experience. In some optional implementations, the server obtains a target instruction indicating the quantity of N. The target instruction can be triggered by the system or by the user; this is not limited here.

[0123] Optionally, the value of N can be in the range [8, 64]. Optionally, the value of N can also be greater than 64. It is understandable that the value of N is related to the computing power of the hardware device; the more powerful the device, the larger the range of values ​​of N that it can theoretically support.

[0124] For example, suppose M is 256 and N is 64. This can be understood as the dataset to be compressed containing 4 groups of data segments, each group containing 64 data segments, and the server compressing each group of data segments separately.

[0125] The preceding text briefly explained that before compressing the data segment, it is necessary to obtain the numerical distribution of the dataset to be compressed. The following explanation uses the dataset to be compressed as the first dataset as an example, in conjunction with Figure 6. Please refer to Figure 6, which is a flowchart illustrating the data compression method provided in this embodiment of the application.

[0126] In summary, compressing N first data segments is achieved based on the numerical distribution of the first dataset containing these N first data segments. The numerical distribution of the first dataset reflects the quantity or frequency of each value in the first data segment. Based on this quantity or frequency, the sort value of each value in the first data segment can be determined. To ensure the accuracy of data compression, each value in the first data segment uniquely corresponds to a sort value, and each sort value in the first data segment uniquely corresponds to a value in the original data segment; that is, there is a one-to-one correspondence between the value and the sort value of the first data segment.

[0127] As shown in Figure 6, after the server acquires N first data segments (i.e., a set of data segments), it determines whether the current calculation conditions meet the truth value calculation conditions and selects different methods to obtain the numerical distribution of the first dataset. If the current calculation conditions meet the truth value calculation conditions, the numerical distribution of the first dataset containing these N first data segments is calculated. If the current calculation conditions do not meet the truth value calculation conditions, the numerical distribution of the first dataset containing these N first data segments is estimated based on historical truth value information.

[0128] The truth value calculation conditions include the current time being at the start of the next calculation cycle, or the interval between the current calculation step and the calculation step corresponding to the previous truth value being a preset interval. For example, in an AI training scenario, if data compression is performed during the forward-backward computation phase, each forward-backward computation is considered a completed calculation step. If it's during the optimizer update phase, each update of the optimizer's relevant parameters is considered a completed calculation step. The calculation cycle and preset interval can be set based on the needs of the actual application. In AI scenarios, they can also be flexibly changed based on different AI tasks; specific details are not limited here.

[0129] In a scheme where the current computational conditions satisfy the truth value calculation conditions, the server traverses all data segments in the first dataset and calculates the numerical distribution of the first dataset. The numerical distribution calculated at this time is the true numerical distribution of the first dataset, that is, the true value of the numerical distribution.

[0130] In schemes where current computational conditions do not meet the true value calculation conditions, the server estimates the numerical distribution of the first dataset based on historical true value information. Historical true value information, also known as historical numerical distribution information, indicates the number of data segments with each value in Z historical data segments. The bit positions of these Z historical data segments are the same as the bit positions of the M first data segments, and Z is an integer greater than 1. In other words, historical numerical distribution information is the historical numerical distribution information of some or all data segments in the first dataset containing these N first data segments. The server can call an estimation model or related functions to estimate the numerical distribution.

[0131] In this application's embodiments, there are multiple possibilities for obtaining the numerical distribution of the first dataset, enriching the implementation methods of the technical solution of this application. Furthermore, in solutions that do not meet the true value calculation conditions, the numerical distribution of the target dataset can be estimated based on historical numerical distribution information, avoiding the computational overhead caused by frequently calculating the true value of the numerical distribution and saving computational resources. Additionally, the true value calculation conditions are diverse, flexibly applicable to different application scenarios, further enriching the implementation methods of the technical solution of this application.

[0132] In this embodiment, the server compresses N first data segments. Besides obtaining the sorting values ​​(i.e., second data segments) of these N first data segments, it also stores metadata indicating the length of each second data segment. This allows for accurate decompression by determining the length of each second data segment based on the metadata.

[0133] For example, in the embodiment shown in Figure 5, the length of the maximum sort value is 3. The server stores this information as metadata, so that when the server decompresses a row of sort values ​​shown in Figure 5, it determines each sort value sequentially according to a length of 3 bits, that is, it determines each second data segment sequentially.

[0134] In some optional implementations, the server also stores a mapping set that indicates a one-to-one correspondence between the value of a first data segment in the first dataset and the sort value of the first data segment. That is, multiple first data segments with the same value correspond to the same sort value, and first data segments with different values ​​correspond to different sort values.

[0135] The sorting value of the first data segment indicates the order of the number of first data segments with the same value in the first dataset, which in turn indicates the value of the corresponding second data segment. Furthermore, the content of the mapping relationship set can have several possibilities:

[0136] Optionally, the mapping relationship set first includes the values ​​of the first data segment in the dataset and the sorting values ​​of the first data segment for each value. For example, in the embodiment shown in Figure 5, the mapping relationship set includes the first column and the third column of the dictionary M.

[0137] Optionally, the mapping set includes the value of the first data segment in the first dataset and the number of first data segments with each value in the first dataset. For example, in the embodiment shown in Figure 5, the mapping set includes the first column and the second column of dictionary M. Based on the number of first data segments with each value in the first dataset, the server further determines the sorting value of the first data segments with each value, and ensures that first data segments with different values ​​correspond to different sorting values.

[0138] Optionally, the mapping set includes the value of the first data segment in the first dataset and the frequency of the first data segment with each value in the first dataset. Based on the frequency of the first data segment with each value in the first dataset, the server further determines the sort value of the first data segment with each value, and ensures that the first data segment with different values ​​corresponds to different sort values.

[0139] Optionally, the mapping relationship set includes the value of the first data segment in the first dataset, the number of the first data segments with each value in the first dataset, and the sort value of the first data segment, and the value of the first data segment corresponds one-to-one with the sort value of the first data segment. For example, in the embodiment shown in Figure 5, the mapping relationship set includes the contents shown in dictionary M.

[0140] Optionally, the mapping relationship set includes the value of the first data segment in the first dataset, the frequency of each value of the first data segment in the first dataset, and the sorting value of the first data segment.

[0141] In this embodiment, the mapping relationship set corresponds to the dataset. Therefore, the overhead of storing one mapping relationship set is sufficient to compress the first data segment included in the corresponding dataset. During decompression, the decompression value corresponding to the second data segment can also be determined based on this mapping relationship set, reducing the consumption of storage resources. Furthermore, the values ​​of the first and second data segments indicated by the mapping relationship set are in a one-to-one correspondence, ensuring the accuracy of data compression. The mapping relationship set can include various possible contents, enriching the implementation methods and application scenarios of the technical solution in this application and improving the flexibility of the technical solution.

[0142] It's important to note that in schemes where the first dataset includes multiple sets of first data segments, for each set of first data segments, the N first data segments within that set are compressed into their corresponding sort values ​​to obtain the second data segment. The length of the second data segment is determined by the maximum sort value of the N first data segments included in that set within the first dataset. The maximum sort values ​​corresponding to the first data segments in different sets of first data segments may be the same or different; therefore, the lengths of the second data segments corresponding to different sets of first data segments may be the same or different. This is not specifically limited here.

[0143] In some alternative implementations, the server may compress an additional number of data segments for a single binary sequence. In other words, the M binary sequences may include one or more datasets to be compressed, and the bit segments of the data segments included in different datasets to be compressed are not the same and do not overlap.

[0144] In summary, the M binary sequences also include a second dataset, which contains M third data segments. These M third data segments have the same bit position in the M binary sequences and do not overlap with the bit positions of the first data segments. In the second dataset, the number of third data segments with the value x1 is n³. Compressing the third data segments with the value x1 into fourth data segments with the value y1 results in n³ fourth data segments, where y1 indicates the order of the number of third data segments with the value x1 in the second dataset. The principle of compressing the third data segments is similar to that of compressing the first data segments, and will not be repeated here.

[0145] In some alternative implementations, the server may also group some or all of the third data segments in the second dataset into a set of data segments. Each set of data segments includes X third data segments, where X may be the same as or different from N. The value of X can be set based on the needs of the actual application; optionally, the longer the original data segment, the larger the value of X can be.

[0146] For example, please refer to Figure 7, which is a schematic diagram provided in an embodiment of this application. In the example shown in Figure 7, assuming the physical unit of the binary sequence is FP32, each binary sequence is divided into 3 data segments based on physical units of UNIT8, UNIT8, and FP16 respectively. The data segments included in the M binary sequences are reassembled to obtain three datasets, dataset 5 to dataset 7. Based on the aforementioned method of filtering datasets to be compressed, the server determines that dataset 5 and dataset 7 are both datasets to be compressed. Among them, dataset 5 includes the data segment of the highest byte in the M binary sequences, and dataset 7 includes the data segment of the lowest two bytes in the M binary sequences.

[0147] Taking dataset 5 as the first dataset and dataset 7 as the second dataset as an example. Dataset 7 includes M third data segments. The server selects X third data segments from these as a group of data segments, where 2 ≤ X ≤ M. The method by which the server selects X third data segments is similar to the method for determining N first data segments from the first dataset described earlier, and will not be repeated here. After determining the X third data segments, the server compresses them into a fourth data segment. The length of the fourth data segment is less than the length of the third data segments. The fourth data segment is actually the sorted value of these X third data segments in dataset 7, and its length is the number of effective bits of the largest sorted value of these X third data segments in dataset 7. The principle by which the server compresses the third data segments into the fourth data segment is the same as the principle by which the first data segment is compressed into the second data segment, and will not be repeated here.

[0148] In this embodiment of the application, different data segments of the binary sequence can be compressed separately, further reducing the storage space occupied and increasing the compression benefits.

[0149] In this embodiment, the binary sequence is divided into multiple data segments. During compression, some or all of the data segments in the binary sequence may be compressed. After compressing multiple binary sequences, the arrangement of the compressed data can be varied, which will be explained below with reference to the schematic diagrams.

[0150] Please refer to Figures 8 and 9, which are schematic diagrams of the compression results provided in the embodiments of this application.

[0151] In the embodiment shown in Figure 8, six binary sequences are used as an example. These six binary sequences include six data segments to be compressed, numbered D1 to D6. Each binary sequence includes one data segment to be compressed and one uncompressed data segment. For example, the first binary sequence includes the data segment to be compressed, D1, and the uncompressed data segment, D1. The server compresses each data segment to be compressed, resulting in compressed data segments S1 to S6.

[0152] Optionally, the compression result can be as shown in Compression Result 1 in Figure 8, arranging the data segments included in the compression result according to the order of the data segments in the binary sequence. The corresponding uncompressed data segments in the binary sequence are arranged after each compressed data segment, with these six data segments to be compressed spaced apart. For example, the compression result of the first binary sequence includes compressed data segment S1 and uncompressed data segment 1. In the compression result, compressed data segment S1 and uncompressed data segment 1 are adjacent.

[0153] Optionally, the compression result can be as shown in compression result 2 in Figure 8. In the compression result, the compressed data segments are contiguous, and the uncompressed data segments in the binary sequence are arranged sequentially after the compressed data segments. For example, the compression result of the first binary sequence includes compressed data segment S1 and uncompressed data segment 1, and the compression result of the second data includes compressed data segment S2 and uncompressed data segment 2. In the compression result, compressed data segment S1 and compressed data segment S2 are adjacent, and uncompressed data segment 1 and uncompressed data segment 2 are adjacent.

[0154] In the embodiment shown in Figure 9, N binary sequences are used as an example. Each binary sequence includes two data segments to be compressed and two uncompressed data segments. For example, the first binary sequence includes data segments A1 and B1 to be compressed, and uncompressed data segments 1.1 and 1.2. The server compresses the N data segments A1 to AN to obtain compressed data segments C1 to CN, and compresses the N original data segments B1 to BN to obtain compressed data segments E1 to EN.

[0155] Optionally, the compression result can be as shown in Compression Result 1 in Figure 9, arranging the data segments included in the compression result according to the order of the data segments in the binary sequence. For example, the compression result of the first binary sequence includes compressed data segments C1 and E1, and uncompressed data segments 1.1 and 1.2. In the compression result, they are arranged in the order of compressed data segment C1, uncompressed data segment 1.1, compressed data segment E1, and uncompressed data segment 1.2.

[0156] Optionally, the compression result can be as shown in Compression Result 2 in Figure 9. The compression results of data segments with the same bit range can be arranged sequentially according to the order of the data segments in the binary sequence. That is, first arrange the compression results of the first bit range, i.e., arrange C1 to CN sequentially. Then arrange the compression results of the second bit range, i.e., arrange 1.1 to N.1 sequentially. Next, arrange the compression results of the third bit range, i.e., arrange E1 to EN sequentially. Finally, arrange the compression results of the fourth bit range, i.e., arrange 1.2 to N.2 sequentially. Here, a bit range refers to the position of a bit.

[0157] Optionally, the compression result can be as shown in compression result 3 in Figure 9. Based on the order of the data segments in the binary sequence, and arranging the compression results of data segments with the same bit segments sequentially, the compression results of the data segments to be compressed are prioritized. That is, the compression results of the first bit segment are arranged first, i.e., C1 to CN sequentially. Then the compression results of the third bit segment are arranged, i.e., E1 to EN sequentially. Next, the compression results of the second bit segment are arranged, i.e., 1.1 to N.1 sequentially. Finally, the compression results of the fourth bit segment are arranged, i.e., 1.2 to N.2 sequentially.

[0158] The preceding text introduced a data compression method provided in the embodiments of this application. This application also provides a data decompression method. Please refer to Figure 10, which is a flowchart illustrating the data decompression method provided in the embodiments of this application.

[0159] 1001. Obtain the compressed sequence, which includes N second data segments. The value of the second data segment indicates the order of the number of the corresponding first data segment in the first dataset. The first dataset includes M first data segments with the same bit position in M ​​binary sequences, where M is an integer greater than 1 and N is an integer greater than 1 and less than M.

[0160] In the data decompression method provided in this application embodiment, the data to be decompressed obtained by the server is a plurality of second data segments, i.e., a compression sequence, obtained based on the data compression method provided in the aforementioned application embodiment. Each second data segment included in the compression sequence indicates the quantity order of the corresponding first data segment in the first dataset.

[0161] The accompanying descriptions in Figures 8 and 9 illustrate various representations of the compression result. After compression, the server also stores compression mode information, which indicates the representation of the compression result and the start position of each second data segment. When decompressing data, the server can identify which parts of the compressed result are the second data segments based on the compression mode information, thus obtaining the compressed sequence.

[0162] 1002. Obtain metadata, which indicates the length of each second data segment.

[0163] The server also obtains metadata indicating the length of the second data segment. For example, in the embodiment shown in Figure 5, the metadata value is 3, indicating that the length of each second data segment in the compressed sequence is 3 bits.

[0164] 1003. Based on the metadata, determine the value of each second data segment.

[0165] Based on the length of the second data segment indicated by the metadata, the server determines the value of each second data segment, which is also the sorting value of the corresponding first data segment. For example, assuming the metadata is 3, the compressed sequence is 001000010011001001000100. By reading the second data segments in 3-bit increments, the values ​​are 1, 0, 2, 3, 1, 1, 0, 4, thus determining the sorting value of the first data segment before compression.

[0166] 1004. Obtain the mapping relationship set, which indicates that the value of the first data segment corresponds one-to-one with the sort value of the first data segment.

[0167] The server also retrieves a set of mapping relationships, which may include a variety of contents:

[0168] Optionally, the mapping relationship set includes the value of the first data segment in the first dataset and the sorting value of the first data segment for each value.

[0169] Optionally, the mapping relationship set includes the value of the first data segment in the first dataset, and the quantity or frequency of the first data segment with each value in the first dataset. Based on the quantity or frequency of the first data segment with each value in the first dataset, the server further determines the sorting value of the first data segment with each value, and ensures that the first data segment with different values ​​corresponds to different sorting values.

[0170] Optionally, the mapping relationship set includes the value of the first data segment in the first dataset, the quantity or frequency of each value of the first data segment in the first dataset, and the sorting value of the first data segment.

[0171] 1005. Based on the values ​​of each second data segment and the mapping relationship set, determine the value of the first data segment corresponding to the value of each second data segment.

[0172] The server determines the value of each second data segment, which is also the sorting value of the first data segment. Then, by combining this with the mapping set, it determines the value of the first data segment corresponding to the sorting value, thus obtaining the first data segment. Taking the example of the second data segments being 1, 0, 2, 3, 1, 1, 0, 4, if the mapping set is the dictionary M shown in Figure 5, then the server determines the values ​​of the first data segments corresponding to the compressed sequence to be 1, 2, 3, 4, 1, 1, 2, 2 respectively.

[0173] In this embodiment of the application, the sorting value of the first data segment indicated by the mapping relationship set corresponds one-to-one with the value of the first data segment, so that the first data segment with different values ​​corresponds to different sorting values, and one sorting value uniquely corresponds to one data segment, ensuring that the decompression result is error-free and realizing lossless decompression.

[0174] For example, taking the training scenario of an AI model as an example, the data compression and decompression methods provided in this application embodiment will be described. Please refer to Figure 11, which is a schematic flowchart provided in this application embodiment.

[0175] As shown in Figure 11, the training of the AI ​​model includes forward and backward computation, gradient calculation, and optimizer update. In the forward and backward computation stage, activation values ​​are obtained by performing forward computation on the model's input data. These activation values ​​can be understood as intermediate results from the forward and backward computation stage. They are frequently generated and destroyed during training. In this embodiment, the activation values ​​can be compressed and decompressed.

[0176] The data generated after forward and backward computation then enters the gradient calculation stage, producing first and second order momentum. This compressed first and second order momentum can be stored in high bandwidth memory (HBM). During the optimizer update stage, the compressed first and second order momentum is first decompressed, and the decompressed data is used to optimize the model parameters. After the model parameters are optimized, the first and second order momentum is compressed again and stored in HBM. In other words, the first and second order momentum is persistently stored, unlike the activation values.

[0177] Based on the data compression method provided in the embodiments of this application, the data compression device executing the method can be divided into a numerical bit reconstruction unit, a numerical distribution estimation unit, and a compression unit based on logical functions. Please refer to Figure 12, which is a schematic diagram of the structure of the data compression device provided in the embodiments of this application.

[0178] As shown in Figure 12, the data compression device 1200 includes a numerical bit reconstruction unit 1201, a numerical distribution estimation unit 1202, and a compression unit 1203.

[0179] The bit reassembly unit 1201 is used to divide each binary sequence into multiple data segments and reassemble the data segments with the same bit positions in the M binary sequences into a dataset. The dataset to be compressed is then selected from the multiple datasets included in the M binary sequences.

[0180] The numerical distribution estimation unit 1202 is used to determine the numerical distribution of the dataset to be compressed. The determination of the numerical distribution can be achieved by calculating the true value of the numerical distribution, or by estimating the numerical distribution based on historical numerical distribution information.

[0181] For example, the working principle of the numerical distribution estimation unit 1202 will be further explained by taking the estimation of numerical distribution through an AI model as an example. Please refer to Figure 13, which is a schematic flowchart of numerical distribution estimation provided in an embodiment of this application.

[0182] During the inference phase, the numerical distribution estimation unit 1202 acquires N data segments to be compressed and determines whether the current computation conditions meet the true value calculation conditions, thus determining whether it is necessary to calculate the true value. If yes, the true value of the numerical distribution is calculated. If no, historical true value information is estimated to obtain the numerical distribution of the target dataset.

[0183] During the training phase of the model included in the numerical distribution estimation unit 1202, the initial truth value calculation condition is set to step interval S = 1, and the initial historical truth value information is the numerical distribution calculated in the first step. A feedback adjustment parameter δ is also set. The current training step step is obtained. When Step%N == 0, that is, when the remainder of step step divided by the step interval is 0, the true value P of the numerical distribution is calculated. Then, the cross-entropy between H and P is calculated. When the cross-entropy calculation result is less than δ, N = 2N; otherwise, N = Max(1, N / 2). This achieves the adjustment of the truth value calculation condition.

[0184] Furthermore, each time the true value of the numerical distribution is calculated, it is input into the historical true value information to update the historical true value information. It can be understood that the smaller δ is, the more accurate the numerical distribution predicted by the model. The larger δ is, the fewer times the true value is calculated, and the greater the computational cost savings.

[0185] Compression unit 1203 is used to compress N data segments to be compressed based on the numerical distribution of the dataset, resulting in N compressed second data segments. In practical applications, multiple sets of data segments to be compressed can be processed in parallel based on the parallel capabilities of the computing card, and the number of data segments included in each set is not limited. Typically, a computing card with one core processes one set of data segments to be compressed.

[0186] The data compression device 1200 is used to implement the data compression method described in the foregoing embodiments, as detailed above, and will not be repeated here.

[0187] In addition, this application also provides a data decompression apparatus, the structure of which is similar to the data compression apparatus shown in FIG12. The difference is that the data decompression apparatus includes a decompression unit instead of a compression unit. The functions performed by each unit can be understood as the inverse operations of each unit in the data compression apparatus shown in FIG12, thereby realizing the data decompression method provided in this application. See the relevant description above for details, which will not be repeated here.

[0188] The structures of the data compression device and the data decompression device can also be other possibilities. Please refer to Figure 14, which is a schematic diagram of the structure of a data compression device provided in an embodiment of this application. As shown in Figure 14, the data compression device 1400 includes an acquisition unit 1401 and a processing unit 1402. Optionally, the data compression device 1400 also includes a storage unit 1403.

[0189] In some optional implementations, the acquisition unit 1401 is used to acquire M binary sequences, the M binary sequences including a first dataset, the first dataset including M first data segments, the M first data segments having the same bit position in the M binary sequences, and M being an integer greater than 1. In the first dataset, the number of first data segments with the value v1 is n1, and the value of the first data segment is determined by the number of bits included in the first data segment.

[0190] Processing unit 1402 is used to compress a first data segment with value v1 into a second data segment with value m1, resulting in n1 second data segments, where m1 indicates the number of the first data segments with value v1 in the first dataset.

[0191] In some alternative implementations, the number of first data segments with the value v2 in the first dataset is n2.

[0192] The processing unit 1402 is further configured to compress the first data segment with a value of v2 into a second data segment with a value of m2, resulting in n2 second data segments, where m2 indicates the number of the first data segments with a value of v2 in the first dataset.

[0193] Wherein, if n2 is greater than n1, then m2 is less than m1. Or, if n2 is equal to n1 and v2 is less than v1, then m2 is less than m1.

[0194] In some alternative implementations, the entropy of the first dataset is less than or equal to the entropy threshold.

[0195] In some alternative implementations, the first dataset includes a first data subset, which includes N first data segments, where N is an integer greater than 1 and less than M.

[0196] The first data subset includes the first data segment with the value v1, and the number of first data segments with the value v1 in the first dataset is the smallest among the N first data segments.

[0197] The length of each second data segment is m1, which is the number of effective bits, and the length of the second data segment is less than the length of the first data segment.

[0198] In some alternative implementations, storage unit 1403 is used to store metadata indicating the length of each second data segment.

[0199] In some optional implementations, storage unit 1403 is used to store a mapping relationship set, which indicates a one-to-one correspondence between the value of a first data segment in the first dataset and the sort value of the first data segment. The sort value of the first data segment indicates the order of the number of first data segments with the same value in the first dataset.

[0200] In some optional implementations, the mapping set includes the values ​​of the first data segment in the first dataset and the sorting values ​​of the first data segments for each value. Alternatively,

[0201] The mapping set includes the value of the first data segment in the first dataset and the number of first data segments with each value in the first dataset. Alternatively,

[0202] The mapping relationship set includes the value of the first data segment in the first dataset, the number of the first data segments with each value in the first dataset, and the sort value of the first data segments with each value.

[0203] In some alternative embodiments, the processing unit 1402 is further configured to:

[0204] If the current calculation conditions satisfy the truth value calculation conditions, then calculate the number of the first data segments for each value in the first dataset.

[0205] If the current calculation conditions do not meet the true value calculation conditions, then based on the historical numerical distribution information, the number of the first data segments for each value in the first dataset is estimated.

[0206] In some optional implementations, the current calculation conditions satisfy the truth value calculation conditions, including: the current time is at the beginning of the next calculation cycle, or the interval between the current calculation step and the calculation step corresponding to the previous truth value is a preset interval.

[0207] In some optional implementations, the M binary sequences include a second dataset, which includes M third data segments. The M third data segments have the same bit position in the M binary sequences, and the bit positions of the M third data segments do not overlap with the bit positions of the M first data segments. In the second dataset, the number of first data segments with the value x1 is n3.

[0208] Processing unit 1402 is used to compress a first data segment with value x1 into a third data segment with value y1, resulting in n3 fourth data segments, where y1 indicates the order of the number of third data segments with value x1 in the second dataset.

[0209] In some alternative implementations, the number of third data segments with a value of x2 in the second dataset is n4.

[0210] The processing unit 1402 is also used to compress the third data segment with a value of x2 into a fourth data segment with a value of y2, resulting in n4 fourth data segments, where y2 indicates the order of the number of the third data segments with a value of x2 in the second dataset.

[0211] Wherein, if n4 is greater than n3, then y2 is less than y1. Or, if n4 equals n3 and x2 is less than x1, then y2 is less than y1.

[0212] The data compression device 1400 is used to implement the data compression method described in the foregoing embodiments, as detailed above, and will not be repeated here.

[0213] Please refer to Figure 15, which is a schematic diagram of the structure of the data decompression apparatus provided in an embodiment of this application. As shown in Figure 15, the data decompression apparatus includes an acquisition unit 1501 and a processing unit 1502.

[0214] In some optional implementations, the acquisition unit 1501 is used to acquire a compressed sequence, which includes N second data segments. The value of each second data segment indicates the order of the corresponding first data segment in a first dataset. The first dataset includes M first data segments with the same bit position in M ​​binary sequences, where M is an integer greater than 1 and N is an integer greater than 1 and less than M. Meta-information is also acquired, indicating the length of each second data segment.

[0215] The processing unit 1502 is used to determine the value of each second data segment based on the metadata.

[0216] The acquisition unit 1501 is also used to acquire a mapping relationship set, which indicates that the values ​​of the first data segment and the values ​​of the second data segment correspond one-to-one.

[0217] The processing unit 1502 is also used to determine the value of the first data segment corresponding to each second data segment based on the value of each second data segment and the mapping relationship set.

[0218] In some optional implementations, the content of the mapping relationship set can be varied: Optionally, the mapping relationship set includes the value of the first data segment in the first dataset and the sorting value of the first data segment for each value. Optionally, the mapping relationship set includes the value of the first data segment in the first dataset and the number of first data segments for each value in the first dataset. Optionally, the mapping relationship set includes the value of the first data segment in the first dataset, the number of first data segments for each value in the first dataset, and the sorting value of the first data segment.

[0219] The data decompression device 1500 is used to implement the data decompression method described in the foregoing embodiments, as detailed above, and will not be repeated here.

[0220] The acquisition unit 1401, processing unit 1402, storage unit 1403, acquisition unit 1501, and processing unit 1502 can all be implemented in software or in hardware. For example, the implementation of processing unit 1402 will be described below. Similarly, the implementation of other units can refer to the implementation of processing unit 1402.

[0221] As an example of a software functional unit, processing unit 1402 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, processing unit 1402 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0222] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0223] As an example of a hardware functional unit, the processing unit 1402 may include at least one computing device, such as a server, computing card, accelerator, etc. Alternatively, the processing unit 1402 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0224] The processing unit 1402 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the processing unit 1402 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the processing unit 1402 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0225] It should be noted that the data compression method provided in this application embodiment is implemented through the acquisition unit 1401, processing unit 1402, and storage unit 1403, and will not be described in detail here. The data decompression method provided in this application embodiment is implemented through the acquisition unit 1501 and processing unit 1502, and will not be described in detail here.

[0226] Please refer to Figure 16, which is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 1600 includes a processor 1601, a communication interface 1602, a bus 1603, and a memory 1604. The processor 1601, communication interface 1602, and memory 1604 communicate via the bus 1603. In practical applications, communication can also be achieved through other means such as wireless transmission; specific methods are not limited here.

[0227] It should be understood that this application does not limit the number of processors and memories in the computing device 1600.

[0228] Processor 1601 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0229] The communication interface 1602 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1600 and other devices or communication networks.

[0230] Bus 1603 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 16, but this does not imply that there is only one bus or one type of bus. Bus 1603 can include pathways for transmitting information between various components of computing device 1600 (e.g., memory 1604, processor 1601, communication interface 1602).

[0231] Memory 1604 may include volatile memory, such as random access memory (RAM). Memory 1604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0232] Optionally, the memory 1604 stores executable program code, and the processor 1601 executes the executable program code to implement the functions of the aforementioned acquisition unit 1401, processing unit 1402, and storage unit 1403, thereby implementing the data compression method provided in this application embodiment. That is, the memory 1604 stores instructions for executing the data compression method provided in this application embodiment.

[0233] Optionally, the memory 1604 stores executable program code, and the processor 1601 executes the executable program code to implement the functions of the aforementioned acquisition unit 1501 and processing unit 1502 (not shown in the figure), thereby realizing the data decompression method provided in this application embodiment. That is, the memory 1604 stores instructions for executing the data decompression method provided in this application embodiment.

[0234] This application also provides a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0235] Please refer to Figures 17 and 18, which are schematic diagrams of the structure of the computing device cluster provided in the embodiments of this application.

[0236] As shown in Figure 17, the computing device cluster includes at least one computing device 1600. Figure 17 illustrates an example where the memory 1604 of one or more computing devices 1600 in the computing device cluster may store the same instructions for executing the data compression method provided in the embodiments of this application.

[0237] In some possible implementations, the memory 1604 of one or more computing devices 1600 in the computing device cluster may also store partial instructions for executing the data compression method. In other words, a combination of one or more computing devices 1600 can jointly execute the instructions for executing the data compression method. It should be noted that the memory 1604 of different computing devices 1600 in the computing device cluster can store different instructions, each used to execute a partial function of the data compression device. That is, the instructions stored in the memory 1604 of different computing devices 1600 can implement the functions of one or more units among the acquisition unit 1401, processing unit 1402, and storage unit 1403.

[0238] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 18 illustrates one possible implementation, and Figure 18 uses a computing device cluster to execute the data compression method provided in the embodiments of this application as an example.

[0239] As shown in Figure 18, two computing devices 1600A and 1600B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 1604 in computing device 1600A stores instructions for executing the functions of the acquisition unit 1401. Simultaneously, the memory 1604 in computing device 1600B stores instructions for executing the functions of the processing unit 1402 and the storage unit 1403.

[0240] The connection method between the computing device clusters shown in Figure 18 can be based on the data compression method provided in this application, which separates the processing operation from the operation outside the processing operation. That is, it is considered that the function of the acquisition unit 1401 is executed by the computing device 1600A, and the functions of the processing unit 1402 and the storage unit 1403 are executed by the computing device 1600B.

[0241] It should be understood that the functions of computing device 1600A shown in Figure 18 can also be performed by multiple computing devices 1600. Similarly, the functions of computing device 1600B can also be performed by multiple computing devices 1600.

[0242] In some alternative implementations, the computing device cluster can also implement the data decompression method provided in the embodiments of this application. Similar to the example in FIG18, the two computing devices are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device. In this type of possible implementation, one computing device stores instructions in its memory to perform the functions of the acquisition unit 1501. Meanwhile, the other computing device stores instructions in its memory to perform the functions of the processing unit 1502.

[0243] Similarly, there may be multiple computing devices that perform the functions of the acquisition unit 1501 and the processing unit 1502.

[0244] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 17 and 18, and will not be repeated here.

[0245] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computer device, it causes the at least one computer device to perform the aforementioned data compression method or data decompression method.

[0246] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the aforementioned data compression method or data decompression method.

[0247] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0248] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A data compression method, characterized in that, include: Obtain M binary sequences, wherein the M binary sequences include a first dataset, the first dataset includes M first data segments, the M first data segments have the same bit position in the M binary sequences, and M is an integer greater than 1; in the first dataset, the number of first data segments with the value v1 is n1, and the value of the first data segment is determined by the number of bits included in the first data segment; The first data segment with value v1 is compressed into a second data segment with value m1, resulting in n1 second data segments, where m1 indicates the order of the number of the first data segment with value v1 in the first dataset.

2. The method according to claim 1, characterized in that, In the first dataset, the number of first data segments with a value of v2 is n2, and the method further includes: The first data segment with a value of v2 is compressed into a second data segment with a value of m2, resulting in n2 second data segments, where m2 indicates the order of the number of the first data segment with a value of v2 in the first dataset. Where, if n2 is greater than n1, then m2 is less than m1; or, If n2 equals n1 and v2 is less than v1, then m2 is less than m1.

3. The method according to claim 1 or 2, characterized in that, The entropy of the first dataset is less than or equal to the entropy threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The first dataset includes a first data subset, which includes N first data segments, where N is an integer greater than 1 and less than M; The first data subset includes the first data segment with the value v1, and the number of the first data segments with the value v1 in the first dataset is the smallest among the N first data segments; In the N second data segments obtained by compressing the N first data segments, the length of each second data segment is the number of effective bits of m1, and the length of the second data segment is less than the length of the first data segment.

5. The method according to claim 4, characterized in that, The method further includes: The metadata is stored, which indicates the length of each second data segment.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: A set of mapping relationships is stored, wherein the set of mapping relationships indicates that the value of the first data segment in the first dataset corresponds one-to-one with the sort value of the first data segment; The sort value of the first data segment indicates the order of the number of first data segments with the same value in the first dataset.

7. The method according to claim 6, characterized in that: The mapping relationship set includes the values ​​of the first data segment in the first dataset and the sorting values ​​of the first data segment for each value; or, The mapping relationship set includes the values ​​of the first data segments in the first dataset and the number of first data segments with each value in the first dataset; or, The mapping relationship set includes the value of the first data segment in the first dataset, the number of the first data segments with each value in the first dataset, and the sorting value of the first data segments with each value.

8. The method according to any one of claims 1 to 7, characterized in that, After obtaining the M binary sequences, the method further includes: If the current calculation conditions meet the truth value calculation conditions, then calculate the number of the first data segments for each value in the first dataset; If the current calculation conditions do not meet the truth value calculation conditions, then based on the historical numerical distribution information, the number of first data segments for each value in the first dataset is estimated. The historical numerical distribution information indicates the number of data segments for each value in Z historical data segments. The bit positions of the Z historical data segments are the same as the bit positions of the M first data segments, and Z is an integer greater than 1.

9. A data compression device, characterized in that, include: An acquisition unit is used to acquire M binary sequences, wherein the M binary sequences include a first dataset, the first dataset includes M first data segments, the M first data segments have the same bit position in the M binary sequences, and M is an integer greater than 1; in the first dataset, the number of first data segments with the value v1 is n1, and the value of the first data segment is determined by the bits included in the first data segment; The processing unit is used to compress the first data segment with a value of v1 into a second data segment with a value of m1, resulting in n1 second data segments, wherein m1 indicates the order of the number of the first data segment with a value of v1 in the first dataset.

10. The apparatus according to claim 9, characterized in that, In the first dataset, the number of first data segments with the value v2 is n2; The processing unit is further configured to compress the first data segment with a value of v2 into a second data segment with a value of m2, resulting in n2 second data segments, wherein m2 indicates the order of the number of the first data segment with a value of v2 in the first dataset; Where, if n2 is greater than n1, then m2 is less than m1; or, If n2 equals n1 and v2 is less than v1, then m2 is less than m1.

11. The apparatus according to claim 9 or 10, characterized in that, The entropy of the first dataset is less than or equal to the entropy threshold.

12. The apparatus according to any one of claims 9 to 11, characterized in that: The first dataset includes a first data subset, which includes N first data segments, where N is an integer greater than 1 and less than M; The first data subset includes the first data segment with the value v1, and the number of the first data segments with the value v1 in the first dataset is the smallest among the N first data segments; In the N second data segments obtained by compressing the N first data segments, the length of each second data segment is the number of effective bits of m1, and the length of the second data segment is less than the length of the first data segment.

13. The apparatus according to claim 12, characterized in that, The device further includes a storage unit for storing metadata indicating the length of each second data segment.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The device further includes a storage unit for storing a mapping relationship set, wherein the mapping relationship set indicates that the value of the first data segment in the first dataset corresponds one-to-one with the sorting value of the first data segment; The sort value of the first data segment indicates the order of the number of first data segments with the same value in the first dataset.

15. The apparatus according to claim 14, characterized in that: The mapping relationship set includes the values ​​of the first data segment in the first dataset and the sorting values ​​of the first data segment for each value; or, The mapping relationship set includes the values ​​of the first data segments in the first dataset and the number of first data segments with each value in the first dataset; or, The mapping relationship set includes the value of the first data segment in the first dataset, the number of the first data segments with each value in the first dataset, and the sorting value of the first data segments with each value.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The processing unit is further configured to: If the current calculation conditions meet the truth value calculation conditions, then calculate the number of the first data segments for each value in the first dataset; If the current calculation conditions do not meet the true value calculation conditions, then based on historical numerical distribution information, the number of first data segments for each value in the first dataset is estimated.

17. A computing device, characterized in that, Including processor and memory; The processor is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device to perform the method as described in any one of claims 1 to 8.

18. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 8.

Images