Data processing method and related apparatus

Abstract

Provided in the embodiments of the present application are a data processing method and a related apparatus. In the present application, a matrix multiplication operation is performed on a rotation factor matrix and an input signal by means of cube units, so as to accelerate in combination with the cube units a fast Fourier transform (FFT) operation performed on the input signal. The method comprises: on the basis of the length of a first input signal, generating a rotation factor matrix, wherein the rotation factor matrix comprises a plurality of rotation factors, which are used for performing an FFT operation on the first input signal; and performing a matrix multiplication operation on the rotation factor matrix and the first input signal, so as to obtain a first matrix-product matrix, wherein the first matrix-product matrix comprises a result of performing the FFT operation on the first input signal.

Description

A data processing method and related apparatus

[0001] This application claims priority to Chinese Patent Application No. 202411677871.7, filed on November 20, 2024, entitled "A Data Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence, and more particularly to a data processing method and related apparatus. Background Technology

[0003] The Short-Time Fourier Transform (STFT) is a variation of the Fast Fourier Transform (FFT) and an important tool for time-frequency analysis. STFT generates a two-dimensional spectrogram by performing a sliding-window FFT on a one-dimensional input signal, transforming untrainable waveform data into a trainable spectrogram. Currently, this STFT is widely used in large-scale speech modeling.

[0004] FFT is a crucial component of STFT and directly impacts its performance. To achieve high-performance STFT implementation in hardware, the industry currently primarily employs a vector-based approach using the FFT butterfly algorithm. Vector-based STFTs can fully leverage vector acceleration engines (such as CPU's SIMD and GPU's SIMT) to enhance hardware performance and accelerate STFT computation.

[0005] Furthermore, since the matrix computation unit cube in NPU / GPU has significantly greater computing power than the vector computation unit vetor, and STFT is currently not compatible with cube units, it cannot fully utilize the computing power of NPU / GPU. Therefore, the performance of STFT on hardware such as NPU / GPU remains relatively poor. Thus, how to fully utilize hardware computing power and accelerate STFT computation on hardware is a pressing issue that needs to be addressed. Summary of the Invention

[0006] This application provides a data processing method and related apparatus, which performs matrix multiplication on the rotation factor matrix and the input signal through a cube unit, so as to realize the fast Fourier operation on the input signal by combining the cube unit.

[0007] In a first aspect, this application provides a data processing method, which includes: generating a rotation factor matrix based on the length of a first input signal, the rotation factor matrix including multiple rotation factors, the multiple rotation factors being used to perform a Fast Fourier Transform (FFT) operation with the first input signal; after obtaining the rotation factor matrix, performing a matrix multiplication operation on the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix, the first matrix multiplication matrix including the result of the FFT operation on the first input signal.

[0008] In this embodiment of the application, a rotation factor matrix can be generated according to the length of the input signal. Multiple rotation factors in the rotation factor matrix are used to perform FFT operation with the input signal. Based on the FFT operation, a cube unit with higher computing power than a vector unit can be used to perform matrix multiplication operation on the generated rotation factor matrix and the input signal to accelerate the FFT operation on the input signal.

[0009] In one possible implementation, the method may further include: acquiring a second input signal; when the length of the second input signal is the same as the length of the first input signal, performing a matrix multiplication operation on the rotation factor matrix and the second input signal to obtain a second matrix multiplication matrix, wherein the second matrix multiplication matrix includes the result of performing an FFT operation on the second input signal.

[0010] In this embodiment, when performing an FFT on the subsequent input signal (the second input signal), it can be determined whether the rotation factor matrix needs to be regenerated based on the length of the second input signal. When the length of the second input signal is the same as the length of the first input signal, the previously generated rotation factor matrix can be directly used without regeneration. This reduces the computational load when performing an FFT on the second input signal, requiring only the matrix multiplication of the second input signal and the rotation factor matrix, effectively improving the efficiency of FFT operations on signals of the same length.

[0011] In one possible implementation, after generating the rotation factor matrix based on the length of the first input signal, the method may further include: performing a windowing operation on the rotation factor matrix according to a window function to obtain a first matrix, the first matrix including the fusion information of the rotation factor matrix and the window function.

[0012] In this embodiment, when performing a short-time Fourier transform on the input signal, it is usually necessary to first divide the input signal into multiple frames to obtain sub-signals, and then perform a windowing operation on each sub-signal. This application uses a window function-based windowing operation on the rotation factor matrix, which avoids the subsequent windowing of each sub-signal, reduces the amount of computation, and thus improves the efficiency of subsequent short-time Fourier transform operations on the input signal.

[0013] In one possible implementation, before generating the rotation factor matrix based on the length of the first input signal, the method may further include: performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal.

[0014] In this embodiment of the application, the first input signal can also be any frame of the third input signal. When performing a short-time Fourier transform on the third input signal, a sliding window operation can be performed on the third input signal to obtain multiple frames of the first input signal, and then a fast Fourier transform can be performed on each frame of the first input signal.

[0015] In one possible implementation, after performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, the method may further include: filling the first input signal to obtain multiple frames of filled first input signal.

[0016] In this embodiment, padding the framed signal ensures that unnecessary spectral distortion is avoided when performing Fourier transform at signal boundaries due to window truncation, thus preventing boundary effects. Furthermore, it standardizes the length of each frame after framing and improves spectral smoothness, resulting in more accurate and reliable spectral analysis.

[0017] In one possible implementation, after filling the first input signal to obtain a first input signal with multiple frames of filling, the method may further include: combining the first input signals with multiple frames of filling to generate a second matrix.

[0018] In one possible implementation, the aforementioned matrix multiplication operation between the rotation factor matrix and the first input signal to obtain the first matrix multiplication matrix may include: performing matrix multiplication operation between the first matrix and the second matrix to obtain the third matrix multiplication matrix, wherein the third matrix multiplication matrix includes the result of performing FFT operation on multiple frames of the first input signal in the first matrix.

[0019] In one possible implementation, the aforementioned matrix multiplication of the first matrix and the second matrix to obtain the third matrix multiplication can include: transposing the second matrix to obtain the transposed second matrix; and performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication.

[0020] In one possible implementation, after performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication, the method may further include: concatenating the real and imaginary data in the third matrix multiplication to obtain the result of performing a short-time Fourier operation on the third input signal.

[0021] Secondly, this application provides a data processing apparatus, comprising: a generation module, configured to generate a rotation factor matrix based on the length of a first input signal, the rotation factor matrix including multiple rotation factors, the multiple rotation factors being used to perform a Fast Fourier Transform (FFT) operation with the first input signal; and a matrix multiplication module, configured to perform a matrix multiplication operation on the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix, the first matrix multiplication matrix including the result of the FFT operation on the first input signal.

[0022] In one possible implementation, the device may further include: an acquisition module for acquiring a second input signal; the aforementioned matrix multiplication module is further configured to perform a matrix multiplication operation on the rotation factor matrix and the second input signal when the length of the second input signal is the same as the length of the first input signal, to obtain a second matrix multiplication matrix, wherein the second matrix multiplication matrix includes the result of performing an FFT operation on the second input signal.

[0023] In one possible implementation, after generating the rotation factor matrix based on the length of the first input signal, the device may further include: a processing module, configured to perform a windowing operation on the rotation factor matrix according to a window function to obtain a first matrix, the first matrix including fusion information of the rotation factor matrix and the window function.

[0024] In one possible implementation, before generating the rotation factor matrix based on the length of the first input signal, the device may further include: a framing module for performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal.

[0025] In one possible implementation, after performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, the aforementioned framing module is further used to fill the first input signal to obtain multiple frames of filled first input signal.

[0026] In one possible implementation, after filling the first input signal to obtain a first input signal with multiple frames of filling, the aforementioned generation module is further used to combine the first input signals with multiple frames of filling to generate a second matrix.

[0027] In one possible implementation, the aforementioned matrix multiplication module is specifically used to: perform matrix multiplication on the first matrix and the second matrix to obtain a third matrix multiplication matrix, wherein the third matrix multiplication matrix includes the result of FFT operation on multiple frames of the first input signal in the first matrix.

[0028] In one possible implementation, the aforementioned matrix multiplication module is specifically used to: transpose the second matrix to obtain the transposed second matrix; and perform matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication.

[0029] In one possible implementation, after performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication, the device may further include: a splicing module for splicing the real and imaginary data in the third matrix multiplication to obtain the result of performing a short-time Fourier operation on the third input signal.

[0030] Thirdly, this application provides a processor that can be connected to a memory for executing instructions stored in the memory, so that the processor performs the method as described in the first aspect or any possible implementation thereof.

[0031] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium stores computer instructions; when these computer instructions are executed on a computer, they cause the computer to perform the method as described in any of the possible implementations of the first aspect.

[0032] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method as described in any of the possible implementations of the first aspect.

[0033] The technical effects of the second to fifth aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0034] Figure 1 is a schematic diagram of a system framework provided in this application;

[0035] Figure 2 is a flowchart of the large speech model and a flowchart of the STFT of the signal in the NPU.

[0036] Figure 3 is a flowchart illustrating a data processing method provided in this application;

[0037] Figure 4 is a schematic diagram of how to combine the real part matrix and the imaginary part matrix;

[0038] Figure 5 is a diagram illustrating the plan mechanism of the Short Time Fourier Transform (STFT).

[0039] Figure 6 is a diagram illustrating the matmul mechanism of the Short Time Fourier Transform (STFT).

[0040] Figure 7 is a performance comparison chart of the data processing method proposed in this application and the vector-based STFT scheme for performing STFT operations on the input signal;

[0041] Figure 8 is a schematic diagram of the structure of a data processing device provided in this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0043] To facilitate understanding, some concepts involved in this application will be introduced first.

[0044] 1. Discrete Fourier Transform (DFT)

[0045] Fourier analysis is the most fundamental method for signal analysis. The Discrete Fourier Transform (DFT) is the core of Fourier analysis. Through the Discrete Fourier Transform, signals can be transformed from the time domain to the frequency domain, thereby allowing the study of the signal's spectral structure and variation patterns.

[0046] 2. Fast Fourier Transform (FFT)

[0047] The Fast Fourier Transform (FFT) is an efficient algorithm for computing the Discrete Fourier Transform (DFT). The core idea of ​​the Fast Fourier Transform is to decompose the original function into a weighted sum of an odd-numbered term and an even-numbered term. Then, the odd-numbered and even-numbered terms are decomposed again until the desired computational precision is achieved. In this way, the complex operations of the original Fourier Transform are completed by repeatedly performing the weighted sum of two terms, thus significantly reducing computation time.

[0048] The Fast Fourier Transform (FFT) has wide applications in signal processing, image processing, quantum mechanics, theoretical physics, military electronic warfare, computer classification and recognition, artificial synthesis of music and speech, medical imaging and diagnosis, seismic exploration data processing, and fault diagnosis of large machinery. It can be used not only for signal spectrum analysis, filtering, noise reduction, compression, and transmission, but also for numerical analysis, constructing fast numerical methods, constructing curves and surfaces, and solving differential equations.

[0049] 3. Short-Time Fourier Transform (STFT)

[0050] The Short-Time Fourier Transform (STFT) is a mathematical transformation method used to determine the frequency and phase of a sinusoidal wave in a local region of a time-varying signal. Its basic idea is to divide a long, non-stationary signal into a series of shorter signals of equal length (or "frames"), assuming that the signal is stationary (or pseudo-stationary) within each short time interval. Then, a Fourier transform is performed on the signal within each short time interval to obtain the spectral information for that time period. By moving the time window, the power spectrum at different times can be calculated, thus obtaining the time-frequency distribution of the entire signal.

[0051] In implementation, the Short-Time Fourier Transform (SFT) is calculated as a series of windowed data frames. The choice of the window function significantly impacts the result. Common window functions include rectangular windows, Hanning windows, and Hamming windows. The window size (frame length) and the sliding step size (frame shift) are also important parameters, determining the balance between time resolution and frequency resolution. The SFT has wide applications in signal processing, such as audio signals, images, and biomedical signals.

[0052] 4. Matrix computation unit (cube)

[0053] The Cube unit is used to perform matrix multiplication. It can only perform matrix multiplication of the same type of data, such as integer matrix multiplication by integer matrix, or floating-point matrix multiplication by floating-point matrix. In general NPU chips, the Cube unit has strong computing power.

[0054] 5. Vector computation unit (vector)

[0055] Vector units are used to compute vector operations and can perform a variety of vector operations, but their computing power is weaker compared to Cube units.

[0056] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0057] Referring to Figure 1, a system architecture 100 is provided for an embodiment of this application. This system architecture 100 may include a computing device cluster 110, which includes at least one computing device. The computing device may be a server, such as a cloud server, a central server, an edge server, or a local server in a local data center. In some possible embodiments, the computing device may also be a desktop computer, a laptop computer, or other terminal device.

[0058] Specifically, the computing units (also called accelerator cards) used for data processing can be distributed in any one or more computing devices in the aforementioned system framework 100. The accelerator cards can be Ascend processors, graphics processing units (GPUs), neural network processing units (NPUs), etc., and are not specifically limited here.

[0059] It is worth noting that the system architecture shown in Figure 1 is only an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, system framework 100 may also include the client, which sends input signals and window functions to the computing device.

[0060] The data processing method provided in this application can be applied to large-scale speech model scenarios. Figure 2 is a flowchart of the large-scale speech model and a flowchart of the STFT process for the signal in the NPU. As shown in Figure 2, in the large-scale semantic model scenario, the input is a series of speech signals. STFT, as the core operation of preprocessing, performs sliding window FFT processing on the speech signals to construct their two-dimensional spectrogram, converting the untrainable waveform data into a trainable spectrogram. The encoder encodes the spectrogram and then outputs it to the decoder for processing. Subsequently, the corresponding large-scale speech model output is generated according to the actual application.

[0061] Taking the Ascend NPU's STFT operation on a one-dimensional speech signal as an example, firstly, the vector unit in the NPU segments the one-dimensional speech signal into frames, obtaining a two-dimensional tensor (i.e., a matrix). This matrix includes multiple frames of sub-signals, with each row representing one frame. Then, the segmented sub-signals are sent to the cube unit, where they are multiplied by a pre-created matmul left matrix to obtain a matrix multiplication matrix. This matrix multiplication matrix includes the result of the STFT operation on the input speech signal, where the matmul left matrix includes the rotation factor matrix and window function fusion information for the one-dimensional speech signal STFT operation. Finally, the vector unit concatenates the real and corresponding imaginary numbers in the matrix multiplication matrix, ultimately obtaining the STFT operation result of the one-dimensional speech signal and the corresponding two-dimensional spectrogram.

[0062] Typically, the computing power of cube units and vector units in NPU / GPU hardware differs significantly, with cube units generally having higher computing power. However, current STFT operations based on NPU / GPUs are not compatible with cube units, preventing them from being used for computation and thus wasting cube hardware resources. Therefore, how to fully leverage the computing power of NPU / GPUs to further accelerate STFT operations has become an urgent problem to be solved.

[0063] To address the current problem, this application proposes a data processing method. This method generates a rotation factor matrix for calculating the FFT of the first input signal based on the length of the first input signal. Subsequently, a cube unit can be used to perform matrix multiplication on the rotation factor matrix and the first input signal, thereby accelerating the FFT operation on the first input signal. Furthermore, since the Short-Time Fourier Transform (STFT) can be viewed as a Fast Fourier Transform of windowed data frames, this further accelerates the STFT operation.

[0064] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0065] Referring to Figure 3, a flowchart of a data processing method provided in this application is shown below.

[0066] Step 301: Generate a rotation factor matrix based on the length of the first input signal;

[0067] In this embodiment, the rotation factor matrix includes multiple rotation factors. In the butterfly operation of the fast Fourier transform, the rotation factors can modulate and demodulate the signal with different frequencies and phases to realize the conversion between the time domain and the frequency domain.

[0068] Specifically, the twitch factor can be expressed as e -2πiknN Where N is the length of the first input signal, k is the sampling point index in the frequency domain, used to represent different frequency components, and the value of k ranges from [0, N-1], and n is the sampling point index in the time domain, and the value of n ranges from [0, N-1]. The expression for the rotation factor can be obtained using Euler's formula e ix =cosx + isinx will e -2πikN Expanding, we get:

[0069] e -2πikn / N =cos(-2πkn / N)+isin(-2πkn / N)

[0070] =cos(2πkn / N)-isin(2πkn / N)

[0071] As can be seen from the expansion of the rotation factor, it includes a real part and an imaginary part. Since the current cube unit does not support complex number operations, the real parts can be combined to obtain a real part matrix, and the imaginary parts can be combined to obtain an imaginary part matrix. Then, the real and imaginary part matrices can be combined again to obtain the rotation factor matrix, ensuring that the rotation factor matrix does not contain imaginary numbers, thus allowing subsequent operations to be performed by the cube unit. In this case, the size of both the real and imaginary part matrices is [N, N].

[0072] The combination of the real and imaginary matrices can be either with the real matrix on top and the imaginary matrix on the bottom, or with the real matrix on the bottom and the imaginary matrix on top. For example, the combination of the real and imaginary matrices can be shown in Figure 4.

[0073] Furthermore, since the first input signal is a real number, the first input signal 0 has conjugate property in the frequency domain of the Fourier transform, X(k) = X(Nk). The derivation of X(k) = X(Nk) is as follows:

[0074] The formula for performing an FFT operation on the first input signal satisfies the following equation:

[0075] again,

[0076] Therefore, when performing FFT on the first input signal, only X(0), X(1), ..., X(N / 2) need to be calculated. The range of k in the real and imaginary matrices can be [0, N / 2+1]. Then, the real and imaginary matrices can be reduced, and a matrix of size [N / 2+1, N] is finally obtained based on the values ​​of k and n, with the rows representing the values ​​of k and the columns representing the values ​​of n.

[0077] In this embodiment of the application, when the first input signal is a real number signal, the rotation factor matrix can be reduced according to the conjugate symmetry of the real number signal in the frequency domain. This can reduce the computational load of the subsequent FFT operation on the first input signal based on the rotation matrix factor, thereby improving computational efficiency.

[0078] Optionally, a second input signal can also be acquired, and its FFT can be calculated. First, based on the length of the second input signal, it can be determined whether a new rotation factor matrix needs to be generated. When the length of the second input signal is the same as the length of the first input signal, the rotation factor matrix can be directly called, and the matrix multiplication of the rotation factor matrix and the second input signal can be calculated to obtain a second matrix multiplication matrix, which includes the result of the FFT operation on the second input signal.

[0079] Optionally, after generating the rotation factor matrix and reducing it, a windowing operation can be performed on the rotation factor matrix according to the window function to obtain a first matrix, which includes the fusion information of the rotation factor matrix and the window function.

[0080] The window function can be a rectangular window, a Hamming window, or a Hanning window, etc., and is not specifically limited here. Furthermore, windowing is applied to the rotation factor matrix, resulting in a first matrix that incorporates the fusion information of the rotation factor matrix and the window function. This allows subsequent FFT operations on input signals of length N to be performed using this first matrix, achieving one-time generation and multiple reuse, thereby effectively reducing the time consumed by subsequent FFT operations on the input signals. As shown in Figure 5, `plan` is the aforementioned constructed first matrix. When the length of the input signal is the same as the length of the signal used to construct the first matrix, the first matrix can be repeatedly called for FFT operations.

[0081] Furthermore, when performing Short Time Fourier Transform (STFT) on the input signal, it is necessary to perform windowing operation on each frame of the signal after framing. The windowing operation on the rotation factor matrix proposed in this application can avoid windowing each frame of the signal, thereby reducing the amount of computation. At the same time, it reduces the high communication complexity caused by the synchronization between multiple operation stages involved in the FFT butterfly algorithm, and improves the efficiency of STFT operation on the input signal.

[0082] Typically, to perform a short-time Fourier transform on an input signal, the input signal can first be divided into frames, which divides the input signal into multiple sub-frames. Then, a fast Fourier transform is performed on each sub-frame to obtain its corresponding spectrum.

[0083] Optionally, before acquiring the first input signal, a third input signal can be acquired, and the third input signal can be framed to obtain multiple frames of the first input signal. Specifically, a sliding window operation can be performed on the third input signal to obtain multiple frames of the first input signal. The third input signal is the input signal to be subjected to the short-time Fourier transform operation. For example, the third input signal can be a speech signal, an image signal, or a similar signal; similarly, the first and second input signals can also be speech signals, image signals, or similar signals.

[0084] Optionally, after performing a sliding window operation on the third input signal, a padding operation can be performed on the first input signal to obtain a first input signal padded with multiple frames. Specifically, when the lengths of the obtained first input signals are inconsistent across multiple frames, the first input signal can be padded. This padding can be performed on each frame of the first input signal, or on a subset of the multiple frames. The resulting first input signal padded with multiple frames can be the signal obtained after padding each frame of the first input signal, or it can include both the unpadded first input signal and the actually padded first input signal; the specific details are not limited here.

[0085] For example, after performing a sliding window operation on the third input signal, an X-frame first input signal is obtained. This first input signal can be padded to obtain an X-frame padded first input signal. Specifically, this can be done by padding the first input signal of each frame in the X-frames, or by padding the first input signals of M frames. The resulting X-frame padded first input signal can be the signal after padding each frame, or it can include the unpadded first input signals of XM frames and the padded first input signals of M frames. Here, X and M are integers, and the value of M ranges from [1, X].

[0086] After obtaining the first input signal after multi-frame padding, the first input signal after multi-frame padding can be combined to obtain a second matrix. One row of data in the second matrix corresponds to the first input signal after one frame processing. The second matrix is ​​a matrix of size [frame, N], where frame is the number of frames.

[0087] It is understandable that after performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, the multiple frames of the first input signal can also be directly combined to obtain the second matrix; the specifics are not limited here.

[0088] Step 302: Perform matrix multiplication on the rotation factor matrix and the first input signal to obtain the first matrix multiplication matrix.

[0089] After obtaining the rotation factor matrix based on the length of the first input signal, matrix multiplication can be performed on the rotation factor matrix and the first input signal to perform an FFT operation on the first input signal. The first matrix multiplication includes the result of the FFT operation on the first input signal.

[0090] When the upper half of the twitch factor matrix is ​​a real matrix and the lower half is an imaginary matrix, the upper half of the first matrix-multiplication matrix represents the real part of the FFT result, and the lower half represents the imaginary part. Similarly, when the upper half of the twitch factor matrix is ​​an imaginary matrix and the lower half is a real matrix, the upper half of the first matrix-multiplication matrix represents the imaginary part of the FFT result, and the lower half represents the real part.

[0091] In this embodiment of the application, when performing STFT operation on the third input signal, the aforementioned framing method is used to frame the third input signal and generate the second matrix. Then, matrix multiplication can be performed on the first matrix and the second matrix to obtain the third matrix multiplication matrix. The third matrix multiplication matrix includes the result of performing FFT operation on multiple frames of the first input signal in the first matrix.

[0092] Specifically, the second matrix can be transposed to obtain a transposed second matrix of size [N, frame]. Then, matrix multiplication is performed on the first matrix and the transposed second matrix to obtain a third matrix multiplication. This third matrix multiplication is a matrix of size [(N / 2+1)*2, frame].

[0093] For example, as shown in Figure 6, Figure 6 is a schematic diagram of matrix multiplication of the right matrix of the matmul for each batch of input signals. The right matrix of matmul is obtained by dividing each batch of input signals into frames and transposing them, and the left matrix of matmul is the first matrix generated above. The left matrix of matmul and the right matrix of matmul for each batch are multiplied by the cube unit to obtain the STFT operation result of each batch.

[0094] Optionally, after obtaining the third matrix multiplication matrix, the real data and the corresponding imaginary data in the third matrix multiplication can be concatenated to obtain the result of the STFT operation on the third input signal.

[0095] In this embodiment, a first matrix (or matmul left matrix) can be generated based on the length of the input signal. This first matrix includes multiple rotation factors, which are used to perform FFT operations with the input signal. Based on the FFT operation, a cube unit with higher computing power than a vector unit can be used to perform matrix multiplication on the generated rotation factor matrix and the input signal, thereby accelerating the FFT operation on the input signal. Furthermore, since the first matrix is ​​obtained by windowing the rotation factor matrix, it includes the fusion information of the rotation factor matrix and the window function. For signals of the same length, the first matrix can be directly used to perform FFT operations on the signal, thus avoiding the need to window each frame of sub-signal during STFT operations. Subsequent STFT operations on the input signal based on the first matrix can reduce the computational load and effectively lower the computation time.

[0096] For example, the data processing method proposed in this application was deployed on an Ascend NPU for testing, and compared with a vector-based STFT scheme deployed on a GPU. The comparison results are shown in Figure 7. As can be seen from Figure 7, the data processing method proposed in this embodiment takes less time to perform STFT operations on the input signal than the vector-based STFT scheme. Therefore, it can be concluded that the data processing method proposed in this embodiment can accelerate the STFT operation of the input signal by combining the cube unit in the NPU / GPU, and can make full use of the computing resources of the NPU / GPU.

[0097] The data processing method proposed in this application can be applied to large speech model scenarios. For example, the data processing method can be deployed on Ascend 910B, and the input speech signal can be used to perform speech recognition. The extracted speech features are then input into an open-source large speech model for inference. The accuracy of the inference results obtained is consistent with that of the inference results obtained based on existing solutions. Furthermore, the time required for speech recognition using the data processing method of this application is shorter.

[0098] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0099] Referring to Figure 8, a schematic diagram of the structure of a data processing device provided in this application is shown below.

[0100] The generation module 801 is used to generate a rotation factor matrix based on the length of the first input signal. The rotation factor matrix includes multiple rotation factors, which are used to perform a Fast Fourier Transform (FFT) operation with the first input signal.

[0101] The matrix multiplication module 802 is used to perform matrix multiplication on the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix, which includes the result of FFT operation on the first input signal.

[0102] In one possible implementation, the device may further include: an acquisition module 803 for acquiring a second input signal; and the aforementioned matrix multiplication module 802 for performing a matrix multiplication operation on the rotation factor matrix and the second input signal when the length of the second input signal is the same as the length of the first input signal, to obtain a second matrix multiplication matrix, wherein the second matrix multiplication matrix includes the result of performing an FFT operation on the second input signal.

[0103] In one possible implementation, after generating the rotation factor matrix based on the length of the first input signal, the device may further include: a processing module 804, configured to perform a windowing operation on the rotation factor matrix according to a window function to obtain a first matrix, the first matrix including fusion information of the rotation factor matrix and the window function.

[0104] In one possible implementation, before generating the rotation factor matrix based on the length of the first input signal, the device may further include: a framing module 805 for performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal.

[0105] In one possible implementation, after performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, the aforementioned framing module 805 is further used to fill the first input signal to obtain multiple frames of filled first input signal.

[0106] In one possible implementation, after filling the first input signal to obtain a first input signal with multiple frames of filling, the aforementioned generation module 801 is further used to combine the first input signal with multiple frames of filling to generate a second matrix.

[0107] In one possible implementation, the aforementioned matrix multiplication module 802 is specifically used to: perform matrix multiplication on the first matrix and the second matrix to obtain a third matrix multiplication matrix, wherein the third matrix multiplication matrix includes the result of FFT operation on multiple frames of the first input signal in the first matrix.

[0108] In one possible implementation, the aforementioned matrix multiplication module 802 is specifically used to: transpose the second matrix to obtain the transposed second matrix; and perform matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication.

[0109] In one possible implementation, after performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication, the device may further include: a splicing module 806, used to splice the real and imaginary data in the third matrix multiplication to obtain the result of short-time Fourier operation on the third input signal.

[0110] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps of the method described in the embodiment shown in FIG3 above.

[0111] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method steps described in the embodiment shown in FIG3 above.

[0112] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0119] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Based on the length of the first input signal, a rotation factor matrix is ​​generated. The rotation factor matrix includes multiple rotation factors, which are used to perform a Fast Fourier Transform (FFT) operation with the first input signal. Perform a matrix multiplication operation on the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix, which includes the result of performing an FFT operation on the first input signal.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the second input signal; When the length of the second input signal is the same as the length of the first input signal, a matrix multiplication operation is performed on the rotation factor matrix and the second input signal to obtain a second matrix multiplication matrix, which includes the result of the FFT operation on the second input signal.

3. The method according to claim 1 or 2, characterized in that, After generating the rotation factor matrix based on the length of the first input signal, the method further includes: The rotation factor matrix is ​​windowed according to the window function to obtain a first matrix, which includes the fusion information of the rotation factor matrix and the window function.

4. The method according to claim 3, characterized in that, Before generating the rotation factor matrix based on the length of the first input signal, the method further includes: A sliding window operation is performed on the third input signal to obtain multiple frames of the first input signal.

5. The method according to claim 4, characterized in that, After performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, the method further includes: The first input signal is filled to obtain a first input signal after multiple frames of filling.

6. The method according to claim 5, characterized in that, After filling the first input signal to obtain a first input signal with multiple filled frames, the method further includes: The first input signal after being filled with multiple frames is combined to generate a second matrix.

7. The method according to claim 6, characterized in that, The step of performing a matrix multiplication operation between the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix includes: Perform matrix multiplication on the first matrix and the second matrix to obtain a third matrix multiplication matrix, which includes the result of FFT operation on multiple frames of the first input signal in the first matrix.

8. The method according to claim 7, characterized in that, The step of performing matrix multiplication on the first matrix and the second matrix to obtain a third matrix multiplication includes: Transpose the second matrix to obtain the transposed second matrix; Perform matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication.

9. The method according to claim 8, characterized in that, After performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication, the method further includes: The real and imaginary data in the third matrix multiplication matrix are concatenated to obtain the result of the short-time Fourier operation on the third input signal.

10. A data processing apparatus, characterized in that, include: The generation module is used to generate a rotation factor matrix based on the length of the first input signal. The rotation factor matrix includes multiple rotation factors, which are used to perform a Fast Fourier Transform (FFT) operation with the first input signal. The matrix multiplication module is used to perform matrix multiplication on the rotation factor matrix and the first input signal to obtain a first matrix multiplication matrix, which includes the result of FFT operation on the first input signal.

11. The apparatus according to claim 10, characterized in that, The device further includes: The acquisition module is used to acquire the second input signal; The matrix multiplication module is further configured to perform a matrix multiplication operation on the rotation factor matrix and the second input signal when the length of the second input signal is the same as the length of the first input signal, to obtain a second matrix multiplication matrix, wherein the second matrix multiplication matrix includes the result of performing an FFT operation on the second input signal.

12. The apparatus according to claim 10 or 11, characterized in that, After generating the rotation factor matrix based on the length of the first input signal, the device further includes: The processing module is used to perform a windowing operation on the rotation factor matrix according to the window function to obtain a first matrix, the first matrix including the fusion information of the rotation factor matrix and the window function.

13. The apparatus according to claim 12, characterized in that, Before generating the rotation factor matrix based on the length of the first input signal, the device further includes: The framing module is used to perform a sliding window operation on the third input signal to obtain multiple frames of the first input signal.

14. The apparatus according to claim 13, characterized in that, After performing a sliding window operation on the third input signal to obtain multiple frames of the first input signal, The framing module is also used to fill the first input signal to obtain a first input signal after multiple frames are filled.

15. The apparatus according to claim 14, characterized in that, After filling the first input signal to obtain a multi-frame filled first input signal, The generation module is further configured to combine the first input signal after the multi-frame filling to generate a second matrix.

16. The apparatus according to claim 15, characterized in that, The matrix multiplication module is specifically used for: Perform matrix multiplication on the first matrix and the second matrix to obtain a third matrix multiplication matrix, which includes the result of FFT operation on multiple frames of the first input signal in the first matrix.

17. The apparatus according to claim 16, characterized in that, The matrix multiplication module is specifically used for: Transpose the second matrix to obtain the transposed second matrix; Perform matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication.

18. The apparatus according to claim 17, characterized in that, After performing matrix multiplication on the first matrix and the transposed second matrix to obtain the third matrix multiplication, the apparatus further includes: The splicing module is used to splice the real and imaginary data in the multiplication matrix of the third matrix to obtain the result of the short-time Fourier operation on the third input signal.

19. A processor, characterized in that, The processor is connected to a memory and is configured to execute instructions stored in the memory to cause the processor to perform the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 9.

21. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 9.

Images