Data processing apparatus and method, and chip, electronic device and storage medium

Abstract

A data processing apparatus, a data processing method, a chip, an electronic device and a computer-readable storage medium. The data processing apparatus comprises a signal processing unit and a neural network processing unit, wherein the signal processing unit is configured with a derivative operator operation strategy, and the neural network processing unit is configured with a neural network model. After the neural network model is updated or modified, operator input data of a preset derivative operator in the neural network model can be sent to the signal processing unit, and by reusing the computing power of the signal processing unit, a derivative operator operation strategy corresponding to the operator input data is executed and an obtained operator operation result is sent to the neural network processing unit.

Description

Data processing apparatus, method, chip, electronic device and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202510203336.6, filed on February 24, 2025, and entitled "Data processing apparatus, method, chip, electronic device and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application belongs to the technical field of data processing, and particularly relates to a data processing apparatus, a data processing method, a chip, an electronic device and a computer readable storage medium. BACKGROUND

[0003] With the rapid development of neural network models, more and more electronic devices are configured with neural network models for optimizing processing of data. For example, noise reduction optimization of audio signals, clarity optimization of pictures, denoising optimization of user physiological data, etc. In related data processing schemes, a processing unit is usually used in combination with a neural network processing unit (NPU) to implement optimization processing of data.

[0004] However, the NPU contains an architecture specially designed for neural network model calculation, such as fixed computing units, which can perform well in neural network model calculation, that is, the NPU has a strong correlation with the neural network model. If the operators in the neural network model are modified or updated, the original NPU may not be able to normally run the updated neural network model. SUMMARY

[0005] The present application aims to provide a data processing apparatus, a data processing method, a chip, an electronic device and a computer readable storage medium, and aims to solve the problem that the NPU cannot be normally used after the neural network model is updated in the existing data processing scheme.

[0006] A first aspect of the embodiments of the present application provides a data processing apparatus, comprising:

[0007] a signal processing unit configured with a derived operator operation strategy;

[0008] a neural network processing unit configured with a neural network model, the neural network processing unit being configured to send operator input data of a preset derived operator in the neural network model to the signal processing unit in a process of processing to-be-processed data by using the neural network model;

[0009] The signal processing unit is configured to perform a derivative operator operation strategy according to the operator input data to obtain an operator operation result, and send the operator operation result to the neural network processing unit.

[0010] The neural network processing unit is further configured to complete a processing operation on the to-be-processed data based on the operator operation result, and obtain neural network model output data of the to-be-processed data.

[0011] A second aspect of the embodiment of the present application provides a data processing method, applied to a signal processing unit, the signal processing unit being connected with a neural network processing unit, the neural network processing unit being configured with a neural network model, and the data processing method comprising the following steps.

[0012] Receiving operator input data sent by the neural network processing unit in a process of processing to-be-processed data by using the neural network model, wherein the operator input data corresponds to a preset derivative operator in the neural network model;

[0013] Performing a derivative operator operation strategy according to the operator input data to obtain an operator operation result;

[0014] Sending the operator operation result to the neural network processing unit, wherein the neural network processing unit is configured to complete a processing operation on the to-be-processed data based on the operator operation result, and obtain neural network model output data of the to-be-processed data.

[0015] In the above scheme, the data processing method further comprises the following steps.

[0016] Receiving neural network model output data sent by the neural network processing unit;

[0017] Outputting corresponding target data according to the neural network model output data.

[0018] In the above scheme, the data processing method further comprises the following steps.

[0019] Performing feature extraction on initial data to obtain first feature data;

[0020] Sending the first feature data to the neural network processing unit as to-be-processed data, wherein the neural network processing unit is configured to receive the first feature data, run the neural network model with the first feature data as the to-be-processed data, receive an operator operation result obtained by the signal processing unit according to the operator input data performing a derivative operator operation strategy, and obtain second feature data according to the operator operation result, and send the second feature data to the signal processing unit as neural network model output data.

[0021] In the above scheme, the receiving of the neural network model output data sent by the neural network processing unit comprises the following steps.

[0022] Receiving second feature data sent by the neural network processing unit.

[0023] output corresponding target data according to the neural network model output data, including:

[0024] output target data corresponding to the initial data according to the second feature data.

[0025] In the above scheme, the neural network processing unit is further configured with a calling interface corresponding to the derivative operator; and a derivative operator operation strategy is executed according to the operator input data, and an operator operation result is obtained, including:

[0026] When the operator input data sent by the neural network processing unit through the calling interface is received, the derivative operator operation strategy is executed according to the operator input data to obtain the operator operation result. The operator input data is a parameter of a preset derivative operator in the neural network model.

[0027] In the above scheme, the operator operation result is sent to the neural network processing unit, including: the operator operation result is sent to the neural network processing unit through the calling interface.

[0028] A third aspect of the embodiments of the present application provides a data processing method applied to a neural network processing unit, the neural network processing unit being connected with a signal processing unit, the neural network processing unit being configured with a neural network model, and the data processing method including:

[0029] In the process of processing the to-be-processed data by using the neural network model, operator input data of a preset derivative operator in the neural network model is sent to the signal processing unit; wherein the signal processing unit is configured with a derivative operator operation strategy, and the signal processing unit is used for executing the derivative operator operation strategy according to the operator input data to obtain an operator operation result, and sending the operator operation result to the neural network processing unit;

[0030] The operator operation result sent by the signal processing unit is received.

[0031] Based on the operator operation result, a processing operation on the to-be-processed data is completed, and neural network model output data of the to-be-processed data is obtained.

[0032] In the above scheme, the data processing method further includes:

[0033] The neural network model output data is sent to the signal processing unit; wherein the signal processing unit is further used for receiving the neural network model output data, and outputting corresponding target data according to the neural network model output data.

[0034] In the above scheme, the data processing method further includes:

[0035] The first feature data sent by the signal processing unit is used as the to-be-processed data to run the neural network model; wherein the first feature data is obtained by the signal processing unit by performing feature extraction on the initial data.

[0036] In the above scheme, the processing operation on the to-be-processed data is completed based on the operator operation result, and the neural network model output data of the to-be-processed data is obtained, including:

[0037] The processing operation on the first feature data is completed according to the operator operation result, and the second feature data is obtained as the neural network model output data.

[0038] In the above scheme, the neural network model output data is sent to the signal processing unit, including:

[0039] The second feature data is sent to the signal processing unit as the neural network model output data.

[0040] In the above scheme, the neural network processing unit is further configured with a calling interface corresponding to the derivative operator; in the process of processing the to-be-processed data by using the neural network model, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit, including:

[0041] In the process of processing the first feature data by using the neural network model, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit through the calling interface.

[0042] In the above scheme, the operator operation result sent by the signal processing unit is received, including:

[0043] The operator operation result sent by the signal processing unit through the calling interface is received.

[0044] The fourth aspect of the embodiment of the present application provides a chip, which is used to execute the steps of the data processing method provided in the second aspect or the third aspect.

[0045] The fifth aspect of the embodiment of the present application provides an electronic device, which includes the data processing device provided in the first aspect or includes the chip provided in the fourth aspect.

[0046] The sixth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the data processing method provided in the second aspect or the third aspect.

[0047] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0048] The data processing apparatus provided in the foregoing embodiment comprises a signal processing unit and a neural network processing unit. The signal processing unit is configured with a derivative operator operation strategy, and the neural network processing unit is configured with a neural network model. In the process of processing the to-be-processed data by using the neural network model, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit, so that the signal processing unit can execute the derivative operator operation strategy according to the operator input data, and the obtained operator operation result is sent to the neural network processing unit. Subsequently, the neural network processing unit can complete the processing operation on the to-be-processed data based on the operator operation result, and obtain corresponding neural network model output data. In this way, after the neural network model is updated or modified, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit, the computing power of the signal processing unit is reused to execute the derivative operator operation strategy corresponding to the operator input data and send the obtained operator operation result to the neural network processing unit, so that the neural network processing unit can obtain the neural network model output data corresponding to the to-be-processed data according to the operator operation result without improving the existing neural network processing unit. The phenomenon that the neural network processing unit cannot be normally used after the neural network model is updated is avoided.

[0049] In addition, after the neural network model is updated, the operation logic in the neural network model that is different from that before the update can be used as a preset derivative operator, and the operator input data of the derivative operator is sent to the signal processing unit. The computing power of the signal processing unit is reused to execute the derivative operator operation strategy corresponding to the operator input data and send the obtained operator operation result to the neural network processing unit, without the need to redesign the neural network processing unit or replace the existing neural network processing unit, thereby reducing the cost of updating the neural network model and providing a data processing scheme that can widen the application range of the neural network processing unit. BRIEF DESCRIPTION OF DRAWINGS

[0050] FIG. 1 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0051] FIG. 2 is a schematic diagram of an application scenario of a data processing apparatus provided by an embodiment of the present application;

[0052] FIG. 3 is an implementation flowchart of a data processing method provided by an embodiment of the present application;

[0053] FIG. 4 is an implementation flowchart of a data processing method provided by another embodiment of the present application;

[0054] FIG. 5 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. EMBODIMENTS OF THE PRESENT INVENTION

[0055] In order to make the technical problems to be solved by the present application, technical solutions and beneficial effects clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0056] It should be noted that the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly and specifically limited.

[0057] For example, in the related art, a processing unit is usually used in combination with a neural network processing unit (NPU) to implement optimized processing of data.

[0058] Here, the largest proportion of neural network model operations is vector or matrix multiplication and accumulation operations, and NPU is a special processor designed to accelerate neural network model operations. NPU contains an architecture designed specifically for neural network model calculations, such as fixed computing units, so it can perform well in neural network model calculations, that is, NPU and neural network model have a strong correlation.

[0059] However, the neural network model is updated and iterated quickly, and in order to improve the performance of the neural network model, the operators in the neural network model must be modified when the neural network model is updated. Because of the strong correlation between NPU and neural network model, once the NPU is designed, the neural network operators that the NPU can support are basically determined, so it is difficult to further extend support for different neural network operators. If the operators in the neural network model are modified or updated, it is easy to cause the original NPU to be unable to normally run the updated neural network model. It can be seen that NPU has poor flexibility and its scope of application is easily limited.

[0060] In addition, if the operators in the neural network model are modified or updated, adjusting the NPU will require a large cost.

[0061] In some embodiments, the present embodiment provides a data processing apparatus, comprising a signal processing unit and a neural network processing unit. The signal processing unit is configured with a derivative operator operation strategy, and the neural network processing unit is configured with a neural network model. In the process of processing the to-be-processed data by using the neural network model via the neural network processing unit, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit, so that the signal processing unit can execute the derivative operator operation strategy according to the operator input data, and the obtained operator operation result is sent to the neural network processing unit. Subsequently, the neural network processing unit can complete the processing operation of the to-be-processed data based on the operator operation result, and obtain the corresponding neural network model output data. In this way, after the neural network model is updated or modified, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit, the computing power of the signal processing unit is reused to execute the derivative operator operation strategy corresponding to the operator input data and send the obtained operator operation result to the neural network processing unit, so that the neural network processing unit can obtain the neural network model output data corresponding to the to-be-processed data according to the operator operation result without improving the existing neural network processing unit. The phenomenon that the neural network processing unit cannot be normally used after the neural network model is updated is avoided.

[0062] In addition, after the neural network model is updated, the operation logic in the neural network model that is different from that before the update can be used as a preset derivative operator, and the operator input data of the derivative operator is sent to the signal processing unit. The computing power of the signal processing unit is reused to execute the derivative operator operation strategy corresponding to the operator input data and send the obtained operator operation result to the neural network processing unit, without the need to redesign the neural network processing unit or replace the existing neural network processing unit, thereby reducing the cost of updating the neural network model and providing a data processing scheme that can widen the application range of the neural network processing unit.

[0063] The data processing apparatus provided by the present embodiment can be configured in an electronic device. In actual use, the electronic device can be a consumer electronic device that needs to process data by using a neural network model. For example, wearable electronic devices such as smart bracelets, smart rings, etc. For another example, cameras, Bluetooth earphones, mobile phones, tablet computers, or notebook computers, etc. When receiving or generating corresponding data, the electronic device can process the to-be-processed data by using the data processing apparatus provided by the present embodiment.

[0064] The data processing apparatus provided by the present embodiment is described in detail through specific implementation manners.

[0065] FIG. 1 shows a structural schematic diagram of a data processing apparatus provided in an embodiment of the present application. As shown in FIG. 1, the data processing apparatus 100 includes a signal processing unit 10 and a neural network processing unit 20. Specifically,

[0066] The signal processing unit 10 is configured with a derivation operator operation strategy. The neural network processing unit 20 is configured with a neural network model. The neural network processing unit 20 is configured to send operator input data of a preset derivation operator in the neural network model to the signal processing unit 10 in a process of processing the to-be-processed data by using the neural network model. The signal processing unit 10 is configured to perform the derivation operator operation strategy according to the operator input data to obtain operator operation results, and send the operator operation results to the neural network processing unit 20. The neural network processing unit 20 is further configured to complete the processing operation on the to-be-processed data based on the operator operation results, and obtain corresponding neural network model output data.

[0067] In the embodiment, the to-be-processed data can be data generated by the signal processing unit 10, or data received by the signal processing unit 10, which is not limited herein. The neural network processing unit 20 is pre-configured with a neural network model. When the to-be-processed data is processed by using the neural network model, the to-be-processed data can be input into the neural network processing unit 20, and the operation process of the neural network model is performed by the neural network processing unit 20 to implement the processing operation on the to-be-processed data.

[0068] For example, the to-be-processed data can be sent to the neural network processing unit 20 by the signal processing unit 10, or sent to the neural network processing unit 20 by other units, which is not limited herein.

[0069] For example, a data generation unit such as an audio data generation module, an image data generation module, etc. generates the to-be-processed data.

[0070] For another example, a communication unit such as a Bluetooth module, an NFC module, a mobile data communication module, etc. receives the to-be-processed data, and sends the received data to the neural network processing unit 20.

[0071] In a specific implementation, the signal processing unit 10 and the neural network processing unit 20 can establish a wired communication link to exchange data according to an existing communication protocol, for example, any one of serial communication protocols I2C, SPI, UART, PCIe and CAN bus communication protocols, or a parallel communication protocol or a DDR protocol.

[0072] In all embodiments of the present application, a derived operator operation strategy is configured in the signal processing unit 10, which is used to describe the method or rule of performing derived operator operation / calculation based on operator input data. Here, the derived operator generally refers to an operator in the neural network model that cannot be directly operated by the neural network processing unit 20.

[0073] It is easy to understand that the operator in the neural network model is the basic calculation unit of the neural network, which is used to perform specific mathematical operations or data processing operations. It can be understood that the operator is the core of data flow and calculation in the neural network, and each operator defines the mapping relationship from input data to output data. Multiple operators are connected according to the network structure to jointly complete the inference or training task of the neural network. Common operators include basic mathematical operations, activation functions, convolution, pooling, normalization, etc.

[0074] In actual implementation, when the neural network model is optimized, the implementation logic of the operator in the neural network model can be changed, or the custom operator can be modified, so that the neural network model framework has a more rich or precise operator structure. Based on this, the original neural network processing unit 20 may not be able to operate the updated neural network model.

[0075] In this embodiment, in order for the data processing apparatus 100 to be able to process the to-be-processed data using the neural network model, in the case of updating the neural network model in the neural network processing unit 20, by defining or declaring a preset derived operator in the neural network model, the operator input data of the preset derived operator in the neural network model is sent to the signal processing unit 10 in the process of processing the to-be-processed data by the neural network processing unit 20 using the neural network model. Here, the operator input data is the input data of the derived operator, and the signal processing unit 10 performs the derived operator operation strategy according to the operator input data, so that the corresponding output data can be obtained as the operator operation result.

[0076] In specific implementation, the neural network model configured in the neural network processing unit 20 can be specifically implemented by configuring a neural network model file in the neural network processing unit 20. Here, the neural network model file can specifically include a header file defining the operator, a custom operator file, etc. Correspondingly, the derived operator can be defined in the header file and / or custom operator file in the model file.

[0077] It can be understood that the model file can also include a bin file for storing model weights to facilitate loading and sharing the model. In some examples, the preset derived operator can also be defined or declared in the bin file of the neural network model to implement the specified operation of the derived operator in the neural network model.

[0078] In the embodiment, since the signal processing unit 10 is configured with the derivative operator operation strategy, the signal processing unit 10 can execute the derivative operator operation strategy according to the operator input data and send the obtained operator operation result to the neural network processing unit 20. In this way, the multiplexing of the signal processing unit 10 is realized, and the utilization rate of the signal processing unit 10 can be improved.

[0079] For example, in specific implementation, the signal processing unit 10 can be implemented by using a processor with programmability. For example, any one of an existing digital signal processor (DSP), a micro controller unit (MCU), and a central processing unit (CPU) can be selected.

[0080] For example, the signal processing unit 10 is a DSP, and the neural network processing unit 20 is an NPU. In specific implementation, since the DSP has programmability, the derivative operator operation strategy corresponding program can be configured in the DSP, so that the DSP can execute the derivative operator operation strategy according to the operator input data and send the obtained operator operation result to the NPU.

[0081] For example, in specific implementation, a convention file of the derivative operator can be configured in the signal processing unit 10, and the convention file is used to describe the correspondence between the operator input data of the derivative operator and the derivative operator operation strategy.

[0082] In combination with the above example, the convention file can be a mapping list between the operator input data of the derivative operator and the derivative operator operation strategy. In specific implementation, when the DSP / MCU / CPU receives the operator input data sent by the NPU, the corresponding derivative operator operation strategy is determined from the mapping list according to the serial number carried by the operator input data, and then the corresponding derivative operator operation strategy can be executed according to the operator input data, and the obtained operator operation result is sent to the NPU.

[0083] It can be understood that in actual use, a plurality of derivative operators can be preset in the neural network model, that is, in the process of processing the to-be-processed data by using the neural network model, the neural network processing unit 20 can send the operator input data of the plurality of preset derivative operators in the neural network model to the signal processing unit 10. Here, the neural network processing unit 20 sends the operator input data of the plurality of preset derivative operators in the neural network model to the signal processing unit 10, which can be specifically according to the operation order of the plurality of preset derivative operators in the neural network model.

[0084] Taking preset derivative operators contained in the neural network model as an example, the first derivative operator, the second derivative operator, and the third derivative operator, it is assumed that the second operator input data of the second derivative operator and the third operator input data of the third derivative operator need to be determined after the operator operation result of the first derivative operator, the first operator input data of the first derivative operator can be sent to the signal processing unit 10 first. After the signal processing unit 10 executes the operation strategy corresponding to the first derivative operator according to the first operator input data, the obtained first operator operation result is sent to the neural network processing unit 20. The neural network processing unit 20 determines the second operator input data of the second derivative operator and the third operator input data of the third derivative operator according to the first operator operation result, and then sends the second operator input data and the third operator input data to the signal processing unit 10. After the signal processing unit 10 executes the operation strategy corresponding to the second derivative operator and the operation strategy corresponding to the third derivative operator according to the second operator input data and the third operator input data, the obtained second operator operation result and the third operator operation result are sent to the neural network processing unit 20. The neural network processing unit 20 obtains the neural network model output data corresponding to the to-be-processed data according to the second operator operation result and the third operator operation result.

[0085] In actual use, the neural network model in the neural network processing unit 20 can be configured according to the to-be-processed data. For example, the to-be-processed data is any one of audio data, picture data, biological data, and sensor collected data, and correspondingly, the neural network model in the neural network processing unit 20 can be any one of an audio processing / recognition model, a picture processing / recognition model, a biological data processing / recognition model, and a sensor data processing model. Here, it can be understood that the to-be-processed data can be data received by the signal processing unit 10, or data generated by the signal processing unit 10. That is, in specific implementation, the to-be-processed data can be sent by the signal processing unit 10 to the neural network processing unit 20.

[0086] As an example, taking the to-be-processed data as audio data, the neural network model in the neural network processing unit 20 can specifically be a voice data processing / optimization model for noise reduction, echo cancellation, or music-vocal separation of audio data. Here, the audio data can be audio data received by the signal processing unit 10 from a microphone, audio data received by a communication chip, or audio data obtained by analyzing memory data, which is not limited here.

[0087] As another example, taking the data to be processed as picture data as an example, the neural network model in the neural network processing unit 20 can specifically be an image data processing / optimization model for feature recognition, image element separation, or image quality improvement of the picture data. Here, the picture data can be picture data received by the signal processing unit 10 from a processor, picture data received by the signal processing unit 10 from a communication chip, or picture data obtained by parsing memory data, which is not limited here.

[0088] As another example, taking the data to be processed as picture data as an example, the neural network model in the neural network processing unit 20 can specifically be an image data processing / optimization model for feature recognition, image element separation, or picture quality improvement of the picture data. Here, the picture data can be picture data received by the neural network processing unit 20 from a processor, picture data received by the neural network processing unit 20 from a communication chip, or picture data obtained by parsing memory data, which is not limited here. As a further example, taking the data to be processed as biological data as an example, the neural network model in the neural network processing unit 20 can specifically biological data processing / optimization model for biological feature recognition, biological feature separation, or biological feature noise reduction of the biological data. Here, the biological data can be biological data received by the signal processing unit 10 from a processor, biological data received by the signal processing unit 10 from a communication chip, or biological data obtained by parsing memory data, which is not limited here.

[0089] In some embodiments, the signal processing unit 10 in the data processing apparatus 100 can be multiple, and the neural network processing unit 20 sends the operator input data of the preset derivative operator in the neural network model to the signal processing unit 10 in the process of processing the data to be processed by the neural network model. Specifically, the operator input data of the derivative operator can be sent to any one of the multiple signal processing units 10, or the operator input data can be sent to a specific signal processing unit 10 in the multiple signal processing units 10 based on a preset calling interface.

[0090] Based on this, in actual application, different derivative operator operation strategies can be configured for one derivative operator in different signal processing units 10, so that the neural network processing unit 20 can send the operator input data of the same derivative operator in multiple neural network models to different signal processing units 10 in the process of processing the data to be processed by the multiple neural network models, and the multiple signal processing units 10 execute different derivative operator operation strategies to obtain different operator operation results according to the operator input data of the same derivative operator. In this way, not only can it provide an implementation basis for the parallel running of the neural network model in the neural network processing unit 20, but also can meet the needs of different derivative operator operation strategies of the same derivative operator in the neural network model in the neural network processing unit 20.

[0091] In some embodiments, one or more derivative operator operation strategies can also be configured in the signal processing unit 10 for the same derivative operator. When the neural network processing unit 20 is processing the data to be processed using the neural network model, the operator input data of the preset derivative operator in the neural network model is sent to the signal processing unit 10, and the operator input data can carry different sub-serial numbers to distinguish different derivative operator operation strategies corresponding to the operator input data.

[0092] For example, a convention file of the derivative operator can be configured in the signal processing unit 10, which is used to describe the correspondence between the operator input data of the derivative operator and the derivative operator operation strategy. For example, the convention file can be a mapping list between the operator input data of the derivative operator and the derivative operator operation strategy. In combination with the above example, when the DSP / MCU / CPU receives the operator input data sent by the NPU, the corresponding derivative operator operation strategy is determined from the mapping list according to the serial number carried by the operator input data. At this time, if the determined corresponding derivative operator operation strategy is multiple, the target derivative operator operation strategy can also be determined from the multiple derivative operator operation strategies according to the sub-serial number carried by the operator input data. Then, the target derivative operator operation strategy can be executed according to the operator input data, and the obtained operator operation result is sent to the NPU. In this way, not only can the parallel running of the neural network model by the neural network processing unit 20 be provided with an implementation basis, but also the different derivative operator operation strategies of the neural network model in the neural network processing unit 20 for the same derivative operator can be met.

[0093] As an embodiment, the neural network processing unit 20 is also configured to send the neural network model output data to the signal processing unit 10. Correspondingly, the signal processing unit 10 is also configured to receive the neural network model output data and output the corresponding target data according to the neural network model output data. It is easy to understand that in actual use, since the neural network processing unit 20 is used to accelerate the calculation of the neural network model, when outputting specific target data, the received neural network model output data needs to be further integrated or processed by the signal processing unit 10, and then the target data corresponding to the data to be processed is output.

[0094] As an example, taking the to-be-processed data as audio data, the neural network model in the neural network processing unit 20 can specifically be a speech data processing / optimization model for noise reduction, echo cancellation, or music voice separation of the audio data. The neural network processing unit 20 sends the neural network model output data obtained by processing the audio data via the neural network model to the signal processing unit 10, the signal processing unit 10 receives the neural network model output data, and outputs corresponding target data according to the neural network model output data, which can specifically be audio data after noise reduction, audio data after echo cancellation, or human voice audio data after music separation, etc.

[0095] As another example, taking the to-be-processed data as picture data, the neural network model in the neural network processing unit 20 can specifically be an image data processing / optimization model for feature recognition, image element separation, or image quality improvement of the picture data. The neural network processing unit 20 sends the neural network model output data obtained by processing the picture data via the neural network model to the signal processing unit 10, the signal processing unit 10 receives the neural network model output data, and outputs corresponding target data according to the neural network model output data, which can specifically be image features, elements, or image data with improved image quality (such as definition, color saturation, etc.) recognized from the image data, etc.

[0096] As another example, taking the to-be-processed data as picture data, the neural network model in the neural network processingunit 20 can specifically be an image data processing / optimization model for feature recognition, image elementseparation, or image quality improvement of the picture data. The neural network processing unit 20 sends theneural network model output data obtained by processing the picture data via the neural network model to thesignal processing unit 10, the signal processing unit 10 receives the neural network model output data, andre-outputs corresponding target data according to the neural network model output data, which can specificallybe image features, elements, or image data with improved image quality (such as definition, color saturation,

[0097] It can be understood that in specific implementation, the to-be-processed data can be audio data, image data, or biological data itself, or can be feature data obtained by feature extraction on the above data.

[0098] As an embodiment, the signal processing unit 10 is further configured to perform feature extraction on the initial data to obtain first feature data, and send the first feature data to the neural network processing unit 20. Correspondingly, the neural network processing unit 20 is specifically configured to receive the first feature data, run the neural network model with the first feature data as the to-be-processed data, receive the operator operation result obtained by the signal processing unit 10 according to the operator input data performing the derived operator operation strategy, and obtain second feature data according to the operator operation result, and send the second feature data as the neural network model output data to the signal processing unit 10.

[0099] In the embodiment, the initial data is the audio data, the image data or the biological data itself. The feature extraction on the initial data by the signal processing unit 10 is to extract the first feature data in the audio data, the first feature data in the image data or the first feature data in the biological data.

[0100] For example, the first feature data obtained by the signal processing unit 10 performing feature extraction on the audio data can be spectral feature data, mel feature data, bark feature data, etc.

[0101] For another example, the first feature data obtained by the signal processing unit 10 performing feature extraction on the image data can be RGB channel feature data, grayscale feature data, etc.

[0102] For still another example, the first feature data obtained by the signal processing unit 10 performing feature extraction on the biological data can be electrical characteristic data of the organism, nine-axis data of the sensor and / or quaternion data obtained from the nine-axis data, etc.

[0103] In the embodiment, the neural network processing unit 20 receives the first feature data as the to-be-processed data, inputs the neural network model, and in the process of executing the neural network model, sends the operator input data of the preset derived operator to the signal processing unit 10, and then receives the operator operation result obtained by the signal processing unit 10 according to the operator input data performing the derived operator operation strategy. The neural network processing unit 20 obtains the second feature data according to the operator operation result, and sends the second feature data as the neural network model output data to the signal processing unit 10.

[0104] As an embodiment, the signal processing unit 10 is specifically configured to output target data corresponding to the initial data according to the second feature data.

[0105] In this embodiment, the target data refers to data obtained after the initial data is processed via the neural network model. In this embodiment, the signal processing unit 10 and the neural network processing unit 20 cooperate to implement the operation of the neural network model. That is, the signal processing unit 10 and the neural network processing unit 20 cooperate to implement the processing of the initial data via the neural network model to obtain the target data. Specifically, the signal processing unit 10 extracts features from the initial data to obtain first feature data, and the neural network processing unit 20 and the signal processing unit 10 cooperate to implement the operation of processing the first feature data via the neural network model to obtain second feature data, and finally the signal processing unit 10 outputs the target data corresponding to the initial data according to the second feature data.

[0106] It should be noted that the neural network model can also be configured in the signal processing unit 10, and the signal processing unit 10 performs the operation of the neural network model. For example, in related solutions, the neural network model can be configured only in the signal processing unit 10, that is, only the signal processing unit 10 performs the operation of the neural network model. In this way, although the target data corresponding to the initial data can also be obtained, the signal processing unit 10 does not have unique optimization for the operation of the neural network model, so the efficiency is lower compared to the processing of the initial data by the signal processing unit 10 and the neural network processing unit 20.

[0107] In specific implementation, the initial data can be at least one of audio data, image data, and biological data.

[0108] In one example, taking the target data as target audio data processed via the neural network model as an example, the target audio data can be audio data after noise reduction, audio data after echo elimination, or human voice audio data after background music is added, etc.

[0109] In another example, taking the target data as target image data processed via the neural network model as an example, the target image data can be image data marked with feature content, image data marked with specific elements, or image data with improved quality compared to the original image data, etc.

[0110] In yet another example, taking the target data as target biological data processed via the neural network model as an example, the target biological data can be certain feature data, all feature data, or biological posture / action data identified from the initial biological data, etc.

[0111] FIG. 2 shows a schematic diagram of an application scenario of the data processing apparatus provided in the embodiments of the present application. As shown in FIG. 2, taking initial data as initial audio data with noise as an example, the signal processing unit 10 performs feature extraction on the initial data, and the obtained first feature data can be at least one of spectral features, mel features and bark features. In a specific implementation, the signal processing unit 10 can perform frame windowing processing on the initial audio data, and after the signal processing unit 10 performs short-time Fourier transform on the audio data after the frame windowing processing, the frequency spectrum corresponding to the audio data can be obtained, and by performing feature extraction on the frequency spectrum, at least one of the spectral features, the mel features and the bark features can be obtained. The signal processing unit 10 sends the extracted spectral features to the neural network processing unit 20 as to-be-processed data. The neural network processing unit 20 processes the spectral features by using the neural network model deployed / configured by the neural network processing unit 20. In this process, the neural network processing unit 20 can send the operator input data of the preset derivative operator to the signal processing unit 10, and the signal processing unit 10 performs the derivative operator operation strategy according to the operator input data to obtain the operator operation result. The neural network processing unit 20 obtains the second feature data, i.e., the processed spectral feature data, by receiving the operator operation result in combination with the overall operation result of the neural network model, and then sends the processed spectral feature data to the signal processing unit 10. Here, the processed spectral feature data can be specifically a predicted amplitude mask obtained by predicting the spectral features by using the neural network model. Here, the preset amplitude mask is used to describe the proportion of the speech content in the initial audio data. Since the short-time Fourier transform obtains the frequency spectrum of the initial speech data, the signal processing unit 10 can multiply the predicted amplitude mask and the audio after the short-time Fourier transform, and thus the denoised frequency spectrum data can be obtained. The target audio data, i.e., the output target data, can be output by performing inverse Fourier transform and audio encoding on the denoised frequency spectrum data.

[0112] It is understood that in other embodiments, when the initial data is image data, the neural network processing unit 20 obtains the second feature data, i.e., the processed image feature data, by receiving the operator operation result and combining it with the overall operation result of the neural network model, and then sends the processed image feature data to the signal processing unit 10. Here, the processed image feature data may specifically be RGB features, grayscale feature data, etc., obtained by predicting the image data through the neural network model. By integrating the processed image feature data, the signal processing unit 10 can obtain the recognition result of the image content, or obtain higher quality image content as target data. Similarly, when the initial data is biological data, the neural network processing unit 20 obtains the second feature data, i.e., the processed biological feature data, by receiving the operator operation result and combining it with the overall operation result of the neural network model, and then sends the processed biological feature data to the signal processing unit 10. Here, the processed biological feature data may specifically be identifiable features obtained by predicting the biological data through the neural network model, such as facial features, fingerprints, or voiceprints, various separable biological features, such as heart rate, blood pressure, blood oxygen content, etc., or biological feature data after noise reduction, etc. The signal processing unit 10 can obtain corresponding recognition results or obtain the action or posture of the organism as target data by integrating the processed biometric data.

[0113] As an example, the neural network processing unit 20 is also configured with a calling interface corresponding to the derived operator. Specifically, the neural network processing unit 20 is used to send the operator input data of the preset derived operator in the neural network model to the signal processing unit 10 through the calling interface during the processing of the first feature data using the neural network model. Correspondingly, the signal processing unit 10 is used to return the operator operation result to the neural network processing unit 20 through the calling interface.

[0114] In this embodiment, the derived operators in the neural network model of the neural network processing unit 20 are configured with calling interfaces. The calling interface is used to send operator input data to the signal processing unit 10, and the signal processing unit 10 returns the operator operation results based on the operator input data.

[0115] In practical implementation, the calling interface can be defined using a function that calls the signal processing unit 10. Specifically, when configuring the calling interface for the derived operator in the neural network model, the interaction format related to the derived operator between the signal processing unit 10 and the neural network processing unit 20 can be defined using a function. For example, the data structure of the output result of the previous layer of the derived operator can be defined, and the output result of the previous layer can be packaged and sent to the signal processing unit 10 as the operator input data. The signal processing unit 10 returns the operator operation result to the neural network processing unit 20 according to the data format defined in the function.

[0116] Understandably, in a practical implementation, the neural network processing unit 20 deploys the model file of the corresponding neural network model, and the pre-defined derivative operators in the neural network model can also be defined or declared in this model file. Correspondingly, the definition of the calling interface can also be included in the model file. Unlike the neural network processing unit 20, the signal processing unit 10 may only deploy a strategy file describing the derivative operator operation strategy. In practical applications, the derivative operator operation strategy can be modified by modifying the content of this strategy file. Here, based on the programmability of the signal processing unit 10, the neural network model can be further updated or optimized by enriching the derivative operator operation strategy, providing a new way to update the neural network model.

[0117] Figure 3 shows a flowchart of a data processing method provided in an embodiment of this application. In the example shown in Figure 3, the data processing method is applied to a signal processing unit. Here, the signal processing unit is connected to a neural network processing unit, the neural network processing unit is configured with a neural network model, and the data processing method includes:

[0118] S110: Receives operator input data sent by the neural network processing unit during the processing of the data to be processed using the neural network model. The operator input data corresponds to a pre-defined derived operator in the neural network model.

[0119] S120: Execute the derived operator operation strategy based on the operator input data to obtain the operator operation result.

[0120] S130: The operator operation result is sent to the neural network processing unit, which is used to complete the processing operation on the data to be processed based on the operator operation result, and obtain the corresponding neural network model output data.

[0121] As one embodiment, the data processing method further includes:

[0122] Receive the output data of the neural network model sent by the neural network processing unit. Output the corresponding target data based on the output data of the neural network model.

[0123] As one embodiment, the data processing method further includes:

[0124] Feature extraction is performed on the initial data to obtain the first feature data. This first feature data is then sent to the neural network processing unit as the data to be processed. The neural network processing unit receives the first feature data, runs a neural network model using it as the data to be processed, and receives signals from the signal processing unit.

[0125] The derivative operator operation strategy is executed based on the operator input data to obtain the operator operation result, and the second feature data is obtained based on the operator operation result. The second feature data is then sent to the signal processing unit as the output data of the neural network model.

[0126] As an example, the above step of receiving neural network model output data sent by the neural network processing unit includes:

[0127] Receive the second feature data sent by the neural network processing unit.

[0128] As an example, the above steps, which output the corresponding target data based on the output data of the neural network model, include:

[0129] Output the target data corresponding to the initial data based on the second feature data.

[0130] As an example, the neural network processing unit is also configured with a calling interface corresponding to the derived operator.

[0131] The above step S120 includes:

[0132] When the neural network processing unit receives operator input data sent through the interface, it executes the derived operator operation strategy according to the operator input data to obtain the operator operation result. The operator input data is the parameter of the pre-set derived operator in the neural network model.

[0133] As an example, step S130 above includes: sending the operator operation result to the neural network processing unit by calling the interface.

[0134] It is understood that the improvements and specific implementation methods related to this application have been described in detail in the embodiments corresponding to Figures 1 and 2. In specific implementation, based on the embodiments corresponding to Figures 1 and 2, the method steps shown in Figure 3 can be executed by the signal processing unit 10 in Figure 1, thereby realizing each step in the data processing method embodiment shown in Figure 3, so it will not be described again here.

[0135] Figure 4 shows a flowchart of a data processing method according to another embodiment of this application. In the example shown in Figure 4, the data processing method is applied to a neural network processing unit, which is connected to a signal processing unit. The neural network processing unit is configured with a neural network model, and the data processing method includes:

[0136] S210: During the process of processing the data to be processed using the neural network model, the operator input data of the pre-defined derived operators in the neural network model is sent to the signal processing unit; wherein, the signal processing unit is configured with a derived operator operation strategy, and the signal processing unit is used to execute the derived operator operation strategy according to the operator input data to obtain the operator operation result, and send the operator operation result to the neural network processing unit.

[0137] S220: Receive the operator operation result sent by the signal processing unit.

[0138] S230: Based on the operator operation results, complete the processing operation on the data to be processed, and obtain the corresponding neural network model output data.

[0139] As one embodiment, the data processing method further includes:

[0140] The output data of the neural network model is sent to the signal processing unit; the signal processing unit is also used to receive the output data of the neural network model and output the corresponding target data according to the output data of the neural network model.

[0141] As one embodiment, the data processing method further includes:

[0142] The system receives first feature data sent by the signal processing unit and runs a neural network model using the first feature data as the data to be processed; wherein, the first feature data is obtained by the signal processing unit from the initial data through feature extraction.

[0143] As an example, step S230 above includes:

[0144] The first feature data is processed based on the operator operation results, and the second feature data is obtained as the output data of the neural network model.

[0145] As an example, the above step of sending the neural network model output data to the signal processing unit includes:

[0146] The second feature data is sent to the signal processing unit as the output data of the neural network model.

[0147] As one embodiment, the neural network processing unit is also configured with a calling interface corresponding to the derived operator. Step S210 above includes:

[0148] During the process of processing the first feature data using a neural network model, the operator input data of the pre-defined derived operators in the neural network model are sent to the signal processing unit through the calling interface.

[0149] As an example, step S220 above includes:

[0150] The signal processing unit receives the operator operation results sent by calling the interface.

[0151] It is understood that the improvements and specific implementation methods related to this application have been described in detail in the embodiments corresponding to Figures 1 and 2. In specific implementation, based on the embodiments corresponding to Figures 1 and 2, the method steps shown in Figure 4 can be executed by the neural network processing unit 20 in Figure 1, thereby realizing each step in the data processing method embodiment shown in Figure 4, so they will not be described again here.

[0152] This application embodiment also provides another chip, which is used to execute the data processing method shown in FIG3, or to execute the data processing method shown in FIG4.

[0153] It is understood that the improvements and specific implementation methods related to this application have been described in detail in the embodiments corresponding to Figures 1 and 2. In specific implementation, based on the embodiments corresponding to Figures 1 and 2, the chip can implement each step in the data processing method embodiment shown in Figure 3 or Figure 4, so it will not be repeated here.

[0154] Figure 5 is a structural block diagram of an electronic device provided in an embodiment of this application. As shown in Figure 5, the electronic device 200 of this embodiment includes the data processing device 100 shown in Figure 1.

[0155] As an example, in a specific implementation, the data processing device 100 may also be packaged as a chip, and correspondingly, the electronic device 200 may include a chip for performing the method steps shown in FIG3 and / or the method steps shown in FIG4.

[0156] It is understood that the improvements and specific implementation methods related to this application have been described in detail in the embodiments corresponding to Figures 1 and 2. In specific implementation, the data processing device 100 can be configured in the electronic device 200 based on the embodiments corresponding to Figures 1 and 2, so it will not be described again here.

[0157] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in each of the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing apparatus, comprising: The signal processing unit is configured with a derivative operator operation strategy; A neural network processing unit, configured with a neural network model, is used for: During the process of processing the data to be processed using the neural network model, the operator input data of the preset derived operators in the neural network model are sent to the signal processing unit; The signal processing unit is used to execute the derived operator operation strategy according to the operator input data to obtain the operator operation result, and send the operator operation result to the neural network processing unit; The neural network processing unit is also used to perform processing operations on the data to be processed based on the operator operation results, and obtain the corresponding neural network model output data.

2. The data processing apparatus as claimed in claim 1, wherein, The neural network processing unit is also used to send the output data of the neural network model to the signal processing unit; The signal processing unit is further configured to receive the output data of the neural network model and output corresponding target data based on the output data of the neural network model.

3. The data processing apparatus as described in claim 2, wherein, The signal processing unit is further configured to extract features from the initial data to obtain first feature data, and send the first feature data to the neural network processing unit; The neural network processing unit is specifically configured to: receive the first feature data, run the neural network model using the first feature data as the data to be processed, receive the operator operation result obtained by the signal processing unit from the derivative operator operation strategy based on the operator input data, obtain the second feature data based on the operator operation result, and send the second feature data as the output data of the neural network model to the signal processing unit.

4. The data processing apparatus as described in claim 3, wherein, The signal processing unit is specifically used to output the target data corresponding to the initial data based on the second feature data.

5. The data processing apparatus according to any one of claims 1 to 4, wherein, The neural network processing unit is also configured with a calling interface corresponding to the derived operator; The neural network processing unit is specifically used to send the operator input data of the preset derivative operators in the neural network model to the signal processing unit through the calling interface during the process of processing the first feature data using the neural network model. Specifically, the signal processing unit is used to return the operator operation result to the neural network processing unit through the calling interface.

6. A data processing method applied to a signal processing unit, wherein the signal processing unit is connected to a neural network processing unit, and the neural network processing unit is configured with a neural network model. The data processing method includes: The system receives operator input data sent by the neural network processing unit during the processing of data to be processed using the neural network model; wherein the operator input data corresponds to a preset derived operator in the neural network model. The derived operator operation strategy is executed based on the operator input data to obtain the operator operation result; The operator operation result is sent to the neural network processing unit; wherein, the neural network processing unit is used to perform processing operations on the data to be processed based on the operator operation result, and obtain the corresponding neural network model output data.

7. The data processing method as described in claim 6, wherein, The data processing method further includes: The system receives the neural network model output data sent by the neural network processing unit and outputs the corresponding target data based on the neural network model output data.

8. The data processing method as described in claim 6, wherein, The data processing method further includes: The initial data is subjected to feature extraction to obtain the first feature data, and the first feature data is sent to the neural network processing unit; The system receives second feature data sent by the neural network processing unit, wherein the second feature data is obtained by the neural network processing unit running the neural network model with the first feature data as the data to be processed, and receives the operator operation result obtained by the signal processing unit executing the derived operator operation strategy according to the operator input data, and obtains the result based on the operator operation result.

9. The data processing method as described in claim 8, wherein, The data processing method further includes: outputting target data corresponding to the initial data based on the second feature data.

10. A data processing method applied to a neural network processing unit, the neural network processing unit being connected to a signal processing unit, the neural network processing unit being configured with a neural network model, the data processing method comprising: During the processing of the data to be processed using the neural network model, the operator input data of the preset derived operators in the neural network model is sent to the signal processing unit; wherein, the signal processing unit is configured with a derived operator operation strategy, and the signal processing unit is used to execute the derived operator operation strategy according to the operator input data to obtain the operator operation result, and send the operator operation result to the neural network processing unit; Receive the result of the operator operation; Based on the results of the operator operations, the processing operation on the data to be processed is completed, and the corresponding neural network model output data is obtained.

11. The data processing method as described in claim 10, wherein, The data processing method further includes: The signal processing unit receives first feature data, wherein the first feature data is obtained by the signal processing unit from initial data by feature extraction. The neural network model is run using the first feature data as the data to be processed, and the operator operation result obtained by the signal processing unit according to the operator input data and the derived operator operation strategy is received, and the second feature data is obtained according to the operator operation result; The second feature data is sent to the signal processing unit as the output data of the neural network model.

12. The data processing method as described in claim 11, wherein, The neural network processing unit is also configured with a calling interface corresponding to the derived operator, and the data processing method further includes: During the process of processing the first feature data using the neural network model, the operator input data of the preset derivative operators in the neural network model are sent to the signal processing unit through the calling interface.

13. A chip for implementing the steps of the data processing method as described in any one of claims 6-12.

14. An electronic device comprising the data processing apparatus according to any one of claims 1 to 5, or comprising the chip according to claim 13.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data processing method as described in any one of claims 6-12.

Images