Model inference method and apparatus, and electronic device

Abstract

Embodiments of the present application provide a model inference method and apparatus, and an electronic device, the method being applicable to a first device, a first model being deployed on the first device, the first model comprising N neural network layers, and the method comprising: determining a type of a first inference task; on the basis of the type of the first inference task, determining, among the N neural network layers, M neural network layers associated with the type of the first inference task, where N and M are each a positive integer greater than or equal to 1, and N is greater than M; and using the M neural network layers to perform inference on first input information, so as to output first output information corresponding to the first input. In the method, the apparatus, or the electronic device, inference is performed on the input information by using only the neural network layers required to be used during inference, and the neural network layers not required to be used are skipped, thus reducing the amount of computation during inference, lowering the power consumption of the device, increasing inference speed, and also saving the operating memory of the device.

Description

Methods, apparatus and electronic devices for model reasoning

[0001] This application claims priority to Chinese Patent Application No. 202510021929.0, filed on January 6, 2025, entitled “Method, Apparatus and Electronic Device for Model Reasoning”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of model reasoning, and more specifically, to a method, apparatus, and electronic device for model reasoning. Background Technology

[0003] With the development of device intelligence, the implementation of many device functions (such as voice question answering, copywriting design, and text summarization generation) relies on large models. Taking the large language model (LLM) as an example, the application of LLM enables users to interact with devices using natural language, which greatly reduces the threshold for users to use these functions.

[0004] However, large models require enormous computing power, and terminal devices will face problems such as high power consumption and slow running speed if they want to run large models. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for model inference. Through this method, apparatus, and electronic device, only the necessary neural network layers are used to infer the input information during the inference process, and unnecessary neural network layers can be skipped. This reduces the amount of computation in the inference process, lowers the power consumption of the device, improves the inference speed, and also saves the device's running memory.

[0006] In a first aspect, a method for model inference is provided, the method being applied to a first device, the first device having a first model deployed thereon, the first model comprising N neural network layers, the method comprising: determining a type of a first inference task; determining, based on the type of the first inference task, M neural network layers from the N neural network layers that are associated with the type of the first inference task, wherein N and M are both positive integers greater than or equal to 1, and N is greater than M; and using the M neural network layers to infer first input information to output first output information corresponding to the first input.

[0007] In this embodiment, the neural network layer associated with the task type can be selected from multiple neural network layers of a large model for inference based on the task type. Neural network layers that are not highly associated with the task type can be automatically skipped. In this way, the computational load of the inference process can be reduced without affecting the inference result, the power consumption of electronic devices can be reduced, and the stuttering of electronic devices during the inference process can be avoided, thereby improving the user experience.

[0008] Furthermore, since it is possible to determine which neural network layers need to be used for inference before inference, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the memory occupied by the inference process. This can further prevent electronic devices from lagging during inference and improve the user experience.

[0009] In conjunction with the first aspect, in one possible implementation, determining the type of the first inference task includes: determining the type of the first inference task based on a user's selection operation on the interface of the first device.

[0010] In this embodiment of the application, before the task inference, the user can select the type of inference task through the user interface. In this way, before inference, it can be determined which neural network layers need to be used for inference. Therefore, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the running memory occupied by the inference process. This can further avoid the electronic device from lag during the inference process and improve the user experience.

[0011] In conjunction with the first aspect, in one possible implementation, determining the type of the first inference task includes: determining the type of the first inference task based on the first input information.

[0012] In some examples, the first input information may be, for example, a voice command, text input, or information input in other ways.

[0013] In this embodiment of the application, before task inference, the device can directly determine the type of task to be inferred based on the user's input information. In this way, before inference, it can be determined which neural network layers need to be used for inference. Therefore, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the running memory occupied by the inference process. This can further avoid the electronic device from lag during the inference process and improve the user experience.

[0014] In conjunction with the first aspect, in one possible implementation, the type of the first reasoning task includes one or more of the following: evaluation task, greeting task, script task, free writing task, daily posting task, speech draft task, meeting minutes task, self-introduction task, praise task, polishing and rewriting task, outline writing task, and group notification task.

[0015] In conjunction with the first aspect, in one possible implementation, determining M neural network layers associated with the type of the first inference task from the N neural network layers according to the type of the first inference task includes: determining the M neural network layers from the N neural network layers according to the type of the first inference task and a first correspondence, wherein the first correspondence is a correspondence between the type of the inference task and the neural network layers, and the type of the inference task includes the type of the first inference task.

[0016] In some embodiments, the correspondence between the types of inference tasks and the neural network layers can be, for example, a one-to-one correspondence between multiple inference task types and multiple lists of neural network layers.

[0017] In this embodiment of the application, when using large model inference, different neural network layers are used for different inference task types. Before inference, the neural network layers to be used in the current inference process can be determined according to the correspondence between the inference task type and the neural network layers. In this way, only the necessary neural network layers can be used for the inference of the current task. This reduces the amount of computation in the inference process and the power consumption of electronic devices without affecting the inference results, thereby avoiding lag or other issues in the electronic devices during the inference process and improving the user experience.

[0018] In conjunction with the first aspect, in one possible implementation, reasoning about the first input information using the M neural network layers includes: reasoning about the first input information using each of the M neural network layers.

[0019] In some embodiments, the first input information may be inferred sequentially using each of the M neural network layers.

[0020] In conjunction with the first aspect, in one possible implementation, the first model includes an input module, an inference module, and an output module, wherein the inference module includes the N neural network layers.

[0021] In some embodiments, the input module is used to receive user input and input first input information to the inference module according to the user input; the output module is used to receive first output information output by the inference module and output inference results according to the first output information.

[0022] In conjunction with the first aspect, in one possible implementation, the first model is a large language model.

[0023] In a second aspect, a model reasoning apparatus is provided, the apparatus comprising a determining module and a first model, the first model comprising N neural network layers, wherein the determining module is configured to determine the type of a first reasoning task; the determining module is further configured to determine, based on the type of the first reasoning task, M neural network layers associated with the type of the first reasoning task from the N neural network layers, wherein N and M are both positive integers greater than or equal to 1, and N is greater than M; the first model is configured to use the M neural network layers to reason about first input information, so as to output first output information corresponding to the first input information.

[0024] In this embodiment, the neural network layer associated with the task type can be selected from multiple neural network layers of a large model for inference based on the task type. Neural network layers that are not highly associated with the task type can be automatically skipped. In this way, the computational load of the inference process can be reduced without affecting the inference result, the power consumption of electronic devices can be reduced, and the stuttering of electronic devices during the inference process can be avoided, thereby improving the user experience.

[0025] Furthermore, since it is possible to determine which neural network layers need to be used for inference before inference, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the memory occupied by the inference process. This can further prevent electronic devices from lagging during inference and improve the user experience.

[0026] In conjunction with the second aspect, in one possible implementation, the determining module is specifically used to: determine the type of the first inference task based on the user's selection operation on the user interface.

[0027] In this embodiment of the application, before the task inference, the user can select the type of inference task through the user interface. In this way, before inference, it can be determined which neural network layers need to be used for inference. Therefore, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the running memory occupied by the inference process. This can further avoid the electronic device from lag during the inference process and improve the user experience.

[0028] In conjunction with the second aspect, in one possible implementation, the determining module is specifically used to: determine the type of the first inference task based on the first input information.

[0029] In some examples, the first input information may be, for example, a voice command, text input, or information input in other ways.

[0030] In this embodiment of the application, before task inference, the device can directly determine the type of task to be inferred based on the user's input information. In this way, before inference, it can be determined which neural network layers need to be used for inference. Therefore, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the running memory occupied by the inference process. This can further avoid the electronic device from lag during the inference process and improve the user experience.

[0031] In conjunction with the second aspect, in one possible implementation, the type of the first reasoning task includes one or more of the following: evaluation task, greeting task, script task, free writing task, daily posting task, speech draft task, meeting minutes task, self-introduction task, praise task, polishing and rewriting task, outline writing task, and group notification task.

[0032] In conjunction with the second aspect, in one possible implementation, the determining module is further specifically used to: determine the M neural network layers from the N neural network layers according to the type of the inference task and the first correspondence, wherein the first correspondence is the correspondence between the type of the inference task and the neural network layers, and the type of the inference task includes the type of the first inference task.

[0033] In some embodiments, the correspondence between the types of inference tasks and the neural network layers can be, for example, a one-to-one correspondence between multiple inference task types and multiple lists of neural network layers.

[0034] In this embodiment of the application, when using large model inference, different neural network layers are used for different inference task types. Before inference, the neural network layers to be used in the current inference process can be determined according to the correspondence between the inference task type and the neural network layers. In this way, only the necessary neural network layers can be used for the inference of the current task. This reduces the amount of computation in the inference process and the power consumption of electronic devices without affecting the inference results, thereby avoiding lag or other issues in the electronic devices during the inference process and improving the user experience.

[0035] In conjunction with the second aspect, in one possible implementation, the first model is specifically used to: infer the first input information using each of the M neural network layers.

[0036] In some embodiments, the first input information may be inferred sequentially using each of the M neural network layers.

[0037] In conjunction with the second aspect, in one possible implementation, the first model includes an input module, an inference module, and an output module, wherein the inference module includes the N neural network layers.

[0038] In some embodiments, the input module is used to receive user input and input first input information to the inference module according to the user input; the output module is used to receive first output information output by the inference module and output inference results according to the first output information.

[0039] In conjunction with the second aspect, in one possible implementation, the first model is a large language model.

[0040] Thirdly, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer program code, and the processor is used to execute the computer program code stored in the memory to implement the method in the first aspect or any possible implementation thereof.

[0041] Fourthly, a computer-readable storage medium is provided, which stores a computer program or instructions that, when executed, implement the method described in the first aspect or any possible implementation thereof.

[0042] Fifthly, a chip is provided, wherein instructions are stored therein, which, when executed on a device, cause the chip to perform the methods of the first aspect or any possible implementation thereof.

[0043] In a sixth aspect, a computer program product is provided, which stores a computer program or instructions that, when executed, implement the method in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0044] Figure 1 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application;

[0045] Figure 2 is a software structure block diagram of the electronic device provided in an embodiment of this application;

[0046] Figure 3 is a schematic diagram of the architecture of a large model provided in an embodiment of this application;

[0047] Figure 4 is a schematic architecture diagram of a large language model provided in an embodiment of this application;

[0048] Figure 5 is a schematic flowchart of a model reasoning method provided in an embodiment of this application;

[0049] Figure 6 is a schematic flowchart illustrating the use of a large language model to reason about two different task types according to an embodiment of this application;

[0050] Figure 7 is a schematic flowchart of an example of using a large language model to reason about an evaluation task according to an embodiment of this application;

[0051] Figure 8 is a schematic diagram of the interface operation corresponding to a method for reasoning using a large language model provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments.

[0053] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "plural" or "multiple" refers to two or more than two.

[0054] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0055] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0056] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "one embodiment," "some embodiments," "another embodiment," "other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0057] The methods provided in this application can be applied to electronic devices such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0058] For example, Figure 1 shows a schematic diagram of the structure of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0059] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0060] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0061] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0062] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0063] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0064] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0065] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0066] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0067] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0068] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0069] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0070] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0071] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0072] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0073] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0074] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0075] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0076] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0077] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0078] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0079] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0080] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0081] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0082] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0083] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0084] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0086] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0087] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0088] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0089] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0090] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0091] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0092] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0093] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an embedded SIM (eSIM) card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0094] It should be understood that the phone cards in the embodiments of this application include, but are not limited to, SIM cards, eSIM cards, universal subscriber identity modules (USIM), universal integrated circuit cards (UICC), etc.

[0095] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0096] Figure 2 is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0097] As shown in Figure 2, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0098] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0099] As shown in Figure 2, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0100] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0101] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0102] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0103] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0104] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0105] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0106] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0107] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0108] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0109] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0110] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0111] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0112] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0113] A 2D graphics engine is a graphics engine for 2D drawing.

[0114] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0115] It should be understood that the technical solutions in the embodiments of this application can be used in systems such as Android, iOS, and HarmonyOS.

[0116] The technical solutions of this application embodiment can be applied to electronic devices capable of running large models. For example, they can be applied to televisions, desktop computers, laptops, in-vehicle screens, and portable electronic devices such as mobile phones, foldable screens, tablets, cameras, camcorders, video recorders, watches, and wristbands. They can also be applied to other electronic devices capable of running large models, and to electronic devices in future networks or in future evolved public land mobile networks (PLMNs).

[0117] With the development of device intelligence, the implementation of many device functions (such as voice question answering, copywriting design, and text summarization generation) relies on large models. Taking the large language model (LLM) as an example, the application of LLM enables users to interact with devices using natural language, which greatly reduces the threshold for users to use these functions.

[0118] Among them, large models refer to neural network models with a very large number of parameters, typically exceeding 1 billion.

[0119] Since the BERT and GPT-1 models validated the effectiveness of masking pre-training, the parameter scale of language models has continued to increase. In particular, the introduction of GPT-3 has also confirmed the application capability of LLM in real-world scenarios, such as applying LLM to writing code, fixing code, generating text summaries, answering questions, and designing copy.

[0120] Among them, the BERT model is a deep learning model used for natural language processing. BERT stands for bidirectional encoder representations from transformers.

[0121] The GPT-1 model is the first generation of generative pre-trained models launched by the OpenAI Research Center in the United States. GPT-1 stands for generative pre-trained transformer 1.

[0122] GPT-3 is a natural language processing model developed by OpenAI. GPT-3 stands for generative pre-trained transformer 3. It has 175 billion parameters and uses a transformer architecture with 96 transformer layers.

[0123] Although the convenient interactive modes and wide range of use cases of large models have attracted the attention of developers, due to their huge number of parameters, running large models is still mainly done on cloud servers.

[0124] For example, Table 1 shows the number of parameters, training data, computational cost for training, and computational cost for outputting each word in the BERT, GPT-1, and GPT-3 models.

[0125] Table 1

[0126] FLOPs stands for floating-point operations per second, which refers to the number of floating-point operations per second, also known as peak speed per second, i.e., the number of floating-point operations performed per second.

[0127] One GFLOPS is equal to one billion (10^9) floating-point operations per second; one TFLOPS is equal to one trillion (10^12) floating-point operations per second.

[0128] As shown in Table 1, the computational cost required for each word output by the GPT-3 model reaches 350 TFLOPs, or 3.5 × 10^14 floating-point calculations. However, the theoretical maximum AI computing power of current mobile phone processors is only 15.8 × 10^12 floating-point calculations, far below the computational power required for the GPT-3 model's inference process.

[0129] Therefore, when faced with the huge computational demands of large models, terminal devices will face problems such as high power consumption and slow speed if they want to run large models, making it difficult to provide users with a good user experience.

[0130] In some embodiments, before deploying the large model on the terminal device, the large model can be sparsely pruned, that is, some modules of low importance in the large model can be removed, and only a portion of the modules can be retained to obtain a sparse model. The sparse model can then be deployed on the terminal device to reduce the number of parameters in the model inference process.

[0131] In this method, after the sparse model is deployed on the terminal device, the inference process and the number of inference parameters of the sparse model are fixed. That is to say, the same sparse model is used for different types of inference tasks, and the calculation method used by the sparse model is the same. Therefore, the computational load and power consumption of the sparse model in the inference process are still high, and the accuracy of the inference results is not high.

[0132] In some embodiments, a gating network is deployed in the large model, which can predetermine whether each neuron in the large model needs to be activated based on user input, and only calculate the neurons that need to be activated during the large model inference process.

[0133] In this method, a gating network needs to be trained before using the large model for inference, which increases the resource requirements for training the large model. Furthermore, the deployment of the gating network during inference also consumes additional computational resources. In addition, since the gating network determines which neurons need to be computed and which do not, it is impossible to know in advance which neurons do not need to be computed. Therefore, all the parameters of the entire large model must still be read into dynamic random-access memory (DRAM), which cannot reduce the running memory of the terminal device.

[0134] Therefore, the current large models, when deployed on terminal devices, have problems such as large computational load and large memory requirements, which can easily lead to problems such as high power consumption and slow operation speed on terminal devices, affecting the user experience.

[0135] In view of this, embodiments of this application provide a method, apparatus, and electronic device for model inference. Through this method, apparatus, and electronic device, neural network layers associated with the task type can be selected from multiple neural network layers of a large model for inference based on the task type. Neural network layers that are not highly associated with the task type can be automatically skipped. In this way, the computational load of the inference process can be reduced without affecting the inference result, the power consumption of the electronic device can be reduced, thereby avoiding lag or other issues during the inference process and improving the user experience.

[0136] Furthermore, since it is possible to determine which neural network layers need to be used for inference before inference, only the parameters of these neural network layers need to be read, which can reduce the number of parameters and thus reduce the memory occupied by the inference process. This can further prevent electronic devices from lagging during inference and improve the user experience.

[0137] For example, Figure 3 shows a schematic diagram of the architecture of a large model provided in an embodiment of this application.

[0138] As shown in Figure 3, the large model includes an input module, a model backbone, and an output module. The model backbone comprises multiple neural network layers, specifically:

[0139] The input module is used to obtain user input and pass it to the model backbone.

[0140] The model backbone is used to infer user input through multiple neural network layers and to pass the inference results to the output module.

[0141] Multiple neural network layers can exist in a stacked manner.

[0142] The output module is used to receive the inference results of the model backbone on the user input and output the inference results to the user.

[0143] For example, taking a large language model as an example, Figure 4 shows a schematic architecture diagram of a large language model provided in an embodiment of this application.

[0144] As shown in Figure 4, a large language model can be based on a transformer architecture, including an input module, a model backbone, and an output module. The input module may include a tokenizer, the model backbone may include multiple transformer modules, and the output module includes a token classifier and a token decoder. Specifically:

[0145] The tokenizer is used to convert input information into input tokens and output a feature vector representing the input tokens.

[0146] One method is to represent tokens as feature vectors using token embedding.

[0147] Among them, the lexical unit is the smallest unit of information processed by the large language model.

[0148] In one example, the input information is "I want to listen to lyrical songs". The corresponding tokenizer converts "I" into a word element represented by feature vector "X1", "want" into a word element represented by feature vector "X2", "listen" into a word element represented by feature vector "X3", "lyrical" into a word element represented by feature vector "X4", "emotion" into a word element represented by feature vector "X5", and "song" into a word element represented by feature vector "X6".

[0149] The marker is also used to transmit feature vector 1 to the model backbone.

[0150] The model backbone contains multiple transformer modules used to obtain feature vector 1 and perform inference on it.

[0151] For example, taking transformer module 1 out of multiple transformer modules as an example, after obtaining feature vector 1, transformer module 1 calculates it and inputs the calculation result into transformer module 2, which then continues to perform calculations, and so on.

[0152] It should be noted that in the solution provided in this application embodiment, for a given input information, not all transformer modules participate in the calculation. Based on the type of input information, one or more transformer modules corresponding to the type of input information can be selected from multiple transformer modules to participate in the calculation, and other transformer modules can be skipped.

[0153] In some embodiments, multiple transformer modules in the model backbone exist in a stacked form, and the stacked multiple transformer modules can also be referred to as multi-layer transformers.

[0154] Each transformer module is essentially a combination of a multi-head self-attention layer and several fully connected feedforward neural networks. Multi-layer transformers allow the model to capture different levels of linguistic information.

[0155] After multiple transformer modules in the model backbone have completed inference on feature vector 1, they output feature vector 2, which is used to represent the output words.

[0156] In one example, the input information is "I want to listen to lyrical songs". Correspondingly, the tokenizer converts "I" into a token represented by the feature vector "X1", converts "want" into a token represented by the feature vector "X2", converts "to listen to" into a token represented by the feature vector "X3", converts "lyrical" into a token represented by the feature vector "X4", converts "songs" into a token represented by the feature vector "X6"; the input module inputs the feature vectors "X1", "X2", "X3", "X4", "X5" and "X6" into the model backbone. After multiple transformer modules in the model backbone sequentially perform inferences on the feature vectors "X1", "X2", "X3", "X4", "X5" and "X6" and complete the inferences, the feature vectors "Y1", "Y2", "Y3", "Y4", "Y5" and "Y6" are output; the token decoders in the output module respectively decode the feature vectors "Y1", "Y2", "Y3", "Y4", "Y5" and "Y6" into output tokens, and then output the inference result "Now play song 1", where song 1 is a lyrical song.

[0157] A token classifier for calculating token probabilities based on feature vector 2.

[0158] In one example, the token classifier calculates that the tokens that may be the first output may include "now", "immediately", "good", where the probability of "now" being the first output token is 57%, the probability of "immediately" being the first output token is 21%, and the probability of "good" being the first output token is 17%.

[0159] A token decoder for converting feature vector 2 into an output token.

[0160] In some embodiments, the token decoder may consider the token probabilities calculated by the token classifier during the process of converting feature vector 2 into an output token.

[0161] In some implementation manners, the token decoder may determine the output token according to the token probabilities calculated by the token classifier and the context.

[0162] The output module is further configured to output an inference result corresponding to the input information.

[0163] In one example, the input information is "I want to listen to lyrical songs", and the corresponding inference result may be, for example, "Now play song 1", where song 1 belongs to lyrical songs.

[0164] For example, Figure 5 shows a schematic flowchart of a model inference method 500 provided in an embodiment of this application. The large model used includes an input module, N transformer modules, and an output module. As shown in Figure 5, the method 500 includes:

[0165] S501: Determine the type of reasoning task.

[0166] In some embodiments, the type of inference task can be determined based on the user's active input. For example, the type of inference task can be determined based on the user's type selection operation on the terminal device.

[0167] In some embodiments, the type of reasoning task can be determined based on user input information, for example, based on information input by the user through voice input or text input.

[0168] In one implementation, for example, the type of reasoning task can be determined by a task classification model, which takes user input information as input and outputs the type of reasoning task.

[0169] In some embodiments, the type of reasoning task may include any one of the following: evaluation, greeting, script, free writing, daily posting, speech, meeting minutes, self-introduction, praise, polishing and rewriting, writing an outline, and group notification. In addition, the type of reasoning task may also include other task types, which are not limited in this application.

[0170] Among them, the N transformer modules can be understood as N neural network layers.

[0171] S502: Based on the type of reasoning task, determine M transformer modules from N transformer modules that are associated with the type of reasoning task, where M and N are both positive integers greater than or equal to 1, and N is greater than M.

[0172] In some embodiments, the transformer modules used may differ for different types of inference tasks when using a large model for inference.

[0173] In some embodiments, a type of inference task corresponds to a list of transformer modules. Before using large model inference, the terminal device has multiple lists of transformer modules that correspond one-to-one with multiple inference task types. Taking inference task type 1 as an example, inference task type 1 corresponds to transformer module list 1. Transformer module list 1 is used to indicate which transformer modules are needed for the inference task belonging to inference task type 1 during the inference process.

[0174] More specifically, it can be based on the type of the current inference task and the one-to-one correspondence between multiple inference task types and multiple lists of transformer modules, to determine M transformer modules associated with the type of inference task from N transformer modules.

[0175] S503: By using M converter modules to perform reasoning on the input information, the output reasoning result corresponding to the input information is generated.

[0176] In some embodiments, each of the M converter modules is used sequentially to perform inference calculations on the input information, outputs the calculation results to the input module, and the input module outputs the inference results corresponding to the input information based on the calculation results.

[0177] In some embodiments, it can be understood that the method provided in this application can be used to prune N transformer modules in a large model, and to prune NM transformer modules that are not related to the type of inference task. Only the parameters of M transformer modules need to be read to obtain a model corresponding to the type of inference task. The model corresponding to the type of inference task is used to infer the input information and output the inference result corresponding to the input information.

[0178] In this embodiment, considering that different inference tasks may require different transformer modules during inference, the transformer modules required for the inference process are determined according to the type of inference task before using a large model for inference. In this way, only the required transformer modules can be used to calculate the input information during the inference process, and the unnecessary transformer modules can be skipped. That is to say, these unnecessary transformer modules do not need to calculate the input information, and the parameters of these unnecessary transformer modules do not need to be read during the inference process. This can reduce the amount of computation in the inference process, reduce the power consumption of the terminal device, improve the inference speed, and save the running memory of the terminal device.

[0179] In one example, taking a large model with 28 stacked transformer modules as an example, when writing copy, four transformer modules can be skipped. Through comparative analysis of the output information, it can be seen that skipping these four transformer modules has no significant impact on the quality of the output information, and can reduce the amount of computation by about 14%, which can significantly improve the model inference speed.

[0180] For example, Figure 6 shows a schematic flowchart of reasoning for two different task types using a large language model provided in an embodiment of this application. The large language model includes an input module, a model backbone, and an output module, wherein 32 transformer modules are stacked in the model backbone.

[0181] Figure 6(a) shows a schematic flowchart of a method for reasoning mathematical problems using a large language model, as provided in an embodiment of this application.

[0182] As shown in Figure 6(a), the process of reasoning about mathematical problems using a large language model may include the following steps:

[0183] (1) When a user asks a math question 1 through a terminal device, the input module obtains the math question 1 and inputs the obtained math question 1 into the model backbone.

[0184] In one implementation, a user can ask a large language model a math problem by typing "math problem 1" into the interface of a terminal device.

[0185] In one implementation, users can ask mathematical questions by issuing voice commands to the terminal device.

[0186] In some embodiments, the input module converts the mathematical problem 1 into lexical units represented as feature vectors and then inputs them into the model backbone.

[0187] (2) Transformer modules 1, 2, 3, ..., 23, 24, 28, 29, 30, 31 and 32 in the main body of the model calculate the mathematical problem 1 in sequence, and output the calculation results to the output module after the calculation is completed.

[0188] It should be noted that the reasoning process for mathematical problem 1 skips transformer modules 25, 26, and 27. In other words, during the reasoning process for mathematical problem 1, it is not necessary to read the parameters of transformer modules 25, 26, and 27. That is, transformer modules 25, 26, and 27 do not participate in the reasoning calculation of mathematical problem 1.

[0189] (3) The output module outputs the answer to mathematical problem 1 based on the calculation results input from the main body of the model.

[0190] As shown in Figure 6(b), the process of reasoning about legal issues using a large language model may include the following steps:

[0191] (1) When a user asks a legal question 1 through a terminal device, the input module obtains the legal question 1 and inputs the obtained legal question 1 into the model backbone.

[0192] In one implementation, a user can ask a legal question ("Legal Question 1") to the large language model by typing it into the interface of a terminal device.

[0193] In one implementation, users can ask legal questions by issuing voice commands to the terminal device.

[0194] In some embodiments, the input module converts the legal issue 1 into lexical units represented as feature vectors and then inputs them into the model backbone.

[0195] (2) Transformer module 1, transformer module 2, transformer module 3, ..., transformer module 18, transformer module 20, transformer module 21, transformer module 23, transformer module 24, transformer module 26, ..., transformer module 32 in the model backbone calculate the legal issue 1 in sequence, and output the calculation results to the output module after the calculation is completed.

[0196] It should be noted that the reasoning process for legal question 1 skips transformer modules 19, 22, and 25. In other words, during the reasoning process for legal question 1, it is not necessary to read the parameters of transformer modules 19, 22, and 25. That is, transformer modules 19, 22, and 25 do not participate in the reasoning calculation for legal question 1.

[0197] (3) The output module outputs the answer to legal question 1 based on the calculation results of the model backbone input.

[0198] In this embodiment, the transformer modules required for different types of input information may vary during the inference process. During inference, only transformer modules related to the type of input information are used for calculation. This eliminates the need to read parameters from transformer modules unrelated to the type of input information, thereby reducing the computational load of the inference process without affecting the accuracy of the inference results. Furthermore, since only parameters from transformer modules related to the type of input information need to be read during the inference process, the operating memory of the terminal device can be saved.

[0199] For example, Figure 7 shows a schematic flowchart of reasoning for an evaluation task using a large language model provided in an embodiment of this application. The large language model includes an input module, a model backbone, and an output module, wherein 32 transformer modules are stacked in the model backbone.

[0200] As shown in Figure 7, taking the input information "Help me write a review of the watch I just bought" as an example, the process of reasoning about this input information using a large language model can include the following steps:

[0201] (1) When a user enters “Write a review for my newly purchased watch” through a terminal device, the input module obtains the input information and inputs the obtained input information into the model backbone.

[0202] In one implementation, a user can trigger a large language model to infer the meaning of "help me write a review of the watch I just bought" by typing "help me write a review of the watch I just bought" on the terminal device's interface.

[0203] For example, a user can trigger the AI ​​writing function on the terminal device, select the "evaluation" function on the corresponding interface of the AI ​​writing function, and then enter the information "help me write an evaluation of the watch I just bought" on the corresponding interface of the "evaluation" function.

[0204] In one implementation, users can trigger a large language model to infer the meaning of "help me write a review of the watch I just bought" by issuing voice commands to the terminal device.

[0205] In some embodiments, the input module converts the information "Help me write a review of the watch I just bought" into lexical units represented as feature vectors, and then inputs them into the model backbone.

[0206] (2) Transformer modules 1, 2, 3, ..., 18, 20, 21, 24, 26, 27, 28, and 29 in the main body of the model calculate the information "Help me write a review of the watch I just bought" in sequence, and output the calculation results to the output module after the calculation is completed.

[0207] It is worth noting that the reasoning process for the information "Help me write a review of the watch I just bought" skips transformer modules 19, 22, 23, 25, 30, 31, and 32. In other words, during the reasoning process for the information "Help me write a review of the watch I just bought," it is not necessary to read the parameters of transformer modules 19, 22, 23, 25, 30, 31, and 32. That is, transformer modules 19, 22, 23, 25, 30, 31, and 32 do not participate in the reasoning calculation for the information "Help me write a review of the watch I just bought."

[0208] In one implementation, the terminal device determines the inference task type as "review" based on the information "Help me write a review of the watch I just bought". Then, based on the correspondence between the inference task type and the transformer module list, it determines the transformer module list corresponding to the "review" type. This transformer module list corresponding to the "review" type includes transformer module 1, transformer module 2, transformer module 3, ..., transformer module 18, transformer module 20, transformer module 21, transformer module 24, transformer module 26, transformer module 27, transformer module 28, and transformer module 29. Further, the large language module is pruned to obtain a model whose main body only includes 25 transformer modules: transformer module 1, transformer module 2, transformer module 3, ..., transformer module 18, transformer module 20, transformer module 21, transformer module 24, transformer module 26, transformer module 27, transformer module 28, and transformer module 29. This model is used to infer the information "Help me write a review of the watch I just bought".

[0209] The 32 converter modules (converter modules 1 to 32) have approximately 7.78 billion parameters, while the 25 converter modules (converter modules 1, 2, 3, ..., 18, 20, 21, 24, 26, 27, 28, and 29) have approximately 6.08 billion parameters.

[0210] (3) The output module outputs the evaluation information corresponding to the information "Help me write a review of the watch I just bought" based on the calculation results input from the model backbone.

[0211] In one example, the review could be something like, "I just bought a watch, and the experience exceeded my expectations. Its accuracy is impressive; it tells the time precisely no matter where I am. The strap is comfortable and durable, and I don't feel uncomfortable even after wearing it for a long time. The design is simple and stylish, suitable for various occasions. I especially love its waterproof feature; I don't have to worry about it when washing my hands or showering. Overall, I'm very satisfied with this purchase!"

[0212] For example, Figure 8 shows a schematic diagram of the interface operation corresponding to a method for reasoning using a large language model provided in an embodiment of this application.

[0213] As shown in Figure 8(a), taking the input information "Write a review of my newly purchased watch" as an example, when a user wants to use the AI ​​function of electronic device 800 to review the newly purchased watch, the user can select the "Review" function in the display interface shown in Figure 8(a); the display interface of electronic device 800 will then switch to the interface shown in Figure 8(b). In this interface, the user can, for example, click icon 801 to adjust the review mode to AI review mode. The user can enter the information "Write a review of my newly purchased watch" in the input box, and then click control 802 to trigger the large language model deployed on electronic device 800 to reason about the information "Write a review of my newly purchased watch". After the reasoning is completed, electronic device 800 can, for example, display the information in the current context. The front display shows pop-up window 803, which includes the review information: "I just bought this watch, and the user experience exceeded my expectations. Its accuracy is impressive; it tells the time accurately no matter where I am. The strap is comfortable and durable, and I don't feel uncomfortable even after wearing it for a long time. The design is simple and elegant, suitable for various occasions. I especially like its waterproof function; I don't have to worry about it when washing my hands or taking a shower. Overall, I'm very satisfied with this purchase!" Pop-up window 803 may also include control 804, which allows the user to edit the displayed review information. Pop-up window 803 may also include control 805, which inserts the review information displayed in pop-up window 803 as content to evaluate the watch.

[0214] In some embodiments, for example, the user can also select the “speech” function in the display interface shown in Figure 8(a) to trigger the terminal device 800 to trim the deployed large model into a model corresponding to the “speech” type, and use the trimmed model to infer the user’s input information.

[0215] In addition, users can, for example, select other types in the display interface shown in Figure 8(a) to trigger the terminal device 800 to trim the deployed large model into a model corresponding to the other type, and use the trimmed model to infer the user's input information.

[0216] The following table, Table 2, compares the relevant data of several different inference task types using the unpruned large language model and the pruned large language model, taking a large language model with 32 stacked neural network layers in the backbone as an example.

[0217] Table 2

[0218] As shown in Table 2, for the "text search" reasoning task, the neural network layers that do not need to be used in the reasoning process are layers 15, 26, 19, 16, 4, 20, 28, 25, 24, 23, and 30. These 11 neural network layers can be pruned to obtain the model corresponding to the "text search" reasoning task. The backbone of this model includes 21 neural network layers, namely layers 1 to 3, layers 5 to 14, 17, 18, 21, 22, 27, 29, 31, and 32. The performance index of the large language model with 32 stacked neural network layers for the "text search" reasoning task is 0.295, with a computational power requirement of 100%. The performance index of the large language model with 21 stacked neural network layers for the "text search" reasoning task is 0.300, with a computational power requirement of 65.6%. It can be seen that the model pruning has little impact on the reasoning results, and the computational power is reduced by 34.4%.

[0219] For the "everyday knowledge question answering" reasoning task, the neural network layers that do not need to be used in the reasoning process are layers 31, 28, 25, 16, 23, 26, 17, and 22. These 8 neural network layers can be pruned to obtain a model corresponding to the "everyday knowledge question answering" reasoning task. The backbone of this model includes 24 neural network layers, namely layers 1 to 15, 18 to 21, 24, 27, 29, 30, and 32. The performance index of the large language model with 32 stacked neural network layers for the "everyday knowledge question answering" reasoning task is 0.472, with a computing power requirement of 100%. The performance index of the large language model with 24 stacked neural network layers for the "everyday knowledge question answering" reasoning task is 0.479, with a computing power requirement of 75.0%. It can be seen that the model pruning has little impact on the reasoning results, and the computing power is reduced by 25.0%.

[0220] For the "document summarization" reasoning task, the neural network layers that are not needed in the reasoning process are layers 23, 25, and 24. These three neural network layers can be pruned to obtain a model corresponding to the "document summarization" reasoning task. The backbone of this model includes 29 neural network layers, namely layers 1 to 22 and layers 26 to 32. The performance index of the large language model with 32 stacked neural network layers for the "document summarization" reasoning task is 2.596, with a computational requirement of 100%. The performance index of the large language model with 29 stacked neural network layers for the "document summarization" reasoning task is 2.404, with a computational requirement of 90.6%. It can be seen that the model pruning has little impact on the reasoning results, and the computational cost is reduced by 9.4%.

[0221] For the "dialogue summarization" reasoning task, the neural network layers that are not needed in the reasoning process are layers 4, 31, 21, 30, and 25. These five neural network layers can be pruned to obtain a model corresponding to the "dialogue summarization" reasoning task. The backbone of this model includes 27 neural network layers, namely layers 1 to 3, 5 to 20, 22 to 24, 26 to 29, and 32. The performance index of the large language model with 32 stacked neural network layers for the "dialogue summarization" reasoning task is 2.278, with a computational requirement of 100%. The performance index of the large language model with 27 stacked neural network layers for the "text search" reasoning task is 2.269, with a computational requirement of 84.3%. It can be seen that the model pruning has little impact on the reasoning results, and the computational cost is reduced by 15.7%.

[0222] One or more modules or units described herein can be implemented in software, hardware, or a combination of both. When any of the above modules or units are implemented in software, the software exists as computer program instructions and is stored in memory. A processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores for executing software instructions to perform calculations or processing, the processor may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0223] When the modules or units described herein are implemented in hardware, the hardware may be any one or any combination of a CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, application-specific digital circuit, hardware accelerator, or non-integrated discrete device, which may run the necessary software or perform the above method flow independently of software.

[0224] When the modules or units described herein are implemented using software, they can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0225] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0226] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0230] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0231] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for model reasoning, characterized in that, The method is applied to a first device, on which a first model is deployed, the first model comprising N neural network layers, the method comprising: Determine the type of the first reasoning task; Based on the type of the first reasoning task, determine M neural network layers from the N neural network layers that are associated with the type of the first reasoning task, where N and M are both positive integers greater than or equal to 1, and N is greater than M; The M neural network layers are used to infer the first input information to output the first output information corresponding to the first input.

2. The method according to claim 1, characterized in that, Determining the type of the first reasoning task includes: The type of the first inference task is determined based on the user's selection operation on the interface of the first device.

3. The method according to claim 1, characterized in that, Determining the type of the first reasoning task includes: The type of the first reasoning task is determined based on the first input information.

4. The method according to any one of claims 1 to 3, characterized in that, The first reasoning task includes one or more of the following: evaluation task, greeting task, script task, free writing task, daily posting task, speech draft task, meeting minutes task, self-introduction task, praise task, polishing and rewriting task, outline writing task, and group notification task.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining M neural network layers associated with the type of the first inference task from the N neural network layers, based on the type of the first inference task, includes: Based on the type of the first reasoning task and the first correspondence, the M neural network layers are determined from the N neural network layers, wherein the first correspondence is the correspondence between the type of the reasoning task and the neural network layer, and the type of the reasoning task includes the type of the first reasoning task.

6. The method according to any one of claims 1 to 5, characterized in that, The inference process using the M neural network layers on the first input information includes: The first input information is inferred using each of the M neural network layers.

7. The method according to any one of claims 1 to 6, characterized in that, The first model includes an input module, an inference module, and an output module, wherein the inference module includes the N neural network layers.

8. The method according to any one of claims 1 to 7, characterized in that, The first model is a large language model.

9. A device for model reasoning, characterized in that, The device includes a determining module and a first model, wherein the first model includes N neural network layers, wherein... The determining module is used to determine the type of the first reasoning task; The determining module is further configured to determine, based on the type of the first inference task, M neural network layers associated with the type of the first inference task from the N neural network layers, wherein N and M are both positive integers greater than or equal to 1, and N is greater than M; The first model is used to reason about the first input information using the M neural network layers, so as to output the first output information corresponding to the first input information.

10. The apparatus according to claim 9, characterized in that, The determining module is specifically used for: The type of the first inference task is determined based on the user's selection on the user interface.

11. The apparatus according to claim 9, characterized in that, The determining module is specifically used for: The type of the first reasoning task is determined based on the first input information.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The first reasoning task includes one or more of the following: evaluation task, greeting task, script task, free writing task, daily posting task, speech draft task, meeting minutes task, self-introduction task, praise task, polishing and rewriting task, outline writing task, and group notification task.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The determining module is also specifically used for: Based on the type of the reasoning task and the first correspondence, the M neural network layers are determined from the N neural network layers, wherein the first correspondence is the correspondence between the type of the reasoning task and the neural network layer, and the type of the reasoning task includes the type of the first reasoning task.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The first model is specifically used for: The first input information is inferred using each of the M neural network layers.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The first model includes an input module, an inference module, and an output module, wherein the inference module includes the N neural network layers.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The first model is a large language model.

17. An electronic device, characterized in that, include: One or more processors; One or more memory units; And one or more computer programs, wherein the one or more computer programs are stored in the one or more memories, the one or more computer programs including instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, The storage medium stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 8.

19. A chip, characterized in that, The chip stores instructions that, when executed, implement the method as described in any one of claims 1 to 8.

20. A computer program product, characterized in that, The computer program product stores a program or instructions that, when executed, implement the method as described in any one of claims 1 to 8.

Images