Electronic device including electronic component for processing multi-modality and operation method and storage medium therefor

Abstract

An embodiment of the present disclosure may provide an electronic device. The electronic device may comprise: a printed circuit board; a processor disposed on the printed circuit board and having multiple processing units embedded therein, the processing units including a neural processing unit; a first memory stacked on the processor in a package-on-package (PoP) manner; a second memory which is disposed on the printed circuit board, has a side surface positioned alongside one side surface of the processor, and includes multiple sub-memories, the second memory being a high bandwidth memory (HBM) stacked via a through-silicon via (TSV) on the printed circuit board on which the processor is disposed; and a memory management accelerator for optimizing bank allocation to the first memory and the second memory. Various other embodiments are possible.

Description

Electronic device including electronic components for processing multimodality, method of operation thereof, and storage medium

[0001] The present disclosure relates to an electronic device comprising an electronic component for processing multimodality, a method of operation thereof, and a storage medium.

[0002] Recently, the performance of artificial intelligence (AI) models has been advancing. For example, an AI model can be trained to provide image analysis results. For instance, an AI model can provide an image reading function as an analysis result. The image reading function may be a function that outputs speech for text (or Braille) contained in the image. For instance, an AI model can provide a translation function as an analysis result. The translation function may be a function that provides a result (or corresponding speech) of translating text (or Braille) contained in the image into another language. For instance, an AI model can provide a summary function as an analysis result. The summary function may be a function that provides a summary of the text (or Braille) contained in the image. There are no restrictions on the type of image analysis.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0004] According to one embodiment of the present disclosure, an electronic device may be provided. The electronic device may include a printed circuit board, a processor having a plurality of processing units including a neural processing unit disposed on the printed circuit board; a first memory stacked on the processor in a package-on-package (PoP) manner; a second memory which is a high bandwidth memory (HBM) stacked on the printed circuit board on which the processor is disposed, having a plurality of sub-memories disposed side-by-side with one side of the processor on the printed circuit board, and which includes a through-silicon via (TSV) manner; and a memory management accelerator for optimizing bank allocation to the first memory and the second memory. The neural processing unit may be connected to the neural processing unit of the processor through a first interface, disposed adjacent to the second memory within the processor, and connected to the neural processing unit of the processor through a second interface.

[0005] According to one embodiment of the present disclosure, a method of operation of an electronic device may be provided. The method of operation of an electronic device may include at least one processor including a memory management accelerator; and a memory for storing instructions, comprising a first memory and a second memory having a relatively higher bandwidth than the first memory. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to perform at least one operation. The at least one operation may include an operation of determining whether the input artificial intelligence model is a plurality or corresponds to a multimodal large-scale language model. The at least one operation may include, when the artificial intelligence model is a plurality or corresponds to a multimodal large-scale language model (multi-modality LLM), an operation of performing bank allocation optimization to the first memory and the second memory by the memory management accelerator based on at least one of the following: bank state information of each of the first memory and the second memory, layer information of the artificial intelligence model, computational characteristics of the artificial intelligence model, memory access patterns occurring during the execution of the artificial intelligence model, and data access information of the at least one processor. The above at least one operation may include: an operation to execute an artificial intelligence model allocated to a first memory and a second memory.

[0006] The aspects, configurations, and / or advantages described above regarding one embodiment of the present disclosure may become more apparent from the following detailed description with reference to the accompanying drawings.

[0007] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0008] FIG. 2 is a block diagram of an electronic device including a processor and memory according to one embodiment.

[0009] FIG. 3 is a diagram illustrating the arrangement relationship between an application processor and memory according to one embodiment.

[0010] FIG. 4 is a cross-sectional view showing the arrangement relationship between an application processor and a memory according to one embodiment.

[0011] FIG. 5 is a cross-sectional view showing the arrangement relationship between an application processor and a memory according to one embodiment.

[0012] FIG. 6 is a cross-sectional view showing the arrangement relationship between an application processor and memory according to one embodiment.

[0013] FIG. 7 is a cross-sectional view showing the arrangement relationship between an application processor and memory according to one embodiment.

[0014] FIG. 8 is a block diagram illustrating a signal processing method between an application processor and a first DRAM and between an application processor and a second DRAM, according to one embodiment.

[0015] FIG. 9 is a block diagram showing the step-by-step operation of a memory management accelerator according to one embodiment.

[0016] FIG. 10 is a block diagram showing a processing method in which a plurality of artificial intelligence models are distributed and processed across a plurality of DRAMs according to one embodiment.

[0017] FIG. 11a is a flowchart illustrating a processing method when a plurality of artificial intelligence models are distributed across a plurality of DRAMs for processing, according to one embodiment.

[0018] FIG. 11b is a flowchart illustrating a processing method when a plurality of artificial intelligence models are distributed across a plurality of DRAMs for processing, according to one embodiment.

[0019] FIG. 12 is a block diagram illustrating a processing method when a multi-modal large-scale language model (multi-modal LLM) is distributed across a plurality of DRAMs for processing according to one embodiment.

[0020] FIG. 13a is a flowchart showing a processing method when a multi-modal large-scale language model (multi-modal LLM) is distributed across multiple DRAMs for processing.

[0021] FIG. 13b is a flowchart showing a processing method when a multi-modal large-scale language model (multi-modal LLM) is distributed across multiple DRAMs for processing.

[0022] FIG. 14 is a block diagram illustrating a hidden layer distributed processing technique for various types of large-scale language models using a memory management accelerator (MMA) according to one embodiment.

[0023] FIG. 15 is a flowchart illustrating a hidden layer distributed processing technique for various types of large-scale language models using a memory management accelerator (MMA) according to one embodiment.

[0024] FIG. 16 is a diagram illustrating a multitasking scenario in which two AI models are executed simultaneously, according to one embodiment.

[0025] FIG. 17a is a diagram showing the arrangement relationship between a processor and a memory according to one embodiment.

[0026] FIG. 17b is a diagram showing the arrangement relationship between a processor and memory according to one embodiment.

[0027] Throughout the attached drawings, similar parts, configurations, and / or structures may be assigned similar reference numbers.

[0028] Neural processing units (NPUs) and memory (e.g., dynamic random access memory (DRAM)) can be used for the efficient operation of artificial intelligence models on small electronic devices such as mobile phones. For instance, memory (e.g., DRAM) can be stacked on top of an application processor (AP) that includes a neural processing unit (NPU).

[0029] However, such a structure may present the following problems if the memory (e.g., DRAM) is not specialized for the neural network processing unit (NPU). First, when the floor area of ​​the application processor (AP) die is expanded to increase capacity, the length of the data signal transmission path between the NPU and the memory (e.g., DRAM) increases, which can lead to signal skew. Signal skew refers to the phenomenon where different signals of the same phase arrive at their destination at different times, which can cause data errors and / or system instability. In particular, in systems implementing the operation of large language models (LLMs) that require high-speed signal processing, time latency or time delay caused by such signal skew can lead to severe performance degradation. Furthermore, stacking memory on top of the application processor (AP) to avoid increasing the length of the data signal transmission path may be disadvantageous to the trend of miniaturization of electronic devices due to thickness limitations. Increasing the size of the application processor (AP) die can increase the probability of defects occurring during the manufacturing process, which may lead to a decrease in production yield.

[0030] Large-scale Language Models (LLMs) are evolving into multi-modal models that learn and process the relationships between various forms of data, such as text, images, videos, and audio, by considering them together. These multi-modal LLMs are very large in size and may require significantly more memory capacity to process diverse data types. Existing DRAM technology has limitations in capacity expansion and can lead to memory shortages when running multi-modal LLMs. Insufficient memory can result in system performance degradation, such as reduced speed and accuracy, when processing multi-modal LLMs. Since LLMs must process large amounts of data at high speeds, they may require high memory bandwidth. However, existing DRAM technology has limitations in bandwidth expansion, and memory access time latency issues may occur, particularly when large-scale data processing is required, such as in multi-modal LLMs.

[0031] The present disclosure can provide various embodiments that effectively connect a neural network processing unit (NPU) and a plurality of memories (e.g., DRAM), thereby reducing and / or preventing time latency.

[0032] The present disclosure provides a memory system optimized for running a multi-modal large-scale language model (multi-modal LLM), thereby solving memory (e.g., DRAM) capacity shortage and time latency issues, while simultaneously providing various embodiments to secure product competitiveness by reducing manufacturing costs.

[0033] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0034] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0035] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication device (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0036] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication device (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0037] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication device (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication device (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0038] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0042] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0043] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0044] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0048] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0050] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication device (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication device (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication device (190) may include a wireless communication device (192) (e.g., cellular communication device, short-range wireless communication device, or GNSS (global navigation satellite system) communication device) or a wired communication device (194) (e.g., LAN (local area network) communication device, or power line communication device). Among these communication devices, the corresponding communication device can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication devices may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication device (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication device (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. The NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication device (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication device (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication device (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication device (192) may support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0053] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication device (190). A signal or power may be transmitted or received between the communication device (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0054] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0055] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0056] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] FIG. 2 is a block diagram of an electronic device including a processor (220) and a memory (230) according to one embodiment.

[0058] The electronic device (200) of the present disclosure (e.g., the electronic device (101) of FIG. 1) may include a hardware structure for processing an artificial intelligence model within the device. If the electronic device (200) includes a hardware structure for processing an artificial intelligence model internally, it may be referred to as an "electronic device capable of on-device artificial intelligence processing." The electronic device (200) of the present disclosure can provide artificial intelligence-processed information to a user at a faster speed compared to processing the artificial intelligence model at an external server (e.g., the server (108) of FIG. 1) that communicates with the electronic device (200).

[0059] The electronic device (200) may include a processor (220) (e.g., the processor (120) of FIG. 1) and a memory (230) (e.g., the memory (130) of FIG. 1).

[0060] A processor (220) (e.g., an application processor) may include a plurality of processing units. The plurality of processing units may be entities for processing that include at least one circuit. The processor (220) may include a plurality of processing units, for example, a central processing unit (CPU) (2201) and a neural processing unit (NPU) (2202) as illustrated in FIG. 2. Additionally, the processor (220) may include a memory controller (2203). The plurality of processing units may operate independently of each other or together. The central processing unit (2201) may control other elements connected to the central processing unit (2201) (e.g., other processing units, other hardware or software components) and may perform data processing or operations. For example, if the processor (220) includes a neural processing unit (2202), the neural processing unit (2202) may be configured to be specialized for processing artificial intelligence models. For example, if the processor (220) includes a memory controller (2203), the memory controller (2203) may be configured to be specialized for accessing and / or controlling memory.

[0061] According to one embodiment, "the processor (220) includes a central processing unit (2201), a neural network processing unit (2202), and a memory controller (2203)" may mean that the central processing unit (2201), the neural network processing unit (2202), and the memory controller (2203) are integrated on a piece of semiconductor wafer (e.g., a die) to form a single integrated circuit chip (IC chip).

[0062] The types and number of multiple processing units are not limited to the embodiment illustrated in FIG. 2 and may include various other multiple processing units. For example, at least one of a graphic processing unit (GPU), a digital signal processor (DSP), a communication processor (CP), and an image signal processor (ISP) may be additionally included. Each processing unit may be configured to use lower power than the central processing unit (2201) included in the processor (e.g., application processor) or to be specialized for a designated function. In the embodiment below FIG. 2, each processing unit may be implemented as part of the processor (220) (e.g., application processor).

[0063] Memory (230) can store various data used by the processor (220) or other components of the electronic device (101) (e.g., the sensor module (176) of FIG. 1). The data may include, for example, input data or output data for software (e.g., the program (140) of FIG. 1) and related commands. Memory (230) may include multiple memories. For example, memory (230) may include volatile memory (232) and / or non-volatile memory (234). In the embodiment of FIG. 2 of the present disclosure, dynamic random access memory (DRAM) is disclosed as the volatile memory (232). For example, the volatile memory (232) of the present disclosure may include two DRAMs (2321, 2322). In the embodiment of FIG. 2, flash memory is disclosed as the non-volatile memory (234). However, it is not necessarily limited to this, and other types of non-volatile memory such as ROM (read-only memory), for example, EEPROM (electrically erasable programmable read-only memory), may also be applied.

[0064] The electronic device (200) of the present disclosure includes two DRAMs (2321, 2322) as volatile memory (232), and these two DRAMs may be of different types. According to one embodiment, the first DRAM (2321) may be a low-capacity memory, and the second DRAM (2322) may be a high-capacity memory. For example, the first DRAM (2321) may serve as a low-capacity, high-speed memory to temporarily store data required for computation of a neural network processing unit (2202). For example, the first DRAM (2321) may serve as a legacy central processing unit (2201) to temporarily store data required for computation. For example, the first DRAM may be a low-power memory (e.g., LPDDR (low-power double data rate)). For example, the second DRAM (DRAM2, 2322) can serve as a high-capacity, high-bandwidth memory (e.g., HBM (high bandwidth memory)) to store parameters of a large language module (LLM) and large amounts of data.

[0065] The present disclosure allows heterogeneous memory (first DRAM (2321), second DRAM (2322)) to be arranged in an optimal 3D structure relative to the processor (220) in order to minimize the physical distance between the processor (e.g., neural network processing unit (2202)) and the memory. Here, "some component is arranged in an optimal 3D structure relative to another component" may mean that all or part of a component is placed in an optimal position relative to all or part of another component in the direction of two axes (e.g., X-axis and Y-axis) of a 3-axis coordinate system, as well as in the direction of the remaining axis (e.g., Z-axis). Through this, signal processing performance can be improved by resolving the signal skew problem that occurs in conventional 2D planar arrangement methods and by reducing and / or preventing time latency. The arrangement structure between the processor (e.g., neural network processing unit (2202)) and the memory will be described in more detail below in the embodiment of FIG. 3.

[0066] The electronic device (200) can minimize the physical distance between components by arranging heterogeneous memory in an optimal 3D stacked structure relative to the processor for rapid signal processing between the processor (e.g., neural network processing unit (2202)) and memory, while maximizing the data transmission speed through a high-speed interface.

[0067] For example, the electronic device (200) may include a plurality of interfaces. In the embodiment of FIG. 2, the electronic device (200) may include an interface (e.g., interface (I1)) for transmitting touch input for some input means (e.g., display module (260)) to the processor (220). Additionally, the electronic device (200) may include an interface (e.g., first interface (I2), second interface (I3)) for transmitting commands or data from the processor (220) to memory (230). Additionally, the electronic device (200) may include an interface (e.g., interface (I4), interface (I5)) for transmitting commands or data stored in volatile memory (232) to non-volatile memory (234). According to one embodiment, other interfaces may be added in addition to the interfaces mentioned above. For example, it may further include an interface for directly transmitting commands or data from the processor (220) to non-volatile memory (234). For convenience, the aforementioned interface has been described as being for transmitting commands or data from one component to another, but conversely, it may also be possible to transmit commands or data from another component to one component.

[0068] The electronic device (200) of the present disclosure may include a first interface (I2) and a second interface (I3) as interfaces for transmitting commands or data between a processor (220) and a memory (230). For example, the first interface (I2) may be provided to transmit commands or data between the processor (220) and a first DRAM (2321), and the second interface (I3) may be provided to transmit commands or data between the processor (220) and a second DRAM (2322). According to one embodiment, the second interface (I3) may be an interface specialized for high-speed command or data transmission. For example, the second interface (I3) may be a Computing Express Link (CXL). According to one embodiment, not only the second interface (I3) but also the first interface (I2) may be configured as a high-speed interface specialized for high-speed command or data transmission.

[0069] The electronic device (200) of the present disclosure may include a memory management accelerator (MMA) (22021) inside a neural network processing unit (NPU) (2202) to enhance on-device artificial intelligence processing performance. The memory management accelerator (22021) may increase memory access efficiency by performing bank allocation of memory in consideration of the non-linear characteristics of data access patterns. Functions of the memory management accelerator (22021) may include a data access pattern analysis function, a bank allocation optimization function, a data prefetching function, a data locality utilization function, and a function to minimize data transfer between banks. The "data access pattern analysis function" may be to extract features by analyzing memory access patterns occurring during the execution of a large-scale language model (LLM) in real time. The "bank allocation optimization function" may be to increase memory access efficiency by dynamically adjusting the bank allocation of the first DRAM and the second DRAM based on the analyzed data access patterns. "Data prefetching function" may be a function that reduces time latency by bringing necessary data into memory (e.g., the first DRAM (2321)) in advance based on predicted data access patterns. "Data locality utilization function" may be a function that reduces memory access time by allocating frequently accessed data to the same bank or adjacent banks. "Inter-bank data transfer minimization function" may be a function that reduces power consumption and improves system performance by reducing unnecessary inter-bank data transfers. Depending on the embodiment, the memory management accelerator (22021) may be configured to omit some of the above functions, or it may be configured to perform additional functions other than those mentioned above.

[0070] The electronic device (200) of the present disclosure may provide distributed processing technology for efficiently executing a large-scale language model (LLM). Distributed processing technology may include model parallelism, pipeline parallelism, data parallelism, and hybrid parallelism. "Model parallelism" may involve executing a large-scale language model (LLM) by dividing it among multiple neural network processing units (NPUs). Here, the operations executed by dividing among the neural network processing units (NPUs) may be executed simultaneously. "Pipeline parallelism" may involve executing layers of a large-scale language model (LLM) in a pipeline manner by allocating them to multiple neural network processing units (NPUs). "Data parallelism" may involve loading copies of the same model into multiple neural network processing units (NPUs) and processing different data inputs. "Hybrid parallelism" may refer to a technology implemented to ensure optimal performance by combining at least two of model parallelism, pipeline parallelism, and data parallelism. The above distributed processing technology(s) may be executed collectively or individually by a processor (220) included in the electronic device (200), and instructions or data for such execution may be stored in memory (230).

[0071] In the description above, the memory management accelerator (MMA) is described as being included in the neural network processing unit (NPU), and for convenience, the memory management accelerator (MMA) may also be described as being included in the neural network processing unit (NPU) in the following embodiments. However, it is not necessarily limited thereto. According to one embodiment, the memory management accelerator (MMA) may be located (or included) in another processing unit within the processor (220) (e.g., application processor), or may be provided independently and distinct from the other processing unit.

[0072] In the description of FIG. 3 and below of the present disclosure, the arrangement relationship and operation method between the processor (220) and the memory (230) can be specifically described with a focus on the volatile memory (232). Therefore, it should be noted that in the following description, "memory" may refer to the volatile memory.

[0073] FIG. 3 is a diagram illustrating the arrangement relationship between a processor (320) and a memory (332) according to one embodiment.

[0074] The embodiment of FIG. 3 can be combined with the embodiments of FIG. 1 and FIG. 2 described above, or the embodiments of FIG. 4 to FIG. 18b described later.

[0075] In describing embodiments below FIG. 3, an orthogonal coordinate system (X, Y, Z) may be illustrated for convenience of explanation. For example, the X-axis direction of the orthogonal coordinate system may represent the length direction of the element to be described in the drawing, the Y-axis direction may represent the width direction, and the Z-axis direction may represent the height direction. In describing the arrangement relationship between the components of the present disclosure, the arrangement of one component above or below another component may refer to an arrangement relationship on the Z-axis. For example, the arrangement of one component below (e.g., the back) another component may be understood as the component being located in the -Z-axis direction relative to the other component.

[0076] Referring to FIG. 3, the electronic device (300) may include a processor (320) (e.g., processor (120) of FIG. 1, processor (220) of FIG. 2) and memory (e.g., volatile memory (332)) (e.g., memory (130) of FIG. 1, memory (230) of FIG. 2). The processor (320) (e.g., application processor) and memory (e.g., volatile memory (332)) may be placed on a single printed circuit board (310). The shape and area of ​​the printed circuit board (310) are not limited, but may be formed in a compact size to have only an area corresponding to the area where the processor (320) and memory (e.g., volatile memory (332)) are placed, in order to fit a small electronic device. The processor (320) and memory (e.g., volatile memory (332)) may be disposed on a surface of the substrate of the printed circuit board (310). According to one embodiment, the processor (320) and memory (e.g., volatile memory (332)) may be placed at substantially the same level (e.g., a surface of the printed circuit board (310)).

[0077] A processor (320) (e.g., an application processor) may include a plurality of processing units. As previously described in FIG. 2, the processor (320) may include a central processing unit (CPU) (3201) and a neural network processing unit (NPU) (3202). In the embodiment of FIG. 3, additionally, a graphics processing unit (GPU) (3203), a communication processor (CP) (3204), an image signal processor (ISP) (3205), and a digital signal processor (DSP) (3206) are shown. According to one embodiment, a central processing unit (CPU) (3201), a neural network processing unit (NPU) (3202), a graphics processing unit (GPU) (3203), a communication processor (CP) (3204), an image signal processor (ISP) (3205), and a digital signal processor (DSP) (3206) may be integrated on a single die of a processor (320) to form a single chip. Here, "die" may refer to a small, thin semiconductor wafer piece typically made of silicon, on which electronic circuits, active components, and passive components may be placed.

[0078] The present disclosure may disclose an embodiment in which a plurality of processing units, for example, a central processing unit (CPU) (3201), a neural network processing unit (NPU) (3202), a graphics processing unit (GPU) (3203), a communication processor (CP) (3204), an image signal processor (ISP) (3205), and a digital signal processor (DSP) (3206) are distributed across a plurality of regions of a die of a single processor (320) (e.g., an application processor). Each processing unit, including the central processing unit (CPU) (3201), the neural network processing unit (NPU) (3202), the graphics processing unit (GPU) (3203), the communication processor (CP) (3204), the image signal processor (ISP) (3205), and the digital signal processor (DSP) (3206), may be mounted on one side of the die of the processor (320).

[0079] According to one embodiment, a central processing unit (CPU) (3201) may be responsible for overall control and computation of an electronic device (and its system). A neural network processing unit (NPU) (3202) may be responsible for the execution and acceleration of an artificial intelligence model. A graphics processing unit (GPU) (3203) may be responsible for large-scale graphics computation. A communication processor (CP) (3204) may be responsible for communication functions with an external electronic device. An image signal processor (ISP) (3205) may be responsible for processing images obtained from other components included in the electronic device (e.g., a camera module). A digital signal processor (DSP) (3206) may be responsible for numerically processing digitized signals by an algorithm. The processor (320) of the present disclosure may additionally include other processing units in addition to the plurality of processing units described above, or some of the plurality of processing units described above may be omitted. According to one embodiment, some of the plurality of processing units described above may be integrated into other parts. For example, the communication processor (CP) (3204) may be integrated into the central processing unit (CPU) (3201). Various other embodiments may also be applied.

[0080] With reference to FIG. 3, the arrangement relationship between the processor (320) and the heterogeneous memory in the electronic device of the present disclosure (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2) is described in detail. Here, the heterogeneous memory describing the arrangement relationship with the processor (320) may correspond to a volatile memory (332). In the present disclosure, heterogeneous memory (first DRAM (3321), second DRAM (3322)) can be arranged with respect to the processor (320) in an optimal 3D stacked structure.

[0081] In the present disclosure, the term "3D stacked structure" may mean that one DRAM (e.g., first DRAM (3321)) is directly stacked on a processor (320) (or a neural network processing unit (NPU) embedded in the processor), and another relatively high-bandwidth DRAM (e.g., second DRAM (e.g., HBM)) ((3322)) is placed side-by-side with the processor (320) on a substrate on which the processor (320) is mounted. Here, being placed side-by-side means that one side of the processor (320) and one side of the other DRAM (e.g., second DRAM (3322)) face each other.

[0082] According to one embodiment, the first DRAM (3321) is a memory specialized for low capacity, low power, and high-speed signal transmission, and can serve as a temporary storage of data required for computation of a neural network processing unit (NPU) (3202) and as a cache memory. According to one embodiment, the first DRAM (3321) can be formed with a structure of one transistor and one capacitor (1T1C).

[0083] According to one embodiment, the first DRAM (3321) may be stacked with respect to the processor (320). According to one embodiment, the processor (320) (e.g., application processor) may be manufactured in the form of a single chip including the above-described plurality of processing devices, and the processor (320) manufactured in the form of a chip may include a top surface (320a) and a bottom surface (320b). According to one embodiment, the first DRAM (3321) may be placed on the top surface (320a) of the processor (320). According to one embodiment, the first DRAM (3321) may be stacked with the processor (320) in a package-on-package (PoP) form.

[0084] According to one embodiment, the second DRAM (3322) is a memory specialized for high capacity, high bandwidth, and high-performance signal transmission, and can serve to store parameters of a large-scale language model (LLM) or large-capacity data. The second DRAM of the present disclosure may be referred to as "high-bandwidth memory (HBM)." According to one embodiment, the second DRAM (3322) may be configured such that a plurality of DRAM dies (33221, 33222, 33223) are stacked in the height direction and electrically connected through through-silicon vias to transmit signals.

[0085] Among the plurality of processing units included in the processor (320) (e.g., application processor), the neural network processing unit (NPU) (3202) may be placed on the die of the processor (320) at a location adjacent to memory that is not stacked on the processor (320) (e.g., second DRAM (3322)). For example, the die of the processor (320) (or the processor (320) manufactured in the form of a chip) may be formed in a rectangular shape, and the die may include four sides (320c, 320d, 320e, 320f) facing different directions. And, referring to FIG. 3, the second DRAM (3322) may be placed adjacent to one of the four sides (320e). For example, the die of the processor (320) may include a first side (320c), a second side (320d), a third side (320e), and a fourth side (320e), and the second DRAM (3322) may be placed adjacent to the third side (320e). In other words, the third side (320e) of the die of the processor (320) may face the second DRAM (3322). In this case, the neural network processing unit (NPU) (3202) may be placed adjacent to the side (320e) closest to the second DRAM (3322) among the four sides (320c, 320d, 320e, 320f) of the die, that is, the side (320e) facing the second DRAM (3322).

[0086] According to one embodiment, a processor (320) (e.g., an application processor) and heterogeneous memory (3321, 3322) are mounted on one side (310a) of a printed circuit board (310), wherein the first DRAM (3321) is mounted on the printed circuit board (310) together with the processor (320) in a stacked state on the processor (320), and the second DRAM (3322) can be mounted on the printed circuit board (310) at a position adjacent to one side (320e) of the processor (320).

[0087] According to one embodiment, the printed circuit board (310) must include an area where a processor (320) (e.g., an application processor) and a second DRAM (3322) are mounted, so the overall shape of the printed circuit board (310) may have a shape in which the length in one direction (length direction (X-axis direction)) is longer than the length in the other direction (width direction (T-axis direction)). When the first side (320c) and the fourth side (320f) of the four sides (320c, 320d, 320e, 320f) of the die of the processor (320) are positioned to face a direction parallel to the width direction of the printed circuit board (310), and the second side (320d) and the third side (320e) are positioned to face a direction parallel to the length direction of the printed circuit board (310), the neural network processing unit (NPU) (3202) can be positioned closer to the third side (320e) facing inward of the die compared to the second side (320d) facing outward of the die of the application processor (320).

[0088] According to the embodiments of FIGS. 2 and 3, the present disclosure can improve the performance and efficiency of processing an artificial intelligence model (e.g., a multi-modal large-scale language model (multi-modal LLM)) through an application processor (320) including a neural network processing unit (3202) and a 3D stacked structure of a first DRAM (3321) and a second DRAM (3322). Additionally, the 3D stacked structure may include a second DRAM that implements high-bandwidth memory by vertically stacking DRAM dies, thereby dramatically increasing data throughput (or throughput). Furthermore, the 3D stacked structure may reduce the area occupied by the memory and / or mounting space by having the first DRAM stacked in a package-on-package (PoP) manner and the second DRAM vertically stacked using through-silicon vias (TSVs). This enables the thinning of the electronic device.

[0089] In the electronic device of the present disclosure, a 3D stacked structure is applied to minimize physical distance, while utilizing a high-speed interface (e.g., CXL) connected between the application processor and the second DRAM to maximize data transfer speed, thereby significantly reducing memory access time latency.

[0090] In addition, according to the present disclosure, various embodiments of intelligent memory distribution technology based on a memory management accelerator (MMA) can be provided. Intelligent memory distribution technology may involve analyzing and predicting data access patterns to optimize bank allocation between a first DRAM and a second DRAM. Through this, time delay can also be further reduced.

[0091] In addition, according to the present disclosure, time delay may be further reduced by applying pipeline parallelization technology, which distributes layers of a large-scale language model across multiple neural network processing units (NPUs) and banks of each memory (first DRAM and second DRAM) in a pipeline form. System throughput may also be increased due to the application of pipeline parallelization technology.

[0092] Hereinafter, the arrangement relationship between the application processor and the memory will be explained in more detail through the cross-sectional views of the embodiments of FIGS. 4 to 7.

[0093] FIG. 4 is a cross-sectional view showing the arrangement relationship between a processor (320) (e.g., an application processor) and a memory (332) according to one embodiment. The embodiment of FIG. 4 may conceptually show the arrangement of a processor (320) and a memory (332) on a printed circuit board (310) in an electronic device (300) (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2).

[0094] According to one embodiment, the processor (320) (e.g., application processor) may be stacked vertically above and below the first DRAM (3321) and arranged side-by-side with the second DRAM (3322). The processor (320) may be assembled in a package-on-package (PoP) manner in which the die of the first DRAM (3321) is directly stacked on top of the first DRAM (3321) and the processor (320). Additionally, the second DRAM (3322) may have a data storage capacity increased by stacking multiple DRAM dies and a data transfer speed improved through vertical connections between DRAM dies using through-silicon vias (TSVs) (V).

[0095] Here, "the processor (320) is positioned next to the second DRAM (3322)" means that one side of the processor (320) and one side of the second DRAM (3322) may be positioned adjacent to each other while facing each other. For example, the third side (320e) of the processor (320) may be positioned adjacent to one side of the second DRAM (3322) while facing each other. According to one embodiment, the processor (320) may be positioned next to the second DRAM (3322) on the same plane of the printed circuit board (310), for example, on one side (310a) of the printed circuit board (310).

[0096] According to one embodiment, an assembly formed by stacking the first DRAM (3321) on the processor (320) may be formed to have substantially the same height as the second DRAM (3322). For example, as shown in FIG. 4, the top surface of the assembly formed by stacking the first DRAM (3321) on the processor (320) and the top surface of the second DRAM (3322) may form substantially the same plane. According to one embodiment, the combined height of the processor (320), the first DRAM (3321), and the solder balls electrically connecting the processor (320) and the first DRAM (3321) may have substantially the same height as the second DRAM (3322).

[0097] According to one embodiment, among a plurality of processing units included in a processor (320) (e.g., an application processor), a neural network processing unit (NPU) (3202) may be positioned adjacent to the second DRAM (3322). For example, if the processor (320) includes another processing unit (e.g., a central processing unit (3201)) together with the neural network processing unit (NPU) (3202), the neural network processing unit (NPU) (3202) may be positioned closer to the second DRAM (3322) than the other processing unit (e.g., the central processing unit (3201)).

[0098] The neural network processing unit (NPU) (3202) can be connected to the first DRAM (3321) via the first interface (I2). Additionally, the neural network processing unit (NPU) (3202) can be connected to the second DRAM (3322) via the second interface (I3). The neural network processing unit (NPU) (3202) is configured to receive data perpendicularly to the surface of the die with respect to the first DRAM (3321), which has a relatively short signal transmission distance, and includes the second interface (I3), which is configured to enable high-speed interfacing with the second DRAM (3322), which has a relatively long signal transmission distance, thereby enabling faster signal processing. For example, the second interface (I3) is connected via a computing express link (CXL) to enable high-speed data transfer between the neural network processing unit (NPU) (3202) and the second DRAM (3322), and multiple stacked DRAM dies (33221, 33222, 33223, 33224) in the second DRAM (3322) are connected to each other via through-silicon vias (TSV) (V), thereby enabling vertical connection between dies and thus providing a fast data transfer speed.

[0099] Referring to FIG. 4, the number of multiple DRAM dies (33221, 33222, 33223, 33224) stacked on the second DRAM (3322) is not limited to any specific embodiment. For example, as shown in FIG. 3, the second DRAM (3322) may include three DRAMs, and as shown in FIG. 4, it may include four DRAMs. Various other embodiments may also be applied.

[0100] According to one embodiment, among the plurality of DRAM dies (33221, 33222, 33223, 33224) stacked on the second DRAM (3322), the DRAM die (33221) positioned at the bottom facing the printed circuit board (310) may be a DRAM that serves as a base die rather than a DRAM in which actual instructions or data are stored. According to one embodiment, the electronic device (300) may further include an interposer positioned between the processor (320) and the printed circuit board (310) and between the second DRAM (3322) and the printed circuit board (310) in the connection between the processor (320) and the second DRAM (3322). By using the interposer, the transfer speed between the processor (320) and the second DRAM (3322) may be further increased. Examples of configurations for multiple DRAM dies (33221, 33222, 33223, 33224) and examples of connections between the processor (320) and the second DRAM (3322) may vary, and these will be explained in more detail below through the examples of FIGS. 5 to 7.

[0101] FIG. 5 is a cross-sectional view showing the arrangement relationship between a processor and memory according to one embodiment. FIG. 6 is a cross-sectional view showing the arrangement relationship between a processor and memory according to one embodiment. FIG. 7 is a cross-sectional view showing the arrangement relationship between a processor and memory according to one embodiment.

[0102] The embodiments of FIGS. 5 to 7 may be combined with the embodiments of FIGS. 1 to 4 described above, or the embodiments of FIGS. 8 to 18b described later. In describing the embodiments of FIGS. 5 to 7, descriptions that overlap with the previously described embodiments may be omitted.

[0103] Referring to FIG. 5, an electronic device (300) according to one embodiment may include a second DRAM (3322) in which a plurality of stacked DRAM dies (33221, 33222, 33223, 33224) are all DRAMs in which instructions or data are stored. In the embodiment of FIG. 5, since a plurality of stacked DRAM dies (33221, 33222, 33223, 33224) are all DRAMs in which actual instructions or data are stored, a component acting as a separate base die may be included. Referring to FIG. 5, a substrate (312) may be included as a component acting as a separate base die for a plurality of DRAM dies (33221, 33222, 33223, 33224) in the electronic device. According to the embodiment of FIG. 5, by replacing the DRAM die at the location where the base die was removed, the capacity can be improved compared to the embodiment using the base die.

[0104] That is, FIG. 5 discloses a base dieless structure, and for this purpose, the electronic device (300) may additionally include a separate substrate (312) distinct from the printed circuit board (310). Meanwhile, the electronic device (300) may additionally include a substrate (311) corresponding to the processor (320) to avoid forming a step difference between the processor (320) (e.g., application processor) and the second DRAM (3322). According to one embodiment, the heights of the substrate (311) corresponding to the processor (320) and the substrate (312) corresponding to the second DRAM (3322) may be substantially the same.

[0105] According to the embodiment of FIG. 5, the base die can be removed and the DRAM can be connected to the printed circuit board (310) using a substrate. This allows the thickness of the electronic device to be designed to be thinner than in the embodiment using the base die. Additionally, time latency can be reduced by shortening the signal transmission path. Furthermore, removing the base die can reduce power consumption, thereby increasing power efficiency and reducing the cost of manufacturing the base die.

[0106] Referring to FIG. 6, the electronic device (300) may include a union substrate (313) integrated into one instead of the substrates (311, 312) of FIG. 5. By integrating the substrate corresponding to the processor (320) and the substrate corresponding to the second DRAM (3322) into one and packaging them, the size of the processor (320) chip can be reduced, and the physical distance between the processor (320) and the second DRAM (3322) can be reduced to improve the data transfer speed and reduce time latency.

[0107] Referring to FIG. 7, the electronic device (300) may include a union substrate (314) instead of the substrates (311, 312) of FIG. 5, and additionally, may further include an interposer (315). By using the interposer (315), signal integrity can be ensured by increasing the signal quality between the processor (320) and the second DRAM (3322), and the transmission speed between the processor (320) and the second DRAM (3322) can be further improved by enabling high-density connections.

[0108] FIG. 8 is a block diagram illustrating a signal processing method between a processor (420), a first DRAM (4321), and a processor (420) and a second DRAM (4322) according to one embodiment. In describing the embodiment of FIG. 8, descriptions that overlap with the previously described embodiment may be omitted.

[0109] Referring to FIG. 8, an electronic device (400) (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3 to 7) may include a processor (420) (e.g., processor (120) of FIG. 1, processor (220) of FIG. 2, processor (320) of FIG. 3 to 7)) and memory (e.g., memory (130) of FIG. 1, memory (230) of FIG. 2, memory (332) of FIG. 3 to 7). The memory may include a first DRAM (4321) (e.g., first DRAM (3321) of FIG. 3 to 7) and a second DRAM (4322) (e.g., second DRAM (3322) of FIG. 3 to 7).

[0110] The signal between the processor (420) (e.g., application processor) and the first DRAM (4321) can be performed independently of the signal processing between the processor (420) and the second DRAM (4322).

[0111] The first DRAM (4321) may consist of a single DRAM die (4321a). The first DRAM (4321) includes a bank having a unique data storage area, and the first DRAM (4321) may include multiple banks. Each bank is configured to read or write data independently, so that parallel processing of multiple banks may be possible. Through independent and parallel processing banks, efficient utilization of bandwidth is possible and bottlenecks are minimized, allowing for high data transfer speeds.

[0112] The first DRAM (4321) may include a row decoder (4321b) and a column decoder (4321c). The row portion of the memory address can be interpreted using the row decoder (4321b) to select the row where the corresponding data is stored, and the column decoder (4321c) can be used to interpret the column portion of the memory address to select the column where the corresponding data is stored.

[0113] The first DRAM (4321) may include a sense amplifier and / or a write driver (4321d). If it includes a sense amplifier, it can be used to detect and amplify data stored in a memory cell and output it. If it includes a write driver, it can be used to write data to a memory cell.

[0114] The second DRAM (4322) may be composed of multiple DRAM dies (4322a). For example, the second DRAM (4322) may include three DRAM dies (43221, 43222, 43223). However, the number of DRAM dies may vary depending on the embodiment. Each die included in the second DRAM (4322) includes a bank having a unique data storage area, and a single DRAM die may include multiple banks. The second DRAM (4322) may have more banks than the first DRAM (4321). Each bank is configured to read or write data independently, so that parallel processing of multiple banks may be possible. Through independent and parallel processing banks, efficient utilization of bandwidth is possible, and bottlenecks are minimized, allowing for a high data transfer speed. The three DRAM dies (43221, 43222, 43223) of the second DRAM (4322) are configured to be stacked vertically, and each die can be connected to each other through a through-silicon via (TSV).

[0115] The second DRAM (4322) may also include a row decoder (4322b) and a column decoder (4322c), just like the first DRAM (4321), and may include a sense amplifier and / or a write driver (4322d).

[0116] A processor (420) (e.g., an application processor) may include a plurality of processing units. As previously described in the embodiments of FIGS. 2 and 3, the processor (420) may include a central processing unit (CPU) (4201), a neural network processing unit (NPU) (4202), and a graphics processing unit (GPU) (4203). In the embodiment of FIG. 8, additionally, a cache memory (4207) and a DMA (direct memory access) (4208) may be included, and a memory controller (4209) may also be included. The cache memory (4207) may be a memory mounted inside the processor (420) to enable fast access to large amounts of memory. The DMA (4208) may be configured to directly perform data transfer between the cache memory and peripheral devices of the processor (420) (e.g., a first DRAM (4321), a second DRAM (4322)) without the intervention of the central processing unit (CPU). The memory controller (4209) may be responsible for accessing and controlling peripheral devices of the processor (420) (e.g., the first DRAM (4321), the second DRAM (4322)), or accessing and controlling the cache.

[0117] According to the present disclosure, the electronic device (400) may include a memory management accelerator (MMA) (42021) inside a neural network processing unit (NPU) (4202). Hereinafter, the role (or operation) of the memory management accelerator will be described in detail with reference to FIG. 9.

[0118] FIG. 9 is a block diagram showing the step-by-step operation of a memory management accelerator (MMA) according to one embodiment.

[0119] According to the present disclosure, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3 to 7, electronic device (400) of FIG. 4) can perform operations such as analyzing memory access patterns, predicting memory access patterns, and optimizing bank allocation using a memory management accelerator (MMA) (e.g., memory management accelerator (22021) of FIG. 2, memory management accelerator (42021) of FIG. 8).

[0120] The step-by-step operation of the memory management accelerator (MMA) may include input data collection, AI model processing, and output and feedback operations.

[0121] In the input data collection operation, data such as information (901) of a target large language model (LLM), data access information (902) of a neural network processing unit (NPU), and / or bank state information (903) of a DRAM may be input to the memory management accelerator (MMA).

[0122] For example, "information of the target large-scale language model (LLM) (901)" may include information such as the layer structure, operation characteristics, and memory access patterns of the large-scale language model (LLM). The layer structure information may include information such as the type of layer, size (e.g., number of nodes per layer), and connection relationships between layers, such as the input layer, hidden layer, and output layer of the large-scale language model. The operation characteristics information may include information such as the type of operation used and the amount of operation. The memory access pattern may include information such as the frequency of memory access and the access size for each layer.

[0123] For example, "data access information (902) of a neural network processing unit (NPU)" may include information such as an access address, access time, access randomness, and access sequentiality. The access address may include information regarding the address of the memory accessed by the neural network processing unit (NPU). The access time may include information regarding the time when the neural network processing unit (NPU) accesses the memory. The access randomness may include information regarding the degree of random distribution of the access address. The access sequentiality may include information regarding the presence or absence of a sequential pattern of the access address. The memory management accelerator may continuously collect information on memory access requests from the neural network processing accelerator (NPU). Furthermore, the memory management accelerator may identify characteristics of the data access pattern by collecting information such as the access address, access time, access randomness, and access sequentiality.

[0124] For example, "bank status information of DRAM (903)" may include information regarding the current capacity, remaining capacity, number of data accesses, and data transmission status of the DRAM. According to one embodiment, when a first DRAM and a second DRAM are included as volatile memory as disclosed in the present disclosure, the bank status information of the DRAM may be stored independently of the first DRAM and the second DRAM, respectively. That is, the bank status information of the DRAM may include bank status information of the first DRAM and bank status information of the second DRAM.

[0125] The data collected during the input data collection operation can be input into an artificial intelligence model within the memory management accelerator (MMA). Then, the memory management accelerator (MMA) can perform AI model processing operations based on the collected input data.

[0126] An artificial intelligence model may have a neural network (NN) structure. In the NN structure of FIG. 9, the input layer is shown to have 3 nodes, the 3 hidden layers each have 5 nodes, and the output layer has 3 nodes. This is merely an example of an NN structure, and the NN structure of the present disclosure is not limited to any specific embodiment.

[0127] According to one embodiment, an AI model processing operation may include a data preprocessing operation, a feature extraction / transformation operation, a predictive model training operation, and a predictive model update operation. The data preprocessing operation may be an operation that normalizes collected input data and transforms it into a required form. Additionally, the data preprocessing operation may be an operation that performs missing value processing, duplicate removal, outlier removal, and / or feature scaling. The feature extraction / transformation operation is an operation that extracts features of data access patterns, and may be an operation such as vectorizing the extracted features or transforming categorical data into numerical data (and vice versa). The predictive model training operation may include a reinforcement model training operation and a time series learning operation. Here, the reinforcement learning operation may be an operation that sets system performance based on bank allocation decisions as a reward and learns an optimal bank allocation policy. Time series learning operations may be operations that predict future access patterns by utilizing time series prediction models such as LSTM (long short-term memory). Predictive model update operations may be operations that improve accuracy by continuously updating the prediction model based on new data access information.

[0128] An electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIG. 3 to 7, electronic device (400) of FIG. 4) can analyze data access patterns and predict and / or infer access patterns by utilizing various AI techniques such as reinforcement learning and time series learning using a memory management accelerator (MMA) (e.g., memory management accelerator (22021) of FIG. 2, memory management accelerator (42021) of FIG. 8). Additionally, the accuracy of analysis, prediction, and / or inference can be improved by updating the prediction model periodically and / or whenever new data is input.

[0129] In the output and feedback operation, the memory management accelerator (MMA) can make an optimal "memory bank allocation decision (904)" based on predicted access patterns and bank state information. And, it can update the "access pattern inference result (905)". The "memory bank allocation decision (904)" is to determine the optimal bank allocation by considering the predicted access patterns and bank load state, and may determine the size and location of the data blocks to be allocated to each bank. The memory management accelerator can maximize memory access efficiency by determining the size and location of the data blocks to be allocated to each bank contained in memory. The "access pattern inference result update" may update the data access pattern information based on the inference result of the prediction model. The updated information can be used for training the next prediction model. This process can be performed in real time and can respond to constantly changing data access patterns.

[0130] Hereinafter, various scenarios for scheduling and priority processing algorithms between a first DRAM and a second DRAM dedicated to a neural network processing unit (NPU) can be provided through the embodiments of FIGS. 10 to 15.

[0131] FIG. 10 is a block diagram illustrating a processing method in which a plurality of artificial intelligence models are distributed and allocated to a plurality of DRAMs for processing, according to one embodiment.

[0132] Referring to FIG. 10, a first artificial intelligence model (AI MODEL_1) may be assigned to the first DRAM (4321), and a second artificial intelligence model (AI MODEL_2) may be assigned to the second DRAM (4322). The artificial intelligence models assigned to the first DRAM (4321) and the second DRAM (4322) may be controlled and computed by a neural network processing unit (NPU). The first artificial intelligence model (AI MODEL_1) may include pre-processing (M11), attention (M12), feed-forward network (FFN) (M13), and post-processing (M14). "Pre-processing (M11)" is a process of preparing and converting data before inputting it into the artificial intelligence model. For example, if text data is input, it may be tokenized, converted into a numeric vector, and then normalized to form a form suitable for the artificial intelligence model. "Attention (M12)" is a mechanism by which an artificial intelligence model processes input data by assigning weights, and may, for example, learn relationships between preprocessed text data, i.e., input tokens. "FFN (M13)" can function to learn more complex patterns based on the results of Attention (M12), transform them non-linearly, and improve the expressiveness of the artificial intelligence model. "Post-processing (M14)" is a process of converting the results predicted by the artificial intelligence model to fit the actual situation—in other words, converting them into an interpretable form—and providing the output prediction results to the user through processes such as visualization. According to one embodiment, a softmax activation function may be used for post-processing (M14), but this is merely one example and is not necessarily limited thereto.The second artificial intelligence model (AI MODEL_2) may also include preprocessing (M21), attention (M22), FFN (M23), and postprocessing (M24), wherein the preprocessing (M11), attention (M12), FFN (M13), and postprocessing (M14) of the first artificial intelligence model (AI MODEL_1) may be applied respectively.

[0133] The first artificial intelligence model (AI MODEL_1) and the second artificial intelligence model (AI MODEL_2) may be different artificial intelligence models. For example, the FFN (M13) of the first artificial intelligence model (AI MODEL_1) may have a neural network (NN) structure different from the FFN (M23) of the second artificial intelligence model (AI MODEL_2). Figure 10 illustrates that both the first artificial intelligence model (AI MODEL_1) and the second artificial intelligence model (AI MODEL_2) have an input layer with 3 nodes, three hidden layers with 5 nodes each, and an output layer with 3 nodes, but this is just one example and various other variations are possible. The first artificial intelligence model (AI MODEL_1) and the second artificial intelligence model (AI MODEL_2) may correspond to the same input parameters, or at least some different input parameters. For example, as illustrated in FIG. 10, in the case where the input layers of the first artificial intelligence model (AI MODEL_1) and the second artificial intelligence model (AI MODEL_2) each have three multiple nodes, the three input parameters corresponding to the three nodes of the input layer of the first artificial intelligence model (AI MODEL_1) may be identical to, or at least partially different from, the three input parameters corresponding to the three nodes of the input layer of the second artificial intelligence model (AI MODEL_1). Meanwhile, those skilled in the art will understand that the examples described above are merely exemplary, and there are no limitations on the type of artificial intelligence model, the NN structure of the artificial intelligence model, and / or the input parameters of the artificial intelligence model.

[0134] FIGS. 11a and FIGS. 11b are flowcharts illustrating a processing method when a plurality of artificial intelligence models are distributed across a plurality of DRAMs for processing, according to one embodiment.

[0135] Note that in the operation of determining whether there are two or more artificial intelligence models of FIG. 11a, if there are two or more artificial intelligence models, the flowchart leads to a memory management accelerator (MMA) processing operation (11031) for the artificial intelligence model assigned to the first DRAM and a memory management accelerator (MMA) processing operation (11041) for the artificial intelligence model assigned to the second DRAM referenced in FIG. 11b.

[0136] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0137] According to one embodiment, the operations 1101, 1102, 1103, 1104, and further subdivided, 11011, 11012, 11013, 1102, 11031, 11032, 11033, 11034, 11035, 11037, 11041, 11042, 11043, 11044, 11045, 11046, 11047, can be understood as being performed in each processor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3 to 7, the electronic device (400) of FIG. 4) (e.g., the processor (120) of FIG. 1, the processor (220) of FIG. 2, the processor (320) of FIG. 3 to 7, the processor (420) of FIG. 8). there is.

[0138] Referring to FIG. 11a and FIG. 11b together, according to one embodiment, in operation 11011, the electronic device may initiate an artificial intelligence model processing operation and a system for the same based on a user requesting an artificial intelligence model service.

[0139] In operation 11012 following operation 11011, the electronic device can initialize the neural network processing unit (NPU) and memory (DRAM, flash memory) to prepare the artificial intelligence model processing operation and the system therefor. Before starting an application or service containing an artificial intelligence model, there may be factors that affect memory (e.g., DRAM) bank allocation depending on the bank size of the area allocated and used by background (BG) apps and foreground apps running on the electronic device. In the initialization phase, it may be necessary to secure memory for the artificial intelligence model processing operation and the system therefor by clearing apps that may affect the initialization in advance. According to one embodiment, the initialization operation may differ if processing of multiple models is required when executing an artificial intelligence model. In the case of multiple models, initialization (erase) may be required for multiple memory areas. As illustrated in FIG. 10 above, if you want to distribute multiple artificial intelligence models across multiple memories (first DRAM, second DRAM), you may need to perform an initialization operation for each memory (first DRAM, second DRAM).

[0140] In operation 11013, the electronic device may perform an artificial intelligence model processing operation when user input occurs. Here, user input can be various, such as text, voice, or image, and there are no limitations according to the embodiment. Inputs other than the user inputs exemplified above are also possible, and it may be possible for multiple user inputs to be input and processed simultaneously. Although operation 11013 related to the occurrence of user input is illustrated in FIG. 11a as being performed after operation 11011 and operation 11012, it is not necessarily limited thereto, and the order may be changed. For example, operation 11013 related to the occurrence of user input may be performed together with the artificial intelligence model service entry operation 11011.

[0141] Based on the occurrence of user input, the electronic device may perform an artificial intelligence model processing operation in operation 1102. If there is only a single artificial intelligence model, processing for that artificial intelligence model is performed, and if there are two or more artificial intelligence models, each model may be processed in a distributed manner in the first DRAM and the second DRAM. For example, as an example of two or more artificial intelligence models, the artificial intelligence model of the memory management accelerator may be, for example, a deep learning model, and may support, for example, text-to-speech (TTS), speech-to-text (STT), translation, braille / image-to-text, LID (language identification) detection, and / or ROI (region of interest) detection, but there is no limit to the type and / or number of supported functions (or sub-artificial intelligence models). As described above, when an electronic device supports various artificial intelligence models, some artificial intelligence models may be allocated to the first DRAM for processing, and other artificial intelligence models may be allocated to the second DRAM (DRMA 2) for processing.

[0142] Referring to FIG. 11a, in operation 11031, the electronic device may perform processing using a memory management accelerator (MMA) on an artificial intelligence model allocated to the first DRAM. Here, processing using a memory management accelerator (MMA) may involve analyzing memory usage and the computational amount of a neural network processing unit (NPU) by considering various factors such as architecture analysis, layer configuration, parameter types and sizes, and computational characteristics for the artificial intelligence model allocated to the first DRAM.

[0143] In operation 11032, the electronic device may load weight and / or bias information and other setting data of the corresponding artificial intelligence model from flash memory to the first DRAM based on the analysis result of operation 11031. In this case, the loading from flash memory to the first DRAM may be performed through a direct memory access (DMA) method.

[0144] In operation 11033, the electronic device can load node values ​​and parameters from the first DRAM to the neural network processing unit (NPU) based on loading from flash memory to the first DRAM.

[0145] In operation 11034, the electronic device may enable a neural network processing unit (NPU) to perform computations of an artificial intelligence model in parallel or sequentially using received node values ​​and parameters based on loading from the first DRAM.

[0146] In operation 11035, the electronic device can transmit intermediate result node values ​​generated during calculation to the first DRAM based on parallel or sequential operation of the neural network processing unit (NPU).

[0147] In relation to operations 11036 and 11037, the electronic device can check whether the operation of the artificial intelligence model allocated to the first DRAM has been completed (operation 11036). If the operation has not been completed, the operation continues until the operation of the artificial intelligence model is completed, and when the operation is completed, a result value is output (operation 11037).

[0148] Meanwhile, referring to FIG. 11b, in operation 11041, the electronic device can perform processing using a memory management accelerator (MMA) on an artificial intelligence model assigned to the second DRAM. The artificial intelligence model assigned to the second DRAM can also perform processing using a memory management accelerator (MMA) in substantially the same way as the first DRAM.

[0149] In operation 11042, the electronic device may load weight and / or bias information and other setting data of the corresponding artificial intelligence model from flash memory to the second DRAM based on the analysis result of operation 11041. In this case, the loading from flash memory to the second DRAM may be performed through a direct memory access (DMA) method.

[0150] In operation 11043, the electronic device can load node values ​​and parameters from the second DRAM to the neural network processing unit (NPU) based on loading from flash memory to the second DRAM.

[0151] In operation 11044, the electronic device may enable a neural network processing unit (NPU) to perform computations of an artificial intelligence model in parallel or sequentially using received node values ​​and parameters based on loading from a second DRAM.

[0152] In operation 11045, the electronic device can transmit intermediate result node values ​​generated during calculation to the second DRAM based on parallel or sequential operation of the neural network processing unit (NPU).

[0153] In relation to operations 11046 and 11047, the electronic device can check whether the operation of the artificial intelligence model allocated to the second DRAM has been completed (operation 11046). If the operation has not been completed, the operation continues until the operation of the artificial intelligence model is completed, and when the operation is completed, a result value is output (operation 11047).

[0154] The artificial intelligence model processing operation using the first DRAM shown in FIG. 11a and the artificial intelligence model processing operation using the second DRAM shown in FIG. 11b can be performed independently of each other. According to one embodiment, the artificial intelligence model processing operation using the first DRAM and the artificial intelligence model processing operation using the second DRAM may be performed at different times, but may also be performed simultaneously. The memory management accelerator (MMA) can optimize the bank allocation of the first DRAM and the second DRAM by considering the characteristics of each artificial intelligence model and memory usage. Through this, the processing performance and processing speed (efficiency) of the electronic device can be increased.

[0155] FIG. 12 is a block diagram illustrating a processing method when a multi-modal large-scale language model (multi-modal LLM) is distributed across a plurality of DRAMs for processing according to one embodiment.

[0156] Referring to FIG. 12, the first DRAM (4321) and the second DRAM (4322) are configured to be allocated and processed together a multi-modal large-scale language model (multi-modal LLM), and during the processing, the artificial intelligence model allocated to the first DRAM (4321) and the second DRAM (4322) can be controlled and computed by a neural network processing unit (NPU).

[0157] A multi-modal large-scale language model (multi-modal LLM) may include pre-processing (M1), attention (M2), feed-forward network (FFN) (M3), and post-processing (M4). The methods for pre-processing (M1), attention (M2), FFN (M3), and post-processing (M4) in FIG. 12 may be based on the embodiments described above in FIG. 10.

[0158] In order to efficiently distribute the processing of a multi-modal large-scale language model (multi-modal LLM), the roles of the first DRAM (4321) and the second DRAM (4322) may be adapted to the characteristics of the internal computations of the model. According to the embodiment illustrated in FIGS. 12 to 13b of the present disclosure, optimized computations can be assigned to the first DRAM (4321) and the second DRAM (4322), respectively, based on the difference in computational characteristics during the Attention (M2) and FFN (M3) stages. According to one embodiment, when the model includes computational processing stages with a large amount of computation in the order of inference progress, the second DRAM (4322) may be assigned to computations with a relatively large amount of computation processing, and the first DRAM (4321) may be assigned to computations with a relatively small amount of computation processing. In cases where a high-throughput operation is performed in a higher priority than a relatively low-throughput operation, the second DRAM (4322) may be assigned to the higher priority operation and the first DRAM (4321) may be assigned to the lower priority operation.

[0159] According to one embodiment, the Attention (M2) may be Multi-Head Attention. While a single Attention calculates one Attention for each word in an input sequence and assigns weights to important information based on it, Multi-Head Attention can execute multiple Attentions in parallel. For example, assuming a state where text input is tokenized, Multi-Head Attention can calculate similarity for all pairs of tokens to identify relationships between input tokens. In this case, each token can be represented by a Query (Q), Key (K), and Value (V) vector, and similarity can be calculated through a dot product operation between Q and K. When the number of input tokens is n, a total of n squared dot product operations are required, so the computational complexity may correspond to the square of n. Therefore, as the number of input tokens increases, the amount of computation increases quadratically, which can lead to a greater computational burden. Since Multi-Head Attention must calculate similarity for all pairs of tokens, memory access patterns may occur randomly and frequently. This can consume a large amount of memory bandwidth and be a factor in increasing time latency. The second DRAM (4322) (e.g., HBM) is a high-bandwidth memory and can be advantageous for random and frequent memory access. Accordingly, according to one embodiment of the present disclosure illustrated in FIG. 12, an Attention (M2) (e.g., Multi-Head Attention) can be allocated to the second DRAM (4322) to alleviate memory bottlenecks and improve computation speed.

[0160] Meanwhile, according to one embodiment of the present disclosure illustrated in FIG. 12, the FFN (M3) can be assigned to the first DRAM (4321). Since the FFN (M3) processes each token independently, the amount of computation may increase in proportion to the number of input tokens n. Therefore, the computational complexity may be proportional to n. That is, the FFN (M3) may have a relatively lower computational burden compared to the computational complexity of Attention (M2) (e.g., Multi-Head Attention). In addition, since the FFN (M3) processes each token sequentially, the memory access pattern may be relatively regular and have high locality. This may be advantageous for consuming less memory bandwidth and reducing latency. The first DRAM (4321) is a low-capacity but high-speed memory and may be suitable for regular and highly local memory access. Therefore, the FFN (M3) operation can be assigned to the first DRAM (4321) to provide fast data processing speed.

[0161] The process of inference in an artificial intelligence model can be divided into an initial stage, an intermediate stage, and a late stage. In the initial stage, the computational amount of Attention (M2) (e.g., Multi-Head Attention) and FFN (M3) in the artificial intelligence model may be similar, or the computational amount of FFN (M3) may be greater. In the intermediate stage, as inference progresses, the computational amount of Attention (M2) (e.g., Multi-Head Attention) increases quadratically as the number of input tokens increases, so it may begin to overwhelm the computational amount of FFN (M3), which increases proportionally as the number of input tokens increases. In the late stage, the computational amount of Attention (M2) (e.g., Multi-Head Attention) in the artificial intelligence model may become much larger than the computational amount of FFN (M3), which may become a bottleneck in the processing of the artificial intelligence model. According to one embodiment, the bank allocation of the first DRAM (4321) and the second DRAM (4322) may be dynamically adjusted in consideration of the computational burden that changes as inference progresses. For example, in the initial stage of the artificial intelligence model, the operation of FFN (M3) may be assigned to the second DRAM (4322) and the operation of Attention (M2) (e.g., Multi-Head Attention) may be assigned to the first DRAM (4321), and in the intermediate and later stages of the artificial intelligence model, the operation of Attention (M2) (e.g., Multi-Head Attention) may be assigned to the second DRAM (4322) and the operation of FFN (M3) may be assigned to the first DRAM (4321).

[0162] As illustrated in FIG. 12, the Attention (M2) (e.g., Multi-Head Attention) and FFN (M3) of the artificial intelligence model are merely examples, and the processing of the artificial intelligence model is not limited thereto. It should be noted that the present disclosure is not limited to any specific algorithm, but provides an embodiment in which optimized operations can be assigned to the first DRAM (4321) and the second DRAM (4322), respectively, by considering the differences in computational characteristics within the artificial intelligence model according to the type of algorithm. Through this, memory access efficiency can be maximized and the overall performance of the system processing the multi-modal large-scale language model (multi-modal LLM) can be improved. In addition, by dynamically adjusting the bank allocation of the first DRAM (4321) and the second DRAM (4322) in consideration of the computational burden that changes as inference progresses, system resources for processing the multi-modal large-scale language model (multi-modal LLM) can be utilized efficiently.

[0163] FIGS. 13a and FIGS. 13b are flowcharts illustrating a processing method when a multi-modal large-scale language model (multi-modal LLM) is distributed across multiple DRAMs for processing.

[0164] Note that the flowchart following the operation of split-loading the artificial intelligence model stored in the flash memory of FIG. 13a into the first DRAM and the second DRAM leads to the operation (1303) in the second DRAM referenced in FIG. 13b.

[0165] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0166] According to one embodiment, the operations 1301, 1302, 1303, 1304, and further subdivided, 13011, 13012, 13013, 13021, 13022, 13031, 13032, 13033, 13034, 13035, 13036, 13041, 13042, 13043, 13044, 13045, 13046 are understood to be performed in each processor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3 to 7, the electronic device (400) of FIG. 4) (e.g., the processor (120) of FIG. 1, the processor (220) of FIG. 2, the processor (320) of FIG. 3 to 7, the processor (420) of FIG. 8). It is possible.

[0167] Referring to FIG. 13a and FIG. 13b together, according to one embodiment, in operation 13011, the electronic device may initiate an artificial intelligence model processing operation and a system for the same based on a user requesting an artificial intelligence model service.

[0168] In operation 13012 following operation 13011, the electronic device can initialize the neural network processing unit (NPU) and memory (DRAM, flash memory) to prepare for the artificial intelligence model processing operation and the system therefor.

[0169] In operation 13013, the electronic device may perform an artificial intelligence model processing operation when user input occurs. Here, the user input may be multimodal input data such as text, voice, or image. Multimodal input data may vary, and there are no limitations according to the embodiments. Inputs other than the user inputs exemplified above are also possible, and it may be possible for multiple user inputs to be input and processed simultaneously.

[0170] In addition, operations 13011, 13012, and 13013 may be applied mutatis mutandis to operations 11011, 11012, and 11013 mentioned earlier in FIG. 11a and FIG. 11b.

[0171] Based on the occurrence of user input, the electronic device can perform an artificial intelligence model processing operation in operation 1302.

[0172] Referring to FIG. 13a, in operation 13021, the electronic device can perform processing using a memory management accelerator (MMA) on an input multimodal artificial intelligence model, i.e., a multimodal large-scale language model. Here, the multimodal artificial intelligence model can be configured to be stored in flash memory when not in use and to move to and reside in DRAM memory when called. Here, processing using a memory management accelerator (MMA) may involve analyzing memory usage and the computational amount of a neural network processing unit (NPU) by considering various factors such as architecture analysis, layer configuration, parameter types and sizes, and computational characteristics of the multimodal large-scale language model.

[0173] In operation 13022, the electronic device may load a multimodal large-scale language model stored in flash memory into a first DRAM and a second DRAM by dividing it based on the analysis result of operation 13021. In this case, loading from flash memory to the first DRAM can be performed via a direct memory access (DMA) method. According to one embodiment, if the multimodal large-scale language model includes preprocessing (M1), attention (M2), FFN (M3), and postprocessing (M4), the attention (M2) operation (e.g., Multi-Head Attention) may be assigned to the second DRAM, and the FFN (M3) operation may be assigned to the first DRAM. According to some embodiments, as described above in the embodiment of FIG. 12, the loading into the first DRAM and the second DRAM by dividing it may be dynamically adjusted according to the inference progress stage.

[0174] According to one embodiment, an Attention (M2) operation (e.g., Multi-Head Attention) may be allocated to a second DRAM. After a preprocessing (M1) operation is performed in a neural network processing unit (NPU), an Attention (M2) operation (e.g., Multi-Head Attention) may be allocated to a second DRAM. Referring to FIG. 13b, for example, when natural language is input as data input, operations 13031, 13032, and 13033 may be performed as natural language processing operations. Operations 13031, 13032, and 13033 may be performed in a neural network processing unit (NPU). Operation 13031 is a tokenization operation, which may involve dividing text into words, sentences, or smaller units. Operation 13032 is an embedding vectorization operation, which may involve embedding words into vectors. The embedding vectorization operation may be identical or similar to the encoding operation that converts words into unique integers. Operation 13033 is a position encoding operation that may provide additional information so that the AI ​​model can recognize the order (position) of the input data. Through the position encoding operation, position information can be added as a vector to the embedding vector. Although not illustrated in the drawings, this preprocessing operation may also include a padding operation to match the lengths if the lengths of the input sequences differ. Furthermore, while the above-described embodiment describes the preprocessing operation based on natural language (e.g., text) being input as input data, if an image rather than natural language is input, resizing and normalization operations may be included to replace the tokenization operation.

[0175] In operation 13034, the second DRAM of the electronic device can perform operations for Multi-Head Attention. In relation to the Linear Projection operation, the input data can be converted into Query (Q), Key (K), and Value (V) vectors. In Multi-Head Attention, the similarity between the Query (Q) and (K) vectors is calculated by Scaled Dot-Product Attention, and a new representation can be generated by applying weights to the Value (V) vector. Multi-Head Attention includes an Add & Normalization operation (13035), which can increase the stability of model training by adding a residual connection to the Attention result and normalizing it.

[0176] In operation 13036, the calculated intermediate result node value can be stored in the second DRAM.

[0177] The FFN (M3) operation can be assigned to the first DRAM. In operation 13041, the first DRAM performs neural network operations to transform input data and extract features through the FFN (M3) operation. In operation 13042, the stability of the model training can be improved by adding residual connections to the result of the FFN and performing normalization through the Add & Normalization operation. In operation 13043, the final output value can be calculated and a probability distribution can be generated through the Linear & Softmax operation, which is a type of post-processing (M4) operation.

[0178] In operation 13044, the calculated intermediate result node value can be stored in the first DRAM. Such operations can be performed repeatedly until the computation of the artificial intelligence model part allocated to the first DRAM and the second DRAM is completed. Since the computation operation of the artificial intelligence model in the first DRAM and the second DRAM is performed based on the stored intermediate node value, it can gradually achieve higher accuracy.

[0179] In relation to operations 13045 and 13046, the electronic device can check whether the operation of the artificial intelligence model assigned to the first DRAM has been completed (operation 13045). If the operation has not been completed, the operation continues until the operation of the artificial intelligence model is completed, and when the operation is completed, a result value is output (operation 13046).

[0180] As illustrated in FIGS. 13a and 13b, the first DRAM and the second DRAM can perform computations by being allocated specific parts of an artificial intelligence model (Multi-Head Attention and / or FFN), and the memory management accelerator (MMA) can optimize the bank allocation of the first DRAM and the second DRAM by analyzing data access patterns. This can increase the processing performance and processing speed (efficiency) of the electronic device.

[0181] FIG. 14 is a block diagram illustrating a hidden layer distributed processing technique for various types of large-scale language models using a memory management accelerator (MMA) according to one embodiment.

[0182] Referring to FIG. 14, the first DRAM (4321) and the second DRAM (4322) are configured to process a large-scale language model (LLM) together, and during the processing, the first DRAM (4321) and the second DRAM (4322) can be controlled by a neural network processing unit (NPU).

[0183] In order to efficiently distribute the processing of a large-scale language model (LLM), the roles of the first DRAM (4321) and the second DRAM (4322) may be adapted to the characteristics of the internal computation of the model. According to the embodiments of FIGS. 14 and 15 of the present disclosure, by analyzing the computational load of the artificial intelligence model (e.g., large-scale language model (LLM)) and the usage patterns of the first DRAM (4321) and the second DRAM (4322), an optimal allocation strategy can be determined, thereby increasing the processing performance and processing speed (efficiency) of the electronic device.

[0184] Distributed processing techniques for a hidden layer such as Fig. 14 may include adaptive bank allocation, non-linear access, and hardware acceleration techniques. Adaptive bank allocation may secure optimal performance by dynamically adjusting the bank allocation of the first DRAM and the second DRAM according to the characteristics of the Large Language Model (LLM). Non-linear access may maximize memory access efficiency by performing bank allocation considering the non-linear characteristics of data access patterns. Hardware acceleration may improve system performance by hardware-accelerating the bank allocation optimization process through a memory management accelerator.

[0185] FIG. 14 illustrates an example of a neural network (NN) having four hidden layers, with the input layer having six nodes, the four hidden layers having 64, 128, 256, and 512 nodes, and the output layer having three nodes. The number of hidden layers and the number of nodes in the present disclosure may vary depending on the embodiment and are not limited to any specific embodiment.

[0186] FIG. 15 is a flowchart illustrating a hidden layer distributed processing technique for various types of large-scale language models using a memory management accelerator (MMA) according to one embodiment.

[0187] In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0188] According to one embodiment, the operations 15011, 15012, 15013, 15021, 15022, 1503, 1504, 1504, 1506, and 1507 may be understood to be performed in each processor of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIG. 3 to 7, the electronic device (400) of FIG. 4) (e.g., the processor (120) of FIG. 1, the processor (220) of FIG. 2, the processor (320) of FIG. 3 to 7, the application processor (420) of FIG. 8).

[0189] Referring to FIG. 15, according to one embodiment, in operation 15011, the electronic device may initiate an artificial intelligence model processing operation and a system for the same based on a user requesting an artificial intelligence model service.

[0190] In operation 15012 following operation 15011, the electronic device can initialize the neural network processing unit (NPU) and memory (DRAM, flash memory) to prepare for the artificial intelligence model processing operation and the system therefor.

[0191] In operation 15013, the electronic device may perform an artificial intelligence model processing operation when user input occurs. Here, the user input may be input data such as text, voice, or image. Here, the input data may be diverse, and there are no limitations according to the embodiment. Inputs other than the user inputs exemplified above are also possible, and it may be possible for multiple user inputs to be input and processed simultaneously.

[0192] In addition, operations 15011, 15012, and 15013 may be applied mutatis mutandis to operations 11011, 11012, and 11013 mentioned earlier in FIG. 11a and FIG. 11b.

[0193] Based on the occurrence of user input, the electronic device can perform an artificial intelligence model processing operation in operation 1502.

[0194] Referring to FIG. 15, in operation 15021, the electronic device can perform processing using a memory management accelerator (MMA) on an input large-scale language model. Here, processing using a memory management accelerator (MMA) may involve analyzing memory usage and the computational amount of a neural network processing unit (NPU) by considering various factors such as architecture analysis of the large-scale language model, layer configuration, parameter types and sizes, and computational characteristics.

[0195] In operation 15022, the electronic device may distribute and allocate a large-scale language model stored in flash memory to a first DRAM and a second DRAM based on the analysis results of operation 13021. Each layer of the large-scale language model is stored in the first DRAM or the second DRAM, and an optimal distribution method may be determined by considering the characteristics of the model and the capacity and speed of the DRAM.

[0196] In relation to operation 1503, a hidden layer suitable for the first DRAM (4321) and the second DRAM (4322) may be assigned. According to one embodiment, hidden layers in a section with a relatively large number of nodes of the hidden layer may be assigned to the second DRAM (4322), and the remaining hidden layers may be assigned to the first DRAM (4321). For example, the first hidden layer (Hidden layer 1) and the second hidden layer (Hidden layer 2) shown in FIG. 14 may be assigned to the first DRAM (4321), and the third hidden layer (Hidden layer 3) and the fourth hidden layer (Hidden layer 4) may be assigned to the second DRAM (4322).

[0197] In relation to operation 1504, the neural network processing unit (NPU) can compute each layer of the model distributed in the first DRAM (4321) and the second DRAM (4322) in parallel or sequentially. Parallel computation is a method of increasing speed by processing multiple layers simultaneously, and sequential computation is a method of processing layers one by one in order. The memory management accelerator (MMA) can select and process the parallel or sequential computation method considering the model characteristics and hardware resource conditions.

[0198] In operation 1505, the calculated intermediate result node value can be stored in the first DRAM and the second DRAM.

[0199] In relation to operations 1506 and 1507, the electronic device can check whether the operation of the artificial intelligence model assigned to the first DRAM has been completed (operation 1506). If the operation has not been completed, the operation continues until the operation of the artificial intelligence model is completed, and when the operation is completed, a result value is output (operation 1507).

[0200] According to the embodiments of FIGS. 14 and 15 of the present disclosure, a single large-scale language model can be efficiently distributed and processed through a memory management accelerator (MMA) via a first DRAM and a second DRAM.

[0201] FIG. 16 is a diagram illustrating a multitasking scenario in which two artificial intelligence models are executed simultaneously, according to one embodiment.

[0202] In a multitasking scenario in which two artificial intelligence models are executed simultaneously as shown in FIG. 16, a processing method may be applied by distributing and allocating a plurality of artificial intelligence models according to the embodiments of FIG. 10 to 11b to a plurality of DRAMs.

[0203] For example, referring to Fig. 16, a scenario can be assumed in which a user wants to generate text (or complete a sentence) from information entered into an electronic device while simultaneously wanting to change a drawing made on the electronic device into a real photograph.

[0204] The electronic device can split the screen into two areas for multitasking, allowing the user to perform text creation and image editing tasks simultaneously. Within the device, the two tasks can run independently of each other, and the user can check the progress of each task and change settings as needed. The results of image editing and text creation are displayed in each area, and the user can save or share the results as required. For example, the user can save a converted photo, such as a picture, along with the generated text to create a travel record.

[0205] For example, referring to FIG. 16, the electronic device may recognize the text "Fu Bao family relationship" written by a user directly in the input window of a text generator (e.g., dictionary) using a part of the body or an input device (e.g., a stylus pen), or may recognize the text "Fu Bao family relationship" corresponding to the user's voice input "Tell me Fu Bao family relationship" and accept it as a first user input. Additionally, the electronic device may accept an image drawn in the input window of an image-to-photo converter that converts an image into a real photo as a second user input different from the first user input.

[0206] The electronic device can perform distributed processing by allocating DRAMs (e.g., a first DRAM and a second DRAM) on which an artificial intelligence model for the first user input and an artificial intelligence model for the second user input will be executed, based on the artificial intelligence prediction results of the memory management accelerator (MMA) for the first user input and the second user input. For example, if the electronic device is classified, based on the learned prediction model of the memory management accelerator (MMA), that the artificial intelligence model for the first user input requires higher parallel processing performance and bandwidth than the artificial intelligence model for the second user input, it may allocate a second DRAM to the artificial intelligence model for the first user input and allocate a first DRAM to the artificial intelligence model for the second user input.

[0207] For example, when a first user input, such as text input "Fu Bao family relationship," is confirmed, the text "Fu Bao is a baby panda born between mother Ai Bao and father Le Bao" corresponding to the text may be output as a result of execution of the artificial intelligence model allocated to the first DRAM, based on a "text generation (or sentence completion)" command for the text. At the same time, when a second user input, such as an image of a "donut shape," is confirmed, a "donut shape" photo corresponding to the image may be output as a result of execution of the artificial intelligence model allocated to the second DRAM, based on a "photo generation" command for the image.

[0208] The example of multitasking according to the embodiment of FIG. 16 above is intended to aid in understanding the invention, and it will be obvious that various other applications are possible and are not limited to the above-described embodiment.

[0209] FIG. 17a is a diagram showing the arrangement relationship between a processor and memory according to one embodiment. FIG. 17b is a diagram showing the arrangement relationship between a processor and memory according to one embodiment.

[0210] FIGS. 17a and 17b disclose an embodiment that improves the artificial intelligence model processing performance and efficiency of an electronic device by incorporating a 3D stacked memory structure, a neural network processing unit (NPU) built-in memory management accelerator (MMA), intelligent memory distribution technology, and various large-scale language model distributed processing technologies into processing-in-memory (PIM) and neuromorphic technologies according to the embodiments of FIGS. 1 to 16 described above.

[0211] Processing-in-memory (PIM) may be a method of performing operations directly within memory without data movement by embedding simple computation functions in the first memory (530) or the second memory (540). For example, referring to FIG. 17a, an embodiment in which PIM is applied as the second memory (540) is illustrated. By applying PIM, unnecessary data movement between the memory and the processor (e.g., CPU) (520) is eliminated, thereby reducing latency caused by data movement and decreasing power consumption, which can have the effect of improving system performance.

[0212] According to one embodiment, PIM technology can be linked with a memory management accelerator (MMA) to improve the accuracy of memory access pattern analysis and prediction, and to maximize the efficiency of bank allocation optimization. By providing computational information performed in the PIM to the MMA, more accurate memory management can be enabled.

[0213] According to one embodiment, although not separately illustrated, a neural network processing unit (NPU) may be embedded inside a processor (e.g., CPU) (520) according to the embodiment of FIG. 17a, and the 3D stacking structure between the neural network processing unit (NPU) and the first memory and second memory described in FIG. 2 to 4 may also be applied to the embodiment of FIG. 17a.

[0214] Neuromorphic technology of FIG. 17b may also be applied to the 3D stacked structure between the neural network processing unit (NPU) and the first memory and second memory described above in FIG. 2 to 4.

[0215] According to neuromorphic technology, the computational efficiency of large-scale language models can be maximized by developing neuromorphic neural network processing units (NPUs) that mimic the neural network structure of the human brain. Neuromorphic NPUs possess excellent parallel processing and learning capabilities and can operate at low power. Furthermore, neuromorphic technology allows for the simultaneous performance of storage and computation by providing neuromorphic memory (DRAM) equipped with neuron and synapse functions. This can have the effect of reducing memory access latency and decreasing power consumption.

[0216] According to one embodiment, by linking neuromorphic technology with a memory management accelerator (MMA), the accuracy of memory access pattern analysis and prediction can be increased, and the efficiency of bank allocation optimization can be maximized.

[0217] The present disclosure provides various embodiments regarding the arrangement of a neural network processing unit (NPU) and memory (e.g., DRAM), wherein the neural network processing unit (NPU) is placed on the die of a processor (e.g., application processor), and is placed at an edge of the processor (e.g., application processor) that is close to the memory located on the side of the processor (e.g., application processor). Furthermore, various embodiments are disclosed in which a memory (e.g., DRAM) dedicated to the neural network processing unit (NPU) is laid out to reduce time latency between the neural network processing unit (NPU) and the memory (e.g., DRAM) and increase efficiency.

[0218] The present disclosure may disclose a method for efficiently operating two memories (e.g., DRAM) by applying a scheduling and priority processing algorithm between memory dedicated to a neural network processing unit (NPU) and general-purpose memory. In addition, various embodiments may be disclosed for processing a workload by physically quantizing a large-scale language model (LLM) from the learning stage and sequentially loading it to suit the capacity of the memory (e.g., DRAM). Through such embodiments of the present disclosure, various existing problems can be solved, and the performance of an electronic device capable of on-device artificial intelligence processing, which includes a hardware structure for processing an artificial intelligence model internally within the electronic device (200), can be improved.

[0219] Various embodiments of the present disclosure can be utilized in various applications and services that require on-device artificial intelligence processing. For example, they can be effectively used in tasks requiring a large number of parameters, such as image recognition, speech recognition, and natural language processing.

[0220] An electronic device according to one embodiment of the present disclosure may be a device of various forms. An electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. An electronic device according to an embodiment of the present disclosure is not limited to the devices described above.

[0221] The embodiments of the present disclosure and the terms used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In the present disclosure, each of phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationally,” it means that said component may be connected to said other component directly (e.g., wired), wirelessly, or through a third component.

[0222] The term “module” as used in embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0223] One embodiment of the present disclosure may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0224] According to one embodiment, the method according to one embodiment of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0225] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0226] According to one embodiment of the present disclosure, an electronic device (101; 200; 300) may be provided. The electronic device comprises: a printed circuit board (310); a processor (120; 220; 320; 420) disposed on the printed circuit board (310) and having a plurality of processing units embedded therein, including a neural processing unit (2202; 3202; 4202); a first memory (2321; 3321; 4321) stacked on the processor in a package-on-package (PoP) manner; and a second memory (2322; 3322; 4322) which is a high-bandwidth memory (HBM) stacked on the printed circuit board on which the processor is disposed, with one side and side of the processor disposed on the printed circuit board (310) side-by-side, and including a plurality of sub-memories, stacked in a through-silicon via (TSV) manner on the printed circuit board on which the processor is disposed. and may include a memory management accelerator (22021) for optimizing bank allocation to the first memory and the second memory based on at least one of the bank state information of each of the first memory and the second memory, layer information of the artificial intelligence model, computational characteristics of the artificial intelligence model, memory access patterns occurring during the execution of the artificial intelligence model, and data access information of the neural network processing device. The neural network processing device may be connected to the neural network processing device of the processor through a first interface, placed adjacent to the second memory within the processor, and connected to the neural network processing device of the processor through a second interface.

[0227] According to one embodiment, the second interface is an interface configured to enable signal transmission at a relatively higher speed than the first interface, and may include a CXL (computing express link).

[0228] According to one embodiment, the processor may include a plurality of processing units embedded in the processor, such as the neural network processing unit and other processing units (e.g., a central processing unit). The neural network processing unit may be positioned closer to the second memory than the other processing units within the processor.

[0229] According to one embodiment, the processor may have a plurality of processing units disposed on a die comprising four different sides. The neural network processing unit may be disposed on the die adjacent to the side closest to the second memory among the sides included in the four different sides.

[0230] According to one embodiment, the second memory is a base dieless high-bandwidth memory and may further include a first substrate (311) disposed between the processor and the printed circuit board and a second substrate (312) disposed between the second memory and the printed circuit board.

[0231] According to one embodiment, the second memory is a base dieless high-bandwidth memory and may further include a union substrate (313) disposed between the processor and the printed circuit board and between the second memory and the printed circuit board.

[0232] According to one embodiment, the second memory is a base dieless high-bandwidth memory and may further include an interposer (315) disposed between the processor and the printed circuit board and between the second memory and the printed circuit board.

[0233] According to one embodiment, the memory management accelerator may be included in the neural network processing device.

[0234] According to one embodiment, when instructions included in the first memory or the second memory are executed individually or collectively by a processor including the plurality of processing devices, the electronic device may be configured to enable the execution of an artificial intelligence model on the device.

[0235] According to one embodiment of the present disclosure, a method of operating an electronic device (101; 200; 300) may be provided. The method of operating the electronic device may include at least one processor (120; 220; 320; 420) including a memory management accelerator; and a memory (130; 230) for storing instructions, the memory including a first memory (2321; 3321; 4321) and a second memory (2322; 3322; 4322) having a relatively higher bandwidth than the first memory. When the instructions are executed individually or collectively by the at least one processor, the electronic device causes at least one operation to be performed, and the at least one operation is: an operation to determine whether an input artificial intelligence model is a plurality of individuals or corresponds to a multimodal large-scale language model; In the case where the above artificial intelligence model is multiple or corresponds to a multi-modal large-scale language model (multi-modality LLM), the method may include: an operation of performing bank allocation optimization to the first memory and the second memory based on at least one of the following information by the memory management accelerator: bank state information of each of the first memory and the second memory, layer information of the artificial intelligence model, computational characteristics of the artificial intelligence model, memory access patterns occurring during the execution of the artificial intelligence model, and data access information of at least one processor; and an operation of executing the artificial intelligence model allocated to the first memory and the second memory.

[0236] According to one embodiment, after performing bank allocation optimization to the first memory and the second memory, the method may further include an operation to determine the size and location of data blocks to be allocated to each bank based on the bank allocation optimization to the first memory and the second memory.

[0237] According to one embodiment, the operation of performing bank allocation optimization for the first memory and the second memory may be configured to learn data access patterns and bank allocation policies using an AI model inside the memory management accelerator based on new data access information, and to update based on the learned data.

[0238] According to one embodiment, the operation of performing bank allocation optimization to the first memory and the second memory may include an operation of analyzing the usage of the memory and the computational amount of the at least one processor.

[0239] According to one embodiment, when there are multiple artificial intelligence models, the operation of performing bank allocation optimization to the first memory and the second memory can be performed in a state where the artificial intelligence models are distributed in the first memory and the second memory.

[0240] According to one embodiment, when the artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), and when the artificial intelligence model includes computational processing steps with a large amount of computation in the order of inference progress, the second memory may be allocated to a high-priority operation with a relatively large amount of computation, and the first memory may be allocated to a low-priority operation with a relatively small amount of computation.

[0241] For example, if the artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), when the artificial intelligence model includes attention and a feed forward network (FFN), the second memory may be allocated to the attention operation and the first memory may be allocated to the FFN operation.

[0242] According to one embodiment, if the artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), prior to the operation of executing the artificial intelligence model allocated to the first memory and the second memory, the operation of loading the artificial intelligence model from the flash memory by dividing it into the first memory and the second memory may be included.

[0243] According to one embodiment, the operation of loading an artificial intelligence model in a flash memory by dividing it into the first memory and the second memory can be performed using a direct memory access (DMA) method.

[0244] According to one embodiment, when the artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), the operation of performing bank allocation optimization to the first memory and the second memory may include a memory allocation operation for the hidden layer of the artificial intelligence model.

[0245] According to one embodiment, the first memory is stacked on the processor in a package-on-package (PoP) manner, and the second memory is stacked on the printed circuit board on which the processor is placed in a through-silicon via (TSV) manner, including a plurality of sub-memories, and the second memory may be a high-bandwidth memory (HBM).

[0246] According to one embodiment, the second memory may be a processing-in-memory (PIM) or neuromorphic.

[0247] Although specific embodiments have been described in the detailed description of the present disclosure, it will be obvious to those skilled in the art that various modifications are possible within the scope of the present disclosure.

[0248] Although the present disclosure has been described by way of example with respect to one embodiment, it should be understood that the embodiment is for illustrative purposes only and is not intended to limit the present disclosure. It will be obvious to those skilled in the art that various changes in form and detailed configuration may be made without departing from the whole context of the present disclosure, including the appended claims and their equivalents.

Claims

1. In an electronic device (101; 200; 300), Printed circuit board (310); A processor (120; 220; 320; 420) disposed on the printed circuit board (310) and having a plurality of processing units embedded therein, including a neural processing unit (2202; 3202; 4202); A first memory (2321; 3321; 4321) stacked on the above processor in a package-on-package (PoP) manner; A second memory (2322; 3322; 4322), which is a high-bandwidth memory (HBM) stacked via a through-silicon via (TSV) on the printed circuit board (310) on which the processor is disposed, and which includes a plurality of sub-memories, wherein the processor is disposed on the printed circuit board; and It includes a memory management accelerator (22021) for optimizing bank allocation to the first memory and the second memory based on at least one of the bank state information of each of the first memory and the second memory, layer information of the artificial intelligence model, computational characteristics of the artificial intelligence model, memory access patterns occurring during the execution of the artificial intelligence model, and data access information of the neural network processing device. The above neural network processing device is an electronic device connected to the neural network processing device of the processor through a first interface, disposed adjacent to the second memory within the processor, and connected to the neural network processing device of the processor through a second interface.

2. In Paragraph 1, The above second interface is an interface configured to enable signal transmission at a relatively higher speed than the above first interface, and is an electronic device including a CXL (computing express link).

3. In Paragraph 1 or 2, The processor is a plurality of processing units embedded in the processor, including the neural network processing unit and other processing units (e.g., a central processing unit), wherein the neural network processing unit is an electronic device positioned closer to the second memory than the other processing units within the processor.

4. In any one of paragraphs 1 to 3, The electronic device wherein the second memory is a base dieless high-bandwidth memory, and further comprises a first substrate (311) disposed between the processor and the printed circuit board and a second substrate (312) disposed between the second memory and the printed circuit board, or further comprises a union substrate (313) disposed between the processor and the printed circuit board and between the second memory and the printed circuit board, or further comprises an interposer (315) disposed between the processor and the printed circuit board and between the second memory and the printed circuit board.

5. In any one of paragraphs 1 to 4, The above memory management accelerator is an electronic device included in the above neural network processing unit.

6. In any one of paragraphs 1 to 5, When the instructions contained in the first memory or the second memory are executed individually or collectively by a processor including the plurality of processing devices, the electronic device, An electronic device configured to enable the execution of an artificial intelligence model on the device.

7. In a method of operating an electronic device (101; 200; 300), At least one processor (120; 220; 320; 420) including a memory management accelerator; and A memory (130; 230) for storing instructions, comprising a first memory (2321; 3321; 4321) and a second memory (2322; 3322; 4322) having a relatively higher bandwidth than the first memory; and When the above instructions are executed individually or collectively by the at least one processor, they cause the electronic device to perform at least one operation, and The above at least one operation is: An action to determine whether the input artificial intelligence model is multiple individuals or corresponds to a multimodal large-scale language model; In the case where the above artificial intelligence model is multiple or corresponds to a multi-modal large-scale language model (multi-modality LLM), an operation of performing bank allocation optimization to the first memory and the second memory by the memory management accelerator based on at least one of the bank state information of each of the first memory and the second memory, layer information of the artificial intelligence model, computational characteristics of the artificial intelligence model, memory access patterns occurring during the execution of the artificial intelligence model, and data access information of the at least one processor; and An operation to execute an artificial intelligence model allocated to the first memory and the second memory; A method of operation including 8. In Paragraph 7, After performing bank allocation optimization for the first memory and the second memory, an operation to determine the size and location of data blocks to be allocated to each bank based on the bank allocation optimization for the first memory and the second memory. A method of operation that further includes 9. In Paragraph 7 or 8, The operation of performing bank allocation optimization for the first memory and the second memory is configured to learn data access patterns and bank allocation policies using an AI model inside the memory management accelerator based on new data access information, and to be updated based on the learned data. Method of operation.

10. In any one of Articles 7 through 9, The operation of performing bank allocation optimization to the first memory and the second memory comprises analyzing the memory usage and the computational amount of the at least one processor. A method of operation including 11. In any one of Articles 7 through 10, In the case where there are multiple artificial intelligence models, the operation of performing bank allocation optimization to the first memory and the second memory is performed in a state where the artificial intelligence models are distributed in the first memory and the second memory. Method of operation.

12. In any one of Articles 7 through 11, In the case where the above artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), when the above artificial intelligence model includes computational processing steps with a large amount of computation in the order of inference progress, the second memory is allocated to a high-priority operation with a relatively high amount of computation, and the first memory is allocated to a low-priority operation with a relatively low amount of computation. Method of operation.

13. In any one of paragraphs 7 through 12, If the above artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), prior to the operation of executing the artificial intelligence model allocated to the first memory and the second memory, the operation of loading the artificial intelligence model from the flash memory by dividing it into the first memory and the second memory; A method of operation including 14. In Paragraph 13, The operation of loading an artificial intelligence model in flash memory by dividing it into the first memory and the second memory is performed using a DMA (direct memory access) method. Method of operation.

15. In Paragraph 13, If the above artificial intelligence model corresponds to a multi-modal large-scale language model (multi-modality LLM), the operation of performing bank allocation optimization to the first memory and the second memory comprises a memory allocation operation for the hidden layer of the artificial intelligence model; A method of operation including

Images