Artificial intelligence neural network computation apparatus and method

Abstract

The present invention relates to artificial intelligence neural network computation apparatus and method, the apparatus comprising: neural network units connected according to a neural network structure and composed of an encoding unit, a decoding unit or a transformer unit connecting the encoding unit and the decoding unit; line buffers that provide input data for processing images to an input neural network unit from among the neural network units; a first weight buffer that provides first weight data to the encoding unit and the decoding unit; a second weight buffer that provides second weight data to the transformer unit; and an output buffer for storing output data from the processing of the images from an output neural network unit from among the neural network units.

Description

Artificial intelligence neural network computing device and method

[0001] The present invention relates to a neural network hardware structure, and more specifically, to an artificial intelligence neural network computation device and method that minimizes access to off-chip memory through line-by-line neural network computation by implementing a U-net structure with pipelined units.

[0002]

[0003] Camera footage inherently contains noise, which can become severe due to weather conditions such as fog, sea fog, yellow dust, and dust. To address this, AI-based real-time ultra-high-speed noise removal technology may be essential. However, real-time processing can be very difficult with existing acceleration technologies because it requires a massive amount of computation and memory access.

[0004] U-net is a representative Encoder-Decoder neural network model proposed for medical image segmentation. Because U-net’s multi-layer structure enables feature extraction from multiple scales, it can be considered a representative neural network that demonstrates excellent performance in noise removal. U-net can be composed of an encoding part containing CNN operations, a decoding part, and a skip-connection part that transmits the encoding results to the decoding part. This configuration enables the restoration of images damaged by noise by preserving detailed image information.

[0005]

[0006] [Prior Art Literature]

[0007] [Patent Literature]

[0008] Korean Registered Patent No. 10-2256288 (May 20, 2021)

[0009]

[0010] One embodiment of the present invention aims to provide an artificial intelligence neural network computing device and method capable of improving real-time noise removal quality by processing data exchange with system memory and controlling the operation of neural network units.

[0011]

[0012] Among the embodiments, the artificial intelligence neural network computing device comprises: neural network units connected according to a neural network structure and composed of an encoding unit, a decoding unit, or a transformer unit connecting the encoding unit and the decoding unit; line buffers providing input data for image processing to an input neural network unit among the neural network units; a first weight buffer providing first weight data to the encoding unit and the decoding unit; a second weight buffer providing second weight data to the transformer unit; and an output buffer storing output data according to image processing from an output neural network unit among the neural network units.

[0013] It may further include a controller that processes data exchange with system memory and controls the operation of the neural network units.

[0014] The above neural network units have an encoder-transformer-decoder structure and may include: an encoding unit that processes an encoding process for compressing the input data into encoding data; a transformer unit that processes a transforming process for converting the encoding data into transforming data; a decoding unit that processes a decoding process for restoring the transforming data and outputs the output data; a skip connection buffer unit (CatBuf) that stores a connection feature map for skip connections for the encoding unit, the transformer unit, and the decoding unit; and a bottleneck unit that connects the encoding units and the decoding units.

[0015] The above encoding unit may include: a first shifter that receives input data from the line buffer and performs a shift operation; a first weight register that receives the first weight data from the weight buffer; and a first convolution unit that performs a convolution operation with the input data on which the shift operation was performed and the first weight data to output encoded output data, and stores the feature map finally generated during the process of the convolution operation as the connected feature map in the skip connected buffer unit.

[0016] The above encoding unit may further include a pool buffer that performs downsampling on the above encoding output data.

[0017] The transformer unit may include: an attention calculation unit that receives a connection feature map from the skip connection buffer unit and calculates attention for the connection feature map based on attention weight data among the second weight data; and a second convolution unit that performs a convolution operation with the attention and convolution weight data among the second weight data to output transforming output data.

[0018] The above encoding unit may further include a second shifter that receives the encoding output data from the above encoding unit and performs a shift operation; and a second weight register that receives the second weight data from the weight buffer (WeightBuf).

[0019] Each of the above-mentioned encoding units may include: a third shifter that receives the transforming output data from the transformer unit and performs a shift operation; a third weight register that receives the weight data from the weight buffer; and a third convolution unit that performs a convolution operation with the transforming output data and the first weight data to output the decoding output data and merges the connection feature map with the decoding output data.

[0020] The skip connection buffer unit directly connects the encoding unit and the decoding unit among the neural network units and can be connected continuously in a number independently calculated according to the neural network depth.

[0021] The above controller can control access to the system memory in the stage before the operation of the neural network units begins to read input data and weight data for the operation of the neural network units, and access to the system memory in the stage after the operation of the neural network units ends to store output data according to the operation of the neural network units.

[0022] Among the embodiments, the artificial intelligence neural network operation method comprises the steps of: providing input data for image processing to an input neural network unit among the neural network units in a line buffer; providing first weight data to an encoding unit and a decoding unit in a weight buffer; providing second weight data to a transformer unit; and storing output data according to image processing from an output neural network unit among the neural network units in an output buffer.

[0023] It may include neural network units connected according to a neural network structure and composed of an encoding unit, a decoding unit, or a transformer unit connecting the encoding unit and the decoding unit, wherein each of the neural network units has an encoder-transformer-decoder structure and includes: an encoding unit that processes an encoding process of compressing input data into encoding data; a transformer unit that processes a transforming process of converting the encoding data into transforming data; a decoding unit that processes a decoding process of restoring the transforming data to output output data; a skip connection buffer unit (CatBuf) that stores a connection feature map for skip connections for the encoding unit, the transformer unit, and the decoding unit; and a bottleneck unit that connects the encoding units and the decoding units.

[0024]

[0025] The disclosed technology may have the following effects. However, this does not mean that a specific embodiment must include all of the following effects or only the following effects; therefore, the scope of the rights of the disclosed technology should not be understood as being limited by this.

[0026] An artificial intelligence neural network computation device and method according to one embodiment of the present invention can improve real-time noise removal quality by processing data exchange with system memory and controlling the operation of neural network units.

[0027]

[0028] FIG. 1 is a diagram illustrating an artificial intelligence neural network computing device according to the present invention.

[0029] Figure 2 is a diagram illustrating the structure of a U-net neural network.

[0030] Figure 3 is a diagram illustrating the block diagram of each neural network unit of Figure 1.

[0031] FIG. 4 is a drawing illustrating a block diagram relating to the hardware structure of an encoding unit according to the present invention.

[0032] FIG. 5 is a diagram illustrating a block diagram regarding the hardware structure of a decoding unit according to the present invention.

[0033] Figure 6 is a diagram illustrating the hardware structure block diagram of a transformer unit.

[0034]

[0035] The description of the present invention is merely an example for structural or functional explanation, and therefore the scope of the present invention should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific example must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0036] Meanwhile, the meaning of the terms described in this application should be understood as follows.

[0037] Terms such as "first," "second," etc., are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0038] When it is stated that one component is "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when it is stated that one component is "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," should be interpreted in the same way.

[0039] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the implemented features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0040] In each step, identifiers (e.g., a, b, c, etc.) are used for convenience of explanation and do not describe the order of the steps; the steps may occur differently from the specified order unless a specific order is clearly indicated in the context. That is, the steps may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

[0041] The present invention may be implemented as computer-readable code on a computer-readable recording medium, and the computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, the computer-readable recording medium may be distributed across networked computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0042] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this application.

[0043]

[0044] FIG. 1 is a diagram illustrating an artificial intelligence neural network computing device according to the present invention.

[0045] Referring to FIG. 1, an artificial intelligence neural network computing device may include a host processor (110), system memory (120), a neural network core (130), a neural network unit (131), a line buffer (132), a weight buffer (133), an output buffer (134), a controller (135), an encoding unit (136), a decoding unit (137), a skip connection buffer unit (138), a bottleneck unit (139), and a transformer unit (140).

[0046] At this time, embodiments of the present invention are not required to include all of the above components simultaneously; depending on each embodiment, some of the components may be omitted, or some or all of the components may be selectively included. The operation of each component will be described in detail below.

[0047]

[0048] Neural network units (131) may be composed of an encoding unit (136), a decoding unit (137), or a transformer unit (140) capable of connecting the encoding unit (136) and the decoding unit (137) according to a neural network structure. Here, the neural network units (131) may be connected to each other according to a specific neural network structure, for example, according to a fully connected, convolutional, and recurrent neural network structure. Additionally, the neural network units (131) may further include a transformer unit (140) that receives data generated from the encoding unit (136), performs image processing operations, and transmits the data to the decoding unit (137).

[0049] The controller (135) can handle data exchange with the system memory (120) and control the operation of the neural network units (131). Here, the controller (135) can load input data from the system memory (120) and provide it to each neural network unit (131). Additionally, the controller (135) can perform the process of processing input data by loading weight data from the system memory (120) and transmitting it to the neural network unit (131).

[0050] In one embodiment, the controller (135) can control access to the system memory (120) before the operation of the neural network units (131) begins to read input data and weight data for the operation of the neural network units (131), and access to the system memory (120) after the operation of the neural network units (131) ends to store output data according to the operation of the neural network units (131). Here, the controller (135) can perform a preparation process to enable the neural network unit (131) to perform image processing for the assigned segmented image by transmitting the input data and weight data from the system memory (120) to the neural network unit (131) through the reading process. Then, the controller (135) can store the output data generated from the neural network unit (131) in the system memory (120).

[0051] In one embodiment, the neural network units (131) may have an encoder-transformer-decoder structure. Here, the encoder-transformer-decoder structure may correspond to a neural network architecture that goes through three stages of encoder-transformer-decoder from data input to output generation. The neural network units (131) may receive original input data through the encoder unit (136) and perform data representation optimization by extracting important patterns of the input data, including color, shape, and boundaries. Additionally, the neural network units (131) may perform a process of converting compressed data generated by the encoder through the transformer unit (140) and process specific parts within the input data more intensively. The neural network units (131) may receive the converted data transformed by the transformer unit (140) through the decoder unit (137) and restore it to the original data form.

[0052] In one embodiment, the neural network units (131) may include an encoding unit (136) that processes an encoding process for compressing input data into encoding data, and a transformer unit (140) that processes a transforming process for converting the encoding data into transforming data. The neural network units (131) can process the original input data through the encoding unit (136) to convert it into compressed encoding data. Additionally, the neural network units (131) can generate transforming data by applying additional weights to the encoding data through the transformer unit (140) to reconstruct the data and enhance the meaning of the encoding data.

[0053] In one embodiment, the neural network units (131) may include a decoding unit (137) that processes a decoding process to restore transformed data and outputs output data. Here, the neural network units (131) may expand the encoded data through the decoding unit (137) and convert it into a final output form. For example, the neural network units (131) may perform an upsampling or deconvolution operation on the encoded data through the decoding unit (137) and restore the data to the resolution and dimensions of the initial input. Here, upsampling is a process of expanding the data to a higher resolution to make it close to the original size, and may include upsampling techniques such as Nearest Neighbor Upsampling, Bilinear Interpolation, and Bicubic Interpolation. Additionally, the deconvolution operation may correspond to the inverse operation of the convolution operation. Neural network units (131) can restore key features while expanding the resolution of an encoded image to a given size by performing deconvolution operations such as padding and stride adjustment based on decoding units (137).

[0054] In one embodiment, the neural network units (131) may include a skip connection buffer unit (138) that stores a connection feature map for skip connections to an encoding unit (136), a transformer unit (140), and a decoding unit (137), and a bottleneck unit (139) that connects the encoding units (136) and the decoding units (137). Here, the skip connection buffer unit (138) performs the role of transmitting some feature information generated during the encoding process to the decoding process via skip connections, and can temporarily store the connection feature map and provide the corresponding data to the decoding unit (137) when skip connections are required. Additionally, the bottleneck unit (139) may be composed of the lowest dimension layers in the neural network and may be composed of CNN layers between the last downsampling and the first upsampling. Thus, the bottleneck unit (139) may have the characteristic of including high-level features. The neural network units (131) can perform an image restoration process that minimizes image data loss and reflects key features of the image by including a skip connection buffer unit (138) and a bottleneck unit (139).

[0055] In one embodiment, the encoding unit (136) may include a first shifter (1361) that receives input data from a line buffer (132) and performs a shift operation. Here, the line buffer (132) may correspond to a device that temporarily stores specific segmented image data of an image. The encoding unit (136) receives input data from the line buffer (132) and can adjust or filter the position of the data during the encoding process through the first shifter (1361).

[0056] In one embodiment, the encoding unit (136) may include a first weight register (1362) and a first temporary buffer (1363) that receive first weight data from a weight buffer (133). Here, the encoding unit (136) receives the first weight data through the first weight register (1362) and may apply specific operations, such as a convolution operation, a linear transformation, and an attention mechanism, to the input data during the encoding process. Additionally, the encoding unit (136) may store intermediate data generated during the encoding process through the first temporary buffer (1363) and utilize it in the next encoding process.

[0057] In one embodiment, the encoding unit (136) can perform a convolution operation with the input data on which a shift operation has been performed and the first weight data to output encoded output data. Here, the encoding unit (136) can store the data derived as a result of the convolution operation as an intermediate feature map. Through this, the encoding unit (136) can reuse the data in the next operation step to increase operation efficiency and maintain feature information of the image data.

[0058] In one embodiment, the encoding unit (136) may include a first convolution unit (1364) that stores an intermediate feature map temporarily generated during the process of convolution operations in a temporary buffer (1363) and stores a finally generated feature map as a connected feature map in a skip connected buffer unit (138). For example, the encoding units (136) may store an intermediate feature map generated during the process of performing a plurality of convolution operations through the first convolution unit in a temporary buffer and reuse it in the next operation process. The encoding units (136) may generate a final feature map summarizing key features extracted from input data after the completion of the convolution operations. Here, the encoding units (136) may store the final feature map in the skip connected buffer unit (138).

[0059] In one embodiment, the encoding unit (136) may further include a pool buffer (1365) for performing downsampling on the encoding output data. Here, downsampling may correspond to a technique of reducing data to retain only important features from the original information and reduce unnecessary parts. Each of the encoding units (136) may perform downsampling on the encoding output data through the pool buffer (1365) and temporarily store the summarized data during the downsampling process. For example, when the encoding unit (136) generates 4x4 size encoding output data, it may perform 2x2 size max pooling on the encoding output data to reduce the size to 2x2 and store the reduced data in the pool buffer (1365).

[0060] In one embodiment, the transformer unit (140) may include an attention calculation unit (1401) that receives a connection feature map from a skip connection buffer unit (138) and calculates attention for the connection feature map based on attention weight data among the second weight data. Here, the transformer unit (140) may assign higher weights to the main features of the input data using an attention mechanism through the attention calculation unit (1401). For example, the transformer unit (140) may evaluate the importance of the connection feature map through the attention calculation unit (1401), weaken unnecessary information, and strengthen necessary information so that the transformer unit can process the data more effectively.

[0061] In one embodiment, the transformer unit (140) may include a second convolution unit (1402) that performs a convolution operation with the attention and the convolution weight data among the second weight data to output transforming output data. Here, the transformer unit (140) may perform a convolution operation using the attention value calculated by the attention calculation unit (1401) and the convolution weight data among the second weight data through the second convolution unit (1402) and finally generate transforming output data.

[0062] In one embodiment, the encoding unit (136) may further include a second shifter (1361) that receives encoding output data from the encoding unit (136) and performs a shift operation, a second weight register (1362) that receives second weight data from the weight buffer (133), and a second temporary buffer (1363). Here, the encoding unit may perform a shift operation on the encoding output data through the second shifter (1361) and transform the data or adjust a specific pattern during the encoding process. Additionally, the encoding unit (136) may perform a process of combining the second weight data and input data and transforming the data through the second weight register (1362). The encoding unit (136) may temporarily store intermediate data generated during the operation in the encoding process through the second temporary buffer (1363), and the stored intermediate data may be removed when the encoding unit (136) generates the final transforming output data.

[0063] In one embodiment, each of the decoding units (137) may include a third shifter (1371) that receives transforming output data from the transformer unit (140) and performs a shift operation, and a third weight register (1372) that receives weight data from the weight buffer (133). Here, the decoding unit (137) can emphasize specific features of the data by transforming the position of the transforming output data by performing a shift operation on the transforming output data. Additionally, the decoding unit (137) may receive weight data from the weight buffer (133) and provide filtering information to be applied when generating the decoding output data.

[0064] In one embodiment, each of the decoding units (137) may include a third convolution unit (1374) that performs a convolution operation with a third temporary buffer (1373), transforming output data, and first weight data to output decoding output data, merges the connected feature map and the decoding output data, and stores an intermediate feature map temporarily generated during the convolution operation in the temporary buffer (1373). Here, the decoding unit (137) can store the intermediate feature map in the temporary buffer through the third convolution unit and utilize it in the next step. For example, the decoding unit (137) can store an intermediate feature map containing edge information or color distribution, etc., in the third temporary buffer (1373) during the process of restoring an image, and utilize the said intermediate feature map in the next step to improve the restoration accuracy.

[0065] In one embodiment, the skip connection buffer unit (138) directly connects the encoding unit (136) and the decoding unit (137) among the neural network units (131) and can be connected consecutively in a number independently calculated according to the neural network depth. Here, the skip connection buffer unit (138) can store a feature map extracted from the encoding unit (136) and connect it so that the decoding unit (137) can directly reference it. For example, the skip connection buffer unit (138) can calculate independent feature maps of the encoding unit (136) and the decoding unit (137) according to the neural network depth, perform connections step by step, and transmit them to the decoding unit (137). For example, the skip connection buffer unit (138) can restore original information by transmitting the output feature map of the encoding unit (137) to the decoding unit (137) when data reduced to a small size through the encoding unit (136) in a neural network such as U-Net is restored during the decoding stage. Here, the number of skip connection buffer units (138) can be determined according to the following mathematical formula 1.

[0066] [Mathematical Formula 1]

[0067]

[0068]

[0069] Here, D may correspond to the number of downsampling cycles, that is, the number of encoding units (136). n may represent the number of each encoding unit (136) and decoding unit (137), and T may represent the number of transformer units (140) to be processed in the skip-connection stage. For example, in FIG. 1, since downsampling is performed twice, D is 2, and since there is one transformer unit (140) between the EN 0 Unit and the De 0 Unit, T is 1. At this time, the size of the skip-connection buffer unit (138) of the En / De 0 Unit is calculated as "2(2-0)-1" and becomes 3, and the size of the skip-connection buffer unit (138) of the En / De 1 Unit is calculated as "2(2-1)-1" and becomes 1.

[0070] Line buffers (132) can provide input data for image processing to an input neural network unit (131) among the neural network units (131). Here, a line buffer (132) may correspond to a device that provides input data for processing a segmented image to an input neural network unit among the neural network units (131). The line buffers (132) can segment image data, temporarily store it, and provide the data in the order required by the neural network unit (131).

[0071] The first weight buffer (1331) can provide first weight data to the encoding unit (136) and the decoding unit (137), and the second weight buffer (1332) can provide second weight data to the transformer unit (140).

[0072] The output buffer (134) can store output data according to image processing from the output neural network unit (131) among the neural network units (131). Here, the output buffer (134) may correspond to a device that stores output data according to image processing of the assigned segmentation image from the output neural network unit among the neural network units (131).

[0073]

[0074] Figure 2 is a diagram illustrating the structure of a U-net neural network.

[0075] Referring to Fig. 2, the U-net can be composed of two main parts. One may correspond to a 'contracting path' or 'encoding path' that extracts features from an image, and the other may correspond to an 'expanding path' or 'decoding path' that generates an output with the same resolution as the original image based on the extracted features.

[0076] The encoder can perform downsampling for the purpose of capturing context and may include multiple convolution layers and max pooling layers similar to a general CNN. In FIG. 2, the encoder may be composed of four encoding blocks (En 0, En 1, ..., En 3), and one encoding block may be composed of three CNN layers and one downsampling step.

[0077] Additionally, the decoder can perform upsampling for fine localization purposes and may include multiple upsampling layers and CNN layers symmetrically with the encoder. In FIG. 2, the decoder may be composed of four decoding blocks (De 0, De 1, ..., De 3), and one decoding block may be composed of three CNN layers and one upsampling step.

[0078] Skip connections can serve to link encoding blocks and symmetric decoding blocks. Through these connections, learning efficiency can be improved by integrating coarse information from the reduced path into the decoding process.

[0079] A bottleneck in a neural network can consist of the lowest-dimensional layers. That is, a bottleneck can be composed of the CNN layers between the last downsampling and the first upsampling. Therefore, a bottleneck can be characterized by containing high-level features.

[0080] The structure of U-Net allows for the acquisition of high-dimensional features through a deep network while effectively preserving fine-grained features (detailed and specific levels of information or structure) of images, thereby enabling pixel-level detailed predictions. U-Net's powerful feature extraction capabilities and ability to generate high-resolution outputs can be highly advantageous for restoring original images from noisy ones.

[0081]

[0082] FIG. 3 is a diagram illustrating a block diagram of each neural network unit of FIG. 1, FIG. 4 is a diagram illustrating a block diagram regarding the hardware structure of an encoding unit according to the present invention, and FIG. 5 is a diagram illustrating a block diagram regarding the hardware structure of a decoding unit according to the present invention.

[0083] Referring to FIG. 3, an artificial intelligence neural network computing device may include neural network units (131) connected according to a neural network structure to implement the operation of a neural network. At this time, the neural network units (131) may include an encoding unit (136), a decoding unit (137), a bottleneck unit (139), and an FC unit. Each unit constituting the neural network may be configured to include basic modules for convolution operations, and the detailed configuration may vary slightly depending on the role of each unit. For example, units including downsampling, residual connection, or concatenation may be implemented with different configurations depending on the detailed function.

[0084] The basis structure commonly applied to each unit may include shifter, convolution unit, weight buffer (133), and temporary buffer modules. Specifically, in the shifter module, a shift operation on the input data may be performed for convolution operations. At this time, the 2D convolution operation may be performed by the filters sliding the input feature map in pixel units.

[0085] Additionally, the weight buffer (133) can receive and store all weights used by each unit from the host processor (110). The weight register module can receive weights required for convolution operations from the weight buffer module. The temporary buffer module can store intermediate feature maps generated from the remaining layers, excluding the last convolution layer of each unit. In the convolution unit module, the actual convolution operations of the CNN layers included in each unit can be performed sequentially. In the convolution unit module, PEs can be configured to be optimized for layer configuration.

[0086] Referring to FIG. 4, each of the encoding units (136) can perform downsampling, and a convolution unit module can generate a concatenation feature map. That is, each encoding unit (136) may include a pool buffer module for downsampling based on the basic structure of FIG. 3. That is, the pool buffer module can perform downsampling on the result of the convolution operation performed by the convolution unit module. In one embodiment, each encoding unit (136) may be implemented to be connected to a skip concatenation buffer unit (138) that stores the concatenation feature map.

[0087] Referring to FIG. 5, each decoding unit (137) may be connected to a skip connection buffer unit (138) that stores a connection feature map to receive the connection feature map. Meanwhile, the bottleneck unit (139) and the FC unit (FC Unit) may be composed only of convolution operations, and thus may be implemented with the same structure as the basic structure of FIG. 3.

[0088]

[0089] Figure 6 is a diagram illustrating the hardware structure block diagram of a transformer unit.

[0090] Referring to FIG. 6, a hardware structure block diagram of the transformer units (140) (Transformer 0, Transformer 1, Transformer 2 Unit) is illustrated. Here, the hardware structure block diagram of the transformer units (140) is based on the block diagram of the neural network unit (131) of FIG. 3, with the attention calculation unit (1401), the first weight register (1362), and the third weight register (1372) newly added. Here, the attention calculation unit (1401) is a module responsible for the attention weight calculation in the transformer unit (140). The first weight register (1362) and the second weight register are modules that store the weights required for the calculation of the attention calculation unit (1401) and the second convolution unit (1372), respectively. The operation method of the modules constituting the transformer unit (140) is as follows.

[0091] First, the first weight buffer (1331) receives and stores all weights used in the encoding unit (136), transformer unit (140), and decoding unit (137) from the host processor (110). The first weight register (1362) receives and stores the weights required by the attention calculation unit (1401) from the weight buffer (133). The third weight register (1372) receives and stores the weights required by the third convolution unit (1374) from the weight buffer (133). Here, the transformer unit (140) can receive the result of the previous unit and perform a shift operation on the input data through the first shifter (1361). Subsequently, the attention calculation unit (1401) performs an attention numerical operation on the input and transmits the result to the third convolution unit (1374). The third convolution unit (1374) sequentially performs complex convolution operations on the received input and temporarily stores the intermediate feature maps generated during the operation process (intermediate layers) in the third temporary buffer (1373). Finally, when the operation for all layers constituting the transformer unit (140) is completed, the final result is transmitted to the next unit.

[0092]

[0093] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0094]

[0095] [Project ID] 2370000151

[0096] [Assignment No.] 00351961

[0097] [Ministry Name] Ministry of Culture, Sports and Tourism

[0098] [Name of Project Management (Specialized) Agency] Korea Content Agency

[0099] [Research Project Name] Cultural Technology Research and Development (R&D)

[0100] [Research Project Title] Development of AI-based Content Creation Technology for the Advancement of Volumetric Studio and Virtual Production

[0101] [Contribution Rate] 50%

[0102] [Name of Project Performing Organization] X-On Studio Co., Ltd.

[0103] [Research Period] 2025.01.01 ~ 2025.12.31

[0104]

[0105] [Project ID] 2710085862

[0106] [Assignment No.] 02315892

[0107] [Ministry Name] Ministry of Science and ICT

[0108] [Name of Project Management (Specialized) Agency] Korea Institute for Science and Technology Commercialization

[0109] [Research Project Name: Promotion of University Technology Management (IP Star Scientist Support Type)

[0110] [Project Title] IP Advancement and Commercialization for the Promotion of Commercialization of Real-time Denoising AI Hardware Technology

[0111] [Contribution Rate] 50%

[0112] [Name of Project Performing Organization] Sejong University Industry-Academic Cooperation Foundation

[0113] [Research Period] April 1, 2025 ~ December 31, 2026

[0114]

[0115] [Explanation of the symbol]

[0116] 100: Artificial intelligence neural network computing unit

[0117] 110: Host Processor 120: System Memory

[0118] 130: Neural Network Core

[0119] 131: Neural network unit 132: Line buffer

[0120] 133: Weight buffer 1331: First weight buffer

[0121] 1332: Second weight buffer 134: Output buffer

[0122] 135: Controller 136: Encoding Unit

[0123] 1361: 1st and 2nd shifters 1362: 1st and 2nd weight registers

[0124] 1363: 1st and 2nd temporary buffers 1364: 1st convolution unit

[0125] 1365: Pool buffer

[0126] 137: Decoding Unit 1371: Third Shifter

[0127] 1372: 3rd weight register 1373: 3rd temporary buffer

[0128] 1374: Third Convolution Unit

[0129] 138: Skip connection buffer unit

[0130] 139: Bottleneck Unit 140: Transformer Unit

[0131] 1401: Attention Calculator 1402: Second Convolution Unit

Claims

1. Neural network units connected according to a neural network structure and composed of an encoding unit, a decoding unit, or a transformer unit connecting the encoding unit and the decoding unit; Line buffers that provide input data for image processing to the input neural network unit among the above neural network units; A first weight buffer that provides first weight data to the encoding unit and the decoding unit; A second weight buffer that provides second weight data to the above transformer unit; An artificial intelligence neural network computing device comprising an output buffer that stores output data according to the processing of the image from an output neural network unit among the above neural network units.

2. In Paragraph 1, An artificial intelligence neural network computing device characterized by further including a controller that processes data exchange with system memory and controls the operation of the neural network units.

3. In paragraph 1, the neural network units It has an encoder-transformer-decoder structure, and An encoding unit that processes an encoding process to compress the above input data into encoded data; A transformer unit that processes a transforming process for converting the above-mentioned encoding data into transforming data; A decoding unit that processes a decoding process to restore the above-mentioned transforming data and outputs the above-mentioned output data; A skip connection buffer unit (CatBuf) storing a connection feature map for skip connections for the encoding unit, the transformer unit, and the decoding unit; and An artificial intelligence neural network computing device characterized by including a bottleneck unit connecting the above-mentioned encoding units and decoding units.

4. In paragraph 3, the encoding unit A first shifter that receives input data from the line buffer and performs a shift operation; A first weight register (WeightReg) that receives the first weight data from the weight buffer (WeightBuf); An artificial intelligence neural network computation device characterized by including a first convolution unit that performs a convolution operation with the input data on which the shift operation is performed and the first weight data to output encoded output data, and stores the feature map finally generated during the process of the convolution operation as the connected feature map in the skip connection buffer unit.

5. In paragraph 4, the encoding unit An artificial intelligence neural network computing device characterized by further including a pool buffer that performs downsampling on the above-mentioned encoded output data.

6. In paragraph 3, the transformer unit An attention calculation unit that receives a connection feature map from the skip connection buffer unit and calculates attention for the connection feature map based on attention weight data among the second weight data; and An artificial intelligence neural network computation device characterized by including a second convolution unit that performs a convolution operation with the attention and the convolution weight data among the second weight data to output transforming output data.

7. In paragraph 6, the above encoding unit A second shifter that receives the encoding output data from the encoding unit and performs a shift operation; and A second weight register (WeightReg) that receives the second weight data from the weight buffer (WeightBuf); An artificial intelligence neural network computing device characterized by further including 8. In paragraph 6, each of the above-mentioned encoding units is A third shifter that receives transforming output data from the transformer unit and performs a shift operation; A third weight register (WeightReg) that receives weight data from the weight buffer (WeightBuf); An artificial intelligence neural network computation device characterized by performing a convolution operation with the transforming output data and the first weight data to output a decoding output data, merging the connection feature map and the decoding output data, and including a third convolution unit.

9. In paragraph 3, the skip connection buffer unit is An artificial intelligence neural network computing device characterized by directly connecting the encoding unit and the decoding unit among the above neural network units and continuously connecting them in a number independently calculated according to the neural network depth.

10. In paragraph 2, the controller An artificial intelligence neural network computing device characterized by controlling access to the system memory to read input data and weight data for the operation of the neural network units in a stage prior to the start of operation of the neural network units, and access to the system memory to store output data according to the operation of the neural network units in a stage prior to the end of operation of the neural network units.

11. A step of forming neural network units, comprising an encoding unit, a decoding unit, or a transformer unit connecting the encoding unit and the decoding unit, into a neural network structure; A step of providing input data for image processing to an input neural network unit among the above neural network units in a line buffer; A step of providing first weight data to the encoding unit and the decoding unit in a weight buffer; The step of providing second weight data to the above transformer unit; and An artificial intelligence neural network operation method comprising the step of storing output data according to the processing of the image from an output neural network unit among the above neural network units in an output buffer.

12. Neural network units comprising an encoding unit, a decoding unit, or a transformer unit connecting the encoding unit and the decoding unit, which are connected according to a neural network structure. Each of the above neural network units is It has an encoder-transformer-decoder structure, and An encoding unit that processes the encoding process of compressing input data into encoded data; A transformer unit that processes a transforming process for converting the above-mentioned encoding data into transforming data; A decoding unit that processes a decoding process to restore the above-mentioned transforming data and outputs output data; A skip connection buffer unit (CatBuf) storing a connection feature map for skip connections for the encoding unit, the transformer unit, and the decoding unit; and An artificial intelligence neural network computing device including a bottleneck unit connecting the above encoding units and decoding units.

Images