Method and apparatus for accelerating artificial intelligence computation using general-purpose artificial intelligence acceleration-dedicated architecture
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-11-11
- Publication Date
- 2026-08-13
Smart Images
Figure KR2025018488_13082026_PF_FP_ABST
Abstract
Description
Method and apparatus for accelerating artificial intelligence computation including a dedicated architecture for general-purpose artificial intelligence acceleration
[0001] The present invention relates to a method and apparatus for accelerating artificial intelligence computation including a general-purpose artificial intelligence acceleration dedicated architecture. More specifically, it relates to a dedicated hardware architecture for accelerating high-efficiency, high-speed artificial intelligence computation, and to a method and apparatus for performing artificial intelligence acceleration computation including mega-cores and mini-cores.
[0002] Artificial intelligence is currently being utilized in various fields such as speech recognition and computer vision. With the development of deep neural networks, early artificial intelligence replaced simple tasks such as pattern recognition and machine language translation, which were performed based on algorithms or programming created directly by humans based on mathematical models, and this can be classified as Discriminative AI.
[0003] Today, as artificial intelligence algorithms have steadily advanced, they have evolved into Generative AI capable of creating new content, as well as simple tasks such as object classification and data analysis.
[0004] Generative AI is evaluated as a technology that can provide people with new perspectives through creative answers and creations, and drastically reduce the cost and time required for content production.
[0005] Artificial intelligence, which can be classified into discriminative AI and generative AI, is subject to different inference requirements depending on its type. For instance, discriminative AI must perform the task of classifying data into one of predefined categories within a very short inference latency of less than 33ms, whereas generative AI allows for the creation of high-quality audio, images, or videos while tolerating long latencies of over one minute. In particular, generative AI, which demands high quality, also differs from discriminative AI in terms of quantization performance. While both types of AI do not experience significant performance degradation even when weights are quantized to a precision of 4 bits or less, generative AI, which requires high quality, demands high bit precision for the input activations (IA) that occur during AI computation. Even discriminative AI was not without its problems in existing quantization methods. Quantization adopts a bit count and quantization that minimizes accuracy loss in the collected dataset; however, the actual environment in which the user utilizes the AI may have data distributions that were not identified in the collected dataset. This ultimately leads to a problem where the AI's performance cannot be guaranteed for cases that were not fully verified in the dataset. In particular, the method utilizing fixed-point numbers, which limits the data range in advance and performs quantization, is highly vulnerable to accuracy degradation issues despite having higher hardware efficiency compared to floating-point numbers, which dynamically adjust the data range using an exponent.
[0006] To solve this problem, one could adopt an FP-FXP hybrid input representation method that separates input values into Inlier and Outlier data, representing Inlier data as Fixed-point (FXP) and Outlier data as Floating-point (FP); however, existing AI acceleration processors could not efficiently support a form where fixed-point and floating-point-based data representation methods are mixed. Furthermore, hardware that supports both fixed-point and floating-point arithmetic units has a problem in that it cannot achieve peak performance due to very low core utilization.
[0007] In order to efficiently support not only discriminative AI but also generative AI without performance degradation, a hardware architecture is required that supports not only the existing fixed-point method but also the FP-FXP hybrid input value representation method.
[0008] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered technology that was known to the general public prior to the filing of the present invention.
[0009] The problem to be solved through the disclosure of the present invention is to efficiently support an FP-FXP hybrid input value representation method by utilizing two heterogeneous cores: a large accelerator MEGA core specialized for fixed-point operations and a small accelerator mini core specialized for floating-point operations.
[0010] The problem to be solved by the disclosure of the present invention is a method performed by an artificial intelligence acceleration device comprising: a step of loading data from at least one user terminal; a step of preprocessing the loaded data; a step of performing a first operation process for an artificial intelligence model on the preprocessed data; a step of performing a second operation process for an artificial intelligence model on the preprocessed data; and a step of calculating the operation efficiency and providing the operation efficiency to the user terminal after the first operation and the second operation process. The artificial intelligence model may include a mini-core architecture that utilizes input sparsity instead of integrating small-scale computing units, utilizing mega-cores and floating-point numbers with increased data reuse rates.
[0011] In one embodiment, the artificial intelligence model may be characterized by integrating a cross-shaped memory architecture for artificial intelligence weights and input data caching between multi-cores.
[0012] In one embodiment, the artificial intelligence model may be characterized by adopting a sparse matrix compression method to minimize memory usage.
[0013] In one embodiment, the artificial intelligence model may be characterized by adopting a sparsity utilization design method in which outlier data is padded with 0 and computation is performed on a MEGA core, and the mini core can perform computation by excluding input '0' in accordance with the sparsity data characteristics.
[0014] In one embodiment, the artificial intelligence model may include a cross-shaped memory architecture to minimize the amount of data read and write when multiple mega cores and mini cores are operated in a single processor.
[0015] In one embodiment, the step of providing computational efficiency to a user terminal may involve supporting an FP-FXP hybrid input value representation method and providing data regarding computational efficiency to the user terminal.
[0016] According to the means for solving the problem of the present invention described above, by utilizing two heterogeneous cores—a large accelerator MEGA core specialized for fixed-point operations and a small accelerator mini core specialized for floating-point operations—it is possible to efficiently support the FP-FXP hybrid input value representation method.
[0017] FIG. 1 illustrates an exemplary environment in which an artificial intelligence acceleration device according to some embodiments of the present disclosure may be applied.
[0018] FIG. 2 is a flowchart relating to certain operations for artificial intelligence model acceleration that can be performed in an artificial intelligence acceleration device according to some embodiments of the present disclosure.
[0019] FIGS. 3 to 11 are drawings for illustrating artificial intelligence acceleration operations according to some embodiments of the present disclosure.
[0020] FIG. 12 is a drawing of an exemplary computing device capable of implementing a device and / or system according to various embodiments of the present disclosure.
[0021] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the technical concept of the present disclosure is not limited to the following embodiments but can be implemented in various different forms. The following embodiments are provided merely to complete the technical concept of the present disclosure and to fully inform those skilled in the art of the scope of the present disclosure, and the technical concept of the present disclosure is defined only by the scope of the claims.
[0022] It should be noted that when assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.
[0023] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing the embodiments and are not intended to limit this disclosure. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.
[0024] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are intended only to distinguish the components from other components and do not limit the nature, order, or sequence of the components. Where it is stated that a component is "connected," "coupled," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be "connected," "coupled," or "joined" between each component.
[0025] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.
[0026] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.
[0027] In addition, terms such as first, second, A, B, (a), (b), etc. may be used when describing the components of the present invention. These terms are intended merely to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by these terms. Throughout the specification, when a part is described as 'comprising' or 'equipped' with a certain component, unless specifically stated otherwise, this means that it does not exclude other components but may include additional components. Furthermore, terms such as 'part' or 'module' described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.
[0028]
[0029] FIG. 1 illustrates an exemplary environment in which an artificial intelligence acceleration device according to some embodiments of the present disclosure may be applied. Through a system including a user terminal (100) and an artificial intelligence acceleration device (200) as shown in FIG. 1, computations performed in an artificial intelligence model can be performed more efficiently and at a higher speed.
[0030] Below, we will explain in more detail the components shown in FIG. 1 related to the high-speed operation of artificial intelligence computation by utilizing the user terminal (100) and the artificial intelligence acceleration device (200) through the system described above.
[0031] FIG. 1 illustrates an example in which a user terminal (100) and an artificial intelligence accelerator (200) are connected via a network, but this is merely for convenience of understanding, and the number of devices that can be connected to the network can vary.
[0032] Meanwhile, FIG. 1 merely illustrates a preferred embodiment for achieving the purpose of the present disclosure, and some components may be added or deleted as needed. Below, the components illustrated in FIG. 1 will be described in more detail.
[0033] The artificial intelligence accelerator (200) can collect and analyze various information generated from the user terminal (100). The various information may include all data generated from the user terminal (100).
[0034] The user terminal (100) illustrated in FIG. 1 may be implemented with one or more computing devices. For example, all functions of the user terminal (100) may be implemented in a single computing device. As another example, the first function of the user terminal (100) may be implemented in a first computing device, and the second function may be implemented in a second computing device. Here, the computing device may be a notebook, a desktop, a laptop, etc., but is not limited thereto and may include all types of devices equipped with computing functions. The user terminal (100) may be connected to the artificial intelligence acceleration device (200) of the present invention via a network to transmit and receive data to and from each other.
[0035] Meanwhile, the artificial intelligence acceleration device (200) may also be implemented as one or more computing devices. For example, all functions of the artificial intelligence acceleration device (200) may be implemented in a single computing device. As another example, the first function of the artificial intelligence acceleration device (200) may be implemented in the first computing device, and the second function may be implemented in the second computing device. Here, the computing device may be a notebook, desktop, laptop, etc., but is not limited thereto and may include all types of devices equipped with computing functions. However, it may be preferable for the artificial intelligence acceleration device (200) to be implemented as a high-performance server-class computing device. An example of a computing device will be described with reference to FIG. 12.
[0036] Additionally, the functions that can be implemented in the artificial intelligence acceleration device (200) may also be implemented by utilizing an electronic device installed in the user terminal (100). Therefore, although the artificial intelligence acceleration device (200) and the user terminal (100) are depicted separately in FIG. 1, it is obvious that according to one embodiment, the artificial intelligence acceleration device (200) is installed in the user terminal (100) and the corresponding device within the user terminal (100) can implement the first function, the second function, etc. Accordingly, it should be noted that this is not limited to an embodiment in which the user terminal (100) and the artificial intelligence acceleration device (200) are externally separated as shown in FIG. 1.
[0037] For convenience of explanation, the present specification describes a situation in which the user terminal (100) and the artificial intelligence acceleration device (200) are separated to implement functions.
[0038] In some embodiments, components included in an environment to which the artificial intelligence accelerator (200) is applied can communicate through a network. The network can be implemented as any type of wired or wireless network, such as a Local Area Network (LAN), a Wide Area Network (WAN), a mobile radio communication network, or Wibro (Wireless Broadband Internet).
[0039] Meanwhile, although the environment illustrated in FIG. 1 is shown as being connected via a network through a user terminal (100) and an artificial intelligence acceleration device (200), it should be noted that the scope of the present disclosure is not limited thereto and that the user terminal (100) may be connected to the artificial intelligence acceleration device (200) via P2P (Peer to Peer).
[0040] Up to now, with reference to FIG. 1, exemplary environments in which a corresponding device (200) according to some embodiments of the present disclosure may be applied have been described. Hereinafter, with reference to FIG. 2 and subsequent drawings, methods according to various embodiments of the present disclosure will be described in detail.
[0041] Each step of the methods described below may be performed by a computing device. In other words, each step of the methods may be implemented by one or more instructions executed by a processor of the computing device. All steps included in these methods may be performed by a single physical computing device, but the first steps of the methods may be performed by a first computing device and the second steps of the methods may be performed by a second computing device.
[0042] In FIG. 2 below, the explanation will continue assuming that each step of the methods is performed by the artificial intelligence acceleration device (200) exemplified in FIG. 1. However, for the convenience of explanation, the subject of the operation of each step included in the methods may be omitted.
[0043]
[0044] FIG. 2 is a flowchart relating to certain operations for artificial intelligence model acceleration that can be performed in an artificial intelligence acceleration device according to some embodiments of the present disclosure.
[0045] In step S100, the artificial intelligence acceleration device (200) may load data from at least one user terminal. The loaded data may be data processed by an artificial intelligence model for performing calculations in an artificial intelligence model included in the artificial intelligence acceleration device (200).
[0046] In step S200, the artificial intelligence accelerator (200) can preprocess the loaded data. The preprocessing process may refer to a process of generating data directly for computation in order to perform artificial intelligence computation in the artificial intelligence model included in the artificial intelligence accelerator (200).
[0047] In step S300, the artificial intelligence accelerator (200) can perform a first computation process for an artificial intelligence model on the preprocessed data. In step S400, the artificial intelligence accelerator (200) can perform a second computation process for an artificial intelligence model on the preprocessed data. In step S500, after the first and second computation processes, the artificial intelligence accelerator (200) can calculate computation efficiency and provide the calculated computation efficiency to a user terminal.
[0048] The above artificial intelligence model may include a mini-core architecture that utilizes input sparsity instead of integrating small-scale computing units, while utilizing mega-cores and floating-point numbers to increase data reuse rates.
[0049] The specific architecture of the artificial intelligence model described in the present invention will be described in detail through FIGS. 3 to 11.
[0050]
[0051] FIGS. 3 to 11 are drawings for illustrating artificial intelligence acceleration operations according to some embodiments of the present disclosure.
[0052] In one embodiment, the artificial intelligence model may be characterized by integrating a cross-shaped memory architecture for artificial intelligence weights and input data caching between multi-cores.
[0053] In one embodiment, the artificial intelligence model may be characterized by adopting a sparse matrix compression method to minimize memory usage.
[0054] In one embodiment, the artificial intelligence model may be characterized by adopting a sparsity utilization design method in which outlier data is padded with 0 and computation is performed on a MEGA core, and the mini core can perform computation by excluding input '0' in accordance with the sparsity data characteristics.
[0055] In one embodiment, the artificial intelligence model may include a cross-shaped memory architecture to minimize the amount of data read and write when multiple mega cores and mini cores are operated in a single processor.
[0056] In one embodiment, the step of providing computational efficiency to a user terminal may involve supporting an FP-FXP hybrid input value representation method and providing data regarding computational efficiency to the user terminal.
[0057] More specifically, regarding the present invention, AI technology, which demonstrated overwhelming performance in ImageNet—a representative object recognition competition—has expanded its scope of application beyond simple object recognition to various areas such as object detection, object region segmentation, speech recognition, and translation. However, since these areas merely provide the results of AI data analysis regarding given inputs, it was thought that they could not be utilized in fields requiring creativity, such as song production or drawing. Yet, with the rapid advancement of AI technology, generative AI has emerged that creates and imagines new things, rather than relying on simple reasoning and analysis.
[0058] When existing artificial intelligence is referred to as discriminative AI, current AI models, which are classified into discriminative AI and generative AI, have different requirements applied during inference depending on their type. Discriminative AI must perform the task of classifying into one of the predefined categories within a very short inference latency of less than 33ms, whereas generative AI must perform the task of creating high-quality sound, images, or videos, while allowing for a long latency of more than 1 minute.
[0059] In particular, generative AI, which demands high quality, differs from discriminative AI in terms of quantization performance. While both types of AI do not experience significant performance degradation even when weights are quantized to a precision of 4 bits or less, generative AI, which requires high quality, demands high bit precision for the input activations (IA) that occur during AI computation. This phenomenon occurs because, as shown in Figure 2, while it is possible to apply and tune optimized quantization methods such as asymmetric or symmetric quantization to unchanging weight values to quantize them to low bits once training is complete, it is difficult to optimize the quantization method for input values that change every time depending on the input. Here, in the case of discriminative AI, even the human eye can easily distinguish images expressed with low quality or low bits; however, for generative AI aimed at creative production activities, low-bit images can make the production quality appear low, so it can be seen that its performance tends to be more sensitive to the bit count of the input values.
[0060] Even for discriminative AI, existing quantization methods were not without their problems. Quantization adopts a bit count and quantization level that minimizes accuracy loss from the collected dataset; however, the actual environment in which users utilize the AI may have data distributions that were not identified in the collected dataset. This ultimately leads to a problem where the AI's performance cannot be guaranteed for cases that were not fully examined in the dataset. In particular, the method utilizing fixed-point numbers, which limits the data range in advance and performs quantization, is highly vulnerable to accuracy degradation issues despite having higher hardware efficiency compared to floating-point numbers, which dynamically adjust the data range using an exponent.
[0061] To address these issues, an FP-FXP hybrid input representation method can be adopted, which separates input values into Inlier and Outlier data and represents Inlier data using Fixed-point (FXP) and Outlier data using Floating-point (FP). However, existing AI acceleration processors have not been able to efficiently support data representations that mix Fixed-point and Floating-point based methods. Furthermore, hardware that supports both Fixed-point and Floating-point arithmetic units suffers from very low core utilization, making it impossible to achieve peak performance. To efficiently support not only Discriminative AI but also Generative AI without performance degradation, a hardware architecture is required that supports both the existing method utilizing only Fixed-point and the Floating-point-Fixed-point hybrid input representation method. To achieve the above-mentioned objective, the present invention proposes a MEGA.mini core architecture that integrates and accelerates two heterogeneous cores: a large accelerator MEGA core specialized for fixed-point operations and a small accelerator mini core specialized for floating-point operations.
[0062] In the floating-point-fixed-point hybrid input representation method, the majority of the data is represented as fixed-point inliers in the center, while the rest of the data, or outliers, is represented as floating-point. Since more than 95% of the data is classified as inliers, outliers account for less than 5%, and to represent this data while minimizing memory usage, an encoding method is adopted that compresses only non-zero values.
[0063] Zero Length Compression (ZLC), a method for compressing non-zero values, is significantly affected by the sparsity of the data. When data is not spars, the compression ratio drops significantly, making it more efficient not to use the compression method at all. This phenomenon intensifies as the number of bits representing non-zero values decreases; in particular, when non-zero values are represented by 4 bits, it can be observed that the data compression ratio is the same as when no compression is performed, even with only 50% sparsity.
[0064] The heterogeneous cores proposed in this invention, namely the MEGA core and the mini core, adopt computation methods optimized for Inlier and Outlier data to which the previously described data compression method is applied. In particular, the MEGA core accepts input values expressed as 8-bit fixed-point numbers and adopts a design that integrates a large number of MAC (Multiply-and-accumulation) units to increase data reuse rates, thereby optimizing acceleration for Inlier data, which accounts for more than 95%. Conversely, the mini core adopts a '0' skip operation method to accelerate 16-bit floating-point Outlier data, which accounts for less than 5%, and integrates only a relatively small number of MAC units compared to the MEGA core. As shown in Fig. 7, the MEGA core is advantageous for processing large amounts of data regardless of the number of bits, so it handles Inlier data with low input sparsity. Conversely, Outlier data, which has high input sparsity and requires a high number of data bits, is exclusively handled and accelerated by the mini core.
[0065] To explain how the floating-point-fixed-point hybrid input representation is mapped to the MEGA.mini core architecture, the maximum data value that can be represented in fixed-point terms is defined for each deep neural network layer based on a predetermined integer part length. Data smaller than this maximum value is represented in fixed-point terms, while data larger is represented in floating-point terms, and compressed using ZLC. Subsequently, each data point is used as input for the MEGA core and the mini core. Even if the input is quantized solely in fixed-point terms without outlier data, support is possible without significant overhead because accelerators that support floating-point have fewer MAC operators. If unexpected outlier data exists, it is represented in floating-point terms, and computation is performed on the mini core. However, since such outlier data appears very rarely, a sparse matrix compression method is adopted to minimize memory usage.
[0066] Looking at a design example of a processor adopting the MEGA.mini core architecture proposed in this invention, the example consists of four MEGA.mini cores, two global I / O memories, two global weighted I / O memories, and a top controller. Among these, the global I / O memories store data to be used by the MEGA.mini cores located on both sides, and two different MEGA.mini cores use the same input value but apply different weights to produce different outputs. However, since the input value is shared, the number of times data is cached from external memory is reduced by half. In the case of the global weighted memories, they store weights to be used by the MEGA.mini cores located at the top and bottom, and the same weight is delivered to two different MEGA.mini cores receiving different input values. Similarly, since the weights are shared, the number of times data is cached from external memory is reduced by half. As such, two global I / O memories and two global weighted memories form a set with four MEGA.mini cores; this memory architecture is named the Cross Memory Architecture because of its cross-shaped form. The Cross Memory Architecture and the four MEGA.mini cores allow for easy size expansion when increasing the number of cores by simply copying and pasting the shape.
[0067] A single MEGA.mini core consists of the previously described MEGA core and mini core, local memory storing compressed outlier data to be used by the mini core, an aggregation core that aggregates the computation results of the two cores, a post-processing unit for data post-processing, and tensor memory (TMEM) that stores input / output tensors. Since most data is represented as fixed-point numbers, outlier data is padded with zeros for computation on the MEGA core, and a design approach that increases data reuse rates is adopted to maximize energy efficiency. In contrast, the mini core adopts a sparsity-based design approach that allows computation by excluding input '0's to suit the characteristics of sparse data.
[0068] Furthermore, to explain the specific structure of the MEGA core, the MEGA core uses MAC units as its basic arithmetic units, which receive 4-bit weights that utilize fixed-point and 8-bit input values that also utilize fixed-point. These MAC units are arranged in a 2D array form and adopt an Output Stationary Dataflow. Data reuse is increased by broadcasting identical pixels or batch data along the horizontal rows of the array and broadcasting identical weight values along the vertical rows. MAC units are arranged in a 32x32 array form; MACs placed in the horizontal rows share a single input value, accept 32 different weight values, and calculate 32 different output values. In contrast, MACs placed in the vertical rows apply a single weight to 32 different input values to calculate 32 different output values.
[0069] This invention proposes a new circuit to maximize the area and energy efficiency of the fixed-point MAC unit adopted by the MEGA core. As shown in Fig. 11, a conventional fixed-point based MAC unit requires one multiplier, one adder, and one register for data accumulation. However, this is the general structure of a fixed-point based MAC that adopts 2's complement representation. The new MAC proposed in this invention adopts sign magnitude for both weights and input values instead of 2's complement, thereby minimizing dynamic power consumption caused by bit toggling in input and weight values that exhibit a normal distribution. Previously, MACs adopting sign magnitude had the problem of having to convert back to 2's complement for accumulation; however, the proposed MAC selects different registers based on the sign values of the weights and input values and simply adds them sequentially without 2's complement data conversion. After the operation for all channels is completed, the values accumulated in the two registers are subtracted to produce the final operation result.
[0070] To explain the method further optimized by adopting sign magnitude, conventional 2's complement accumulators use a full adder for addition and registers for accumulation, and sign extension was essential because multiple values had to be added. The increased number of full adders and registers due to sign extension raises hardware overhead accordingly. However, the proposed sign magnitude-based accumulator eliminates sign extension and replaces the full adder with an asynchronous counter, thereby implementing accumulation operations without registers or a clock network. In particular, the asynchronous counter not only improves area and energy efficiency but also minimizes setup timing violation issues caused by critical paths.
[0071] To explain the specific structure of the mini core, it consists of a MAC unit that accepts 4-bit weights represented as fixed-point and 16-bit or 12-bit input values represented as floating-point. For floating-point operations, this MAC unit integrates not only multipliers, adders, and registers, but also normalization circuits for exponent processing. While the MEGA core is designed with a 2D MAC array to maximize data reuse, the mini core is designed with a 1D MAC array. Like the MEGA core, it employs Output Stationary Dataflow; by retrieving a single non-zero value from local memory, it retrieves different weight values that produce different outputs and performs the operation simultaneously. Although the data reuse rate is lower than that of the MEGA core, it supports zero-skip operations, which can actually result in higher energy efficiency when processing sparsity data. To skip '0's, compressed data is processed using Zero Length Compression or Zero Length Encoding; to utilize Zero Length Encoding, which represents the length of consecutive '0's, the length of '0's is used in address operations to retrieve the next weight.
[0072] Regarding additional optimization measures to further enhance the efficiency of mini-core operations, since utilizing the existing 16-bit floating-point representation results in low memory access volume and multiplier energy efficiency, a new dynamic floating-point representation method is proposed to reduce the required number of bits to 12 bits. The outlier data accepted as input by the mini-core represents only large numbers that cannot be represented by the MEGA core. Therefore, the exponent bias value is adjusted in advance by utilizing the integer part length information used in the MEGA core. Since small values do not need to be represented, the denormalization circuit required by the floating-point unit can be eliminated, and outlier data within a range similar to FP16 can be represented through bias adjustment with a small number of bits.
[0073] The results of computations performed by the MEGA core and mini core are aggregated in the Aggregation core. Looking at the computation process, since the MEGA core's computation results are accumulated in two different registers based on the sign of the input and weight values, the values stored in the two registers are subtracted to first obtain the final result. Subsequently, the MEGA core result is converted to a 16-bit floating-point number and added to be combined with the mini core's result. After aggregating the results of the MEGA core and mini core, the final result is classified into Inlier and Outlier based on the integer part length predetermined for each layer of the deep neural network, and converted into the appropriate data format for each. Inlier data is converted to fixed-point to be used as input for the MEGA core, while Outlier data is converted to dynamic floating-point to be used as input for the mini core. Additionally, Zero Length Compression is applied to the data used as input for the mini core for compression.
[0074] As shown in Fig. 16, the MEGA.mini core architecture, which adopts a floating-point-fixed-point hybrid input representation method, exhibits different computational characteristics between the MEGA core and the mini core. While the MEGA core demonstrates a constant data throughput regardless of input sparsity, the mini core demonstrates higher data throughput as sparsity increases. Furthermore, once the MEGA core starts an operation, it produces outputs for multiple batches simultaneously, whereas the mini core produces outputs for only one batch at a time. When utilizing these two cores with different computational characteristics together, two problems arise. First, since the single operation completion time of the mini core varies depending on data sparsity, there may be a workload imbalance where the operation of one core—either the mini core or the MEGA core—finishes first and waits for the other core to finish. When sparsity is high, the mini core's operation finishes first and waits for the MEGA core's operation. Conversely, when sparsity is low, the MEGA core finishes before the mini core and waits for the mini core's operation to finish. In addition to the issue of workload imbalance, problems with simultaneous access to output memory can also occur. Since the output memory bandwidth is tailored to the bandwidth of the mini-cores, if the MEGA core and mini-cores access the output memory simultaneously, the operation of the other core must be temporarily paused while data is being written. In particular, because the MEGA core outputs multiple batches at once unlike the mini-cores, the mini-cores must wait for a considerable amount of time while the MEGA core is accessing the output memory, unable to write data.
[0075] To solve the aforementioned problem, the present invention presents a total of three solutions. First, algorithmic or software-based solutions are described in FIG. 17. Generative AI models can be broadly classified into Diffusion Models (DM) and Non-diffusion Models (NDM). DM is a method that generates high-quality output by iterating multiple times on a single input, whereas NDM is a method that generates output with only a single iteration. NDM utilizes a predefined integer part, but employs a long-term update method in which the integer part length is reset to a higher value if a consistently high outlier ratio is observed in an environment where the user uses it repeatedly. On the other hand, DM uses an integer part length optimized at Iteration 0 and employs a short-term update method that adjusts the integer part length according to the outlier ratio that gradually changes with each iteration.
[0076] The second method adopts a prefetch method that predefines a single operation tile at the compiler level and performs the next tile operation in advance when the current tile operation is finished. As shown in Fig. 18, to use this method, a buffer is aggregated to temporarily store the outputs of the MEGA core and mini core. When the current tile operation is finished, the result of that tile operation is stored in the buffer, and the next tile operation is started in advance to maximize core utilization.
[0077] The final solution involves integrating the output synchronization circuit. As shown in Fig. 19, this circuit is integrated within the aggregation core to minimize the problem of concurrent access to output memory. The primary role of the output synchronization circuit is to lower the priority of operations for low-sparse batches or pixels and to perform low-sparse operations while the MEGA core accesses output memory, thereby eliminating the need for the mini core to access output memory. The Task Snatching Unit within the output synchronization circuit is designed with a CNT Tracker and a Task FIFO. The CNT Tracker receives the number of non-zero values as input to monitor the lowest-sparse tasks and fetch those operations later. High-sparse tasks are fetched into the FIFO and assigned sequentially, and when the MEGA core is accessing output memory, low-sparse tasks are assigned instead of the FIFO.
[0078] So far, we have mainly described how the MEGA.mini core architecture performs operations, particularly when adopting a floating-point-fixed-point hybrid input representation method, but as explained in Fig. 9, the MEGA.mini core can determine an optimized allocation method according to various data formats and usages.
[0079] To explain the six software usage models available for the MEGA.mini core architecture, the first model utilizes only the MEGA core and can be used when deep neural networks are sufficiently represented using fixed-point numbers alone, or when there is no need to compensate for accuracy using floating-point numbers. The second model utilizes only the mini core and can be used when the input channels of the deep neural network are not sufficiently large, when input sparsity is very high, or when all input values can be represented solely as floating-point numbers. The third model involves the MEGA core and mini core processing independent tasks simultaneously; it is primarily used when it is necessary to perform independent layer operations that do not affect each other concurrently, such as inception or residual paths. The fourth model utilizes the MEGA core and mini core when employing a floating-point-fixed-point hybrid input representation method. The fifth model involves the MEGA core performing the operation first and using the result as the input for the mini core. The sixth model involves the mini core performing the operation first and using the result as the input for the MEGA core. The fifth and sixth computational models can be used to accelerate inference by combining layer fusion concepts.
[0080] Thus, while the MEGA core and mini core proposed by this invention appear to have a structure similar to the existing CPU architecture of Big / Little cores, significant differences exist. Although both the Big / Little core architecture and the MEGA.mini core architecture are similar in that they integrate two cores of different sizes, the existing Big / Little core architecture is optimized for CPU computation, and both the Big core and Little core support 32-bit and 64-bit floating-point numbers. On the other hand, the MEGA core and mini core utilize different data representation methods—8-bit dynamic fixed-point and 12-bit dynamic floating-point—resulting in optimization for NPU computation rather than CPU computation. To minimize instruction processing latency, the existing Big core integrates additional hardware blocks to support Out-of-Order Execution (OoOE), which is a major cause of high hardware complexity. In contrast, the existing Little core is structured to process instructions sequentially without such a structure, exhibiting a simple hardware architecture. In the MEGA.mini core architecture proposed in this invention, the MEGA core actually has a simpler hardware structure. On the other hand, the mini core requires more complex hardware to support the '0' operation skip function. These differences in hardware structure ultimately lead to significant differences in energy efficiency and throughput; while the big core has high throughput but very low energy efficiency, the MEGA core exhibits high throughput and energy efficiency. The little core supports high-efficiency computation at the cost of low throughput, whereas the mini core has low throughput and energy efficiency. However, the mini core supports a different data representation method than the MEGA core, enabling more accurate data representation.
[0081] To explain the two ways the cross-shaped memory architecture adopted from the MEGA.mini core architecture is utilized, the cross-shaped memory architecture, composed of two global I / O memories and two global weighted memories, supports two different memory access modes: Unicasting mode and Broadcasting mode. Unicasting mode allocates a separate area of cross-shaped memory for each MEGA.mini, allowing each core to independently occupy global memory. Broadcasting mode transmits and receives the same data to adjacent MEGA.mini cores, reducing memory access volume by half and external memory access volume by half. However, since Broadcasting mode requires adjacent cores to share the same value, it is not possible to program and utilize cores in various ways.
[0082] This cross-shaped memory architecture is very easy to expand, as shown in Fig. 23, because the four MEGA.mini cores, two global I / O memories, and two global weighted memories are considered as a set, and the structure can be expanded in its original form to match the target performance of the processor being manufactured while maintaining that form. For example, in the case of a chip that communicates between IPs as a Network-on-chip (NoC), the size can be freely increased or decreased by simply increasing the size of the NoC and attaching the cross-shaped memory structure as if copying and pasting it.
[0083] Below, an exemplary computing device in which an artificial intelligence acceleration device (200) can be implemented will be described in detail through FIG. 12.
[0084]
[0085] FIG. 12 is a drawing of an exemplary computing device capable of implementing a device and / or system according to various embodiments of the present disclosure.
[0086] A computing device (1500) may include one or more processors (1510), a bus (1550), a communication interface (1570), a memory (1530) for loading a computer program (1591) executed by the processor (1510), and a storage (1590) for storing the computer program (1591). However, only components related to the embodiments of the present disclosure are illustrated in FIG. 12. Accordingly, a person skilled in the art to which the present disclosure belongs will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 12.
[0087] The processor (1510) controls the overall operation of each component of the computing device (1500). The processor (1510) may be configured to include a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphic Processing Unit), or any form of processor well known in the art of the present disclosure. Additionally, the processor (1510) may perform operations for at least one application or program for executing the method according to the embodiments of the present disclosure. The computing device (1500) may have one or more processors.
[0088] The memory (1530) stores various data, commands and / or information. The memory (1530) may load one or more programs (1591) from storage (1590) to execute a method according to embodiments of the present disclosure. The memory (1530) may be implemented as volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.
[0089] The bus (1550) provides communication functions between components of the computing device (1500). The bus (1550) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.
[0090] The communication interface (1570) supports wired and wireless internet communication of the computing device (1500). Additionally, the communication interface (1570) may support various communication methods other than internet communication. To this end, the communication interface (1570) may be configured to include a communication module well known in the art of the present disclosure.
[0091] According to some embodiments, the communication interface (1570) may be omitted.
[0092] Storage (1590) can store one or more of the above programs (1591) and various data non-temporarily.
[0093] Storage (1590) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which this disclosure belongs.
[0094] A computer program (1591) may include one or more instructions that cause a processor (1510) to perform a method / operation according to various embodiments of the present disclosure when loaded into memory (1530). That is, the processor (1510) may perform a method / operation according to various embodiments of the present disclosure by executing the one or more instructions.
[0095] Various embodiments of the present disclosure and effects according to those embodiments have been described with reference to FIGS. 1 to 12. The effects according to the technical concept of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description in the specification.
[0096] The technical concept of the present disclosure, as described so far with reference to FIGS. 1 to 12, may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). The computer program recorded on the computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed on the other computing device, thereby being used on the other computing device.
[0097] In the foregoing, although all components constituting the embodiments of the present disclosure have been described as being combined or operating together, the technical concept of the present disclosure is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present disclosure, all components may be selectively combined and operated in one or more ways.
[0098] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.
[0099] Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure may be practiced in other specific forms without changing the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present disclosure shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concept defined by the present disclosure.
Claims
1. In a method performed by an artificial intelligence accelerator, A step of loading data from at least one user terminal; Step for preprocessing loaded data; A step of performing a first computation process for an artificial intelligence model on preprocessed data; A step of performing a second computation process for an artificial intelligence model on preprocessed data; and After the first operation and the second operation process, the step of calculating the operation efficiency and providing the operation efficiency to a user terminal; is included. The above artificial intelligence model utilizes mega-cores and floating-point numbers to increase data reuse rates, and includes a mini-core architecture that utilizes input sparsity instead of integrating small-scale computing units. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
2. In Paragraph 1, The above artificial intelligence model is characterized by integrating a cross-shaped memory architecture for artificial intelligence weights and input data caching between multicores. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
3. In Paragraph 2, The above artificial intelligence model is characterized by adopting a sparse matrix compression method to minimize memory usage. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
4. In Paragraph 3, The above artificial intelligence model is characterized by adopting a sparsity-utilizing design method in which outlier data is padded with 0 and computation is performed on the MEGA core, while the mini core performs computation by excluding input '0' to match the sparsity data characteristics. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
5. In Paragraph 4, The above artificial intelligence model includes a cross-shaped memory architecture to minimize the amount of data read and write when multiple mega-cores and mini-cores operate on a single processor. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
6. In Paragraph 5, The step of providing computational efficiency to a user terminal supports an FP-FXP hybrid input value representation method and provides data regarding computational efficiency to the user terminal. Method for accelerating artificial intelligence computations, including a dedicated architecture for general-purpose artificial intelligence acceleration.
7. Processor; Network interface; Memory; and It includes a computer program that is loaded into the memory and executed by the processor, The above processor is, Instructions for loading data from at least one user terminal; Instructions for preprocessing loaded data; Instructions for performing a first operation process for an artificial intelligence model on preprocessed data; Instructions for performing a second computation process for an artificial intelligence model on preprocessed data; and After the first and second operation processes, an instruction to calculate the operation efficiency and provide the operation efficiency to a user terminal is performed, including, The above artificial intelligence model utilizes mega-cores and floating-point numbers to increase data reuse rates, and includes a mini-core architecture that utilizes input sparsity instead of integrating small-scale computing units. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.
8. In Paragraph 7, The above artificial intelligence model is characterized by integrating a cross-shaped memory architecture for artificial intelligence weights and input data caching between multicores. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.
9. In Paragraph 8, The above artificial intelligence model is characterized by adopting a sparse matrix compression method to minimize memory usage. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.
10. In Paragraph 9, The above artificial intelligence model is characterized by adopting a sparsity-utilizing design method in which outlier data is padded with 0 and computation is performed on the MEGA core, while the mini core performs computation by excluding input '0' to match the sparsity data characteristics. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.
11. In Paragraph 10, The above artificial intelligence model includes a cross-shaped memory architecture to minimize the amount of data read and write when multiple mega-cores and mini-cores operate on a single processor. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.
12. In Paragraph 11, The step of providing computational efficiency to a user terminal supports an FP-FXP hybrid input value representation method and provides data regarding computational efficiency to the user terminal. An AI acceleration device that accelerates AI computations, including a dedicated architecture for general-purpose AI acceleration.