Recommendation system and method for accelerating inference

Abstract

A recommendation system and method for accelerating inference are disclosed. The recommendation system comprises a memory processing unit for performing a feature interaction on an embedding vector whenever the embedding vector is looked up on an embedding table with respect to a sparse feature. Here, the feature interaction is performed by an inner product between the embedding vector at the lookup and an embedding vector used in a previous lookup.

Description

Recommendation system and method for accelerating inference

[0001] The present invention relates to a recommendation system and method for accelerating reasoning.

[0002] Deep learning-based recommendation systems perform feature interaction after both the bottom MLP stage and the embedding lookup stage are completed. As a result, inference time is inevitably long.

[0003] The present invention provides a recommendation system and method for accelerating reasoning.

[0004] To achieve the above-mentioned purpose, a recommendation system according to one embodiment of the present invention includes a memory processing unit that performs a feature interaction on an embedding vector whenever an embedding vector is looked up in an embedding table for a sparse feature. Here, the feature interaction is performed by taking the inner product of the embedding vector at the time of the lookup and the embedding vector used at the previous lookup.

[0005] A recommendation system according to another embodiment of the present invention includes a host; and a main memory that performs an embedding table lookup or feature interaction according to a command provided from the host. Here, the main memory performs feature interaction on embedding vectors corresponding to sparse features, and the host does not perform feature interaction on dense features but converts them into vectors of a certain length through a bottom MLP.

[0006] A recommendation method according to one embodiment of the present invention includes the steps of: performing a feature interaction on an embedding vector whenever an embedding vector is looked up in an embedding table for a sparse feature; inputting a dense feature into a bottom MLP to convert the dense feature into a vector of a predetermined length; and inputting the result of the feature interaction of the sparse feature and the output of the bottom MLP into a top MLP to output a Click Through Rate (CTR).

[0007]

[0008] The recommendation system and method according to the present invention can perform a feature interaction on an embedding vector whenever an embedding vector is looked up in an embedding table for a sparse feature. As a result, the time for the embedding table lookup and the time for the feature interaction overlap, so the inference time can be significantly reduced.

[0009] Figure 1 is a diagram illustrating the recommendation process of a general recommendation system.

[0010] FIG. 2 is a diagram illustrating the recommendation process of a recommendation system according to one embodiment of the present invention.

[0011] Figure 3 is a timing diagram illustrating embedding table lookup and feature interaction.

[0012] FIG. 4 is a diagram illustrating a recommendation system from a hardware perspective according to an embodiment of the present invention.

[0013] As used in this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "composed" or "comprising" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as meaning that some of the components or steps may be excluded, or that additional components or steps may be included. Furthermore, terms such as "...part," "module," etc., as used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0014]

[0015] The present invention relates to a recommendation system using a collaborative filtering method, and more specifically to a deep learning-based recommendation system such as a Deep Learning Recommendation Model (DLRM).

[0016] According to one embodiment, the recommendation system may not perform feature interaction for dense features and may perform feature interaction only for sparse features. Here, dense features are information expressed as real numbers (e.g., information such as height, weight, etc.) and are converted into a vector of a certain length through a bottom multi-layer perceptron (bottom MLP), and sparse features are information expressed in one-hot or multi-hot encoding methods (e.g., information such as gender, country, etc.) and are converted into an embedding vector stored in an embedding table.

[0017] In particular, the recommendation system may use a method that performs feature interaction whenever an embedding vector is looked up in an embedding table. That is, the recommendation system pipelins feature interaction and embedding table lookup so that feature interaction can be masked during embedding table lookup time, and as a result, inference time can be significantly reduced.

[0018]

[0019] Hereinafter, various embodiments of the present invention will be described in detail with reference to the attached drawings.

[0020] FIG. 1 is a diagram illustrating the recommendation process of a general recommendation system, FIG. 2 is a diagram illustrating the recommendation process of a recommendation system according to an embodiment of the present invention, FIG. 3 is a timing diagram illustrating embedding table lookup and feature interaction, and FIG. 4 is a diagram illustrating a recommendation system from a hardware perspective according to an embodiment of the present invention.

[0021] Looking at a general recommendation system, as shown in Fig. 1, dense features are converted into vectors of a certain length through a bottom MLP (Bot-MLP), sparse features are converted into embedding vectors, feature interactions receive the vector output from the bottom MLP and the embedding vector as inputs, and each vector is taken as an inner product with other vectors excluding itself, and the top MLP (Top-MLP) receives the output of the feature interaction as input and finally outputs the Click Through Rate (CTR) to complete the inference.

[0022] That is, the above recommendation system performs a lower MLP process, an embedding lookup process, a feature interaction process, and a higher MLP process. However, the feature interaction is performed after both the lower MLP stage and the embedding lookup stage have been completed. Consequently, the inference time was inevitably long.

[0023] In addition, to apply this method to NMP (Near-Memory Processing) accelerators, the output of the lower MLP had to be stored back in memory, which occupied main memory space and bandwidth.

[0024] Accordingly, the recommendation system of the present invention proposes a method to reduce inference time, particularly feature interaction time, in order to shorten inference time and reduce the use of main memory space and bandwidth.

[0025] Referring to FIG. 2, the recommendation system of the present invention inputs dense features into a lower MLP for processing and then directly inputs them into an upper MLP without performing feature interaction, while sparse features are input into an upper MLP after performing embedding lookup and feature interaction. That is, the recommendation system does not perform feature interaction for dense features but performs feature interaction only for sparse features.

[0026] According to one embodiment, the recommendation system can perform feature interaction together during embedding table lookup as shown in FIG. 3. Consequently, the time for embedding table lookup and the time for feature interaction can overlap, that is, embedding table lookup and feature interaction can be pipelined. Thus, inference time can be significantly reduced.

[0027] Referring specifically to FIGS. 3 and 4, first, using an extended kernel, commands required for the operation of a memory processing unit of HBM (High Bandwidth Memory), which is the main memory in the host, such as an NMP, can be provided to the NMP through the memory controller of the host.

[0028] Next, new information can be overwritten in the registers that store information about the number of embedding tables and the dimensions of the embedding vectors prior to the inference operation.

[0029] Continuing, when information about a sparse feature is input while inference is running, the host can provide information to the NMP via a command regarding whether it was a hot vector at the time of training. If the sparse feature is a hot vector, it checks whether the corresponding embedding vector is stored in a cache that follows the LRU page replacement algorithm; if it is not stored, it can read the corresponding embedding vector from the DRAM memory and pass it to the processing unit while simultaneously storing it in the LRU cache, and if it is stored, it can immediately provide the embedding vector to the processing unit.

[0030] Next, when a new embedding vector is input, the processing unit performs a dot product with the embedding vectors stored in the internal cache and stores the result in an output buffer, and can store the new embedding vector in the internal cache. Here, the embedding vectors are the embedding vectors used during the previous embedding lookup.

[0031] This process is repeated whenever a new embedding vector is input until all embedding vectors in the embedding table are looked up.

[0032] For example, let's assume that four embedding vectors (v1, v2, v3, v4) are sequentially looked up and read from the embedding table of the cache above.

[0033] When the second embedding vector (v2) is input to the processing unit, the processing unit can perform an inner product of the second embedding vector (v2) and the first embedding vector (v1) read from the cache as shown in Equation 1 below. Here, the first embedding vector (v1) is an embedding vector previously used for embedding lookup.

[0034]

[0035] Next, when the third embedding vector (v3) is input to the processing unit, the processing unit performs an inner product of the third embedding vector (v3) and the first embedding vector (v1) and second embedding vector (v2) read from the cache as shown in Equation 2 below.

[0036]

[0037] Continuing, when the fourth embedding vector (v4) is input to the processing unit, the processing unit performs an inner product of the fourth embedding vector (v4) and the first embedding vector (v1), second embedding vector (v2), and third embedding vector (v3) read from the cache, as shown in Equation 3 below.

[0038]

[0039] Next, the processing unit or host can terminate the feature interaction by combining the inner product results into an element-wise sum as shown in Equation 4 below.

[0040]

[0041] Furthermore, the host provides the result of the feature interaction to the upper MLP, and the upper MLP can terminate the inference process by predicting the CTR using the result of the lower MLP and the result of the feature interaction.

[0042] For this internal process, the processing unit may include a multiplier, an adder tree, and an accumulator.

[0043]

[0044] Meanwhile, the components of the aforementioned embodiments can be easily identified from a process perspective. That is, each component can be identified as a separate process. Furthermore, the processes of the aforementioned embodiments can be easily identified from the perspective of the device components.

[0045] In addition, the technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiments, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0046] The embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art with ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.

Claims

1. A memory processing unit that performs a feature interaction on an embedding vector whenever an embedding vector is looked up from an embedding table for a sparse feature, wherein A recommendation system characterized in that the above feature interaction is performed by taking the inner product of the embedding vector at the time of the lookup and the embedding vector used at the previous lookup.

2. In paragraph 1, the memory processing unit is NMP (Near-Memory Processing), wherein The above memory processing unit includes a processing unit and a cache, and A recommendation system characterized by, when a new embedding vector is input to the processing unit, the processing unit performs the feature interaction by taking the inner product of the input new embedding vector and the embedding vectors stored in the cache, and storing the input new embedding vector in the cache.

3. In Paragraph 2, The host further includes a command to the memory processing unit to perform the embedding lookup or the feature interaction, wherein A recommendation system characterized by the above host completing feature interaction by combining the inner product results for all embedding vectors into an element-wise sum.

4. A recommendation system according to claim 1, characterized in that dense features are converted into vectors of a certain length through a bottom MLP without performing feature interaction, and the result of the feature interaction between the output of the bottom MLP and the sparse features is input to a top MLP.

5. A recommendation system according to claim 1, characterized in that the time of the embedding table lookup and the time of the feature interaction overlap.

6. Host; and It includes a main memory that performs an embedding table lookup or feature interaction according to a command provided from the above host, The above main memory performs feature interaction with embedding vectors corresponding to sparse features, and A recommendation system characterized by the above host converting dense features into a vector of a certain length through a bottom MLP without performing feature interaction.

7. In paragraph 6, the NMP (Near-Memory Processing) of the main memory includes a processing unit, A recommendation system characterized by, when a new embedding vector is input to the processing unit, the processing unit performs the feature interaction by taking the inner product of the input new embedding vector and the embedding vectors stored in the cache, and storing the input new embedding vector in the cache.

8. A recommendation system according to claim 6, characterized in that the host inputs the result of the feature interaction of the sparse feature and the output of the lower MLP into the upper MLP to output a Click Through Rate (CTR).

9. A step of performing a feature interaction on an embedding vector whenever an embedding vector is looked up from an embedding table for a sparse feature; A step of inputting a dense feature into a bottom MLP to convert the dense feature into a vector of a certain length; and A recommendation method characterized by including the step of inputting the result of the feature interaction of the above sparse feature and the output of the above lower MLP into a top-MLP to output a Click Through Rate (CTR).

10. In claim 9, the step of performing the feature interaction is, For all embedding vectors, a step of taking an inner product of the corresponding embedding vector and one or more embedding vectors stored in the cache; and A recommendation method characterized by including a step of combining the above internal results.

Images