Apparatus and method for arithmetic processing for function approximation using look-up table

Abstract

An apparatus and method for arithmetic processing for function approximation using a look-up table are disclosed. An arithmetic processing apparatus according to one embodiment of the present invention comprises: a look-up table processing unit that presets and stores, in a look-up table format, a 1-1 index and a 2-1 index for obtaining an activation function across an entire range of preset bits, the entire range including a pre-interval section (pre-itv) and a point-count section corresponding to a single affine function; and a computation unit that generates the activation function using a piecewise-linear function that applies differentiated precision to each segment of the entire range, by applying the 1-1 index and the 2-1 index of the look-up table.

Description

Operation processing device and method for function approximation using a lookup table

[0001] The disclosed embodiments relate to an operation processing apparatus and method for function approximation using a lookup table.

[0002] The activation function of the current layer is used by approximating functions such as the swish function and sigmoid function through a lookup table (LUT) into a piecewise-linear function.

[0003] For functions with singular points, such as exponential x(exp(x)) and 1 / x, or functions where the slope varies significantly across intervals, there may be limitations in support using existing lookup tables with uniform intervals. In particular, for lightweight activation of the int type, the aforementioned phenomenon is relatively more severe; for functions going from int to int, if the interval is fixed, it may be difficult to express changes according to scale.

[0004] The disclosed embodiments aim to provide an apparatus and method for computational processing for function approximation using a lookup table, which implements an activation function using a non-uniform lookup table based on a piecewise-linear function that applies different precision for each section.

[0005] A computation processing device according to one embodiment includes: a lookup table processing unit that pre-sets and stores in the form of a lookup table a first-1 index and a second-1 index for obtaining an activation function for an entire interval of bits, which includes a pre-interval interval (pre-itv) and a point count interval corresponding to one affine function; and a computation unit that generates the activation function using a piecewise-linear function that applies a precision differentiated by interval in the entire interval, and applies the first-1 index and the second-1 index of the lookup table.

[0006] The above lookup table processing unit sets the prior interval section according to a first criterion pre-set in the entire section, and the set prior interval section is the first interval section (itv res ) and the first X interval (X res It can be set by distinguishing with ).

[0007] The above lookup table processing unit sets the 1-1 index according to a second criterion pre-set based on the first interval section, and can set it in the form of 1 / (power of 2).

[0008] The above lookup table processing unit can determine the number of points in the point count range corresponding to the one affine function according to the above 1-1 index.

[0009] The above operation unit calculates the above segment-specific linear function using a preset mathematical formula 1, and the above mathematical formula 1 is a piecewise-linear function( ) and the above itv may be the value of the interval of the linear function for each interval, and the above pre-itv may be the value of the prior interval.

[0010] The above mathematical formula 1 is implemented through mathematical formula 2, and the above mathematical formula 2 is, and, above a res is the 1-1 index, above b res is the 2-1 index, the above itv res is the first interval section and the above X res can be the first X interval.

[0011] The above operation unit generates the activation function using a preset mathematical formula 3, and the mathematical formula 3 is, And, above, a is the first-2nd index, above, b is the second-2nd index, above, itv is the value of the interval of the linear function by interval, and above, x may be the value of the number of points interval corresponding to one affine function.

[0012] The above x can have its interval values ​​differentiated according to the above 1-1 index.

[0013] A method for processing operations according to another embodiment, in a method performed by an operation processing device, comprises: a step of setting a point count interval corresponding to a pre-interval interval (pre-itv) and a single affine function in a pre-set entire interval of bits; a step of pre-setting a first-1 index and a second-1 index to obtain an activation function for the entire interval and storing them in the form of a lookup table; and a step of generating the activation function using a piecewise-linear function that applies a precision differentiated by interval in the entire interval, wherein the first-1 index and the second-1 index of the lookup table are applied.

[0014] The above operation processing method, in the step of setting the prior interval section, sets the prior interval section according to a first criterion pre-set in the entire section, and sets the set prior interval section as a first interval section (itv res ) and the first X interval (X res It can be set by distinguishing with ).

[0015] In addition to this, a computer-readable recording medium may be further provided for recording a computer program for executing a method for implementing the disclosed embodiments.

[0016] According to the disclosed embodiments, an activation function can be processed using a non-uniform lookup table based on a piecewise-linear function that applies different precision to each section.

[0017] FIG. 1 is a block diagram for explaining a computational processing device according to one embodiment.

[0018] FIGS. 2 to 5 are exemplary diagrams for explaining a calculation processing method according to one embodiment.

[0019] FIG. 6 is a flowchart for explaining a calculation processing method according to one embodiment.

[0020] FIG. 7 is a block diagram illustrating a computing environment including a computing device according to one embodiment.

[0021] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0022] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0023] FIG. 1 is a block diagram for explaining a computational processing device according to one embodiment.

[0024] Hereinafter, a method for processing operations according to one embodiment will be described with reference to FIGS. 2 to 5, which are exemplary diagrams for explaining the method.

[0025] Referring to FIG. 1, the operation processing unit (100) includes a lookup table processing unit (110) and an operation unit (120). Since the components illustrated in FIG. 1 are not essential for implementing the operation processing unit (100) according to the present disclosure, the operation processing unit (100) described in this specification may have more or fewer components than those listed above.

[0026] The components illustrated in FIG. 1 may be connected to each other so as to communicate through a communication network (not shown). In some embodiments, the communication network may include the Internet, one or more local area networks, wire area networks, cellular networks, mobile networks, other types of networks, or a combination of these networks.

[0027] Referring to FIG. 2, a general processing unit (not shown) can generate a piecewise linear function when an N+M bit activation is input. N and M can be arbitrary natural numbers. When the entire interval is N+M bits, the function of 2 to the power of N+M may be formed in a non-uniform state on a piecewise basis rather than a uniform linear state. Accordingly, the general processing unit sets the front part of the entire interval as an interval interval (itv), determines the indices of c and d in output_1 = c[itv] x X + d[itv], which indicates which piecewise linear function is used, and approximates it with a linear function related to X in the latter part. This can be defined as computing a lookup table-based activation function. For example, the indices of the interval interval (itv) in FIG. 2 can be approximated by connecting them to 2^N linear functions (affine functions). In each affine function, depending on X, each value can be approximated by outputting it as output_1 from X to X's 00 to 01. That is, in the entire interval of Fig. 2, 2 to the power of N+M points are composed of 2^N affine functions, so one affine function can correspond to 2^M points. At this time, if the activation function has a gradient that does not exceed 1, such as sigmoid or softmax, the projection can be relatively smooth. However, in the case of activation functions that have singular points, such as exponential x (exp(x)) or 1 / x, or where the gradient differs significantly from interval to interval, the approximation must be done relatively more densely, and there may be limitations in support through a lookup table that has a general uniform interval.

[0028] Accordingly, the computation processing device (100) of the disclosed embodiment may be configured to implement a technique for using a non-uniform lookup table (non-uniform LUT) for activation function approximation in which the difference in gradient between intervals is relatively large when performing deep learning computations, by applying an activation function to use non-linear computations. At this time, the computation processing device (100) may apply a technique for applying different precision levels for each interval, and a detailed explanation thereof will be provided later.

[0029] Referring to FIG. 1, the operation processing unit (100) includes a lookup table processing unit (110) and an operation unit (120).

[0030] Specifically, the lookup table processing unit (110) can store a first-1 index and a second-1 index for obtaining an activation function for an entire interval in a pre-set bit interval including a pre-interval interval (pre-itv) and a point count interval corresponding to one affine function, by pre-setting the first-1 index and the second-1 index to obtain an activation function for the entire interval in the form of a lookup table (LUT).

[0031] For example, the above 1-1 index (a res When ) is stored in a lookup table, it can be stored in the form 0, 1, 3, 4, 5(>>) rather than 1, 1 / 2, 1 / 4, 1 / 8, 1 / 16, 1 / 32.

[0032] The lookup table processing unit (110) can set a pre-interval interval (pre-itv) according to a first criterion pre-set in the entire interval. At this time, the pre-interval interval (pre-itv) is set arbitrarily by the operator, but can be set based on the function to be approximated and a pre-set center interval. In addition to the pre-interval interval (pre-itv), the first-1 index (a res ), 2-1 Index (b res) and a and b of Equation 3 described below may also be arbitrarily set by the operator, but may be set based on the function to be approximated and a pre-set midpoint interval.

[0033] For example, as illustrated in FIG. 3, if N+M bit-based activation is applied, the entire interval msb (most significant bit) ~ 1sb can be divided and set into a pre-interval interval (pre-itv) and a point count interval (X in FIG. 5) corresponding to one affine function of the remaining interval excluding the pre-interval interval.

[0034] The lookup table processing unit (110) sets the pre-interval interval as the first interval interval (itv res ) and the first X interval (X res It can be set by distinguishing as ). The lookup table processing unit (110) can determine the precision per section using the set first interval section and the first X section. At this time, determining the precision per section may mean determining the degree of denseness described above.

[0035] That is, a computation processing device (100) according to one embodiment introduces a prior interval section into a general ITV section as in FIG. 2, thereby providing a first interval section (itv res ) and the first X interval (X res It is possible to perform the process of operating the function to which ) is applied once again.

[0036] The lookup table processing unit (110) sets a first-1 index according to a second criterion pre-set based on a first interval section, and may set it in the form of 1 / (power of 2). This may be to fully utilize the X section of FIG. 5. At this time, referring to FIG. 5, the number of points corresponding to one affine function is the first-1 index (a resIt can be differentiated according to ). In Fig. 5, the number of points corresponding to one affine function can be varied, such as when the bit of the X interval is relatively large, a relatively large number of points correspond to one affine function, and when it is relatively small, a relatively small number of points correspond to one affine function.

[0037] Referring to 401 in FIG. 4, an activation function using a generally uniform lookup table can have uniform precision per interval. For example, as illustrated in FIG. 2, when N+M bit-based activation is applied, 2^M points can correspond to one affine function. In the entire interval of FIG. 2, the second interval (itv) can represent the number of affine functions, and the second X interval (x) can represent the number of points per affine function (e.g., 2^M). That is, a total of 2^(N+M) points can correspond to 2^N affine functions. In this case, approximation can be performed by outputting each value as an output from each affine function according to x, from 0 to 1.

[0038] In contrast, referring to 403 in FIG. 4, the computation processing unit (100) of the disclosed embodiment is configured to use a non-uniform lookup table (non-uniform LUT) for activation function approximation where the difference in slope between intervals is relatively large, and can be applied by setting a prior interval interval as in the lookup table processing unit (110) described above.

[0039] In the whole range based on bits P+Q+R, there may be a condition where 1 interval is 2^R points and 1 group is 2^Q intervals. P, Q, and R may be natural numbers.

[0040] The above entire interval can satisfy the conditions of 2^P groups, 2^(P+Q) intervals, and 2^(P+Q+R) points.

[0041] Referring to Fig. 3, the first interval section (itv) based on the P+Q+R bits res ) is P bit, 1st X interval (X res When ) is Q bits, the first interval (itv res ) with the 1-1 index (a res You can set ).

[0042] For example, the lookup table processing unit (110) uses the first-1 index (a) based on the entire section of FIG. 3. res )cast It can be set as follows.

[0043] In addition, the lookup table processing unit (110) has a second-1 index (b res ) can be set as [-2^(N-1), 2^(N-1)-1]. This can be implemented assuming that there are lookup tables of the same size, since the number of affine functions in the previous general method was 2^N, separate from PQR.

[0044] The lookup table processing unit (110) can determine the number of points in a point count range corresponding to one affine function according to the 1-1 index.

[0045] For example, referring to Fig. 5, a res [itv res When ] == 1, the number of points in the interval (X) corresponding to a single affine function is 2^(R+0) ((1)), a res [itv res When ] == 1 / 2, the number of points corresponding to a single affine function in the interval (X) is 2^(R+1) points ((2)), a res [itv resWhen ] == 1 / 4, the number of points in the interval (X) corresponding to one affine function is 2^(R+2) ((3)), a res [itv res When ] == 1 / 8, the number of points in the interval (X) corresponding to one affine function is 2^(R+3) ((4)), a res [itv res When ] == 1 / 16, the number of points in the interval (X) corresponding to one affine function is 2^(R+4) ((5)), a res [itv res When ] == 1 / 32, the number of points in the interval (X) corresponding to one affine function may be 2^(R+5) ((6)). In this case, the above-described case (4) may be a case where activation function approximation using a uniform lookup table is possible, such as the origin of 401 in FIG. 4.

[0046] A computation processing device (100) of one embodiment can implement a non-uniform lookup table activation function through a piecewise-linear function that obtains precision for each section through a prior interval section. The piecewise-linear function may also be named a piecewise-linear function.

[0047] The operation unit (120) generates an activation function using a piecewise-linear function that applies a differentiated precision to each section in the entire section, and may apply the first-1 index and the second-1 index of the lookup table.

[0048] The operation unit (120) can calculate a linear function for each interval using a pre-set mathematical formula 1.

[0049] [Mathematical Formula 1]

[0050] piecewise-linear function( )

[0051] The above itv may be the value of the interval of the linear function for each interval, and the above pre-itv may be the value of the prior interval.

[0052] Specifically, the above mathematical formula 1 can be implemented through mathematical formula 2.

[0053] [Mathematical Formula 2]

[0054]

[0055] The above a res is the 1-1 index, above b res is the 2-1 index, the above itv res is the first interval section and the above X res can be the first X interval.

[0056] Specifically, in the case where the 2-1 index of the above-described mathematical formula 2 is [-2^(N-1), 2^(N-1)-1], mathematical formula 2 may be equal to ∈ [-2^(N-1), 2^(N-1)-1].

[0057] As shown in FIG. 2, generally 2^M points correspond to one affine function, but as shown in FIG. 3, the operation processing unit (100) of the disclosed embodiment corresponds R bits excluding the pre-interval period (pre-itv) to one affine function (case: a res == 1) Since 2^R points can be mapped to a single affine function.

[0058] The operation unit (120) An activation function can be generated by applying the itv value calculated using a function. The above itv value may refer to the value of the interval of the linear function for each section.

[0059] Specifically, the operation unit (120) can generate an activation function using a preset mathematical formula 3.

[0060] [Mathematical Formula 3]

[0061]

[0062] The above a may be the 1st-2nd index, the above b may be the 2nd-2nd index, the above itv may be the value of the interval of the linear function by section, and the above x may be the value of the point count interval corresponding to a single affine function. The above x may be the above 1st-1st index (a res The values ​​for each interval can be differentiated depending on ).

[0063] A service for processing operations for function approximation using a lookup table may be provided, for example, in the form of a web page or through an application installed on a user device (not shown).

[0064] According to one embodiment, the lookup table processing unit (110) and the operation unit (120) may each be implemented using one or more physically separated devices, or by one or more hardware processors or a combination of one or more hardware processors and software, and unlike the illustrated example, they may not be clearly distinguished in specific operation.

[0065] FIG. 6 is a flowchart illustrating a method for processing operations according to one embodiment. The method illustrated in FIG. 6 can be performed, for example, by the aforementioned processing device (100). Although the method is described in the illustrated flowchart by dividing it into a plurality of steps, at least some of the steps may be performed in a different order, combined with other steps and performed together, omitted, divided into detailed steps, or performed with one or more steps not illustrated added.

[0066] In step 1100, the operation processing unit (100) can set a pre-interval interval (pre-itv) and a point count interval corresponding to one affine function in the entire interval of the pre-set bits through the lookup table processing unit (110).

[0067] In the step of setting the above-mentioned prior interval section, the computation processing device (100) sets the prior interval section according to a first criterion pre-set in the entire section through the lookup table processing unit (110), and the set prior interval section is the first interval section (itv res ) and the first X interval (X res It can be set by distinguishing with ).

[0068] In step 1200, the computation processing unit (100) can pre-set a first-1 index and a second-1 index to obtain an activation function for the entire range through the lookup table processing unit (110) and store them in the form of a lookup table.

[0069] In step 1300, the computation processing unit (100) generates an activation function using a piecewise-linear function that applies a differentiated precision to each section in the entire section through the computation unit (120), and may apply the first-1 index and the second-1 index of the lookup table.

[0070] FIG. 7 is a block diagram illustrating a computing environment including a computing device according to one embodiment. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0071] The illustrated computing environment (10) includes a computing device (12). The computing device (12) may be one or more components included in a computation processing device (100) according to one embodiment.

[0072] The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) can cause the computing device (12) to operate according to the exemplary embodiment described above. For example, the processor (14) can execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to cause the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0073] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0074] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0075] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0076] The disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0077] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

1. A lookup table processing unit that pre-sets and stores a first-1 index and a second-1 index in the form of a lookup table for obtaining an activation function for an entire interval of pre-set bits, including a pre-interval interval (pre-itv) and a point count interval corresponding to a single affine function; and A computation processing device comprising a computation unit that generates the activation function using a piecewise-linear function applying differentiated precision to each section in the entire section above, and applies the first-1 index and the second-1 index of the lookup table.

2. In Claim 1, The above lookup table processing unit is, In the entire section above, the prior interval section is set according to a first criterion that is pre-set, and the set prior interval section is the first interval section (itv res ) and the first X interval (X res A computational processing unit configured by distinguishing as ).

3. In Claim 2, The above lookup table processing unit is, An operation processing device that sets the 1-1 index according to a second criterion pre-set based on the first interval section above, and sets it in the form of 1 / (power of 2).

4. In Claim 3, The above lookup table processing unit is, A processing unit for determining the number of points in a point count range corresponding to the one affine function according to the above 1-1 index.

5. In Claim 1, The above operation unit is, The above linear function for each interval is calculated using the preset mathematical formula 1, and The above mathematical formula 1 is, piecewise-linear function( ) and, A computational processing unit in which the above itv is the value of the interval of a linear function for each interval, and the above pre-itv is the value of the prior interval.

6. In Claim 5, The above mathematical formula 1 is implemented through mathematical formula 2, and The above mathematical formula 2 is, And, The above a res is the 1-1 index, above b res is the 2-1 index, the above itv res is the first interval section and the above X res is a computational processing unit, which is the first X interval.

7. In Claim 6, The above operation unit is, The above activation function is generated using preset mathematical formula 3, and The above mathematical formula 3 is, And, A processing unit wherein a is the first-2nd index, b is the second-2nd index, itv is the value of the interval of a linear function by interval, and x is the value of the point count interval corresponding to a single affine function.

8. In Claim 7, The above x is a computation processing device in which the interval values ​​are differentiated according to the above 1-1 index.

9. In a method performed by a computational processing unit, A step of setting a point count interval corresponding to a pre-interval interval (pre-itv) and a single affine function within the entire interval of a preset bit; A step of pre-setting a 1-1 index and a 2-1 index to obtain an activation function for the entire above interval and storing them in the form of a lookup table; and A method for processing operations, comprising the step of generating the activation function using a piecewise-linear function that applies differentiated precision to each section in the entire section, and applying the first-1 index and the second-1 index of the lookup table.

10. In Claim 9, In the step of setting the aforementioned prior interval, In the entire section above, the prior interval section is set according to a first criterion that is pre-set, and the set prior interval section is the first interval section (itv res ) and the first X interval (X res A method for processing operations, configured by separating with ).

Images