Arithmetic device and electronic apparatus

The arithmetic unit design in DCiM systems addresses power consumption and transmission delays by optimizing storage and signal line placement, resulting in low-power and high-speed multiply-accumulate operations.

WO2026239906A1PCT designated stage Publication Date: 2026-11-19SONY SEMICON SOLUTIONS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/016518
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-04-24
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing Digital In-Memory-Compute (DCiM) technologies face increased power consumption and transmission delays due to long wiring lengths between shared memory and arithmetic units, leading to reduced calculation speed and operating frequency.

Method used

An arithmetic unit design with multiple first and second storage units, shorter signal lines, and a sum-of-accumulate circuit to perform operations efficiently, reducing power consumption and transmission delays by storing frequently changing operands closer to the multiply-accumulate operation circuit.

Benefits of technology

The design achieves low-power and high-speed multiply-accumulate operations by minimizing power consumption and transmission delays, enhancing the efficiency of DCiM systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026016518_19112026_PF_FP_ABST
    Figure JP2026016518_19112026_PF_FP_ABST
Patent Text Reader

Abstract

[Problem] To realize a high-speed sum-of-products operation with low power consumption. [Solution] An arithmetic device comprises: a plurality of first arithmetic modules that each perform a sum-of-products operation; and a second arithmetic module that adds results of a plurality of sum-of-products operations performed by the plurality of first arithmetic modules. Each of the first arithmetic modules comprises: a first storage unit that stores an inputted first digital signal; a second storage unit that stores an inputted second digital signal; a sum-of-products arithmetic circuit that performs a sum-of-products operation on the first digital signal and the second digital signal; a first signal line that transmits the first digital signal from the first storage unit to the sum-of-products arithmetic circuit; and a second signal line that transmits the second digital signal from the second storage unit to the sum-of-products arithmetic circuit and has a wiring length shorter than that of the first signal line.
Need to check novelty before this filing date? Find Prior Art

Description

Computing units and electronic devices

[0001] This disclosure relates to computing devices and electronic equipment.

[0002] DCiM (Digital Compute-in-Memory, or DIMC: Digital In-Memory-Compute) technology is known to significantly reduce power consumption, such as data movement in memory, by integrating memory and arithmetic units or placing them in close proximity. For example, a simplified DIMC macroarchitecture has been proposed by integrating essential components such as a bit cell array, multiplier, and first-stage adder tree (Non-Patent Literature 1).

[0003] HUNBN, a 1.77MB Digital In-Memory-Compute SoC for edge applications achieving 126 TOPs / W (4b) at macro level and 24 TOPs / W at SoC level. (https: / / ieeexplore.ieee.org / document / 10719477)

[0004] In DCiM, multiply-accumulate operations are typically performed using multiple types (for example, two types) of operands. Furthermore, DCiM's multiply-accumulate operations utilize multiple macros, which are collections of functional blocks. In Non-Patent Literature 1, one of the two operands is stored in a memory located outside the macro. This memory is shared by multiple macros (hereinafter also referred to as the shared memory).

[0005] The macros in Non-Patent Document 1 require the operator to be retrieved from the shared storage. The wiring length between the shared storage and the macros increases as the number of macros increases. This can lead to increased power consumption due to data transmission from the shared storage. Furthermore, long wiring lengths can cause transmission delays. In this case, the calculation speed by the macros decreases, which may necessitate a reduction in the operating frequency (bandwidth) of the entire arithmetic unit containing the macros.

[0006] Therefore, this disclosure provides an arithmetic unit capable of low-power and high-speed multiply-accumulate operations.

[0007] To solve the above problems, the present disclosure provides an arithmetic device comprising: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit, which adds up a plurality of multiply-accumulate operation results performed by the plurality of first arithmetic units, wherein each of the plurality of first arithmetic units comprises: a first storage unit for storing an input first digital signal; a second storage unit for storing an input second digital signal; a multiply-accumulate operation circuit for performing a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line for transmitting the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line for transmitting the second digital signal from the second storage unit to the multiply-accumulate operation circuit.

[0008] The first storage unit stores one or more of the first digital signals, and the second storage unit stores a plurality of the second digital signals. The second storage unit further includes a first selection circuit for selecting some of the first digital signals used in a sum-of-accumulate operation from the first storage unit, and a second selection circuit for selecting some of the second digital signals used in a sum-of-accumulate operation from the second storage unit. The sum-of-accumulate operation circuit may perform a sum-of-accumulate operation on the first digital signals selected by the first selection circuit and the second digital signals selected by the second selection circuit.

[0009] The sum-of-accumulate circuit may further include a third storage unit that holds the first digital signal selected by the first selection circuit, and a fourth storage unit that holds the second digital signal selected by the second selection circuit, wherein the sum-of-accumulate circuit may perform a sum-of-accumulate operation based on the first digital signal held by the third storage unit and the second digital signal held by the fourth storage unit.

[0010] The system may further include a first sense amplifier that senses the first digital signal selected by the first selection circuit, and a second sense amplifier that senses the second digital signal selected by the second selection circuit.

[0011] The first storage unit and the second storage unit may each have SRAM (Static Random Access Memory).

[0012] The first selection circuit selects a plurality of first digital signals, the second selection circuit selects a plurality of second digital signals, the sum-of-accumulate circuit has a plurality of multiplication circuits, and each of the plurality of multiplication circuits may perform a multiplication operation between one of the plurality of first digital signals selected by the first selection circuit and one of the plurality of second digital signals selected by the second selection circuit.

[0013] The sum-of-accumulate circuit may perform at least one of the following: fully connected layer operations, MLP (Multi-Layer Perceptron) operations, attention operations, or optical flow operations.

[0014] The sum-of-accumulate circuit may perform multiple sum-of-accumulate operations based on the fixed first digital signal and the selected multiple second digital signals when the first digital signal selected by the first selection circuit is fixed and the second selection circuit is switching the selection of the second digital signal.

[0015] The first storage unit may have SRAM, and the second storage unit may have memory other than SRAM.

[0016] The second storage unit may have a shift register for storing the second digital signal.

[0017] The sum-of-accumulate circuit performs a convolution operation, and the shift register may shift the second digital signal for each unit of the convolution operation.

[0018] The reading period of the second digital signal may be shorter than the reading period of the first digital signal.

[0019] The sum-of-products circuit includes an product circuit that performs a product operation between the first digital signal and the second digital signal, and a sum circuit that adds the results of the product operation circuit. The first signal line may connect the first storage unit and the product circuit, and the second signal line may connect the second storage unit and the product circuit.

[0020] The product operation circuit may be arranged between the first storage unit and the second storage unit, and the sum operation circuit may be arranged between the product operation circuit and the first storage unit.

[0021] The second storage unit may be located between the product operation circuit and the first storage unit.

[0022] The system comprises a first arithmetic unit group having a plurality of the first arithmetic units, and a fifth storage unit that supplies at least one of the first digital signal or the second digital signal to the plurality of first arithmetic units in the first arithmetic unit group, wherein each of the first arithmetic units in the first arithmetic unit group may have at least one of a first switch that switches between reading the first digital signal from the fifth storage unit or reading the first digital signal from the first storage unit, or a second switch that switches between reading the second digital signal from the fifth storage unit or reading the second digital signal from the second storage unit.

[0023] The system may further include an output circuit that switches the output destination based on the sum-accumulate result of a calculation based on the first digital signal read from the first storage unit and the second digital signal read from the second storage unit, and the sum-accumulate result of a calculation based on the first digital signal or the second digital signal read from the fifth storage unit.

[0024] The system comprises a plurality of second arithmetic units and a group of second arithmetic units having a plurality of first arithmetic units connected to any of the plurality of second arithmetic units, wherein the fifth storage unit is located in the center of the plurality of first arithmetic units in the group of first arithmetic units, and the second arithmetic units may be located in the center of the plurality of first arithmetic units in the group of second arithmetic units.

[0025] If the sum-of-accumulate circuit and the storage unit including the first or second storage unit have substantially the same width in the direction in which the first signal line or the second signal line is arranged, the sum-of-accumulate circuit is arranged so that one side of the storage unit and one side of the storage unit substantially coincide. If the sum-of-accumulate circuit and the storage unit have different widths, the sum-of-accumulate circuit and the storage unit may be arranged so that the smaller side of the storage unit touches the center of the larger side of the sum-of-accumulate circuit and the storage unit.

[0026] Furthermore, according to this disclosure, an electronic device is provided comprising: an imaging unit for capturing image data; and a signal processing unit for performing signal processing on the image data, wherein the signal processing unit comprises: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit for adding up a plurality of multiply-accumulate operation results calculated by the plurality of first arithmetic units, and each of the plurality of first arithmetic units comprises: a first storage unit for storing an input first digital signal; a second storage unit for storing an input second digital signal; a multiply-accumulate operation circuit for performing a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line for transmitting the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line for transmitting the second digital signal from the second storage unit to the multiply-accumulate operation circuit.

[0027] A block diagram showing the configuration of the arithmetic unit according to the first embodiment of this disclosure. A block diagram showing the configuration of the macro in Figure 1. A block diagram showing a more detailed configuration of the macro according to the first embodiment of this disclosure. A block diagram showing the configuration of the macro relating to the first modified example. A block diagram showing the configuration of the macro relating to the second modified example. A block diagram showing the configuration of the macro relating to the third modified example. A block diagram showing the configuration of the macro relating to the fourth modified example. A block diagram showing the configuration of the macro relating to the fifth modified example. A block diagram showing the configuration of the macro relating to the sixth modified example. A block diagram showing the configuration of the macro relating to the seventh modified example. A block diagram showing the configuration of the macro relating to the eighth modified example. A first diagram showing the layout configuration of the macro according to the first embodiment of this disclosure. A second diagram showing the layout configuration of the macro according to the first embodiment of this disclosure. A first diagram showing the layout configuration of the macro relating to the fifth modified example. A second diagram showing the layout configuration of the macro relating to the fifth modified example. A wiring layout configuration of the macro according to the first embodiment of this disclosure. A first diagram showing the layout configuration of the macro relating to a comparative example. A second diagram showing the layout configuration of the macro relating to a comparative example. A diagram showing the wiring layout configuration of the macro relating to a comparative example. A diagram showing a first application example of the macro according to the first embodiment of this disclosure. A diagram illustrating the operation of the macro in Figure 20 according to the first embodiment of this disclosure. A diagram illustrating the operation of the macro in Figure 20 according to a modified example. A diagram showing a second application example of the macro according to the first embodiment of this disclosure. A diagram showing the layout configuration of the arithmetic unit according to a comparative example. A block diagram showing the configuration of the arithmetic unit according to the second embodiment of this disclosure. A block diagram showing the configuration of the shared storage unit and macro according to the basic example. A block diagram showing the configuration of the shared storage unit and macro according to the second embodiment of this disclosure. A block diagram showing the configuration of the shared storage unit and macro according to the first modified example. A block diagram showing the configuration of the shared storage unit and macro according to the second modified example. A block diagram showing the configuration of the shared storage unit and macro according to the third modified example. A diagram showing a first application example of the macro in Figure 27. A diagram showing a second application example of the macro in Figure 27. A block diagram showing the configuration of the arithmetic unit according to the fourth modified example. A block diagram showing the configuration of the arithmetic unit according to the fifth modified example. A block diagram showing the configuration of the arithmetic unit according to the sixth modified example. A diagram showing the layout configuration of multiple macros, a shared storage unit, and an OR operation circuit.A block diagram showing an example configuration of an imaging device according to the third embodiment of this disclosure. A perspective view showing an overview of the external configuration example of the imaging device in Figure 37. A block diagram showing an example of a schematic configuration of a vehicle control system. An explanatory diagram showing an example of the installation positions of the external information detection unit and the imaging unit.

[0028] Embodiments of the arithmetic unit and electronic equipment will be described below with reference to the drawings. While the main components of the arithmetic unit and electronic equipment will be described below, there may be components and functions not shown or described. The following description does not exclude any components or functions not shown or described.

[0029] (First Embodiment) Figure 1 is a block diagram showing the configuration of a computing device 1 according to the first embodiment of the present disclosure. The computing device 1 in Figure 1 has a plurality of DCIM macros (DCIM macrocells, hereinafter also referred to as macro 10). The computing device 1 can be configured with, for example, an IC (Integrated Circuit).

[0030] The arithmetic unit 1 can be applied to any multiply-accumulate operation in machine learning, etc. Specifically, the arithmetic unit 1 can be applied to, for example, DNNs (Deep Neural Networks). The arithmetic unit 1 can perform various multiply-accumulate operations, such as multiply-accumulate operations in convolutional layers or fully-connected layers.

[0031] A macro (first arithmetic unit) 10 is, for example, a collection of functional blocks that combine memory cells and arithmetic cells. Multiple macros 10 with identical internal configurations and sizes can be placed in the arithmetic unit 1. Such macros 10 may be registered, for example, in a design library. In this case, the macros 10 can be easily duplicated during the design phase of the arithmetic unit 1. Furthermore, the macros 10 may be reused in other arithmetic units. Note that the arithmetic unit 1 may contain multiple macros 10 with different internal configurations or sizes.

[0032] Macro 10 can perform at least a portion of the sum-of-products operation of the arithmetic unit 1. For example, multiple macros 10 perform sum-of-products operations in parallel. The sum-of-products results of multiple macros 10 can be added together, for example, using an adder circuit (not shown in Figure 1), to obtain the sum-of-products result of the arithmetic unit 1. Note that the arithmetic unit 1 may contain a mixture of macros 10 that perform different sum-of-products operations. Also, the arithmetic unit 1 may contain a mixture of macros 10 that perform sum-of-products operations on different layers. In this case, one group of macros 10 (belonging to the first layer) may perform sum-of-products operations based on the sum-of-products results of another group of macros 10 (belonging to the second layer).

[0033] The macro 10 in Figure 1 comprises a first storage unit 2, a second storage unit 3, and a sum-of-accumulate circuit 4, respectively.

[0034] The first storage unit 2 stores operands (first operands) A1, A2, . . . n The second memory unit 3 stores the operands (hereinafter collectively referred to as operand A). The second memory unit 3 stores operands (second operands) B1, B2, ... B m The sum-of-products circuit 4 stores (hereinafter collectively referred to as operand B). The sum-of-products circuit 4 performs a sum-of-products operation on operand A and operand B. The sum-of-products circuit 4 in Figure 1 is located between the first storage unit 2 and the second storage unit 3.

[0035] The first storage unit 2 and the second storage unit 3 include, for example, SRAM (Static Random Access Memory), flip-flops, shift registers, or latch arrays. Hereinafter, unless otherwise specified, the specification will describe examples in which the first storage unit 2 and the second storage unit 3 have SRAM. SRAM enables high-speed operation in data writing and reading and does not require refreshing.

[0036] Multiple macros 10 are arranged, for example, within the calculation area 5. Alternatively, the calculation unit 1 may have a configuration that includes a shared storage unit (fifth storage unit) 6 shared by the multiple macros 10.

[0037] As shown in Figure 1, in macro 10, operands A and B used in the multiply-accumulate operation are stored in the first storage unit 2 and the second storage unit 3, respectively, located near the multiply-accumulate operation circuit 4. This allows the arithmetic unit 1 to reduce the power consumption of data transfer compared to when operands A or B are transferred from the shared storage unit 6.

[0038] Figure 2 is a block diagram showing the configuration of macro 10 in Figure 1. The multiply-accumulate circuit 4 includes an product circuit 7 and a sum circuit 8.

[0039] The product circuit 7 calculates the product of each element of operand A and each element of operand B. The product circuit 7 includes logic elements such as an AND circuit, a NAND circuit, or a NOR circuit. The product circuit 7 may also include a lookup table (LUT).

[0040] The sum operation circuit 8 adds up the results of the multiplication operation circuit 7. The sum operation circuit 8 includes, for example, a full adder or a half adder. The sum operation circuit 8 includes, for example, a combination of an adder tree and an accumulator.

[0041] As shown in Figure 2, the product operation circuit 7 is located on the second storage unit 3 side. The sum operation circuit 8 is located between the first storage unit 2 and the product operation circuit 7.

[0042] A signal line (first signal line) La extends from the first memory unit 2 to the multiplication circuit 7, transmitting a signal containing information about operand A. A signal line (second signal line) Lb extends from the second memory unit 3 to the multiplication circuit 7, transmitting a signal containing information about operand B. Signal line La passes through the summation circuit 8 and connects to the multiplication circuit 7. Signal line La is longer than signal line Lb.

[0043] The second memory unit 3 stores the operand with the higher change frequency (shorter reading cycle) among the multiple operands used in the sum-of-accumulate operation (i.e., operands A and B). Here, the change frequency refers to the frequency at which the content of the operand read by the product operation circuit 7 is changed (hereinafter also referred to as the reading frequency).

[0044] For example, suppose the first product operation is performed in the product circuit 7 between operands A1, A2, and A3 and operands B1, B2, and B3. Then, suppose the second product operation is performed in the product circuit 7 between operands A1, A2, and A3 and operands B4, B5, and B6. In this case, the contents of operand B have been changed.

[0045] In the example above, different signals are transmitted on the signal line Lb between the first and second multiplication operations. Therefore, it is necessary to set different voltages on the signal line Lb between the first and second multiplication operations. On the other hand, on the signal line La, the signals transmitted between the first and second multiplication operations are the same, so it is not necessary to set different voltages.

[0046] The second memory unit 3 stores operands that change more frequently (i.e., operand B). Therefore, the signal line Lb experiences a higher frequency of voltage charging and discharging during sum-of-products operations than the signal line La. On the other hand, the signal line Lb has a shorter wiring length than the signal line La. Consequently, the power consumption due to voltage charging and discharging during sum-of-products operations on the signal line Lb is smaller than the power consumption due to voltage charging and discharging during sum-of-products operations on the signal line La.

[0047] In other words, in macro 10 of Figure 2, the operand B, which is more frequently changed, is stored in the second storage unit 3, which is located closer to the multiplication circuit 7 than the first storage unit 2. This reduces the power consumption of the multiply-accumulate operation compared to storing the operand B in the first storage unit 2.

[0048] The frequency of changes between operand A and operand B may be approximately the same. In this case, operand A may be stored in the second storage unit 3, and operand B may be stored in the first storage unit 2.

[0049] Figure 3 is a block diagram showing a more detailed configuration of macro 10 according to the first embodiment of the present disclosure. The first storage unit 2 includes a bit cell array 11, a decoder (first selection circuit) 12, a plurality of sense amplifiers (SAs) 13, a plurality of latches (third storage units) 14, and a control circuit 15.

[0050] Furthermore, the horizontal direction in Figure 3 is called the first direction X, and the vertical direction is called the second direction Y. The first direction X is the direction in which the multiple signal lines La and Lb are arranged. The second direction Y is the direction in which the multiple signal lines La and Lb extend.

[0051] The bit cell array 11 stores the operand A. The decoder 12 decodes the address of the bit cell array 11. Based on the decoded address, the decoder 12 outputs the operand A used in the multiply-accumulate operation from the bit cell array 11.

[0052] The sense amplifier 13 amplifies the signal from the bit cell array 11 and transmits it to the latch 14. The latch 14 temporarily holds the signal output by the bit cell array 11. That is, the latch 14 stores the operand A selected by the decoder 12. At the timing of the product operation, the latch 14 outputs the held signal to the product operation circuit 7. The control circuit 15 controls the sense amplifier 13 and the latch 14. The sense amplifier 13 and the latch 14 are arranged in the first direction X.

[0053] Furthermore, Figure 3 illustrates multiple latch groups 16, each containing multiple latches 14. Each of the multiple latch groups 16 stores, for example, one operand (first digital signal) A. The multiple latches 14 within the latch group 16 correspond, for example, to each bit of the operand A.

[0054] The second storage unit 3, like the first storage unit 2, includes a bit cell array 21, a decoder (second selection circuit) 22, a plurality of sense amplifiers 23, a plurality of latches (fourth storage unit) 24, and a control circuit 25. The bit cell array 21 stores the operand (second digital signal) B. Figure 3 also illustrates a plurality of latch groups 26, each containing a plurality of latches 24, and each storing one operand B.

[0055] Figure 3 illustrates multiple multiplication circuits 7. Each multiplication circuit 7 performs the multiplication operation on operands A and B read from the corresponding latch groups 16 and 26. Multiple signal lines La and signal lines Lb are connected to each multiplication circuit 7. The number of signal lines La connected to the multiplication circuit 7 corresponds, for example, to the number of bits in one operand A. The number of signal lines Lb connected to the multiplication circuit 7 corresponds, for example, to the number of bits in one operand B.

[0056] Figure 3 shows an example where the number of bits (bit width) of operand A and the number of bits of operand B are the same (for example, X bits). In this case, the same number of signal lines La and Lb are connected to the product circuit 7.

[0057] In this specification, the bit cell arrays 11 and 21 may be referred to as the first storage unit and the second storage unit, respectively. Furthermore, the sense amplifier 13 is arranged, for example, for each bit of the operand A. However, if the operand A consists of multiple bits, the group of sense amplifiers for each operand A may be referred to as a single sense amplifier 13 (in this case, a sense amplifier 13 is arranged for each operand A). The same applies to the sense amplifier 23. Also, the latch groups 16 and 26 may be referred to as a single latch 14 and 24.

[0058] Figure 4 is a block diagram showing the configuration of macro 10a according to the first modified example. In Figure 4, the number of bits of operand A and the number of bits of operand B are different. In Figure 4, the number of bits of operand A is X bits, while the number of bits of operand B is Y bits. The product operation circuit 7 is connected, for example, to a number of signal lines La corresponding to X bits and a number of signal lines Lb corresponding to Y bits. Figure 4 shows an example where X > Y.

[0059] The macro 10a in Figure 4 has multiple switches 31 between the second storage unit 3 and the product operation circuit 7. Each switch 31 is connected to multiple latch groups 26. When X is n times Y, the switch 31 is connected to, for example, n latch groups 26. The switch 31 selects a latch group 26 according to the required product operation and connects it to the product operation circuit 7.

[0060] In macro 10a of Figure 4, the number of signal lines La and Lb arranged in the first direction X can be made the same. Also, the number of latches 14 and 24 can be made the same. As a result, the width of the first storage unit 2 and the width of the second storage unit 3 can be made to match.

[0061] Note that the number of bits Y of operand B may be greater than the number of bits X of operand A (Y > X). In this case, a switch 31 connected to a group of latches 16 may be placed between the first storage unit 2 and the product operation circuit 7. Also, in macro 10a, a switch 31 may be placed between the first storage unit 2 and the product operation circuit 7, and between the second storage unit 3 and the product operation circuit 7.

[0062] Figure 5 is a block diagram showing the configuration of macro 10b according to the second modification. In macro 10b of Figure 5, sense amplifiers 13 and 23 are omitted. Macro 10b can be applied when the number of data points (e.g., number of data rows) in the bit cell arrays 11 and 21 is small and the charging and discharging times of signal lines La and Lb are short. Macro 10b can be made more space-saving compared to macro 10 and the like.

[0063] Figure 6 is a block diagram showing the configuration of macro 10c according to a third modified example. In macro 10c of Figure 6, latches 14 and 24 are omitted compared to macro 10b of Figure 5. As a result, macro 10c is more space-efficient than macro 10b, etc. Signal lines La and Lb directly connect the product operation circuit 7 and the bit cell arrays 11 and 21.

[0064] Figure 7 is a block diagram showing the configuration of macro 10d according to the fourth modification. In macro 10d of Figure 7, the configurations of the first storage unit 2 and the second storage unit 3a are different. Specifically, the first storage unit 2 has SRAM, and the second storage unit 3a has memory other than SRAM (for example, a shift register). In macro 10d of Figure 7, the advantages of both SRAM and other memory can be utilized. In macro 10d of Figure 7, the latch 14 may be omitted. Alternatively, macro 10d may have a configuration that includes a sense amplifier 13.

[0065] Figure 8 is a block diagram showing the configuration of macro 10e according to the fifth modified example. Macro 10e in Figure 8 differs from macro 10a in Figure 5 in the arrangement of the first storage unit 2, the second storage unit 3, and the multiply-accumulate circuit 4. Specifically, the second storage unit 3 is placed between the first storage unit 2 and the multiply-accumulate circuit 4 (also called a vertically stacked storage unit configuration). The multiplicative circuit 7 is placed between the second storage unit 3 and the summation circuit 8.

[0066] The signal line La extending from the first memory unit 2 passes through the second memory unit 3 and is connected to the multiplication circuit 7. Unlike the signal line La in Figure 3, the signal line La in Figure 8 does not need to pass through the summation circuit 8.

[0067] The configuration shown in Figure 8 is preferably applicable when the vertical width ht of the sum operation circuit 8 is large. The sum operation circuit 8 has a vertical width ht corresponding to, for example, the number of stages in the adder tree. For example, an adder tree that calculates the sum of eight data points has 8 = 2 3 It has a three-stage adder.

[0068] For example, if the vertical width ht of the sum arithmetic circuit 8 is greater than the vertical width hm of the second memory unit 3, the configuration in Figure 8 may allow for a shorter wiring length of the signal line La than the configuration in Figure 3. This can reduce the power consumption of the arithmetic unit 1. However, even if the vertical width ht of the sum arithmetic circuit 8 is small, the sum arithmetic circuit 8 may have factors that increase the wiring length, such as the signal line La needing to make a large detour to avoid adders, etc. In such cases, the configuration in Figure 8 may be preferable to the configuration in Figure 3. On the other hand, if the vertical width hm of the second memory unit 3 is greater than the vertical width ht of the sum arithmetic circuit 8, the configuration in Figure 3 may be preferable to the configuration in Figure 8.

[0069] Furthermore, in macro 10e of Figure 8, the first storage unit 2 and the second storage unit 3 are arranged in close proximity, which allows for a reduction in the wiring length of the signal lines BL and BLB, which will be described later.

[0070] Figure 9 is a block diagram showing the configuration of macro 10f according to the sixth modified example. Macro 10f in Figure 9 has a vertical stacking configuration of storage units, similar to Figure 8. In macro 10f in Figure 9, latch 14 and latch 24 are arranged adjacent to each other. Latches 14 and 24 are arranged between bit cell array 11 and bit cell array 21.

[0071] In Figure 9, the first storage unit 2 and the second storage unit 3 share a common control circuit because the latches 14 and 24 are arranged adjacent to each other. The macro 10f in Figure 9 has a control circuit 32 that controls the latches 14 and 24.

[0072] In macro 10f of Figure 9, by sharing the control circuit between the first storage unit 2 and the second storage unit 3 (i.e., control circuit 32), the control circuit can be made more compact than in a configuration where each unit has its own control circuit (i.e., control circuits 15 and 25 in Figure 8, etc.).

[0073] Figure 10 is a block diagram showing the configuration of macro 10g according to the seventh modified example. Macro 10g in Figure 10 has a laminated structure including a first layer Sa and a second layer Sb. Bit cell arrays 11 and 21 are arranged in the first layer Sa. The second layer Sb is, for example, a metal layer (wiring layer).

[0074] Bit cell arrays 11 and 21 each contain a plurality of bit cells. A bit cell has, for example, a flip-flop circuit. Each bit cell is connected to a signal line BL that transmits the signal of operand A or B. Each bit cell is also connected to a signal line BLB. A signal complementary to signal line BL is transmitted on signal line BLB.

[0075] Figure 10 illustrates the signal lines BLa and BLBa connected to the bit cells in the bit cell array 11, and the signal lines BLb and BLBb connected to the bit cells in the bit cell array 21. Signal lines BLa and BLb correspond to the signal line BL mentioned above. Signal lines BLBa and BLBb correspond to the signal line BLB mentioned above.

[0076] In the second layer Sb of Figure 10, either signal line BLa or BLb (for example, signal line BLb) is placed. Also, either signal line BLBa or BLBb (for example, signal line BLBa) is placed in the second layer Sb. The other of signal line BLLa or BLb, and the other of signal lines BLBa and BLBb are placed in the first layer Sa.

[0077] In macro 10g of Figure 10, signal lines BLa and BLb are placed on different layers, and signal lines BLBa and BLBb are placed on different layers. This reduces the area of ​​macro 10g.

[0078] Figure 11 is a block diagram showing the configuration of macro 10h according to the eighth modified example. Macro 10h in Figure 11 has a first layer Sa and a second layer Sb. Bit cell arrays 11 and 21 and latches 14 and 24 are arranged in the first layer Sa of Figure 11. At least a portion of signal lines La and Lb are arranged in the second layer Sb of Figure 11.

[0079] In macro 10h of Figure 11, signal lines La and Lb are placed on a different layer from the bit cell arrays 11 and 21, etc. This eliminates the need for signal lines La and Lb to pass through the bit cell arrays 11 and 21, latches 14 and 24.

[0080] In macros 10g and 10h of Figures 10 and 11, latches 14 and 24 can be placed adjacent to the product operation circuit 7. As shown in Figures 10 and 11, latches 14 and 24 may be placed between the bit cell arrays 11 and 21 and the product operation circuit 7.

[0081] In macros 10g and 10h of Figures 10 and 11, latches 14 and 24 are positioned closer to the product circuit 7 than in macro 10f of Figure 9. This shortens the wiring length between latches 14 and 24 and the product circuit 7, thereby reducing power consumption.

[0082] In macro 10g of Figure 10, the signal lines La and Lb may be placed in the second layer Sb, similar to macro 10h of Figure 11. Alternatively, they may be placed in layers different from the first layer Sa and the second layer Sb.

[0083] Figure 12 is a first diagram showing the layout configuration of macro 10 according to the first embodiment of the present disclosure. In Figure 12, the width (hereinafter also referred to as the horizontal width) of the multiply-accumulate circuit 4 in the first direction X is approximately the same as the horizontal width of the second storage unit 3.

[0084] In this case, it is desirable that the side Ea of the multiply-accumulate circuit 4 facing the second memory unit 3 (hereinafter also referred to as one side of the memory unit) and the side Eb of the second memory unit 3 facing the multiply-accumulate circuit 4 coincide. Here, "sides Ea and Eb coincide" means that they are arranged so that side Eb (or Ea) does not shift in the first direction X relative to side Ea (or Eb). Sides Ea and Eb may be separated in the second direction Y. This allows for optimization of the wiring layout, as will be described later. It is also preferable to arrange the first memory unit 2 and the multiply-accumulate circuit 4 in a similar manner.

[0085] Figure 13 is a second diagram showing the layout configuration of macro 10 according to the first embodiment of the present disclosure. In Figure 12, the widths of the multiply-accumulate circuit 4 and the second storage unit 3 are different. In this case, for the optimization of the wiring layout, it is desirable that the shorter of the two sides, Ea and Eb, be placed in the center of the longer side.

[0086] Figure 14 is the first diagram showing the layout configuration of macro 10e according to the fifth modified example. Figure 15 is the second diagram showing the layout configuration of macro 10e according to the fifth modified example. The macro 10e in Figures 14 and 15 have a vertical stacking configuration of storage units, similar to Figure 8. In this case as well, similar to Figures 12 and 13, it is desirable that sides Ea and Eb are arranged to coincide, or that the shorter side is placed in the center of the longer side.

[0087] Figure 16 shows the wiring layout configuration of macro 10 according to the first embodiment of this disclosure. Below, an example in which the second storage unit 3 and the multiply-accumulate circuit 4 are arranged as shown in Figures 12 to 15 will be described. When the widths of the second storage unit 3 and the multiply-accumulate circuit 4 are the same, the signal line Lb can be wired in a substantially straight line parallel to the second direction Y, as shown in Figure 16. This shortens the wiring length of the signal line Lb and reduces the variation in the wiring lengths of multiple signal lines Lb (i.e., the wiring layout can be optimized). Even if the widths of the second storage unit 3 and the multiply-accumulate circuit 4 are not the same, it is still possible to shorten the wiring length of the signal line Lb and reduce the variation in wiring length. The same applies to the first storage unit 2 and the signal line La.

[0088] Figure 17 is a first diagram showing the layout configuration of macro 100 according to one comparative example. Figure 18 is a second diagram showing the layout configuration of macro 100 according to one comparative example. In the macro 100 of Figure 17, similar to Figure 12, the multiply-accumulate circuit 4 is arranged between the first storage unit 2 and the second storage unit 3. In the macro 100 of Figure 18, similar to Figure 14, the storage units are stacked vertically.

[0089] In macro 100 in Figures 17 and 18, edges Ea and Eb do not coincide. Furthermore, the shorter edge of Ea and Eb is not positioned in the center of the longer edge. Edge Eb is offset from edge Ea in the first direction X.

[0090] Figure 19 shows the wiring layout configuration of macro 100 according to one comparative example. In Figure 19, side Eb is positioned to the right of side Ea. As a result, there is variation in the wiring length of multiple signal lines Lb. Specifically, the signal lines Lb positioned on the left side of Figure 19 have shorter wiring lengths, while the signal lines Lb positioned on the right side of Figure 19 have longer wiring lengths.

[0091] Furthermore, the multiplication circuit 7 is positioned to the right of the second memory unit 3. Therefore, the signal line Lb cannot be routed in a straight line parallel to the second direction Y, and thus the wiring length cannot be shortened. The same applies to the signal line La.

[0092] In other words, the wiring layout configuration shown in Figure 16 allows for shorter wiring lengths and reduces variations in wiring lengths compared to the wiring layout configuration shown in Figure 19.

[0093] Figure 20 shows a first application example of macro 10 according to the first embodiment of this disclosure. Figure 20 illustrates an example of applying macro 10 to a fully-connected layer operation. The left side of Figure 20 shows a fully-connected layer.

[0094] The fully coupled layer in Figure 20 has input data (first data, or first digital signal) a1, a2, ... a n The following is input. Also, in a fully connected layer, any input coefficient a i Regarding this, weight data (second data, or second digital signal) bi,1 , b i,2 , ... b i,n and a product operation therewith is performed. The output data of the fully connected layer (third data, or third digital signal) c₁, c₂, ... c n are respectively calculated by the sum-of-products operation of Σb i,1 a i , Σb i,2 a i , ... Σb i,n a i .

[0095] The first storage unit 2 in FIG. 20 stores input data a₁ to a n . In addition, the second storage unit 3 stores weight data b 1,1 to b n,n . Further, the sum-of-products operation circuit 4 performs a sum-of-products operation on the input data a₁ to a n and b 1,j to b n,j in one sum-of-products operation, and outputs output data c j .

[0096] In the above case, in the sum-of-products operation for the output data c₁ to c n (that is, n sum-of-products operations), after the input data a₁ to a n are read once from the first storage unit 2, there is no need to change the input data. On the other hand, the weight data b 1,j to b n,j are changed for each sum-of-products operation.

[0097] Specifically, assume that the weight data b 1,1 to b n,1 (that is, j=1) are read from the second storage unit 3, and the sum-of-products operation for the output data c₁ is performed. Next, in order to perform the sum-of-products operation for the output data c₂ (that is, j=2), the weight data b 1,2 to b n,2 are read out.

[0098] That is, the weight data b 1,j to b n,j have a higher change frequency than the input data a₁ to a n .

[0099] Note that the input data a₁ to a nThe data may be read from the first memory unit 2 in multiple steps.

[0100] Figure 21 is a diagram illustrating the operation of macro 10 in Figure 20 according to the first embodiment of the present disclosure. The horizontal axis of Figure 21 represents the timeline. In Figure 21, input data a1 to a n However, an example is shown where the data is read from the first memory unit 2 in two separate steps.

[0101] First, the second memory unit 3 stores the weight data b 1,1 ~b n,n The data is written (Write Factor, step S1). Subsequently, the input data a1 to a are stored in the first storage unit 2. n The data is written (Write Data, step S2). The execution order of steps S1 and S2 may be reversed, or steps S1 and S2 may be executed approximately simultaneously. In steps S1 and S2, for example, data may be written from the shared storage unit 6.

[0102] Next, a portion of the input data a1 to a from the first storage unit 2 k The data is read (Read Data, step S3). At the same time (for example, almost simultaneously), a portion of the weight data b is read from the second storage unit 3. 1,1 ~b k,1 The data is read out (Read Factor, step S4). The sum-of-accumulate circuit 4 calculates Σb based on the data read out in steps S3 and S4. i,1 a i A sum-of-products operation is performed on (1 ≤ i ≤ k) (Calculation, step S5). The result of the sum-of-products operation in step S5 is stored in a predetermined storage unit. The result of the sum-of-products operation in step S5 may be stored in the first storage unit 2, or it may be stored in another temporary storage unit, etc.

[0103] Next, a portion of the weight data b from the second memory unit 3. 1,2 ~b k,2 The data is read out (Read Factor, step S6). The sum-of-accumulate circuit 4 reads out the input data a1 to a in step S3. k And the weight data b read out in step S6 1,2 ~b k,2 Based on this, Σb i,2a i Perform a sum-of-products operation on (1 ≤ i ≤ k) (Calculation, step S7).

[0104] Even after steps S6 and S7, the weight data b from the second storage unit 3 continues to be transmitted. 1,j ~b k,j Readout and sum-of-products operation Σb by the sum-of-products operation circuit 4. i,j a i The condition (1 ≤ i ≤ k) is repeated multiple times. During the nth sum-of-products operation, weight data b is taken from the second memory unit 3. 1,n ~b k,n The sum of products is read out, and the sum of products circuit 4 performs the sum of products operation Σb i,n a i The operation (1 ≤ i ≤ k) is performed (steps S11 and S12). The sum-of-products results are stored in a predetermined storage unit (for example, the first storage unit 2).

[0105] Next, a portion of the remaining input data from the first storage unit 2. k+1 ~a n The data is read out (step S13). At the same time (for example, almost simultaneously), a portion of the weight data b is read out from the second storage unit 3. k+1,1 ~b n,1 The data is read out (step S14). Based on the data read out in steps S13 and S14, the sum-of-accumulate circuit 4 calculates Σb i,1 a i Perform a sum-of-products operation on (k+1 ≤ i ≤ n) (step S15).

[0106] After steps S14 and S15, the reading of data from the second storage unit 3 and the sum-of-products operation by the sum-of-products operation circuit 4 are repeated in the same manner. During the nth sum-of-products operation, weight data b is read from the second storage unit 3. k+1,n ~b n,n The sum of products is read out, and the sum of products circuit 4 performs the sum of products operation Σb i,n a i Perform the following steps (k+1≦i≦n): (Steps S21 and S22).

[0107] Next, the sum-of-accumulate circuit 4 outputs data c1 to c based on the results of multiple calculations up to step S22. n Output (step S23). For example, output data c1 = Σb i,1 a i(1 ≤ i ≤ n) can be calculated by adding the sum-of-products result of step S5 and the sum-of-products result of step S15. Similarly, by adding multiple sum-of-products results from step S5 onwards to the corresponding multiple sum-of-products results from step S15 onwards, output data c2 to c n These are calculated respectively. Output data c1 to c n This may be stored, for example, in the first memory unit 2.

[0108] The sum-of-products operation in step S15 may include adding the sum-of-products result to the sum-of-products result of step S5 read from the first storage unit 2, etc. The added sum-of-products result may also be stored in the first storage unit 2, etc. Similarly, the sum-of-products operations from step S15 onward may include adding the sum-of-products result to the corresponding sum-of-products result read from the first storage unit 2, etc., and storing it in the first storage unit 2, etc. In this case, step S23 may be omitted.

[0109] Macro 10 can perform the next fully connected layer operation after step S23 (steps S31 to S33). In step S31, input data a1 to a k For example, c1 to c calculated in step S23 k However, data may be read from the first storage unit 2. In addition, other input data stored in the first storage unit 2 may be read. Alternatively, before step S31, new input data may be stored in the first storage unit 2, similar to step S2.

[0110] In step S32, weight data b is received from the second storage unit 3. 1,1 ~b k,1 The data is read out. The data read out in step S32 may be the same as the data read out in step S4, or it may be different. Also, before step S32, new weight data may be stored in the second storage unit 3, similar to step S1. In step S33, the sum-of-products operation circuit 4 performs a sum-of-products operation based on the data read out in steps S31 and S32. Thereafter, the sum-of-products operation is repeated in the same manner as in steps S4 to S23, thereby generating new output data c1 to c n This can be calculated.

[0111] The frequency of Read Data in FIG. 21 corresponds to the change frequency of the input data a1 to a n . The frequency of the Read Factor corresponds to the change frequency of the weight data b 1,j to b n,j . As shown in FIG. 21, even when the input data a1 to a n are read out from the first storage unit 2 in a plurality of times, the weight data b 1,j to b n,j have a higher change frequency than the input data a1 to a n .

[0112] That is, in the operation of FIG. 21, the frequency at which the sum-of-products operation circuit 4 reads the weight data b 1,j to b n,j from the second storage unit 3 (read frequency) is higher than the frequency at which the sum-of-products operation circuit 4 reads the input data a1 to a n from the first storage unit 2 (read frequency).

[0113] In other words, in the operation of FIG. 21, while the decoder 12 fixes the selection of the input data a1 to a n , the decoder 22 switches the selection of the weight data b 1,j to b n,j . The sum-of-products operation circuit 4 performs a plurality of sum-of-products operations based on the fixed input data a1 to a n and the plurality of selected weight data b 1,j to b n,j .

[0114] FIG. 21 illustrates an example in which the input data a1 to a n are read out from the first storage unit 2 in two separate times. The present invention is not limited to this, and the input data a1 to a n may be read out in one time, or may be read out in three or more separate times.

[0115] In this specification, the input data a1 to a k and the weight data b 1,j to b k,j used for one sum-of-products operation may be referred to as one first digital signal and one second digital signal, respectively. In the operation of FIG. 21, the weight data b 1,j to bk,j This is read out multiple times. In other words, the second storage unit 3 stores multiple second digital signals. Input data a1 to a k The data may be read in multiple steps or in a single step (i.e., k = n). In other words, the first storage unit 2 stores one or more first digital signals.

[0116] As shown in Figure 3, the sum-of-accumulate circuit 4 may have multiple product circuits 7. In this case, in this specification, the input data and weight data used in the product operation of each of the multiple product circuits 7 may be referred to as one first digital signal and one second digital signal, respectively. That is, multiple first digital signals and multiple second digital signals may be used in a single sum-of-accumulate operation.

[0117] Furthermore, macro 10 in Figure 20 can also be applied to MLP (Multi-Layer Perceptron) operations. MLP includes multiple fully connected layers. In other words, the left side of Figure 20 can be said to illustrate a part of the MLP operation. Macro 10 can perform MLP operations by performing the above fully connected layer operations multiple times.

[0118] Figure 22 is a diagram illustrating the operation of macro 10 in Figure 20 in a modified example. Figure 22 illustrates an example where the frequency of Read Data and the frequency of Read Factor are approximately the same.

[0119] Specifically, as in Figure 21, data is written to the first storage unit 2 and the second storage unit 3 (steps S1 and S2). Subsequently, a portion of the input data a1 to a is written from the first storage unit 2. k The data is read out (step S3). In addition, a portion of the weight data b is read out from the second storage unit 3. 1,1 ~b k,1 The data is read out (step S4). The sum-of-accumulate circuit 4 performs a sum-of-accumulate operation based on the data read out in steps S3 and S4, and stores the calculation result in the first storage unit 2, etc. (step S5).

[0120] Next, a portion of the remaining input data from the first storage unit 2. k+1 ~a nThe data is read out (step S41). Also, a portion of the weight data b is read from the second storage unit 3. k+1,1 ~b n,1 The data is read out (step S42). Based on the data read out in steps S41 and S42, the sum-of-accumulate circuit 4 calculates Σb i,1 a i Perform a sum-of-products operation on (k+1 ≤ i ≤ n) (step S43). The result of the sum-of-products operation in step S43 may be added to the result of the sum-of-products operation in step S5.

[0121] In the operation shown in Figure 22, based on the sum-of-products calculation results in steps S5 and S43, the output data c1 = Σb i,1 a i It is possible to calculate (1 ≤ i ≤ n) (step S44). Similarly, by repeatedly reading data from the first storage unit 2 and the second storage unit 3 and performing the sum-of-products operation by the sum-of-products operation circuit 4, output data c2 to c n These are calculated sequentially.

[0122] The operation shown in Figure 22 is as follows: output data c1 to c n This is effective when you want to calculate a part of it quickly. For example, it is effective when the fully connected layer calculation of one layer and the fully connected layer calculation of the next layer are performed in parallel. In this case, the output data c1 to c calculated during the processing in Figure 22 n This can then be used in the fully connected layer calculations of the next layer, thereby speeding up the overall calculations.

[0123] On the other hand, the operation in Figure 21 can reduce power consumption compared to the operation in Figure 22. In Figure 22, the frequency (number of read data operations) and the frequency (number of read factor operations) are approximately the same. In contrast, in Figure 21, the frequency (number of read data operations) can be made n / 2 times that of the frequency (number of read factor operations). That is, in the operation in Figure 21, the power consumption for reading from the first storage unit 2 can be made n / 2 times that of the operation in Figure 22.

[0124] Macro 10 in Figure 20 and the operation in Figure 21 (or Figure 22) can also be applied to operations other than fully connected layers. For example, in the Attention operation (Transformer), Q = XWq K = XW k , and V = XW v The following calculation is performed. Q is the query, K is the key, V is the value, X is the input word, and W q , W k , and W v This is the weight data. In the three equations above, the input word X is used repeatedly. For this reason, the input word X is stored in the first storage unit 2, and the weight data W is stored in the second storage unit 3. q , W k , and W v It is preferable to store this information.

[0125] Furthermore, macro 10 in Figure 20 can be applied to Optical Flow, for example. In Optical Flow, feature maps that are changed less frequently (used repeatedly) may be stored in the first storage unit 2, and feature maps that are changed more frequently may be stored in the second storage unit 3.

[0126] The macro 10 in Figure 20 may be applied to various calculations, but is not limited to the above.

[0127] Figure 23 shows a second application example of macro 10 according to the first embodiment of this disclosure. Figure 23 illustrates an example of applying macro 10d to a CNN (Convolutional Neural Network) operation.

[0128] Macro 10d in Figure 23, similar to Figure 7, allows for different configurations of the first storage unit 2 and the second storage unit 3a. The first storage unit 2 may include, for example, an SRAM. The second storage unit 3a may include, for example, a shift register 3b.

[0129] CNN operations involve one or more convolution operations. Figure 23 illustrates the kernel (filter) and input data used in the convolution operation. For example, a k x k size rectangular kernel is used as the kernel. The kernel in Figure 23 is element data a 1,1 ~a k,k It has.

[0130] The first storage unit 2 contains the kernel element data a in the convolution operation.1,1 ~a k,k It can store the following. Multiple kernels (input channels) may be used in the convolution operation. Multiple kernels may be stored in the first storage unit 2 at once, or multiple kernels may be stored in multiple separate operations.

[0131] As input data for Figure 23, for example, n x m size (pixel) image data is used. Alternatively, as input data, a feature map or the like, obtained by extracting features from image data through convolution operations, may also be used. The input data is element data (hereinafter also referred to as pixel data) b 1,1 ~b n,m It has.

[0132] In a convolution operation, a sum-of-products operation is performed between a subset of elements extracted from the input data and elements of the kernel. In a convolution operation, the extraction position of the input data is shifted one element at a time in one direction (for example, to the right in Figure 23) (hereinafter also referred to as sliding), and the above sum-of-products operation is performed.

[0133] For example, first, pixel data b 1,1 ~b k,k The extracted element data and the kernel element data a are extracted. 1,1 ~a k,k A sum-of-products operation is performed. Next, the extraction position of the input data is slid. Next is the pixel data b 2,1 ~b k+1,k The extracted element data and the kernel element data a are extracted. 1,1 ~a k,k A sum-of-products operation is performed.

[0134] In macro 10d of Figure 23, the above sliding operation is realized using the shift register 3b. First, the pixel data is stored in the second storage unit 3a. As described above, the pixel data is changed more frequently than the kernel element data.

[0135] First, pixel data b 1,1 ~b k,k This is stored in the shift register 3b. The multiply-accumulate circuit 4 reads out the kernel element data a from the first storage unit 2. 1,1 ~a k,kAnd the pixel data b read from shift register 3b 1,1 ~b k,k Based on this, Σa i,j b i,j Perform a sum-of-products operation on (1 ≤ i ≤ k, 1 ≤ j ≤ k).

[0136] The shift register 3b performs FIFO (First In First Out) operation. The first pixel data stored is b. 1,1 ~b k,k Of these, pixel data b 1,1 ~b 1,k It is treated as the oldest (i.e., the first data entered) data. Also, pixel data b k,1 ~b k,k It is treated as the most recent (i.e., last entered) data.

[0137] Next, the shift register 3b receives new pixel data b k+1,1 ~b k+1,k The shift register 3b stores the stored pixel data b. 1,1 ~b k,k Shift the pixel data b. 1,1 ~b 1,k The newly stored pixel data b is erased. k+1,1 ~b k+1,k However, it is treated as the most recent data.

[0138] As a result, the shift register 3b stores pixel data b 2,1 ~b k+1,k The data is retained. In the next multiply-accumulate operation, the kernel element data a 1,1 ~a k,k And the pixel data b read from shift register 3b 2,1 ~b k+1,k The terms "to" and "to" are used.

[0139] As described above, the shift register 3b stores new pixel data for each multiply-accumulate operation, shifts the stored pixel data, and deletes the oldest stored pixel data, thereby achieving a sliding operation. In other words, the shift register 3b shifts the pixel data for each unit of the convolution operation (for example, at regular intervals).

[0140] Here, as a comparative example, let's consider an example where SRAM is used for the second storage unit 3a. In the case of SRAM, data changes are basically made to the data of each bit cell row. Therefore, when used in CNN, the same data will be included across multiple bit cell rows, resulting in wasted capacity. For example, in order to realize the above sliding operation, in the case of SRAM, the pixel data b 1,1 ~b k,k And, pixel data b 2,1 ~b k+1,k It is necessary to store this in separate bit cell rows. Pixel data b 1,1 ~b k,k and b 2,1 ~b k+1,k This is pixel data b 2,1 ~b k,k The part is duplicated. That is, the same pixel data b is in two bit cell rows. 2,1 ~b k,k It includes.

[0141] Furthermore, in the case of SRAM, all data on the latch must be changed, and the power required for data rewriting is also wasted. For example, in the sliding operation described above, the pixel data b initially held on the latch 1,1 ~b k,k Delete and remove pixel data b 2,1 ~b k+1,k I need to re-memorize it.

[0142] In contrast, the shift register 3b does not require storing the same data on multiple lines, thus reducing storage capacity. Furthermore, with the shift register 3b, it is not necessary to rewrite all the data with each slide operation. In the slide operation described above, the pixel data b 2,1 ~b k,k Pixel data can be modified without erasing it. This reduces power consumption.

[0143] The slides are performed, for example, in a Z-shape. The position of the extracted input data is slid, for example, to the right. When the extracted input data reaches the right edge of the input data, it is then shifted one element downwards, and the slide is performed again from the left edge to the right edge of the input data.

[0144] As described above, in a slide that traces a Z shape, when the input data reaches the end in the first slide direction (for example, to the right in Figure 23) and then slides in the second slide direction (for example, downward in Figure 23), it becomes necessary to rewrite all the stored data in the shift register 3b. In order to reduce the number of times the entire stored data in the shift register 3b is rewritten, it is preferable that the first slide direction be the longitudinal direction of the input data.

[0145] Alternatively, the sliding operation may be, for example, when the extraction position of the input data reaches the right end, the extraction position is shifted downward by one element, and then the input data is slid from the right end to the left end. In other words, with each slide in the second sliding direction (for example, downward), the sliding in the first sliding direction (for example, left-right) may be reversed (for example, changing from a slide to the right to a slide to the left, or from a slide to the left to a slide to the right). In this case, it becomes unnecessary to rewrite all the stored data in the shift register 3b. The shift register 3b shown in Figure 23 has the function of shifting the stored data to the right in Figure 23. In addition to this, in order to realize the above sliding operation, the shift register 3b may also have the function of shifting the stored data to the left and downward in Figure 23.

[0146] If the kernel size is large compared to the storage capacity of the first storage unit 2, macro 10d may be configured to read the kernel elements in multiple steps. In this case, the operation shown in Figure 21 or the operation shown in Figure 22 may be applied.

[0147] New pixel data may be transferred to the shift register 3b from the shared storage unit 6 or the like for each slide. The shift register 3b may also have an area for storing surplus pixel data in addition to the pixel data transferred to the multiply-accumulate circuit 4. In this case, even if there is a delay in the transfer of data from the shared storage unit 6 or the like, the slide operation can be performed without delay using the surplus pixel data. In this case, the frequency of transferring pixel data from the shared storage unit 6 or the like to the shift register 3b may be less than the slide frequency. By reducing the frequency of transferring pixel data from the shared storage unit 6 or the like to the shift register 3b, the power consumption of the arithmetic unit 1 can be reduced. In addition, the second storage unit 3a may be equipped with a temporary storage unit that holds new pixel data transferred from the shared storage unit 6 or the like, and the shift register 3b may acquire pixel data from the temporary storage unit for each slide. The update frequency of the pixel data in the temporary storage unit may be less than the slide frequency.

[0148] Figure 24 shows the layout configuration of a computing device 200 according to one comparative example. The computing device 200 has multiple macros 201, similar to the computing device 1 in Figure 1. The macros 201 have a single storage unit 202 instead of the first storage unit 2 and the second storage unit 3 in Figure 1.

[0149] The multiply-accumulate circuit 4 in Figure 24 performs a multiply-accumulate operation on operands A and B, similar to the multiply-accumulate circuit 4 in Figure 1. The storage unit 202 stores operand A, for example. The shared storage unit 6 stores operand B, for example. The multiply-accumulate circuit 4 reads operands A and B from the storage unit 202 and the shared storage unit 6, respectively.

[0150] In the arithmetic unit 200 of Figure 24, data needs to be transferred from the shared storage unit 6 each time the operand B is read. Therefore, the power consumption for data transfer is higher than when reading the operand B from the second storage unit 3 of Figure 1.

[0151] Furthermore, the arithmetic unit 200 in Figure 24 includes macros 201 located near the shared storage unit 6 and macros 201 located far from the shared storage unit 6. Of these, macros 201 located far from the shared storage unit 6 may experience delays in reading the operand B. This can lead to variations in data reading delays among multiple macros 201, and may necessitate a reduction in the operating frequency of the arithmetic unit 200 due to macros 201 with large delays.

[0152] In contrast, in macro 10 of Figure 1, both operands A and B can be read from the first storage unit 2 and the second storage unit 3, which are located near the sum-of-accumulate circuit 4. This reduces the power consumption of data transfer.

[0153] Below, we compare the power consumption efficiency of macro 10 in Figure 1 and macro 201 in Figure 24. First, the power consumption E of writing to and reading from the second storage unit 3 of the operand B. L The power E is the charge / discharge power of the signal line Lb. L write and the power E of the second memory unit 3 L SRAM It can be divided into two parts. That is, E L = E L write +E L SRAM This is the result.

[0154] Of these, the distance between the second memory unit 3 and the multiply-accumulate circuit 4 is extremely small compared to the distance between the shared memory unit 6 and the multiply-accumulate circuit 4 in Figure 24. Therefore, the power E of the charge and discharge of the signal line Lb L write It can be considered to be approximately 0. That is, E L write = 0, E L = E L SRAM Let's assume that.

[0155] In macro 201 of Figure 24, the power E is used for writing and reading the operand B to the shared storage unit 6. G Regarding the power E of the signal line during charging and discharging, G write and the power E of the shared storage unit 6 GSRAM It can be divided into two parts. That is, E G = E G write +E G SRAM This is the result.

[0156] The signal line between the shared memory unit 6 and macro 201 has a significantly longer wiring length compared to the signal line Lb. Therefore, the power E of the signal line during charging and discharging is G write The power E of the shared storage unit 6 G SRAM It can be several times that amount. Therefore, using the coefficient α, E G write = αE G SRAM , E G = (1 + α)E G SRAM Let's assume that.

[0157] The shared storage unit 6 is larger in size compared to the second storage unit 3 within macro 10. Therefore, the power consumption E of the shared storage unit 6 is G SRAM The power E of the second memory unit 3 L SRAM It can be several times that. Therefore, using the coefficient β, E G SRAM = βE L SRAM , E G = (1 + α)βE L SRAM Let's assume that.

[0158] From the above, E G / E L = (1 + α)βE L SRAM / E L SRAM = (1 + α)β. In other words, macro 10 in Figure 1 has (1 + α)β times the power efficiency compared to macro 201 in Figure 24. If we set the coefficient α = 4 and the coefficient β = 2, the power efficiency becomes 10 times higher.

[0159] Furthermore, in the multiple macros 10 shown in Figure 1, the distance between the multiply-accumulate circuit 4 and the second storage unit 3 can be made uniform. Therefore, reading the operand B from the second storage unit 3 suppresses variations in data reading delay more effectively than reading the operand B from the shared storage unit 6. In the arithmetic unit 1 shown in Figure 1, since the data reading delay can be made uniform among the multiple macros 10, it becomes possible to improve the operating frequency of the arithmetic unit 1.

[0160] Furthermore, in the arithmetic unit 200 of Figure 24, multiple macros 201 must use the same operand B stored in the shared storage unit 6. In contrast, in the multiple macros 10 of Figure 1, different operands B can be stored in the second storage unit 3. This improves the flexibility of the data flow.

[0161] Thus, in the arithmetic unit 1 according to the first embodiment of this disclosure, the first storage unit 2 and the second storage unit 3, which store operands A and B respectively, are arranged within the macro 10. As a result, the multiply-accumulate circuit 4 within the macro 10 does not need to obtain operands A and B from the shared storage unit 6 outside the macro 10. By obtaining operands A and B from the first storage unit 2 and the second storage unit 3, which are located closer than the shared storage unit 6, the multiply-accumulate circuit 4 can suppress power consumption for data transmission, shorten the time required for data transmission, and improve the operating frequency of the arithmetic unit 1. Furthermore, by arranging the first storage unit 2 and the second storage unit 3 for each macro 10, multiple macros 10 can store different operands A and different operands B. This improves the flexibility of the data flow of the arithmetic unit 1.

[0162] Furthermore, in the arithmetic unit 1, the operand B, which is frequently changed, is stored in the second storage unit 3, which is located closer to the product arithmetic circuit 7. The signal line Lb connecting the second storage unit 3 and the product arithmetic circuit 7 has a shorter wiring length than the signal line La connecting the first storage unit 2 and the product arithmetic circuit 7, and therefore consumes less power when changing the operand. For this reason, the power consumption of the arithmetic unit 1 can be reduced compared to storing the operand B in the first storage unit 2.

[0163] For example, the operation shown in Figure 21 can be applied to the data reading process from the first storage unit 2. This reduces the frequency of data reading from the first storage unit 2 and thus reduces power consumption.

[0164] The arithmetic unit 1 according to the first embodiment of this disclosure can be applied, for example, to fully connected layer calculations. In fully connected layer calculations, as shown in Figure 20, input data a1 to a that change infrequently n The first storage unit 2 stores the frequently changing weight data b. 1,1 ~b n,n This can be stored in the second memory unit 3. This reduces the power consumption of fully connected layer operations. The arithmetic unit 1 can also be applied to MLP operations, and power consumption can be reduced in a similar manner.

[0165] Furthermore, the arithmetic unit 1 can also be applied to Transformer operations. In the attention operation in Transformer, input words X that change infrequently are stored in the first storage unit 2, and weight data W that changes infrequently are stored. q , W k , and W v This can be stored in the second memory unit 3. Furthermore, the arithmetic unit 1 can also be applied to Optical Flow.

[0166] Furthermore, the arithmetic unit 1 can also be applied to CNN calculations. In CNN calculations, kernel element data that is not frequently changed can be stored in the first storage unit 2, and pixel data that is frequently changed can be stored in the second storage unit 3. In addition, a shift register 3b can be applied to the second storage unit 3 in CNN calculations. By shifting the stored data in the shift register 3b according to the sliding operation, the pixel data used for multiply-accumulate calculations can be changed. The shift register 3b can retain a portion of the stored data before and after the change in pixel data. As a result, the shift register 3b can repeatedly use a portion of the stored pixel data in multiply-accumulate calculations without erasing it. The shift register 3b can speed up the writing of pixel data and reduce power consumption during pixel data writing.

[0167] (Second Embodiment) In the second embodiment of the present disclosure, various modifications of the arithmetic unit 1 having macros 10, 10a to 10h described in the first embodiment will be described.

[0168] Figure 25 is a block diagram showing the configuration of the arithmetic unit 1 according to a second embodiment of the present disclosure. As shown in Figure 25, the arithmetic unit 1 may have a plurality of shared storage units 6. Each shared storage unit 6 is shared by a plurality of macros 10 arranged in one direction (for example, a first direction X).

[0169] Furthermore, the arithmetic unit 1 may have a configuration that includes multiple summation circuits (second arithmetic units) 41 that add up the sum-of-products results of multiple macros 10. Each summation circuit 41 adds up the sum-of-products results of multiple macros 10 arranged in one direction (for example, the second direction Y).

[0170] The shared storage unit 6 (hereinafter also called GIB: Global Input Buffer) can transfer data (i.e., operands A or B) to at least one of the first storage unit 2 or the second storage unit 3. The multiply-accumulate circuit 4 can read the data transferred from the shared storage unit 6 from the first storage unit 2 and the second storage unit 3 (hereinafter collectively called LIB: Local Input Buffer). In other words, the shared storage unit 6 can indirectly transfer data to the multiply-accumulate circuit 4 via the first storage unit 2 and the second storage unit 3.

[0171] In addition to the above, the shared storage unit 6 may directly transfer data to the sum-of-accumulate circuit 4.

[0172] In the following, the power E when data is indirectly transferred from the shared storage unit 6 to the multiply-accumulate circuit 4 (hereinafter also referred to as indirect transfer) is described. LIB The power consumption E in the case of directly transferring data (hereinafter also referred to as direct transfer) GIB Compare the following: Power E LIB and E GIB This is the power E of data transfer from GIB to the multiply-accumulate circuit 4 (reading from GIB). GIB read Data transfer from GIB to LIB (writing to LIB) Power E LIB write , data transfer from LIB to multiply-accumulate circuit 4 (read from LIB) power E LIB read, read count Nrep, number of macros Nmacro, and number of data transferred N IA Using this, it can be expressed as follows: E LIB = E LIB write ×Nmacro×N IA +E LIB read ×Nmacro×N IA ×Nrep...(1) E GIB = E GIB read ×Nmacro×N IA ×Nrep... (2)

[0173] For simplicity, the wiring length between GIB and each macro 10 will be assumed to be the same. Power E for each macro 10 LIB write , E LIB read , and E GIB read Assume there is no variation. Also, power E LIB write , E LIB read , and E GIB read This includes the power for charging and discharging the signal lines.

[0174] Furthermore, the number of reads Nrep is the number of times the same data is read. In the case of transferring (reading) data from the LIB to the multiply-accumulate circuit 4, the number of reads Nrep refers to the number of times data in the LIB is read without being modified (i.e., without data being transferred again from the GIB). Also, the number of transferred data N IA This includes the number of data transferred almost simultaneously using multiple signal lines and the number of data transferred sequentially using a single signal line. For example, if three operands A (A1, A2, and A3) are transferred twice to the multiply-accumulate circuit 4, the number of transferred data N is... IA The value is 3, and the number of read operations Nrep is 2.

[0175] Here, E GIB > E LIBIn this case, indirect transfer (first mode) is more power-efficient than direct transfer (second mode). From equations (1) and (2) above, indirect transfer is preferable when the number of readouts Nrep satisfies the following: Nrep > E LIB write / (E GIB read -E LIB read ) ... (3)

[0176] Furthermore, the distance between GIB and macro 10 is greater than the distance between LIB and macro 10, therefore E GIB read > E LIB read Furthermore, in reality, the average distance between GIB and macro 10 increases as the number of macro 10s increases. Therefore, the condition for the number of reads Nrep shown in equation (3) actually depends on the number of macro 10s, etc.

[0177] Furthermore, if the number of read operations Nrep does not satisfy equation (3), direct transfer may be preferable. Therefore, it is preferable to use direct transfer and indirect transfer depending on the arithmetic processing.

[0178] The following describes an example of the configuration of macro 10 according to the second embodiment of this disclosure. Figure 26 is a block diagram showing the configuration of the shared storage unit 6 and macro 10 according to a basic example. In Figure 26, the shared storage unit 6 transfers operands A and B to the multiply-accumulate circuit 4 via the first storage unit 2 and the second storage unit 3 (i.e., indirect transfer).

[0179] Figure 27 is a block diagram showing the configuration of the shared storage unit 6 and macro 10i according to the second embodiment of the present disclosure. The macro 10i in Figure 27 has a switch (second switch) 42. The switch 42 switches whether to transfer the operand B from the shared storage unit 6 to the multiply-accumulate circuit 4 (i.e., direct transfer) or from the second storage unit 3 to the multiply-accumulate circuit 4 (i.e., indirect transfer).

[0180] Figure 28 is a block diagram showing the configuration of the shared storage units 6a, 6b and macro 10i according to the first modified example. Figure 28 illustrates the shared storage unit (sixth storage unit) 6a that stores operand A and the shared storage unit (seventh storage unit) 6b ​​that stores operand B.

[0181] Figure 29 is a block diagram showing the configuration of the shared storage unit 6 and macro 10j according to a second modified example. Figure 30 is a block diagram showing the configuration of the shared storage units 6a, 6b and macro 10i according to a third modified example. Macro 10j in Figures 29 and 30 has a switch 42 and a switch (first switch) 43. Switch 43 switches whether to transfer the operand A from the shared storage unit 6 to the multiply-accumulate circuit 4 (i.e., direct transfer) or from the first storage unit 2 to the multiply-accumulate circuit 4 (i.e., indirect transfer). Note that macro 10j may also have a configuration in which switch 42 is omitted.

[0182] The configurations shown in Figures 26 to 30 are applicable to macros 10 and 10a to 10h according to the first embodiment of this disclosure.

[0183] Figure 31 shows a first application example of macro 10i of Figure 27. The arithmetic unit 1 in Figure 31 has a plurality of shared storage units 6 and a plurality of macros 10i. Note that macro 10j, such as in Figure 29, may be placed in place of macro 10i in the arithmetic unit 1. Alternatively, macros 10i, macro 10j, and macro 10 of Figure 26 may be mixed within the arithmetic unit 1. In addition, shared storage units 6a and 6b, such as in Figure 28, may be placed in place of (or together with) the shared storage units 6. Note that the illustration of the summation circuit 41 is omitted in Figure 31.

[0184] The arithmetic unit 1 in Figure 31 has a plurality of macro groups (first macro group) 40. Macro group 40 includes a plurality of macros 10 (macro 10i in Figure 31) connected to the same shared storage unit 6. Figure 31 shows macro groups 40_1 to 40_4 as the plurality of macro groups 40. In macro 10i in Figure 31, the direct transfer path and the indirect transfer path of the operand B are illustrated. Of these, the path used for transfer by the multiply-accumulate circuit 4 is shown with a solid line, and the path not used for transfer by the multiply-accumulate circuit 4 is shown with a dashed line.

[0185] Figure 31 also shows a macro 10i that does not transfer the operand B to the sum-of-accumulate circuit 4, i.e., does not perform a sum-of-accumulate operation. Both the direct and indirect paths for such a macro 10i are shown with dashed lines.

[0186] In the macro group 40_1, multiple macros 10i have the operand B, etc., directly transferred to the multiply-accumulate circuit 4. When the same operand is used among multiple macros 10i, a configuration like macro group 40_1 may be preferable.

[0187] In the macro group 40_2, multiple macros 10i indirectly transfer the operand B, etc., to the multiply-accumulate circuit 4. When different operands are used among the multiple macros 10i, a configuration like macro group 40_2 is preferable. Also, when the number of readouts Nrep satisfies equation (3), a configuration like macro group 40_2 may be preferable from the viewpoint of power efficiency.

[0188] Macro group 40_3 contains a mix of macros 10i that perform multiply-accumulate operations and macros 10i that do not perform multiply-accumulate operations. The arithmetic unit 1 can individually perform multiply-accumulate operations on some of the macros 10i using operands stored in the LIB (hereinafter also referred to as individual drive). Individual drive can be achieved by individually turning on or off switches 42, etc., of the multiple macros 10i. In addition, operands may be directly transferred from the shared storage unit 6 to the macros 10i that are driven individually.

[0189] Macro group 40_4 contains a mix of macros 10i that directly transfer the operand to the multiply-accumulate circuit 4 and macros 10i that transfer it indirectly. As in macro group 40_4, direct transfer macros 10i and indirect transfer macros 10i may be flexibly combined. Macro group 40_4 may also be implemented by individually turning on or off switches such as 42.

[0190] Figure 32 shows a second application example of macro 10i in Figure 27. The arithmetic unit 1 in Figure 32 has a plurality of shared storage units 6 and a plurality of macro groups 40 (i.e., a plurality of macros 10i). Note that the arithmetic unit 1 in Figure 32 may also include macro 10 in Figure 26, macro 10j in Figure 29, etc., and shared storage units 6a and 6b in Figure 28, etc. Also, Figure 32 shows macro group 40a, which is one of the macro groups 40.

[0191] The sum-of-products results of each macro 10i within the macro group 40 are added together by the sum operation circuit 41. The sum-of-products results of macro group 40a may also be added together by the sum operation circuit 41. Alternatively, the sum-of-products results of macro group 40a may not be input to the sum operation circuit 41 but output to another sum operation circuit or signal processing circuit (third arithmetic unit). If there are multiple such macro groups 40a, multiple other sum operation circuits or signal processing circuits may be provided, or a single sum operation circuit (or signal processing circuit) that adds up the multiple calculation results of multiple macro groups 40a may be provided.

[0192] The arithmetic unit 1 may also have an output circuit 44 that switches the output destination of the sum-of-accumulate calculation result of the macro group 40a.

[0193] The macro group 40a may be configured to input the sum-of-accumulate result based on the operands directly transferred from the shared storage unit 6 to the sum operation circuit 41. For example, it is conceivable that multiple macro groups 40 are used for the same arithmetic operation (e.g., CNN operation). In this case, macro group 40a may be used for the same arithmetic operation as multiple macro groups 40. That is, the number of macro groups 40 used for arithmetic operations can be increased.

[0194] The macro group 40a may be configured to output the sum-of-products result based on the operands stored in the LIB to other summation circuits or signal processing circuits, etc. In this case, the macro group 40a may be used for different arithmetic operations than the other macro groups 40. That is, the arithmetic unit 1 can execute multiple arithmetic operations in parallel using the macro groups 40 and 40a.

[0195] The macro group 40a may also be configured to input the sum-of-accumulate result based on the operands stored in the LIB to the sum operation circuit 41. Alternatively, the macro group 40a may be configured to output the sum-of-accumulate result based on the operands directly transferred from the shared storage unit 6 to another sum operation circuit or signal processing circuit, etc.

[0196] Figure 33 is a block diagram showing the configuration of the arithmetic unit 1a according to the fourth modified example. The arithmetic unit 1a has an adder (second adder) 51 that further adds the calculation results of a plurality of sum calculation circuits 41. The arithmetic unit 1a may also have a switch 52 that switches whether or not to input the calculation results of the sum calculation circuits 41 to the adder 51. If the switch 52 does not output the calculation results of the sum calculation circuits 41, it may output "0" to the adder 51. Multiple adders 51 and switches 52 may be arranged depending on the number of sum calculation circuits 41.

[0197] The arithmetic unit 1a in Figure 33 has a plurality of macro groups (second macro group) 50. Each macro group 50 includes a plurality of macros 10 connected to the same summation circuit 41. The arithmetic unit 1a has a number of summation circuits 41 corresponding to the number of macro groups 50.

[0198] In the arithmetic unit 1a shown in Figure 33, the sum-of-products results of two macro groups 50 can be added together by two summation circuits 41 and an adder 51. The adder 51 allows the arithmetic unit 1a to calculate a number of input data corresponding to the number of macro groups 50. When there are two macro groups 50, the number of operable inputs is doubled compared to when there is one macro group 50. Furthermore, by arranging three or more macro groups 50, the number of operable inputs may be tripled or more.

[0199] If the adder 51 outputs "0" (i.e., the results of one or more summation circuits 41 are not added), the multiple macro groups 50 may be used for different arithmetic operations. Alternatively, power consumption may be reduced by not using some of the multiple macro groups 50 for calculations. Macro groups 50 not used for calculations may be used as replacement macro groups 50 in case of failure of other macro groups 50 (i.e., redundant configuration).

[0200] Figure 34 is a block diagram showing the configuration of the arithmetic unit 1b according to the fifth modified example. The arithmetic unit 1b has a shift circuit 53 that shifts the calculation result of the sum calculation circuit 41 and inputs it to a switch 52 or an adder 51. The adder 51 and the shift circuit 53 constitute a shift-add circuit 54. The switch 52 may be omitted from the arithmetic unit 1b. The shift circuit 53 may shift the calculation result of the sum calculation circuit 41 by a number corresponding to the number of macros 10 in the macro group 50. In Figure 34, an example is shown in which four macros 10 are connected to the sum calculation circuit 41, and the shift circuit 53 shifts the calculation result of the sum calculation circuit 41 by 4 bits (or 4 bytes). Multiple shift circuits 53 may be arranged depending on the number of sum calculation circuits 41. The arithmetic unit 1b may also have a switch or the like to switch whether to input the calculation result of the sum calculation circuit 41 to the shift circuit 53 or to input it to the switch 52 or adder 51.

[0201] The shift circuit 53 enables the arithmetic unit 1b to perform calculations on the number of input bits or the bit width within the macro 10, depending on the number of macro groups 50. When there are two macro groups 50, the number of input bits that can be calculated or the bit width within the macro 10 is doubled compared to when there is one macro group 50. Furthermore, by arranging three or more macro groups 50, the number of input bits or the bit width may be tripled or more.

[0202] Figure 35 is a block diagram showing the configuration of the arithmetic unit 1c according to the sixth modified example. The arithmetic unit 1c stores the calculation result of the summation circuit 41 in an external storage device 55. The arithmetic unit 1c may also have a communication unit 56 that transfers the calculation result to the storage device 55. The communication unit 56 may be located outside the arithmetic unit 1c (for example, as a router). However, the arithmetic unit 1c may also be directly connected to the storage device 55 by signal lines or the like.

[0203] The calculation result of the summation circuit 41 may be output to any GIB or LIB via the communication unit 56.

[0204] Furthermore, the storage device 55 may be connected to any GIB or LIB. The GIB or LIB may receive data from the storage device 55.

[0205] Figure 36 shows the layout configuration of multiple macros 10, multiple shared storage units 6, and multiple summation circuits 41. It is preferable that the shared storage units 6 are arranged so that they are at equal distances from each macro 10 in the macro group 40. Specifically, it is preferable that the shared storage units 6 be located in the center of the macro group 40 (for example, in the center of the first direction X).

[0206] Similarly, it is preferable that the summation circuit 41 be arranged such that it is at equal distances from each macro 10 within the macro group 50. Specifically, it is preferable that the summation circuit 41 be located in the center of the macro group 50 (for example, in the center of the second direction Y).

[0207] The layout of the multiple macros 10, multiple shared storage units 6, and multiple summation circuits 41 in Figure 36 can be arranged in a rectangular shape overall.

[0208] Thus, in the computing device 1 according to the second embodiment of this disclosure, a GIB that can transfer data common to multiple macros 10 and a LIB that can store individual data for multiple macros 10 can be used in combination. This improves the flexibility of the data flow.

[0209] As shown in Figures 26 to 30, macro 10 may transfer data directly from the shared storage unit 6 to the multiply-accumulate circuit 4, or it may transfer data indirectly to the multiply-accumulate circuit 4 via the LIB. Macro 10 may also have a switch to switch between transferring data from the GIB or from the LIB to the multiply-accumulate circuit 4.

[0210] Multiple macros 10 may contain a mix of macros 10 that perform multiply-accumulate operations based on GIB data, macros 10 that perform multiply-accumulate operations based on LIB data, and macros 10 that do not perform multiply-accumulate operations.

[0211] For example, the arithmetic unit 1 may have an output circuit 44 that inputs the sum-of-products calculation result based on data from the GIB to a sum calculation circuit 41, and outputs the sum-of-products calculation result based on data stored in the LIB to another sum calculation circuit or signal processing circuit, etc. This makes it possible to switch between using an arbitrary macro group 40a for the same calculation as multiple macro groups 40, or executing multiple calculations in parallel with macro groups 40 and 40a.

[0212] (Third Embodiment) The computing device 1 according to the first or second embodiment of this disclosure can be applied to, for example, an imaging device. Figure 37 is a block diagram showing an example configuration of an imaging device 60 according to the third embodiment of this disclosure. In Figure 37, the imaging device 60 has an imaging block 70 and a signal processing block 80. The computing device 1 according to the first or second embodiment of this disclosure can be applied to at least a part of the signal processing block 80.

[0213] The imaging block 70 and the signal processing block 80 are electrically connected by connecting lines (internal buses) CL1, CL2, and CL3.

[0214] The imaging block 70 includes an imaging unit 71, an imaging processing unit 72, an output control unit 73, an output interface 74, and an imaging control unit 75, and captures images.

[0215] The signal processing block 80 includes a CPU (Central Processing Unit) 81, a DSP (Digital Signal Processor) 82, a memory 83, a communication interface 84, an image compression unit 85, and an input interface 86, and performs predetermined signal processing using the captured image obtained by the imaging block 70.

[0216] The imaging unit 71 is composed of multiple pixels arranged in two dimensions. The imaging unit 71 is driven by the imaging processing unit 72 to capture an image.

[0217] In other words, light from the optical system (not shown in Figure 37) enters the imaging unit 71. The imaging unit 71 receives the incident light from the optical system at each pixel, performs photoelectric conversion, and outputs an analog image signal corresponding to the incident light. The optical system includes, for example, a zoom lens, a focus lens, or an aperture.

[0218] The size of the image (signal) output by the imaging unit 71 can be selected from several sizes, such as 12M (3968 x 2976) pixels or VGA (Video Graphics Array) size (640 x 480 pixels).

[0219] Furthermore, the image output by the imaging unit 71 can be selected to be, for example, an RGB (red, green, blue) color image or a black and white image showing only brightness. These selections can be made as a type of shooting mode setting.

[0220] The imaging processing unit 72 performs imaging processing related to image acquisition by the imaging unit 71, such as driving the imaging unit 71, performing analog-to-digital (AD) conversion of the analog image signal output by the imaging unit 71, and processing the imaging signal, in accordance with the control of the imaging control unit 75.

[0221] Here, the imaging signal processing includes, for example, a process to determine the brightness of each small region by calculating the average value of the pixel values ​​for each predetermined small region of the image output by the imaging unit 71, a process to convert the image output by the imaging unit 71 into an HDR (High Dynamic Range) image, defect correction, and development.

[0222] The imaging processing unit 72 outputs a digital image signal (in this case, for example, a 12M pixel or VGA-sized image) obtained by A / D conversion of the analog image signal output by the imaging unit 71 as the captured image.

[0223] The captured image output by the imaging processing unit 72 is supplied to the output control unit 73 and also to the image compression unit 85 of the signal processing block 80 via the connecting line CL2.

[0224] The output control unit 73 is supplied with captured images from the imaging processing unit 72, and also with signal processing results from the signal processing block 80 via the connecting line CL3, which are obtained from signal processing using the captured images, etc.

[0225] The output control unit 73 performs output control to selectively output the captured image from the imaging processing unit 72 and the signal processing results from the signal processing block 80 to an external source (for example, memory) through a (single) output I / F 74.

[0226] In other words, the output control unit 73 selects either the captured image from the imaging processing unit 72 or the signal processing result from the signal processing block 80 and supplies it to the output I / F 74.

[0227] The output I / F 74 is an interface that outputs the captured image and signal processing results supplied from the output control unit 73 to the outside. For the output I / F 74, a relatively high-speed parallel interface such as MIPI (Mobile Industry Processor Interface) can be used.

[0228] At the output interface 74, the captured image from the imaging processing unit 72 or the signal processing result from the signal processing block 80 is output externally according to the output control of the output control unit 73. Therefore, for example, if only the signal processing result from the signal processing block 80 is needed externally and the captured image itself is not needed, only the signal processing result can be output, thereby reducing the amount of data output externally from the output interface 74.

[0229] Furthermore, the signal processing block 80 performs signal processing to obtain the signal processing results required externally, and outputs these signal processing results from the output I / F 74. This eliminates the need for external signal processing, thereby reducing the load on external blocks.

[0230] The imaging control unit 75 has a communication interface 76 and a register group 77. The communication interface 76 is a first communication interface, such as a serial communication interface such as I2C (Inter-Integrated Circuit), and exchanges necessary information, such as information to be read from and written to the register group 77, with an external device (such as a control unit).

[0231] The register group 77 has multiple registers and stores imaging information related to image acquisition by the imaging unit 71, as well as various other information. For example, the register group 77 stores imaging information received from the outside via the communication interface 76, and the results of imaging signal processing by the imaging processing unit 72 (for example, the brightness of each small area of ​​the captured image).

[0232] The imaging information stored in the register group 77 includes, for example, ISO sensitivity (analog gain during AD conversion in the imaging processing unit 72), exposure time (shutter speed), frame rate, focus, shooting mode, cropping range, etc.

[0233] Shooting modes include manual mode, where settings such as exposure time and frame rate are manually set, and automatic mode, where settings are automatically adjusted according to the scene. Automatic mode includes modes for various shooting scenes, such as nightscapes and portraits.

[0234] Furthermore, the cropping range refers to the range from which the image processing unit 72 crops a portion of the image output by the imaging unit 71 and outputs it as a captured image. By specifying the cropping range, it becomes possible, for example, to crop only the area in which a person is visible from the image output by the imaging unit 71. In addition to cropping from the image output by the imaging unit 71, there is also a method of reading only the image (signal) of the cropping range from the imaging unit 71.

[0235] The imaging control unit 75 controls the imaging processing unit 72 according to the imaging information stored in the register group 77, thereby controlling the imaging of images by the imaging unit 71.

[0236] The register group 77 can store imaging information, the results of imaging signal processing by the imaging processing unit 72, and output control information related to output control by the output control unit 73. The output control unit 73 can perform output control to selectively output the captured image and signal processing results according to the output control information stored in the register group 77.

[0237] Furthermore, in the imaging device 60, the imaging control unit 75 and the CPU 81 of the signal processing block 80 are connected via a connection line CL1, and the CPU 81 can read and write information to the register group 77 via the connection line CL1. In other words, in the imaging device 60, reading and writing information to the register group 77 can be done not only from the communication interface 76 but also from the CPU 81.

[0238] The CPU 81 and input I / F 86 within the signal processing block 80 are connected to each other via a bus, and can exchange information as needed.

[0239] The CPU 81 executes a program stored in memory 83 to control the signal processing block 80, read and write information to the register group 77 of the imaging control unit 75 via the connection line CL1, and perform various other processes.

[0240] For example, the CPU 81 functions as an imaging information calculation unit that calculates imaging information using the signal processing results obtained by signal processing in the DSP 82 by executing a program, and feeds back the new imaging information calculated using the signal processing results to the register group 77 of the imaging control unit 75 via the connecting line CL1 for storage.

[0241] Therefore, the CPU 81 can, as a result, control the imaging by the imaging unit 71 and the imaging signal processing by the imaging processing unit 72 according to the signal processing results of the captured image.

[0242] Furthermore, the imaging information stored by the CPU 81 in the register group 77 can be provided (output) to the outside via the communication interface 76. For example, the focus information among the imaging information stored in the register group 77 can be provided via the communication interface 76 to a focus driver (not shown in Figure 37) that controls the focus.

[0243] The DSP 82 functions as a signal processing unit that performs signal processing using the captured image supplied from the imaging processing unit 72 to the signal processing block 80 via the connection line CL2, and information received from the input I / F 86 from the outside, by executing a program stored in the memory 83.

[0244] The memory 83 is composed of SRAM or DRAM, and stores data necessary for the processing of the signal processing block 80. For example, the memory 83 stores programs received from the outside via the communication I / F 84, captured images compressed by the image compression unit 85 and used in signal processing by the DSP 82, the signal processing results of signal processing performed by the DSP 82, and information received by the input I / F 86.

[0245] Communication I / F 84 is a second communication interface, such as a serial communication interface like SPI (Serial Peripheral Interface), and exchanges necessary information, such as programs executed by the CPU 81 or DSP 82, with an external device (such as memory or a control unit).

[0246] For example, the communication interface 84 downloads a program from an external source that the CPU 81 or DSP 82 will execute, and supplies it to memory 83 for storage. Therefore, the CPU 81 or DSP 82 can perform various processes depending on the program downloaded by the communication interface 84.

[0247] Furthermore, the communication interface 84 can exchange not only programs but also arbitrary data with external devices. For example, the communication interface 84 can output the signal processing results obtained by the signal processing of the DSP 82 to an external device. In addition, the communication interface 84 can output information according to the instructions of the CPU 81 to an external device, thereby enabling the external device to be controlled according to the instructions of the CPU 81.

[0248] Here, the signal processing results obtained by signal processing in the DSP 82 can be output externally from the communication I / F 84, or written to the register group 77 of the imaging control unit 75 by the CPU 81. The signal processing results written to the register group 77 can be output externally from the communication I / F 76. The same applies to the processing results of the processing performed by the CPU 81.

[0249] The image compression unit 85 receives the captured image from the imaging processing unit 72 via the connection line CL2. The image compression unit 85 performs a compression process to compress the captured image, generating a compressed image with less data than the original captured image. The compressed image generated by the image compression unit 85 is supplied to the memory 83 via the bus and stored there.

[0250] Here, signal processing in the DSP 82 can be performed using the captured image itself, or using the compressed image generated from the captured image by the image compression unit 85. Since the compressed image has less data than the captured image, the load on signal processing in the DSP 82 can be reduced, and the storage capacity of the memory 83 that stores the compressed image can be saved.

[0251] The image compression unit 85 can perform compression processing such as scaling down a 12M (3968 x 2976) pixel image to a VGA-sized image. Furthermore, if the signal processing in the DSP 82 is performed on luminance and the captured image is an RGB image, the compression processing can perform YUV conversion, for example, converting the RGB image to a YUV image. The image compression unit 85 can be implemented using software or dedicated hardware.

[0252] The input interface 86 is an interface that receives information from the outside. For example, the input interface 86 receives the output of an external sensor (external sensor output) from an external sensor and supplies it to the memory 83 via the bus for storage.

[0253] For the input interface 86, for example, a parallel interface such as MIPI can be used, similar to the output interface 74.

[0254] Furthermore, as an external sensor, for example, a distance sensor that senses distance information can be employed. In addition, as an external sensor, for example, an image sensor that senses light and outputs an image corresponding to that light, that is, an image sensor separate from the imaging device 60, can be employed.

[0255] In addition to using captured images (or compressed images generated from them), the DSP82 can perform signal processing using external sensor outputs received by the input I / F86 from the aforementioned external sensors and stored in the memory 83.

[0256] In the single-chip imaging device 60 configured as described above, signal processing using the captured image (or the compressed image generated therefrom) obtained by imaging in the imaging unit 71 is performed by the DSP 82, and the signal processing results and the captured image are selectively output from the output I / F 74. Therefore, an imaging device that outputs the information required by the user can be configured in a compact size.

[0257] Figure 38 is a perspective view showing an overview of an example of the external configuration of the imaging device 60 shown in Figure 37. The imaging device 60 can be configured as a single-chip semiconductor device having a stacked structure in which multiple dies are stacked, for example, as shown in Figure 38. In Figure 38, the imaging device 60 is configured by stacking two dies, dies 61 and 62.

[0258] In Figure 38, the upper die 61 is equipped with an imaging unit 71, and the lower die 62 is equipped with an imaging processing unit 72 or imaging control unit 75, and a CPU 81 or input I / F 86.

[0259] The upper die 61 and the lower die 62 are electrically connected, for example, by forming a through hole that penetrates die 61 and reaches die 62, or by performing a Cu-Cu joint that directly connects the Cu wiring exposed on the lower surface of die 61 to the Cu wiring exposed on the upper surface of die 62.

[0260] Here, the imaging processing unit 72 can employ, for example, a column-parallel AD method or an area AD method as the method for performing A / D conversion of the image signal output by the imaging unit 71.

[0261] In a column-parallel AD method, for example, an ADC (AD Converter) is provided for each row of pixels constituting the imaging unit 71, and the ADC for each row is responsible for the AD conversion of the pixel signals of the pixels in that row, so that the AD conversion of the image signals of the pixels in each row of a row is performed in parallel. When a column-parallel AD method is adopted, a part of the imaging processing unit 72 that performs the AD conversion in the column-parallel AD method may be mounted on the upper die 61.

[0262] In the area AD method, the pixels constituting the imaging unit 71 are divided into multiple blocks, and an ADC is provided for each block. The ADC of each block is responsible for the AD conversion of the pixel signals of the pixels in that block, so that the AD conversion of the image signals of the pixels in multiple blocks is performed in parallel. In the area AD method, the image signal AD conversion (readout and AD conversion) can be performed only for the necessary pixels among the pixels constituting the imaging unit 71, using the block as the smallest unit. If a large area of ​​the imaging device 60 is acceptable, the imaging device 60 can be constructed from a single die.

[0263] Furthermore, although Figure 38 shows that two dies 61 and 62 are stacked to form a single-chip imaging device 60, a single-chip imaging device 60 can be formed by stacking three or more dies. For example, when three dies are stacked to form a single-chip imaging device 60, the memory 83 shown in Figure 38 can be mounted on another die.

[0264] In an imaging device in which sensor chips, memory chips, and DSP chips are connected in parallel by multiple bumps (hereinafter also referred to as a bump-connected imaging device), the thickness increases significantly and the device becomes larger compared to a single-chip imaging device 60 configured in a stacked structure. Furthermore, in a bump-connected imaging device, signal degradation at the bump connection points may make it difficult to secure a sufficient rate for outputting the captured image from the imaging processing unit 72 to the output control unit 73.

[0265] The stacked structure of the imaging device 60 prevents the device from becoming larger as described above, and prevents the inability to secure a sufficient rate between the imaging processing unit 72 and the output control unit 73. Therefore, the stacked structure of the imaging device 60 makes it possible to configure a compact imaging device that outputs the information required by the user. If the information required by the user is an captured image, the imaging device 60 can output the captured image. Also, if the information required by the user is obtained by signal processing using the captured image, the imaging device 60 can perform the signal processing in the DSP 82 to obtain the signal processing result as the information required by the user and output it.

[0266] The signal processing performed in the imaging device 60, that is, the signal processing of the DSP 82, can, for example, employ recognition processing to recognize a predetermined object from the captured image.

[0267] Furthermore, for example, the imaging device 60 can receive the output of a distance sensor, such as a ToF (Time of Flight) sensor, which is positioned in a predetermined positional relationship with the imaging device 60, via the input I / F 86. In this case, the signal processing of the DSP 82 can employ a fusion process that integrates the output of the distance sensor and the captured image to determine a distance with good accuracy. This process may include, for example, using the captured image to remove noise from the distance image obtained from the output of the distance sensor received via the input I / F 86.

[0268] Furthermore, for example, the imaging device 60 can receive images output by an image sensor positioned in a predetermined positional relationship with the imaging device 60 via the input I / F 86. In this case, the signal processing of the DSP 82 can, for example, employ a self-localization and mapping (SLAM) process that uses the image received via the input I / F 86 and the captured image as a stereo image.

[0269] In the DSP82, recognition processing, fusion processing, and self-localization processing can be implemented, for example, by multiple macros 10 shown in Figure 1.

[0270] When the arithmetic unit 1 is applied to the DSP 82, the results of image recognition processing, etc., in the DSP 82 may be output via the communication I / F 84. Alternatively, the output control unit 73 may be replaced by a signal processing block such as the arithmetic unit 1. This signal processing block includes a CPU, DCIM, and memory, etc. The output control unit 73 may perform high-resolution processing on the captured image, and the output I / F 74 may output a high-resolution image.

[0271] As described above, the calculation device 1 in Figure 1 is applicable to the imaging device 60, etc. Furthermore, when convolution calculations are performed within the imaging device 60, macro 10d in Figure 23 is preferably applicable. Also, when fully connected layer calculations are performed in the imaging device 60, macro 10 in Figure 20 and the processing in Figure 21 are preferably applicable.

[0272] (Application Examples) The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein may be implemented as a device mounted on any type of mobile vehicle, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, construction machinery, or agricultural machinery (tractors).

[0273] Figure 39 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile control system to which the technology of this disclosure may be applied. The vehicle control system 7000 comprises a plurality of electronic control units connected via a communication network 7010. In the example shown in Figure 39, the vehicle control system 7000 comprises a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these plurality of control units may be an in-vehicle communication network conforming to any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay®.

[0274] Each control unit comprises a microcomputer that performs calculations according to various programs, a storage unit that stores programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit is equipped with a network interface for communication with other control units via the communication network 7010, and a communication interface for communication with devices or sensors inside or outside the vehicle via wired or wireless communication. Figure 39 shows the functional configuration of the integrated control unit 7600, which includes a microcomputer 7610, a general-purpose communication interface 7620, a dedicated communication interface 7630, a positioning unit 7640, a beacon receiver 7650, an in-vehicle equipment interface 7660, an audio / image output unit 7670, an in-vehicle network interface 7680, and a storage unit 7690. Other control units similarly include a microcomputer, a communication interface, and a storage unit.

[0275] The drivetrain control unit 7100 controls the operation of devices related to the vehicle's drivetrain according to various programs. For example, the drivetrain control unit 7100 functions as a control device for generating driving force for the vehicle, such as an internal combustion engine or a drive motor; a driving force transmission mechanism for transmitting driving force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device such as an ABS (Antilock Brake System) or an ESC (Electronic Stability Control).

[0276] A vehicle state detection unit 7110 is connected to the drive system control unit 7100. The vehicle state detection unit 7110 includes, for example, a gyro sensor for detecting the angular velocity of the axial rotation motion of the vehicle body, an acceleration sensor for detecting the acceleration of the vehicle, or at least one of the sensors for detecting the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine speed, or the rotational speed of the wheels. The drive system control unit 7100 performs calculation processing using the signals input from the vehicle state detection unit 7110 and controls the internal combustion engine, drive motor, electric power steering system, brake system, etc.

[0277] The body system control unit 7200 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window system, or various lamps such as headlights, reverse lights, brake lights, turn signals, or fog lights. In this case, the body system control unit 7200 may receive radio waves transmitted from a portable device that replaces a key or signals from various switches. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock system, power window system, lamps, etc.

[0278] The battery control unit 7300 controls the secondary battery 7310, which is the power source for the drive motor, according to various programs. For example, the battery control unit 7300 receives information such as battery temperature, battery output voltage, or remaining battery capacity from the battery device equipped with the secondary battery 7310. The battery control unit 7300 uses these signals to perform calculations and controls the temperature of the secondary battery 7310 or the cooling device provided in the battery device.

[0279] The external information detection unit 7400 detects information from outside the vehicle equipped with the vehicle control system 7000. For example, at least one of the imaging unit 7410 and the external information detection unit 7420 is connected to the external information detection unit 7400. The imaging unit 7410 includes at least one of the following: a Time of Flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The external information detection unit 7420 includes at least one of the following: an environmental sensor for detecting the current weather or climate, or an ambient information detection sensor for detecting other vehicles, obstacles, or pedestrians around the vehicle equipped with the vehicle control system 7000.

[0280] The environmental sensor may be at least one of the following: a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunlight, and a snow sensor for detecting snowfall. The ambient information detection sensor may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. These imaging unit 7410 and external information detection unit 7420 may be provided as independent sensors or devices, or as a device in which multiple sensors or devices are integrated.

[0281] Here, Figure 40 shows an example of the installation location of the imaging unit 7410 and the external information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are installed, for example, at least one of the following locations on the vehicle 7900: the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield inside the passenger compartment. The imaging unit 7910 installed on the front nose and the imaging unit 7918 installed on the upper part of the windshield inside the passenger compartment mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 installed on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 installed on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 installed on the upper part of the windshield inside the passenger compartment is mainly used for detecting preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, or lanes.

[0282] Figure 40 shows an example of the imaging range of each imaging unit 7910, 7912, 7914, and 7916. Imaging range a shows the imaging range of imaging unit 7910 located on the front nose, imaging ranges b and c show the imaging ranges of imaging units 7912 and 7914 located on the side mirrors, respectively, and imaging range d shows the imaging range of imaging unit 7916 located on the rear bumper or back door. For example, by superimposing the image data captured by imaging units 7910, 7912, 7914, and 7916, an overhead view image of the vehicle 7900 can be obtained.

[0283] The external information detection units 7920, 7922, 7924, 7926, 7928, and 7930, which are installed on the front, rear, sides, corners, and the upper part of the windshield inside the vehicle 7900, may be, for example, ultrasonic sensors or radar devices. The external information detection units 7920, 7926, and 7930, which are installed on the front nose, rear bumper, back door, and the upper part of the windshield inside the vehicle 7900, may be, for example, LIDAR devices. These external information detection units 7920 to 7930 are mainly used for detecting preceding vehicles, pedestrians, or obstacles.

[0284] Returning to Figure 39, the explanation continues. The external information detection unit 7400 causes the imaging unit 7410 to capture images of the area outside the vehicle and receives the captured image data. The external information detection unit 7400 also receives detection information from the connected external information detection unit 7420. If the external information detection unit 7420 is an ultrasonic sensor, radar device, or LIDAR device, the external information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the external information detection unit 7400 may perform object detection processing such as detecting people, vehicles, obstacles, signs, or characters on the road surface, or distance detection processing. Based on the received information, the external information detection unit 7400 may perform environmental recognition processing to recognize rainfall, fog, or road surface conditions. Based on the received information, the external information detection unit 7400 may calculate the distance to an object outside the vehicle.

[0285] Furthermore, the external information detection unit 7400 may perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The external information detection unit 7400 may perform distortion correction or alignment processing on the received image data, and may also synthesize image data captured by different imaging units 7410 to generate an overhead view image or a panoramic image. The external information detection unit 7400 may also perform viewpoint transformation processing using image data captured by different imaging units 7410.

[0286] The in-vehicle information detection unit 7500 detects information inside the vehicle. The in-vehicle information detection unit 7500 is connected to, for example, a driver status detection unit 7510 that detects the driver's state. The driver status detection unit 7510 may include a camera that images the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sounds inside the vehicle. The biosensor is installed, for example, on the seat or steering wheel and detects the biometric information of an occupant sitting in the seat or a driver holding the steering wheel. Based on the detection information input from the driver status detection unit 7510, the in-vehicle information detection unit 7500 may calculate the driver's level of fatigue or concentration, or determine whether the driver is dozing off. The in-vehicle information detection unit 7500 may perform processing such as noise cancellation on the collected audio signals.

[0287] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 is implemented by a device that can be operated by the passenger, such as a touch panel, buttons, a microphone, a switch, or a lever. The integrated control unit 7600 may also receive data obtained by voice recognition of voice input from the microphone. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or PDA (Personal Digital Assistant) that is compatible with the operation of the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information by gesture. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on the information input by the passenger using the above input unit 7800 and outputs it to the integrated control unit 7600. Passengers and others can input various data or instruct the vehicle control system 7000 to perform processing operations by operating this input unit 7800.

[0288] The storage unit 7690 may include a ROM (Read Only Memory) for storing various programs executed by a microcomputer, and a RAM (Random Access Memory) for storing various parameters, calculation results, or sensor values. The storage unit 7690 may also be implemented using a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0289] The general-purpose communication interface 7620 is a general-purpose communication interface that mediates communication between the external environment 7750 and various devices present in the external environment 7750. The general-purpose communication interface 7620 may implement cellular communication protocols such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (registered trademark) (Long Term Evolution), or LTE-A (LTE-Advanced), or other wireless communication protocols such as wireless LAN (also known as Wi-Fi (registered trademark)) and Bluetooth (registered trademark). The general-purpose communication interface 7620 may connect, for example, to devices (e.g., application servers or control servers) located on an external network (e.g., the Internet, a cloud network, or a carrier-specific network) via a base station or access point. Furthermore, the general-purpose communication I / F 7620 may connect to terminals located near the vehicle (for example, terminals belonging to the driver, pedestrians, or shops, or MTC (Machine Type Communication) terminals) using, for example, P2P (Peer To Peer) technology.

[0290] The dedicated communication interface 7630 is a communication interface that supports communication protocols developed for use in vehicles. The dedicated communication interface 7630 may implement standard protocols such as WAVE (Wireless Access in Vehicle Environment), DSRC (Dedicated Short Range Communications), or cellular communication protocols, which are combinations of lower-layer IEEE 802.11p and upper-layer IEEE 1609. The dedicated communication interface 7630 typically performs V2X communication, a concept that includes one or more of the following: vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0291] The positioning unit 7640 performs positioning by receiving, for example, GNSS (Global Navigation Satellite System) signals from GNSS satellites (for example, GPS signals from GPS (Global Positioning System) satellites) and generates location information including the vehicle's latitude, longitude, and altitude. The positioning unit 7640 may also determine its current location by exchanging signals with a wireless access point, or it may acquire location information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0292] The beacon receiver 7650 receives radio waves or electromagnetic waves transmitted from, for example, a radio station installed on a road, and acquires information such as the current location, traffic congestion, road closures, or travel time. The functions of the beacon receiver 7650 may also be included in the dedicated communication interface 7630 described above.

[0293] The in-vehicle equipment interface 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle equipment 7760 located inside the vehicle. The in-vehicle equipment interface 7660 may establish a wireless connection using wireless communication protocols such as wireless LAN, Bluetooth®, NFC (Near Field Communication), or WUSB (Wireless USB). Furthermore, the in-vehicle equipment I / F 7660 may establish a wired connection such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-definition Link) via connection terminals (and, if necessary, cables) not shown. The in-vehicle equipment 7760 may include, for example, at least one of the following: a mobile device or wearable device owned by a passenger, or an information device brought into or installed in the vehicle. The in-vehicle equipment 7760 may also include a navigation device that searches for a route to any destination. The in-vehicle equipment I / F 7660 exchanges control signals or data signals with these in-vehicle equipment 7760s.

[0294] The in-vehicle network interface 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network interface 7680 transmits and receives signals and other data in accordance with a predetermined protocol supported by the communication network 7010.

[0295] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information acquired via at least one of the general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the drive force generator, steering mechanism, or braking device based on acquired information from inside and outside the vehicle, and output control commands to the drive system control unit 7100. For example, the microcomputer 7610 may perform coordinated control aimed at realizing ADAS (Advanced Driver Assistance System) functions, including vehicle collision avoidance or impact mitigation, following driving based on distance between vehicles, maintaining vehicle speed, vehicle collision warning, or vehicle lane departure warning. Furthermore, the microcomputer 7610 may perform cooperative control for purposes such as autonomous driving, where the vehicle drives autonomously without driver intervention, by controlling the drive force generating device, steering mechanism, or braking device, etc., based on the acquired information about the vehicle's surroundings.

[0296] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and surrounding structures, people, and other objects based on information acquired via at least one of the general-purpose communication I / F 7620, dedicated communication I / F 7630, positioning unit 7640, beacon receiver 7650, in-vehicle equipment I / F 7660, and in-vehicle network I / F 7680, and create local map information including surrounding information of the vehicle's current location. The microcomputer 7610 may also predict dangers such as vehicle collision, proximity of pedestrians, or entry into a closed road based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal to generate a warning sound or to illuminate a warning lamp.

[0297] The audio-image output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying the vehicle's occupants or those outside the vehicle. In the example in Figure 39, the output devices are exemplified as an audio speaker 7710, a display unit 7720, and an instrument panel 7730. The display unit 7720 may include, for example, at least one of an onboard display and a head-up display. The display unit 7720 may also have an AR (Augmented Reality) display function. The output device may be other devices other than these, such as headphones, wearable devices such as glasses-type displays worn by occupants, projectors, or lamps. If the output device is a display device, the display device visually displays the results obtained from various processes performed by the microcomputer 7610 or information received from other control units in various formats such as text, images, tables, and graphs. If the output device is an audio output device, the audio output device converts the audio signal, consisting of reproduced audio data or sound data, into an analog signal and outputs it audibly.

[0298] In the example shown in Figure 39, at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include other control units not shown. Also, in the above description, some or all of the functions performed by one control unit may be assigned to other control units. In other words, as long as information is transmitted and received via the communication network 7010, predetermined calculation processing may be performed by any of the control units. Similarly, a sensor or device connected to one control unit may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.

[0299] In the vehicle control system 7000 described above, the arithmetic unit 1 according to this embodiment, as described with reference to Figure 1, can be applied to the imaging unit 7410 and the like in the application example shown in Figure 39. For example, by applying the arithmetic unit 1 according to this embodiment to the image processing of the imaging unit 7410, power consumption can be reduced and the operating frequency can be improved.

[0300] Furthermore, this technology can take the following configuration: (1) A arithmetic device comprising: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit that adds up a plurality of multiply-accumulate operation results performed by the plurality of first arithmetic units, wherein each of the plurality of first arithmetic units comprises: a first storage unit that stores an input first digital signal; a second storage unit that stores an input second digital signal; a multiply-accumulate operation circuit that performs a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line that transmits the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line that transmits the second digital signal from the second storage unit to the multiply-accumulate operation circuit. (2) The arithmetic device according to (1), wherein the first storage unit stores one or more of the first digital signals, the second storage unit stores a plurality of the second digital signals, and further comprises a first selection circuit for selecting some of the first digital signals used in a sum-of-accumulate operation from the first storage unit, and a second selection circuit for selecting some of the second digital signals used in a sum-of-accumulate operation from the second storage unit, and the sum-of-accumulate operation circuit performs a sum-of-accumulate operation on the first digital signals selected by the first selection circuit and the second digital signals selected by the second selection circuit. (3) The arithmetic device according to (2), further comprising a third storage unit for holding the first digital signals selected by the first selection circuit, and a fourth storage unit for holding the second digital signals selected by the second selection circuit, and the sum-of-accumulate operation circuit performs a sum-of-accumulate operation on the first digital signals held by the third storage unit and the second digital signals held by the fourth storage unit. (4) The arithmetic device according to (2) or (3), further comprising: a first sense amplifier for sensing the first digital signal selected by the first selection circuit; and a second sense amplifier for sensing the second digital signal selected by the second selection circuit. (5) The arithmetic device according to any one of (2) to (4), wherein the first storage unit and the second storage unit each have SRAM (Static Random Access Memory).(6) The arithmetic device according to any one of (2) to (5), wherein the first selection circuit selects a plurality of first digital signals, the second selection circuit selects a plurality of second digital signals, the sum-of-accumulate circuit has a plurality of product circuits, and each of the plurality of product circuits performs a product operation on one of the plurality of first digital signals selected by the first selection circuit and one of the plurality of second digital signals selected by the second selection circuit. (7) The arithmetic device according to any one of (2) to (6), wherein the sum-of-accumulate circuit performs at least one of fully connected layer operations, MLP (Multi Layer Perceptron) operations, Attention operations, or Optical Flow operations. (8) The arithmetic device according to any one of (2) to (7), wherein the sum-of-accumulate circuit performs multiple sum-of-accumulate operations on the fixed first digital signal and the plurality of selected second digital signals when the first digital signal selected by the first selection circuit is fixed and the second selection circuit is switching the selection of the second digital signals. (9) The arithmetic device according to (1), wherein the first storage unit has SRAM and the second storage unit has memory other than SRAM. (10) The arithmetic device according to (1) or (9), wherein the second storage unit has a shift register for storing the second digital signal. (11) The arithmetic device according to (10), wherein the multiply-accumulate circuit performs a convolution operation and the shift register shifts the second digital signal for each unit of the convolution operation. (12) The arithmetic device according to (10) or (11), wherein the first storage unit stores one or more of the first digital signals and further comprises a first selection circuit for selecting some of the first digital signals used in the multiply-accumulate operation from the first storage unit, and the multiply-accumulate circuit performs a multiply-accumulate operation between the first digital signal selected by the first selection circuit and the second digital signal stored in the shift register.(13) The arithmetic device according to (12), further comprising a third storage unit that holds the first digital signal selected by the first selection circuit, wherein the sum-of-accumulate circuit performs a sum-of-accumulate operation based on the first digital signal stored in the third storage unit and the second digital signal stored in the shift register. (14) The arithmetic device according to (12) or (13), further comprising a first sense amplifier that senses the first digital signal selected by the first selection circuit. (15) The arithmetic device according to any one of (1) to (14), wherein the reading period of the second digital signal is shorter than the reading period of the first digital signal. (16) The arithmetic device according to any one of (1) to (15), wherein, during the sum-of-accumulate operation of the first arithmetic unit, the frequency at which the second signal line is charged and discharged is greater than the frequency at which the first signal line is charged and discharged. (17) The arithmetic device according to any one of (1) to (16), wherein the multiply-accumulate circuit comprises an multiplicative circuit that performs a multiplicative operation between the first digital signal and the second digital signal, and a summation circuit that adds the results of the multiplicative circuit, the first signal line connects the first storage unit and the multiplicative circuit, and the second signal line connects the second storage unit and the multiplicative circuit. (18) The arithmetic device according to (17), wherein the multiplicative circuit is arranged between the first storage unit and the second storage unit, and the summation circuit is arranged between the multiplicative circuit and the first storage unit. (19) The arithmetic device according to (18), wherein the height of the second storage unit in the direction in which the second signal line extends is greater than the height of the summation circuit in the direction in which the second signal line extends. (20) The arithmetic device according to (17), wherein the second storage unit is arranged between the multiplicative circuit and the first storage unit. (21) The arithmetic apparatus according to (20), wherein the height of the second memory unit in the direction in which the second signal line extends is smaller than the height of the summation circuit in the direction in which the second signal line extends.(22) The arithmetic apparatus according to (20) or (21), wherein the first storage unit stores one or more of the first digital signals, the second storage unit stores a plurality of the second digital signals, the third storage unit holds some of the first digital signals used in the sum-accumulate operation from among the first digital signals stored in the first storage unit, and the fourth storage unit selects some of the second digital signals used in the sum-accumulate operation from among the first digital signals stored in the second storage unit, and the third storage unit and the fourth storage unit are arranged adjacent to each other. (23) The arithmetic device according to any one of (1) to (22), comprising: a first arithmetic unit group having a plurality of first arithmetic units; and a fifth storage unit that supplies at least one of the first digital signal or the second digital signal to a plurality of first arithmetic units in the first arithmetic unit group, wherein each of the first arithmetic units in the first arithmetic unit group has a first mode of reading at least one of the first digital signal or the second digital signal from the fifth storage unit; and a second mode of reading the first digital signal from the first storage unit and reading the second digital signal from the second storage unit. (24) The arithmetic device according to any one of (1) to (23), comprising: a first arithmetic unit group having a plurality of the first arithmetic units; and a fifth storage unit that supplies at least one of the first digital signal or the second digital signal to a plurality of the first arithmetic units in the first arithmetic unit group, wherein each of the first arithmetic units in the first arithmetic unit group has at least one of: a first switch for switching between reading the first digital signal from the fifth storage unit or reading the first digital signal from the first storage unit; or a second switch for switching between reading the second digital signal from the fifth storage unit or reading the second digital signal from the second storage unit. (25) The arithmetic device according to (23) or (24), wherein the fifth storage unit comprises: a sixth storage unit for storing the first digital signal; and a seventh storage unit for storing the second digital signal.(26) The arithmetic device according to any one of (23) to (25), further comprising an output circuit that switches the output destination between the sum-accumulate result based on the first digital signal read from the first storage unit and the second digital signal read from the second storage unit and the sum-accumulate result based on the first digital signal or the second digital signal read from the fifth storage unit. (27) The arithmetic device according to (26), further comprising a plurality of second arithmetic units that output a first sum-accumulate result based on the first digital signal or the second digital signal read from the fifth storage unit, and one or more third arithmetic units that output a second sum-accumulate result based on the first digital signal read from the first storage unit and the second digital signal read from the second storage unit in parallel with the first sum-accumulate result. (28) The arithmetic device according to any one of (23) to (27), comprising: a plurality of second arithmetic units; and a group of second arithmetic units having a plurality of first arithmetic units connected to any of the plurality of second arithmetic units, wherein the fifth storage unit is located in the center of the plurality of first arithmetic units in the group of first arithmetic units, and the second arithmetic unit is located in the center of the plurality of first arithmetic units in the group of second arithmetic units. (29) The arithmetic device according to any one of (1) to (28), wherein if the width of the sum-of-accumulate arithmetic circuit and the storage unit including the first storage unit or the second storage unit are substantially the same in the direction in which the first signal line or the second signal line is arranged, the sum-of-accumulate arithmetic circuit is arranged such that one side of the storage unit substantially coincides with one side of the storage unit, and if the width of the sum-of-accumulate arithmetic circuit and the storage unit are different, the sum-of-accumulate arithmetic circuit and the storage unit are arranged such that the smaller side of the storage unit touches the center of the larger side of the sum-of-accumulate arithmetic circuit and the storage unit.(30) Electronic device comprising: an imaging unit for capturing image data; and a signal processing unit for performing signal processing on the image data, wherein the signal processing unit comprises: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit for adding up a plurality of multiply-accumulate operation results calculated by the plurality of first arithmetic units, and each of the plurality of first arithmetic units comprises: a first storage unit for storing an input first digital signal; a second storage unit for storing an input second digital signal; a multiply-accumulate operation circuit for performing a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line for transmitting the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line for transmitting the second digital signal from the second storage unit to the multiply-accumulate operation circuit.

[0301] The aspects of this disclosure are not limited to the individual embodiments described above, but include various modifications that a person skilled in the art could conceive, and the effects of this disclosure are not limited to those described above. In other words, various additions, modifications, and partial deletions are possible, as long as they do not depart from the conceptual idea and spirit of this disclosure derived from the claims and their equivalents.

[0302] 1, 1a, 1b, 1c, 200 Arithmetic unit, 2 First memory unit, 3, 3a Second memory unit, 3b Shift register, 4 Multiply-accumulate circuit, 5 Arithmetic area, 6, 6a, 6b Shared memory unit, 7 Multiply circuit, 8, 41 Sum circuit, 9 Memory element, 10, 10a, 10b, 10c, 10d, 10e, 10f, 10g, 10h, 10i, 10j, 100, 201 Macro, 11, 21 Bit cell array, 12, 22 Decoder, 13, 23 Sense amplifier, 14, 24 Latch, 15, 25 Control circuit, 16, 26 Latch group, 31, 42, 43, 52 Switch, 32 Control circuit, 40, 40_1, 40_2, 40_3, 40_4, 40a, 50 Macro group, 44 Output circuit, 51 Adder, 53 Shift circuit, 54 Shift-add circuit, 55 Memory device, 56 Communication unit, 60 Imaging device, 61, 62 Die, 70 Imaging block, 71 Imaging unit, 72 Imaging processing unit, 73 Output control unit, 74, 86 Output I / F, 75 Imaging control unit, 76, 84 Communication I / F, 77 Register group, 80 Signal processing block, CPU 81, DSP 82, 83 Memory, 85 Image compression unit, 202 Storage unit

Claims

1. An arithmetic device comprising: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit, which adds up a plurality of multiply-accumulate operation results performed by the plurality of first arithmetic units, wherein each of the plurality of first arithmetic units comprises: a first storage unit for storing an input first digital signal; a second storage unit for storing an input second digital signal; a multiply-accumulate operation circuit for performing a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line for transmitting the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line for transmitting the second digital signal from the second storage unit to the multiply-accumulate operation circuit.

2. The arithmetic device according to claim 1, wherein the first storage unit stores one or more first digital signals, the second storage unit stores a plurality of second digital signals, and further comprises a first selection circuit for selecting some of the first digital signals used in a sum-of-accumulate operation from the first storage unit, and a second selection circuit for selecting some of the second digital signals used in a sum-of-accumulate operation from the second storage unit, and the sum-of-accumulate operation circuit performs a sum-of-accumulate operation with the first digital signals selected by the first selection circuit and the second digital signals selected by the second selection circuit.

3. The arithmetic device according to claim 2, further comprising: a third storage unit that holds the first digital signal selected by the first selection circuit; and a fourth storage unit that holds the second digital signal selected by the second selection circuit, wherein the sum-of-accumulate operation circuit performs a sum-of-accumulate operation based on the first digital signal held by the third storage unit and the second digital signal held by the fourth storage unit.

4. The arithmetic device according to claim 2, further comprising: a first sense amplifier for sensing the first digital signal selected by the first selection circuit; and a second sense amplifier for sensing the second digital signal selected by the second selection circuit.

5. The arithmetic apparatus according to claim 2, wherein the first storage unit and the second storage unit each have SRAM (Static Random Access Memory).

6. The arithmetic device according to claim 2, wherein the first selection circuit selects a plurality of first digital signals, the second selection circuit selects a plurality of second digital signals, the sum-of-accumulate circuit has a plurality of multiplication circuits, and each of the plurality of multiplication circuits performs a multiplication operation between one of the plurality of first digital signals selected by the first selection circuit and one of the plurality of second digital signals selected by the second selection circuit.

7. The arithmetic device according to claim 2, wherein the sum-of-accumulate circuit performs at least one of the following: fully coupled layer arithmetic, MLP (Multi Layer Perceptron) arithmetic, attention arithmetic, or optical flow arithmetic.

8. The arithmetic device according to claim 2, wherein the sum-of-accumulate operation circuit performs multiple sum-of-accumulate operations based on the fixed first digital signal and a plurality of selected second digital signals when the first digital signal selected by the first selection circuit is fixed and the second selection circuit is switching the selection of the second digital signal.

9. The arithmetic apparatus according to claim 1, wherein the first storage unit has SRAM, and the second storage unit has memory other than SRAM.

10. The arithmetic apparatus according to claim 1, wherein the second storage unit has a shift register for storing the second digital signal.

11. The arithmetic device according to claim 10, wherein the sum-of-accumulate circuit performs a convolution operation, and the shift register shifts the second digital signal for each unit of the convolution operation.

12. The arithmetic device according to claim 1, wherein the reading period of the second digital signal is shorter than the reading period of the first digital signal.

13. The arithmetic device according to claim 1, wherein the sum-of-products circuit comprises an product circuit that performs a product operation between the first digital signal and the second digital signal, and a sum circuit that adds the results of the product operation circuit, the first signal line connects the first storage unit and the product circuit, and the second signal line connects the second storage unit and the product circuit.

14. The arithmetic device according to claim 13, wherein the product operation circuit is arranged between the first storage unit and the second storage unit, and the sum operation circuit is arranged between the product operation circuit and the first storage unit.

15. The arithmetic device according to claim 13, wherein the second storage unit is disposed between the product operation circuit and the first storage unit.

16. The arithmetic device according to claim 1, comprising: a group of first arithmetic units having a plurality of the first arithmetic units; and a fifth storage unit that supplies at least one of the first digital signal or the second digital signal to a plurality of first arithmetic units in the group of first arithmetic units, wherein each of the first arithmetic units in the group of first arithmetic units has at least one of a first switch for switching between reading the first digital signal from the fifth storage unit or reading the first digital signal from the first storage unit, or a second switch for switching between reading the second digital signal from the fifth storage unit or reading the second digital signal from the second storage unit.

17. The arithmetic device according to claim 16, further comprising an output circuit that switches the output destination based on the sum-accumulate result based on the first digital signal read from the first storage unit and the second digital signal read from the second storage unit, and the sum-accumulate result based on the first digital signal or the second digital signal read from the fifth storage unit.

18. The arithmetic device according to claim 16, comprising: a plurality of second arithmetic units; and a group of second arithmetic units having a plurality of first arithmetic units connected to any of the plurality of second arithmetic units, wherein the fifth storage unit is located in the central part of the plurality of first arithmetic units in the group of first arithmetic units; and the second arithmetic unit is located in the central part of the plurality of first arithmetic units in the group of second arithmetic units.

19. The arithmetic device according to claim 1, wherein, when the width of the sum-of-accumulate circuit and the storage unit including the first storage unit or the second storage unit are substantially the same in the direction in which the first signal line or the second signal line is arranged, the sum-of-accumulate circuit is arranged such that one side of the storage unit and one side of the storage unit substantially coincide, and when the width of the sum-of-accumulate circuit and the storage unit are different, the sum-of-accumulate circuit and the storage unit are arranged such that the smaller side of the storage unit touches the center of the larger side of the sum-of-accumulate circuit and the storage unit.

20. Electronic device comprising: an imaging unit for capturing image data; and a signal processing unit for performing signal processing on the image data, wherein the signal processing unit comprises: a plurality of first arithmetic units, each performing a multiply-accumulate operation; and a second arithmetic unit for adding up a plurality of multiply-accumulate operation results calculated by the plurality of first arithmetic units, and each of the plurality of first arithmetic units comprises: a first storage unit for storing an input first digital signal; a second storage unit for storing an input second digital signal; a multiply-accumulate operation circuit for performing a multiply-accumulate operation between the first digital signal and the second digital signal; a first signal line for transmitting the first digital signal from the first storage unit to the multiply-accumulate operation circuit; and a second signal line having a shorter wiring length than the first signal line for transmitting the second digital signal from the second storage unit to the multiply-accumulate operation circuit.