Image analysis system and vehicle

Abstract

This image analysis system comprises a first processor, a second processor, and a third processor, each capable of executing image analysis. The bandwidth of a first transmission channel for the first processor to communicate with a memory is different from both the bandwidth of a second transmission channel for the second processor to communicate with a memory and the bandwidth of a third transmission channel for the third processor to communicate with a memory.

Description

Image Analysis System and Vehicle

[0001] The present invention relates to an image analysis system and a vehicle.

[0002] Vehicle control using artificial intelligence has begun to spread. In such vehicle control, an image of the environment around the vehicle is analyzed. Patent Document 1 describes a SoC including a main processor and a GPU.

[0003] Japanese Patent Application Laid-Open No. 2022-548643

[0004] In a vehicle, various images can be analysis targets. The capabilities required for analyzing an image can vary depending on the analysis content. Some aspects of the present invention provide a technique for enabling efficient execution of image analysis.

[0005] According to some embodiments, there is provided an image analysis system including a first processor, a second processor, and a third processor each capable of executing image analysis, wherein a bandwidth of a first transmission path for the first processor to communicate with a memory is different from both a bandwidth of a second transmission path for the second processor to communicate with the memory and a bandwidth of a third transmission path for the third processor to communicate with the memory.

[0006] According to some embodiments, a technique for enabling efficient execution of image analysis is provided.

[0007] Other features and advantages of the present invention will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar configurations are denoted by the same reference numerals.

[0008] The attached drawings are included in the specification and constitute a part thereof, illustrating embodiments of the present invention and are used to explain the principles of the present invention together with the description thereof. Block diagram illustrating an example of the configuration of a control device and a vehicle according to some embodiments. Block diagram illustrating an example of the arrangement of cameras in a vehicle according to some embodiments. Block diagram illustrating an example of the configuration of an ECU in a vehicle according to some embodiments. Flow diagram illustrating an example of a method for controlling a semiconductor device according to some embodiments. Block diagram illustrating an example of the mounting of a semiconductor device according to some embodiments. Block diagram illustrating an example of the mounting of a semiconductor device according to some embodiments.

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more of the features described in the embodiments may be combined in any way. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0010] <Control device and its application examples> Figure 1 is a block diagram of a control device CNT according to a part of the present invention and a schematic diagram of a vehicle V, which is an example of its application. In Figure 1, the schematic of the vehicle V is shown in a plan view and a side view. The vehicle V in this embodiment is, as an example, a sedan-type four-wheeled passenger car, and may be, for example, a parallel hybrid vehicle. The vehicle V is not limited to a four-wheeled passenger car, and may be a saddle-type vehicle (motorcycle, motorized tricycle), or a large vehicle such as a truck or bus.

[0011] The control device CNT includes a controller 1, which is an electronic circuit that performs control of the vehicle V, including automatic driving control of the vehicle V. The controller 1 comprises a plurality of ECUs (Electronic Control Units). The ECUs are provided, for example, for each function of the control device CNT. Each ECU includes a processor, such as a CPU (Central Processing Unit), a storage device such as semiconductor memory, and an interface with an external device. The storage device stores programs executed by the processor and data used by the processor for processing. The interface includes input / output interfaces and communication interfaces. Each ECU may have a plurality of processors, a plurality of storage devices, and a plurality of interfaces. Programs stored in the storage device may be stored in the storage device by being installed in the control device CNT using a storage medium such as a CD-ROM. In addition to or instead of this, programs stored in the storage device may be downloaded from an external server via wireless communication, or written by a scan tool or a factory writing device.

[0012] Controller 1 controls the driving (acceleration) of vehicle V by controlling power unit (power plant) 2. Power unit 2 is a driving unit that outputs driving force to rotate the drive wheels of vehicle V, and may include an internal combustion engine, a motor, and an automatic transmission. The motor can be used as a driving source to accelerate vehicle V, and can also be used as a generator during deceleration, etc. (regenerative braking).

[0013] In this embodiment, the controller 1 controls the output of the internal combustion engine and motor, or switches the gear of the automatic transmission, in response to the driver's driving operations detected by the operation detection sensor 2a provided on the accelerator pedal AP and the operation detection sensor 2b provided on the brake pedal BP, as well as the vehicle speed of the vehicle V detected by the rotation speed sensor 2c. The automatic transmission is equipped with a rotation speed sensor 2c that detects the rotation speed of the output shaft of the automatic transmission as a sensor for detecting the driving state of the vehicle V. The vehicle speed of the vehicle V can be calculated from the detection result of the rotation speed sensor 2c.

[0014] Controller 1 controls the braking (deceleration) of the vehicle V by controlling the hydraulic system 3. The driver's braking operation on the brake pedal BP is converted into hydraulic pressure in the brake master cylinder BM and transmitted to the hydraulic system 3. The hydraulic system 3 is an actuator that can control the hydraulic pressure of the hydraulic fluid supplied to the brake systems 3a (e.g., disc brake systems) provided on each of the four wheels, based on the hydraulic pressure transmitted from the brake master cylinder BM.

[0015] Controller 1 can control the braking of the vehicle V by controlling the drive of solenoid valves and other components of the hydraulic system 3. Controller 1 can also configure an electric servo brake system by controlling the distribution of braking force from the brake device 3a and braking force from regenerative braking of the motor in the power unit 2. Controller 1 may also illuminate the brake lamp 3b during braking.

[0016] Controller 1 controls the steering of the vehicle V by controlling the electric power steering system 4. The electric power steering system 4 includes a mechanism that steers the front wheels in response to the driver's driving operation (steering operation) on the steering wheel ST. The electric power steering system 4 includes a drive unit 4a that provides driving force (sometimes referred to as steering assist torque) to assist the steering operation or to automatically steer the front wheels of the vehicle V. The drive unit 4a is equipped with a motor as a drive source. The electric power steering system 4 also includes a steering angle sensor 4b that detects the steering angle, and a torque sensor 4c that detects the steering torque borne by the driver (referred to as steering load torque and distinguished from steering assist torque).

[0017] Controller 1 controls the electric parking brake device 3c located on the rear wheels of the vehicle V. The electric parking brake device 3c includes a mechanism for locking the rear wheels. Controller 1 can control the locking and unlocking of the rear wheels by the electric parking brake device 3c.

[0018] Controller 1 controls an information output device 5 that provides information to the vehicle. The information output device 5 includes, for example, a display device 5a that provides information to the driver by image, and / or an audio output device 5b that provides information to the driver by voice. The display device 5a includes, for example, a display device provided on the instrument panel or a display device provided on the steering wheel ST. The display device 5a may also include a head-up display. The information output device 5 may also provide information to the occupants by vibration or light.

[0019] The controller 1 receives instruction input from the occupant (e.g., the driver) via the input device 6. The input device 6 is positioned in a location where the driver can operate it and includes, for example, a group of switches 6a for the driver to give instructions to the vehicle V, and / or a turn signal lever 6b for activating the turn signals.

[0020] Controller 1 recognizes and determines the current position and course (attitude) of the vehicle V. In this embodiment, the vehicle V is equipped with a gyro sensor 7a, a GNSS (Global Navigation Satellite System) sensor 7b, and a communication device 7c. The gyro sensor 7a detects the rotational motion (yaw rate) of the vehicle V. The GNSS sensor 7b detects the current position of the vehicle V. The communication device 7c also communicates wirelessly with a server that provides map information and traffic information to acquire this information. In this embodiment, Controller 1 determines the course of the vehicle V based on the detection results of the gyro sensor 7a and the GNSS sensor 7b, and sequentially acquires map information related to the course from the server via the communication device 7c and stores it in the database 7d (storage device). The vehicle V may also be equipped with other sensors to detect the state of the vehicle V, such as an acceleration sensor that detects the acceleration of the vehicle V.

[0021] Controller 1 executes automatic driving control of vehicle V based on the detection results of various detection units installed in vehicle V. Vehicle V is equipped with external sensors, which are ambient detection units 8a to 8b, that detect the external environment (surrounding conditions) of vehicle V, and internal sensors, which are internal detection units 9a to 9b, that detect the conditions inside the vehicle (the state of the occupants (especially the driver)). Controller 1 can grasp the surrounding conditions of vehicle V based on the detection results of ambient detection units 8a to 8b and execute automatic driving control according to those surrounding conditions. In addition, controller 1 can determine whether the driver is performing the predetermined actions required of the driver when executing automatic driving control, based on the detection results of internal detection units 9a to 9b.

[0022] The surrounding detection unit 8a is an imaging device that captures images of the area in front of the vehicle V (hereinafter sometimes referred to as the front camera 8a), and is mounted, for example, on the interior side of the front windshield at the front of the roof of the vehicle V. The controller 1 can extract the outline of an object, identify the type of object, or extract lane markings (white lines, etc.) on the road by analyzing the images captured by the front camera 8a.

[0023] The surrounding detection unit 8b is a millimeter-wave radar (hereinafter sometimes referred to as radar 8b), which uses radio waves to detect targets around the vehicle V and detects (measures) the distance to the targets and the direction (azimuth) of the targets relative to the vehicle V. In the example shown in Figure 1, five radar 8b units are provided: one in the center of the front of the vehicle V, one at each of the left and right corners of the front, and one at each of the left and right corners of the rear.

[0024] The surrounding detection unit installed on the vehicle V is not limited to the above configuration; the number of cameras and radars may be changed, and a LiDAR (Light Detection and Ranging) for detecting targets around the vehicle V may also be provided.

[0025] The in-vehicle detection unit 9a is an imaging device that captures images of the interior of the vehicle (hereinafter sometimes referred to as the in-vehicle camera 9a), and is installed, for example, on the interior side of the front part of the roof of the vehicle interior V. In this embodiment, the in-vehicle camera 9a is a driver monitor camera that captures images of the driver (for example, the driver's eyes and face). The controller 1 can determine the driver's gaze and the direction of their face by analyzing the image (driver's face image) captured by the in-vehicle camera 9a.

[0026] The in-vehicle detection unit 9b is a grip sensor that detects the driver's grip on the steering wheel ST (hereinafter sometimes referred to as the grip sensor 9b), and is provided, for example, on at least a part of the steering wheel ST. A torque sensor 4c that detects the driver's steering torque may also be used as the in-vehicle detection unit.

[0027] <Examples of Automatic Driving Control> Automatic driving control of a vehicle V includes, for example, acceleration / deceleration control, lane keeping control, and lane change control. Acceleration / deceleration control is a function (ACC: Adaptive Cruise Control) in which the controller 1 automatically controls the power unit 2 and hydraulic system 3 based on the detection results of the surrounding detection unit 8 and map information to automatically control the acceleration and deceleration of the vehicle V within a predetermined vehicle speed. With ACC, if there is a vehicle in front, it is also possible to accelerate or decelerate the vehicle V to maintain a safe distance from the vehicle in front. ACC reduces the burden on the driver in terms of acceleration and deceleration operations (operations on the accelerator pedal AP and brake pedal BP).

[0028] Lane keeping control is a function (LKAS: Lane Keeping Assist System) in which the controller 1 automatically controls the electric power steering system 4 based on the detection results of the surrounding detection unit 8 and map information to keep the vehicle V inside the lane. LKAS reduces the burden on the driver in terms of steering operations (operations on the steering wheel ST) while the vehicle V is moving straight.

[0029] Lane change control is a function (ALC: Auto Lane Changing, ALCA: Active Lane Change Assist) in which the controller 1 automatically controls the power unit 2, hydraulic system 3, and electric power steering system 4 based on the detection results of the surrounding detection unit 8 and map information to change the lane the vehicle V is traveling in to an adjacent lane. ALC is lane change control based on system requests, and ALCA is lane change control based on occupant requests. Examples of system requests include when a navigation system that guides the vehicle V to a destination requests a lane change for the vehicle V, or when overtaking a preceding vehicle regardless of whether route guidance is provided. When an occupant request is made, the driver instructs the lane change by operating an input device (e.g., the turn signal lever 6b). ALC or ALCA reduces the driver's operational burden of accelerating, decelerating, and steering the vehicle V when changing lanes.

[0030] Other examples of automated driving control may include, for example, collision mitigation braking, ABS function, traction control, and / or vehicle attitude control, which avoid collisions with road targets (e.g., pedestrians, other vehicles, or obstacles) by controlling the hydraulic system 3.

[0031] Automated driving control where the decision-making entity is a human (driver) can be called driver assistance. Driver assistance corresponds to levels 1 and 2 of the five levels defined by the Society of Automotive Engineers (SAE). Automated driving control where the decision-making entity is the vehicle can be called autonomous driving. Autonomous driving corresponds to levels 3 to 5 of the five levels defined by the SAE.

[0032] <Camera Arrangement and Imaging Range> The arrangement and imaging range of the multiple cameras mounted on vehicle V will be described with reference to Figure 2. The upper part of Figure 2 shows a plan view of vehicle V, and the lower part of Figure 2 shows a cross-sectional view of vehicle V. Some cameras are shown in only one of the plan view or the cross-sectional view. The camera arrangement and imaging range shown in Figure 2 is an example. Each camera may be arranged in a different position or have a different imaging range, as long as the object being imaged is maintained. Also, vehicle V may not include some of the cameras shown in Figure 2, and may include cameras not shown in Figure 2. In the following description, the images captured by the cameras are typically moving images, but may also be still images.

[0033] Cameras 201 to 203 are configured to capture images of the environment outside the vehicle V. Specifically, cameras 201 and 202 are configured to capture images of the environment in front of the vehicle V. Camera 202 is equipped with a telephoto lens, allowing it to capture images at a greater distance than camera 201. Images captured by cameras 201 and 202 may be analyzed in automatic driving control to recognize traffic participants (vehicles and pedestrians) and road infrastructure (traffic lights, signs, lane markings, etc.) in front of the vehicle V. Camera 203 is configured to capture images of the environment behind the vehicle V. Images captured by camera 203 may be analyzed in automatic driving control to recognize traffic participants (vehicles and pedestrians) behind the vehicle V. Images captured by camera 203 may be displayed on the display device 5a while the vehicle V's shift position is in reverse. Furthermore, cameras 201 to 203 may be used as part of a drive recorder, and images captured by cameras 201 to 203 may be stored in the vehicle V's storage device.

[0034] Cameras 211 to 214 are each configured to capture images of the environment outside the vehicle V. Specifically, camera 211 is configured to capture images of the environment in front of the vehicle V, camera 212 is configured to capture images of the environment behind the vehicle V, camera 213 is configured to capture images of the environment to the right of the vehicle V, and camera 214 is configured to capture images of the environment to the left of the vehicle V. Each of cameras 211 to 214 is fitted with a fisheye lens. Each of cameras 211 to 214 has a wider field of view than camera 201. The images captured by cameras 211 to 214 may be processed to generate an image (overhead view image) that looks as if the vehicle V is viewed from above. The overhead view image may be displayed on the display device 5a. In addition, the overhead view image may be transmitted to the user's mobile terminal so that the user can remotely observe the environment around the vehicle V while the vehicle V is parked. Furthermore, in addition to or instead of the overhead view image, a wide-angle image captured by any of the cameras 211 to 214 may be displayed on the display device 5a or the mobile terminal.

[0035] Cameras 221 to 223 are each configured to capture images of the environment outside the vehicle V. Specifically, camera 221 is configured to capture images of the environment behind the vehicle V. The image captured by camera 221 is displayed in the rearview mirror. In this way, camera 221 is used as part of a digital rearview mirror. Camera 222 is configured to capture images of the environment to the right rear of the vehicle V. The image captured by camera 222 is displayed in the right side mirror. Camera 223 is configured to capture images of the environment to the left rear of the vehicle V. The image captured by camera 223 is displayed in the left side mirror. In this way, cameras 222 and 223 are used as part of a digital side mirror. Image processing such as backlight correction and color correction may be performed on the images captured by cameras 221 to 223.

[0036] Cameras 231 to 234 are each configured to capture images of the environment outside of vehicle V. Specifically, camera 231 is configured to capture images of the environment in front of vehicle V, camera 232 is configured to capture images of the environment behind vehicle V, camera 233 is configured to capture images of the environment to the right of vehicle V, and camera 234 is configured to capture images of the environment to the left of vehicle V. Cameras 231 to 234 are each configured to capture an area closer to vehicle V than camera 201. Images captured by cameras 231 to 234 may be analyzed to recognize people around vehicle V. Controller 1 may unlock vehicle V if the recognized person is a registered person (e.g., the owner of vehicle V).

[0037] Cameras 241 to 244 are each configured to capture images of the environment outside of vehicle V. Specifically, camera 241 is configured to capture images of the environment to the right rear of vehicle V, camera 242 is configured to capture images of the environment to the right front of vehicle V, camera 243 is configured to capture images of the environment to the left rear of vehicle V, and camera 244 is configured to capture images of the environment to the left front of vehicle V. Images captured by cameras 241 and 242 may be analyzed in automatic driving control to recognize traffic participants (vehicles and pedestrians) and road traffic infrastructure (traffic lights, signs, lane markings, etc.) on the right side of vehicle V. Images captured by cameras 243 and 244 may be analyzed in automatic driving control to recognize traffic participants (vehicles and pedestrians) and road traffic infrastructure (traffic lights, signs, lane markings, etc.) on the left side of vehicle V. Images captured by cameras 241 to 244 may be analyzed to monitor the surroundings of vehicle V while vehicle V is parked. In Sentry Mode, Controller 1 may output an alarm sound towards the area around Vehicle V if a suspicious person is present around Vehicle V, or it may notify the owner of Vehicle V of the presence of a suspicious person. Sentry Mode is a mode in which, when the user is outside the vehicle, the system detects when a suspicious person approaches the vehicle and records video or sounds an alarm.

[0038] Cameras 251 to 253 are each configured to capture images of the interior of vehicle V or the occupants of vehicle V. Specifically, camera 251 is configured to capture the face of the driver of vehicle V. Camera 252 is configured to capture images of the entire driver's seat of vehicle V. Camera 253 is configured to capture images of the occupants in the front row and second row of vehicle V. Images captured by cameras 251 to 253 may be analyzed in automatic driving control to monitor the driver of vehicle V (for example, to determine whether surrounding area monitoring is being performed). Images captured by cameras 251 to 253 may be transmitted to the user's mobile terminal for the user to remotely observe the interior of vehicle V while the vehicle V is parked (for example, for pet mode or remote in-cabin). Pet mode is a mode that allows the user to check on their pet inside the vehicle via camera while outside the vehicle, or to turn on the air conditioner and set the interior temperature appropriately. Remote in-cabin is a mode that allows the user to check on the interior of the vehicle via camera while outside the vehicle. Images captured by cameras 251-253 may be analyzed to estimate the emotions of the occupants of vehicle V. Cameras 251-253 may be used as part of a drive recorder, and images captured by cameras 251-253 may be stored in the vehicle V's memory device.

[0039] <ECU for Autonomous Driving Control> Referring to Figure 3, an example configuration of the ECU 300 that performs automatic driving control will be described. The ECU 300 that performs automatic driving control may also be called an AD / ADAS (Autonomous Driving / Advanced Driver-Assistance Systems) ECU. The ECU 300 performs automatic driving control by controlling the actuators of the vehicle V based on various information detected by the vehicle V's detection unit. The ECU 300 is used as part of the vehicle V's controller 1.

[0040] The ECU 300 includes a circuit board 301 and a plurality of circuit components mounted on the circuit board 301. The plurality of circuit components mounted on the ECU 300 include a plurality of deserializer (Des) circuits 311 to 316, a semiconductor device 320, a power supply circuit 330, and a communication interface (I / F) 340. The ECU 300 may also include other circuit components (such as a power management integrated circuit and flash ROM).

[0041] The semiconductor device 320 performs automatic driving control of the vehicle V. The semiconductor device 320 may also be called a semiconductor chip, integrated circuit, or semiconductor integrated circuit. Multiple functions are implemented in the semiconductor device 320. In the example in Figure 3, the semiconductor device 320 implements the functions of a general-purpose processor 321, an image analysis processor 322, an image analysis processor 323, and DRAMs (Dynamic Random Access Memory) 324 to 326. These multiple functions may be implemented by a SoC (System on a Chip), SiP (System in Package), SoP (System on Package), MCM (Multi Chip Module), etc. The multiple functions implemented in the semiconductor device 320 constitute an image analysis system.

[0042] The general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 are each capable of performing image analysis. Image analysis is image processing for understanding or interpreting information represented by an image. Image analysis is different from image processing that does not involve understanding or interpretation, such as image transformation or filtering. The general-purpose processor 321 is also capable of performing image processing other than image analysis. The general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 constitute an image analysis system. The general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 may each be capable of performing image analysis using AI (Artificial Intelligence) algorithms, for example, they may be capable of performing image analysis using a trained model. DRAMs 324 to 326 are memories that store data used in these image analyses. In the example in Figure 3, the DRAMs 324 to 326 are included in the semiconductor device 320. Alternatively, the DRAMs 324 to 326 may be mounted on the circuit board 301 outside the semiconductor device 320.

[0043] In addition to image analysis, the general-purpose processor 321 has the function of controlling the entire semiconductor device 320. For example, the general-purpose processor 321 instructs the image analysis processors 322 and 323 on processing content, and controls the power supply from the power supply circuit 330 to the image analysis processors 322 and 323, respectively. The general-purpose processor 321 includes an SRAM (Static Random Access Memory) 327, the image analysis processor 322 includes an SRAM 328, and the image analysis processor 323 includes an SRAM 329. The SRAM of each processor is used as cache memory.

[0044] The Des circuits 311 to 316 each deserialize the image data supplied from the camera and supply it to the semiconductor device 320. The Des circuit 311 is connected to the cameras 201 to 203 by individual cables 370, connected to the image analysis processor 322 by the transmission line 361, and connected to the general-purpose processor 321 by the transmission line 362. The Des circuit 311 is not connected to the image analysis processor 323. In FIG. 3, only one cable 370 is labeled with a reference numeral, but all components connecting the camera and the Des circuit are the cable 370. The transmission line 361 is composed of a printed wiring formed on the circuit board 301, a lead formed on the package of the semiconductor device 320, and a wire connecting the pad of the semiconductor die of the semiconductor device 320 to the lead. The same applies to the transmission lines 362 to 368. Serialized image data is supplied to the Des circuit 311 from each of the cameras 201 to 203 through individual cables 370. The Des circuit 311 deserializes the image data supplied from the cameras 201 to 203 and supplies the deserialized image data to each of the general-purpose processor 321 and the image analysis processor 322.

[0045] The Des circuit 312 is connected to the cameras 211 to 214 by individual cables 370, connected to the image analysis processor 322 by the transmission line 363, and connected to the general-purpose processor 321 by the transmission line 364. The Des circuit 312 is not connected to the image analysis processor 323. Serialized image data is supplied to the Des circuit 312 from each of the cameras 211 to 214 through individual cables 370. The Des circuit 312 deserializes the image data supplied from the cameras 211 to 214 and supplies the deserialized image data to each of the general-purpose processor 321 and the image analysis processor 322.

[0046] The Des circuit 313 is connected to cameras 221 to 223 by individual cables 370 and is connected to the general-purpose processor 321 by the transmission line 365. The Des circuit 313 is not connected to either the image analysis processor 322 or the image analysis processor 323. Serialized image data is supplied to the Des circuit 313 from each of cameras 221 to 223 through the individual cables 370. The Des circuit 313 deserializes the image data supplied from cameras 221 to 223 and supplies the deserialized image data to the general-purpose processor 321.

[0047] The Des circuit 314 is connected to cameras 231 to 234 by individual cables 370 and is connected to the general-purpose processor 321 by the transmission line 366. The Des circuit 314 is not connected to either the image analysis processor 322 or the image analysis processor 323. Serialized image data is supplied to the Des circuit 314 from each of cameras 231 to 234 through the individual cables 370. The Des circuit 314 deserializes the image data supplied from cameras 231 to 234 and supplies the deserialized image data to the general-purpose processor 321.

[0048] The Des circuit 315 is connected to cameras 241 to 244 by individual cables 370 and is connected to the image analysis processor 323 by the transmission line 367. The Des circuit 315 is not connected to either the general-purpose processor 321 or the image analysis processor 322. Serialized image data is supplied to the Des circuit 315 from each of cameras 241 to 244 through the individual cables 370. The Des circuit 315 deserializes the image data supplied from cameras 241 to 244 and supplies the deserialized image data to the image analysis processor 323.

[0049] Des circuit 316 is connected to cameras 251-253 by individual cables 370 and to image analysis processor 323 by transmission line 368. Des circuit 315 is not connected to either the general-purpose processor 321 or the image analysis processor 322. Serialized image data is supplied to Des circuit 316 from each of the cameras 251-253 via individual cables 370. Des circuit 316 deserializes the image data supplied from cameras 251-253 and supplies the deserialized image data to image analysis processor 323.

[0050] As described above, each of the above-mentioned cameras is connected to at least one of the general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323. Specifically, each of the cameras 201-203 and 211-214 is connected to both the general-purpose processor 321 and the image analysis processor 322, but not to the image analysis processor 323. Each of the cameras 221-223 and 231-234 is connected to the general-purpose processor 321 and not to either the image analysis processor 322 or the image analysis processor 323. Each of the cameras 241-244 and 251-253 is connected to the image analysis processor 323 and not to either the general-purpose processor 321 or the image analysis processor 322.

[0051] The connection relationships between the multiple cameras and multiple processors described above are just one example; they may be connected in other ways. For example, some of the multiple cameras may be connected to all of the multiple processors, and some of the multiple processors may be connected to all of the multiple cameras.

[0052] The power supply circuit 330 converts the voltage supplied from the vehicle's power supply, located outside the ECU 300, into the operating voltage for each circuit component of the ECU 300 and supplies it to the ECU. The communication interface 340 is an interface used by the semiconductor device 320 to communicate with other ECUs in the vehicle.

[0053] The semiconductor device 320 will be described in detail below. The general-purpose processor 321 and the image analysis processor 322 are connected to each other by a transmission line 351. The general-purpose processor 321 and the image analysis processor 323 are connected to each other by a transmission line 352. The image analysis processor 322 and the image analysis processor 323 are connected to each other by a transmission line 353. The general-purpose processor 321 is connected to the DRAM 324 by a transmission line 354. The image analysis processor 322 is connected to the DRAM 325 by a transmission line 355. The image analysis processor 323 is connected to the DRAM 326 by a transmission line 356. Each of the transmission lines 351 to 356 is composed of, for example, a wiring layer formed on the semiconductor die of the semiconductor device 320, or a wire connecting the two semiconductor dies. In the example in Figure 3, the general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 are connected to separate DRAMs. Alternatively, the general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 may share DRAM.

[0054] The general-purpose processor 321 has different performance characteristics from both the image analysis processor 322 and the image analysis processor 323. The image analysis processors 322 and 323 may have equal or different performance characteristics. For example, the bandwidth of the transmission path 355 for the image analysis processor 322 to communicate with the DRAM 325 may be equal to the bandwidth of the transmission path 356 for the image analysis processor 323 to communicate with the DRAM 326. The same applies to other performance characteristics.

[0055] The following describes the differences between the performance of the general-purpose processor 321 and the performance of the image analysis processor 322. The differences between the performance of the general-purpose processor 321 and the performance of the image analysis processor 323 may be similar. For at least some of the performances described below, the order of magnitude may be reversed, or they may have equal performance.

[0056] The image processing capability of the image analysis processor 322 (e.g., 14 gigapixels / second) is higher than that of the general-purpose processor 321 (e.g., 3.6 gigapixels / second). The computing power of the image analysis processor 322 (e.g., 400 TPOS (Tera Operations Per Second)) is higher than that of the general-purpose processor 321 (e.g., 100 TOPS). Therefore, the image analysis processor 322 can perform image analysis faster than the general-purpose processor 321, but it also consumes more power.

[0057] The capacity of the SRAM 328 in the image analysis processor 322 (e.g., 12 megabits / tile) is smaller than the capacity of the SRAM 327 in the general-purpose processor 321 (e.g., 16 megabits / tile). The bandwidth of the transmission path 355 for the image analysis processor 322 to communicate with the DRAM 325 (e.g., 9.6 gigabits / second) is larger than the bandwidth of the transmission path 354 for the general-purpose processor 321 to communicate with the DRAM 324 (e.g., 8.5 gigabits / second). Therefore, the general-purpose processor 321 can perform simple processing faster than the image analysis processor 322. On the other hand, the image analysis processor 322 can perform complex processing that frequently accesses the DRAM 325 faster than the general-purpose processor 321.

[0058] The image analysis processor 322 incorporates technologies to accelerate image analysis using AI algorithms (e.g., softmax optimization, floating-point arithmetic, microscaling, and grid sampling improvement), which are not applied to the general-purpose processor 321. Therefore, the image analysis processor 322 can perform image analysis using AI algorithms faster than the general-purpose processor 321, but it also consumes more power.

[0059] In summary, the general-purpose processor 321 excels at image analysis with small input data and small pre-trained models, while the image analysis processor 322 excels at image analysis with large input data and large pre-trained models. Furthermore, the general-purpose processor 321 is better suited to small-scale image processing than the image analysis processor 322. Based on these performance differences, the processor responsible for analyzing images captured by multiple cameras is determined.

[0060] The processing of images captured by cameras 201-203 and 211-214 may require different processor performance depending on the state of the vehicle V. Therefore, it may be possible to change whether the general-purpose processor 321 or the image analysis processor 322 processes these images. For example, when the vehicle V is in manual driving mode or parked mode, no advanced processing is required for these images. In such cases, the general-purpose processor 321 performs image processing. This image processing may or may not include image analysis. On the other hand, when the vehicle V is in an automatic driving control state (especially an automatic driving state), advanced analysis is required for these images. In such cases, the image analysis processor 322 performs image analysis.

[0061] As described above, images captured by cameras 201-203 and 211-214 are supplied to both the general-purpose processor 321 and the image analysis processor 322. The general-purpose processor 321 may stop image analysis by the image analysis processor 322 when it is performing image processing itself. To stop image analysis by the image analysis processor 322, the general-purpose processor 321 may send a signal to the image analysis processor 322 requesting it to stop image analysis, or it may stop supplying power to the image analysis processor 322. The general-purpose processor 321 may stop its own image analysis when it is having the image analysis processor 322 perform image analysis. To have the image analysis processor 322 perform image analysis, the general-purpose processor 321 may send a signal to the image analysis processor 322 requesting it to perform image analysis, or it may supply power to the image analysis processor 322. Alternatively, images captured by cameras 201-203 and 211-214 may be analyzed by both the general-purpose processor 321 and the image analysis processor 322. Instead of supplying images captured by cameras 201-203 and 211-214 to both the general-purpose processor 321 and the image analysis processor 322, a switch circuit may be used to switch the destination of these images.

[0062] Since the images captured by cameras 221-223 are displayed in the rearview mirror or side mirror, processing these images requires immediate image processing and display output in a short time after camera activation. Specifically, the vehicle V is required to be activated immediately by an activation trigger from a power-off state, perform facial recognition processing from the acquired camera images, and unlock the doors. On the other hand, complex analysis of these images is not performed. Therefore, the images captured by these cameras are processed by a general-purpose processor 321.

[0063] Since the images captured by cameras 231 to 234 are used to unlock vehicle V, the analysis of these images requires responsive image processing and display output in a short time after camera activation, similar to the analysis of images from cameras 221 to 223. For this reason, the images captured by these cameras are analyzed by a general-purpose processor 321.

[0064] The processing of images captured by cameras 241-244 and 251-253 may require different processor performance depending on the state of the vehicle V. Therefore, it may be possible to change whether the general-purpose processor 321 or the image analysis processor 323 processes the images captured by these cameras. For example, when the vehicle V is in manual driving mode or parked mode, no advanced processing is required for these images. In such cases, the general-purpose processor 321 performs image processing. On the other hand, when the vehicle V is in automatic driving control mode (especially automatic driving mode), advanced analysis is required for these images. In such cases, the image analysis processor 323 performs image analysis.

[0065] As described above, the images captured by cameras 241-244 and 251-253 are supplied to the image analysis processor 323 and not to the general-purpose processor 321. The general-purpose processor 321 may send a signal to the image analysis processor 323 requesting it to stop image analysis in order to stop image analysis when it is performing image processing itself. The general-purpose processor 321 may also request the image analysis processor 323 to transfer the images supplied from these cameras to it in order to perform image processing itself. The general-purpose processor 321 may send a signal to the image analysis processor 323 requesting it to perform image analysis when it wants the image analysis processor 323 to perform image analysis.

[0066] <Method for controlling semiconductor device 320> A method for controlling the semiconductor device 320 will be described with reference to Figure 4. Each step in Figure 4 is executed by the general-purpose processor 321. The method in Figure 4 starts when power is supplied to the general-purpose processor 321 from the power supply of the vehicle V and continues as long as power is supplied to the general-purpose processor 321. At the start of the method in Figure 4, it is assumed that the general-purpose processor 321, the image analysis processor 322, and the image analysis processor 323 are all operational and performing image processing.

[0067] In S401, the general-purpose processor 321 determines whether the vehicle V has transitioned to the parked state. If the general-purpose processor 321 determines that the vehicle V has transitioned to the parked state (YES in S401), it proceeds to S402; otherwise (NO in S401), it repeats S401. In other words, the general-purpose processor 321 waits until the vehicle V transitions to the parked state. The general-purpose processor 321 may determine that the vehicle V has transitioned to the parked state based on the driver of the vehicle V inputting the shift position to parking, or it may determine that the vehicle V has transitioned to the parked state based on the electric parking brake device 3c being activated in response to an instruction from the driver.

[0068] In S402, the general-purpose processor 321 determines whether monitoring of the parked vehicle V is enabled. If the general-purpose processor 321 determines that monitoring of the parked vehicle V is enabled (YES in S402), it proceeds to S403; otherwise (NO in S402), it proceeds to S404. Monitoring of vehicle V may involve monitoring the environment outside of vehicle V, or monitoring the interior of vehicle V or its occupants. The determination of whether monitoring of vehicle V is enabled may be based on settings set by the user of vehicle V and stored in the vehicle V's storage device. For example, if pet mode, remote in-cabin mode, or sentry mode is enabled, it is determined that monitoring of vehicle V is enabled.

[0069] S403 is executed when vehicle V is parked and monitoring of vehicle V is enabled. The general-purpose processor 321 transitions the image analysis processor 322 to a non-operational state and the image analysis processor 323 to a transfer state. The non-operational state of the image analysis processor means that it does not perform image analysis and does not transfer image data. The transfer state of the image analysis processor means that it does not perform image analysis but does transfer image data. For example, the general-purpose processor 321 may stop supplying power to the image analysis processor 322 in order to transition it to a non-operational state. The general-purpose processor 321 may send a request to the image analysis processor 322 to transition it to a transfer state in order to transition the image analysis processor 323 to a transfer state. In response to this instruction, the image analysis processor 322 stops performing image analysis and starts transferring image data.

[0070] Subsequently, the general-purpose processor 321 processes the images supplied by cameras 201-203, 211-214, 221-223, 231-234, 241-244, and 251-253. For example, the general-purpose processor 321 may store the images captured by cameras 201-203 in the vehicle V's storage device. The general-purpose processor 321 may generate an overhead view image from the images captured by cameras 251-253 in response to instructions from the vehicle V's user and transmit it to the user's mobile terminal. The general-purpose processor 321 may display the images captured by cameras 221-223 on the vehicle V's rearview mirror and side mirrors. The general-purpose processor 321 may analyze the images captured by cameras 231-234 to recognize the presence of registered persons around the vehicle V. The general-purpose processor 321 may analyze the images captured by cameras 241 to 244 to recognize the presence of a suspicious person around vehicle V or any suspicious action taken against vehicle V. The general-purpose processor 321 may also analyze the images captured by cameras 251 to 253 to monitor the conditions inside vehicle V, or it may transmit these images to the user's mobile terminal.

[0071] S404 is executed when vehicle V is parked and monitoring of vehicle V is disabled. The general-purpose processor 321 transitions image analysis processors 322 and 323 to a non-operational state. In this case, images from cameras 241-244 and 251-253 are not processed. Therefore, the general-purpose processor 321 may also transition cameras 241-244 and 251-253 to a non-operational state. A non-operational state for a camera means that it is not capturing images.

[0072] Subsequently, the general-purpose processor 321 processes the images supplied from cameras 201-203, 211-214, 221-223, and 231-234. This image processing may be the same as the image processing described in S403.

[0073] In S405, the general-purpose processor 321 determines whether the vehicle V has transitioned from the parked state to another state. If the general-purpose processor 321 determines that the vehicle V has transitioned to another state (YES in S405), it proceeds to S406; otherwise (NO in S405), it repeats S405. In other words, the general-purpose processor 321 waits until the vehicle V transitions from the parked state to another state. The general-purpose processor 321 may determine that the vehicle V has transitioned to another state based on the driver of the vehicle V inputting a shift position other than parking, or it may determine that the vehicle V has transitioned to another state based on the electric parking brake device 3c ceasing to operate in response to an instruction from the driver.

[0074] S405 is executed when the vehicle V is in a state other than parked. The general-purpose processor 321 transitions the image analysis processors 322 and 323 to an operational state. The operational state of an image analysis processor is a state in which image analysis is performed. For example, the general-purpose processor 321 may supply power to each image analysis processor to transition the image analysis processor 322 to an operational state, or it may send a request to each image analysis processor to transition to an operational state. The image analysis processors 322 and 323 each start analyzing images supplied from the connected camera in response to transitioning to an operational state. For example, the image analysis processors 322 and 323 each perform image analysis for automatic driving control.

[0075] Subsequently, the general-purpose processor 321 processes the images supplied from cameras 221-223 and 231-234. This image processing may be the same as the image processing described in S403. The general-purpose processor 321 may also process the images supplied from cameras 201-203 and 211-214, or it may not.

[0076] In the method shown in Figure 4, the image analysis processors 322 and 323 become operational when the vehicle V is in a state other than parked. Alternatively, the image analysis processors 322 and 323 may become operational only when the vehicle V is in an automatic driving control state (i.e., an automatic driving state and a driving assistance state), or only when the vehicle V is in an automatic driving state.

[0077] <Structure of Semiconductor Device 320> An example of the structure of the semiconductor device 320 will be described with reference to Figures 5 and 6. In each figure, the upper part shows a plan view of the semiconductor device 320, and the lower part shows a side view of the semiconductor device 320. In Figure 5, the semiconductor device 320 is mounted by an SoC. The semiconductor device 320 includes one semiconductor die 500 and one package 510. Figure 5 shows the case where the package 510 is of the BGA (Ball Grid Array) type, but other types of packages may be used. The same applies to Figure 6.

[0078] A general-purpose processor 321 is formed in region 501 of the semiconductor die 500. An image analysis processor 322 is formed in region 502 of the semiconductor die 500. An image analysis processor 323 is formed in region 503 of the semiconductor die 500. DRAMs 324 to 326 are formed in region 504 of the semiconductor die 500. Thus, in the example of Figure 5, multiple functional blocks of the semiconductor device 320 are formed on a single semiconductor die 500. In Figure 5, the DRAMs 324 to 326 are formed in one region 504, but instead, the DRAMs 324 to 326 may be formed distributed across multiple regions. Regions 501 to 504 are connected to each other by wiring layers formed on the semiconductor die 500.

[0079] The package 510 includes one substrate 511, a plurality of solder balls 512, and one sealing resin 513. In the plan view of Figure 5, the sealing resin 513 is shown transparently for illustrative purposes. The semiconductor die 500 is mounted on the substrate 511 and sealed by the sealing resin 513. The electrodes of the semiconductor die 500 are electrically connected to the solder balls 512 through a wiring layer formed on the substrate 511. The semiconductor die 500 is housed in the package 510.

[0080] In Figure 6, the semiconductor device 320 is mounted by an MCM. The semiconductor device 320 includes a plurality of semiconductor dies 601 to 604 and one package 610.

[0081] A general-purpose processor 321 is formed on semiconductor die 601. An image analysis processor 322 is formed on semiconductor die 602. An image analysis processor 323 is formed on semiconductor die 603. DRAMs 324 to 326 are formed on semiconductor die 604. Thus, in the example of Figure 6, multiple functional blocks of the semiconductor device 320 are formed on separate semiconductor dies 601 to 604. In Figure 6, DRAMs 324 to 326 are formed on a single semiconductor die 604, but instead, DRAMs 324 to 326 may be distributed and formed on multiple semiconductor dies.

[0082] The package 610 includes one substrate 611, a plurality of solder balls 612, and one sealing resin 613. In the plan view of Figure 6, the sealing resin 613 is shown transparently for illustrative purposes. The semiconductor dies 601 to 604 are mounted on the substrate 611 and sealed by the sealing resin 613. The semiconductor dies 601 to 604 are connected to each other by wiring formed on the substrate 611. The electrodes of the semiconductor dies 601 to 604 are electrically connected to the solder balls 612 through the wiring layer formed on the substrate 611. The semiconductor dies 601 to 604 are housed in the package 610.

[0083] In the example shown in Figure 5 above, multiple processors constituting the image analysis system (general-purpose processor 321, image analysis processor 322, and image analysis processor 323) are housed in a single package 610. Alternatively, these multiple processors may be housed in separate packages and formed as separate semiconductor devices.

[0084] <Summary of Embodiments> (Item 1) An image analysis system (320) comprising a first processor (321), a second processor (322), and a third processor (323), each capable of performing image analysis, wherein the bandwidth of a first transmission line (354) for the first processor to communicate with a memory (324) is different from the bandwidth of a second transmission line (355) for the second processor to communicate with a memory (324) and the bandwidth of a third transmission line (356) for the third processor to communicate with a memory (324). According to this item, since the image analysis system has multiple processors with different performance levels, the processor corresponding to the performance required for image analysis can perform the image analysis, and the image analysis can be performed efficiently. In addition, since the image analysis system has multiple processors, the load of image analysis is distributed and a redundant configuration can be constructed. (Item 2) The image analysis system according to Item 1, wherein the bandwidth of the second transmission line is greater than the bandwidth of the first transmission line. According to this item, image analysis can be efficiently performed by having the second processor perform image analysis that requires a large transmission line bandwidth. (Item 3) The image analysis system according to Item 1 or 2, wherein the first processor includes a first SRAM (325), the second processor includes a second SRAM (326), and the capacity of the second SRAM is different from the capacity of the first SRAM. According to this item, image analysis can be efficiently performed by assigning processing to the processor according to the SRAM capacity required for the image analysis. (Item 4) The image analysis system according to Item 3, wherein the capacity of the first SRAM is greater than the capacity of the second SRAM. According to this item, image analysis can be efficiently performed by having the first processor perform image analysis that requires a large SRAM capacity. (Item 5) The image analysis system according to any one of Items 1 to 4, wherein the bandwidth of the second transmission line is equal to the bandwidth of the third transmission line. According to this item, two processors having the same performance with respect to the transmission line bandwidth are provided.(Item 6) An image analysis system according to any one of items 1 to 5, wherein the first processor controls the power supply to the second processor and the third processor, respectively. According to this item, power supply to the second processor can be stopped when the second processor and the third processor are not performing image analysis, thereby reducing power consumption. (Item 7) An image analysis system according to any one of items 1 to 6, wherein one or more semiconductor dies (500, 601-604) on which the first processor, the second processor and the third processor are formed are housed in a single package (510, 610). According to this item, the above system can be implemented in a single semiconductor device. (Item 8) A vehicle (V) comprising an image analysis system as described in any one of items 1 to 7, and a plurality of cameras (201-203, 211-214, 221-223, 231-234, 241-244, 251-253), wherein each of the plurality of cameras is connected to at least one of the first processor, the second processor, and the third processor, and at least one of the first processor, the second processor, and the third processor is connected to only some of the plurality of cameras. According to this item, image analysis can be performed with an appropriate processor depending on the purpose of analyzing the images captured by the cameras. (Item 9) The vehicle according to item 8, wherein the plurality of cameras include a first camera (201-203, 211-214) configured to capture the environment outside the vehicle, and the first camera is connected to both the first processor and the second processor. According to this item, image analysis can be performed with an appropriate processor depending on the state of the vehicle. (Item 10) The vehicle according to Item 9, wherein it is possible to change whether the first processor or the second processor processes the image captured by the first camera. According to this item, image analysis can be performed by the appropriate processor depending on the state of the vehicle. (Item 11) The vehicle according to Item 9 or 10, wherein when the vehicle is parked, the image captured by the first camera is processed by the first processor.According to this item, image analysis can be performed with an appropriate processor depending on the state of the vehicle. (Item 12) The vehicle according to any one of items 9 to 11, wherein when the vehicle is in an automated driving state or a driver assistance state, the image captured by the first camera is analyzed by the second processor. According to this item, image analysis can be performed with an appropriate processor depending on the state of the vehicle. (Item 13) The vehicle according to any one of items 8 to 12, wherein the plurality of cameras include second cameras (251 to 253) configured to capture images of the interior of the vehicle or the occupants of the vehicle, and the second cameras are connected to the third processor. According to this item, image analysis can be performed with an appropriate processor depending on the state of the vehicle. (Item 14) The vehicle according to item 13, wherein when the vehicle is in a state where it is monitoring the interior of the vehicle or the occupants of the vehicle, the image captured by the second camera is processed by the third processor, and when the vehicle is not in a state where it is monitoring the interior of the vehicle or the occupants of the vehicle, the image captured by the second camera is not processed. According to this section, only the necessary processes can be performed depending on the vehicle's condition.

[0085] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention.

Claims

1. An image analysis system comprising a first processor, a second processor, and a third processor, each capable of performing image analysis, wherein the bandwidth of a first transmission path for the first processor to communicate with memory is different from the bandwidth of a second transmission path for the second processor to communicate with memory and the bandwidth of a third transmission path for the third processor to communicate with memory.

2. The image analysis system according to claim 1, wherein the bandwidth of the second transmission line is greater than the bandwidth of the first transmission line.

3. The image analysis system according to claim 1 or 2, wherein the first processor includes a first SRAM, the second processor includes a second SRAM, and the capacity of the second SRAM is different from the capacity of the first SRAM.

4. The image analysis system according to claim 3, wherein the capacity of the first SRAM is greater than the capacity of the second SRAM.

5. The image analysis system according to any one of claims 1 to 4, wherein the bandwidth of the second transmission line is equal to the bandwidth of the third transmission line.

6. The image analysis system according to any one of claims 1 to 5, wherein the first processor controls the power supply to the second processor and the third processor, respectively.

7. The image analysis system according to any one of claims 1 to 6, wherein one or more semiconductor dies on which a first processor, a second processor, and a third processor are formed are housed in a single package.

8. A vehicle comprising: an image analysis system according to any one of claims 1 to 7; and a plurality of cameras, each of the plurality of cameras being connected to at least one of the first processor, the second processor, and the third processor, and at least one of the first processor, the second processor, and the third processor being connected to only some of the cameras among the plurality of cameras.

9. The vehicle according to claim 8, wherein the plurality of cameras include a first camera configured to capture images of the environment outside the vehicle, and the first camera is connected to both the first processor and the second processor.

10. The vehicle according to claim 9, wherein it is possible to change whether the first processor or the second processor processes the image captured by the first camera.

11. The vehicle according to claim 9 or 10, wherein when the vehicle is parked, an image captured by the first camera is processed by the first processor.

12. The vehicle according to any one of claims 9 to 11, wherein when the vehicle is in an autonomous driving state or a driver assistance state, an image captured by the first camera is analyzed by the second processor.

13. The vehicle according to any one of claims 8 to 12, wherein the plurality of cameras include a second camera configured to image the interior of the vehicle or the occupants of the vehicle, and the second camera is connected to the third processor.

14. The vehicle according to claim 13, wherein when the vehicle is in a state of monitoring the interior of the vehicle or the occupants of the vehicle, an image captured by the second camera is processed by the third processor, and when the vehicle is not in a state of monitoring the interior of the vehicle or the occupants of the vehicle, an image captured by the second camera is not processed.

Images