Information processing system, information processing method, and program

Abstract

The present technology relates to an information processing system, an information processing method, and a program that make it possible to generate image data close to image data actually obtained using a camera. The information processing system comprises: a first image generation unit that generates, from first image data obtained by a first camera, second image data in which the influence of characteristics of the first camera has been reduced; and a second image generation unit that generates third image data in which the influence of characteristics of a second camera has been added to the second image data. The present technology can be applied to, for example, a system that generates learning data for machine learning.

Description

Information Processing System, Information Processing Method, and Program

[0001] The present technology relates to an information processing system, an information processing method, and a program, and particularly relates to an information processing system, an information processing method, and a program suitable for use in generating learning image data.

[0002] For the development of autonomous driving technology, learning data including a large amount of image data is required. However, the work of collecting a large amount of image data corresponding to various situations (for example, accidents, etc.) in the real world is very difficult, and the required cost also increases.

[0003] On the other hand, conventionally, a technique has been proposed in which, by applying the parameters of a target camera to a CG (Computer Graphics) model, image data corresponding to the camera is generated (for example, see Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2023-56056

[0005] However, since the CG model is different from the real world, the difference between the image data generated using the technique described in Patent Document 1 and the image data actually obtained by the target camera becomes large.

[0006] The present technology has been made in view of such a situation, and enables the generation of image data close to the image data actually obtained using a camera.

[0007] An information processing system according to one aspect of the present technology includes a first image generation unit that generates second image data with the influence of the characteristics of the first camera reduced from first image data obtained by the first camera, and a second image generation unit that generates third image data with the influence of the characteristics of the second camera added to the second image data.

[0008] One aspect of this technology is an information processing method in which an information processing system generates a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced, and generates a third image data by adding the influence of the characteristics of the second camera to the second image data.

[0009] One aspect of this technology involves a program that causes a computer to perform a process that includes generating a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced, and generating a third image data by adding the influence of the characteristics of the second camera to the second image data.

[0010] In one aspect of this technology, a second image data is generated from the first image data obtained by the first camera, with the influence of the characteristics of the first camera reduced, and a third image data is generated by adding the influence of the characteristics of the second camera to the second image data.

[0011] This is a block diagram showing an example of the configuration of a vehicle control system. This is a diagram showing an example of the sensing area of ​​an external recognition sensor in the vehicle control system of Figure 1. This is a block diagram showing an example of the configuration of an information processing system to which this technology is applied. This is a block diagram showing an example of the configuration of the image processing unit of Figure 3. This is a flowchart for explaining the learning model generation process. This is a diagram showing an example of the configuration of a computer.

[0012] The following describes the embodiments for implementing this technology. The explanation will proceed in the following order: 1. Example of a vehicle control system configuration 2. Embodiment 3. Modified examples 4. Others

[0013] <<1. Example of Vehicle Control System Configuration>> Figure 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is a non-limiting example of a mobile device control system to which this technology is applied.

[0014] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to the automation of the vehicle's operation. This automation includes Level 1 to Level 5 driving automation, as well as remote driving and / or remote assistance of the vehicle 1 by a remote driver. The levels of driving automation may refer to the Society of Automotive Engineers (SAE) J3016™ APL2021 Levels of Driving Automation, where SAE Level 0 represents the lowest level of driving automation and SAE Level 5 represents the highest level of driving automation. For example, SAE Level 1 driving automation may consist of driver assistance functions that provide the driver with steering or brake / acceleration support, and SAE Level 5 driving automation may consist of an automated driving function that can drive the vehicle under all conditions.

[0015] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, an automated driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0016] Two or more (or, in some cases, all) of the following components are connected to communicate with each other via a communication network 41: the vehicle control ECU 21, the communication unit 22, the map information storage unit 23, the location information acquisition unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the memory unit 28, the driving automation control unit 29, the DMS 30, the HMI 31, and the vehicle control unit 32. The communication network 41 is composed of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay®, and Ethernet®. In some embodiments, the communication network 41 may have two or more types of communication networks, and different types of communication networks may be used depending on the type of data being transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-capacity data. In some embodiments, two or more (or possibly all) units of the vehicle control system 11 may be directly connected using wireless communication (e.g., relatively short-range communication) without going through the communication network 41. In some embodiments, the wireless communication may use near-field wireless communication technology. Non-limiting examples of near-field wireless communication technology include near-field communication (NFC) and Bluetooth®. In some embodiments, two or more (or possibly all) units of the vehicle control system 11 may be connected using the communication network 41 and wireless communication technology (e.g., near-field wireless communication technology).

[0017] In the following embodiment, where two or more units of the vehicle control system 11 communicate via the communication network 41, the description of the communication network 41 will be omitted. For example, in an embodiment where the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the vehicle control ECU 21 and the communication unit 22 communicating.

[0018] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0019] The communication unit 22 communicates with various devices inside the vehicle 1 (hereinafter referred to as in-vehicle devices), various devices outside the vehicle 1 (hereinafter referred to as external devices), other vehicles, base stations, etc., and transmits and receives various types of data. In some embodiments, the communication unit 22 may use multiple communication technologies to perform communication.

[0020] A non-limiting example of communication between the communication unit 22 and external equipment will be briefly described. In some embodiments, the communication unit 22 may communicate with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication technology. Examples of non-limiting wireless communication technologies include 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. External networks that the communication unit 22 can communicate with may include, for example, the internet, a cloud network, or a network specific to a carrier. The communication technology used by the communication unit 22 to communicate with an external network is not particularly limited, as long as it is a wireless communication technology that enables digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0021] In some embodiments, the communication unit 22 may communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by relatively slow-moving objects such as pedestrians and cyclists, terminals installed in fixed locations such as stores, and / or MTC (Machine Type Communication) terminals. In some embodiments, the communication unit 22 may perform V2X (Vehicle to Everything) communication. V2X communication generally refers to communication between the vehicle and other entities. Non-exclusive examples of V2X communication include vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with homes, and vehicle-to-pedestrian communication with terminals carried or worn by pedestrians.

[0022] In some embodiments, the communication unit 22 may receive a program from outside the vehicle 1 to update the software that controls the operation of the vehicle control system 11 (for example, over the air). In some embodiments, the communication unit 22 may receive map information, traffic information, information about the vehicle 1's surroundings, etc., from outside the vehicle 1. In some embodiments, the communication unit 22 may transmit information about the vehicle 1, information about the vehicle 1's surroundings, etc., to an external device or external network. Non-limiting examples of information about the vehicle 1 that the communication unit 22 transmits to an external device or external network include data indicating the status of the vehicle 1, recognition results from the recognition unit 73, etc. In some embodiments, the communication unit 22 may communicate with a vehicle emergency call system. Non-limiting examples of a vehicle emergency call system include e-Call, etc.

[0023] In some embodiments, the communication unit 22 may receive electromagnetic waves transmitted by a road traffic information communication system. In some embodiments, such electromagnetic waves may be transmitted using radio beacons, optical beacons, FM multiplex broadcasting, etc.

[0024] A non-limiting example of communication with in-vehicle equipment that the communication unit 22 can perform will be outlined below. In some embodiments, the communication unit 22 may communicate with in-vehicle equipment using wireless communication. For example, in some embodiments, the communication unit 22 may communicate with in-vehicle equipment wirelessly using wireless communication technology that enables digital bidirectional communication at a predetermined or higher communication speed. Non-limiting examples of wireless communication technologies include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 may also communicate with in-vehicle equipment using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 22 may communicate with in-vehicle equipment via wired communication through a cable connected to a connection terminal (not shown). In some embodiments, the communication unit 22 may communicate with in-vehicle equipment using wired communication technology that enables digital bidirectional communication at a predetermined or higher communication speed. Non-exclusive examples of wired communication technologies include USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link).

[0025] Here, in-vehicle equipment refers to, for example, equipment located inside vehicle 1 that is not connected to the communication network 41. In-vehicle equipment is divided into equipment that constitutes the vehicle control system 11 and equipment that does not. Non-exclusive examples of in-vehicle equipment that does not constitute the vehicle control system 11 include mobile devices and wearable devices owned by users of vehicle 1 (e.g., the driver, passengers), and information equipment temporarily installed inside vehicle 1. These devices can, for example, be moved outside vehicle 1 and become external equipment.

[0026] The map information storage unit 23 stores maps acquired from external devices or external networks and / or maps created by the vehicle 1. For example, the map information storage unit 23 may store three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, etc.

[0027] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map may be a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and may be provided to vehicle 1 from an external server or the like. A point cloud map may be a map composed of point clouds (point cloud data). A vector map may be a map adapted for automated driving by associating traffic information, such as the locations of lanes and traffic lights, with a point cloud map.

[0028] The point cloud map and vector map may be provided from, for example, an external server, or they may be created in the vehicle 1 as maps for matching with the local map described later, based on sensing results from the camera 51, radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred meters square, relating to the planned route that the vehicle 1 will travel may be obtained from the external server.

[0029] The location information acquisition unit 24 acquires location information of the vehicle 1. The acquired location information may be supplied to the driving automation control unit 29. In some embodiments, the location information acquisition unit 24 may receive GNSS (Global Navigation Satellite System) signals from GNSS satellites. In some embodiments, the location information acquisition unit 24 may receive signals from beacons or the like.

[0030] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the external recognition sensor 25 are arbitrary.

[0031] In some embodiments, the external recognition sensor 25 may include a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0032] Camera 51 can use any suitable shooting method. In some embodiments, camera 51 may use a shooting method capable of distance measurement. Non-limiting examples of cameras using a shooting method capable of distance measurement include ToF (Time of Flight) cameras, stereo cameras, monocular cameras, and infrared cameras. However, camera 51 may not be limited to these and may simply be for acquiring images, regardless of distance measurement.

[0033] In some embodiments, the external recognition sensor 25 may include environmental sensors for detecting characteristics of the environment around the vehicle 1. Non-limiting examples of detectable environmental characteristics include weather, climate, brightness, etc. In some embodiments, the environmental sensors may include various sensors such as raindrop sensors, fog sensors, sunshine sensors, snow sensors, and illuminance sensors.

[0034] In some embodiments, the external recognition sensor 25 may include a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0035] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle 1, and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the in-vehicle sensor 26 are not particularly limited, as long as they are of a type and number that can be realistically installed in the vehicle 1.

[0036] In some embodiments, the in-vehicle sensor 26 may include one or more sensors from among a camera, radar, seat sensor, microphone, and biosensor. In some embodiments, the camera included in the in-vehicle sensor 26 may use a distance-measuring shooting method. Non-limiting examples of cameras using a distance-measuring shooting method include ToF cameras, stereo cameras, monocular cameras, and infrared cameras. However, the camera included in the in-vehicle sensor 26 may not be for distance measurement and may simply be for acquiring captured images. The biosensor included in the in-vehicle sensor 26 may be installed, for example, on the seat or steering wheel, and may detect various biometric information of the user.

[0037] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the vehicle sensor 27 are not particularly limited, as long as they are of a type and number that can be realistically installed on the vehicle 1.

[0038] In some embodiments, the vehicle sensor 27 may include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU) integrating them. In some embodiments, the vehicle sensor 27 may include a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal (e.g., pedal force, pedal stroke), and / or a brake sensor for detecting the amount of operation of the brake pedal (e.g., pedal force, pedal stroke). In some embodiments, the vehicle sensor 27 may include a rotation sensor for detecting the rotational speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and / or a wheel speed sensor for detecting the rotational speed of the wheels. In some embodiments, the vehicle sensor 27 may include a battery sensor for detecting the remaining charge and temperature of the battery, and / or an impact sensor capable of detecting external impacts.

[0039] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of storage mediums include magnetic storage devices such as EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access Memory), and / or HDD (Hard Disc Drive), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The storage unit 28 stores various programs and data used by one or more (or in some cases all) units of the vehicle control system 11. In some embodiments, the storage unit 28 may include an EDR (Event Data Recorder) or a DSSAD (Data Storage System for Automated Driving) to store information about the vehicle 1 before and after an event such as an accident, and information acquired by the in-vehicle sensors 26.

[0040] The driving automation control unit 29 controls the driving automation function of the vehicle 1. In some embodiments, the driving automation control unit 29 may include an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0041] The analysis unit 61 performs analysis processing of the situation of the vehicle 1 and / or its surroundings. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0042] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on sensor data from the external recognition sensor 25 and the high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 may generate a local map based on sensor data from the external recognition sensor 25 and perform matching between the local map and the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 may be based on, for example, the center of the rear wheels relative to the axle.

[0043] In some embodiments, the local map may be a three-dimensional high-precision map created using technologies such as SLAM (Simultaneous Localization and Mapping), an occupancy grid map, etc. The three-dimensional high-precision map may be, for example, the point cloud map described above. The occupancy grid map is a map that divides the three-dimensional or two-dimensional space around the vehicle 1 into grids of a predetermined size and indicates the occupancy state of objects in grid units. The occupancy state of an object may be indicated, for example, by the presence or absence of the object or the probability of existence. In some embodiments, the local map may also be used, for example, in the detection processing and / or recognition processing of the situation outside the vehicle 1 by the recognition unit 73.

[0044] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and / or sensor data from the vehicle sensor 27.

[0045] The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Examples of methods for combining different types of sensor data include, but are not limited to, compounding, integrating, fusing, associating, and the like.

[0046] The recognition unit 73 executes detection processing for detecting the situation outside the vehicle 1 and / or recognition processing for recognizing the situation outside the vehicle 1.

[0047] For example, the recognition unit 73 may perform detection processing and / or recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like.

[0048] Specifically, for example, the recognition unit 73 may perform detection processing and / or recognition processing of objects around the vehicle 1. Examples of the object detection processing may include processing for detecting the presence or absence, size, shape, position, movement, and the like of an object. Examples of the object recognition processing may include processing for recognizing attributes such as the type of an object or identifying a specific object. The detection processing and the recognition processing are not necessarily clearly separated and may overlap at least partially.

[0049] In some embodiments, the recognition unit 73 may detect an object around the vehicle 1 by performing clustering that classifies a point cloud based on sensor data from the radar 52 and / or the LiDAR 53, etc. into clusters for each group of point clouds. Thereby, the presence or absence, size, shape, and position of an object around the vehicle 1 can be detected.

[0050] In some embodiments, the recognition unit 73 may detect the movement of an object around the vehicle 1 by tracking the movement of a group of point clouds classified by clustering. Thereby, the speed and / or traveling direction (movement vector) of an object around the vehicle 1 can be detected.

[0051] In some embodiments, the recognition unit 73 may detect and / or recognize vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., based on image data supplied from the camera 51. In some embodiments, the recognition unit 73 may recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0052] In some embodiments, the recognition unit 73 may perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and / or the recognition result of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 may recognize the location and / or status of traffic signals, the content of traffic signs and / or road markings, the content of traffic regulations, and / or drivable lanes.

[0053] In some embodiments, the recognition unit 73 may perform recognition processing of the environment surrounding the vehicle 1. In some embodiments, the recognition unit 73 may recognize weather characteristics (temperature, humidity, brightness), and / or road surface conditions, etc.

[0054] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 may create an action plan by performing route planning and route following.

[0055] In some embodiments, the path planning may include global path planning and local path planning. Global path planning may include the process of planning a rough route from the start to the goal. Local path planning, also called trajectory planning, may include the generation of a trajectory that allows the vehicle 1 to travel safely and smoothly along the planned route in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 and the presence of any obstacles.

[0056] In some embodiments, route following may involve planning actions to safely and accurately travel along the route planned by the route planner within a planned time. The action planning unit 62 may, for example, calculate the target speed and / or target angular velocity of the vehicle 1 based on the results of this route following process.

[0057] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0058] For example, in some embodiments, the motion control unit 63 may control the steering control unit 81, brake control unit 82, and / or drive control unit 83, which are included in the vehicle control unit 32 described later, to perform lateral vehicle motion control and / or longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 may perform one or more driver assistance functions and / or control for the purpose of driving automation (e.g., lateral vehicle motion control, longitudinal vehicle motion control). Non-limited examples of driver assistance functions include collision avoidance or impact mitigation, inter-vehicle distance control (e.g., control to maintain a specific distance from a vehicle traveling in front of the vehicle 1), vehicle speed control (e.g., control to maintain a specific speed), vehicle collision warning, and lane departure warning. Non-limited examples of driving automation include driving without operation by the driver or remote driver.

[0059] In some embodiments, the DMS 30 may perform driver authentication processing and / or driver status recognition processing based on sensor data from the in-vehicle sensor 26 and / or input data input to the HMI 31, which will be described later. Non-limited examples of driver status that may be recognized include physical condition, alertness level, concentration level, fatigue level, gaze direction, intoxication level, driving operation, posture, etc.

[0060] In some embodiments, the DMS 30 may perform authentication processing for users other than the driver (e.g., passengers) and / or recognition processing for the status of such users. In some embodiments, the DMS 30 may perform recognition processing for the internal conditions of the vehicle 1 based on sensor data from the in-vehicle sensors 26. Non-limiting examples of characteristics of the internal conditions of the vehicle 1 that may be recognized include temperature, humidity, brightness, odor, etc.

[0061] HMI31 receives various data and instructions as input and presents various data to the user.

[0062] A brief overview of data input to the HMI 31 is provided. The HMI 31 is equipped with an input device for a person to input data, instructions, etc. Based on the data, instructions, etc. input by the input device, the HMI 31 generates an input signal and supplies it to one or more (or, in some cases, all) units of the vehicle control system 11. In some embodiments, the HMI 31 may be equipped with a touch panel, buttons, switches, and / or levers as input devices. Not limited to these, the HMI 31 may be equipped with an input device that allows information to be input by methods other than manual operation, such as voice or gestures. In some embodiments, the HMI 31 may be equipped with a remote control device using infrared and / or radio waves, or an external connection device that corresponds to the operation of the vehicle control system 11, as an input device. Non-limited examples of external connection devices include mobile devices (e.g., smartphones) and wearable devices (e.g., smartwatches).

[0063] A brief explanation of data presentation by HMI31 is provided below. HMI31 generates visual, auditory, and / or tactile information for the user and / or people outside of vehicle 1. HMI31 may also perform output control to control the output, output content, output timing, and / or output method of each generated piece of information. Non-limited examples of visual information that can be generated and output by HMI31 include information shown by images and light, such as operation screens, vehicle 1 status displays, warning displays, and monitor images showing the surroundings of vehicle 1. Non-limited examples of auditory information that can be generated and output by HMI31 include voice guidance, warning sounds, and warning messages. Non-limited examples of tactile information that can be generated and output by HMI31 include information given to the user's sense of touch through force, vibration, movement, etc.

[0064] In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image itself, or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be a device that displays visual information within the user's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality, in addition to or as an alternative to a normal display device. In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device provided in the vehicle 1, such as a navigation device, instrument panel, CMS (Camera Monitoring System), electronic mirror, lamp, etc.

[0065] In some embodiments, the HMI31 may include an audio speaker, headphones, or earphones as an output device capable of outputting auditory information.

[0066] In some embodiments, the HMI 31 may include a haptic element using haptic technology as an output device capable of outputting tactile information. The haptic element may be provided, for example, on a part of the vehicle 1 that the user comes into contact with, such as the steering wheel or the seat.

[0067] The vehicle control unit 32 controls one or more (or, in some cases, all) units of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0068] The steering control unit 81 detects and / or controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0069] The brake control unit 82 detects and / or controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0070] The drive control unit 83 detects and / or controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a drive ECU for controlling the drive system and an actuator for driving the drive system.

[0071] The body system control unit 84 detects and / or controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 also includes, for example, a body system ECU that controls the body system, actuators that drive the body system, etc.

[0072] The light control unit 85 detects and / or controls the state of various lights on the vehicle 1. Non-exclusive examples of lights that can be controlled by the light control unit 85 include headlights, taillights, fog lights, turn signals, brake lights, projector lights, bumper indicators, etc. The light control unit 85 includes a light ECU for controlling the lights, actuators for driving the lights, etc.

[0073] The horn control unit 86 detects and / or controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU for controlling the car horn, an actuator for driving the car horn, and the like.

[0074] Figure 2 shows an example of the sensing area of ​​the external recognition sensor 25 in Figure 1, including the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54. In Figure 2, a schematic view of the vehicle 1 from above is shown.

[0075] Sensing regions 101F and 101B show examples of sensing regions for ultrasonic sensors 54. Sensing region 101F (for example, the sensing region of multiple ultrasonic sensors 54) covers the area around the front end of vehicle 1. Sensing region 101B (for example, the sensing region of multiple ultrasonic sensors 54) covers the area around the rear end of vehicle 1.

[0076] The sensing results in sensing region 101F and / or sensing region 101B may be used, for example, for parking assistance of vehicle 1.

[0077] Sensing areas 102F, 102B, 102L, and 102R are examples of sensing areas for short-range or medium-range radar 52. Sensing area 102F covers a position further in front of vehicle 1 than sensing area 101F. Sensing area 102B covers a position further in rear of vehicle 1 than sensing area 101B. Sensing area 102L covers the area around the left rear of vehicle 1. Sensing area 102R covers the area around the right rear of vehicle 1.

[0078] The sensing results in sensing region 102F may be used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region 102B may be used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing region 102L and / or sensing region 102R may be used, for example, to detect one or more objects in the blind spots on the left and / or right sides of vehicle 1.

[0079] Sensing areas 103F, 103B, 103L, and 103R show examples of sensing areas by camera 51. Sensing area 103F covers a position further in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position further in rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the left side of vehicle 1. Sensing area 103R covers the right side of vehicle 1.

[0080] The sensing results in sensing region 103F may be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 103B may be used, for example, for parking assistance and / or surround view systems. The sensing results in sensing region 103L and / or sensing region 103R may be used, for example, for surround view systems.

[0081] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position further in front of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction of vehicle 1 than sensing area 103F.

[0082] The sensing results in the sensing region 104 may be used, for example, to detect objects such as surrounding vehicles.

[0083] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 105 covers a position further in front of vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction of vehicle 1 than sensing area 104.

[0084] The sensing results in the sensing area 105 may be used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0085] In some embodiments, the sensing areas of each sensor of the external recognition sensor 25 (e.g., camera 51, radar 52, LiDAR 53, ultrasonic sensor 54) may take various configurations other than those shown in Figure 2. Specifically, in some embodiments, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1. Furthermore, the installation positions of each sensor are not limited to the examples described above. Also, the number of sensors may be one or multiple.

[0086] This technology generates image data used, for example, in the training process of a learning model used for the autonomous driving of vehicle 1. Furthermore, this technology executes the training process of the learning model using the generated image data.

[0087] Furthermore, this technology generates image data used for training a learning model used for autonomous driving of other vehicles, based on image data obtained by the camera 51 of vehicle 1. This technology also executes the training process of the learning model using the generated image data.

[0088] <<2. Embodiments>> Next, embodiments of this technology will be described with reference to Figures 3 to 5.

[0089] <Example Configuration of Information Processing System 201> Figure 3 shows an example configuration of an information processing system 201 to which this technology is applied.

[0090] The information processing system 201 includes an imaging unit 211, an input unit 212, an information processing unit 213, and an output unit 214.

[0091] The shooting unit 211 includes a camera (hereinafter referred to as the shooting camera) and a sensor for detecting the shooting conditions of the shooting camera. For example, the shooting unit 211 may include multiple shooting cameras capable of shooting the same scene from multiple viewpoints. For example, the shooting unit 211 may shoot the same scene from multiple viewpoints by moving the shooting camera or a mobile body equipped with a shooting camera.

[0092] Sensors for detecting the shooting conditions of the camera include, for example, an IMU for detecting the movement of the camera and a temperature sensor for detecting the temperature around the camera. The sensors may be installed on the camera or around the camera.

[0093] The imaging unit 211 generates imaging data that includes image data obtained by the imaging camera (hereinafter referred to as "imaging image data"), setting parameters of the imaging camera, and imaging condition data related to the imaging conditions of the imaging image data. The imaging unit 211 supplies the imaging data to the curation unit 221 of the information processing unit 213.

[0094] For example, if the imaging unit 211 is provided on the vehicle 1, the imaging unit 211 is composed of a portion of the external recognition sensor 25 and vehicle sensor 27 of the vehicle 1 shown in Figure 1.

[0095] The input unit 212 is used, for example, for a user to operate the information processing unit 213 or to input data necessary for processing by the information processing unit 213. The input unit 212 supplies the input data to the information processing unit 213.

[0096] The input data includes, for example, user instructions, data for annotating image data, data for generating scenarios, and virtual setting parameters (hereinafter referred to as virtual setting parameters) and virtual shooting condition data (hereinafter referred to as virtual shooting condition data) for the learning camera described later.

[0097] The information processing unit 213 generates training data based on the captured data and input data, and uses the generated training data to perform machine learning and generate a learning model.

[0098] The information processing unit 213 includes a curation unit 221, an image data storage unit 222, an object model storage unit 223, an image processing unit 224, an annotation unit 225, a learning data generation unit 226, a learning data storage unit 227, and a learning unit 228.

[0099] The curation unit 221 stores the shooting data supplied from the shooting unit 211 in the shooting data storage unit 222. The curation unit 221 selects the shooting image data to be used for the learning process from among the shooting image data contained in each of the shooting data stored in the shooting data storage unit 222. The curation unit 221 supplies the selected shooting image data to the annotation unit 225 and the learning data generation unit 226. The curation unit 221 also supplies the shooting data, including the selected shooting image data, to the image processing unit 224.

[0100] The object model storage unit 223 stores object models, which are 3D (Dimensions) models of various objects.

[0101] The image processing unit 224 generates a 3D model, which is three-dimensional image data corresponding to a scene in the captured image data included in the captured data, based on the captured data and input data. Based on the 3D model, the object model stored in the object model storage unit 223, and the input data, the image processing unit 224 generates training image data (hereinafter referred to as training image data) that simulates the image data obtained by capturing with the target camera (hereinafter referred to as the training camera). The image processing unit 224 supplies the generated training image data to the annotation unit 225 and the training data generation unit 226.

[0102] The learning camera may be a different model of camera from the camera equipped in the imaging unit 211. Alternatively, the learning camera may be the same model as the camera equipped in the imaging unit 211. Furthermore, the learning camera may be a virtual camera that does not actually exist.

[0103] The annotation unit 225 adds annotations to the captured image data and training image data based on the input data. The annotation unit 225 supplies the annotation data related to the added annotations to the training data generation unit 226.

[0104] The learning data generation unit 226 stores the captured image data and annotation data dataset, and the learning image data and annotation data dataset, in the learning data storage unit 227. The learning data generation unit 226 generates learning data using the datasets stored in the learning data storage unit 227. The learning data generation unit 226 stores the generated learning data in the learning data storage unit 227.

[0105] The learning unit 228 performs machine learning using the learning data stored in the learning data storage unit 227 and generates a learning model.

[0106] The output unit 214 outputs various types of information. For example, the output unit 214 displays images based on captured image data, 3D models, or training image data.

[0107] <Example of the configuration of the image processing unit 224> Figure 4 shows an example of the configuration of the image processing unit 224 shown in Figure 3.

[0108] The image processing unit 224 includes a characteristic reduction image generation unit 301, a 3D model storage unit 302, a scenario generation unit 303, an object data generation unit 304, and a learning image generation unit 305.

[0109] The characteristic reduction image generation unit 301 performs processing on the captured image data included in the shooting data supplied from the curation unit 221 to reduce the influence of the characteristics of the shooting camera, and generates a 3D model corresponding to the scene in the captured image data based on the reduced-process captured image data. The characteristics of the shooting camera include, for example, at least one of optical characteristics, imaging characteristics, and image processing characteristics. From another viewpoint, the characteristics of the shooting camera include, for example, at least one of the functions, specifications, performance, and setting parameters of the shooting camera. For example, the optical characteristics, imaging characteristics, and image processing characteristics of the shooting camera are determined by one or a combination of two or more of the functions, specifications, performance, and setting parameters of the shooting camera. The characteristic reduction image generation unit 301 comprises a characteristic reduction unit 311 and a 3D model generation unit 312.

[0110] The characteristic reduction unit 311 uses an inverse model of the camera used for shooting (hereinafter referred to as the inverse shooting camera model) to generate image data from the captured image data in which the influence of the characteristics of the shooting camera has been reduced.

[0111] An inverse camera model is, for example, a model that infers image data from input image data before the influence of the characteristics of the camera is added. An inverse camera model comprises, for example, at least one of an inverse image processing circuit model, an inverse image sensor model, and an inverse optical system model.

[0112] An inverse image processing circuit model is a model that estimates image data before the influence of the characteristics of the image processing circuit, such as the ISP (Image Signal Processor), of the camera is added, based on the input image data. An inverse image sensor model is a model that estimates image data before the influence of the characteristics of the image sensor, of the camera is added, based on the input image data. An inverse optical system model is a model that estimates image data before the influence of the characteristics of the optical system, of the camera is added, based on the input image data.

[0113] The characteristic reduction unit 311 includes an image processing characteristic reduction unit 321, an imaging characteristic reduction unit 322, and an optical characteristic reduction unit 323.

[0114] The image processing characteristic reduction unit 321 generates image data from captured image data with reduced influence from the image processing characteristics of the camera used for capturing. For example, based on the setting parameters and shooting condition data included in the captured data, the image processing characteristic reduction unit 321 uses an inverse image processing circuit model to perform processing on the captured image data to reduce the influence from the image processing characteristics of the camera used for capturing. The image processing characteristic reduction unit 321 supplies the captured image data after the reduction processing to the imaging characteristic reduction unit 322.

[0115] The imaging characteristic reduction unit 322 generates image data from captured image data with reduced influence from the imaging characteristics of the camera. For example, based on the setting parameters and shooting condition data included in the shooting data, the imaging characteristic reduction unit 322 uses an inverse image sensor model to perform processing on the captured image data to reduce the influence from the imaging characteristics of the camera. The imaging characteristic reduction unit 322 supplies the processed captured image data to the optical characteristic reduction unit 323.

[0116] The optical characteristics reduction unit 323 generates image data from captured image data with the influence of the optical characteristics of the camera reduced. For example, based on the setting parameters and shooting condition data included in the shooting data, the optical characteristics reduction unit 323 uses an inverse optical system model to perform processing on the captured image data to reduce the influence of the optical characteristics of the camera. The optical characteristics reduction unit 323 supplies the processed captured image data to the 3D model generation unit 312.

[0117] The 3D model generation unit 312 generates a 3D model, which is a three-dimensional image data representing a scene, based on multiple image data of the same scene taken from different angles. The 3D model generation unit 312 stores the generated 3D model in the 3D model storage unit 302.

[0118] The scenario generation unit 303 displays images based on captured image data supplied from the curation unit 221 on the output unit 214 as needed. The scenario generation unit 303 generates scenarios to be used for learning processing based on data input by the user via the input unit 212. The scenario generation unit 303 supplies scenario data related to the generated scenarios to the object data generation unit 304 and the rendering unit 331.

[0119] The object data generation unit 304 generates object data, which is image data of objects appearing in the scenario, based on the object models stored in the object model storage unit 223. The object data generation unit 304 supplies the generated object data to the rendering unit 331.

[0120] The learning image generation unit 305 generates learning image data based on scenario data, object data, 3D models stored in the 3D model storage unit 302, and virtual setting parameters and virtual shooting condition data for the learning camera input by the user via the input unit 212. The learning image generation unit 305 includes a rendering unit 331 and a characteristics addition unit 332.

[0121] The rendering unit 331 generates training image data corresponding to scenes in a scenario based on the scenario data, object data, and 3D models stored in the 3D model storage unit 302. The rendering unit 331 supplies the generated training image data to the optical properties addition unit 341 of the properties addition unit 332.

[0122] The characteristic addition unit 332 uses a forward model of the learning camera (hereinafter referred to as the learning camera model) to generate image data in which the influence of the characteristics of the learning camera is added to the learning image data. The characteristics of the learning camera include, for example, at least one of optical characteristics, imaging characteristics, and image processing characteristics. From another viewpoint, the characteristics of the learning camera include, for example, at least one of the functions, specifications, performance, and setting parameters of the learning camera. For example, the optical characteristics, imaging characteristics, and image processing characteristics of the learning camera are determined by one or a combination of two or more of the functions, specifications, performance, and setting parameters of the learning camera.

[0123] A training camera model is, for example, a model that generates image data in which the characteristics of the training camera are added to the input image data. The training camera model comprises, for example, an optical system model, an image sensor model, and an image processing circuit model.

[0124] The optical system model is, for example, a model that generates image data in which the characteristics of the optical system of the learning camera are added to the input image data. The image sensor model is, for example, a model that adds the characteristics of the image sensor of the learning camera to the input image data. The image processing circuit model is, for example, a model that adds the characteristics of the image processing circuit, such as the ISP, of the learning camera to the input image data.

[0125] The characteristic addition unit 332 includes an optical characteristic addition unit 341, an imaging characteristic addition unit 342, and an image processing characteristic addition unit 343.

[0126] The optical characteristics addition unit 341 generates image data in which the influence of the optical characteristics of the learning camera is added to the learning image data. For example, based on the virtual setting parameters and virtual shooting condition data of the learning camera supplied from the input unit 212, the optical characteristics addition unit 341 uses an optical system model to perform a process that adds the influence of the optical characteristics of the learning camera to the learning image data. The optical characteristics addition unit 341 supplies the learning image data after the addition process to the imaging characteristics addition unit 342.

[0127] The imaging characteristics addition unit 342 generates image data in which the influence of the imaging characteristics of the training camera is added to the training image data. For example, based on the virtual setting parameters and virtual shooting condition data of the training camera supplied from the input unit 212, the imaging characteristics addition unit 342 uses an image sensor model to perform a process that adds the influence of the imaging characteristics of the training camera to the training image data. The imaging characteristics addition unit 342 supplies the training image data after the addition process to the image processing characteristics addition unit 343.

[0128] The image processing characteristics addition unit 343 generates image data in which the influence of the image processing characteristics of the training camera is added to the training image data. For example, based on the virtual setting parameters and virtual shooting condition data of the training camera supplied from the input unit 212, the image processing characteristics addition unit 343 uses an image processing circuit model to perform processing to add the influence of the image processing characteristics of the training camera to the training image data. The image processing characteristics addition unit 343 supplies the training image data after the addition processing to the training data generation unit 226.

[0129] <Learning Model Generation Process> Next, the learning model generation process performed by the information processing system 201 will be explained with reference to the flowchart in Figure 5.

[0130] In step S1, the information processing system 201 collects the captured data. Specifically, the shooting unit 211 takes pictures of the surroundings and supplies the captured data, including the captured image data obtained from the shooting, to the curation unit 221.

[0131] For example, if the camera unit 211 is installed on the vehicle 1, the camera unit 211 photographs the area around the vehicle 1 while the vehicle 1 is in motion and supplies the photographic data, including the obtained photographic image data, to the curation unit 221.

[0132] As mentioned above, the shooting data includes not only the captured image data but also the camera settings parameters and shooting condition data.

[0133] For example, setting parameters include the exposure time (shutter speed), aperture value (F-number), and ISO sensitivity of the camera used for shooting.

[0134] The shooting conditions data includes, for example, data on the environment around the camera during shooting (hereinafter referred to as "environmental data") and data on the movement of the camera during shooting (hereinafter referred to as "motion data"). The data on the environment around the camera during shooting includes, for example, data on at least one of the following: temperature, humidity, and brightness around the camera during shooting.

[0135] Environmental data includes, for example, the temperature around the camera used for shooting.

[0136] Motion data includes, for example, the velocity or acceleration of the camera in the forward / backward, left / right, and up / down directions, as well as angular velocity in the roll, pitch, and yaw directions.

[0137] The curation unit 221 stores the acquired shooting data in the shooting data storage unit 222.

[0138] As this process is repeated, the image data obtained by the imaging unit 211 is stored in the image data storage unit 222.

[0139] In step S2, the curation unit 221 selects the captured image data. For example, the curation unit 221 selects the captured image data contained in each of the captured data stored in the captured data storage unit 222 into captured image data to be used in the subsequent learning process and captured image data not to be used. For example, captured image data that is predicted to have a high learning effect in machine learning is preferentially selected as the captured image data to be used in the subsequent learning process.

[0140] The curation unit 221 may select captured image data based on user instructions input via the input unit 212, or it may select captured image data using a predetermined algorithm or a trained model obtained through machine learning.

[0141] The curation unit 221 supplies captured image data to be used in the subsequent learning process to the annotation unit 225, the learning data generation unit 226, and the scenario generation unit 303. The curation unit 221 also supplies captured data, including captured image data to be used in the subsequent learning process, to the image processing characteristic reduction unit 321.

[0142] In step S3, the annotation unit 225 adds annotations to the captured image data.

[0143] For example, the annotation unit 225 displays an image based on the captured image data supplied from the curation unit 221 on the output unit 214.

[0144] For example, the user, while viewing the image displayed on the output unit 214, inputs data for adding annotations to the captured image data to the annotation unit 225 via the input unit 212.

[0145] For example, the annotation unit 225 adds annotations to the captured image data based on data entered by the user. For instance, the position and type of each object in the captured image data are added as annotations.

[0146] The annotation unit 225 generates annotation data related to the assigned annotations and supplies it to the learning data generation unit 226.

[0147] The learning data generation unit 226 stores the captured image data and the corresponding annotation data dataset in the learning data storage unit 227.

[0148] In step S4, the characteristic reduction unit 311 performs a process on the captured image data to reduce the influence of the characteristics of the camera used for capturing.

[0149] Specifically, the image processing characteristic reduction unit 321 uses an inverse image processing circuit model to reduce the influence of the image processing characteristics of the camera on the captured image data, based on the setting parameters and shooting condition data included in the captured data.

[0150] For example, image processing characteristics include the characteristics of various image processing operations (e.g., correction, adjustment, conversion, etc.) performed by the image processing circuit in the camera. For example, the image processing performed by the image processing circuit includes one or more of the following: aberration correction, shading correction, gain adjustment, defect correction, noise reduction, demosaicing, white balance adjustment, color correction, gamma correction, sharpness correction, contrast correction, etc.

[0151] This allows, for example, the reverse processing performed by the image processing circuit of the camera—such as correction, adjustment, and conversion—to be applied to the captured image data.

[0152] The image processing characteristic reduction unit 321 supplies the image data, including the image data after reduction processing, to the imaging characteristic reduction unit 322.

[0153] The imaging characteristic reduction unit 322 uses an inverse image sensor model to reduce the influence of the imaging characteristics of the camera on the captured image data, based on the setting parameters and shooting condition data included in the captured data.

[0154] For example, imaging characteristics include data relating to the characteristics of the image sensor in the camera. Image sensor characteristics include, for example, exposure time, pixel arrangement (e.g., CFA (Color Filter Array)), shutter method (e.g., rolling shutter or global shutter), and characteristics relating to HDR (High Dynamic Range) composite processing.

[0155] This performs an inverse transformation of the image data transformation process performed by the image sensor of the camera. In addition, noise, distortion, blur, and pixel saturation (overexposure and underexposure) generated by the image sensor of the camera are reduced.

[0156] Specifically, for example, if the image sensor of the camera is performing HDR (High Dynamic Range) compositing, the inverse transformation of the HDR compositing process is performed. This separates the captured image data into, for example, long-exposure image data with a long exposure time before compositing, and short-exposure image data with a short exposure time.

[0157] For example, based on the pixel arrangement of the image sensor of a camera, captured image data is converted into RAW image data where each pixel has all the color information (e.g., RGB).

[0158] For example, motion blur and motion artifacts in captured image data are reduced based on the exposure time of the image sensor of the camera and the motion data of the camera.

[0159] For example, if the shutter method of the image sensor of a camera is a rolling shutter method, the rolling shutter distortion of the captured image data is reduced based on the characteristics of the rolling shutter method and the exposure time of the image sensor.

[0160] For example, based on the exposure time of the image sensor of the camera and the temperature during shooting, noise in the captured image data generated by the image sensor, as well as pixel saturation (overexposure and underexposure), are reduced.

[0161] The imaging characteristic reduction unit 322 supplies the imaging data, including the image data after reduction processing, to the optical characteristic reduction unit 323.

[0162] The optical characteristic reduction unit 323 uses an inverse optical system model to reduce the influence of the optical characteristics of the camera on the captured image data included in the captured data, based on the setting parameters and shooting condition data included in the captured data.

[0163] Optical characteristics include the characteristics of the optical system, such as the lens, of a camera used for taking photographs. For example, optical characteristics include the type of lens, focal length, aperture value, aberration characteristics, etc., of the camera.

[0164] This reduces noise and distortion in captured image data caused by the optical system. For example, it reduces noise and distortion in captured image data caused by lens aberrations.

[0165] The optical characteristics reduction unit 323 supplies the captured image data after the reduction process to the 3D model generation unit 312.

[0166] In step S5, the 3D model generation unit 312 generates a 3D model. For example, the 3D model generation unit 312 uses a technology such as NeRF (Neural Radiance Fields) to generate a 3D model corresponding to a scene based on multiple captured image data of the same scene taken from different viewpoints. The 3D model generation unit 312 stores the generated 3D model in the 3D model storage unit 302.

[0167] In step S6, the scenario generation unit 303 generates a scenario.

[0168] For example, the scenario generation unit 303 displays an image based on the captured image data supplied from the curation unit 221 on the output unit 214 as needed.

[0169] For example, the user, while viewing the image displayed on the output unit 214, inputs data for generating a scenario to be used in the learning process to the scenario generation unit 303 via the input unit 212.

[0170] The scenario generation unit 303 generates a scenario based on the input data. For example, when generating a training scenario for an autonomous driving learning model, the scenario includes descriptions of the vehicle's movement, objects around the vehicle, and the environment around the vehicle.

[0171] The scenario generation unit 303 generates scenario data related to the generated scenario and supplies it to the object data generation unit 304 and the rendering unit 331.

[0172] In step S7, the object data generation unit 304 generates object data. For example, based on the scenario generated by the scenario generation unit 303, the object data generation unit 304 obtains object models from the object model storage unit 223 that correspond to objects that appear in the scenario but do not exist in the captured image data.

[0173] The object data generation unit 304 generates object data corresponding to objects to be superimposed on the training image data generated by the rendering unit 331, based on the scenario, from the object model. For example, the object data generation unit 304 generates object data with adjusted orientation, size, shape, texture, shading, etc. The object data generation unit 304 supplies the generated object data to the rendering unit 331.

[0174] In step S8, the rendering unit 331 generates training image data. Specifically, the rendering unit 331 acquires a 3D model necessary for generating training image data based on a scenario from the 3D model storage unit 302. The rendering unit 331 generates training image data from the 3D model from a viewpoint based on the scenario. The rendering unit 331 also superimposes object data onto the training image data at the positions indicated by the scenario. The rendering unit 331 supplies the generated training image data to the annotation unit 225 and the optical properties addition unit 341.

[0175] In step S9, the characteristic addition unit 332 performs a process to add the influence of the characteristics of the learning camera to the learning image data.

[0176] For example, the user inputs virtual setting parameters and virtual shooting condition data for the learning camera to the characteristics addition unit 332 via the input unit 212.

[0177] The contents of the virtual setting parameters and virtual shooting condition data are substantially the same as the setting parameters and shooting condition data included in the shooting data supplied from the shooting unit 211 to the curation unit 221 in the processing of step S1.

[0178] For example, virtual setting parameters include the virtual exposure time (shutter speed), aperture value (F-number), ISO sensitivity, etc., of the learning camera.

[0179] The virtual shooting condition data includes, for example, data relating to the virtual environment around the learning camera (hereinafter referred to as virtual environment data) and data relating to the virtual movement of the learning camera (hereinafter referred to as virtual movement data).

[0180] The optical characteristics addition unit 341 uses an optical system model, based on virtual setting parameters and virtual shooting condition data, to perform a process that adds the influence of the optical characteristics of the training camera to the training image data.

[0181] As a result, in step S4, processing is performed on the training image data that is almost the reverse of the processing performed on the captured image data by the optical characteristic reduction unit 323. For example, noise and distortion that are presumed to be generated by the optical system of the training camera are added to the training image data.

[0182] The optical characteristics addition unit 341 supplies the training image data after the addition process to the imaging characteristics addition unit 342.

[0183] The imaging characteristics addition unit 342 uses an image sensor model, based on virtual setting parameters and virtual shooting condition data, to perform a process that adds the influence of the imaging characteristics of the training camera to the training image data.

[0184] As a result, in step S4, processing is performed on the training image data that is almost the reverse of the processing performed on the captured image data by the imaging characteristic reduction unit 322. For example, the conversion processing of the training image data performed by the image sensor of the training camera is performed. In addition, for example, noise, distortion, blur, and pixel saturation that are presumed to be generated by the image sensor of the training camera are added to the training image data.

[0185] Specifically, for example, when a training camera performs HDR synthesis processing, long-exposure image data and short-exposure image data are generated based on the training image data. Furthermore, the long-exposure image data and short-exposure image data are combined to generate HDR training image data.

[0186] For example, the color information of each pixel in the training image data is converted based on the pixel arrangement of the image sensor of the training camera.

[0187] For example, virtual motion blur and motion distortion (motion artifacts) are added to the training image data based on the virtual exposure time of the training camera's image sensor and virtual motion data.

[0188] For example, if the image sensor of the training camera uses a rolling shutter system, a virtual rolling shutter distortion is added to the training image data based on the characteristics of the rolling shutter system and the virtual exposure time of the image sensor.

[0189] For example, virtual noise and pixel saturation are added to the training image data based on a virtual exposure time for the image sensor and a virtual temperature during shooting.

[0190] The imaging characteristics addition unit 342 supplies the training image data after the addition processing to the image processing characteristics addition unit 343.

[0191] The image processing characteristics addition unit 343 uses an image processing circuit model, based on virtual setting parameters and virtual shooting condition data, to add the influence of the image processing characteristics of the training camera to the training image data.

[0192] As a result, in step S4, processing that is almost the reverse of the processing performed on the captured image data by the image processing characteristic reduction unit 321 is performed on the training image data. For example, correction processing, adjustment processing, conversion processing, etc., performed by the image processing circuit of the training camera are performed on the training image data.

[0193] The image processing characteristics addition unit 343 supplies the training image data after the addition processing to the training data generation unit 226.

[0194] In step S10, the annotation unit 225 adds annotations to the training image data by the same process as in step S3.

[0195] For example, the annotation unit 225 may add annotations to the training image data based on a scenario corresponding to the training image data.

[0196] The annotation unit 225 generates annotation data related to the assigned annotations and supplies it to the learning data generation unit 226.

[0197] The training data generation unit 226 stores the training image data and the corresponding annotation data dataset in the training data storage unit 227.

[0198] For example, by repeatedly executing the processes in steps S2 to S10, a dataset used to generate training data is stored in the training data storage unit 227.

[0199] In step S11, the learning data generation unit 226 generates learning data. Specifically, the learning data generation unit 226 generates labels, which are the correct data for the image data (captured image data or learning image data) stored in the learning data storage unit 227, based on the annotation data corresponding to the image data. The content of the labels differs depending on the type of learning model to be generated. The learning data generation unit 226 stores the learning data, including the image data and labels, in the learning data storage unit 227.

[0200] The learning data generation unit 226 generates learning data containing each image data by performing the above processing on each image data stored in the learning data storage unit 227.

[0201] Whether both captured image data and training image data are used as training data, or whether only one of them is used, depends on factors such as the type of training model being generated and the device (e.g., vehicle) on which the training model is used. For example, training data is generated using image data corresponding to a camera of the same type as the camera installed in the device to which the training model is applied.

[0202] In step S12, the learning unit 228 executes the learning process. Specifically, the learning unit 228 generates a target learning model (trained model) by performing machine learning using the learning data stored in the learning data storage unit 227.

[0203] Furthermore, the machine learning method is not particularly limited. For example, the machine learning method may be selected based on the type of learning model being targeted.

[0204] After that, the learning model generation process ends.

[0205] As described above, a large amount of image data to be used as training data can be efficiently acquired. For example, a large amount of image data corresponding to various scenes in autonomous driving can be efficiently acquired.

[0206] Furthermore, training data is generated by first reducing the influence of the characteristics of the camera used for capturing the image data, and then adding the influence of the characteristics of the training camera. This makes it possible to generate training image data that is close to the image data obtained using the training camera in reality. In other words, the difference between the image data obtained using the training camera in reality and the generated training image data can be reduced.

[0207] As a result, the accuracy of machine learning improves, and the accuracy of the generated learning models also improves. For example, the performance of learning models for autonomous driving, such as object recognition models and models for autonomous driving control, improves. In particular, the performance of learning models for autonomous driving of vehicles using learning cameras improves.

[0208] Furthermore, by changing the training camera model, for example, it is possible to easily generate training image data that closely resembles the image data obtained by actually taking pictures with various cameras.

[0209] Note that the camera used for shooting and the camera used for training may be the same model. In this case, the amount of image data used for training can be increased by generating new image data of a new scene, or image data in which at least one of the following is different: setting parameters, shooting conditions, image processing, etc., based on the image data obtained from the shooting camera.

[0210] <<3. Modified Examples>> Below, modified examples of the embodiments of the present technology described above will be explained.

[0211] <Modifications of the inverse camera model and the camera model for learning> For example, an inverse camera model may be used for each camera model used for shooting, or a common inverse camera model may be used across multiple models by changing the parameters for each model.

[0212] Similarly, for example, a separate learning camera model may be used for each model of learning camera, or a common learning camera model may be used across multiple models by changing the parameters for each model.

[0213] For example, if the division of processing between the image sensor and the image processing circuit differs depending on the model of the camera used for shooting, the division of processing between the inverse image sensor model and the inverse image processing circuit model of the inverse shooting camera model will also be changed.

[0214] Similarly, if, for example, the division of processing between the image sensor and the image processing circuit differs depending on the model of the training camera, the division of processing between the image sensor model and the image processing circuit model of the training camera model will also be changed.

[0215] For example, if part of the image processing on the image data is performed outside the camera, the processing corresponding to the externally performed image processing may or may not be included in the inverse image processing model.

[0216] Similarly, if, for example, part of the image processing on image data is performed outside the training camera, the processing corresponding to the externally performed image processing may or may not be included in the image processing model.

[0217] The processing of the inverse image processing circuit model, inverse image sensor model, and inverse optical system model described above is an example and can be modified as appropriate. Furthermore, for example, an inverse camera model may comprise only one or two of the inverse image processing circuit model, inverse image sensor model, and inverse optical system model.

[0218] The processing of the image processing circuit model, image sensor model, and optical system model described above is just an example and can be modified as appropriate. Furthermore, for example, the learning camera model may comprise only one or two of the image processing circuit model, image sensor model, and optical system model.

[0219] For example, the inverse camera model may be held in advance by the characteristic reduction unit 311, or it may be provided to the characteristic reduction unit 311 from an external source during processing.

[0220] For example, the camera model for learning may be held in advance by the characteristics addition unit 332, or it may be provided to the characteristics addition unit 332 from an external source during processing.

[0221] For example, the characteristic reduction unit 311 may generate image data in which the influence of at least one of the functions, specifications, performance, and setting parameters of the camera used for capturing images has been reduced. Conversely, for example, the characteristic addition unit 332 may generate image data in which the influence of at least one of the functions, specifications, performance, and setting parameters of the camera used for learning has been added.

[0222] <Variations regarding the division of processing, etc.> The processing of the information processing unit 213 may be performed by one device or by being divided and performed by multiple devices. Similarly, the processing of the image processing unit 224 may be performed by one device or by being divided and performed by multiple devices.

[0223] For example, the process of generating training data and the training process may be performed on different devices. For example, the process of generating a 3D model, the process of generating training data using the 3D model, and the training process may be performed on different devices.

[0224] For example, the annotation unit 225 may be configured to add annotations to the 3D model. In this case, the annotation unit 225 may also be configured to add annotations to the training image data based on the annotations added to the 3D model.

[0225] For example, the learning data generation unit 226 may generate learning data using a 3D model.

[0226] <Examples of application of this technology> This technology can be applied to all cases where image data for training data is generated, regardless of the type of learning model to be generated, the type of machine learning (e.g., supervised learning, unsupervised learning, reinforcement learning, etc.), and the intended use of the training data (e.g., training data, validation data, test data, etc.).

[0227] Furthermore, this technology can be applied to generating image data for purposes other than art book data. For example, it can be applied to generating image data for various simulations.

[0228] <<4. Others>> <Example of Computer Configuration> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on the computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers that can perform various functions by installing various programs.

[0229] Figure 6 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0230] In computer 1000, the CPU (Central Processing Unit) 1001, ROM (Read-Only Memory) 1002, and RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0231] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0232] The input unit 1006 consists of input switches, buttons, a microphone, an image sensor, etc. The output unit 1007 consists of a display, a speaker, etc. The storage unit 1008 consists of a hard disk or non-volatile memory, etc. The communication unit 1009 consists of a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0233] In the computer 1000 configured as described above, the CPU 1001 loads, for example, a program stored in the memory unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

[0234] The program executed by the computer 1000 (CPU 1001) can be provided by recording it on a removable medium 1011, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0235] In computer 1000, programs can be installed in the storage unit 1008 via the input / output interface 1005 by inserting the removable media 1011 into the drive 1010. Alternatively, programs can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, programs can be pre-installed in the ROM 1002 or the storage unit 1008.

[0236] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0237] Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

[0238] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0239] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0240] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0241] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0242] <Examples of configuration combinations> This technology can also be configured as follows:

[0243] (1) An information processing system comprising: a first image generation unit that generates second image data from first image data obtained by a first camera, with reduced influence due to the characteristics of the first camera; and a second image generation unit that generates third image data by adding the influence due to the characteristics of the second camera to the second image data. (2) The information processing system according to (1), wherein the characteristics of the first camera include at least one of the optical characteristics, imaging characteristics, and image processing characteristics of the first camera; the characteristics of the second camera include at least one of the optical characteristics, imaging characteristics, and image processing characteristics of the second camera; the optical characteristics of the first camera include the characteristics of the first optical system provided by the first camera; the imaging characteristics of the first camera include the characteristics of the first image sensor provided by the first camera; the image processing characteristics of the first camera include the characteristics of the first image processing circuit provided by the first camera; the optical characteristics of the second camera include the characteristics of the second optical system provided by the second camera; the imaging characteristics of the second camera include the characteristics of the second image sensor provided by the second camera; and the image processing characteristics of the second camera include the characteristics of the second image processing circuit provided by the second camera. (3) The information processing system according to (2), wherein the characteristics of the first image sensor include characteristics relating to at least one of the exposure time, shutter method, pixel arrangement, and HDR (High Dynamic Range) synthesis processing of the first image sensor, and the characteristics of the second image sensor include characteristics relating to at least one of the exposure time, shutter method, pixel arrangement, and HDR synthesis processing of the second image sensor. (4) The information processing system according to (2) or (3), wherein the characteristics of the first optical system include aberration characteristics of the first lens provided in the first optical system, and the characteristics of the second optical system include aberration characteristics of the second lens provided in the second optical system. (5) The information processing system according to any one of (2) to (4), wherein the characteristics of the first image processing circuit include characteristics relating to at least one of the correction processing, adjustment processing, and conversion processing of the first image processing circuit, and the characteristics of the second image processing circuit include characteristics relating to at least one of the correction processing, adjustment processing, and conversion processing of the second image processing circuit.(6) The information processing system according to any one of (1) to (5), wherein the first image generation unit further generates the second image data from the first image data based on the shooting conditions of the first image data. (7) The information processing system according to (6), wherein the shooting conditions include at least one of the environment surrounding the first camera and the movement of the first camera. (8) The information processing system according to any one of (1) to (7), wherein the first image generation unit generates the second image data from the first image data using an inverse camera model that estimates the image data before the influence of the characteristics of the first camera is added from the input image data. (9) The information processing system according to (8), wherein the inverse camera model comprises at least one of the following: an inverse optical system model that estimates image data before the influence of the characteristics of the optical system of the first camera is added to the input image data; an inverse image sensor model that estimates image data before the influence of the characteristics of the image sensor of the first camera is added to the input image data; and an inverse image processing circuit model that estimates image data before the influence of the characteristics of the image processing circuit of the first camera is added to the input image data. (10) The information processing system according to any one of (1) to (9), wherein the second image generation unit generates the third image data from the second image data using a camera model that generates image data with the influence of the characteristics of the second camera added to the input image data. (11) The information processing system according to (10), wherein the camera model comprises at least one of the following: an optical system model that adds the influence of the characteristics of the optical system of the second camera to the input image data; an image sensor model that adds the influence of the characteristics of the image sensor of the second camera to the input image data; and an image processing circuit model that adds the influence of the characteristics of the image processing circuit of the second camera to the input image data. (12) The information processing system according to any one of (1) to (11), wherein the first image generation unit reduces at least one of the noise, distortion, blur, and pixel saturation of the first image data.(13) The information processing system according to any one of (1) to (12), wherein the characteristics of the first camera include at least one of the functions, specifications, performance, and setting parameters of the first camera, and the characteristics of the second camera include at least one of the functions, specifications, performance, and setting parameters of the second camera. (14) The information processing system according to any one of (1) to (13), further comprising a learning data generation unit that generates learning data including the third image data. (15) The information processing system according to (14), further comprising a learning unit that performs machine learning using the learning data. (16) The information processing system according to any one of (1) to (15), wherein the second image data is a three-dimensional model corresponding to a scene in the first image data. (17) The information processing system according to any one of (1) to (16), wherein the first camera and the second camera are of different models. (18) An information processing method comprising an information processing system that generates a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced; and generates a third image data by adding the influence of the characteristics of the second camera to the second image data. (19) A program for causing a computer to perform a process that includes generating a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced; and generating a third image data by adding the influence of the characteristics of the second camera to the second image data.

[0244] Furthermore, the effects described herein are merely illustrative and not limiting; other effects may also occur.

[0245] 1 Vehicle, 11 Vehicle control system, 201 Information processing system, 211 Imaging unit, 212 Input unit, 213 Information processing unit, 214 Output unit, 221 Curation unit, 224 Image processing unit, 225 Annotation unit, 226 Learning data generation unit, 228 Learning unit, 301 Characteristic reduction image generation unit, 303 Scenario generation unit, 304 Object data generation unit, 311 Characteristic reduction unit, 312 3D model generation unit, 321 Image processing characteristic reduction unit, 322 Imaging characteristic reduction unit, 323 Optical characteristic reduction unit, 331 Rendering unit, 332 Characteristic addition unit, 341 Optical characteristic addition unit, 342 Imaging characteristic addition unit, 343 Image processing characteristic addition unit

Claims

1. An information processing system comprising: a first image generation unit that generates second image data from first image data obtained by a first camera, with reduced influence due to the characteristics of the first camera; and a second image generation unit that generates third image data by adding the influence due to the characteristics of the second camera to the second image data.

2. The information processing system according to claim 1, wherein the characteristics of the first camera include at least one of the optical characteristics, imaging characteristics, and image processing characteristics of the first camera; the characteristics of the second camera include at least one of the optical characteristics, imaging characteristics, and image processing characteristics of the second camera; the optical characteristics of the first camera include the characteristics of the first optical system provided by the first camera; the imaging characteristics of the first camera include the characteristics of the first image sensor provided by the first camera; the image processing characteristics of the first camera include the characteristics of the first image processing circuit provided by the first camera; the optical characteristics of the second camera include the characteristics of the second optical system provided by the second camera; the imaging characteristics of the second camera include the characteristics of the second image sensor provided by the second camera; and the image processing characteristics of the second camera include the characteristics of the second image processing circuit provided by the second camera.

3. The information processing system according to claim 2, wherein the characteristics of the first image sensor include characteristics relating to at least one of the exposure time, shutter method, pixel arrangement, and HDR (High Dynamic Range) synthesis processing of the first image sensor, and the characteristics of the second image sensor include characteristics relating to at least one of the exposure time, shutter method, pixel arrangement, and HDR synthesis processing of the second image sensor.

4. The information processing system according to claim 2, wherein the characteristics of the first optical system include the aberration characteristics of the first lens provided in the first optical system, and the characteristics of the second optical system include the aberration characteristics of the second lens provided in the second optical system.

5. The information processing system according to claim 2, wherein the characteristics of the first image processing circuit include characteristics relating to at least one of the correction processing, adjustment processing, and conversion processing of the first image processing circuit, and the characteristics of the second image processing circuit include characteristics relating to at least one of the correction processing, adjustment processing, and conversion processing of the second image processing circuit.

6. The information processing system according to claim 1, wherein the first image generation unit further generates a second image data from the first image data based on the shooting conditions of the first image data.

7. The information processing system according to claim 6, wherein the shooting conditions include at least one of the environment surrounding the first camera and the movement of the first camera.

8. The information processing system according to claim 1, wherein the first image generation unit generates the second image data from the first image data using an inverse camera model that estimates the image data before the influence of the characteristics of the first camera is added from the input image data.

9. The information processing system according to claim 8, wherein the inverse camera model comprises at least one of the following: an inverse optical system model that estimates image data before the influence of the characteristics of the optical system of the first camera is added to the input image data; an inverse image sensor model that estimates image data before the influence of the characteristics of the image sensor of the first camera is added to the input image data; and an inverse image processing circuit model that estimates image data before the influence of the characteristics of the image processing circuit of the first camera is added to the input image data.

10. The information processing system according to claim 1, wherein the second image generation unit generates the third image data from the second image data using a camera model that generates image data in which the characteristics of the second camera are added to the input image data.

11. The information processing system according to claim 10, wherein the camera model comprises at least one of the following: an optical system model that adds the influence of the characteristics of the optical system of the second camera to the input image data; an image sensor model that adds the influence of the characteristics of the image sensor of the second camera to the input image data; and an image processing circuit model that adds the influence of the characteristics of the image processing circuit of the second camera to the input image data.

12. The information processing system according to claim 1, wherein the first image generation unit reduces at least one of noise, distortion, blur, and pixel saturation of the first image data.

13. The information processing system according to claim 1, wherein the characteristics of the first camera include at least one of the functions, specifications, performance, and setting parameters of the first camera, and the characteristics of the second camera include at least one of the functions, specifications, performance, and setting parameters of the second camera.

14. The information processing system according to claim 1, further comprising a learning data generation unit that generates learning data including the third image data.

15. The information processing system according to claim 14, further comprising a learning unit that performs machine learning using the aforementioned training data.

16. The information processing system according to claim 1, wherein the second image data is a three-dimensional model corresponding to a scene in the first image data.

17. The information processing system according to claim 1, wherein the first camera and the second camera are of different models.

18. An information processing method comprising: generating a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced; and generating a third image data by adding the influence of the characteristics of the second camera to the second image data.

19. A program for causing a computer to perform a process that includes generating a second image data from a first image data obtained by a first camera, with the influence of the characteristics of the first camera reduced; and generating a third image data by adding the influence of the characteristics of the second camera to the second image data.

Images