Electronic apparatus and robot system

Abstract

This electronic apparatus comprises a control unit that controls the operation of a robot device on the basis of the results of image recognition processing performed by an AI processing unit provided inside a camera and performed on captured images obtained by capturing images of the robot device exterior by using the camera.

Description

Electronic device, robot system

[0007]

[0001] This technology relates to an electronic device and a robot system, and particularly relates to a technology for controlling the operation of a robot device based on the result of image recognition processing using an AI model for a captured image.

[0002] For example, as disclosed in Patent Document 1 and Patent Document 2 below, there is a technology for performing image recognition processing using an AI (Artificial Intelligence) model on a captured image of the outside of a robot device, and controlling the operation of the robot device based on the result of the image recognition processing.

[0003] Japanese Patent Application Laid-Open No. 2019-528535, Japanese Patent No. 6549545

[0004] However, in any of the inventions described in Patent Documents 1 and 2 above, the control unit (control device) that controls the operation of the robot device inputs a captured image from a camera and performs image recognition processing on the input captured image. With such a configuration, it is necessary to transfer image data from the camera to the control unit, resulting in a time lag until the result of the image recognition processing is obtained, and as a result, a delay occurs in the operation control of the robot device.

[0005] This technology has been made in view of the above circumstances, and an object thereof is to suppress a delay in the operation control of a robot device in a system that controls the operation of the robot device based on the result of image recognition processing using an AI model.

[0006] The electronic device according to this technology includes a control unit that controls the operation of the robot device based on the result of image recognition processing performed by an AI processing unit provided in the camera for a captured image obtained by imaging the outside of the robot device with the camera. Thereby, when obtaining the image recognition result by the AI model, it is not necessary to transmit the captured image outside the camera.

[0007] This diagram shows the general configuration of a robot system as an embodiment, equipped with electronic equipment as an embodiment of this technology. This diagram explains an example of camera placement in a robot device. This is a block diagram showing an example of the configuration of a robot device. This is a block diagram showing an example of the internal configuration of an image sensor. This diagram explains an example of a safety judgment method. This diagram explains an example of a method for identifying a person's position based on captured images. This diagram explains measures to prevent delays in safety judgment timing. This is a flowchart showing an example of a processing procedure for realizing an action control method as an embodiment. This is an explanatory diagram of a first modification relating to the location of the AI ​​processing unit and control unit. This is an explanatory diagram of a second modification relating to the location of the AI ​​processing unit and control unit. This is an explanatory diagram of a third modification relating to the location of the AI ​​processing unit and control unit. This is an explanatory diagram of a fourth modification relating to the location of the AI ​​processing unit and control unit.

[0008] The embodiments will be described below in the following order: <1. Overview of the robot system as an embodiment> <2. Configuration of the electronic equipment as an embodiment> <3. Motion control method as an embodiment> <4. Processing procedure> <5. Modified examples> <6. Summary of embodiments> <7. This technology>

[0009] <1. Overview of the Robot System as an Embodiment> Figure 1 is a diagram showing the overview configuration of a robot system as an embodiment, which is configured with electronic equipment as an embodiment of the present technology. The robot system of the embodiment is configured with at least a robot device 1. The robot device 1 corresponds to an example of electronic equipment as per the present technology.

[0010] In this example, the robot system is configured to support product manufacturing operations using the robot device 1 in a manufacturing plant that produces various products. For example, the robot system in this example is configured to support semiconductor manufacturing operations using the robot device 1 in a semiconductor equipment manufacturing plant.

[0011] In the robot system of this example, the robot device 1 has a partially movable part, which is a movable part that can displace a part of the device. Specifically, the robot device 1 in this example is configured to have an arm portion 1a, and the joint portion J for moving this arm portion 1a is formed as the above-mentioned partially movable part.

[0012] In this example, the arm portion 1a has multiple joints J from the base to the tip. When the joint J at the tip is driven, it is possible to grasp an object, and when the joints J closer to the base are driven, the bending angle of the arm can be adjusted, making it possible to handle the grasped object, such as moving it to a different location. The arm portion 1a can also be configured to have a turntable at the base to enable the arm to swing. In this case, the turntable is an example of the partially movable part described above.

[0013] In the robot system of this example, the robot device 1 is assumed to be used as an AGV (Automated Guided Vehicle). For this purpose, the robot device 1 is provided with an AGV unit 1b. Although not shown in the diagram, the AGV unit 1b is configured to include wheels, a driving motor that drives the wheels, a steering mechanism for steering the wheels and a steering actuator that drives the steering mechanism, and a driving control unit that controls the driving motor and steering actuator to control the movement of the robot device 1.

[0014] The robot system in this example includes a dispatch control device 100 for remotely controlling the dispatch of the robot device 1, which functions as an AGV. The dispatch control device 100 is configured with a computer and instructs the aforementioned travel control unit in the AGV unit 1b to move the robot device 1 to a predetermined position within the factory.

[0015] In this example, it is assumed that multiple robotic devices 1 will each perform a task within the factory, and the robot system is comprised of multiple robotic devices 1. The dispatch control device 100 in this example can individually instruct each robotic device 1 to move to a different location within the factory and perform tasks individually.

[0016] The robot device 1 is equipped with a camera 2 that captures images of the area outside the robot device 1 (not shown in Figure 1). The camera 2 is provided for monitoring the external conditions of the robot device 1, and in this example, as illustrated in Figure 2, multiple cameras 2 are provided on the robot device 1. In Figure 2, the direction in front of the robot device 1 is indicated by the arrow G, and in this example, the robot device 1 is equipped with four cameras 2, each capturing images of the front, right side, left side, and rear. Each camera 2 is configured to perform wide-angle imaging using, for example, a fisheye lens, and as shown in the figure, the imaging fields of adjacent cameras 2 overlap. This is done to prevent blind spots.

[0017] <2. Configuration of Electronic Devices as Embodiments> Figure 3 is a block diagram showing an example of the configuration of the robot device 1. Figure 3 shows an example of the electrical configuration related to the operation of the arm portion 1a, other than the AGV portion 1b described in Figure 1, for the example of the configuration of the robot device 1.

[0018] As shown in the figure, the robot device 1 includes a camera 2, an arm control unit 3, and an arm actuator group 4. Although only one camera 2 is shown in Figure 3, the robot device 1 in this example is actually equipped with four cameras 2, as illustrated in Figure 2.

[0019] The arm actuator group 4 comprehensively represents the actuators that drive the multiple movable parts (partially movable parts) of the arm 1a, which are provided as the aforementioned joint J, etc. Examples of these actuators include motors and solenoids.

[0020] The arm control unit 3 controls the movement of the arm 1a by controlling the drive of each actuator provided as the arm actuator group 4. For example, the arm control unit 3 may include a computer device, and the computer device may control the arm actuator group 4 based on the image captured by the camera 2 so that the arm 1a handles a predetermined target object in a predetermined manner. For example, the arm actuator group 4 may be controlled to obtain the movement of the arm 1a to move a target object placed at a predetermined position outside the robot device 1 onto a base provided on the robot device 1, or conversely, the arm actuator group 4 may be controlled to obtain the movement of the arm 1a to move a target object placed on the base to a predetermined position outside the robot device 1.

[0021] Camera 2 comprises an imaging optical system OP, an image sensor 21, a memory unit 22, a camera control unit 23, and a communication unit 24. The image sensor 21, memory unit 22, camera control unit 23, and communication unit 24 are connected via a bus BS, enabling them to communicate data with each other.

[0022] The image sensor 21 is configured as a solid-state image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) type. The imaging optical system OP includes lenses such as a cover lens and a focus lens, as well as an aperture (iris) mechanism. This imaging optical system OP guides light (incident light) from the subject and focuses it onto the light-receiving surface (imaging surface) of the image sensor 21.

[0023] The camera control unit 23 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). The CPU performs various processes according to the programs stored in the ROM and the programs loaded into the RAM, thereby performing overall control of the camera 2. For example, the camera control unit 23 realizes the AF (Auto Focus) function by controlling a focus actuator (not shown) that drives the focus lens based on the light received signal obtained from the image sensor 21. The camera control unit 23 also realizes the AE (Auto Exposure) function by controlling the gain of the AGC (Automatic Gain Control) circuit in the image sensor 21, the shutter speed, and the aperture value based on the light received signal obtained from the image sensor 21.

[0024] Furthermore, the camera control unit 23 controls the writing and reading of various data to and from the memory unit 22. The memory unit 22 is a non-volatile storage device such as a flash memory device or an HDD (Hard Disc Drive), and is used to store data used by the camera control unit 23 when performing various processes. The memory unit 22 can also be used to store image data output from the image sensor 21.

[0025] Furthermore, the camera control unit 23 is also capable of performing various data communications with devices outside the camera 2 via the communication unit 24. In this example, the communication unit 24 is configured to perform data communications with the arm control unit 3 located outside the camera 2, and the camera control unit 23 is capable of transmitting various data to the arm control unit 3 and receiving data transmitted by the arm control unit 3 via the communication unit 24. It should be noted that the communication unit 24 may also be configured to perform network communications, such as communications via the Internet or a LAN (Local Area Network).

[0026] In this example, the camera 2 includes an AI (Artificial Intelligence) processing unit 44, which is configured to perform image recognition processing on captured images. Furthermore, the image sensor 21 in this example has a safety determination / control unit F1 that performs control to prevent the arm 1a from coming into contact with a person, based on the results of this image recognition processing.

[0027] Figure 4 is a block diagram showing an example of the internal configuration of the image sensor 21. As shown in the figure, the image sensor 21 comprises an imaging unit 41, an image signal processing unit 42, an internal sensor control unit 43, an AI processing unit 44, a memory unit 45, and a communication interface (I / F) 46, each connected via a bus 47 to enable data communication between them.

[0028] The imaging unit 41 includes a pixel array unit formed by arranging pixels, each having a photoelectric conversion element (light-receiving element) such as a photodiode, in a two-dimensional array, and a readout circuit that reads out electrical signals (received signals) obtained by photoelectric conversion from each pixel in the pixel array unit. This readout circuit performs processes such as CDS (Correlated Double Sampling) processing and AGC (Automatic Gain Control) processing on the electrical signals obtained by photoelectric conversion, and further performs A / D (Analog to Digital) conversion processing.

[0029] In the imaging unit 41 of this example, a color filter that selectively transmits light of one of the colors R, G, or B is formed for each pixel so that an RGB color image can be obtained as the captured image. In this example, the arrangement of the color filters in the imaging unit 41 is, for example, an RGB Bayer arrangement. Note that the Bayer arrangement is just one example, and other arrangements of color filters (mosaic arrangements) such as RYYB or RGBW can be adopted.

[0030] The imaging unit 41 outputs a RAW image as the captured image. Here, a RAW image refers to a digital captured image immediately after A / D conversion of the signal read from the pixel array unit. The captured image as a RAW image output from the imaging unit 41 is input to the image signal processing unit 42.

[0031] The image signal processing unit 42 performs preprocessing, simulcasting, YC generation, codec processing, etc., on the captured image as a RAW image. Preprocessing includes clamping to clamp the black level to a predetermined level, and correction processing between the R, G, and B color channels. Preprocessing also includes brightness adjustment processing such as gamma correction, and color adjustment processing such as white balance adjustment and linear matrix processing. Linear matrix processing is a color reproduction error correction process that corrects the color to suit the desired color space by performing predetermined matrix operations on RGB.

[0032] Furthermore, in the simultaneous processing, a color separation process is applied so that the image data for each pixel contains all color components: R, G, and B. For example, in the case of an image sensor using a Bayer array color filter as in this example, demosaicing is performed as the color separation process. In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the R, G, and B image data.

[0033] In codec processing, the image data that has undergone the various processing steps described above is subjected to encoding for purposes such as recording and communication, and file generation. Codec processing makes it possible to generate video files in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264. It also makes it possible to generate still image files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format).

[0034] The sensor control unit 43 is configured with a microcomputer, for example, which includes a CPU, ROM, RAM, etc., and comprehensively controls the operation of the image sensor 21. For example, the sensor control unit 43 issues instructions to the imaging unit 41 to control the execution of imaging operations. It also controls the execution of processing to the image signal processing unit 42.

[0035] The AI ​​processing unit 44 is configured with programmable arithmetic processing units such as a DSP (Digital Signal Processor) or FPGA (Field Programmable Gate Array), and performs inference processing using an AI model on captured images, that is, image recognition processing such as object detection processing and object recognition processing. Image recognition processing here broadly refers to processing that recognizes the content of an image. Examples of image recognition processing include object detection processing that detects the area in which an object exists, object recognition processing that recognizes what kind of object is depicted in the image, semantic segmentation processing, and anomaly detection processing such as PatchCore. Here, object detection processing may include not only detecting the area in which an object exists, but also processing that recognizes what kind of object it is, such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0036] In this example, the AI ​​model used by the AI ​​processing unit 44 is assumed to be an AI model that performs object detection processing, including the detection of the region where an object exists and the processing of recognizing what kind of object it is, similar to YOLO and SSD mentioned above. Specifically, the AI ​​model in this example is assumed to be an AI model trained to recognize "people".

[0037] The memory unit 45 is used to store data necessary for the AI ​​processing unit 44 to perform image recognition processing (object detection processing in this example). Specifically, the memory unit 45 stores data of the AI ​​model used by the AI ​​processing unit 54 for image recognition processing. This AI model data includes parameters that indicate the structure of the neural network, and data of parameters used as filter coefficients in convolution processing, etc.

[0038] The communication interface (I / F) 46 is an interface that communicates with various parts connected via the bus BS, such as the camera control unit 23 and the memory unit 22, located outside the image sensor 21. For example, the communication interface 46 can output various data to the outside of the image sensor 21, such as the result information of object detection processing by the AI ​​processing unit 44 and captured images, based on the control of the sensor's internal control unit 43. The result information of object detection processing refers to information indicating the area where an object exists, such as a bounding box, and information indicating the recognition result of the object (in this example, "person"). The captured image referred to here may be data converted to a predetermined file format by codec processing in the image signal processing unit 42, or a RAW image.

[0039] <3. Operation control method as an embodiment> In this embodiment, the robot device 1 has a control function as a safety judgment / control unit F1 in the image sensor 21 in order to implement control to prevent the arm portion 1a from coming into contact with a person.

[0040] The safety determination and control unit F1 controls the operation of the robot device 1 based on the results of image recognition processing performed by the AI ​​processing unit 44 located inside the camera 2, using the captured images obtained by the camera 2 capturing images of the outside of the robot device 1.

[0041] Specifically, when a person is recognized in the captured image by the object detection process of the AI processing unit 44, the safety determination / control unit F1 in this example performs a process of specifying the positional relationship between the person and the robot device 1 based on the captured image, and based on the result of determining whether the person has approached the robot device 1 within a predetermined distance or less, performs control to limit the movable range of the robot device 1. More specifically, as the control to limit the movable range of the robot device 1, control is performed to limit the movable range of the movable part (partially movable part) of the arm part 1a. The safety determination / control unit F1 in this example performs control to stop the operation of the robot device 1 as the control to limit the movable range. Specifically, control is performed to stop the operation of each partially movable part in the arm part 1a.

[0042] Hereinafter, a specific processing example of the safety determination / control unit F1 will be described. Here, the processing of the safety determination / control unit F1 described below can be performed for each camera 2 when a plurality of cameras 2 that image different directions are provided as in the robot device 1 of this example.

[0043] FIG. 5 is a diagram for explaining an example of a specific safety determination method by the safety determination / control unit F1. As the safety determination, it is performed as a determination of whether the distance Dh to the person shown in the figure is less than or equal to the safety margin distance Dm. For confirmation, it should be noted that the person here is the person recognized by the image recognition process by the AI processing unit 44. The method of obtaining the distance Dh to the person will be described again later.

[0044] As shown in the figure, the safety margin distance Dm is calculated as a value obtained by adding the safety override distance Ds to the distance from the device position, which is the position of the robot device 1, to the outer end position of the movable range, which is the outer end position of the movable range of the arm part 1a.

[0045] The device position can be estimated based on images captured by camera 2, for example, by performing self-position estimation processing such as SLAM (Simultaneous Localization and Mapping). Alternatively, if the dispatch control device 100 controls the dispatch of the robot device 1, as in this example, the position information of the robot device 1 managed by the dispatch control device 100 can be used.

[0046] The safety margin distance Ds is calculated as a value that changes dynamically according to the speed of the person's movement, as shown in the following formula: "Ds = person's movement speed × (arm deceleration time + image recognition lag time) + additional safety distance" In this example, "arm deceleration time," "image recognition lag time," and "additional safety distance" are fixed values. "Arm deceleration time" means the time required from the time a stop command is given until the arm 1a actually stops moving. "Image recognition lag time" means the delay time until the AI ​​processing unit 44 obtains the image recognition result for an object that is a person. Specifically, for example, it could be the time from the start of the frame period (exposure period) of the captured image (frame image) that the AI ​​processing unit 44 targets for image recognition processing until the AI ​​processing unit 44 obtains the image recognition result. "Additional safety distance" is an additional margin distance to further enhance safety, and in this example, a predetermined fixed value is used.

[0047] We will explain specific methods for determining a person's movement speed at a later date.

[0048] Here, as understood from the above description, the safety margin distance Dm used for safety determination in the present embodiment includes the delay time until an image recognition result is obtained as the "image recognition lag time". In the robot device 1 of the present embodiment, since the AI processing unit 44 that performs image recognition processing is provided inside the camera 2, when obtaining an image recognition result, it is not necessary to transmit the captured image outside the camera 2. Therefore, a reduction in the time lag until an image recognition result is obtained can be achieved, and the above-mentioned "image recognition lag time" can be significantly reduced compared to the conventional case, and accordingly, the safety margin distance Dm can be set shorter. As a result, according to the present embodiment, the frequency of restricting the movable range of the robot device 1, that is, in this example, the frequency of forced stop of the operation of the arm unit 1a can be reduced, and the operation efficiency of the robot device 1 can be improved.

[0049] In particular, in this example, the AI processing unit 44 is provided inside the image sensor 21. Thereby, it is not necessary to transmit the captured image between different chips for image recognition processing, and a further reduction in the time lag until an image recognition result is obtained can be achieved, and a further suppression of the operation control delay of the robot device can be achieved.

[0050] Also, as understood from the above description, the safety determination and control unit F1 in this example sets a safety margin distance Dm according to the moving speed of a person, and based on the result of determining whether the person is approaching the robot device 1 within the safety margin distance Dm or less, performs control to restrict the movable range of the robot device 1.

[0051] This allows for dynamic adjustment of the safety margin distance in response to the person's movement speed when controlling the robot device 1 to limit its range of motion when a person approaches it within a predetermined safety margin distance. Specifically, the slower the person's movement speed, the shorter the safety margin distance becomes. Therefore, it is possible to prevent excessive restriction of the robot device 1's range of motion when the person's movement speed is slow, thereby further improving the operational efficiency of the robot device 1. In addition, safety can be ensured by shortening the safety margin distance when the person's movement speed is fast.

[0052] Furthermore, there are various methods for calculating the safety margin Ds, and it is not limited to the calculation method exemplified above. For example, although the above example uses fixed values ​​for "arm deceleration time," "image recognition lag time," and "additional safety distance," it is also possible to make at least one of these variable values. For example, if there are multiple operating modes with different operating speeds for the arm 1a, it is possible to use different values ​​for "arm deceleration time" depending on the operating mode. Similarly, for "image recognition lag time," for example, if there are multiple processing modes with different processing times for the image recognition processing by the AI ​​processing unit 44, it is possible to use different values ​​for "image recognition lag time" depending on the processing mode. Also, for "additional safety distance," it is possible to make it variablely configurable according to user operation. Furthermore, it is also possible to omit at least one of "arm deceleration time" and "additional safety distance," or to use another value in addition to, or replace at least one of, "arm deceleration time" and "additional safety distance" in the calculation of the safety margin Ds.

[0053] This section describes an example of a method for determining the location of a person recognized by image recognition processing. In this example, the safety determination / control unit F1 determines the location of the person based on the image captured by the camera 2. In this example, the safety determination / control unit F1 determines the location of the person based on the image captured after image signal processing by the image signal processing unit 42.

[0054] Figure 6 illustrates an example of a method for determining a person's position based on captured images. To determine a person's position based on captured images, multiple grid lines are defined on the floor surface visible in the image, as shown in the figure. In this example, it is assumed that camera 2 uses a fisheye lens for wide-angle imaging, resulting in image distortion. Therefore, the grid lines are not straight lines, but rather distorted to reflect the distortion of the image.

[0055] For each cell on the image defined as described above, the correspondence between the "position on the image" and the "position in the real world" is determined in advance through calibration. For example, if it is assumed that the robot device 1 moves between predetermined work positions as appropriate, as in this example, this calibration can be performed when the robot device 1 has completed moving to a work position.

[0056] When a person is recognized by the AI ​​processing unit 44, the safety determination and control unit F1 identifies which of the aforementioned grids the recognized person's feet are located in. Then, using the correspondence information obtained through the calibration described above, it identifies the "real-world location" corresponding to the identified grid, and obtains this "real-world location" information as the location information of the recognized person.

[0057] It should be noted that determining the distance Dh to a person does not necessarily require identifying the person's position based on the captured image, as described above. For example, the distance Dh to a person can be measured using a distance measuring sensor capable of measuring the distance to an object. Examples of distance measuring sensors include ToF (Time of Flight) sensors that measure distance using the ToF method, stereo sensors that measure distance using the stereo method, and radar sensors that measure distance using the radar method.

[0058] While the above example uses distance values ​​as the basis for safety determination, it is also possible to perform safety determination based on location. Specifically, it is possible to determine whether the location of a identified person is within a safety margin range with a radius of the safety margin distance Dm, that is, whether the coordinates indicating the person's location are included within that safety margin range. However, even if such a method is adopted, the substance remains the same: determining whether the distance to the person is less than or equal to the safety margin distance.

[0059] Next, we will explain an example of a method for determining a person's movement speed. A person's movement speed is calculated based on the positional difference between the current frame and previous past frames for a person recognized by image recognition processing. In this example, the person's movement speed is calculated based on the positional difference between the current frame and the frame immediately preceding it. Specifically, the positional difference, i.e., the distance the person moves within one frame period, is determined, and the person's movement speed is calculated by dividing this distance by the time length of one frame period.

[0060] In this example, since camera 2 uses a fisheye lens for imaging, distortion occurs in the captured image. However, image recognition processing using an AI model will experience a significant decrease in recognition performance at the edges of the captured image where the distortion is greater, making it extremely difficult to correctly recognize a person.

[0061] When considering a scenario where a person gradually approaches the center of the imaging range from outside the imaging range, if the person cannot be correctly recognized at the outer edge of the image, they will only be recognized as a person when they approach an area inside the outer edge. In this case, at the time of recognition, the person's position in the immediately preceding frame is unknown, making it impossible to determine the person's speed of movement, which delays the timing of safety determination.

[0062] Therefore, in this example, we will implement measures to prevent delays in safety assessments that occur due to the inability to immediately calculate the distance a person travels.

[0063] Figure 7 illustrates measures to prevent such delays in safety judgment timing. In Figure 7, the region labeled "Ei" represents the outer edge of the captured image (hereinafter referred to as "outer edge Ei"). Figure 7A shows a state where a person is captured in the outer edge Ei, and Figure 7B shows a state where the person has moved to an area inside the outer edge Ei.

[0064] In this example, the AI ​​processing unit 44 does not perform image recognition processing on the outer edge Ei of the captured image. Instead, the safety judgment / control unit F1 performs object detection processing by detecting the image difference between frames, and then tracks the object detected by this object detection processing in the time direction. Specifically, it determines the position and movement speed of the detected object in the current frame and records them in memory (for example, the RAM of the sensor's internal control unit 53). At this time, the movement speed of the detected object can be calculated based on the object's position in the current frame and its position in the previous frame. The area of ​​the object in the previous frame can be detected as an area where the image difference value from the current frame is greater than or equal to a predetermined value. Here, the object detected by the object detection processing of the outer edge Ei as described above can be rephrased as an object that was not recognized as a person by the image recognition processing of the AI ​​processing unit 44.

[0065] Subsequently, in response to the recognition of a person in the captured image through image recognition processing, the safety determination and control unit F1 determines whether the tracked object and the newly recognized person are the same object, based on the tracking information, specifically the information on the movement speed of the object detected at the outer edge Ei in this example. The movement speed of the newly recognized person is determined based on the tracking information and the location information of the newly recognized person.

[0066] Here, the determination of whether the object being tracked and the newly recognized person are the same object can be made based on the movement speed of the object being tracked from frame n-2 to frame n-1 (hereinafter referred to as "tracked object velocity Vt"), the position of the object being tracked in frame n-1 (hereinafter referred to as "tracked object immediate position Pt"), and the position of the person recognized by the image recognition process in frame n (i.e., the newly recognized person) (hereinafter referred to as "person recognition position Pr"). Specifically, first, assuming that the object being tracked and the newly recognized person are the same object, the movement speed from the tracked object immediate position Pt to the person recognition position Pr is calculated as the assumed movement speed Va. If the difference between this assumed movement speed Va and the tracked object velocity Vt is small, the newly recognized person can be considered to be the same object as the object being tracked. In this regard, the safety determination and control unit F1 in this example determines whether the difference between the assumed movement speed Va and the tracking target object speed Vt is within a predetermined value, and determines whether the tracking target object and the newly recognized person are the same object.

[0067] If the safety determination and control unit F1 determines that the difference between the assumed movement speed Va and the tracking target object speed Vt is within a predetermined value, and that the tracking target object and the newly recognized person are the same object, it sets the assumed movement speed Va as the movement speed of the newly recognized person.

[0068] <4. Processing Procedure> Figure 8 is a flowchart showing a specific example of a processing procedure for realizing the operation control method as described above. In this example, the processing shown in Figure 8 is executed by the CPU of the sensor internal control unit 43 based on a program stored in the ROM of the sensor internal control unit 43. However, for the purposes of this explanation, the entity executing the processing shown in Figure 8 will be the sensor internal control unit 43.

[0069] First, in step S101, the sensor control unit 43 inputs the image recognition result. That is, it inputs information indicating the result of the image recognition processing performed by the AI ​​processing unit 44 on the new frame image.

[0070] In step S102, following step S101, the sensor control unit 43 determines whether or not a person has been recognized. That is, based on the image recognition result input in step S101, it determines whether or not a person has been recognized in the new frame.

[0071] If it is determined in step S102 that no person was recognized, the sensor control unit 43 proceeds to step S103 and executes object detection processing on the outer edge of the captured image. That is, it performs object detection processing by image difference detection on the outer edge Ei of the captured image of the current frame.

[0072] In step S104, following step S103, the sensor control unit 43 determines whether or not an object has been detected at the outer edge Ei. If it determines that no object was detected at the outer edge Ei, the sensor control unit 43 returns to step S101. Depending on the processing in step S101 in this case, processing to input the image recognition result for the next new frame may be performed. In other words, if the process returns to step S101, a wait of one frame occurs.

[0073] On the other hand, if it is determined that an object has been detected at the outer edge Ei, the sensor control unit 43 proceeds to step S105 and calculates the object's movement speed based on the position difference from the previous frame. This corresponds to the calculation process of the tracking target object's velocity Vt as described above.

[0074] In step S106, following step S105, the sensor control unit 43 performs a process to record the object's position and speed in memory. That is, it performs a process to identify the position of the detected object (the "position in the real world" mentioned above), and then records the identified position information and the speed information calculated in step S105 in a predetermined memory such as the RAM of the sensor control unit 43.

[0075] The sensor control unit 43 returns to step S101 after executing the process in step S106.

[0076] As long as an object continues to be detected at the outer edge Ei, the processes from steps S101 to S106 are repeated, and the memory records information about the tracking target object, including its movement speed (tracking target object velocity Vt) and position information (position information that functions as the aforementioned tracking target object's immediate position Pt).

[0077] If it is determined in step S102 that a person has been recognized, the sensor control unit 43 proceeds to step S107. In step S107, the sensor control unit 43 determines whether or not a person was recognized in the previous frame.

[0078] In step S107, if it is determined that no person was recognized in the previous frame, the sensor control unit 43 proceeds to step S108 to determine whether the recorded object and the recognized person are the same object. In other words, in this example, after calculating the assumed movement speed Va mentioned above, the tracking target object velocity Vt is obtained as the latest movement speed recorded in memory in the processing of step S106, and it is determined whether the difference between the assumed movement speed Va and the tracking target object velocity Vt is less than or equal to a predetermined value.

[0079] Furthermore, the determination of whether or not two objects are the same is not limited to being based solely on their speed of movement, as described above; it is also conceivable to use positional information in addition. For example, one could determine the direction of movement of the tracked object based on the recorded position history of the object, i.e., the object being tracked, and simultaneously calculate the direction of movement from the position Pt immediately before the tracked object to the newly recognized person's position as a hypothetical direction of movement. The difference between these directions of movement could then be added as one of the conditions for determining that two objects are the same, provided that the difference is less than or equal to a predetermined value.

[0080] In step S108, if the sensor control unit 43 determines that the recorded object and the recognized person are not the same object, it returns to step S101. In other words, in this case, the person's movement speed is not set based on the tracking information.

[0081] On the other hand, if in step S108 the sensor control unit 43 determines that the recorded object and the recognized person are the same object, it proceeds to step S109 and performs the process of setting the movement speed of the newly recognized person. In other words, in this example, the process of setting the assumed movement speed Va calculated in the determination in step S108 as the movement speed of the newly recognized person is performed. In response to having performed the setting process in step S109, the sensor control unit 43 proceeds to step S111, which will be described later.

[0082] If, in step S107, it is determined that a person was also recognized in the previous frame, the sensor control unit 43 proceeds to step S110 and calculates the person's movement speed based on the position difference with the previous frame. That is, it calculates the person's movement speed based on the difference (distance between) between the position of the person recognized in the previous frame and the position of the person recognized in the current frame. In response to having performed the calculation process in step S110, the sensor control unit 43 proceeds to step S111.

[0083] In step S111, the sensor control unit 43 calculates the safety margin distance Dm based on the movement speed. Specifically, based on the movement speed set in step S109 or the movement speed calculated in step S110, the safety margin distance Ds is calculated using the formula shown above, and then the safety margin distance Dm is calculated by adding the safety margin distance Ds to the distance from the device position to the outer end of the movable range of the arm portion 1a (see Figure 5).

[0084] In step S112, following step S111, the sensor control unit 43 calculates the distance Dh to the person. In other words, in this example, the position of the newly recognized person (position in the real world) is identified using the method described with reference to Figure 6, and then the distance between the identified position of the person and the position of the device is calculated as the distance Dh to the person.

[0085] In step S113, following step S112, the sensor control unit 43 determines whether the distance Dh to the person is less than or equal to the safety margin distance Dm. If it determines that the distance Dh to the person is not less than or equal to the safety margin distance Dm, the sensor control unit 43 returns to step S101.

[0086] On the other hand, if the sensor control unit 43 determines that the distance Dh to the person is less than or equal to the safety margin distance Dm, it proceeds to step S114 and outputs a forced stop instruction signal for the arm unit 1a. That is, it outputs a forced stop instruction signal to the arm control unit 3 via the communication unit 24. This allows the movement of the arm unit 1a to be forcibly stopped in response to a person approaching the robot device 1 within the safety margin distance Dm.

[0087] The sensor's internal control unit 43 completes the series of processes shown in Figure 8 in accordance with the execution of the process in step S114.

[0088] <5. Modifications> Here, the embodiments are not limited to the specific examples described above, but a variety of modified configurations can be adopted. For example, although not specifically mentioned above, if the movement control of multiple robot devices 1 is performed by the dispatch control device 100, as in the embodiment illustrated above, it is conceivable that the safety determination could be made based on the position information of each robot device 1 managed by the dispatch control device 100. For example, in image recognition processing, it is possible that another robot device 1 may be mistakenly recognized as a person. Therefore, it is conceivable that the safety determination could be made by using the position information of other robot devices 1 managed by the dispatch control device 100, and if another robot device 1 is present at the location of the object that was mistakenly identified as a person, the recognition result of it being a person could be canceled.

[0089] Furthermore, while the above example shows an instance where both the AI ​​processing unit 44, which performs image recognition processing, and the safety judgment / control unit F1 are located within the image sensor 21 of the camera 2, this is merely one example. Variations in the location of the AI ​​processing unit 44 and the safety judgment / control unit F1 are also conceivable, as illustrated in Figures 9 to 12 below.

[0090] Figure 9 is an explanatory diagram of a first modified example relating to the location where the AI ​​processing unit 44 and the safety judgment / control unit F1 are installed. In the first modified example, the safety judgment / control unit F1, which was installed inside the sensor, is installed in the camera control unit outside the sensor.

[0091] As shown in the figure, the first modified robot device 1A differs from the robot device 1 in that it is equipped with camera 2A instead of camera 2. Camera 2A differs from camera 2 in that it is equipped with image sensor 21A instead of image sensor 21, and camera control unit 23A instead of camera control unit 23.

[0092] The image sensor 21A differs from the image sensor 21 in that it does not have a safety determination / control unit F1, while the camera control unit 23A differs from the camera control unit 23 in that it does have a safety determination / control unit F1.

[0093] Figure 10 is an explanatory diagram of a second modified example. In the second modified example, both the AI ​​processing unit 44 and the safety judgment / control unit F1, which were located inside the sensor, are located in a camera control unit outside the sensor. As shown in the figure, the robot device 1B of the second modified example differs from the robot device 1 in that it has a camera 2B instead of camera 2. Camera 2B differs from camera 2 in that it has an image sensor 21B instead of image sensor 21, and a camera control unit 23B instead of camera control unit 23.

[0094] The image sensor 21B differs from the image sensor 21 in that it does not have an AI processing unit 44 and a safety determination / control unit F1, while the camera control unit 23B differs from the camera control unit 23 in that it has an AI processing unit 44 and a safety determination / control unit F1.

[0095] Figure 11 is an explanatory diagram of a third modified example. In this third modified example, the safety judgment and control unit F1 is located on a processor 5 outside the camera. As shown in the figure, the robot device 1C of the third modified example differs from the robot device 1 in that it is equipped with camera 2C instead of camera 2. Camera 2C differs from camera 2 in that it is equipped with image sensor 21A (i.e., an image sensor with the safety judgment and control unit F1 omitted) instead of image sensor 21.

[0096] As shown in the figure, the robot device 1C is equipped with a processor 5, for example, a CPU, and this processor 5 is equipped with a safety determination and control unit F1.

[0097] In addition, in the third modified robot device 1C, the AI ​​processing unit 44 may be provided in the camera control unit 23.

[0098] Figure 12 is an explanatory diagram of the fourth modified example. The fourth modified example is an example in which the safety judgment / control unit F1 is provided in a separate device from the camera equipped with the AI ​​processing unit 44. Figure 12 shows an example in which the camera 2C equipped with an image sensor 21A having an AI processing unit 44 is provided as a separate device from the robot device 1D equipped with an arm control unit 3 and an arm actuator group 4. In this example, it is assumed that the robot device 1D does not have an AGV unit 1b and is fixed in a predetermined position, and that captured images of the outside of the robot device 1D are obtained by the camera 2C located outside the robot device 1D. Furthermore, in this example, it is assumed that the operation instructions for the robot device 1D are given by a control device 10, which is a computer device separate from the robot device 1D and the camera 2C. In addition, in this example, the safety judgment / control unit F1 is provided in this control device 10 as shown in the figure.

[0099] As can be understood from the above explanation, the robot device relating to this technology may also be configured without the AGV unit 1b (i.e., without self-propulsion function). Furthermore, the device form of the electronic equipment equipped with the safety judgment / control unit F1 is not limited to the form of a camera or robot device, but can also be a control device 10 or other form other than a camera or robot device.

[0100] In the explanations so far, we have used the example of restricting the movement of a robot device based on the results of a safety assessment as an example of restricting the movement of a partially movable part, such as the movable part of the arm. However, it is also conceivable that restrictions on the movement of the entire robot device, that is, the movement of the robot device, could be imposed based on the results of a safety assessment.

[0101] Furthermore, the operation control of the robot device based on the results of the image recognition processing performed by the AI ​​processing unit is not limited to control that restricts the operation of the robot device. For example, it is conceivable to perform control other than operation restriction, such as switching whether or not to overcome an object depending on the type of object recognized, as in the invention described in Patent Document 1, or operating the robot device in a pattern corresponding to human movement, as in the invention described in Patent Document 2, or controlling it to move while avoiding people.

[0102] <6. Summary of Embodiments> As described above, the electronic equipment of the embodiment (robot devices 1, 1A, 1B, 1C, control device 10, cameras 2, 2A, 2B, processor 5) is equipped with a control unit (safety judgment / control unit F1) that controls the operation of the robot device based on the results of image recognition processing performed by an AI processing unit located inside the camera on the captured images obtained by the camera capturing images of the outside of the robot device. As a result, it is not necessary to transmit the captured images outside the camera when obtaining image recognition results by the AI ​​model. Therefore, the time required to transmit the captured images when obtaining image recognition results can be shortened, the time lag until image recognition results are obtained can be reduced, and the delay in the operation control of the robot device can be suppressed.

[0103] Furthermore, in the electronic device of this embodiment, when a person is recognized in the captured image by image recognition processing, the control unit performs a process to determine the positional relationship between the person and the robot device based on the captured image or the output of the distance measuring sensor, and controls the range of motion of the robot device based on the result of determining whether the person has approached the robot device to a predetermined distance or less. This makes it possible to restrict the range of motion of the robot device in response to when a person approaches the robot device to a predetermined safety margin distance or less. Therefore, it is possible to prevent a person from coming into contact with the robot device and improve safety. Here, considering that there is a considerable time lag until the image recognition result from the AI ​​model is obtained, the above safety margin distance should be determined taking this time lag into consideration. Specifically, the greater the time lag until the image recognition result is obtained, the longer the safety margin distance should be. According to this embodiment, since the time lag until the image recognition result is obtained is reduced, the safety margin distance can be set to a shorter distance. If the safety margin distance is shortened, the frequency of the range of motion of the robot device being restricted will also be reduced, so the operating efficiency of the robot device can be improved.

[0104] Furthermore, the process of determining the positional relationship between a person and a robotic device is equivalent to the process of determining the person's position if the robotic device's position is known. Alternatively, the process of determining the distance from the robotic device to the person is also a form of determining the positional relationship between a person and a robotic device. In addition, the determination of whether a person has approached the robotic device to a predetermined distance or less is not limited to a distance-based determination of whether the distance to the person is less than or equal to a predetermined distance (safety margin distance), but also includes a position-based determination of whether the person is located within a predetermined distance range.

[0105] Furthermore, in the electronic device of this embodiment, the control unit performs processing to determine the person's movement speed based on the captured image, sets a safety margin distance according to the movement speed, and controls the range of motion of the robot device based on the result of determining whether the person is approaching the robot device within the safety margin distance. With the above configuration, when controlling the range of motion of the robot device when a person approaches the robot device within the safety margin distance, it is possible to dynamically change the safety margin distance according to the person's movement speed, specifically by shortening the safety margin distance as the person's movement speed decreases. Therefore, it is possible to prevent excessive restriction of the range of motion of the robot device even when the person's movement speed is slow, and further improvements in the operational efficiency of the robot device can be made. In addition, since it is possible to shorten the safety margin distance when the person's movement speed is fast, safety can also be ensured.

[0106] Furthermore, in the electronic device of this embodiment, the robot device has a partially movable part which is a movable part that can displace a part of the robot device, and the control unit controls the range of motion of the partially movable part of the robot device. This makes it possible to prevent a person from coming into contact with a part of the robot device when the robot device is configured to operate in a manner that displaces a part of it, thereby improving safety.

[0107] Furthermore, in the electronic device of this embodiment, the robot device has an arm, the partially movable part is configured as a movable part of the arm, and the control unit controls the range of motion of the partially movable part as a movable part of the arm. This makes it possible to prevent a person from coming into contact with the arm when the robot device has an arm, thereby improving safety.

[0108] Furthermore, in the electronic device of this embodiment, the control unit performs control to stop the robot's operation as a control to limit the robot's range of motion. This makes it possible to stop the robot's operation in response to a person approaching the robot below the safety margin distance, thereby further improving safety. In this embodiment, the safety margin distance can be set shorter by reducing the time lag until the image recognition result is obtained, thus reducing the frequency of the robot's operation being stopped. In other words, operational efficiency can be improved by increasing the opportunities for the robot to operate.

[0109] Furthermore, in the electronic device of the embodiment, the control unit performs temporal tracking of an object detected at the outer edge of the captured image taken at a first time point, which was not recognized as a person by image recognition processing. When a person is newly recognized by image recognition processing in the captured image taken at a second time point, after the first time point, the control unit determines, based on the tracking information, whether the tracked object and the newly recognized person are the same object, and determines the movement speed of the newly recognized person based on the tracking information and the position information of the newly recognized person. As a result, even if it is difficult to recognize a person by image recognition processing for objects captured at the outer edge of the captured image, such as when the captured image is taken using a fisheye lens, it is possible to immediately determine the movement speed of the person (without waiting for the person to move) as soon as the person is recognized by tracking from the outer edge. Therefore, the speed of determination regarding whether or not to restrict the range of motion of the robot device can be increased, and safety can be improved.

[0110] Furthermore, the electronic devices of the embodiment (robot devices 1, 1A, 1B, 1C) are equipped with cameras. That is, they are electronic devices in which a camera with an AI processing unit and a control unit are implemented. When controlling the operation of the robot devices, it becomes necessary to transmit image recognition results between devices, and the speed of operation control based on image recognition results can be improved.

[0111] Furthermore, in the electronic device of this embodiment, the AI ​​processing unit is located within the image sensor of the camera. This eliminates the need to transmit captured images between different chips for image recognition processing. Consequently, the time lag until image recognition results are obtained can be further reduced, and the delay in the operation control of the robot device can be further suppressed.

[0112] The robot system as an embodiment is a robot system that includes at least a robot device, and further includes a control unit that controls the operation of the robot device based on the results of image recognition processing performed by an AI processing unit located inside the camera on the captured image obtained by imaging the outside of the robot device with a camera. The same functions and effects as those of the electronic device as described above can be obtained with such a robot system.

[0113] Furthermore, the effects described herein are merely illustrative and not limited to those described herein, and other effects may also occur.

[0114] <7. This Technology> This technology can also be configured as follows: (1) An electronic device comprising a control unit that controls the operation of the robot device based on the results of image recognition processing performed by an AI processing unit located in the camera on an image obtained by capturing an image of the outside of the robot device with the camera. (2) The electronic device according to (1), wherein when a person is recognized in the image obtained by the image recognition processing, the control unit performs processing to determine the positional relationship between the person and the robot device based on the image obtained or the output of a distance measuring sensor, and controls the range of motion of the robot device based on the result of determining whether the person has approached the robot device to a predetermined distance or less. (3) The electronic device according to (2), wherein the control unit performs processing to determine the speed of movement of the person based on the image obtained, sets a safety margin distance according to the speed of movement, and controls the range of motion of the robot device based on the result of determining whether the person has approached the robot device to a safety margin distance or less. (4) The robot device has a partially movable part which is a movable part that can displace a part of the robot device, and the control unit is an electronic device according to any one of (2) to (4) above that performs control to limit the range of motion of the partially movable part of the robot device. (5) The robot device has an arm, and the partially movable part is configured as a movable part of the arm, and the control unit is an electronic device according to (4) above that performs control to limit the range of motion of the partially movable part as a movable part of the arm. (6) The control unit is an electronic device according to any one of (2) to (5) above that performs control to stop the operation of the robot device as control to limit the range of motion of the robot device.(7) The electronic device according to (3), wherein the control unit performs temporal tracking of an object detected at the outer edge of the captured image captured at a first time point, which was not recognized as a person by the image recognition processing, and when a person is newly recognized by the image recognition processing in the captured image captured at a second time point later than the first time point, the control unit determines, based on the tracking information, whether the object to be tracked and the newly recognized person are the same object, and determines the movement speed of the newly recognized person based on the tracking information and the position information of the newly recognized person. (8) The electronic device according to any one of (1) to (7), which is equipped with the camera. (9) The electronic device according to (8), wherein the AI ​​processing unit is provided in the image sensor of the camera. (10) A robot system comprising at least a robot device, wherein the control unit controls the operation of the robot device based on the results of image recognition processing performed by the AI ​​processing unit provided in the camera on an image obtained by capturing the outside of the robot device with the camera.

[0115] 1, 1A, 1B, 1C, 1D Robot device 1a Arm section J Joint section 1b AGV section 100 Dispatch control device 2, 2A, 2B, 2C Camera 3 Arm control section 4 Arm actuator group OP Imaging optical system 21, 21A, 21B Image sensor 41 Imaging section 42 Image signal processing section 43 Sensor internal control section 44 AI processing section 45 Memory section 46 Communication interface (I / F) 47 Bus 22 Memory section 23, 23A, 23B Camera control section 24 Communication section BS Bus F1 Safety judgment / control section Ei Outer edge section 5 Processor 10 Control device

Claims

1. An electronic device comprising a control unit that controls the operation of a robot device based on the results of image recognition processing performed by an AI processing unit located within the camera on an image obtained by capturing an image of the outside of the robot device using the camera.

2. The electronic device according to claim 1, wherein, when a person is recognized in the captured image by the image recognition process, the control unit performs a process to determine the positional relationship between the person and the robot device based on the captured image or the output of the distance measuring sensor, and performs control to limit the movable range of the robot device based on the result of determining whether the person has approached the robot device to a predetermined distance or less.

3. The electronic device according to claim 2, wherein the control unit performs a process to determine the movement speed of the person based on the captured image, sets a safety margin distance according to the movement speed, and performs control to limit the range of motion of the robot device based on the result of determining whether the person is within the safety margin distance of the robot device.

4. The electronic device according to claim 2, wherein the robot device has a partially movable part which is a movable part that can displace a part of the robot device, and the control unit performs control to limit the range of motion of the partially movable part of the robot device.

5. The electronic device according to claim 4, wherein the robot device has an arm, the partially movable part is configured as a movable part of the arm, and the control unit performs control to limit the range of motion of the partially movable part as a movable part of the arm.

6. The electronic device according to claim 2, wherein the control unit performs control to stop the operation of the robot device as a control to limit the range of motion of the robot device.

7. The electronic device according to claim 3, wherein the control unit performs temporal tracking of an object detected at the outer edge of the captured image captured at a first time point, which was not recognized as a person by the image recognition process, and when a person is newly recognized by the image recognition process in the captured image captured at a second time point later than the first time point, the control unit determines, based on the tracking information, whether the object being tracked and the newly recognized person are the same object, and determines the movement speed of the newly recognized person based on the tracking information and the location information of the newly recognized person.

8. The electronic device according to claim 1, comprising the camera.

9. The electronic device according to claim 8, wherein the AI ​​processing unit is provided within the image sensor of the camera.

10. A robot system comprising at least a robotic device, wherein the robot system comprises a control unit that controls the operation of the robotic device based on the results of image recognition processing performed by an AI processing unit located within the camera on an image obtained by capturing an image of the outside of the robotic device using a camera.

Images