Mobile apparatus control method, mobile apparatus control system, information processing method, information processing apparatus, mobile apparatus, article manufacturing method, and recording medium

Abstract

A control method for controlling, by a processing unit (101), a mobile apparatus (200) capable of executing a first mode for performing work and being movable, wherein the mobile apparatus (200) is capable of executing a second mode for performing an operation to respond to the appearance of a subject (306) satisfying a predetermined condition, and the processing unit (101) switches the operation mode of the mobile apparatus (200) from the first mode to the second mode when the processing unit (101) has detected the subject (306).

Description

Control method for a mobile device, control system for a mobile device, information processing method, information processing apparatus, mobile device, method for manufacturing an article, and recording medium

[0001] The present disclosure relates to a control method for a mobile device, a control system for a mobile device, an information processing method, an information processing apparatus, a mobile device, a method for manufacturing an article, a program, and a recording medium.

[0002] In recent years, there has been a proposal to cause a work robot that performs a predetermined task to execute an abnormal operation when an abnormality occurs (see Patent Document 1). In this Patent Document 1, it is proposed that when an abnormality occurs, the robot retreats from the user's movement route, moves to the location where the abnormality occurred, or returns to the home position.

[0003] Japanese Unexamined Patent Application Publication No. 2022 - 139018

[0004] By the way, in recent years, automation has been progressing in stores, factories, etc. using robots, and it has become possible for a small number of workers to perform operations. However, for example, if a person who causes an incident such as robbery appears and an attempt is made to deal with it with a small number of people, there is a problem that the damage may increase.

[0005] Therefore, an object of the present disclosure is to provide a control method for a mobile device, a control system for a mobile device, an information processing method, an information processing apparatus, a mobile device, a method for manufacturing an article, a program, and a recording medium that can reduce damage caused by a target person.

[0006] One aspect of the present disclosure is a control method for controlling a mobile device that can execute a first mode for performing work and can move, by a processing unit, wherein the mobile device can execute a second mode for performing an operation to deal with the appearance of a target person who satisfies a predetermined condition, and when the processing unit detects the target person, the processing unit switches the operation mode of the mobile device from the first mode to the second mode.

[0007] One aspect of the present disclosure is a control system for a mobile device, comprising: a mobile device that is capable of performing a first mode for performing work and a second mode for performing actions in response to the appearance of a person who meets predetermined conditions; and an information processing device having a processing unit for controlling the mobile device, wherein the processing unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the person.

[0008] One aspect of the present disclosure is an information processing method for processing information using an information processing apparatus having a processing unit that controls a mobile device capable of performing a first mode of work and being movable, wherein the mobile device is capable of performing a second mode of operation that deals with the appearance of a subject that meets predetermined conditions, and the processing unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the subject.

[0009] One aspect of the present disclosure is an information processing device having a processing unit that controls a mobile device capable of performing a first mode of work and being movable, the mobile device being capable of performing a second mode of action in response to the appearance of a subject that meets predetermined conditions, and the processing unit switching the operating mode of the mobile device from the first mode to the second mode when the subject is detected.

[0010] One aspect of the present disclosure is a mobile device having a control unit that is capable of performing a first mode of work and is mobile, wherein the control unit is capable of performing a second mode of operation that deals with the appearance of a subject that meets predetermined conditions, and when the subject is detected, the mobile device switches the operation mode from the first mode to the second mode. The effects of disclosure

[0011] According to this disclosure, it is possible to reduce the harm caused by the subject.

[0012] Other features and advantages of this disclosure will become apparent from the following description with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are given the same reference numeral.

[0013] This is a diagram showing the surveillance system according to the first embodiment. This is a block diagram showing the configuration of the surveillance device according to the first embodiment. This is a block diagram showing the configuration of the robot according to the first embodiment. This is a flowchart showing the thief detection process in the surveillance device according to the first embodiment. This is a flowchart showing the processing of the first mode in the robot according to the first embodiment. This is a flowchart showing the processing of the surveillance system when a thief is detected according to the first embodiment. This is a flowchart showing the processing of the second mode in the robot according to the first embodiment. This is a diagram showing the layout of the store. This is a diagram showing the arrangement of seats in the store. This is a diagram showing the surveillance cameras in the store and the imaging range of each surveillance camera. This is a diagram showing the paths that the robot can move. This is a diagram showing the position of the robot at the time a thief is detected. This is a diagram showing the movement path when the robot is moved toward the thief. This is a diagram showing the case when the robot is operated to move into the imaging range of the surveillance camera. This is a diagram showing the case when the robot is operated to move toward the thief. This is a diagram showing the case when the robot is operated to intercept between the thief and the customer. This is a diagram showing the case when one robot is operated to prevent the thief from escaping and another robot is operated to encourage the customer to evacuate. This is a diagram showing the case when two robots are operated to prevent the thief from escaping and encourage the customer to evacuate. This is a block diagram showing the configuration of the surveillance device according to the second embodiment. This is a block diagram showing the configuration of the robot according to the second embodiment. This flowchart shows the processing of the monitoring system when a thief is detected according to the second embodiment. This flowchart shows the processing of the second mode in the robot according to the second embodiment.

[0014] <First Embodiment> A first embodiment for implementing this disclosure will be described below with reference to the figures.

[0015] [Outline Configuration of the Monitoring System] First, the outline configuration of the monitoring system 1 according to the first embodiment will be explained using Figure 1. Figure 1 is a diagram showing the monitoring system according to the first embodiment.

[0016] As shown in Figure 1, the monitoring system 1, which acts as a control system for mobile devices, is a system that monitors a store 400, such as a restaurant. The monitoring system 1 includes a monitoring device 100, which acts as an information processing device for monitoring the store 400, and a plurality of robots 200-1, 200-2, which act as mobile devices that perform serving work (serving services) as a service in the store 400. The robots 200-1, 200-2 shown in Figure 1 are what are commonly called service robots or serving robots. Although Figure 1 shows a case where two robots are deployed in the store 400, the number may be one or three or more. Furthermore, in the following description, when it is not necessary to distinguish between individual robots, they will simply be referred to as robot 200, and when it is necessary to distinguish them, they will be referred to as robot 200-1, 200-2, etc., using a hyphen.

[0017] Furthermore, in the store 400 shown in Figure 1, we will explain assuming that there is an employee 301, customers 302, 303, 304, 305, and a thief 306. The thief 306 is, for example, a person who has committed a crime such as robbery in the past, or a person who is likely to commit a crime such as robbery in the future, and in this embodiment, is a target person who meets predetermined conditions. These predetermined conditions refer to matching the information in the database, which will be described in more detail later. On the other hand, the employee 301 and customers 302, 303, 304, and 305 are persons other than the target persons. In particular, customers 302, 303, 304, and 305 are persons who should be given priority in urging evacuation.

[0018] [Configuration of the monitoring device and robot control system] Next, the configuration of the monitoring device 100 and the control system of the robot 200 will be explained using Figures 1, 2A, and 2B. Figure 2A is a block diagram showing the configuration of the monitoring device according to the first embodiment. Figure 2B is a block diagram showing the configuration of the robot according to the first embodiment.

[0019] (Control system of the monitoring device) As shown in Figures 1 and 2A, the monitoring device 100 is broadly composed of a processing unit 101, a storage unit 102, a detection unit 103, a communication unit 104, and a network unit 105. The processing unit 101 has a CPU 101a, and is configured to perform its function as a processing unit by executing a program stored in the storage unit 102, and performs various processes in the monitoring device 100 (see Figures 3 and 5), which will be described in detail later.

[0020] The storage unit 102 is configured to achieve its function as a storage unit by having recording media such as ROM 102a, RAM 102b, and HDD (hard disk drive) 102c. The storage unit 102 records data (information) used in various processes, which will be described in detail later, and functions as a database.

[0021] The detection unit 103 is configured to function as a detection unit that detects people, including the thief 306, by having a plurality of surveillance cameras 103A that are fixed above the ceiling of the store 400 and act as imaging devices to acquire images (still images, moving images), as will be described later. In the following description, if it is not necessary to distinguish between individual surveillance cameras, they will simply be referred to as surveillance camera 103A, and if it is necessary to distinguish between them, they will be referred to as surveillance camera 103A-1 to 103A-5 using a hyphen.

[0022] The communication unit 104 is configured to function as a communication unit that communicates wirelessly with the robot 200, by having an antenna and communication equipment (not shown in the figure). The communication method of the communication unit 104 can be any wireless communication method, such as wireless LAN, infrared communication, Bluetooth (registered trademark), etc.

[0023] The network unit 105 is configured to function as a network unit that transmits and receives data to and from an external network by having an interface (I / F) 105a. The network unit 105 refers to a WAN, etc., and is connected to the so-called internet. As will be described in more detail later, it can receive input such as a learning model that has learned the operation of the continuation 06 by AI (artificial intelligence), and publicly available information from the police.

[0024] (Robot Control System) As shown in Figures 1 and 2B, the robot 200 is broadly composed of a control unit 201, an input unit 203, a communication unit 204, a drive unit 206, and an output unit 207. The control unit 201 has a CPU 201a, a ROM 202b, a RAM 202c, etc., and is configured to perform its function as a control unit by having the CPU 201a execute a program stored in the ROM 202b, etc. That is, the control unit 201 performs various processes in the robot 200 (see Figures 4, 5, and 6), which will be described in more detail later. The control unit 201 is configured to be able to execute two operation modes, which will be described in more detail later: a first mode that performs operations related to work (tasks), and a second mode that performs operations to deal with the appearance of a thief 306 as a target person that meets predetermined conditions.

[0025] The input unit 203 includes a camera 203A, a microphone 203B as an audio input unit, a distance sensor 203C, an operation unit 203D, etc., and outputs the information acquired by these as input data to the control unit 201. Specifically, the camera 203A is mounted on the robot 200 and, for example, takes images of the area in front. This acquired image information is output to the control unit 201 and used for autonomous driving (collision avoidance) of the robot 200, and is also output to the monitoring device 100 via the communication unit 204 and can be used for detecting the thief 306, which will be described in more detail later. The microphone 203B is mounted on the robot 200 and, for example, acquires (records) surrounding sounds. This acquired sound information is output to the monitoring device 100 via the communication unit 204 and can be used for detecting the thief 306, which will be described in more detail later. The distance sensor 203C measures the distance to surrounding obstacles. This measured distance information is output to the control unit 201 and is used for autonomous driving (collision avoidance) of the robot 200. The control unit 203D consists of, for example, an operation panel and buttons mounted on the robot 200, and accepts various operations from the store clerk 301 and customers 302, 303, 304, and 305 (see S201, S205, S226, etc.).

[0026] The communication unit 204 is configured to function as a communication unit that communicates wirelessly with the monitoring device 100, by having an antenna and communication equipment (not shown in the figure). In other words, the communication unit 204 is configured to receive commands from the processing unit 101 of the monitoring device 100 and input them to the control unit 201. The communication method of the communication unit 204 can be any method that is compatible with the communication method of the monitoring device 100.

[0027] The drive unit 206 is configured to function as a drive unit that enables the robot 200 to move within the store 400, by having an electric motor, battery, wheels, etc. (not shown). The drive state of the drive unit 206 is controlled by the control unit 201 based on the selected path and target coordinates, and the control unit 201 also controls the robot to avoid collisions based on images captured by the camera 203A.

[0028] The output unit 207 is configured to have a display unit 207A, which is composed of a display or the like, in order to achieve the function of displaying various information to the store clerk 301, customers 302, 303, 304, 305, and thief 306. In this embodiment, the display unit 207A and the operation unit 203D are described as separate units, but the system is not limited to this, and they may be integrated using a touch panel or the like.

[0029] [Processing of Monitoring System 1] Next, the various processes in the monitoring system 1 described above will be explained with reference to the diagram. In recent years, for example, in restaurants 400, the replacement of food service work from employees 301 to so-called service robots (serving robots) is progressing with the aim of reducing manpower. As a result, the number of restaurant employees 301 is decreasing, but if a store is attacked by thieves 306, for example, the number of employees 301 who can respond will decrease, which may lead to greater damage. Therefore, in this embodiment, the monitoring system 1 uses a robot 200 that performs normal service work (tasks) to prevent crimes committed by thieves 306 or reduce damage when the store is attacked. The processing of the monitoring system 1 (control method of the mobile device, information processing method for processing information by the monitoring system 1) will be explained below, divided into the processing of the monitoring device 100 and the processing of the robot 200.

[0030] (Store Details) Here, we will first explain the details of the store 400 layout, surveillance camera 103A and surveillance area Arn, robot 200 movement path RT, etc., using Figures 7A to 8B. Figure 7A is a diagram showing the store layout. Figure 7B is a diagram showing the seating arrangement in the store. Figure 8A is a diagram showing the surveillance cameras in the store and the range in which each surveillance camera can capture images of intent to kill. Figure 8B is a diagram showing the paths that the robot can move along.

[0031] As shown in Figure 7A, the store 400 located in the security area ArX, which is a security zone, has a layout in which the indoor area ArIn is enclosed as a room by a structural wall 400W. The store 400 is also equipped with an automatic door 403 as a device. Opening the automatic door 403 connects the indoor area ArIN to the outdoor area ArOUT, which is an area within the store 400's premises or a public area facing the premises and is part of the security area ArX. In other words, customers 302-305 and thieves 306 enter the indoor area ArIn from the outdoor area ArOUT by passing through the automatic door 403.

[0032] As shown in Figure 7B, the indoor area ArIN of the store 400 is equipped with multiple seats 401a to 401o, which are structural components, in each section separated by the wall 400W. The indoor area ArIN of the store 400 is also equipped with a waiting position 402 where the robot 200 waits. The robot 200 waits at the waiting position 402 until, for example, food is placed on it. Once the food is placed on it by the employee 301 and serving is instructed via the control unit 203D, the robot 200 begins to move. When the robot 200 moves, it moves along the movement path RT in the movable area ArM shown in Figure 8B, heading towards the designated seat among the multiple seats 401a to 401o. After serving is complete, it moves along the movement path RT back to the waiting position 402. The robot 200 is capable of autonomous movement within the movable area ArM and is not necessarily restricted to the movement path RT, but can move while avoiding obstacles and people.

[0033] Furthermore, as shown in Figure 8A, four surveillance cameras 103A-1 to 103A-4 are installed on the ceiling of the indoor area ArIN of the store 400, and one surveillance camera 103A-5 is also installed in the outdoor area ArOUT. Of these, surveillance camera 103A-1 is capable of capturing images within the surveillance area Ar1, in other words, surveillance area Ar1 is within the field of view of surveillance camera 103A-1. Similarly, surveillance camera 103A-2 is capable of capturing images within the surveillance area Ar2, surveillance camera 103A-3 within the surveillance area Ar3, surveillance camera 103A-4 within the surveillance area Ar4, and surveillance camera 103A-5 within the surveillance area Ar5.

[0034] In the indoor area ArIN, the monitoring areas Ar1 to Ar4 of the surveillance cameras 103A-1 to 103A-4 are positioned to slightly overlap with adjacent monitoring areas. In other words, the indoor area ArIN is within the imaging range of any of the surveillance cameras 103A. Furthermore, the monitoring area Ar5 of surveillance camera 103A-5 is designed to image the passageway leading to the indoor area ArIN (automatic door 403) in the outdoor area ArOUT. That is, for example, customers 302 to 305 and thieves 306 would pass through monitoring area Ar5 to reach the indoor area ArIN. If the indoor area ArIN is directly connected to electrically powered moving equipment such as elevators or escalators, the monitoring area Ar5 of surveillance camera 103A-5 may be designed to image the inside of the elevator or the top of the escalator.

[0035] (Detection of thieves in the monitoring device) First, the detection process of thieves 306 processed by the monitoring device 100 will be explained using Figure 3. Figure 3 is a flowchart showing the detection process of thieves in the monitoring device according to the first embodiment.

[0036] For example, when the power of the monitoring device 100 is turned ON, the process of detecting the thief 306 is executed as shown in Figure 3. First, in step S101, the processing unit 101 of the monitoring device 100 acquires video (which may also be a still image) captured by each of the multiple monitoring cameras 103A located in the security area ArX. Similarly, in step S101, the processing unit 101 acquires video (which may also be a still image) captured by the camera 203A of the robot 200 located in the security area ArX. Furthermore, similarly, in step S101, the processing unit 101 acquires audio picked up by the microphone 203B of the robot 200 located in the security area ArX.

[0037] Next, the processing unit 101 performs image analysis on the captured video footage to detect people from the video footage using image recognition, or performs voice analysis on the recorded audio to detect people from the audio using voice recognition (S102). This image recognition or voice recognition of people can be performed using commonly used image recognition technologies such as face recognition or recognition of full-body features, or commonly used voice recognition technologies. Then, the processing unit 101 compares (verifies) the video and audio information of the detected people with the information of the person 306 stored in a database, for example, in the HDD 102c of the storage unit 102 (S103).

[0038] Here, we will explain the information about the thief 306 stored in the database (i.e., HDD 102c). The database stores information related to the image of the thief 306, such as information on the appearance of the perpetrator released by the police, a learning model for image recognition of the criminal's actions, and a learning model for image recognition of the actions of people around the criminal. The database also stores information related to the image of the thief 306, such as information on the appearance of a person who has been banned from entering store 400. Furthermore, the database stores information related to the voice of the thief 306, such as a learning model for speech recognition of the criminal's statements, and a learning model for speech recognition of the statements of people around the criminal. This information is acquired via the network unit 105 and stored, for example, on HDD 102c, to form a database.

[0039] The information about the suspect's appearance released by the police specifically refers to image information such as facial photographs and sketches of wanted criminals (i.e., the perpetrators) released on the internet by organizations such as the National Police Agency or International Criminal Police Organization. This information released by the police may also be a machine learning model created using AI or similar technologies. Furthermore, this machine learning model obtained from the police may include not only the perpetrator's physical characteristics but also behavioral characteristics such as distinctive movements and habits.

[0040] Furthermore, the learning models used for image recognition of criminal behavior are, for example, learning models of human actions that represent the behavior of criminals, created through machine learning using AI at research institutions, security companies, and surveillance system development companies. Criminal behavior here refers to actions such as intentional acts of vandalism, destruction, and violence. Specifically, for example, a customer or store clerk might drop a plate by accident, but a criminal might intentionally throw a plate and drop it, and even though the result is that the plate breaks, there are differences in the actions leading up to that point. Learning models for image recognition of criminal behavior are created by learning and modeling such criminal actions.

[0041] Furthermore, the learning models used for image recognition of the behavior of those surrounding criminals are also learning models of human actions that show the behavior of those around criminals, created through machine learning using AI, etc., at research institutions, security companies, surveillance system development companies, etc. Specifically, for example, customers and store clerks may behave in fear or flee when faced with a criminal. Learning models that model the behavior of such people around criminals are used for image recognition of the behavior of those around criminals. In short, if the people around a criminal can be identified, then anyone behaving differently can be detected as a criminal.

[0042] Furthermore, the learning models used for speech recognition of criminals' statements are human speech models that represent the statements of criminals, created through machine learning using AI, etc., at research institutions, security companies, surveillance system development companies, etc. Specifically, criminals may make statements that involve shouting, making strange noises, or making threats, and learning models for speech recognition of criminals are created by learning and modeling such statements made by criminals.

[0043] In addition, a learning model for voice recognition of the statements of people around the criminal is also a learning model of human voices representing the statements of people around the criminal created by machine learning using AI or the like in, for example, research institutions, security companies, surveillance system development companies, etc. Specifically, for example, people around the criminal may make statements such as screaming or stopping the criminal, and a model created by learning and modeling such statements of people around the criminal serves as a learning model for voice recognition of the statements of people around the criminal.

[0044] As described above, the processing unit 101 compares the images of the detected person from the plurality of surveillance cameras 103A or the camera 203A of the robot 200 with the information of the thief 306 in the database to determine whether the detected person (target person) is the thief 306 (S104). Alternatively, the processing unit 101 compares the voice of the detected person from the microphone 203B of the robot 200 with the information of the thief 306 in the database to determine whether the detected person (target person) is the thief 306 (S104). In other words, it is determined whether it matches the information in the database, and if it matches the information in the database, it is considered that the person satisfies the predetermined conditions, and the detection of the appearance of the thief 306 is performed. Here, if it is determined that the detected person is not the thief 306 (usually determined as the store clerk 301 or the customers 302 to 305) (No in S104), the process returns to step S101, and the processes described above are repeated.

[0045] On the other hand, if it is determined that the detected person is the thief 306, that is, if the thief 306 is detected from the acquired video and voice information (Yes in S104), first, the detected person (target person) is recognized as the thief 306 (S105). That is, the processing unit 101, for example, turns on various flags and proceeds to the implementation of the thief countermeasure process (S106). Here, for example, if the customers 302 to 305 take the same actions as the thief 306, that is, suspicious actions, they may be recognized as thieves. In that case, the store clerk 301 may rejudge the customers and perform a process to cancel this recognition. The details of the thief countermeasure process will be described later as the process when a thief is detected in the surveillance system.

[0046] In the above description, it has been explained that when the detected person matches the information in the database, it is determined that the detected person is the thief 306. However, it is not limited to this. For example, when a part of the information in the database matches, the detected person may be determined as a candidate for the thief 306. In this embodiment, also in this case, the operation mode of the robot 200 is switched from the first mode to the second mode. Then, in order to deal with the candidate for the thief 306, the robot 200 may be moved, for example, the robot 200 may be made to perform operations such as waiting near the candidate.

[0047] (Processing in the First Mode of the Robot) Next, the processing in the first mode of the robot 200, that is, the processing in the mode for performing normal work (service), will be described with reference to FIG. 4. When the control unit 201 of the robot 200 is controlled as the first mode which is a mode for performing normal work, the processing shown in FIG. 4 is executed.

[0048] In the first mode, the control unit 201 of the robot 200 first waits for an instruction input from the store clerk 301. Then, when the store clerk 301, for example, places a loaded item such as a dish on a table (not shown) of the robot 200 and inputs an instruction to perform food service via the operation unit 203D, the control unit 201 receives the input (S201). Then, the control unit 201 moves from the current position toward the location instructed by the store clerk 301 from the in-store map (see FIGS. 7A and 7B) read and stored in the ROM 201b, etc. (S202). That is, a route RT (see FIG. 8B) is set from the current position to the location (the position of the destination seat) where the loaded item (dish) is to be served, and the robot moves along that route RT. During this movement, the control unit 201 drives the drive unit 206 to perform autonomous driving while avoiding collisions with the customers 302 to 305 and obstacles based on the video of the camera 203A and the distance to the approaching objects by the distance sensor 203C. Then, the movement process is repeated until the instructed location is reached (No in S203), and when the instructed location is reached (Yes in S203), the drive unit 206 is stopped and the movement is stopped (S204).

[0049] Next, the control unit 201 waits until, for example, a customer at the seat where the food was served takes the food and the customer inputs the end of serving via the operation unit 203D, or until it detects that the food or other items on the vehicle have been removed from the camera 203A image or the like (No. in S205). When the control unit 201 detects that the customer has input the end of serving or that the items on the vehicle have been removed (Yes in S205), it moves toward the waiting position 402 (see Figure 8B) (S206). During this time, the control unit 201 also drives the drive unit 206 to autonomously navigate, avoiding collisions with customers 302-305 or obstacles, based on the image from the camera 203A and the distance to nearby objects detected by the distance sensor 203C. In addition, although it has been explained that the image from the camera 203A is used to detect that the items on the vehicle have been removed, it may also be possible to simply provide a weight sensor or pressure sensor and detect it using a sensor.

[0050] (Processing flow of the monitoring system when a thief is detected) Next, the processing flow of the monitoring system 1 when the monitoring device 100 detects a thief 306 as described above (see S106) will be explained using Figure 5. Figure 5 is a flowchart showing the processing of the monitoring system when a thief is detected according to the first embodiment.

[0051] As described above, when the monitoring device 100 detects the thief 306 (Yes in S104), it proceeds to execute the countermeasures against the thief 306 (S106). Then, as shown in Figure 5, the overall processing of the monitoring system 1 is as follows: First, the processing unit 101 of the monitoring device 100 commands the control unit 201 of the robot 200, and the robot 200 transitions from the first mode to the second mode. As a result, as will be described in more detail later, the processing in the second mode causes the robot 200 to move into the field of view of the monitoring camera 103A that shows the thief 306.

[0052] Next, the processing unit 101 of the monitoring device 100 acquires the layout of the store 400 (i.e., location information of the building's structure) (see Figures 7A and 7B) that has been pre-recorded in the HDD 102c (S111). Then, the processing unit 101 detects and acquires the locations of the thief 306, the store clerk 301, and the customers 302-305 by performing image recognition on the video footage captured by the multiple monitoring cameras 103A (S112). Then, as will be described in more detail later, it determines the position for the robot 200 to deal with the thief 306 and instructs the robot 200 to move to that position (S113). Upon receiving this command, the robot 200 moves to the position calculated by the monitoring device 100 according to the command (S212).

[0053] (Processing in the second mode of the robot) Next, the processing when the robot 200 switches from the first mode to the second mode, that is, the processing in the second mode, will be explained using Figures 6, 9A, and 9B. Figure 6 is a flowchart showing the processing in the second mode of the robot according to the first embodiment. Figure 9A shows the position of the robot at the time the thief is detected. Figure 9B shows the movement path when the robot moves toward the thief.

[0054] As shown in Figure 6, first, the control unit 201 of the robot 200 receives a command from the monitoring device 100 to transition the robot 200 to the second mode and to move to the field of view of the monitoring camera 103A that shows the thief 306 (see S113) (S221). Then, if the control unit 201 is working in the first mode, it interrupts that work (serving) (S222). The control unit 201 then transitions the robot 200's operating mode from the first mode to the second mode and performs the process of moving to the field of view of the monitoring camera 103A instructed by the processing unit 101 of the monitoring device 100 until it arrives at that location (No. in S223 and S224). After that, once the robot 200 arrives at that location, it proceeds to the next process (Yes in S224). Up to this point, this corresponds to the process of step S211 explained in Figure 5.

[0055] A specific example of the process from step S221 to step S224 will be explained. For example, the robot 200 that is serving food is located outside the monitoring area Ar4, which is the field of view of the surveillance camera 103A-4 that captures the thief 306, in the movable area ArM of the store 400, as shown in Figure 9A. When the monitoring device 100 detects the thief 306 while the robot 200 is in this position, the processing unit 101 of the monitoring device 100 commands the control unit 201 of the robot 200 to transition from the first mode to the second mode and to move into the monitoring area Ar4 (i.e., within the field of view). Upon receiving this command, the robot 200 sets a movement path RT4, as shown in Figure 9B, and moves to the monitoring area Ar4, that is, moves so that the robot 200 is visible to the surveillance camera 103A-4. In short, the processing unit 101 calculates the position of the monitoring area Arn to which the robot 200 should move in response to the surveillance camera 103A-n showing the thief 306, commands the robot 200, and the robot 200 moves to the commanded monitoring area Arn (see Figure 10A).

[0056] Next, as described above, when the robot 200 arrives at the designated field of view of the surveillance camera 103A, the control unit 201 moves to the position designated by the processing unit 101 of the monitoring device 100 (S225), that is, it actually performs the process of dealing with the thief 306. Specific details of this dealing with the thief 306 will be described later. This process is repeated until the processing unit 101 of the monitoring device 100 issues an instruction to deactivate the second mode (No. in S228). That is, when the processing unit 101 of the monitoring device 100 determines that it has finished dealing with the thief 306, the processing unit 101 instructs the control unit 201 of the robot 200 to deactivate the second mode, that is, to switch from the second mode to the first mode (Yes in S228). Also, when the store clerk 301 determines that it has finished dealing with the thief 306, it inputs the command to deactivate the second mode using the operation unit 203D of the robot 200 (Yes in S226). Then, the control unit 201 of the robot 200 sends a request to the processing unit 101 of the monitoring device 100 to deactivate the second mode (S227). Upon receiving this, the processing unit 101 of the monitoring device 100 determines that the second mode should be deactivated and instructs the control unit 201 of the robot 200 to deactivate the second mode, that is, to switch from the second mode to the first mode (Yes in S228). As a result, the control unit 201 of the robot 200 switches the operating mode to the first mode (S229), and thus the processing of the second mode is completed. During this time, the processing unit 101 of the monitoring device 100 repeatedly moves the robot 200 relative to the position of the thief 306, that is, controls it using visual feedback, so the robot 200 needs to be within the field of view of the monitoring camera 103A. Also, the processing in step S225 corresponds to the processing in step S212 explained in Figure 5.

[0057] Furthermore, in the monitoring system 1, if there are multiple robots 200, the processing unit 101 of the monitoring device 100 commands each of the control units 201 of the multiple robots 200 to switch to the first mode all at once. Also, for example, if a store employee 301 makes an input to deactivate the second mode, the operation is performed on one of the multiple robots 200. In response, the processing unit 101 of the monitoring device 100 commands each of the control units 201 of the multiple robots 200 to switch to the first mode all at once.

[0058] [Specific Examples of Handling Thieves] Next, specific examples of handling thieves 306 by the robot 200 will be explained using Figures 10A to 12. Figure 10A shows the case where the robot is operated to move into the imaging range of the surveillance camera. Figure 10B shows the case where the robot is operated to move closer to the thief. Figure 11A shows the case where the robot is operated to intervene between the thief and the customer. Figure 11B shows the case where one robot is operated to prevent the thief from escaping, and another robot is operated to encourage the customer to evacuate. Figure 12 shows the case where both robots are operated to prevent the thief from escaping and to encourage the customer to evacuate.

[0059] As shown in Figure 10A, the robot 200 is moved to the monitoring area Arn, which is within the field of view of the monitoring camera 103A-n that shows the thief 306, by instruction from the processing unit 101 of the monitoring device 100 (see S211). Then, in this first embodiment, the process of dealing with the thief 306 by moving the robot 200 is initiated (see S212). The process of dealing with the thief 306 can be broadly considered as follows: the process of securing the thief 306, the process of prioritizing the safety of customers 302-305, the process of prioritizing the evacuation of customers 302-305, and the process of preventing the thief 306 from entering the room. Note that the process of dealing with the thief 306 is not necessarily limited to the robot 200 directly dealing with the thief 306, but refers to the process in which the robot 200 deals with the thief 306 as a result of its appearance, that is, the process of dealing with customers 302-305 is also included.

[0060] (Process of securing the thief) First, the process of securing the thief 306 by moving the robot 200 will be explained. Suppose the processing unit 101 of the monitoring device 100 has decided to perform the process of securing the thief 306 when it has switched the robot 200 to the second mode (i.e., when it has determined that there is an emergency). Then the processing unit 101 controls the movement of the robot 200 to perform actions that prevent the thief 306 from escaping from the robot 200's movable area ArM. That is, for example as shown in Figure 10B, the robot 200 moves close to the thief 306, and in particular moves to position itself in the thief 306's escape path, thereby preventing the thief 306 from escaping.

[0061] In this case, the processing unit 101 can control the movement of the robot 200 according to the layout of the store 400 (i.e., the position information of the structures) in the robot 200's movable area ArM. That is, it can be controlled to move the robot 200 in such a way that it corners the thief 306 between the robot 200 and structures such as the walls 400W and seats 401a to 401o of the store 400, thereby preventing his escape and capturing him.

[0062] Furthermore, in the monitoring system 1, if there are multiple robots 200, the processing unit 101 switches the operating mode of two or more of the multiple robots 200, or all of them, to the second mode. Then, the processing unit 101 can control a selected robot 200 from among the multiple (two or more) robots 200 that have been switched to the second mode to perform an action to prevent the escape of the thief 306. That is, for example, as shown in Figure 11B, the selected robot 200-1 is controlled to move in a way that prevents the escape of the thief 306 and secures him, while the other robots 200-2 are controlled to move in a way that prioritizes the evacuation of the customer 302, as will be described later. It is conceivable that robot 200-1 would be the robot that is closest to the thief 306 at the time of detection, or a robot that was not serving food.

[0063] Furthermore, it is conceivable that the processing unit 101 may control two or more robots 200 selected from among the multiple robots 200 switched to the second mode to work in coordination without interfering with each other, thereby hindering the escape of the thief 306. That is, for example, as shown in Figure 12, it is conceivable that two selected robots 200-1 and 200-2 may be controlled to move in a way that hinders the escape of the thief 306 and secures him, in other words, the thief 306 may be secured by narrowing his escape route.

[0064] Furthermore, it is conceivable that the processing unit 101 of the monitoring device 100 may detect multiple thieves 306. In this case, the processing unit 101 may coordinate multiple robots 200 to prevent the escape of multiple thieves 306 and capture them. For example, one robot 200 may be moved to capture the first thief 306, and another robot 200 may be moved to capture the second thief 306. Of course, if there are three or more thieves 306, three or more robots 200 may be moved and assigned to each thief 306 to prevent their escape.

[0065] Furthermore, the processing unit 101 may control devices such as automatic doors 403 that are on the escape route of the thief 306 in the security area ArX, and perform actions to prevent the thief 306 from escaping. That is, for example, as shown in Figure 11B, as will be described in more detail later, it is conceivable that the automatic doors 403 could be opened when the customer 302 evacuates, and that the automatic doors 403 could be closed when the thief 306 attempts to escape, thereby securing the thief 306. Although automatic doors 403 are described as an example of a device on the escape route of the thief 306, it is not limited to this, and could also be an electric mobility device such as an elevator or escalator, and it is conceivable that the thief 306 could be secured by controlling the stopping of these devices.

[0066] (Processing to prioritize customer safety) Next, we will explain the process of prioritizing the safety of customers 302-305 through the movement of the robot 200. Suppose the processing unit 101 of the monitoring device 100 has determined that when it switches the robot 200 to the second mode (i.e., when it has determined that an emergency has occurred), it will perform processing to prioritize the safety of customers 302-305. Then, the processing unit 101 can control the movement of the robot 200 to perform an action that prevents the approach of the thief 306 and customers 302-305, who are not the thief 306. That is, for example, as shown in Figure 11A, the robot 200 moves to intercept between the thief 306 and customer 302, preventing them from approaching each other. This can be considered to prioritize the safety of customer 302. Note that the robot 200 may also move to intercept between the thief 306 and the store clerk 301, who is not the thief 306, not just customers 302-305.

[0067] Furthermore, in the monitoring system 1, if there are multiple robots 200, the processing unit 101 switches the operating mode of two or more of the multiple robots 200, or all of them, to the second mode. Then, the processing unit 101 can control a selected robot 200 from among the multiple (two or more) robots 200 that have been switched to the second mode to perform an action that prevents the thief 306 and the customers 302-305 from approaching each other. That is, for example, as shown in Figure 11B, the selected robot 200-1 is controlled to move in a way that prevents the thief 306 and the customers 302 from approaching each other. Then, the other robots 200-2 can be controlled to move in a way that prioritizes the evacuation of the customers 302, as will be described later.

[0068] Furthermore, it is conceivable that the processing unit 101 may select two or more robots 200 from among the multiple robots 200 that have been switched to the second mode and control their movement to coordinate actions that prevent the thief 306 from approaching the customer 302. That is, for example, as shown in Figure 12, it is conceivable that two selected robots 200-1 and 200-2 may be controlled to move in a manner that prevents the thief 306 from approaching the customer 302.

[0069] (Processing to prioritize the evacuation of customers) Next, we will explain the process of prioritizing the evacuation of customers 302 to 305 by moving the robot 200. Suppose the processing unit 101 of the monitoring device 100 has determined that when it switches the robot 200 to the second mode (i.e., when it has determined that there is an emergency), it will prioritize the evacuation of customers 302 to 305, who are not the thief 306. Then the processing unit 101 controls the movement of the robot 200 to perform an action that encourages the evacuation of customers 302 to 305 from the robot 200's movable area ArM. That is, for example as shown in Figure 11B, the robot 200-2 moves away from the evacuation route of customer 302 and moves the robot 200 to encourage the evacuation of customers 302 to 305. This can be used to prioritize the evacuation of customer 302. Alternatively, the robot 200 may be moved to encourage the evacuation of store employee 301, who is not the thief 306.

[0070] Furthermore, in the monitoring system 1, if there are multiple robots 200, the processing unit 101 switches the operating mode of two or more of the multiple robots 200, or all of them, to the second mode. Then, the processing unit 101 can control the movement of a selected robot 200 from among the multiple (two or more) robots 200 that have been switched to the second mode, so that it performs an action to encourage the evacuation of customers 302 to 305. That is, for example, as shown in Figure 11B, the selected robot 200-2 is controlled to move so that it performs an action to encourage customer 302 to evacuate. The other robot 200-1 is then controlled to move so that it prevents the escape of the thief 306 and secures him.

[0071] Furthermore, the processing unit 101 may control devices such as automatic doors 403 located along the evacuation route for customers 302 to 305 in the security area ArX, and perform actions to assist in the evacuation of customers 302 to 305. That is, for example, as shown in Figure 11B, the automatic door 403 may be opened when customer 302 evacuates. Note that the automatic door 403 is described as an example of a device located along the evacuation route for customers 302 to 305. However, it is not limited to this, and other electrically powered moving devices such as elevators and escalators may also be used, and it is conceivable to assist in the evacuation of customers 302 to 305 by controlling the drive of these devices in the direction of evacuation for customers 302 to 305.

[0072] (Process to prevent the intruder from entering) Finally, we will explain the process of preventing the intruder 306 from entering the store 400 by moving the robot 200, that is, the process of preventing entry. Suppose the processing unit 101 of the monitoring device 100 has decided to perform the process of preventing the intruder 306 from entering when it has switched the robot 200 to the second mode (that is, when it has determined that there is an emergency). That is, the processing unit 101 may detect the intruder 306 based on the image captured by the monitoring camera 103A-5 (see Figure 8A) which is located outside the robot 200's movable area ArM in the store 400, that is, in the outdoor area ArOUT. In this case, the processing unit 101 may command the control unit 201 of the robot 200 to move in a way that prevents the intruder 306 from entering the store 400 (indoor area ArIN), such as blocking the automatic door 403.

[0073] Furthermore, the processing unit 101 may be configured to control devices such as automatic doors 403 in the path through which the intruder 306 could enter the room, thereby preventing the intruder 306 from entering. While automatic doors 403 are used as an example of devices in the path through which the intruder 306 could enter, the device is not limited to this; for example, elevators, escalators, etc., could also be used to prevent the intruder 306 from entering by controlling their stopping.

[0074] [Summary of the First Embodiment] As described above, in the monitoring system 1 according to this first embodiment, when the processing unit 101 of the monitoring device 100 detects a thief 306 in the security area ArX, it switches the operating mode of the robot 200 from the first mode to the second mode. As a result, even if there are only a few employees in the store 400, for example, the robot 200 will deal with the thief 306, thereby reducing the damage caused by the thief 306.

[0075] In particular, when the robot 200 is switched to the second mode, it performs actions to prevent the thief 306 from approaching the customers 302-305, that is, it moves to intercept the thief 306 and the customers 302-305. This allows the safety of the customers 302-305 to be prioritized.

[0076] Furthermore, the robot 200, when switched to the second mode, performs an action to encourage customers 302-305 to evacuate from the robot 200's movable area ArM, that is, the robot 200 moves to evacuate customers 302-305 from their evacuation route in the store 400. This allows for prioritizing the evacuation of customers 302-305.

[0077] Furthermore, the robot 200, when switched to the second mode, performs actions to prevent the thief 306 from escaping from the robot 200's movable area ArM, that is, the robot 200 moves to position itself in the thief 306's escape route within the store 400. This makes it possible to apprehend the thief 306.

[0078] <Second Embodiment> Next, a second embodiment, which is a modified version of the first embodiment, will be described using Figures 13A to 15. Figure 13A is a block diagram showing the configuration of a monitoring device according to the second embodiment. Figure 13B is a block diagram showing the configuration of a robot according to the second embodiment. Figure 14 is a flowchart showing the processing of the monitoring system when a thief is detected according to the second embodiment. Figure 15 is a flowchart showing the processing of the second mode in the robot according to the second embodiment. In this description of the second embodiment, the same reference numerals are used for parts similar to those in the first embodiment, and their explanations are omitted.

[0079] In the monitoring system 1 according to this second embodiment, as shown in Figure 13A, the monitoring device 100 is equipped with an output unit 107, and this output unit 107 has a speaker 107B as an audio output unit. Also, in this monitoring system 1, as shown in Figure 13B, the robot 200 has a speaker 207B as an audio output unit in its output unit 207. These speakers 107B and 207B are used to output sound to the thief 306, thereby persuading or warning the thief 306 by sound. Note that speaker 107B may be, for example, a speaker installed on the ceiling of the store 400, and may be a speaker that plays music or announcements during normal business hours.

[0080] (Processing flow of the monitoring system when a thief is detected according to the second embodiment) First, in the second embodiment, the processing flow of the monitoring system 1 when the monitoring device 100 detects a thief 306 (see S106) will be explained with reference to Figure 14.

[0081] Similar to the first embodiment, when the monitoring device 100 detects the thief 306 (Yes in S104), the system proceeds to execute the countermeasures against the thief 306 (S106). Then, as shown in Figure 14, the overall processing of the monitoring system 1 is as follows: First, the processing unit 101 of the monitoring device 100 commands the control unit 201 of the robot 200, and the robot 200 transitions from the first mode to the second mode. This causes the robot 200 to move into the field of view of the monitoring camera 103A that shows the thief 306.

[0082] Next, the processing unit 101 of the monitoring device 100 acquires the layout of the store 400 (S111). Then, the processing unit 101 detects and acquires the positions of the thief 306, the store clerk 301, and the customers 302-305 by performing image recognition on the video captured by the multiple monitoring cameras 103A (S112). Then, it determines the position where the robot 200 will deal with the thief 306 and instructs the robot 200 to move to that position (S123). In addition, in this second embodiment, it also calculates and determines whether or not to speak to the thief 306 from the speaker 107B and / or the speaker 207B of the robot 200, and outputs an instruction to the robot 200 accordingly.

[0083] Then, in the monitoring device 100, the processing unit 101 determines whether or not to speak to the thief 306 using the speaker 107B (S124). If it does not speak (No in S124), it returns to step S112. If it does speak to the thief 306 using the speaker 107B (Yes in S124), it speaks into the store 400 with the content determined by the processing unit 101 (S125), and returns to step S112.

[0084] Meanwhile, in step S123, the robot 200 receives a command from the processing unit 101 of the monitoring device 100 and moves to the position calculated by the monitoring device 100 according to that command (S212). Then, according to the content of the command from the processing unit 101, the robot 200 determines whether or not to speak to the thief 306 using its speaker 207B (S213). If it does not speak (No in S213), it returns to step S112. If it does speak to the thief 306 using the robot 200's speaker 207B (Yes in S213), it speaks to the thief 306 according to the content determined by the processing unit 101 (S214) and returns to step S112.

[0085] (Processing of the second mode in the robot in the second embodiment) Next, the processing of the robot 200 in the second mode in the second embodiment will be explained using Figure 15. As shown in Figure 15, first, the control unit 201 of the robot 200 receives a command from the monitoring device 100 to transition the robot 200 to the second mode and to move to the field of view of the monitoring camera 103A that shows the thief 306 (see S113) (S221). Then, if the control unit 201 is working in the first mode, it interrupts that work (serving) (S222). The control unit 201 then transitions the operating mode of the robot 200 from the first mode to the second mode and performs the process of moving to the field of view of the monitoring camera 103A instructed by the processing unit 101 of the monitoring device 100 until it arrives at that location (No. in S223 and S224). After that, once the robot 200 arrives at that location, it proceeds to the next process (Yes in S224). This corresponds to the process of step S211 as explained in Figure 14.

[0086] Next, as described above, when the robot 200 arrives at the designated field of view of the surveillance camera 103A, the control unit 201 moves to the position indicated by the processing unit 101 of the surveillance device 100 (S225), that is, it actually performs the process of dealing with the thief 306. This movement process corresponds to the process of step S212 described in Figure 14.

[0087] In this second embodiment, the control unit 201 of the robot 200 determines whether or not there is an instruction to speak from the processing unit 101 of the monitoring device 100 (S230). This determination corresponds to the process in step S213 as described in Figure 14. If there is an instruction to speak from the processing unit 101 of the monitoring device 100 (Yes in S230), the robot plays the voice data received from the processing unit 101 of the monitoring device 100, that is, speaks to the thief 306 (S231), and proceeds to step S226. This determination corresponds to the process in step S214 as described in Figure 14. If there is no instruction to speak from the processing unit 101 of the monitoring device 100 (No in S230), the robot proceeds directly to step S226. Specific details of this speaking to the thief 306 will be described later.

[0088] The process is then repeated until the processing unit 101 of the monitoring device 100 issues an instruction to deactivate the second mode (No. in S228). That is, when the processing unit 101 of the monitoring device 100 determines that it has finished dealing with the thief 306, it instructs the control unit 201 of the robot 200 to deactivate the second mode, i.e., to switch from the second mode to the first mode (Yes in S228). Also, when the store clerk 301 determines that it has finished dealing with the thief 306, it inputs to deactivate the second mode using the operation unit 203D of the robot 200 (Yes in S226). Then, the control unit 201 of the robot 200 sends a request to the processing unit 101 of the monitoring device 100 to deactivate the second mode (S227). In response, the processing unit 101 of the monitoring device 100 determines that the second mode has been deactivated, and instructs the control unit 201 of the robot 200 to deactivate the second mode, that is, to switch from the second mode to the first mode (Yes in S228). As a result, the control unit 201 of the robot 200 switches the operating mode to the first mode (S229), and thus the processing of the second mode is completed.

[0089] Also, similar to the first embodiment, if there are multiple robots 200 in the monitoring system 1, the processing unit 101 of the monitoring device 100 commands each control unit 201 of the multiple robots 200 to switch to the first mode all at once. Furthermore, for example, if a store employee 301 makes an input to deactivate the second mode, the operation is performed on one of the multiple robots 200. In response, the processing unit 101 of the monitoring device 100 commands each control unit 201 of the multiple robots 200 to switch to the first mode all at once.

[0090] [Specific Examples of Handling Thieves] Next, a specific example of outputting sound to the thief 306 via the speaker 107B of the monitoring device 100 (speaker of the store 400) or the speaker 207B of the robot 200 will be explained.

[0091] (Process to persuade or warn the thief) Suppose the processing unit 101 of the monitoring device 100 determines that when the robot 200 is switched to the second mode (i.e., when it determines that an emergency has occurred), it will perform the process of outputting sound to the thief 306 using speaker 107B or speaker 207B. Then, the processing unit 101 outputs sound to the thief 306 using speaker 107B or speaker 207B. In this case, when outputting sound from speaker 107B, speaker 107B will be directed towards the thief 306 in the form of a so-called in-store announcement. Also, when outputting sound from the robot 200's speaker 207B, as described above, the robot 200 will be moved so that it is within the field of view of the monitoring camera 103A that shows the thief 306, and then speaker 207B will be directed towards the thief 306.

[0092] The content that these speakers 107B or 207B emit towards the thief 306 is likely to first output a message persuading the thief 306 not to commit the crime. The processing unit 101 obtains information about the thief 306 from the database information recorded in the HDD 102c or via the network unit 105 (i.e., the external network), and generates persuasive content as voice data based on the obtained information. By emitting this message, the thief 306 can be persuaded not to commit the crime, thereby reducing the damage caused by the thief 306.

[0093] Furthermore, the processing unit 101 retrieves information about the sentence the thief 306 would receive if he committed the crime, either from the database information recorded on the HDD 102c or via the network unit 105 (i.e., the external network), and generates voice data containing the retrieved sentence. This voice notification warns the thief 306 of the risks of committing the crime and, in a broader sense, persuades him not to commit the crime, thereby reducing the damage caused by the thief 306.

[0094] Furthermore, in the monitoring system 1, if there are multiple robots 200, the processing unit 101 switches the operating mode of two or more of the multiple robots 200, or all of them, to the second mode. Then, the processing unit 101 can output sound to the thief 306 through the speaker 207B installed in a robot 200 selected from among the multiple robots 200 that have been switched to the second mode. In other words, the processing unit 101 can use the robot 200 that has moved toward the thief 306, or in other words, the robot 200 closest to the thief 306, to deliver persuasion or warning.

[0095] Furthermore, the processing unit 101 may determine the language used by the thief 306 from, for example, the video footage of the thief 306 captured by the camera 203A of the robot 200, or from the voice of the thief 306 acquired by the microphone 203B. In other words, the language used by the thief 306 may be determined from the information in the database recorded on the HDD 102c, or from the information acquired via the network unit 105 (i.e., the external network). When the processing unit 101 generates audio data containing persuasive or warning messages, it may translate the audio data into that language so that the thief 306 can understand it. This makes it possible to persuade the thief 306 not to commit the crime and reduce the damage caused by the thief 306.

[0096] [Summary of the Second Embodiment] As described above, in the monitoring system 1 according to this second embodiment, when the processing unit 101 of the monitoring device 100 detects a thief 306 in the security area ArX, it outputs a sound to the thief 306 via speaker 107B or speaker 207B. This makes it possible to persuade the thief 306 not to commit the crime and reduce the damage caused by the thief 306.

[0097] Furthermore, the configuration, operation, and effects of the second embodiment described above are the same as those of the first embodiment, so their description will be omitted.

[0098] <Possibility of other embodiments> In the first and second embodiments described above, a robot 200 that performs services such as serving food was described as an example of a mobile device, but it is not limited to this. For example, any mobile device that performs the first mode of work, such as a mobile industrial robot, cleaning robot, transport robot, or drone, may be used.

[0099] Furthermore, the moving device may be a robot used to manufacture goods (an industrial robot). In this case, for example, a robot that transports parts necessary for manufacturing goods, or a robot that transports finished goods, are possible. In other words, a manufacturing method can be carried out using a robot in the first mode to manufacture goods. Industrial robots can also come in various forms, such as vertical articulated robots, parallel link robots, and linear articulated robots.

[0100] Furthermore, although the monitoring device 100 was described as being installed in the store 400 in the first and second embodiments, it is not limited to this. For example, the monitoring device 100 may be a server device located in a different location from the store 400 via the internet or the like. Also, the configuration of the monitoring device 100 does not need to consist of all the configurations shown in Figures 2A and 2B, and each configuration may be located in multiple different locations. For example, the processing unit 101 may be located in the store 400, and the storage unit 102 may be located in a different location from the store 400 as an external server or cloud server, etc.

[0101] Furthermore, although the second embodiment described a voice directed at the thief 306, it is not limited to this, and could also be a voice directed at the store clerk 301 or customers 302-305 to evacuate.

[0102] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., ASIC) that implements one or more functions.

[0103] This disclosure can be used in a method for controlling a mobile device, a control system for a mobile device, an information processing method, an information processing device, a mobile device, a method for manufacturing an article, and a recording medium.

[0104] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are attached to make the scope of the invention public.

[0105] This application claims priority based on Japanese Patent Application No. 2024-221170, filed on 17 December 2024, and all of its contents are incorporated herein by reference.

[0106] 1... Surveillance system (control system for mobile devices) / 100... Surveillance device (information processing device) / 101... Processing unit / 103A... Surveillance camera (imaging device) / 107B... Speaker (audio output unit) / 200... Robot (mobile device) / 201... Control unit / 203A... Camera (imaging device) / 203B... Microphone (audio input unit) / 207B... Speaker (audio output unit) / 301... Shop staff (persons other than the target) / 302-305... Customers (persons other than the target) / 306... Thief (target) / 400W... Wall (structure) / 401a-401o... Seats (structure) / 403... Automatic door (device) / ArIN... Indoor area (room) / ArM... Mobility area / ArX... Security area (security area)

Claims

1. A control method for controlling a mobile device that is capable of performing a first mode of operation and is movable, wherein the mobile device is capable of performing a second mode of operation that deals with the appearance of a person who meets predetermined conditions, and the control unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the person.

2. The method for controlling a mobile device according to claim 1, wherein when the processing unit switches the mobile device to the second mode, the mobile device controls the mobile device to perform an action that prevents the target person from approaching other persons.

3. The method for controlling a mobile device according to claim 2, wherein the processing unit, upon detecting the target person, switches the operating mode of a plurality of mobile devices from the first mode to the second mode, and controls a mobile device selected from the plurality of mobile devices switched to the second mode to perform an operation that prevents the target person from approaching other persons.

4. The method for controlling a mobile device according to any one of claims 1 to 3, wherein when the mobile device is switched to the second mode, the processing unit controls the mobile device to perform an action to prompt the evacuation of persons other than the target person from the mobile area of ​​the mobile device.

5. The method for controlling a mobile device according to claim 4, wherein the processing unit, upon detecting the target person, switches the operating mode of the plurality of mobile devices from the first mode to the second mode, and controls the mobile device selected from the plurality of mobile devices switched to the second mode to perform an action to encourage the evacuation of persons other than the target person.

6. The method for controlling a mobile device according to any one of claims 1 to 5, wherein when the mobile device is switched to the second mode, the processing unit controls a device located on a route from which persons other than the target person can evacuate, thereby assisting the evacuation of persons other than the target person.

7. The method for controlling a mobile device according to any one of claims 1 to 6, wherein the processing unit controls the mobile device to perform an action that prevents the subject from escaping from the mobile area of ​​the mobile device when the mobile device is switched to the second mode.

8. The method for controlling a moving device according to claim 7, wherein the processing unit controls the moving device to move the target person between the structure and the structure in accordance with the positional information of the structure in the movable area.

9. The method for controlling a mobile device according to claim 7 or 8, wherein the processing unit, upon detecting the target person, switches the operating mode of the plurality of mobile devices from the first mode to the second mode, and controls a mobile device selected from the plurality of mobile devices switched to the second mode to perform an action that prevents the target person from escaping.

10. The method for controlling a mobile device according to claim 9, wherein the processing unit controls two or more mobile devices selected from the plurality of mobile devices switched to the second mode to coordinate and perform an action to prevent the escape of the target person.

11. The method for controlling a mobile device according to claim 10, wherein when the processing unit detects multiple subjects who meet predetermined conditions, it controls two or more selected mobile devices to work in conjunction to prevent the escape of the subjects.

12. The method for controlling a mobile device according to any one of claims 1 to 11, wherein when the mobile device is switched to the second mode, the processing unit controls a device in a path from which the subject can escape, thereby preventing the subject from escaping.

13. The method for controlling a mobile device according to any one of claims 1 to 12, wherein the processing unit outputs sound to the target person via the sound output unit when the mobile device is switched to the second mode.

14. The control method for a mobile device according to claim 13, wherein the processing unit outputs a voice message to the target person to persuade them not to commit the crime.

15. The control method for a mobile device according to claim 14, wherein the processing unit acquires information about the subject via an external network and generates persuasive content based on the acquired information.

16. The control method for a mobile device according to claim 14 or 15, wherein the processing unit determines the sentence to be imposed if the subject commits a crime and generates content to convey the determined sentence.

17. The method for controlling a mobile device according to any one of claims 13 to 16, wherein the processing unit, when it detects the target person, switches the operating mode of the plurality of mobile devices from the first mode to the second mode, and outputs sound to the target person using the sound output unit provided in the mobile device selected from among the plurality of mobile devices that have been switched to the second mode.

18. The method for controlling a mobile device according to any one of claims 1 to 17, wherein the processing unit detects the target person by comparing the image captured by the imaging device located in the security area with information in a database.

19. The control method for a mobile device according to claim 18, wherein the processing unit image-recognizes the appearance of the subject from the video and detects the subject by comparing it with the information in the database.

20. The method for controlling a mobile device according to claim 19, wherein the information in the database includes information on the appearance of the suspect released by the police.

21. The control method for a mobile device according to claim 18, wherein the processing unit image-recognizes the movements of the subject from the video and detects the subject by comparing it with the information in the database.

22. The method for controlling a mobile device according to claim 21, wherein the information in the database includes a learning model for image recognition of the actions of criminals.

23. The method for controlling a mobile device according to any one of claims 18 to 22, wherein the processing unit image-recognizes the movements of persons surrounding the subject from the video and detects the subject by comparing them with information in the database.

24. The method for controlling a mobile device according to claim 23, wherein the information in the database includes a learning model for image recognition of the actions of persons in the vicinity of a criminal.

25. The control method for a mobile device according to any one of claims 18 to 24, wherein the information in the database is obtained via an external network.

26. A method for controlling a moving device according to any one of claims 18 to 25, wherein a plurality of imaging devices are installed in the security area, and the processing unit controls the moving device, which has been switched to the second mode, to move into the field of view of one of the plurality of imaging devices in which the subject is within the field of view.

27. A method for controlling a mobile device according to any one of claims 18 to 26, wherein a plurality of imaging devices are installed in the security area, and at least one of the imaging devices is also installed outside the room in the security area where the mobile device is located.

28. The method for controlling a mobile device according to claim 27, wherein when the processing unit detects the subject based on the image captured by the imaging device located outside the room, it switches the mobile device to the second mode and controls the mobile device to perform an operation to prevent the subject from entering the room.

29. The method for controlling a mobile device according to any one of claims 18 to 28, wherein the imaging device is a surveillance camera installed above the security area.

30. The method for controlling a mobile device according to any one of claims 18 to 28, wherein the imaging device is a camera installed on the mobile device.

31. The control method for a mobile device according to any one of claims 1 to 30, wherein the processing unit detects the target person by comparing the voice acquired by the voice input unit located in the security area with information in a database.

32. The control method for a mobile device according to claim 31, wherein the processing unit performs speech recognition on the audio to identify the subject's statements and compares them with the information in the database to detect the subject.

33. The method for controlling a mobile device according to claim 32, wherein the information in the database includes a learning model for speech recognition of the statements of criminals.

34. The method for controlling a mobile device according to any one of claims 31 to 33, wherein the processing unit performs voice recognition on the audio to identify statements made by persons in the vicinity of the target person, and detects the target person by comparing them with information in the database.

35. The method for controlling a mobile device according to claim 34, wherein the information in the database includes a learning model for speech recognition of speeches made by persons in the vicinity of a criminal.

36. The control method for a mobile device according to any one of claims 31 to 35, wherein the information in the database is obtained via an external network.

37. The method for controlling a mobile device according to any one of claims 31 to 36, wherein the audio input unit is a microphone installed on the mobile device.

38. The method for controlling a mobile device according to any one of claims 1 to 37, wherein the processing unit detects a candidate who may be the target person in the security area, and switches the operating mode of the mobile device from the first mode to the second mode.

39. The method for controlling a mobile device according to claim 38, wherein the processing unit controls the mobile device to move near the candidate when the mobile device is switched to the second mode.

40. The method for controlling a mobile device according to any one of claims 1 to 39, wherein the mobile device is a robot that performs work on a person other than the target person in the first mode.

41. A control system for a mobile device comprising: a mobile device that is capable of performing a first mode for performing work and a second mode for performing actions in response to the appearance of a person who meets predetermined conditions; and an information processing device having a processing unit for controlling the mobile device, wherein the processing unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the person.

42. The control system for a mobile device according to claim 41, wherein the mobile device is a robot that manufactures articles as an operation in the first mode.

43. An information processing method for processing information using an information processing device having a processing unit that controls a mobile device capable of performing a first mode of work and being mobile, wherein the mobile device is capable of performing a second mode of operation that deals with the appearance of a subject that meets predetermined conditions, and the processing unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the subject.

44. An information processing device having a processing unit that controls a mobile device capable of performing a first mode of work and being movable, wherein the mobile device is capable of performing a second mode of action in response to the appearance of a target person that meets predetermined conditions, and the processing unit switches the operating mode of the mobile device from the first mode to the second mode when it detects the target person.

45. A mobile device having a control unit, which is capable of performing a first mode of work and is mobile, wherein the control unit is capable of performing a second mode of action to deal with the appearance of a person who meets predetermined conditions, and when the person is detected, the mobile device switches the operating mode from the first mode to the second mode.

46. ​​A method for manufacturing an article, comprising manufacturing an article using the control system for a mobile device described in claim 42.

47. A computer-readable recording medium that stores a program for causing a computer to execute the control method of a mobile device according to any one of claims 1 to 40.

Images