Monitoring system, monitoring method, and program

Abstract

The present invention makes it possible to reduce the monitoring workload of an operator monitoring a monitoring target. This monitoring system includes: a video acquisition means for acquiring video from a camera associated with a monitoring target; an audio acquisition means for acquiring audio associated with the monitoring target; a determination means for determining, on the basis of the acquired video and audio, whether a prescribed event has occurred; an identification means for identifying the type of the prescribed event determined to have occurred; a selection means for selecting notification content according to the identified type of the prescribed event; and a notification means for converting the selected notification content into text and issuing a notification.

Description

Monitoring System, Monitoring Method, and Program

[0001] The present disclosure relates to a monitoring system, a monitoring method, and a program.

[0002] As a related technology, Patent Document 1 discloses an emergency notification system for a vehicle. In the emergency notification system described in Patent Document 1, when an abnormality such as an accident occurs, the vehicle transmits an emergency notification to an operation center. The vehicle also transmits position information and inside / outside vehicle information to the operation center. The inside / outside vehicle information includes information on the number of occupants of the vehicle, the gender of each occupant, classification information of the facial expressions of the occupants, the degree of injury of the occupants, audio inside the vehicle, and video inside and outside the vehicle. An operator at the operation center who has received the emergency notification determines the situation of the accident based on the inside / outside vehicle information. The operator notifies the occurred abnormality to various emergency organizations such as the police, ambulance services, or roadside services as necessary.

[0003] Japanese Unexamined Patent Publication No. 2017-117194

[0004] In Patent Document 1, when an abnormality occurs, inside / outside vehicle information including numerical values, audio, and video is provided from the vehicle to the operation center. For this reason, an operator at the operation center can accurately grasp the situation at the site where the abnormality has occurred. However, in Patent Document 1, the operator needs to judge the situation from information such as numerical values, audio, or video included in the inside / outside vehicle information. Therefore, Patent Document 1 has a problem that the monitoring load on the operator at the operation center is high.

[0005] One exemplary object of the present disclosure is to provide a monitoring system, a monitoring method, and a program that can reduce the monitoring load on an operator who monitors a monitoring target.

[0006] A monitoring system according to a first aspect of the present disclosure includes: video acquisition means for acquiring video from a camera related to a target to be monitored; audio acquisition means for acquiring audio related to the target to be monitored; determination means for determining whether a predetermined event has occurred based on the acquired video and audio; determination means for determining the type of the predetermined event that has been determined to have occurred; selection means for selecting notification content according to the determined type of predetermined event; and notification means for converting the selected notification content into text and notifying the recipient.

[0007] A monitoring method according to a second aspect of this disclosure includes acquiring video from a camera related to the object being monitored, acquiring audio related to the object being monitored, determining whether a predetermined event has occurred based on the acquired video and audio, determining the type of predetermined event that has been determined to have occurred, selecting notification content according to the determined type of predetermined event, and notifying the selected notification content in text form.

[0008] A program according to a third aspect of this disclosure acquires video from a camera related to a monitored object, acquires audio related to the monitored object, determines whether a predetermined event has occurred based on the acquired video and audio, determines the type of predetermined event that has been determined to have occurred, selects notification content according to the determined type of predetermined event, and causes a computer to execute a process of converting the selected notification content into text and notifying the computer.

[0009] The monitoring system, monitoring method, and program related to this disclosure can reduce the monitoring burden on operators who monitor the monitored targets.

[0010] This is a block diagram showing an example configuration of the monitoring system related to this disclosure. This is a block diagram showing an example configuration of the monitoring system related to this disclosure. This is a block diagram showing an example configuration of a mobile unit. This is a block diagram showing an example configuration of a monitoring device. This is a diagram showing an example of a notification displayed on the monitoring screen. This is a flowchart showing the operating procedure of the monitoring device. This is a block diagram showing an example configuration of a computer device.

[0011] Prior to describing embodiments of this disclosure, an overview of this disclosure will be provided. Figure 1 is a block diagram showing an example configuration of a monitoring system according to this disclosure. The monitoring system 10 includes video acquisition means 11, audio acquisition means 12, judgment means 13, decision means 14, selection means 15, and notification means 16.

[0012] The video acquisition means 11 acquires video from a camera related to the monitored object. The audio acquisition means 12 acquires audio related to the monitored object. The determination means 13 determines whether a predetermined event has occurred based on the acquired video and audio. The decision means 14 determines the type of predetermined event that has been determined to have occurred. The selection means 15 selects notification content according to the determined type of predetermined event. The notification means 16 converts the selected notification content into text and notifies the recipient.

[0013] In this disclosure, the notification means 16 verbalizes or texts the notification content according to the type of predetermined event that has occurred. By looking at the text of the notification content, the operator performing the monitoring of the monitored target can easily find out what kind of event has occurred in the monitored target. For this reason, the monitoring system 10 can reduce the monitoring burden on the operator who is monitoring the monitored target.

[0014] The embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. Furthermore, in each drawing, the same elements and similar elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0015] Figure 2 is a block diagram showing an example configuration of a monitoring system according to the present disclosure. An embodiment of the present disclosure will be described with reference to Figure 2. The monitoring system 100 shown in Figure 2 includes a monitoring device 110 and a monitoring screen display device 130. In the following embodiment, an example of remotely monitoring a mobile body 200 using the monitoring device 110 will be described. However, the object of monitoring is not limited to the mobile body 200.

[0016] The monitoring device 110 is connected to the mobile body 200 via the network 150. The network 150 includes, for example, a wireless communication network using a communication line standard such as Long Term Evolution (LTE). The network 150 may also include a wireless communication network such as Wi-Fi® or a fifth-generation mobile communication system. The mobile body 200 is configured as, for example, a land vehicle such as a car, bus, taxi, or truck. The monitoring device 110 corresponds to the monitoring system 10 shown in Figure 1.

[0017] The mobile vehicle 200 is configured to enable autonomous operation based on information from sensors mounted on the mobile vehicle. The mobile vehicle 200 may be configured to allow switching between automatic operation and manual operation by an operator inside the vehicle. The mobile vehicle 200 may be switched from manual operation to automatic operation, or from automatic operation to manual operation, in response to instructions transmitted from, for example, a monitoring device 110. The mobile vehicle 200 may be a train, a ship, or an aircraft, or a mobile robot such as an Automated Guided Vehicle (AGV). The mobile vehicle 200 does not necessarily need to be configured to enable automatic operation.

[0018] Figure 3 is a block diagram showing an example configuration of the mobile unit 200. The mobile unit 200 includes a camera 201, a microphone 202, a vehicle sensor 203, a vehicle control electronic control unit (ECU) 204, an autonomous driving ECU 205, and a communication device 206. In the mobile unit 200, these components are configured to communicate with each other via a network such as an in-vehicle Local Area Network (LAN) or a Controller Area Network (CAN).

[0019] Camera 201 captures images of the area around the mobile body 200. Camera 201 may include multiple cameras that capture images of, for example, the front, rear, right side, and left side of the vehicle. Camera 201 may also include a camera that captures images of the interior of the mobile body 200. Microphone 202 acquires sound from the mobile body 200. Microphone 202 may include a microphone that acquires sounds from outside the vehicle and a microphone that acquires sounds from inside the vehicle. The mobile body 200 may include surrounding monitoring sensors such as radar and Light Detection and Ranging (LiDAR).

[0020] The vehicle sensor 203 is a sensor for detecting various states of the moving body 200. The vehicle sensor 203 includes, for example, a vehicle speed sensor for detecting vehicle speed, a steering sensor for detecting steering angle, an accelerator pedal position sensor for detecting accelerator pedal opening, and a brake pedal force sensor for detecting brake pedal depression.

[0021] The vehicle control ECU 204 is an electronic control unit that performs functions such as driving control of the mobile unit 200. Generally, an electronic control unit has a processor, memory, input / output (I / O) circuits, and a bus connecting these. Based on sensor information output by the vehicle sensor 203, the vehicle control ECU 204 performs various controls, such as controlling the fuel injection amount, controlling the engine ignition timing, and controlling the amount of power steering assist.

[0022] The autonomous driving ECU 205 is an electronic control unit that controls the autonomous driving of the mobile unit 200. The autonomous driving ECU 205 acquires video and sensor information from the camera 201 and vehicle sensor 203, and controls the autonomous driving of the mobile unit 200 based on the acquired video and sensor information.

[0023] The communication device 206 is configured as a device for wireless communication between the mobile unit 200 and the network 150. The hardware configuration of the communication device 206 includes a wireless communication antenna, a transmitter, and a receiver. The communication device 206 also has a processor, memory, I / O circuits, and a bus connecting these. The functions of each part within the communication device 206 are realized, for example, by executing a control program stored in memory using the processor.

[0024] The communication device 206 transmits image data or video data acquired by the camera 201 to the monitoring device 110 via the network 150. The communication device 206 also transmits audio data acquired by the microphone 202 to the monitoring device 110 via the network 150. The communication device 206 may also acquire sensor information such as vehicle speed information from the vehicle sensor 203 and transmit the acquired sensor information to the monitoring device 110 via the network 150.

[0025] The communication device 206 may receive remote control information, i.e., information for remotely controlling the mobile unit 200, from the monitoring device 110. The remote control information includes, for example, information indicating the accelerator opening, the amount of steering wheel operation, and the amount the brake pedal is pressed. When the communication device 206 receives remote control information, it transmits the received remote control information to the vehicle control ECU 204 via the in-vehicle LAN or the like. The vehicle control ECU 204 controls the mobile unit 200 based on the received remote control information.

[0026] The monitoring device 110 is a device used to remotely monitor the mobile unit 200. The monitoring device 110 may also be capable of remotely controlling the mobile unit 200. The monitoring screen display device 130 is a display device for showing information used to monitor the mobile unit 200 to an operator performing monitoring duties. The monitoring device 110 causes the monitoring screen used to monitor the mobile unit 200 to be displayed on the monitoring screen display device 130.

[0027] The monitoring screen includes, for example, a video display area that displays video transmitted from the mobile device 200. The monitoring screen display device 130 may also display vehicle information, such as vehicle speed, transmitted from the mobile device 200, on the monitoring screen. The monitoring screen display device 130 does not necessarily have to be a device independent of the monitoring device 110, and may be part of the monitoring device 110. The monitoring screen display device 130 is configured as, for example, a liquid crystal display device.

[0028] Figure 4 is a block diagram showing an example configuration of the monitoring device 110. The monitoring device 110 includes a video acquisition unit 111, an audio acquisition unit 112, a judgment unit 113, a decision unit 114, a selection unit 115, a notification unit 116, a specific unit 117, a suggestion unit 118, a prediction unit 119, and a storage unit 120. The monitoring device 110 is configured, for example, as a device having one or more memories and one or more processors. The monitoring device 110 can be realized by the processor executing processing according to instructions read from the memory. In the following embodiments, "-unit" is also called "-means".

[0029] The video acquisition unit 111 acquires video data from the mobile body 200 via the network 150, which is acquired by the camera 201. The video acquisition unit 111 acquires video data from, for example, the front, rear, right side, and left side of the mobile body. The audio acquisition unit 112 acquires audio data from the mobile body 200 via the network, which is acquired by the microphone 202. The video acquisition unit 111 corresponds to the video acquisition means 11 shown in Figure 1. The audio acquisition unit 112 corresponds to the audio acquisition means 12 shown in Figure 1.

[0030] The determination unit 113 determines, based on the acquired video and audio data, whether a predetermined event has occurred in the mobile body 200. The predetermined event may relate to, for example, an abnormality or danger in the mobile body 200. The predetermined event may also relate to a scene that causes the driver operating the mobile body to raise their attention level. For example, the determination unit 113 determines whether an abnormality or an event that could potentially cause an abnormality has occurred in the mobile body 200. The determination unit 113 may also determine whether a danger or an event that could potentially lead to a dangerous situation has occurred in the mobile body 200.

[0031] The determination unit 113 may, for example, detect abnormal behavior of a person on board the mobile vehicle 200, such as a crew member or passenger, based on the acquired video data and audio data. The determination unit 113 may also detect abnormal conditions of luggage loaded on the mobile vehicle based on the acquired video data and audio data. The determination unit 113 may, for example, determine that an event that could create a dangerous situation has occurred if the mobile vehicle 200 is traveling down an alley without a sidewalk and there are pedestrians in the alley. Alternatively, it may determine that an event that could create a dangerous situation has occurred if the mobile vehicle 200 is traveling in an area with heavy pedestrian traffic. The determination unit 113 may include, for example, an artificial intelligence (AI) model that determines danger or alerts from the input video data and audio data.

[0032] The judgment unit 113 analyzes video data, for example, and recognizes, for example, traffic participants, signal light status, and signs from the video data. Based on the results of the video data analysis, the judgment unit 113 determines whether a predetermined event, such as an abnormality or danger, has occurred to the mobile vehicle 200. For example, the judgment unit 113 analyzes video data and determines that a passenger fell at the bus entrance while the bus was stopped. In that case, the judgment unit 113 determines that an abnormality or danger has occurred to a person on board the mobile vehicle 200.

[0033] Furthermore, the judgment unit 113 analyzes audio data, for example, and recognizes sounds related to a predetermined event from the audio data. The judgment unit 113 also recognizes abnormal sounds occurring inside or outside the mobile vehicle 200 from the audio data. For example, the judgment unit 113 analyzes audio data acquired by a microphone inside the vehicle and determines that a scream has occurred inside the vehicle. In that case, the judgment unit 113 determines that an abnormality or danger has occurred to a person riding in the mobile vehicle 200.

[0034] The determination unit 113 may analyze the video data and recognize that there are many people around the mobile body 200. The determination unit 113 may also analyze the audio data and recognize that there are many voices outside the mobile body 200. In that case, the determination unit 113 may determine that the mobile body 200 is traveling in an area with many pedestrians. If the determination unit 113 determines that a predetermined event such as an abnormality has occurred, it may store the video data and audio data at the time the predetermined event occurred in the storage unit 120. The determination unit 113 corresponds to the determination means 13 shown in Figure 1.

[0035] The determination unit 114 determines the type of predetermined event that has occurred when the judgment unit 113 determines that a predetermined event has occurred in the mobile body 200. For example, if the judgment unit 113 determines that a scream has occurred, the determination unit 114 determines that a scream inside the vehicle is the type of predetermined event. If the judgment unit 113 determines that a passenger has fallen at the entrance or exit of the bus, the determination unit 114 determines that a passenger has fallen is the type of predetermined event. If the judgment unit 113 determines that the mobile body 200 is traveling in an area with many pedestrians, the determination unit 114 determines that traveling in a dangerous area is the type of predetermined event. For example, the determination unit 114 has a table that associates predetermined events determined by the judgment unit 113 with the types of predetermined events. The determination unit 114 uses the table to determine the type of predetermined event that corresponds to the predetermined event determined by the judgment unit 113. The determination unit 114 corresponds to the determination means 14 shown in Figure 1.

[0036] The selection unit 115 selects the notification content to the operator according to the type of predetermined event determined by the decision unit 114. The selection unit 115 selects the notification content according to the type of predetermined event, for example, by using a table that associates the type of predetermined event with the notification content. The selection unit 115 corresponds to the selection means 15 shown in Figure 1.

[0037] The notification unit 116 converts the notification content selected by the selection unit 115 into text and notifies the operator of the notification content. The notification unit 116 also notifies the operator of the notification content by displaying the converted text notification content on the monitoring screen display device 130. The notification unit 116 corresponds to the notification means 16 shown in Figure 1.

[0038] If the determination unit 113 determines that a predetermined event has occurred, the identification unit 117 identifies the cause of the predetermined event. For example, the identification unit 117 analyzes video data and audio data from a short time before the predetermined event is determined to have occurred to identify the event that caused the predetermined event. In addition to video data and audio data, the identification unit 117 may also use vehicle sensor information received from the mobile body 200 and video from a roadside camera to identify the cause of the predetermined event. If the identification unit 117 identifies the cause of the predetermined event, the notification unit 116 may notify the operator of the identified cause of the predetermined event in text format.

[0039] The proposal unit 118 proposes a method for responding to a predetermined event when the judgment unit 113 determines that a predetermined event has occurred. For example, if the operator responds to a predetermined event that has occurred, the proposal unit 118 records the operator's response as a response history. The proposal unit 118 may also record a response history for each type of predetermined event determined by the decision unit 114. The proposal unit 118 obtains responses to predetermined events similar to the predetermined event that the judgment unit 113 determined to have occurred from the response history. The proposal unit 118 proposes the obtained responses to the operator as a method for responding to the predetermined event that the judgment unit 113 determined to have occurred. The operator can then decide on a method for responding to a newly occurring predetermined event, referring to the responses proposed by the proposal unit 118.

[0040] The prediction unit 119 may predict a predetermined future event based on video data and audio data. For example, the prediction unit 119 learns video data and audio data from past predetermined event occurrences as past predetermined event occurrence patterns. The predetermined event occurrence patterns may include vehicle information at the time of the predetermined event occurrence, as well as additional data added manually. The prediction unit 119 predicts a future predetermined event if, although it has not yet been determined that a predetermined event has occurred, the situation of the moving object is approaching a situation in the past where a predetermined event was determined to have occurred, and therefore the predetermined event may occur.

[0041] The prediction unit 119 may compare the video and audio data acquired from the mobile body 200 with past predetermined event occurrence patterns and calculate a similarity score to past predetermined event occurrences as a risk score. For example, if the risk score is above a threshold, the prediction unit 119 predicts a future predetermined event. If the prediction unit 119 predicts a predetermined event, the notification unit 116 notifies the operator that a predetermined event has been predicted. In this case, the operator can be aware of the occurrence of the predetermined event before it actually occurs.

[0042] The determination unit 114 may determine the degree of a predetermined event in addition to the type of predetermined event. If the moving body 200 is a bus, the determination unit 114 may determine the degree of the predetermined event depending on whether the event that occurred is a predetermined event that interferes with the operation of the bus. For example, if the predetermined event that occurred is a predetermined event that interferes with the operation of the bus, the determination unit 114 may determine that the degree of the predetermined event is high. If the predetermined event that occurred is a predetermined event that does not interfere with the operation of the bus, the determination unit 114 may determine that the degree of the predetermined event is low.

[0043] The determination unit 114 may determine the severity of a predetermined event depending on whether or not the event is a predetermined event that involves human life. For example, the determination unit 114 may determine that a predetermined event that involves human life is of high severity, and that a predetermined event that does not involve human life is of low severity. The determination unit 114 may, for example, analyze video data and audio data and determine that a predetermined event is of high severity if a passenger has fallen and is not moving, or if a passenger is bleeding.

[0044] If the degree of a predetermined event has been determined in the determination unit 114, the notification unit 116 may select the notification format and notification priority based on the type and degree of the predetermined event that has occurred. The notification unit 116 may, for example, change the font size and color of the text notification content according to the type and degree of the predetermined event. Alternatively, the notification unit 116 may, for example, determine the notification priority according to the type and degree of the predetermined event and change the display format according to the priority. If a predetermined event has occurred in multiple mobile units 200, the notification unit 116 may prioritize notifying the operator of the predetermined event with the highest priority.

[0045] Fig. 5 is a diagram showing an example of a notification displayed on a monitoring screen. In the example of Fig. 5, the vehicle number indicates identification information of the moving object 200. The status indicates whether the vehicle is in automatic driving or manual driving. In addition, the status indicates information such as whether the vehicle is traveling or stopped. The alert indicates information indicating a predetermined event such as an abnormality or danger occurring in the moving object 200. When the determination unit 113 determines that a predetermined event has occurred, the notification unit 116 displays the text-based notification content selected by the selection unit 115 in the alert.

[0046] In the example of Fig. 5, the monitoring screen display device 130 integrates information of a plurality of moving objects 200, and displays a status and an alert for each moving object 200. An operator can grasp the status of each moving object 200 and what kind of abnormality or danger has occurred to the moving object by looking at the monitoring screen. On the monitoring screen, the moving objects 200 may be sorted according to the priority of notifications. When the operator selects a symbol shown in the display field, the monitoring screen display device 130 displays an individual monitoring screen of the selected moving object. The individual monitoring screen includes, for example, an area for displaying video data acquired from the moving object 200 and an area for displaying vehicle information of the moving object 200.

[0047] Next, the operation procedure will be described. Fig. 6 is a flowchart showing the operation procedure of the monitoring device 110. The operation procedure of the monitoring device 110 corresponds to a monitoring method. A video acquisition unit 111 acquires video data captured by a camera 201 from the moving object 200 (step S1). An audio acquisition unit 112 acquires audio data acquired by a microphone from the moving object 200 (step S2). A determination unit 113 determines whether a predetermined event has occurred in the moving object 200 based on the video data acquired in step S1 and the audio data acquired in step S2 (step S3). If it is determined in step S3 that no predetermined event has occurred, the process returns to step S1.

[0048] If it is determined that a predetermined event has occurred in step S3, the determining unit 114 determines the type of the occurred predetermined event (step S4). The selecting unit 115 selects notification content based on the type of the predetermined event determined in step S4 (step S5). The notifying unit 116 notifies the operator of a text indicating the notification content selected in step S5 (step S6).

[0049] In one embodiment, the determining unit 113 determines occurrence of a predetermined event based on video data and audio data. The determining unit 114 determines the type of the occurred predetermined event. The selecting unit 115 selects notification content according to the type of the predetermined event. When a predetermined event occurs, the notifying unit 116 converts the notification content corresponding to the occurred predetermined event into text and notifies the operator of the text as an alert. The operator can know what kind of event is occurring in the moving object 200 by viewing the text-converted notification content. Therefore, the monitoring apparatus 110 according to one embodiment can reduce the monitoring load on an operator who monitors a monitoring target.

[0050] In one embodiment, the operator can monitor the moving object 200 by checking video of the moving object 200 in which the predetermined event has occurred as necessary. In one embodiment, the operator does not need to continuously watch video of the moving object 200 at all times. Therefore, a single operator can monitor a plurality of moving objects 200.

[0051] In one embodiment, when simultaneously monitoring a plurality of moving objects 200, the operator can easily know which moving object 200 has what kind of event occurred. As necessary, the operator can take an appropriate response according to the content of the occurred event. In one embodiment, the monitoring apparatus 110 not only detects and reports an occurred event, but also enables the operator to take an appropriate response according to the content of the occurred event.

[0052] In this disclosure, the monitoring device 110 may be configured using a computer device or a server device. Figure 7 is a block diagram showing an example configuration of a computer device that can be used as the monitoring device 110. The computer device 500 has a processor 510 such as a CPU (Central Processing Unit), a storage unit 520, a ROM (Read Only Memory) 530, a RAM (Random Access Memory) 540, a communication interface (IF: Interface) 550, and a user interface 560.

[0053] The communication interface 550 is an interface for connecting the computer device 500 to a communication network via wired communication means or wireless communication means. The user interface 560 includes a display unit, such as a display. The user interface 560 also includes input units such as a keyboard, mouse, and touch panel.

[0054] The memory unit 520 is an auxiliary storage device capable of holding various types of data. The memory unit 520 does not necessarily have to be part of the computer device 500; it may be an external storage device or cloud storage connected to the computer device 500 via a network.

[0055] ROM 530 is a non-volatile memory device. For example, a semiconductor memory device such as a relatively small-capacity flash memory is used for ROM 530. The program executed by the processor 510 can be stored in the storage unit 520 or ROM 530. The storage unit 520 or ROM 530 stores various programs, for example, for realizing the functions of each part of the monitoring device 110.

[0056] The program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technologies, Compact Disc (CD), digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes an electrical, optical, acoustic or other form of propagating signal.

[0057] The RAM 540 is a volatile memory device. Various semiconductor memory devices such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory) can be used for the RAM 540. The RAM 540 can be used as an internal buffer for temporarily storing data, etc. The processor 510 loads the program stored in the memory unit 520 or ROM 530 into the RAM 540 and executes it. By executing the program, the functions of each part within the monitoring device 110 can be realized. The processor 510 may have an internal buffer that can temporarily store data, etc.

[0058] In this disclosure, the monitoring device 110 does not necessarily have to be configured as a single computer device. The monitoring device 110 may be configured using a plurality of physically separated devices. For example, the monitoring device 110 may be configured by a plurality of computer devices interconnected via a network. The monitoring device 110 may include, for example, a first device mounted on the mobile body 200 and a second device located outside the mobile body 200. The monitoring device 110 may be mounted on the mobile body 200. In that case, the notification unit 116 may notify the occupant of the mobile body 200 of a predetermined event that has occurred.

[0059] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Any embodiment can be combined with other embodiments as appropriate.

[0060] Each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated with one or more other embodiments, rather than being associated with only one specific embodiment. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0061] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0062] [Note 1] A monitoring system comprising: video acquisition means for acquiring video from a camera related to the subject being monitored; audio acquisition means for acquiring audio related to the subject being monitored; determination means for determining whether a predetermined event has occurred based on the acquired video and audio; determination means for determining the type of predetermined event abnormality that has been determined to have occurred; selection means for selecting notification content according to the determined type of predetermined event; and notification means for converting the selected notification content into text and notifying the recipient.

[0063] [Note 2] The monitoring system according to Note 1, further comprising a means for identifying the cause of the predetermined event that is determined to have occurred, wherein the notification means notifies the identified cause of the predetermined event in text form.

[0064] [Note 3] The monitoring system according to Note 1 or 2, wherein the determination means determines the degree of the predetermined event, and the selection means selects the notification content and the priority of the notification according to the type and degree of the predetermined event.

[0065] [Appendix 4] The monitoring system according to any one of Appendix 1 to 3, further comprising prediction means for predicting a predetermined future event in the monitored object based on the acquired video and audio.

[0066] [Note 5] The monitoring system according to Note 4, wherein the prediction means predicts the future predetermined event from the acquired video and audio using a predetermined event occurrence pattern that includes video and audio from past predetermined event occurrences.

[0067] [Note 6] The notification means is a monitoring system according to any one of the items in Notes 1 to 5, which integrates information on multiple monitored targets and notifies each monitored target of the notification content in text format.

[0068] [Note 7] The monitoring system according to any one of Notes 1 to 6, further comprising a proposal means for proposing a method of responding to a predetermined event that has been determined to have occurred by the determination means, based on the response history of past occurrences of predetermined events.

[0069] [Note 8] The monitoring system according to any one of Notes 1 to 7, wherein the determination means determines that a predetermined event has occurred and saves the video and audio to the storage means.

[0070] [Note 9] The surveillance system according to any one of Notes 1 to 8, wherein the object being monitored is a moving object, the video acquisition means acquires video from a camera mounted on the moving object, and the audio acquisition means acquires audio from a microphone mounted on the moving object.

[0071] [Note 10] The monitoring system according to Note 9, wherein the determination means detects at least one of abnormal human behavior inside the mobile body and abnormality of luggage in the mobile body as the predetermined event.

[0072] [Note 11] A monitoring method which involves acquiring video from a camera related to the subject being monitored, acquiring audio related to the subject being monitored, determining whether a predetermined event has occurred based on the acquired video and audio, determining the type of predetermined event that has been determined to have occurred, selecting notification content according to the determined type of predetermined event, and notifying the selected notification content in text form.

[0073] [Note 12] A program that causes a computer to perform the following processes: acquire video from a camera related to the subject being monitored; acquire audio related to the subject being monitored; determine whether a predetermined event has occurred based on the acquired video and audio; determine the type of predetermined event that has been determined to have occurred; select notification content according to the determined type of predetermined event; and convert the selected notification content into text and notify the computer.

[0074] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 10 that are dependent on Appendice 1 may also be dependent on Appendices 11 and 12 in the same way as in Appendices 2 to 10. Some or all of the elements described in any appendice may be applicable to various hardware, software, recording means, systems, and methods for recording software.

[0075] 10: Monitoring system 11: Video acquisition means 12: Audio acquisition means 13: Judgment means 14: Decision means 15: Selection means 16: Notification means 100: Monitoring system 110: Monitoring device 111: Video acquisition unit 112: Audio acquisition unit 113: Judgment unit 114: Decision unit 115: Selection unit 116: Notification unit 117: Identification unit 118: Proposal unit 119: Prediction unit 120: Storage unit 130: Monitoring screen display device 150: Network 200: Mobile device 201: Camera 202: Microphone 203: Vehicle sensor 204: Vehicle control ECU 205: Autonomous driving ECU 206: Communication device 500: Computer device 510: Processor 520: Memory unit 530: ROM 540: RAM 550: Communication interface 560: User interface

Claims

1. A surveillance system comprising: video acquisition means for acquiring video from a camera related to the subject being monitored; audio acquisition means for acquiring audio related to the subject being monitored; determination means for determining whether a predetermined event has occurred based on the acquired video and audio; determination means for determining the type of the predetermined event that has been determined to have occurred; selection means for selecting notification content according to the determined type of predetermined event; and notification means for converting the selected notification content into text and notifying the system.

2. The monitoring system according to claim 1, further comprising a means for identifying the cause of a predetermined event that has been determined to have occurred, wherein the notification means notifies the identified cause of occurrence in text form.

3. The monitoring system according to claim 1 or 2, wherein the determination means determines the degree of the predetermined event, and the selection means selects the notification content and the priority of the notification according to the type and degree of the predetermined event.

4. The monitoring system according to any one of claims 1 to 3, further comprising prediction means for predicting a predetermined future event in the monitored object based on the acquired video and audio.

5. The monitoring system according to claim 4, wherein the prediction means predicts the future predetermined event from the acquired video and audio using a predetermined event occurrence pattern that includes video and audio from past predetermined event occurrences.

6. The monitoring system according to any one of claims 1 to 5, wherein the notification means integrates information on multiple monitored targets and notifies each monitored target of the notification content in text form.

7. The monitoring system according to any one of claims 1 to 6, further comprising a proposal means for proposing a method of responding to a predetermined event that has been determined to have occurred by the determination means, based on the history of responses to past predetermined events.

8. The monitoring system according to any one of claims 1 to 7, wherein the determination means determines that a predetermined event has occurred and stores the video and audio in the storage means.

9. The monitoring system according to any one of claims 1 to 8, wherein the object to be monitored is a mobile object, the video acquisition means acquires video from a camera mounted on the mobile object, and the audio acquisition means acquires audio from a microphone mounted on the mobile object.

10. The monitoring system according to claim 9, wherein the determination means detects at least one of abnormal human behavior inside the mobile body and abnormality of luggage in the mobile body as the predetermined event.

11. A monitoring method that acquires video from a camera related to the subject being monitored, acquires audio related to the subject being monitored, determines whether a predetermined event has occurred based on the acquired video and audio, determines the type of predetermined event that has been determined to have occurred, selects notification content according to the determined type of predetermined event, and converts the selected notification content into text and notifies the system.

12. A program that causes a computer to perform the following processes: acquire video from a camera related to the subject being monitored, acquire audio related to the subject being monitored, determine whether a predetermined event has occurred based on the acquired video and audio, determine the type of predetermined event that has been determined to have occurred, select a notification content according to the determined type of predetermined event, and convert the selected notification content into text and notify the computer.

Images