Information processing system, information processing method, and program

The information processing system addresses the challenge of understanding camera imaging areas by analyzing images and providing tailored responses, improving situational awareness and response effectiveness.

WO2026240165A1PCT designated stage Publication Date: 2026-11-19X-BRIDGE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/018239
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-05-14
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Conventional monitoring techniques using cameras fail to understand the situation of the imaging area, making it impossible to take appropriate measures.

Method used

An information processing system that includes an image acquisition unit, additional information assigning unit, language model input unit, language model output acquisition unit, and notification control unit, which together analyze camera images and provide targeted responses based on anomaly detection results.

Benefits of technology

Enables effective monitoring and response strategies tailored to the camera's imaging area, enhancing situational awareness and enabling timely and appropriate actions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026018239_19112026_PF_FP_ABST
    Figure JP2026018239_19112026_PF_FP_ABST
Patent Text Reader

Abstract

The present invention addresses the problem of providing monitoring technology or surveillance technology with which it is possible to take measures corresponding to the situation within an imaging region of a camera. In a CPU 11 of a management server 1: an information acquiring unit 31 acquires a captured image captured by a camera 2 together with a camera identifier for identifying the camera 2; an associating unit 33 associates additional information corresponding to the camera identifier with the captured image; a model input / output control unit 35 inputs the captured image, the additional information, and an instruction for designating a predetermined output format to a language model capable of accepting input of at least images and text and outputting at least natural language; the information acquiring unit 31 acquires an abnormality detection result conforming to the designated output format from the language model; and a notification control unit 36 executes control related to notification on the basis of the abnormality detection result. The abovementioned problem is thus resolved.
Need to check novelty before this filing date? Find Prior Art

Description

Information Processing System, Information Processing Method, and Program

[0001] The present invention relates to an information processing system, an information processing method, and a program.

[0002] Conventionally, a technique for monitoring an imaging area using a monitoring camera is known (for example, Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2021-103815

[0004] However, with the conventional technique, since the situation of the imaging area of the camera cannot be understood, it is not possible to take measures corresponding to the situation of the imaging area.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to provide a monitoring technique or a watching technique that can take measures corresponding to the situation of the imaging area of a camera.

[0006] To achieve the above object, one aspect of the present invention provides an information processing system comprising: an image acquisition unit that acquires a captured image captured by a camera together with a camera identifier for identifying the camera; an additional information assigning unit that associates additional information corresponding to the camera identifier with the captured image; a language model input unit that inputs the captured image, the additional information, and an instruction specifying a predetermined output format to a language model capable of receiving at least image and text inputs and outputting at least natural language; a language model output acquisition unit that acquires an abnormality detection result conforming to the output format from the language model; and a notification control unit that executes control related to notification based on the abnormality detection result.

[0007] An information processing method and a program corresponding to the information processing system according to one aspect of the present invention are also provided as an information processing method and a program corresponding to the information processing system according to one aspect of the present invention.

[0008] According to the present invention, it is possible to provide a monitoring technique or a watching technique that can take measures corresponding to the situation of the imaging area of a camera.

[0009] This figure shows an example of the overall configuration of an information processing system according to one embodiment of the present invention. This is a block diagram showing an example of the hardware configuration of the management server constituting the information processing system of Figure 1. This is a functional block diagram showing an example of the functional configuration of the management server. This is a flowchart showing an example of the processing of the management server. This figure shows an example of the application of this service.

[0010] The following description of this embodiment will be made with reference to the drawings. <Information Processing System S> Figure 1 is a diagram showing an example of the overall configuration of an information processing system S according to one embodiment of the present invention. The information processing system S is a system that enables responses according to the conditions of the camera's imaging area, and more specifically, it is a system that enables responses according to the conditions of the imaging area by utilizing a large-scale language model (LLM).

[0011] As shown in Figure 1, the information processing system S is a system configured to include a management server 1, a camera 2, a model provisioning server 3, and user devices 4-1 to 4-n (where n is a natural number of 1 or more) connected via a network N. The network N is, for example, the Internet, a LAN (Local Area Network), a VPN (Virtual Private Network), etc., and its communication method may be wireless communication such as Wi-Fi® or Bluetooth®, or wired communication. In the following, when it is not necessary to explain each of the user devices 4-1 to 4-n individually, they will be collectively referred to as "user device 4".

[0012] [Management Server 1] Management Server 1, which constitutes the information processing system S, is an information processing device that acts as a server for managing the entire information processing system S. Management Server 1 is capable of executing predetermined application programs that make the information processing system S available for use. Management Server 1 is capable of acquiring various types of information transmitted from Camera 2, Model Provision Server 3, User Device 4, and external sources, and performing various types of processing. Management Server 1 is also capable of causing Camera 2, Model Provision Server 3, and User Device 4 to perform various types of processing.

[0013] For example, the management server 1 acquires the captured image taken by camera 2 along with an identifier that identifies camera 2 (hereinafter referred to as the "camera identifier"), and manages the image by associating it with the information added according to the camera identifier (hereinafter referred to as the "additional information"). The additional information may include information about the area of ​​the captured image that is captured (hereinafter referred to as the "target area") that is associated with the camera identifier.

[0014] The management server 1 receives an image, additional information, and instructions specifying a predetermined output format as input to a predetermined language model (hereinafter referred to as "this model") provided by the model provision server 3, and obtains anomaly detection results output from this model in accordance with the specified output format. This model is a language model that can accept at least image and text input and can output at least natural language.

[0015] The management server 1 executes control related to notifications based on the acquired anomaly detection results. Specifically, the management server 1 determines the notification destination based on the acquired camera identifier and executes control related to notifications to that destination. Furthermore, the management server 1 is capable of executing control related to notifications that include images that have undergone a predetermined anonymization process (hereinafter referred to as "processed images") on the acquired captured images. Here, "notification" is a concept that is not limited to sending alarms and alerts to terminals (smartphones, etc.) used by the user (the person to be notified), but also includes sending control commands and information to cooperating external systems and devices.

[0016] [Camera 2] Camera 2, which constitutes the information processing system S, is a camera that captures images of the target area. Camera 2 is installed so as to be able to capture images of the target area and transmits the captured images of the target area to the management server 1. When capturing images of the target area, Camera 2 may capture images as still images or as moving images. Camera 2 may be, for example, a general visible light camera, an infrared camera, or a multi-wavelength spectroscopic camera using special sensors and filters. Examples of multi-wavelength spectroscopic cameras include hyperspectral cameras and multispectral cameras.

[0017] [Model Provision Server 3] The model provision server 3, which constitutes the information processing system S, provides this model. This model is a machine learning model that, upon input of an captured image, additional information, and instructions specifying a predetermined output format, can output anomaly detection results according to the specified output format. This model outputs anomaly detection results according to the specified output format by performing natural language processing based on the input prompt. The format of the input prompt is not particularly limited and may be data of a type such as text data, image data, or audio data. This model may, for example, utilize a large-scale language model and deep learning. The information output from this model may be transmitted to the user device 4 via the management server 1, or it may be transmitted directly from the model provision server 3 to the user device 4.

[0018] [User Device 4] User device 4, which constitutes the information processing system S, is an information processing device operated by users who utilize the services provided by the information processing system S (hereinafter referred to as "the Service"). User device 4 consists of, for example, smartphones, tablet terminals, personal computers, as well as information output devices such as autonomous robots and speakers, and devices that perform various actions. Specific examples of users of the Service will be described later.

[0019] User device 4 executes a predetermined application program that enables the use of this service. User device 4 is capable of performing various processes based on various information transmitted from the management server 1, the model provision server 3, and external sources, as well as various input information entered by the user. For example, some user devices 4 are capable of acquiring and displaying the output information of this model transmitted from the management server 1 or the model provision server 3.

[0020] Furthermore, some user devices 4 are capable of transmitting various types of information to the management server 1, the model provision server 3, and external sources. For example, some user devices 4 are capable of transmitting various types of input information entered by the user to the management server 1, the model provision server 3, and external sources.

[0021] The processing performed by the management server 1, camera 2, model provision server 3, and user device 4, which constitute the information processing system S, is merely an example. In other words, as long as the information processing system S as a whole has the functionality to realize the above-mentioned processing, some or all of the functions for realizing the above-mentioned processing may be shared or performed collaboratively within the information processing system S.

[0022] For example, some or all of the functions of the management server 1 may be assigned to other information processing devices within the information processing system S. Alternatively, some or all of the functions of other information processing devices within the information processing system S may be assigned to the management server 1. Furthermore, some or all of the functions of the management server 1 may be transferred to other servers, etc., not shown in the diagram. This facilitates processing within the entire information processing system S and allows for complementary processing.

[0023] <Hardware Configuration> [Hardware Configuration of Management Server 1] Figure 2 is a block diagram showing an example of the hardware configuration of the management server 1 that constitutes the information processing system S shown in Figure 1. The management server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20.

[0024] The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data necessary for the CPU 11 to execute various processes. The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0025] The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 consists of a display, speaker, etc., and outputs various information as images, sounds, etc. The input unit 17 consists of a keyboard, mouse, touch panel, etc., and accepts input of various information. The storage unit 18 consists of a hard disk, DRAM (Dynamic Random Access Memory), etc., and stores various data. The communication unit 19 communicates with other devices via the aforementioned network N, which is configured as the Internet, etc.

[0026] The drive 20 is appropriately equipped with removable media 21, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as needed. The removable media 21 can also store various types of data stored in the storage unit 18, just like the storage unit 18.

[0027] [Hardware Configuration of User Device 4] User device 4 has the same hardware configuration as the management server 1 shown in Figure 2 above. That is, user device 4 has a CPU, ROM, RAM, bus, input / output interface, output unit, input unit, storage unit, communication unit, drive, and removable media, which correspond to the CPU 11, ROM 12, RAM 13, bus 14, input / output interface 15, output unit 16, input unit 17, storage unit 18, communication unit 19, drive 20, and removable media 21 in Figure 2, respectively.

[0028] <Functional Configuration of Management Server 1> Figure 3 is a functional block diagram showing an example of the functional configuration of Management Server 1. When Management Server 1's CPU 11 is operating, for example, the information acquisition unit 31, the information management unit 32, the association unit 33, the model input / output control unit 35, the notification control unit 36, and the transmission control unit 37 are all functioning.

[0029] Furthermore, the storage unit 18 of the management server 1 is equipped with various databases. For example, it is equipped with databases such as the image capture DB 41 which stores captured images, the identifier DB 42 which stores camera identifiers, the additional information DB 43 which stores additional information, and the processed image DB 44 which stores processed images. The image capture DB 41, identifier DB 42, additional information DB 43, and processed image DB 44 are all related to each other. Note that each database does not necessarily have to be configured as a physically separate database; for example, the identifier DB 42 and the additional information DB 43 may be logically separated and stored within a single database.

[0030] The information acquisition unit 31 acquires various types of information. For example, the information acquisition unit 31 acquires various types of information transmitted from the camera 2. Examples of information transmitted from the camera 2 include captured images and camera identifiers. The information acquisition unit 31 also acquires various types of information transmitted from the model provision server 3. Examples of information transmitted from the model provision server 3 include output information of this model. The information acquisition unit 31 also acquires various types of input information transmitted from the user device 4.

[0031] The information management unit 32 stores and manages various types of information in the database of the storage unit 18. For example, the information management unit 32 stores and manages captured images in the captured image DB 41. The information management unit 32 also stores and manages camera identifiers in the identifier DB 42. The information management unit 32 also stores and manages additional information in the additional information DB 43. The information management unit 32 also stores and manages processed images in the processed image DB 44.

[0032] The association unit 33 associates and manages additional information with the captured image acquired by the information acquisition unit 31. The additional information may include information about the target area.

[0033] The model input / output control unit 35 controls the input and output of information to the model. Specifically, the model input / output control unit 35 generates a prompt for the model that includes instructions specifying additional information and a predetermined output format, and controls the input of the prompt and the captured image to the model. Based on the input prompt and the captured image, the model infers the state of the captured image according to the instructions specifying the output format included in the prompt, and outputs the result in a structured format. That is, the model input / output control unit 35 controls the model to output an anomaly detection result in accordance with the structured output format based on the input prompt and the captured image. The model input / output control unit 35 also controls the acquisition of the anomaly detection inference result (hereinafter appropriately referred to as "structured inference result") output from the model in accordance with the structured output format. When the model input / output control unit 35 outputs the anomaly detection result as output information of the model, it may output the anomaly detection result for each of multiple captured images that are at different times.

[0034] The anomaly detection result output here may include information indicating the type of anomaly. The anomaly detection result may also include information indicating one of several statuses related to the state in which the anomaly was detected. Furthermore, the information acquisition unit 31 may verify whether the output information of this model conforms to a predetermined output format, and if it conforms to the predetermined output format, it may acquire it as an anomaly detection result.

[0035] The model input / output control unit 35 enables input control (context role injection) and output control (structured schema constraints) as input / output control to enable the model to function as a stable event detection sensor specialized for a specific environment or purpose. By dynamically assigning specific social roles and purposes to the model, even when the same captured image is input, the recognition output (definition of the detection target) and operation execution (action) can be changed according to the social role assigned to the model.

[0036] Specifically, the model input / output control unit 35 has functions such as dynamic prompt generation and context injection, structured schema-constrained inference, time-series state evaluation (conditional state transitions), dynamic policy determination of subsequent actions, privacy protection processing, schema verification and standardization of events, and dynamic routing control. Of these, dynamic prompt generation and context injection is a function that not only performs simple image recognition but also dynamically synthesizes and inputs the environment unique to each camera 2 (for example, the arrangement of furniture in the target area) and the social role assigned to this model. This function allows for flexible changes in the objects to be recognized and evaluation criteria depending on the social role, even for the same event, thereby suppressing the occurrence of environment-dependent false positives. As a result, it can handle a variety of use cases.

[0037] Furthermore, structured schema-constrained inference is a function that forces the free-text output of this model to be bound to a specific schema such as JSON (JavaScript Object Notation). This function improves the stability of system processing such as importance determination, notification, physical control, and action execution. In addition, time-series state evaluation (conditional state transitions) is a function that absorbs false detections caused by temporary fluctuations unique to this model on the system side. With this function, by transitioning the state only when specific conditions (for example, consecutive occurrences within a certain period) are met, stable detection accuracy that can withstand actual operation can be achieved.

[0038] Furthermore, the dynamic policy determination for subsequent actions is a function that automatically switches between policies of immediate execution, continuous evaluation, and execution suppression based on the nature and severity of the detected event, using structured inference results. This function allows only truly necessary processing to be executed, thus suppressing unnecessary system operations. In addition, the privacy protection processing is a function that applies anonymization processing (e.g., face blurring) only to captured images sent externally. This function lowers the barrier to use in indoor or private spaces, and allows for a balance between privacy and the convenience of situation confirmation.

[0039] Furthermore, schema validation and standard event conversion are functions that verify whether structured data is in the correct format and convert it into a standard event format within the system. This function ensures the robustness of subsequent processing. In addition, dynamic routing control is a fundamental function that routes information to the appropriate user's user device 4 or the controlled camera 2 based on the camera identifier.

[0040] The notification control unit 36 ​​executes notification control based on the anomaly detection result acquired by the information acquisition unit 31. Specifically, the notification control unit 36 ​​determines the notification destination based on the camera identifier acquired by the information acquisition unit 31 and executes notification control to that destination. Here, if the information acquisition unit 31 has acquired anomaly detection results for each of several captured images that are in different time periods, the notification control unit 36 ​​may execute notification control based on the progression of the multiple anomaly detection results within a predetermined time period.

[0041] Furthermore, if the acquired anomaly detection result includes information indicating the type of anomaly, the notification control unit 36 ​​may change the content of the notification control based on the type of anomaly. Also, if the acquired anomaly detection result includes information indicating one of several statuses related to the state in which an anomaly was detected, the notification control unit 36 ​​may change the content of the notification control according to the status. Furthermore, the notification control unit 36 ​​may execute control related to notifications that include processed images. Specific examples of notification control by the notification control unit 36 ​​will be described later.

[0042] The transmission control unit 37 controls the transmission of various types of information to the camera 2, the model provision server 3, the user device 4, and external sources. For example, the transmission control unit 37 controls the transmission of prompts input to the model by the model input / output control unit 35 to the model provision server 3. The transmission control unit 37 also controls the transmission of output information of the model acquired by the information acquisition unit 31 to the user device 4.

[0043] <Processing Flow of Management Server 1> FIG. 4 is a flowchart illustrating an example of processing of the management server 1. When a captured image and a camera identifier are transmitted from the camera 2 (YES in Step S1), the management server 1 acquires the transmitted captured image and camera identifier (Step S2), and associates additional information with the captured image for management (Step S3). On the other hand, when no captured image and no camera identifier have been transmitted from the camera 2 (NO in Step S1), the management server 1 repeats the determination processing of Step S1.

[0044] The management server 1 inputs predetermined information into the present model provided from the model providing server 3 (Step S4). Specifically, the management server 1 inputs the captured image, the additional information, and an instruction specifying a predetermined output format into the present model. Then, when an abnormality detection result conforming to the specified output format is output from the present model (YES in Step S5), the management server 1 acquires the output abnormality detection result (Step S6), and executes control related to notification based on the abnormality detection result (Step S7). Thereby, the processing of the management server 1 ends (END). On the other hand, when no abnormality detection result is output from the present model (NO in Step S5), the management server 1 repeats the determination processing of Step S5.

[0045] <Specific Examples> Figure 5 shows an example of how this service can be applied. As shown in Figure 5, this service enables monitoring and supervision of a person 500 according to the situation in the captured image 400 of the target area 200 captured by the camera 2, by having the management server 1 control the input and output of information to the model 300 provided by the model provision server 3. For example, this service can be applied to home care and monitoring of the elderly, restoration of disaster sites to their original state by autonomous robots, smart monitoring of public infrastructure, crime prevention and customer service support in retail stores, safety management at construction sites, understanding the condition of crops and livestock in agriculture, inventory and work management in logistics warehouses, patient monitoring in medical facilities, monitoring the condition of roads and bridges in transportation infrastructure, recognition of the surrounding environment of autonomous vehicles, environmental monitoring of energy facilities, safety management of children in schools and childcare facilities, monitoring of factory production lines, home monitoring of pets, indoor monitoring of infants, comprehensive security for general households, exam supervision in cram schools, congestion management at event venues and theme parks, crime prevention management in office and corporate lobbies, attendance and presence management in offices, etc. Specific application examples will be explained individually below.

[0046] [Home care and monitoring of the elderly] This service can be applied to home care and monitoring of the elderly. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a butler who monitors the elderly. The subject 500 has a habit of napping on the sofa on a daily basis. Detect falls, signs of falls (e.g., sliding off a chair), remaining in the same position for a long time, and changes in physical condition (vomiting, showing signs of pain, accidental ingestion of a foreign object, etc.)."

[0047] When this service is applied to in-home care and monitoring for elderly people, this model 300 changes recognition outputs and action executions according to the assigned social role. For example, this model 300 recognizes a state where the subject 500 of in-home care and monitoring has fallen on the floor, and outputs that the possibility of a fall is high and the necessity of response is at a medium level. In this case, according to the continuous evaluation policy, this model 300 transmits a processed image subjected to privacy protection processing (for example, face mosaic processing) to the user device 4 (smartphone or the like) of a user (family member, caregiver, predetermined operator, or the like) only when the state has been evaluated multiple times within a certain period, and requests the user for visual confirmation.

[0048] In addition, this model 300 recognizes an unstable posture in which the subject 500 of in-home care and monitoring is about to slip off a chair or bed, and outputs that the state is highly likely to be a premonition of a fall (unstable body position) and the necessity of response is at a high level. In this case, according to the immediate execution policy, this model 300 transmits an alert to the user device 4 of a user (for example, a family member or caregiver of the subject 500) before a fall occurs. Further, this model 300 recognizes that the subject 500 of in-home care and monitoring has remained in the same posture on a bed or sofa for a long time (for example, 2 hours or more), and outputs that the state is highly likely to be a long-time same body position (pressure ulcer risk) and the necessity of response is at a medium level. In this case, according to the continuous evaluation policy, this model 300 transmits a notification for prompting a position change to the user device 4 at the stage when a predetermined time is exceeded.

[0049] Furthermore, Model 300 recognizes when the person receiving home care or monitoring, 500, is clutching their chest in pain, vomiting, or attempting to put a foreign object other than food into their mouth, and outputs that there is a high risk of acute health change or accidental ingestion, and that the need for action is at the highest level. In this case, Model 300 immediately sends an emergency alert to the user device 4 according to the immediate execution policy. Also, Model 300 recognizes when the person receiving home care or monitoring, 500, is lying down in a safe position on a sofa, and outputs that there is a high probability that they are in a normal state (resting), and that the need for action is at a low level. In this case, Model 300 does not notify the user device 4 according to the execution suppression policy.

[0050] [Restoring a Disaster Site to its Current State Using Autonomous Robots] This service can be applied to restoring a disaster site to its current state using an autonomous robot as the user device 4. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a worker responsible for restoring the accident site to its current state. Prioritize detecting the presence of rubble and the leakage of hazardous materials."

[0051] Even when this service is applied to the restoration of a disaster site using an autonomous robot, Model 300 will change its recognition output and action execution according to the social role it has been assigned. For example, Model 300 will recognize debris blocking a passage in the disaster site and output that there is a high probability that the debris is an obstacle and that the need for action is at a high level. In this case, Model 300 will send a control command to the autonomous robot's drive system to "grasp the object and move it to the specified location" according to the immediate execution policy.

[0052] Furthermore, Model 300 detects liquid leaks from drums left at the disaster site and outputs a message indicating a high probability of hazardous material leakage and a high level of need for action. In this case, Model 300, in accordance with its immediate execution policy, sends a control command to the autonomous robot's drive system to reverse, and also sends an emergency alert to the user's (disaster site manager's) user device 4 (smartphone, etc.).

[0053] [Smart Monitoring of Public Infrastructure] This service can be applied to smart monitoring of public infrastructure. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a security guard patrolling the train station at night. The current time is 2:00 AM, the time when cleaning staff are mopping. Detect any suspicious activity outside of business hours."

[0054] Even when this service is applied to smart monitoring of public infrastructure, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes a person loitering outside of business hours and outputs that there is a high probability of warning behavior (loitering) and that the need for action is at a medium level. In this case, Model 300 notifies the user's (security guard, etc.) user device 4 if the person remains for a certain period of time, according to the continuous evaluation policy. Also, Model 300 recognizes a person (cleaner) moving in the target area 200 and outputs that there is a high probability that this is normal work (cleaning) that is not a warning target and that the need for action is at a low level. In this case, Model 300 does not notify the user device 4, according to the execution suppression policy.

[0055] [Security and Customer Service Support in Retail Stores] This service can be applied to security and customer service support in retail stores. In this case, the management server 1 controls the input to the model 300, for example, by inputting role and environmental information such as, "During business hours, you are an excellent store clerk. Detect customers who are having trouble finding products during business hours," or "Outside of business hours, you are a security guard. Detect intruders and suspicious activity outside of business hours."

[0056] Even when this service is applied to security and customer service support in retail stores, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes a person loitering outside of business hours and outputs that there is a high probability of warning behavior (loitering) and that the need for action is at a medium level. In this case, Model 300 notifies the user's (security guard, etc.) user device 4 if the person stays for a certain period of time, according to the continuous evaluation policy. Also, Model 300 recognizes a person staring at merchandise for a long time and outputs that there is a high probability of a customer service opportunity and that the need for action is at a medium level. In this case, Model 300 sends a notification to the user's (store clerk in their assigned area within the store) user device 4 (e.g., intercom) indicating that there is a customer service opportunity, according to the immediate execution policy.

[0057] Furthermore, Model 300 recognizes a person attempting to pry open a locked door and outputs that there is a high probability that the person is an intruder and that a response is needed at a high level. In this case, Model 300, in accordance with the immediate action policy, notifies the user's (security company's) user device 4 (a terminal of a designated security system, etc.). At the same time, it also emits a warning sound from the user device 4 (store speaker). In addition, Model 300 recognizes a person repeatedly peering outside the store and outputs that there is a high probability that this is a suspicious behavior (peeping) and that a response is needed at a medium level. In this case, Model 300, in accordance with the continuous evaluation policy, flags the recorded data and notifies the user device 4 if it continues for a certain period of time.

[0058] [Safety Management at Construction Sites] This service can be applied to safety management at construction sites. In this case, the management server 1 controls the input of roll and environmental information to the model 300, for example, "You are the safety manager at the construction site. Wearing a helmet and safety harness is mandatory in this area, and entry under the crane is prohibited. Detect instances of helmet not being worn, safety harness not being worn, and entry under the crane."

[0059] Even when this service is applied to safety management at a construction site, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes a worker not wearing a helmet and outputs that there is a high possibility of non-compliance with safety standards (not wearing protective equipment) and that the need for action is high. In this case, Model 300, according to the immediate execution policy, will output voice information from the user device 4 (work site speaker), such as "Please check your safety equipment" (notification will be sent by transmitting various information to trigger the output).

[0060] Furthermore, Model 300 recognizes workers not using safety harnesses while working at heights and outputs a message indicating a high probability of serious non-compliance with safety standards and a high level of need for action. In this case, Model 300, under an immediate action policy, sends an emergency notification to the user's (site supervisor's) user device 4 (smartphone, etc.) and also causes a warning to be output from the user device 4 (work site speaker) (notification is provided by sending various information to trigger the output). Also, Model 300 recognizes a person who has entered under a crane and outputs a message indicating a high probability of intrusion into a dangerous area and a high level of need for action. In this case, Model 300, under an immediate action policy, causes the user device 4 (work site speaker) to output voice information, such as "Please evacuate from the dangerous area" (notification is provided by sending various information to trigger the output).

[0061] [Monitoring the condition of crops and livestock in agriculture] This service can be applied to monitoring the condition of crops and livestock in agriculture. In this case, the management server 1 controls the input of role and environmental information to the model 300, such as, "You are the farm manager. Tomatoes are being grown in this field. Detect signs of pest damage, signs of disease, signs of water shortage, intrusion by harmful animals such as bears, monkeys, and wild boars, and abnormal water levels," or "You are a veterinarian's assistant. Detect changes in the gait and recumbent time of cattle."

[0062] Even when this service is applied to monitoring the condition of crops and livestock in agriculture, the model 300 changes its recognition output and action execution according to the assigned social role. For example, the model 300 recognizes leaf feeding patterns and outputs that there is a high probability of pest damage and that the need for action is at a medium level. In this case, according to the continuous evaluation policy, the model 300 notifies the user's (farm manager's) user device 4 (smartphone, etc.) if damage is detected in multiple locations or multiple times.

[0063] Furthermore, Model 300 recognizes leaf discoloration patterns and outputs that there is a high probability of signs of poor growth (disease spots) and that the need for action is at a moderate level. In this case, Model 300 notifies the user's (farm manager's) user device 4 (smartphone, etc.) if progression is confirmed, according to the continuous evaluation policy. Also, Model 300 recognizes plant wilting and outputs that there is a high probability of signs of poor growth (wilting) and that the need for action is at a moderate level. In this case, Model 300, according to the continuous evaluation policy, will cooperate with the user device 4 (e.g., an automatic irrigation system) if the wilting continues for a certain period of time and will also notify the user's (farm manager's) user device 4 (smartphone, etc.).

[0064] Furthermore, Model 300 detects the intrusion of pests such as bears and monkeys into the vicinity of farmland and outputs that there is a high probability of pest intrusion and that a response is needed at a high level. In this case, Model 300 activates user devices 4 (e.g., deterrent sound devices or flashlights) according to the immediate action policy and notifies the user's (farm manager's) user device 4 (smartphone, etc.). Also, Model 300 detects that the water level exceeds or falls below the set level and outputs that there is a high probability of an abnormal water level and that a response is needed at a high level. In this case, Model 300 sends an emergency alert to the user's (farm manager's) user device 4 (smartphone, etc.) according to the immediate action policy and controls the operation of user devices 4 (water supply and drainage gates) if possible.

[0065] Furthermore, Model 300 recognizes lameness (a limping gait) in cattle and outputs that there is a high probability of a gait abnormality and that the need for intervention is at a moderate level. In this case, Model 300 notifies the user's (veterinarian or administrator's) user device 4 (smartphone, etc.) if it is detected multiple times, according to the continuous evaluation policy. Also, Model 300 recognizes that a cattle does not stand up for a long period of time and outputs that there is a high probability of a significant decrease in activity level and that the need for intervention is at a moderate level. In this case, Model 300 notifies the user's (veterinarian or administrator's) user device 4 (smartphone, etc.) if the cattle remain lying down for longer than the normal resting time, according to the continuous evaluation policy.

[0066] [Management of inventory and operations in logistics warehouses] This service can be applied to the management of inventory and operations in logistics warehouses. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a warehouse operations supervisor. Detect pedestrians entering forklift aisles, signs of cargo collapse, and inventory placement in unauthorized areas."

[0067] Even when this service is applied to inventory and work management in a logistics warehouse, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes that a pedestrian has entered a forklift aisle and outputs that there is a high possibility of entering a restricted area and that the need for action is high. In this case, Model 300, in accordance with the immediate execution policy, sends a stop command to User Device 4 (forklift control system) and causes User Device 4 (workplace speaker) to output a warning (notified by sending various information to trigger the output).

[0068] Furthermore, Model 300 recognizes that the cargo on the pallet is tilted and outputs that there is a high risk of cargo collapse and that the need for action is at a medium level. In this case, Model 300, under the policy of immediate action, notifies the user's (field worker's) user device 4 (smartphone, etc.) and instructs them to check the safety of the area. Also, Model 300 recognizes a pallet left in a passageway and outputs that there is a high possibility of improper placement and that the need for action is at a low level. In this case, under the policy of continuous evaluation, Model 300 notifies the user's (floor manager's) user device 4 (terminal of a designated safety management system, etc.) if the pallet is left there for a certain period of time.

[0069] [Patient Monitoring in Medical Facilities] This service can be applied to patient monitoring in medical facilities. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a ward nurse. The patient in this room is post-operative and requires rest. Detect when the patient gets out of bed, operates medical equipment themselves, and shows signs of pain."

[0070] Even when this service is applied to patient monitoring in a medical facility, the model 300 changes its recognition output and action execution according to the assigned social role. For example, the model 300 recognizes that a patient is attempting to climb over a bed rail and outputs that there is a high probability of safety risk behavior (getting out of bed) and that the need for action is at a high level. In this case, the model 300 sends an alert to the user device 4 (ward management system, etc.) at the nurse station according to the immediate execution policy.

[0071] Furthermore, Model 300 recognizes when a patient touches an IV tube or buttons on equipment and outputs that there is a high probability of dangerous behavior (operation of equipment) and that a high level of response is required. In this case, Model 300 sends an alert to the user device 4 (ward management system, etc.) at the nurse station according to the immediate execution policy. Also, Model 300 recognizes the patient's face and outputs that there is a high probability of a sudden change in condition (expression of pain) based on changes in facial expression such as grimacing and that a high level of response is required. In this case, Model 300 sends a notification to the user device 4 (smartphone, etc.) of the user (attending nurse) according to the immediate execution policy.

[0072] [Condition Monitoring of Roads and Bridges in Transportation Infrastructure] This service can be applied to the condition monitoring of roads and bridges in transportation infrastructure. In this case, the management server 1 controls the input of roll and environmental information to the model 300, for example, "You are an infrastructure inspection engineer. Detect crack progression on the concrete surface of this bridge, exposed rebar, and blockage of drainage outlets."

[0073] Even when this service is applied to monitoring the condition of roads and bridges in transportation infrastructure, Model 300 changes its recognition output and operation execution according to the assigned social role. For example, Model 300 recognizes minute cracks on the concrete surface and outputs that there is a high possibility of surface deterioration progressing and that the need for action is low. In this case, according to the continuous evaluation policy, Model 300 notifies the user's (inspector, etc.) user device 4 (smartphone, etc.) if the change continues during an evaluation over several days to several weeks.

[0074] Furthermore, Model 300 recognizes the exposure of reinforcing bars due to concrete spalling and outputs that there is a high possibility of structural deterioration (reinforcing bar exposure) and that the need for action is at a medium level. In this case, Model 300 records this in the user device 4 (inspection report system) and notifies the user's (person in charge's) user device 4 (smartphone, etc.). Also, Model 300 recognizes puddles around drain outlets and outputs that there is a high possibility of malfunction (poor drainage) and that the need for action is at a medium level. In this case, if puddles do not disappear for a certain period of time during rainfall, Model 300 notifies the user's (maintenance department's) user device 4 (designated maintenance system, etc.).

[0075] [Environmental Recognition for Autonomous Vehicles] This service can be applied to the environmental recognition of autonomous vehicles. In this case, the management server 1 controls the input of role and environmental information to the model 300, such as, "You are a driver driving in a residential area. Prioritize detecting children, bicycles, and blind spots caused by parked cars," or "You are a driver driving on a highway. Detect sudden deceleration of vehicles ahead, falling objects, and lane departures."

[0076] Even when this service is applied to the surrounding environment recognition of an autonomous vehicle, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes pedestrians or cyclists appearing from behind objects and outputs that there is a high risk of unpredictable intrusion and a high level of need for action. In this case, Model 300 sends a rapid deceleration / stop command to the user device 4 (vehicle control system, etc.) according to the immediate execution policy.

[0077] Furthermore, Model 300 recognizes a parked vehicle on the road and outputs that the area behind it is a high-risk area with a high blind spot risk and a medium level of need for action. In this case, Model 300 sends a slow-down command to the user device 4 (vehicle control system, etc.) according to the immediate execution policy. Also, Model 300 recognizes the brake lights of the vehicle ahead and a sudden decrease in the distance between vehicles and outputs that the collision risk is high and a high level of need for action. In this case, Model 300 sends a deceleration / avoidance command to the user device 4 (vehicle control system, etc.) according to the immediate execution policy.

[0078] Furthermore, Model 300 recognizes an obstacle on the lane and outputs that there is a high probability that it is a moving obstacle and that the need for action is high. In this case, Model 300 sends a lane change command to User Device 4 (vehicle control system, etc.) according to the immediate execution policy. Also, Model 300 recognizes that its own vehicle is about to cross a white line and outputs that there is a high probability that the driving trajectory is abnormal and that the need for action is high. In this case, Model 300 sends a steering correction command to User Device 4 (vehicle control system, etc.) according to the immediate execution policy.

[0079] [Environmental Monitoring of Energy Facilities] This service can be applied to environmental monitoring of energy facilities. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a power plant operation monitor. Detect equipment liquid leaks, gas leaks from piping, and out-of-spec readings on instruments."

[0080] Even when this service is applied to environmental monitoring of energy facilities, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes an image in which a liquid film is spreading around equipment and outputs that there is a high possibility of fluid leakage and that the need for action is at a high level. In this case, Model 300 sends an alert to user device 4 (device in the control room, etc.) and automatically closes user device 4 (related valves, etc.) according to the immediate execution policy (notifying the user by sending various information to cause automatic closing).

[0081] Furthermore, Model 300 recognizes the emission of steam or gas from piping and outputs that there is a high possibility of gas leakage and that the need for action is at a high level. In this case, Model 300, under the policy of immediate action, sends an alert to User Device 4 (devices in the control room, etc.) and forces User Device 4 (ventilation system, etc.) to start up (notifying it by sending various information to force it to start up). Also, Model 300 recognizes that the needle of an analog instrument is in the red zone and outputs that there is a high possibility of an abnormal value being displayed and that the need for action is at a medium level. In this case, under the policy of continuous evaluation, Model 300 notifies User Device 4 (devices in the control room, etc.) if this is confirmed multiple times.

[0082] [Child Safety Management in Schools and Childcare Facilities] This service can be applied to child safety management in schools and childcare facilities. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a childcare worker. Detect the risk of falling from playground equipment, approaching the boundary fence, and signs of interpersonal trouble among children playing in the playground."

[0083] Even when this service is applied to child safety management in schools and childcare facilities, the Model 300 changes its recognition output and action execution according to the assigned social role. For example, the Model 300 recognizes a child in an unstable position at a high point on a jungle gym and outputs that there is a high risk of falling from a height and that the need for action is at a high level. In this case, the Model 300 sends an alert to the user's (childcare worker in the playground) user device 4 (smartphone, etc.) according to the immediate execution policy.

[0084] Furthermore, Model 300 recognizes when a child approaches the fence and outputs that there is a high probability of approaching the boundary and that the need for action is at a medium level. In this case, according to the continuous evaluation policy, Model 300 will notify the user's (the nursery teacher in the playground) user device 4 (smartphone, etc.) if the child remains near the fence for a certain period of time. Also, Model 300 recognizes unnatural closeness or pushing between children and outputs that there is a high probability of a collision or trouble and that the need for action is at a medium level. In this case, according to the continuous evaluation policy, Model 300 will notify the user's (the nursery teacher in the playground) user device 4 (smartphone, etc.) if the behavior continues for a certain period of time.

[0085] [Factory Production Line Monitoring] This service can be applied to monitoring factory production lines. In this case, the management server 1 controls the input of roll and environmental information to the model 300, for example, "You are the quality control manager for the production line. Detect changes in belt conveyor speed and deflection, product appearance defects (deformation, discoloration, foreign matter adhesion), and dangerous actions by workers."

[0086] Even when this service is applied to monitoring a factory production line, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes a slight deflection in a conveyor belt and outputs that there is a high probability of a change in equipment operation and that the need for action is at a medium level. In this case, according to the continuous evaluation policy, Model 300 notifies the user's (equipment maintenance personnel's) user device 4 (smartphone, etc.) if the change continues within a certain period.

[0087] Furthermore, Model 300 recognizes deformation or discoloration of products flowing down the line and outputs that there is a high possibility of quality abnormality and that the need for action is at a medium level. In this case, Model 300 sends a command to user device 4 (discharge mechanism, etc.) to automatically remove the defective product from the line, according to the immediate execution policy. Also, Model 300 recognizes a worker approaching a machine in operation and outputs that there is a high possibility of non-compliance with safety standards and that the need for action is at a high level. In this case, Model 300 sends a stop command to user device 4 (line control system, etc.) and emits a warning sound from user device 4 (factory speaker), according to the immediate execution policy.

[0088] [Pet In-Home Monitoring] This service can be applied to in-home pet monitoring. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a pet sitter. There is one Shiba Inu (elderly, with heart disease) in this room. Detect prolonged lying down, vomiting, abnormal breathing, and furniture destruction."

[0089] Even when this service is applied to in-home pet monitoring, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes that a pet is lying in the same position for a long period of time and outputs that there is a high possibility of a significant decrease in activity level and that the need for intervention is at a medium level. In this case, Model 300, according to the continuous evaluation policy, notifies the user's (pet owner's) user device 4 (smartphone, etc.) if no activity is detected for a certain period of time. Also, Model 300 recognizes that a pet is vomiting and outputs that there is a high possibility of a sudden change in health (vomiting) and that the need for intervention is at a high level. In this case, Model 300, according to the immediate action policy, immediately notifies the user's (pet owner's) user device 4 (smartphone, etc.).

[0090] Furthermore, Model 300 recognizes rapid breathing from the movement of the pet's abdomen and outputs that there is a high possibility of a sudden change in health (respiratory abnormality) and that the need for action is at a high level. In this case, Model 300 immediately notifies the user's (pet owner's) user device 4 (smartphone, etc.) according to the immediate action policy. Also, Model 300 recognizes the pet's behavior of chewing on sofas or cushions and outputs that there is a high possibility of problem behavior (destruction) and that the need for action is at a medium level. In this case, Model 300 notifies the user's (pet owner's) user device 4 (smartphone, etc.) if the behavior continues, according to the continuous evaluation policy.

[0091] [Indoor Monitoring of Infants] This service can be applied to indoor monitoring of infants. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are a babysitter. There is a 10-month-old infant in this room. Detect when the baby climbs out of the crib, sleeps face down for extended periods, puts small objects near the baby's mouth, and is dressed inappropriately for the room temperature."

[0092] Even when this service is applied to indoor monitoring of infants, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes that an infant has its feet on the railing of a crib and outputs that there is a high probability of this being a fall risk behavior and that the need for intervention is high. In this case, Model 300 immediately sends an alert to the user's (guardian's) user device 4 (smartphone, etc.) according to the immediate action policy. Also, Model 300 recognizes that an infant is lying face down and outputs that there is a high probability of suffocation risk (face down) and that the need for intervention is high. In this case, Model 300, according to the continuous evaluation policy, sends an alert to the user's (guardian's) user device 4 (smartphone, etc.) if the condition persists after a short grace period for the infant to roll over.

[0093] Furthermore, Model 300 recognizes when an infant attempts to put a small object that has fallen on the floor into its mouth and outputs that there is a high probability of this being a choking hazard and that the need for intervention is high. In this case, Model 300 immediately sends an alert to the user's (guardian's) user device 4 (smartphone, etc.) according to the immediate action policy. Also, Model 300 compares room temperature information with the infant's clothing (e.g., light clothing in a cold room) and outputs that there is a high probability of environmental incompatibility (clothing) and that the need for intervention is low. In this case, Model 300 notifies the user's (guardian's) user device 4 (smartphone, etc.) if the situation continues for a certain period of time, according to the continuous evaluation policy.

[0094] [Comprehensive Security for General Households] This service can be applied to comprehensive security for general households. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are the caretaker of this household. Detect intruders, forgotten fires, plumbing malfunctions, and confirm the children's return home."

[0095] Even when this service is applied to the comprehensive security of a typical home, Model 300 will change its recognition output and action execution according to the assigned social role. For example, Model 300 may recognize a person attempting to break a front door window and output that there is a high probability of this being an intruder and that a response is needed at a high level. In this case, Model 300 will, according to the immediate action policy, send an emergency notification to the user's (resident's) user device 4 (smartphone, etc.) and activate the user device 4 (indoor alarm). Also, Model 300 may recognize that the stove is left on and unoccupied, and output that there is a high probability of fire risk (forgetting to turn it off) and that a response is needed at a high level. In this case, according to the immediate action policy, Model 300 will send an emergency notification to the user's (resident's) user device 4 (smartphone, etc.).

[0096] Furthermore, Model 300 recognizes that water is continuously flowing from the faucet and outputs that there is a high risk of water leakage and that the need for action is at a medium level. In this case, Model 300, under the policy of continuous evaluation, will notify the user's (resident's) user device 4 (smartphone, etc.) if the situation continues for a certain period of time. Also, Model 300 recognizes the face of a registered child and outputs that there is a high probability that it is a specific person returning home and that the need for action is at a low level. In this case, Model 300, under the policy of immediate execution, records the return home log and sends a daily notification to the user's (guardian's) user device 4 (smartphone, etc.).

[0097] [Exam Supervision in Cram Schools] This service can be applied to exam supervision in cram schools. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are an exam supervisor. Detect the use of smartphones, frequent shifts of gaze to the adjacent seat, unnatural hand movements under the desk, and reference to unauthorized materials."

[0098] Even when this service is applied to exam supervision in cram schools, Model 300 changes its recognition output and action execution according to the assigned social role. For example, if Model 300 recognizes that a test-taker is operating a smartphone under the desk, it outputs that there is a high possibility of cheating (use of communication device) and that the need for action is high. In this case, Model 300 immediately notifies the user's (exam supervisor's) user device 4 (smartphone, etc.) according to the immediate action policy. Also, if Model 300 recognizes that a test-taker is repeatedly looking in the direction of the adjacent seat, it outputs that there is a high possibility of a biased gaze pattern and that the need for action is medium. In this case, according to the continuous evaluation policy, Model 300 notifies the user's (exam supervisor's) user device 4 (smartphone, etc.) if it is detected multiple times within a certain period.

[0099] Furthermore, Model 300 detects when a test-taker appears to be hiding something under the desk and outputs that there is a high probability of suspicious behavior and that the need for action is at a medium level. In this case, according to the continuous evaluation policy, Model 300 will notify the user's (exam proctor's) user device 4 (smartphone, etc.) if the behavior continues. Also, Model 300 detects when a test-taker is looking at a memo hidden in their pencil case and outputs that there is a high probability of cheating (referring to materials) and that the need for action is at a high level. In this case, Model 300 will immediately notify the user's (exam proctor's) user device 4 (smartphone, etc.).

[0100] [Congestion Management at Event Venues and Theme Parks] This service can be applied to congestion management at event venues and theme parks. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, by saying, "You are the operations manager of the theme park. Detect queue lengths, congestion levels in aisles, occurrences of people feeling unwell, and instances of people leaving or cutting in line."

[0101] Even when this service is applied to congestion management at event venues and theme parks, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes that a queue extends beyond a designated area and outputs that there is a high possibility of exceeding capacity and that the need for action is at a medium level. In this case, Model 300 automatically updates the waiting time on user device 4 (digital signage, etc.) if a certain period of time is exceeded, according to the continuous evaluation policy. Also, Model 300 recognizes that the population density in a specific area is approaching the capacity limit and outputs that there is a high possibility of increasing congestion and that the need for action is at a medium level. In this case, Model 300 notifies the user (staff) user device 4 (smartphone, etc.) if congestion continues for a certain period of time, according to the continuous evaluation policy.

[0102] Furthermore, Model 300 recognizes a person crouching in a queue or aisle and outputs that there is a high possibility of a sudden change in their condition and that the need for intervention is high. In this case, Model 300, following the immediate action policy, transmits the location information of the person 500 and sends an emergency notification to the user's (medical staff's) user device 4 (smartphone, etc.). Also, Model 300 recognizes a person cutting in line and outputs that there is a high risk of trouble (cutting in line) and that the need for intervention is low. In this case, following the continuous evaluation policy, Model 300 instructs the user's (surrounding staff's) user device 4 (smartphone, etc.) to patrol if this occurs multiple times.

[0103] [Security Management for Office and Corporate Lobbies] This service can be applied to security management for office and corporate lobbies. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are the security manager of this office building. Detect unauthorized entry without going through reception, loitering outside business hours, abandonment of suspicious objects, and unauthorized removal from loading docks."

[0104] Even when this service is applied to security management in office and corporate lobbies, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes a person attempting to force their way through a security gate and outputs that there is a high probability of unauthorized entry and a high level of need for action. In this case, Model 300, in accordance with the immediate action policy, seals the user device 4 (security gate) (notifying the user by sending various information to seal it) and sends an emergency notification to the user device 4 (a designated security management system) in the user (security room). Also, Model 300 recognizes a person moving through the lobby outside of business hours and outputs that there is a high probability of the person being present outside of business hours and a high level of need for action. In this case, Model 300, in accordance with the immediate action policy, sends an alert to the user device 4 (a designated security management system) in the user (security room).

[0105] Furthermore, Model 300 recognizes bags and luggage left in the lobby and outputs that there is a high probability that they are suspicious items and that a response is needed at a high level. In this case, Model 300, under the policy of continuous evaluation, notifies the security room if the owner does not appear after a certain period of time. Also, Model 300 recognizes the action of a person without a permit carrying out an item and outputs that there is a high risk of unauthorized removal and that a response is needed at a high level. In this case, under the policy of immediate action, Model 300 sends an alert to the user's (security room's) user device 4 (a designated security management system) and automatically increases the recording resolution of the camera 2 installed in the area.

[0106] [Attendance and Presence Management in the Office] This service can be applied to attendance and presence management in the office. In this case, the management server 1 controls the input of role and environmental information to the model 300, for example, "You are the person in charge of attendance management in the office. Understand the presence status of each desk area, the usage status of meeting rooms, and the congestion level of shared spaces."

[0107] Even when this service is applied to attendance and presence management in an office, Model 300 changes its recognition output and action execution according to the assigned social role. For example, Model 300 recognizes that a particular desk has been unoccupied for a long period of time and outputs that there is a high probability of prolonged absence and that the need for action is low. In this case, Model 300 notifies the user's (administrator's) user device 4 (smartphone, etc.) if the absence continues for a certain period of time during business hours, according to the continuous evaluation policy.

[0108] Furthermore, Model 300 recognizes that there are people in a meeting room that appears vacant on the reservation system, and outputs that there is a high probability of unreserved use and that the need for action is low. In this case, according to the continuous evaluation policy, if the use continues for a certain period of time, Model 300 will cooperate with user device 4 (reservation system) and send a notification prompting automatic reservation or departure. Also, Model 300 recognizes that there is a high probability of overcrowding in the refresh space and outputs that there is a low probability of overcrowding and that the need for action is low. In this case, according to the continuous evaluation policy, if the congestion continues, Model 300 will display information on user device 4 (office signage, etc.) encouraging dispersed use.

[0109] <Notification Control Processing Flow> Next, the processing flow for notification control by the notification control unit 36 ​​will be explained in detail. The model input / output control unit 35 inputs the captured image acquired from camera 2 and additional information corresponding to the camera identifier into the model 300 and acquires a structured inference result according to a predetermined output format. The additional information may include information about the environment unique to each camera 2 (for example, the arrangement of equipment in the target area) and information about the social role to be assigned to the model 300. The structured inference result includes at least information indicating the type of anomaly, a status indicating the urgency of the detected state, and the camera identifier of the camera 2 that acquired the captured image. The model input / output control unit 35 verifies whether the acquired structured inference result conforms to a predetermined output format (schema) (schema verification), and if it conforms to the output format, it causes the information acquisition unit 31 to acquire the structured inference result as an anomaly detection result. The captured image, metadata, and detection metrics are stored in the respective databases by the information management unit 32.

[0110] Based on the structured inference results obtained as an anomaly detection result, the notification control unit 36 ​​first determines whether the status indicating the urgency is above a predetermined threshold. If the urgency is below the threshold, the notification control unit 36 ​​does not execute any control related to notification (hereinafter referred to as "execution suppression"). If the urgency is above the threshold, the notification control unit 36 ​​determines one of the following notification policies based on the type of anomaly: "immediate notification," "notification after continuous confirmation," and "execution suppression."

[0111] If the notification policy is "immediate notification," the notification control unit 36 ​​will assume that a high-priority anomaly such as a fire or unauthorized intrusion has been detected and will immediately execute control to notify the recipient. If the notification policy is "notify after continuous confirmation," the notification control unit 36 ​​will assume that an anomaly requiring confirmation, such as a suspected fall, has been detected and will further determine whether the detection result of the anomaly has been obtained a certain number of times consecutively within a predetermined time window. If a certain number of consecutive detections are confirmed within the predetermined time window, the notification control unit 36 ​​will execute control to notify the recipient. If the number of consecutive detections within the predetermined time window is less than the certain number, the notification control unit 36 ​​will only record the number of detections and will not execute control to notify. If the notification policy is "execution suppression," the notification control unit 36 ​​will not execute control to notify.

[0112] The notification control unit 36, when executing control related to notifications, determines the notification destination based on the camera identifier and enables the inclusion of a processed image, which has undergone a predetermined anonymization process on the captured image, in the notification. The notification destination and notification means may be pre-set in association with the camera identifier, and examples include notification to a mobile terminal such as a smartphone, notification via a messaging service including SNS, or linkage to an emergency call (such as a 119 emergency call).

[0113] <Advantageous Effects of This Embodiment> According to the above embodiment, it is possible to provide a monitoring technology or surveillance technology that can take appropriate action according to the conditions of the camera's imaging area.

[0114] <Other> Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc., within the scope that can achieve the objective of the present invention are included in the present invention. For example, the series of processes described above can be executed by hardware or by software. In other words, the above-described functional configuration is merely illustrative and not particularly limited. That is, it is sufficient that the information processing system S is equipped with a function that can execute the series of processes described above as a whole, and the type of functional block used to realize this function is not particularly limited to the above-described example.

[0115] Furthermore, the location of the functional blocks is not particularly limited and can be arbitrary. For example, the functional blocks of management server 1 may be transferred to other devices, or the functional blocks of other devices may be transferred to servers. Also, a single functional block may consist of hardware alone, software alone, or a combination of both.

[0116] When a series of processes are executed by software, the programs that make up that software are installed on a computer or other device from a network or storage medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer capable of performing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0117] Such recording media containing programs may consist not only of removable media (not shown) distributed separately from the main device to provide programs to users, but also of recording media provided to users in a state where they are pre-installed in the main device. Since programs can be distributed via a network, the recording media may be installed on or accessible from a computer connected to or capable of connecting to a network.

[0118] In this specification, the step of describing a program to be recorded on a recording medium includes not only processes that are performed chronologically in that order, but also processes that are not necessarily performed chronologically, but are executed in parallel or individually. Furthermore, in this specification, the term "system" refers to an overall system composed of multiple devices, means, etc.

[0119] In other words, the information processing system to which the present invention is applied can take various forms having the following configurations: (1) That is, the information processing system to which the present invention is applied (for example, the information processing system S in Figure 1) is an information processing system comprising: an image acquisition means (for example, an information acquisition unit 31 in Figure 3) that acquires an image captured by a camera (for example, "camera 2" in Figure 1) together with a camera identifier that identifies the camera; an additional information assignment means (for example, an association unit 33 in Figure 3) that associates additional information corresponding to the camera identifier with the image captured; a language model input means (for example, a model input / output control unit 35 in Figure 3) that accepts input of at least images and text and outputs at least natural language, and inputs the image captured, the additional information, and an instruction to specify a predetermined output format; a language model output acquisition means (for example, an information acquisition unit 31 in Figure 3) that acquires an anomaly detection result from the language model according to the output format; and a notification control means (for example, a notification control unit 36 ​​in Figure 3) that executes control related to notification based on the anomaly detection result. This enables the provision of monitoring or surveillance technology that can take appropriate action according to the conditions in the camera's imaging area.

[0120] (2) The notification control means can determine the notification destination based on the camera identifier and execute the notification control to the notification destination. This makes it possible to control notifications to the notification destination according to the camera identifier, so that appropriate responses can be taken depending on the situation.

[0121] (3) The anomaly detection result also includes information indicating the type of anomaly, and the notification control means can change the content of the control related to the notification based on the type of anomaly. This makes it possible to take appropriate action according to the type of anomaly.

[0122] (4) The abnormality detection result also includes information indicating one of several statuses related to the detected state, and the notification control means can change the content of the control relating to the notification according to the status. This makes it possible to take appropriate action according to the status.

[0123] (5) Furthermore, the language model output acquisition means can acquire the anomaly detection result for each of the multiple captured images that are different in time, and the notification control means can execute the control related to the notification based on the progression of the multiple anomaly detection results within a predetermined time. This makes it possible to respond in accordance with the progression of the situation over time, rather than responding based on a single timing.

[0124] (6) The additional information may also include information relating to the region of the captured image that is to be captured, which is associated with the camera identifier. This makes it possible to take appropriate action depending on the region to be captured.

[0125] (7) The notification control means can also perform control over the notification, including a processed image obtained by applying a predetermined anonymization process to the captured image. This ensures the protection of privacy.

[0126] (8) The language model output acquisition means can also verify whether the output information of the language model conforms to the output format, and if it conforms to the output format, it can be acquired as the anomaly detection result. This ensures the robustness of subsequent processing.

[0127] Furthermore, the information processing method to which the present invention is applied can take various forms having the following configurations. (9) That is, the information processing method to which the present invention is applied is an information processing method executed by an information processing device, and includes: an image acquisition step of acquiring an image captured by a camera together with a camera identifier that identifies the camera; an additional information assignment step of associating additional information corresponding to the camera identifier with the image captured; an input control step of inputting the image captured, the additional information, and an instruction specifying a predetermined output format to a language model that can accept input of at least images and text and can output at least natural language; a result acquisition step of acquiring an anomaly detection result from the language model in accordance with the output format; and a notification control step of executing control related to notification based on the anomaly detection result. This makes it possible to provide a monitoring technology or surveillance technology that can take action according to the status of the camera's imaging area.

[0128] Furthermore, the program to which the present invention is applied can take various forms having the following configurations. (10) That is, the program to which the present invention is applied is a program that causes a computer to execute a control process including: an image acquisition step of acquiring an image captured by a camera along with a camera identifier that identifies the camera; an additional information assignment step of associating additional information corresponding to the camera identifier with the captured image; an input control step of inputting the captured image, the additional information, and an instruction specifying a predetermined output format to a language model that can accept input of at least images and text and can output at least natural language; a result acquisition step of acquiring an anomaly detection result from the language model according to the output format; and a notification control step of executing control related to notification based on the anomaly detection result. This makes it possible to provide a monitoring technology or surveillance technology that can take action according to the status of the camera's imaging area.

[0129] 1: Management server, 2: Camera, 3: Model provision server, 4: User device, 11: CPU, 16: Output unit, 17: Input unit, 18: Storage unit, 19: Communication unit, 31: Information acquisition unit, 32: Information management unit, 33: Association unit, 35: Model input / output control unit, 36: Notification control unit, 37: Transmission control unit, 41: Captured image DB, 42: Identifier DB, 43: Addition DB, 44: Processed image DB, N: Network, S: Information processing system

Claims

1. An information processing system comprising: an image acquisition means for acquiring an image captured by a camera along with a camera identifier that identifies the camera; an additional information assignment means for associating additional information corresponding to the camera identifier with the captured image; a language model input means for inputting the captured image, the additional information, and an instruction specifying a predetermined output format to a language model that can accept at least image and text inputs and can output at least natural language; a language model output acquisition means for acquiring an anomaly detection result from the language model in accordance with the output format; and a notification control means for executing control related to notification based on the anomaly detection result.

2. The information processing system according to claim 1, wherein the notification control means determines a notification recipient based on the camera identifier and performs control related to the notification on the notification recipient.

3. The information processing system according to claim 1, wherein the anomaly detection result includes information indicating the type of anomaly, and the notification control means changes the content of the control relating to the notification based on the type of anomaly.

4. The information processing system according to claim 1, wherein the abnormality detection result includes information indicating one of a plurality of statuses relating to the detected state, and the notification control means changes the content of the control relating to the notification according to the status.

5. The information processing system according to claim 1, wherein the language model output acquisition means acquires the anomaly detection result for each of the multiple captured images that are different in time, and the notification control means performs control related to the notification based on the progression of the multiple anomaly detection results within a predetermined time.

6. The information processing system according to claim 1, wherein the additional information includes information relating to the region of the captured image that is to be captured, associated with the camera identifier.

7. The information processing system according to claim 1, wherein the notification control means performs control relating to the notification, which includes a processed image obtained by applying a predetermined anonymization process to the captured image.

8. The information processing system according to claim 1, wherein the language model output acquisition means verifies whether the output information of the language model conforms to the output format, and acquires it as the anomaly detection result if it conforms to the output format.

9. An information processing method performed by an information processing device, comprising: an image acquisition step of acquiring an image captured by a camera along with a camera identifier that identifies the camera; an additional information assignment step of associating additional information corresponding to the camera identifier with the image captured; an input control step of inputting the image captured, the additional information, and an instruction specifying a predetermined output format to a language model that can accept at least image and text inputs and can output at least natural language; a result acquisition step of acquiring an anomaly detection result from the language model in accordance with the output format; and a notification control step of performing control related to notification based on the anomaly detection result.

10. A program that causes a computer to execute control processing including: an image acquisition step of acquiring an image captured by a camera along with a camera identifier that identifies the camera; an additional information assignment step of associating additional information corresponding to the camera identifier with the captured image; an input control step of inputting the captured image, the additional information, and an instruction specifying a predetermined output format to a language model that is capable of accepting at least image and text input and outputting at least natural language; a result acquisition step of acquiring an anomaly detection result from the language model in accordance with the output format; and a notification control step of executing control related to notification based on the anomaly detection result.