Information processing device, information processing system, information processing method, and program

Abstract

Provided are an information processing device, an information processing method, an information processing system, and a program capable of outputting appropriate information as a chronology. An information processing device according to an embodiment comprises: a recognition unit that recognizes events using sensor information acquired from a sensor; a report input unit that accepts input of report content from a reporter; and an output unit that outputs a chronology on the basis of the recognition result of the events and the report content.

Description

Information Processing Apparatus, Information Processing System, Information Processing Method, and Program

[0001] The present disclosure relates to an information processing apparatus, an information processing system, an information processing method, and a program.

[0002] Patent Document 1 discloses a system for disaster monitoring and crowd behavior monitoring.

[0003] Japanese Patent Application Laid-Open No. 2019-79216

[0004] The system of Patent Document 1 collects information using various information collection means. Examples of the collected information include image data collected by a camera / video sensor, voice data collected by a microphone, etc., radio wave information collected by a radio wave monitoring device. Further, examples of the collected information include meteorological information collected by a meteorological sensor such as temperature, weather, wind direction, etc., public information published on SNS (social networking service), websites, etc. The system structures the collected information and performs tagging. The system performs individual analysis with the collected information collected by each information collection means as input.

[0005] Furthermore, the system assigns the date and time of occurrence, coordinates, and information unique to the information source to the information extracted by the analysis (single source analysis). The system stores the single source analysis results and models (knowledge) in a database as knowledge integration results. The system displays the search results obtained by searching a plurality of databases in chronological order (time series change).

[0006] In the chronological display, the collected information is displayed in chronological order. Therefore, it is desirable to display appropriate information in chronological order so that the monitor can easily grasp the situation.

[0007] The information processing apparatus according to the present disclosure includes a recognition unit that recognizes an event using sensor information acquired from a sensor, a report input unit that receives an input of a report content from a reporter, and an output unit that outputs a chronology based on the recognition result of the event and the report content.

[0008] The information processing system relating to this disclosure includes a sensor provided for collecting information, a recognition means for recognizing an event using sensor information acquired from the sensor, a terminal for receiving input from a reporter, a report input means for inputting the report content from the terminal, and an output means for outputting a chronology based on the event recognition result and the report content.

[0009] The information processing method for outputting a chronology according to this disclosure uses sensor information acquired from a sensor to recognize an event, accepts input of the report content from the reporter, and outputs a chronology based on the event recognition result and the report content.

[0010] The program relating to this disclosure causes a computer to perform the following steps: to recognize an event using sensor information acquired from a sensor; to receive input of the report content from the reporter; and to output a chronology based on the event recognition result and the report content.

[0011] We can provide information processing devices, information processing systems, information processing methods, and programs that enable efficient monitoring.

[0012] This is a block diagram schematically showing an information processing system. This is a diagram showing an example of a system configuration using an information processing device. This is a diagram showing an example of a system configuration using an information processing device. This is a diagram showing an example of a system configuration using an information processing device. This is a diagram showing an example of a system configuration using an information processing device. This is a diagram showing an example of a system configuration using an information processing device. This is a diagram showing an example of a system configuration using an information processing device. This is a block diagram schematically showing the hardware configuration of an information processing device.

[0013] Embodiment 1 Hereinafter, this embodiment will be described with reference to the drawings. Figure 1 is a schematic block diagram showing the configuration of the information processing device.

[0014] The information processing device 100 comprises a recognition unit 101, a report input unit 102, and an output unit 103. The recognition unit 101 recognizes an event using sensor information acquired from the sensor 300. The report input unit 102 receives input of the report content from the reporter. The output unit 103 outputs a chronology based on the event recognition result and the report content.

[0015] The information processing device 100 can display events recognized using sensor information and reports from reporters in a chronological order. The information processing device 100 can output appropriate information as a chronology, enabling efficient monitoring.

[0016] Embodiment 2 Figure 2 is a schematic diagram showing the overall configuration of the information processing system. The information processing system 1 is a system for monitoring the situation by an observer. The information processing system 1 is used for purposes such as disaster, fire, security, crime prevention, and intruder detection. The information processing system 1 is equipped with an information processing device 10 and a plurality of sensors 30. The information processing system 1 has a terminal 42 used by a reporter 41.

[0017] The information processing device 10 is installed in security rooms, disaster response headquarters, fire stations, police stations, etc. The information processing device 10 processes information collected at disaster sites, security facilities, etc., to display a chronological overview. The information processing device 10 may also have application programs (also called apps) installed to execute each function. Here, the person monitoring the situation through the chronological overview of the information processing device 10 is also called the user. In other words, the user monitors the overall situation by looking at the display device 131 that displays the chronological overview.

[0018] The information processing device 10 is, for example, a server, cloud server, personal computer, or smartphone. The information processing device 10 has hardware such as a processor, memory, and input / output devices. The processor executes a program stored in memory, thereby performing the processing described later.

[0019] The information processing device 10 is not limited to a single physical device. For example, multiple devices having communication functions may cooperate to function as the information processing device 100. The communication function may support multiple communication standards such as Wi-Fi®, Bluetooth®, 5G (fifth generation), and LTE (Long Term Evolution).

[0020] Sensor 30 is installed to collect various types of information from monitoring facilities, disaster sites, etc. Sensor 30 is a camera that captures images of facilities, sites, etc. For example, sensor 30 may be an RGB camera, a monochrome camera, an infrared camera, or a depth camera. Sensor 30 captures moving images. Furthermore, sensor 30 is not limited to a camera, but may also be a distance measuring sensor such as LiDAR (Light Detection And Ranging), or a radar.

[0021] Each of the multiple sensors 30 is assigned an ID (Identifier) ​​to identify it. Sensor 30 transmits video data, along with the date and time of capture and the ID, as sensor information to the information processing device 10. Sensor information includes video data, date and time of capture, and the ID. Sensor information may also include information indicating the location where the information was collected. Alternatively, sensor 30 may include information extracted from video data, rather than the video data itself. Sensor 30 has a communication function for transmitting the collected information data to the information processing device 10.

[0022] The following are examples of sensors 30 for monitoring and guarding a facility. Here, sensors 31, 32, 33, 34, 35, and 36 are shown as examples of sensors 30. Sensors 31 and 32 are cameras such as surveillance cameras and security cameras installed within the facility. Sensor 33 is a camera worn by security guards, etc. Alternatively, security guards may hold sensor 33 in their hands. Or, sensor 33 may be attached to the security guard's helmet or uniform, etc.

[0023] Sensor 34 is a camera mounted on a mobile robot such as a cleaning robot, security robot, or transport robot. Sensor 35 is a camera mounted on a vehicle. This vehicle may be, for example, a vehicle capable of autonomous driving. Sensor 36 is a camera mounted on an aircraft such as a drone. Since sensors 34 to 36 are mounted on the mobile aircraft, they move with the aircraft. Based on commands from the user, mobile aircraft such as vehicles, robots, and aircraft are dispatched to the collection location. The mobile aircraft may move autonomously or be remotely controlled.

[0024] Sensors 31 and 32, such as surveillance cameras, are fixed in predetermined locations within the facility. Therefore, the information processing device 10 can identify the location where information is collected using the IDs of sensors 31 and 32.

[0025] Mobile robots, vehicles, and aircraft estimate their own position using encoders, accelerometers, satellite positioning systems, etc. The position of the mobile object when sensors 34, 35, and 36 capture images becomes the information collection location. Sensors 34, 35, and 36 transmit sensor information, including the collection location, to the information processing device 10. Of course, sensors 34, 35, and 36 themselves may also estimate their own position.

[0026] The sensor 33 worn by security guards or other personnel may estimate its own position, similar to a robot. Alternatively, the security guard or other personnel may input the collection location. The sensor 33 transmits sensor information, including the collection location, to the information processing device 10. Furthermore, the sensor 33 transmits the collection time, indicating the date and time of the image capture, to the information processing device 10.

[0027] Reporters 41 include security guards, facility staff, firefighters, police officers, civil servants, and local government employees. Reporters 41 patrol facilities and collect information. Alternatively, reporters 41 are dispatched to locations where an anomaly has been detected to collect information. Reporters 41 possess terminals 42 such as smartphones, tablet computers, and wireless devices. Terminals 42 are mobile communication terminals capable of sending and receiving data with the information processing device 10.

[0028] Terminal 42 has input devices such as a keyboard, touch panel, and microphone. Terminal 42 has a communication device for sending and receiving data to and from the information processing device 10. Terminal 42 may also have a function to estimate its own position using a satellite positioning system. In this case, the terminal 42's own position becomes the collection position. Furthermore, a camera mounted on terminal 42 may function as a sensor 30. For example, terminal 42 may photograph a facility or the like with its camera. Terminal 42 may transmit the captured data to the information processing device 10.

[0029] Reporter 41 makes a report by operating terminal 42. For example, they input the report content by operating a touch panel. The report content may be, for example, a report stating that a suspicious person was spotted. Alternatively, the report content may be a report stating that no abnormalities were found. Of course, reporter 41 may also input the report content by voice input. For example, reporter 41 may use voice input into the microphone to make a report.

[0030] When reporter 41 inputs a report, terminal 42 transmits the report content, including the report text, to the information processing device 10. The report text input by reporter 41 may also include information indicating the collection time and collection location. Alternatively, terminal 42 may add information indicating the collection time and collection location to the report content. Terminal 42 transmits the report content, including the report text, collection time, and collection location, to the information processing device 10.

[0031] The information processing device 10 includes a recognition unit 11, a report input unit 12, and an output unit 13. The recognition unit 11 acquires sensor information from the sensor 30. The recognition unit 11 recognizes events using the sensor information acquired from the sensor 30. For example, the sensor information includes video data. The recognition unit 11 detects events and other phenomena by performing image processing on the video data. The recognition unit 11 outputs the occurred events as recognition results to the output unit 13. The recognition unit 11 recognizes people and objects included in the video data.

[0032] The recognition unit 11 can detect the presence of new people or objects by comparing past and present video footage. This allows the recognition unit 11 to identify unidentified people or objects, thereby enabling the detection of suspicious individuals or objects at the scene.

[0033] For example, if the recognition unit 11 detects a suspicious person or object, it outputs a recognition result indicating that a suspicious person or object has been detected. Alternatively, if it does not detect a suspicious person or object, it outputs a recognition result indicating that there is no problem. The recognition unit 11 can output recognition results based on pre-registered template documents. For example, the recognition unit 11 reads a template document corresponding to the recognition result and outputs it.

[0034] Alternatively, the recognition unit 11 may have a machine learning model that takes video data as input and outputs a recognition result. The recognition unit 11 has an AI (Artificial Intelligence) video recognition model. For example, the machine learning model is generated by machine learning using a dataset containing multiple training data. The recognition unit 11 inputs video data into the pre-generated machine learning model. The machine learning model outputs a recognition result. The recognition result may be text data indicating an event. For example, the machine learning model may output as a recognition result that it has detected a suspicious person or object. Alternatively, the machine learning model may output as a recognition result that it has not detected a suspicious person or object.

[0035] The recognition unit 11 can detect whether or not there are abnormalities in the installed equipment or facilities. The recognition unit 11 can also recognize the extent of damage in the event of a disaster. Alternatively, the recognition unit 11 can detect intruders or intruder objects. The recognition unit 11 may also use a Visual Language Model (VLM). In that case, the recognition unit 11 takes video as input and outputs text, words, etc., related to the content of the video from the Visual Language Model.

[0036] The report input unit 12 receives input of report content from the reporter 41. For example, the report input unit 12 may have a communication interface for sending and receiving data with the terminal 42. The reporter 41 inputs a report text appropriate to the situation at the site using the terminal 42. The report content includes the text data of the report text. Furthermore, the date and time of report input becomes information indicating the collection time. The location of report input becomes information indicating the collection location. The report input unit 12 receives report content data, including the report text, collection time, and collection location, from the terminal 42. The report input unit 12 then outputs the report content to the output unit 13.

[0037] The output unit 13 performs chronological output based on the recognition results and reported content. The chronological output is time-series data in which the recognition results and reported content are arranged in chronological order. Therefore, the output unit 13 displays the recognition results and reported content in order of collection time. The output unit 13 has a display device or display interface for performing chronological display. Specifically, the output unit 13 outputs the chronology to the display device 131 via a general-purpose interface such as HDMI (High-Definition Multimedia Interface). The display device 131 is an LCD display or a projector. Of course, the output unit 13 may have a wireless connection interface instead of a wired connection such as HDMI (registered trademark).

[0038] Here, the display device 131 is a separate device from the information processing device 10, but the information processing device 10 may also have the display device 131. In other words, the information processing device 10 and the display device 131 may be an integrated device. For example, the information processing device 10 may be a laptop computer with a display device 131. The output unit 13 performs display output for displaying the chronology. Display output includes the process of outputting a signal for display, or the process of displaying it on the display device 131.

[0039] Figure 2 shows an example of the display screen for the chronology display on the display device 131. In the chronology display, the recognition results and report contents are displayed in chronological order. Specifically, the report contents and recognition results are displayed in order of collection time. Here, each item of the report contents and the recognition result of the event is called a chronology item. For example, in Figure 2, four chronology items are displayed. Each chronology item includes the collection time, collection location, and collected information. The most recent chronology item is displayed at the bottom of the display screen. Furthermore, when new report contents or recognition results are input to the output unit 13, the previous chronology items move up one level, and the most recent chronology item is added to the bottom.

[0040] In Figure 2, the recognition unit 11 recognizes that there is no particular problem from the video captured at 12:22:31 by camera #1 installed in area C on the first floor. Therefore, the display device 131 displays "12:22:31 / 1st floor area C / Camera #1: No particular problem" as the recognition result. Subsequently, the recognition unit 11 recognizes the presence of a person and a suspicious object from the video captured at 12:23:32 by camera #3 installed in area E on the third floor. Therefore, the display device 131 displays "12:23:32 / 3rd floor area E / Camera #3: Person detected. Suspicious object detected." as the recognition result.

[0041] Next, the recognition unit 11 detects a person from the video captured at 12:23:32 by robot #1 while it was moving around area B on the second floor. Therefore, the display device 131 displays "12:23:32 / 2nd floor area B / robot #1: person detected" as the recognition result. Subsequently, security guard #2, who was patrolling area C, reports at 12:24:31 that he has spotted a person running. Therefore, the display device 131 displays "12:24:31 / area C / security guard #2: person running detected" as the report content. Here, "12:24:31 / area C / security guard #2: person running detected" is the latest chronology item.

[0042] In this way, the output unit 13 displays a chronology in which the recognition result by the recognition unit 11 and the report content by the reporter 41 are mixed. The output unit 13 outputs the chronology so that the report content is appended before or after the recognition result. That is, the display device 131 chronologically displays information including the recognition result and the report content. Therefore, by visually recognizing the display device 131 on which the chronology display is being performed by a supervisor or the like, the overall situation can be easily grasped. Further, the output unit 13 may output the chronology to the terminal 42 of the reporter 41 such as a security guard. Thereby, the reporter 41 can take appropriate actions while confirming the latest situation.

[0043] Embodiment 3 FIG. 3 is a block diagram schematically showing the overall configuration of the information processing system 1. In FIG. 3, a metadata storage unit 14 and a summary generation unit 15 are added to the configuration of FIG. 2. Since the configuration of the information processing apparatus 10 is the same as that in FIG. 2, the description will be omitted as appropriate.

[0044] The recognition unit 11 generates metadata used for summary generation. The metadata storage unit 14 stores metadata indicating the recognition result in the recognition unit 11. The summary generation unit 15 generates a summary of the recognized event based on the metadata. The summary generation unit 15 has, for example, a language model such as an LLM (Large Language Models). The summary generation unit 15 generates a summary sentence summarizing the recognized event. For example, the summary generation unit 15 generates a summary sentence summarizing the current event by referring to the chronology items of past events. The output unit 13 outputs the summary and the report content as a chronology.

[0045] The metadata storage unit 14 accumulates the recognition result in the recognition unit 11 as metadata. That is, the metadata storage unit 14 stores metadata indicating past recognition results. The metadata is data indicating the recognition result and includes information on the collection position, collection location, and collection content. The metadata may include important phrases and words in the recognition result in the recognition unit 11. Alternatively, the metadata storage unit 14 may store metadata extracted from the summary generated by the summary generation unit 15.

[0046] The summary generation unit 15 generates a summary using the metadata. The summary generation unit 15 can generate a summary of events including the currently occurring events and the events that occurred in the past. The summary generation unit 15 can generate a summary including the relationship between events before and after in time. The output unit 13 outputs the recognized events, reports, or summaries in chronological order. By combining the video recognition AI in the recognition unit 11 and the LLM in the summary generation unit 15, the information processing device 10 can generate an appropriate summary.

[0047] The summary generation unit 15 can generate a summary sentence that summarizes a plurality of events. That is, the summary generation unit 15 generates one chronological item that summarizes the recognition results of a plurality of events. Hereinafter, an example of a summary will be described. Assume that the recognition unit 11 recognizes that there were a person and a suspicious object from the video captured at 12:22:32 by the camera #3 installed in the 3rd floor E area. In this case, the recognition unit 11 generates metadata indicating that a person was detected and metadata indicating that a suspicious object was detected, and writes them to the metadata storage unit 14.

[0048] The summary generation unit 15 uses this metadata to generate a summary sentence "A person installed a suspicious object". That is, since a person and an object were discovered almost simultaneously, the summary generation unit 15 presumes that the person installed the suspicious object. Then, the output unit 13 causes the display device 131 to display a chronological item including the summary sentence. Here, as one chronological item, "12:23:32 / 3rd floor E area / Camera #3: A person installed a suspicious object" is displayed.

[0049] In this way, the summary generation unit 15 generates a summary sentence that summarizes a plurality of events. The summary generation unit 15 generates a summary from metadata and the like. The output unit 13 outputs the summary sentence as a chronological item. By doing so, the user can easily grasp the situation. For example, the user can grasp a plurality of events together. Therefore, the user can easily grasp the situation.

[0050] Embodiment 4 Figure 4 is a schematic block diagram showing the overall configuration of the information processing system 1. In Figure 4, the information processing device 10 performs a process to track objects included in the video. For example, the recognition unit 11 identifies people and objects shown in the video as objects. The information processing device 10 then tracks the objects based on the video data from each sensor 30. The summary generation unit 15 generates a summary that includes the object tracking results.

[0051] The recognition unit 11 extracts features of objects and people. The features include information indicating the type of object, such as whether it is a person or an object. Objects may be distinguished as, for example, drones, vehicles, or mobile objects. Furthermore, the features include information indicating a person's clothing, multiple colors, gender, height, and body shape. Alternatively, the recognition unit 11 may perform facial recognition of a person and use its feature quantities as features. The features of an object include information indicating the object's size, color, shape, model, and product name. The recognition unit 11 then identifies the object based on its features. Based on the identification result, the recognition unit 11 assigns an ID to the object. In this way, the recognition unit 11 can identify the object, and therefore can track objects detected by multiple sensors 30.

[0052] For example, the recognition unit 11 recognizes features such as a person's clothing and face. The metadata storage unit 14 stores the features as metadata for each target. In other words, the metadata is data that indicates the features of each target. By referring to the metadata indicated by past recognition results, targets can be tracked. In other words, the summary generation unit 15 generates a summary of targets whose features match those of the current recognition result and past recognition result, treating them as the same target to be tracked.

[0053] In the example of chronology display shown in Figure 4, one chronology item displays "12:23:32 / 3rd floor E area / Camera #3: Person #1 placed a suspicious object". At this time, since the person detected by camera #3 in the 3rd floor E area is assigned an ID, the chronology item displays "Person #1". The recognition unit 11 extracts the person's characteristics from the video captured by camera #3. The metadata storage unit 14 stores metadata indicating the person's characteristics.

[0054] Suppose the same person #1 then moves to area B on the second floor. At 12:23:42, robot #1, moving through area B on the second floor, captures a picture of person #1 using sensor 30. The recognition unit 11 detects person #1 from the image captured at 12:23:42 by sensor 30 of robot #1 as it moves through area B on the second floor. The recognition unit 11 performs recognition processing on the images from both sensors 30. If the features extracted by the recognition unit 11 match, the summary generation unit 15 generates a summary sentence identifying them as the same person #1.

[0055] Next, the output unit 13 outputs a chronology including a summary. Then, the display device 131 displays a chronology item that reads "12:23:42 / 2nd floor B area / Robot #1: Person #1 who placed a suspicious object in E area was found." In this way, the output unit 13 can output appropriate information as a chronology. This makes it easier for the user to understand the situation, enabling more efficient monitoring.

[0056] Furthermore, at 12:24:31, reporter 41 reported finding the fleeing person #1 in area C. Therefore, the report should read, "12:24:31 / Area C / Security Guard #2: Fleeing person #1 found." This allows the user to identify the entry and escape routes of person #1. Thus, the user can quickly grasp the situation and take appropriate action.

[0057] Embodiment 5 Figure 5 is a block diagram schematically showing the overall configuration of the information processing system 1. In Figure 5, a report storage unit 16 is added to the configuration in Figure 4. Note that the basic configuration of the information processing system 1 is the same as the configuration described above, so a description is omitted.

[0058] The report storage unit 16 stores the report content entered into the report input unit 12. In other words, the report storage unit 16 accumulates past report content. The reporter 41 inputs a report that includes the characteristics of the subject. For example, the reporter inputs information indicating the characteristics of the subject, such as clothing, body type, and face, and submits a report. The report storage unit 16 saves the characteristics for each subject.

[0059] The summary generation unit 15 generates a summary based on the report content stored in the report storage unit 16 and the metadata stored in the metadata storage unit 14. In this way, the summary generation unit 15 can generate a summary by referring to past report content and past events. For example, the summary generation unit 15 generates a summary by referring to report content and metadata from the present to a certain period of time ago. In this way, the output unit 13 can output appropriate information as a chronology, enabling efficient monitoring.

[0060] In the example of chronology display shown in Figure 5, the display device 131 displays a chronology item that reads, "12:23:42 / 2nd floor B area / Robot #1: Person #1 who placed a suspicious object in E area was found." Upon seeing this chronology item, the user instructs the security guards to close the entrances and exits. Specifically, the user instructs security guard group 5 to be dispatched to the 2nd floor to close the entrances and exits. The user, as the reporter, inputs a report indicating that instruction. Therefore, the report indicating that instruction is displayed chronologically in chronology item C1.

[0061] The display device 131 displays chronology item C1, which reads, "12:23:45 / 2nd floor / Command: Dispatch security guard group 5 to the 2nd floor and close the entrance." The report text here may be a summary text generated by the summary generation unit 15. In other words, the summary generation unit 15 generates a summary text that summarizes the user's command.

[0062] Next, let's assume that at 12:24:31, security guard #2, who was guarding area C on the first floor, discovered a person. Here, since the person was discovered in an area other than the closed area, the summary generation unit 15 generates a summary of this person as a different person #2. Alternatively, because the characteristics do not match those of person #1, the summary generation unit 15 generates a summary of this person as a different person #2. Therefore, the display device 131 displays the summary text as, "12:24:31 / 1st floor area C / Robot #1: Discovered person #2".

[0063] Next, security guard #4, who was dispatched to the second floor, discovers a person in area G on the second floor at 12:27:11. At this time, security guard #4 reports that he has discovered a person. When the report input unit 12 receives the input of the report content, the report storage unit 16 saves the report content. The summary generation unit 15 generates a summary based on the saved report content. The summary is a warning message to users and security guards because the person is in an area with a closed entrance.

[0064] Specifically, the display device 131 displays the following as chronology item C2: "12:27:11 / 2nd floor G area / Security guard #4: Person #3 spotted. The fifth group, which is closed off, is in an area with a blocked entrance / exit, so be careful."

[0065] In this way, the summary generation unit 15 generates a summary by combining the recognition results and the information from the report. This allows the output unit 13 to output more appropriate information in a chronological order. This makes it easier for the user to understand the situation, enabling more efficient monitoring. Of course, the output unit 13 may also output the recognition results or the report content before summarization as chronological items.

[0066] Figure 6 is a flowchart illustrating the information processing method. First, the recognition unit 11 recognizes an event using sensor information from the sensor 30 (S11). For example, the recognition unit 11 outputs a recognition result by inputting video data into the AI ​​generation model. The metadata storage unit 14 stores metadata corresponding to the recognition result of the event (S12). This allows metadata about past events to be accumulated. The report input unit 12 receives input of the report content from the reporter (S13). The report storage unit 16 stores the report content (S14). This allows report content to be accumulated.

[0067] The summary generation unit 15 generates a summary based on the stored metadata and report content (S15). The output unit 13 outputs the summary as a chronology (S16). The display device 131 displays the summary text as a chronology item. In this way, the output unit 13 can output more appropriate information as a chronology. This makes it easier for the user to understand the situation, enabling more efficient monitoring.

[0068] Embodiment 6 Figure 7 is a block diagram schematically showing the overall configuration of the information processing system 1. In Figure 7, a matching unit 19 is added to the configuration of Figure 5. The basic configuration of the information processing system 1 is the same as the configuration described above, so its explanation will be omitted as appropriate.

[0069] The output unit 13 outputs the chronology to the alignment unit 19. The alignment unit 19 aligns the chronology items. Then, the summary generation unit 15 updates the summary based on the alignment results. Furthermore, the alignment unit 19 may update the recognition results and metadata. Specifically, the alignment unit 19 extracts information that should be added to or corrected from past chronology items. The alignment unit 19 compares the extracted information with the information stored in the metadata storage unit 14.

[0070] If the extracted information is not stored in the metadata storage unit 14, the recognition unit 11 performs recognition processing using the extracted information. The summary generation unit 15 generates a summary using the extracted information. The metadata storage unit 14 stores metadata based on the extracted information. For example, the metadata storage unit 14 updates past metadata according to the extracted information. Furthermore, the summary generation unit 15 generates a summary based on the recognition results using the extracted information. For example, it updates past summaries according to the extracted information. As a result, past recognition results, metadata, or summaries are modified according to the alignment results. In other words, past recognition results, metadata, or summaries are updated to align with the newly extracted information in the alignment results.

[0071] Figure 7 shows an example of chronology display on the display device 131. In Figure 7, the screen of the display device 131 before the update is referred to as the display screen 132, and the screen of the display device 131 after the update is referred to as the update screen 133. In other words, when the chronology items shown on the display screen 132 are updated based on the alignment results, they will be shown on the update screen 133.

[0072] From the video captured at 12:22:31 by camera #1 installed in area C, the recognition unit 11 recognizes that there are no particular problems. Therefore, as shown on the display screen 132, the display device 131 displays "12:22:31 / Area C / Camera #1: No particular problems" as the recognition result. Subsequently, from the video captured at 12:23:32 by camera #3 installed in area E, the recognition unit 11 recognizes that drone #1 is flying. Therefore, the display device 131 displays "12:23:32 / Area E / Camera #3: Drone #1 is flying" as the recognition result.

[0073] The recognition unit 11 recognizes that drone #1 is flying based on the video captured at 12:23:42 by the sensor of robot #1 as it travels through area B. Therefore, the display device 131 displays "12:23:42 / Area B / Robot #1: Drone #1 is flying" as the recognition result.

[0074] From the video footage captured by camera #4 at 12:24:41 in area A, the recognition unit 11 recognizes that drone #1 is flying. Furthermore, the recognition unit 11 identifies the drone's model as XXX. The summary generation unit 15 generates a summary indicating that the drone's model has been identified. Therefore, the display device 131 displays the summary as, "12:24 / Area A / Camera #4: Drone #1 is flying. Model identified as XXX, a surveillance drone." In other words, the drone's model indicates that it is a surveillance drone. Thus, the summary generation unit 15 generates a summary including the drone's model and purpose.

[0075] Since the drone model and purpose have been identified, the matching unit 19 updates the past chronology entries. For example, the updated chronology entry displayed on the display device 131 is: "12:23:32 / Area E / Camera #3: Drone #1 may have flown and photographed the area around Area E." The updated chronology entry displayed on the display device 131 is: "12:23:42 / Area B / Robot #1: Drone #1 may have flown and photographed Area B from Area E."

[0076] Thus, having identified drone #1 as a surveillance drone, the matching unit 19 outputs a command to the recognition unit 11 and the metadata storage unit 14 to re-recognize the area that drone #1 can photograph. The recognition unit 11 re-recognizes the event based on the update result. The metadata storage unit 14 stores metadata indicating the updated information. The summary generation unit 15 updates the content of the relevant chronology items based on the updated information.

[0077] This allows for the chronological display of more appropriate information. For example, if sensor 30 photographs a distant aircraft, the image quality is low, making it difficult to identify the aircraft. Specifically, the recognition unit 11 cannot identify the type of aircraft. Alternatively, the recognition unit 11 cannot determine whether the aircraft is a bird or an artificial object such as a drone. Similarly, if reporter 41 spots a distant aircraft, it is difficult to identify the aircraft.

[0078] In such cases, the information processing device 10 can identify the aircraft based on the video footage and reports acquired after the aircraft approaches. Then, the matching unit 19 updates the past chronology entries using the newly identified information. The summary generation unit 15 may generate a summary using the extracted information. The output unit 13 displays the updated chronology entries. For example, if the aircraft is found to be a bird, the chronology entries are modified to indicate that there is no particular problem as it is a bird. Also, if the aircraft is identified as a transport drone, the user dispatches security guards or mobile units to the area through which the transport drone passed. The security guards or mobile units then investigate whether there are any suspicious objects. In this way, the output unit 13 can output more detailed information as a chronology. This makes it easier for the user to understand the situation, enabling more efficient monitoring.

[0079] Embodiment 7 Figure 8 is a block diagram schematically showing the overall configuration of the information processing system 1. In Figure 8, a question input unit 21, an answer generation unit 22, and an answer output unit 23 are added to the configuration of Figure 5. The basic configuration of the information processing system 1 is the same as that shown in Figures 1 to 5, so the explanation will be omitted as appropriate.

[0080] The question input unit 21 receives questions from users and reporters 41. For example, the display device 131 displays a question input field 134 and an answer output field 135. Users enter their questions as text in the question input field 134. The answer output field 135 outputs the answer as text.

[0081] The user enters the question into the question input field 134 using an input device such as a keyboard or touch panel. Alternatively, the user may input the question by voice using a microphone or the like. The question input unit 21 receives the question input for the question input field 134.

[0082] The answer generation unit 22 generates answers to user questions. For example, the answer generation unit 22 generates answers using a language model such as LLM. The answer generation unit 22 may also have an AI model such as a chatbot. The AI ​​model takes the question text as input and outputs the answer text. The AI ​​model can be generated in advance using machine learning. Alternatively, if the user specifies a video they want to watch, the answer generation unit 22 may output that video.

[0083] For example, an AI model is generated based on training data that includes questions and answers as a dataset. The answer to a question may be a follow-up question to the user. In other words, if the user's question is unclear, the answer generation unit 22 may generate a follow-up question to the user.

[0084] The answer output unit 23 outputs the answer to the display device 131. The answer output unit 23 has a display interface similar to the output unit 13. The answer output unit 23 displays the answer on the display device 131. Of course, the answer output unit 23 may also output the answer by voice. As a result, the text sentence of the answer is displayed in the answer output field 135.

[0085] For example, a user might input a question such as, "What did the person the security guard found in area C do?" When the question input unit 21 receives this question, the answer generation unit 22 generates an answer to the question. The answer generation unit 22 generates the answer by referring to past metadata, recognition results, report content, and summaries. For example, the answer generation unit 22 extracts information from metadata and report content and performs inference. The answer generation unit 22 may also have a generating AI or the like.

[0086] When the answer output unit 23 outputs an answer, the answer output field 135 displays the answer text. This makes it easier for the user to understand the situation. The user can quickly find out what they want to know and the relationships between past events.

[0087] An example of a GUI (Graphical User Interface) for chronology display is described below. Figure 9 is a schematic diagram showing a display window 136 for displaying chronology items. The display window 136 includes a status report field 137 and an action display field 138. Furthermore, the display screen shows an input / output field 139 for inputting and outputting questions and answers.

[0088] The status report section 137 displays the report text, recognition results, or summary as chronology items. Furthermore, the status report section 137 is accompanied by video footage captured by the sensor 30 and terminal 42. By displaying the chronology items together with the video footage in this way, users can more easily understand the situation.

[0089] For example, at 12:23:32, the recognition unit 11 recognizes that person #1 has placed a suspicious object in area E on the third floor. Subsequently, camera #3 detects person #1 in area B on the second floor. Further down the line, security guard #2 reports finding person #1 in area C. The situation report column 137 displays these reports and recognition results in chronological order. Of course, the summary generation unit 15 may also automatically generate a summary from the video.

[0090] The action display area 138 shows the actions that security guards and others should take. The output unit 13 outputs action information indicating the actions to take in response to the event, based on the recognition results and report contents. The action information may, for example, indicate that an alarm should be activated in area B on the second floor. Alternatively, the action information may indicate that further action is needed in area C. Specifically, the action information may include instructions to dispatch security guards, drones, robots, etc., to area C or its surroundings.

[0091] Thus, the output unit 13 may output action information based on the recognition results and reported content. Action information is information indicating the action to take in response to an event. For example, the output unit 13 may output a command as a chronology item to dispatch security guards or drones to the area where a suspicious person or object was found. Alternatively, the summary generation unit 15 may generate a summary sentence that includes commands such as closing entrances and activating security systems. In this way, the user can take an appropriate response to the situation.

[0092] Figure 10 shows an example of the hardware configuration of an information processing device 10 according to several embodiments. In the example shown in the figure, the information processing device 10 includes a memory 191, a processor 192, a communication interface 193, and a user interface 194. A terminal 42 and other components may also have a similar configuration.

[0093] The communication interface 193 is used to communicate with other devices that make up the system (e.g., user terminals, digital forensic tools). The communication interface 193 is an interface for connecting to other devices such as sensors 30 and terminals 42 via wired communication means or wireless communication means.

[0094] The user interface 194 includes an input device for receiving input from the user. The user interface 194 includes input devices such as a keyboard, mouse, microphone, and touch panel. The user interface 194 includes an output device for providing output to the user. The user interface 194 includes an output unit 13 for providing chronological output. The user interface 194 may also include a speaker for outputting alarm sounds, etc.

[0095] The processor 192 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 192 may include multiple processors.

[0096] Memory 191 is comprised of a combination of volatile and non-volatile memory. Memory 191 may include a plurality of physically independent memory devices. Volatile memory is, for example, Static Random Access Memory (SRAM) or Dynamic RAM (DRAM), or a combination thereof. Non-volatile memory is Mask Read Only Memory (MROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, hard disk drive, or solid-state drive (SSD), or any combination thereof. Memory 191 may include storage located away from the processor 192. In this case, the processor 192 may access memory 191 via an I / O (Input / Output) interface, which is not shown.

[0097] The memory 191 may store a software module (computer program) containing a set of instructions and data for performing processing by the information processing device 10 as described in some of the embodiments described above. In some implementations, the processor 192 may be configured to read the software module from the memory 191 and execute it to perform the processing of the information processing device 10 as described in some of the embodiments described above.

[0098] The program described above includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more of the functions described in this disclosure. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, SSD (Solid-State Drive), or other memory technologies, CD-ROM, DVD (Digital Versatile Disc), Blu-ray® disc, or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0099] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure can be made as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0100] Furthermore, each drawing is merely illustrative to illustrate one or more embodiments. Each drawing may be associated not only with one specific embodiment but also with one or more other embodiments. As those skilled in the art will understand, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings, for example, to create embodiments not explicitly shown or described. Not all features or steps shown in any one drawing to illustrate an exemplary embodiment are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be changed as appropriate.

[0101] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.

[0102] Some or all of the above embodiments may also be described as follows, but are not limited to the following: (Note 1) An information processing device comprising: recognition means for recognizing an event using sensor information acquired from a sensor; report input means for receiving input of report content from a reporter; and output means for outputting a chronology based on the recognition result of the event and the report content. (Note 2) The information processing device according to Note 1, further comprising: metadata storage means for storing metadata corresponding to the recognition result of the recognition means; and summary generation means for generating a summary of an event based on the metadata, wherein the output means outputs the chronology based on the summary of the event and the report content. (Note 3) The information processing device according to Note 2, wherein the recognition means recognizes the features of an object in an image captured by the sensor, and associates the object in the image with the object included in the report content based on the features. (Note 4) The information processing device according to Note 3, wherein the recognition means tracks an object in an image captured by the sensor, and the summary generation means generates a summary including information about the tracked object. (Note 5) The information processing device according to Note 2, further comprising a report storage means for storing the report contents, wherein the summary generation means generates a summary based on the report contents and the recognition results over a predetermined period. (Note 6) The information processing device according to Note 2, further comprising a matching means for matching the recognition results and the report contents, wherein the summary generation means updates the summary based on the matching results from the matching means. (Note 7) The information processing device according to Note 2, wherein the output means outputs action information indicating an action to take in response to the event based on the recognition results and the report contents. (Note 8) The information processing device according to Notes 1 to 7, further comprising a question input means for receiving input of a question from a user, an answer generation means for generating an answer to the question, and an answer output means for outputting the answer. (Note 9) The information processing device according to any one of Notes 1 to 8, wherein the recognition means has a machine learning model that takes the sensor information as input and outputs the recognition result of an event.(Note 10) An information processing system comprising: a sensor provided for collecting information; a recognition means for recognizing an event using sensor information acquired from the sensor; a terminal for receiving input from a reporter; a report input means for receiving report content from the terminal; and an output means for outputting a chronology based on the event recognition result and the report content. (Note 11) An information processing method for recognizing an event using sensor information acquired from a sensor, receiving input of report content from a reporter, and outputting a chronology based on the event recognition result and the report content. (Note 12) A program for causing a computer to perform the steps of: recognizing an event using sensor information acquired from a sensor; receiving input of report content from a reporter; and outputting a chronology based on the event recognition result and the report content.

[0103] Some or all of the elements (e.g., configuration and function) described in Appendices 2 to 9 that are subordinate to Appendice 1 may also be subordinate to Appendices 11 and 12 in the same manner as those described in Appendices 2 to 9. Some or all of the elements described in any appendice may be applied to various hardware, software, recording means, systems, and methods for recording software.

[0104] This application claims priority based on Japanese Patent Application No. 2024-190738, filed on 30 October 2024, and incorporates all of its disclosures herein.

[0105] 1 Information Processing System 10 Information Processing Device 11 Recognition Unit 12 Report Input Unit 13 Output Unit 14 Metadata Storage Unit 15 Summary Generation Unit 19 Alignment Unit 21 Question Input Unit 22 Answer Generation Unit 23 Answer Output Unit 30 Sensor 41 Reporter 42 Terminal 100 Information Processing Device 101 Recognition Unit 102 Report Input Unit 103 Output Unit 131 Display Device 132 Display Screen 133 Update Screen 134 Question Input Field 135 Answer Output Field 136 Display Window 137 Status Report Field 300 Sensor

Claims

1. An information processing device comprising: a recognition means for recognizing an event using sensor information acquired from a sensor; a report input means for receiving input of report content from a reporter; and an output means for outputting a chronology based on the event recognition result and the report content.

2. The information processing apparatus according to claim 1, further comprising: metadata storage means for storing metadata corresponding to the recognition result of the recognition means; and summary generation means for generating an event summary based on the metadata, wherein the output means outputs the chronology based on the event summary and the report content.

3. The information processing apparatus according to claim 2, wherein the recognition means recognizes the features of an object in the image captured by the sensor, and associates the object in the image with the object included in the report content based on the features.

4. The information processing apparatus according to claim 3, wherein the recognition means tracks an object in the image captured by the sensor, and the summary generation means generates a summary containing information about the tracked object.

5. The information processing apparatus according to claim 2, further comprising a report storage means for storing the report contents, wherein the summary generation means generates a summary based on the report contents and the recognition results over a predetermined period.

6. The information processing apparatus according to claim 2, further comprising a matching means for matching the recognition result with the report content, wherein the summary generation means updates the summary based on the matching result of the matching means.

7. The information processing apparatus according to claim 2, wherein the output means outputs action information indicating an action to take in response to the event based on the recognition result and the report content.

8. An information processing apparatus according to any one of claims 1 to 7, further comprising: a question input means for receiving a question from a user; an answer generation means for generating an answer to the question; and an answer output means for outputting the answer.

9. The information processing apparatus according to any one of claims 1 to 8, wherein the recognition means has a machine learning model that takes the sensor information as input and outputs the recognition result of an event.

10. An information processing system comprising: a sensor provided for collecting information; a recognition means for recognizing an event using sensor information acquired from the sensor; a terminal for receiving input from a reporter; a reporting input means for inputting the content of the report from the terminal; and an output means for outputting a chronology based on the recognition result of the event and the content of the report.

11. The information processing system according to claim 10, further comprising: metadata storage means for storing metadata corresponding to the recognition result of the recognition means; and summary generation means for generating an event summary based on the metadata, wherein the output means outputs the chronology based on the event summary and the report content.

12. The information processing system according to claim 11, wherein the recognition means recognizes the features of an object in the image captured by the sensor, and associates the object in the image with the object included in the report content based on the features.

13. The information processing system according to claim 12, wherein the recognition means tracks an object in an image captured by the sensor, and the summary generation means generates a summary containing information about the tracked object.

14. An information processing method that recognizes an event using sensor information obtained from a sensor, accepts input of the report content from the reporter, and outputs a chronology based on the event recognition result and the report content.

15. The information processing method according to claim 14, which involves storing metadata corresponding to the recognition result, generating an event summary based on the metadata, and outputting the chronology based on the event summary and the report content.

16. The information processing method according to claim 15, which recognizes the characteristics of an object in an image captured by the sensor and associates the object in the image with the object included in the report based on the characteristics.

17. The information processing method according to claim 16, which tracks an object in an image captured by the sensor and generates a summary including information about the tracked object.

18. A program that causes a computer to perform the following steps: to recognize an event using sensor information acquired from a sensor; to receive input of a report from a reporter; and to output a chronology based on the event recognition result and the report content.

19. The program according to claim 18, which causes the computer to perform the steps of storing metadata corresponding to the recognition result, generating an event summary based on the metadata, and outputting the chronology based on the event summary and the report content.

20. The program according to claim 19, which recognizes the features of an object in an image captured by the sensor and associates the object in the image with the object included in the report based on the features.

Images