Information processing device and information processing method

The information processing device addresses the limitations of existing technologies by updating event scores based on user interactions and responses, improving the accuracy of memory recall and event extraction.

WO2026234823A1PCT designated stage Publication Date: 2026-11-12NT T INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2025/017141
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-05-09
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing technologies fail to accurately update event scores reflecting changes in user impressions over time and do not consider memory retention and engagement levels, limiting their effectiveness in memory recall and event extraction.

Method used

An information processing device that includes an acquisition unit, estimation unit, and score update unit to gather user information, estimate memory retention and engagement levels, and update event scores based on user interactions and responses.

Benefits of technology

The device accurately updates event scores by reflecting fluctuations in user memory strength and engagement, enhancing the accuracy of event scoring and recall.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2025017141_12112026_PF_FP_ABST
    Figure JP2025017141_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An information processing device (1) holds information related to a plurality of candidate events used for information presentation, and comprises an acquisition unit, an estimation unit, and a score update unit. The acquisition unit acquires one or both of user presentation information, which is information presented by a user, and response information indicating the user's response to information presentation using information related to a candidate event. The estimation unit estimates one or more of the degree of memory retention, the degree of understanding, and the degree of engagement of the user on the basis of one or both of the response information and the user presentation information when a related candidate event is retrieved from among the plurality of candidate events on the basis of the user presentation information. The score update unit updates one or both of the score of the retrieved candidate event and the score of the candidate event on which the response has been based, on the basis of the estimation result by the estimation unit.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] Conventionally, with the spread of smart devices and other technologies, research has been conducted to extract information necessary for memory recall, monitoring, or memorable experiences from the increasing amount of personal behavioral history (see, for example, Non-Patent Documents 1-3).

[0003] Wang Sihan et al., "Memory Recall Support through Life Log Ranking," Proceedings of the 86th National Conference, Information Processing Society of Japan, 2024, 2024(1), pp.467-468. Matsuo Yuji et al., "A Method for Extracting Living Areas Focusing on Places of Stay and Routes," IEICE Technical Report, Institute of Electronics, Information and Communication Engineers, 2015, 114(500), pp.77-82. Hayashi Yukifumi, "Memorable Tourist Experiences: An Exploratory Study of Experiential Content Using Quantitative Text Analysis," Journal of Social Psychology, Japanese Society of Social Psychology, 2024, Vol. 40, No. 2, pp.162-170.

[0004] However, the technology described in Non-Patent Document 1 only calculates a score for each event once, and does not reflect subsequent changes in the user's impression of the event. Furthermore, Non-Patent Document 2 is limited to a technology for extracting living areas and does not consider scoring events. In addition, the technology described in Non-Patent Document 3 is limited to analysis based on psychology and has not reached the stage of systematization.

[0005] The present invention has been made in view of the above, and aims to update event scores with high accuracy.

[0006] To solve the above-mentioned problems and achieve the objective, the present invention provides an information processing device that holds information related to a plurality of candidate events used for information presentation, and comprises an acquisition unit, an estimation unit, and a score update unit. The acquisition unit acquires either or both of user-presented information, which is information presented by the user, and response information, which indicates the user's response to the information presentation using information related to the candidate events. The estimation unit estimates one or more of the user's memory retention level, comprehension level, and engagement level based on either or both of the user-presented information and the response information when a relevant candidate event is found among a plurality of candidate events based on the user-presented information. The score update unit updates either or both of the score of the found candidate event and the score of the candidate event that was the source of the response, based on the estimation result by the estimation unit.

[0007] The information processing device according to the present invention has the effect of being able to update event scores with high accuracy.

[0008] Figure 1 is a diagram showing an example of the configuration of an information processing device according to the embodiment. Figure 2 is a diagram showing an example of input information used for filtering by an LLM by the acquisition unit in the processing unit of the information processing device according to the embodiment. Figure 3 is a diagram showing an example of the configuration of the location information processing unit in the processing unit of the information processing device according to the embodiment. Figure 4 is a flowchart showing the processing flow of the outside living area determination unit in the location information processing unit shown in Figure 3. Figure 5 is a diagram showing an image of the outside living area determination result by the determination processing unit shown in Figure 3. Figure 6 is a flowchart showing the processing flow of the outside living area event extraction unit in the location information processing unit shown in Figure 4. Figure 7 is a diagram showing an example of a simple algorithm for extracting outside living area events in the outside living area event extraction unit in the location information processing unit shown in Figure 3. Figure 8 is a diagram showing an example of multiple location information from which outside living area events are extracted by the simple algorithm shown in Figure 7. Figure 9 is a diagram showing another example of a simple algorithm for extracting outside living area events in the outside living area event extraction unit in the location information processing unit shown in Figure 3. Figure 10 is a diagram showing an example of multiple location information from which outside living area events are extracted by the simple algorithm shown in Figure 9. Figure 11 is a diagram showing a visualized image of the distance score calculated in the distance score calculation unit shown in Figure 3. Figure 12 is a flowchart showing the processing flow of the event information processing unit shown in Figure 1. Figure 13 is a diagram showing an example of the configuration of the event information processing unit in the processing unit of the information processing device according to the embodiment. Figure 14 is a diagram showing an example of event detection using a threshold by the event detection unit of the event information processing unit shown in Figure 13. Figure 15 is a diagram showing another example of event detection using a threshold by the event detection unit of the event information processing unit shown in Figure 13. Figure 16 is a diagram illustrating the machine learning model used for event detection by the event detection unit of the event information processing unit shown in Figure 13. Figure 17 is a diagram illustrating the machine learning model used for score calculation by the score calculation unit of the event information processing unit shown in Figure 13. Figure 18 is a diagram showing an example of the configuration of the response processing unit shown in Figure 1. Figure 19 is a flowchart showing an example of the processing flow by the first processing unit shown in Figure 18.Figure 20 is a diagram illustrating the processing performed by the first processing unit shown in Figure 18. Figure 21 is a diagram illustrating an example of a score update rule in the first processing unit shown in Figure 18. Figure 22 is a flowchart illustrating an example of the processing flow by the processing unit including the second processing unit shown in Figure 18. Figure 23 is a diagram illustrating the processing performed by the processing unit including the second processing unit shown in Figure 18. Figure 24 is a diagram illustrating an example of a score update rule in the second processing unit shown in Figure 18. Figure 25 is a diagram illustrating another example of a score update rule in the second processing unit shown in Figure 18. Figure 26 is a diagram illustrating an example of a method for generating presentation information by the presentation information generation unit shown in Figure 1. Figure 27 is a diagram illustrating another example of a method for generating presentation information by the presentation information generation unit shown in Figure 1. Figure 28 is a diagram illustrating an example of a GUI presented to the user by the information presentation unit shown in Figure 1. Figure 29 is a diagram illustrating another example of a GUI presented to the user by the information presentation unit shown in Figure 1. Figure 30 is a diagram illustrating an example of a computer that executes an information processing program.

[0009] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0010] [Embodiment] Figure 1 is a diagram showing an example of the configuration of an information processing device according to an embodiment. The information processing device 1 shown in Figure 1 is an information processing device that holds information related to a plurality of candidate events used for information presentation, and presents information to the user based on the information related to the candidate events.

[0011] The information processing device 1 collects the user's behavioral history and, based on the collected behavioral history, detects candidate events at times when it is estimated that memorable events or events the user wishes to recall occurred. The behavioral history includes, but is not limited to, the user's location information, information indicating the user's purchase amount, the user's count values, and the aesthetic score of the user's photos. Candidate events are events in which relevant information is used for information presentation.

[0012] For example, the information processing device 1 determines which clusters obtained by clustering multiple consecutive location information of the user are within the user's living area, and which clusters are outside the living area, based on the number of location information obtained in descending order. Then, based on the location information outside the living area, the information processing device 1 extracts the time when the user is continuously located outside the living area as an outside-living-area event. In this way, the information processing device 1 can appropriately extract outside-living-area events. The information processing device 1 also calculates a distance score, which is a score based on the distance from the location information within the living area to the location information outside the living area.

[0013] The information processing device 1 detects off-site events that are relevant to the user and calculates a score for the off-site events that are candidates, based on event-related information, which is information about the user associated with the off-site event. The event-related information is information included in the collected behavioral history and includes, for example, one or more pieces of information such as distance score, user purchase amount, user count, and aesthetic score of the user's photos.

[0014] In this way, the information processing device 1 can automatically detect and score events that users will remember and want to recall by using event-related information corresponding to events outside their living area to detect candidate events and calculate scores for those candidate events. Since activities involving movement, such as travel, are often events that users will remember and want to recall, the information processing device 1 detects candidate events from one or more events outside their living area.

[0015] The scores of candidate events may be used, for example, to select candidate events to be used when the information processing device 1 presents information, or to decide whether or not to maintain them as candidate events, but are not limited to such examples.

[0016] Furthermore, the information processing device 1 updates the scores of candidate events using information from the user. For example, the information processing device 1 acquires user-provided information, which is information presented by the user, and estimates one or more of the user's memory retention level, comprehension level, and engagement level based on the acquired user-provided information. Based on the estimated results, the information processing device 1 updates the scores of the retrieved candidate events.

[0017] In this way, the information processing device 1 updates the score of a candidate event when the user mentions a topic related to that candidate event while interacting with the user. This allows the device to reflect fluctuations in the strength of the user's memory in the score of the candidate event, thereby improving the accuracy of the candidate event score.

[0018] Furthermore, the information processing device 1 acquires response information indicating the user's response to information presentation using information related to candidate events, and estimates one or more of the user's memory retention level, comprehension level, and engagement level based on the acquired response information. Based on the estimated results, the information processing device 1 updates the score of the retrieved candidate events.

[0019] In this way, the information processing device 1 touches upon the topic of candidate events while interacting with the user and updates the score of the candidate event in accordance with the user's response. This allows fluctuations in the strength of the user's memory to be reflected in the score of the candidate event, thereby improving the accuracy of the candidate event score.

[0020] In this way, the information processing device 1 updates the score of candidate events based on one or more estimated results from the user's memory retention level, understanding level, and engagement level. This allows the information processing device 1 to accurately update event scores that reflect fluctuations in the user's memory strength and other factors.

[0021] As shown in Figure 1, the information processing device 1 comprises a communication unit 2, a storage unit 3, and a processing unit 4. The information processing device 1 transmits and receives various information from external devices, for example, via the communication unit 2. For example, the information processing device 1 acquires the user's activity history from the user's terminal device or a cloud server that manages user information.

[0022] Furthermore, the information processing device 1 engages in dialogue with the user (for example, voice dialogue, chat-style dialogue, etc.). For example, the information processing device 1 obtains user-provided information, which is information presented by the user, via the communication unit 2, and obtains response information, which shows the user's response to information presented using information related to candidate events.

[0023] [Communication Unit 2] Communication Unit 2 is implemented, for example, by a communication module. Communication Unit 2 is connected to the network by wire or wireless and transmits and receives information with various other devices. For example, Communication Unit 2 transmits and receives information via the network with user terminal devices and various servers.

[0024] [Storage Unit 3] The storage unit 3 is implemented by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, or by storage devices such as hard disks and optical discs. The storage unit 3 includes a location information storage unit 10, an event-related information storage unit 11, and a candidate event information storage unit 12.

[0025] [Location Information Storage Unit 10] The location information storage unit 10 stores location information acquired by the processing unit 4 via the communication unit 2. The location information includes, for example, information such as time, latitude, and longitude. The time is the time when the latitude and longitude were detected (for example, year, month, day, hour, minute, second).

[0026] A user ID (IDentifier) ​​is associated with the location information. The user ID is, for example, the identification information of the user to whom the information is presented by the information processing device 1. In this way, the event-related information storage unit 11 stores the location information for each user. Note that the user ID may also be included in the location information.

[0027] [Event-related information storage unit 11] The event-related information storage unit 11 stores event-related information acquired by the processing unit 4 via the communication unit 2.

[0028] Event-related information is information related to events outside the living area detected by the processing unit 4. Event-related information includes, for example, one or more pieces of information such as purchase information, count values, and aesthetic scores of photographs. Purchase information includes, but is not limited to, information such as the purchase amount per store, the name of the store, the type of store, and the purchase amount per product and store type.

[0029] The count data includes information such as media information, SNS (Social Networking Service) posting information, and step count information. Media information is information about media data (e.g., data such as photos and videos), and is not limited to, for example, the number of media data or the number of people extracted from the media data. Media data is not limited to, for example, one or more media data from media data stored on the user's terminal device and media data of the user stored on a cloud server.

[0030] The number of people extracted from the media data is, for example, the number of people recognized using an image processing engine (see, for example, Reference 1) or a multimodal LLM (Large Language Models), but is not limited to such examples. The processing unit 4 obtains information indicating the number of people extracted from the media data using an image processing engine or a multimodal LLM.

[0031] Reference 1: “Next-generation media processing AI MediaGnosis” [Accessed April 30, 2025], Internet <URL: https: / / www.rd.ntt / mediagnosis / >

[0032] Social media post-related information includes, for example, the number of posts a user has made on social media, the number of reactions from others to posts made by a user on social media, and the number of media data items and user mentions included in posts made by a user. Posts include, for example, messages, media data, or a combination thereof, but are not limited to these examples. The number of reactions includes, for example, the number of likes, the number of replies, and their total, but are not limited to these examples.

[0033] SNS post-related information can be obtained, for example, by using the attributes of the post information obtained in JSON (JavaScript® Object Notation) format from the SNS's API (Application Programming Interface). For example, processing unit 4 obtains SNS post-related information through the SNS's API. The SNS API is, for example, the API of X (formerly Twitter®), but is not limited to such examples.

[0034] Step count information is information that indicates the user's step count, and is, for example, the total number of steps during an event outside the user's living area or the number of steps per unit of time during an event outside the user's living area, but is not limited to such examples.

[0035] The aesthetic score of a photograph is an index that numerically evaluates the visual beauty and appeal of the photograph. For example, processing unit 4 calculates the aesthetic score of a photograph using the technology described in Reference 2 or an Aesthetic Assessment Model using deep learning. The Aesthetic Assessment Model is, for example, NIMA (Neural Image Assessment), but is not limited to such examples.

[0036] Reference 2: Y. Deng, “Image Aesthetic Assessment - An experimental Survey”, IEEE Signal Processing Magazine, July 2017, pp.80-106

[0037] [Candidate Event Information Storage Unit 12] The candidate event information storage unit 12 stores multiple candidate event information. For example, the candidate event information includes time information indicating a candidate event, information indicating the user's location in the candidate event, the score of the candidate event, candidate event-related information which is information related to the candidate event, and the cumulative number of detections. The time information includes, for example, information indicating the start time and information indicating the end time. The cumulative number of detections is the total number of times the candidate event information has been referenced by the processing unit 4.

[0038] The candidate event information stored in the candidate event information storage unit 12 is referenced and updated by the processing unit 4. For example, the score of the candidate event, candidate event-related information, and cumulative detection count included in the candidate event information are updated by the processing unit 4. The candidate event-related information includes, but is not limited to, the event-related information mentioned above, as well as SNS posting information, information entered by the user on online services other than SNS, and information on the interaction between the information processing device 1 and the user.

[0039] [Processing Unit 4] Processing Unit 4 comprises an acquisition unit 20, a location information processing unit 21, an event information processing unit 22, a response processing unit 23, a presentation information generation unit 24, and an information presentation unit 25.

[0040] [Acquisition Unit 20] The acquisition unit 20 acquires various information from external devices via the communication unit 2. For example, the acquisition unit 20 acquires location information and event-related information from external devices. The acquisition unit 20 stores the acquired location information in the location information storage unit 10. The acquisition unit 20 also stores the acquired event-related information in the event-related information storage unit 11.

[0041] The acquisition unit 20 can acquire, for example, user location information, purchase information, media data, media information, SNS post information, SNS post-related information, etc., from the user's terminal device, a cloud server that manages user information (for example, a server that provides cloud storage services), and other online servers. The acquisition unit 20 can acquire location information, purchase information, media data, media information, SNS post information, etc., for example, through the API of the cloud server or the API of other online servers.

[0042] The acquisition unit 20 can acquire information indicating the number of people extracted from media data, for example, using an image processing engine or a multimodal LLM. The acquisition unit 20 can use multimodal LLM through an API of an AI (Artificial Intelligence) service that supports multimodal LLM. In the following, when LLM is mentioned, it may include multimodal LLM.

[0043] Furthermore, the acquisition unit 20 filters the information posted to SNS and can extract SNS post-related information from the posted information that is not excluded by the filtering process. Because SNS posts are made manually, there is often a discrepancy between the time the facts related to the posted content occurred and the time the post was made. Therefore, even if a post is made during an event, it may be SNS post information unrelated to the event. To address this, the acquisition unit 20 checks the correspondence with location information and other event-related information and performs filtering to exclude SNS post information unrelated to the event.

[0044] The acquisition unit 20 performs filtering by, for example, reverse geocoding of location information. For example, the acquisition unit 20 excludes SNS posts that do not contain part of the address information obtained by reverse geocoding from the location information corresponding to the event. Reverse geocoding is the process of converting latitude and longitude coordinates into the corresponding address. For example, the acquisition unit 20 can obtain address information corresponding to the location information corresponding to the event through a geographic information provision service API (for example, Google® Map API).

[0045] Furthermore, the acquisition unit 20 can perform filtering using media data (such as photos and videos) included in the information posted to social media. For example, the acquisition unit 20 can exclude social media posts from which the photos and videos were taken during the event.

[0046] Furthermore, the acquisition unit 20 can also perform filtering using LLM. For example, the acquisition unit 20 inputs input information, including address information obtained by reverse geocoding from location information corresponding to an event, SNS post information, and instruction text, into the LLM, and excludes the post information based on the output from the LLM. The acquisition unit 20 can use LLM through an API of an AI service that can use LLM (for example, OpenAI® API), but LLM may also be provided within the information processing device 1. The same applies to other processes that use LLM in the processing unit 4.

[0047] Figure 2 shows an example of input information used for filtering by the LLM by the acquisition unit 20 in the processing unit 4 of the information processing device 1 according to the embodiment. In Figure 2, the text contained in the post information is inserted into <Post Body>. When the acquisition unit 20 inputs the input information shown in Figure 2 into the LLM, one of the following characters is output from the LLM: "Yes", "No", or "I don't know". The acquisition unit 20 excludes post information other than the one for which "Yes" is output from the LLM.

[0048] Furthermore, if the submitted information includes images such as photographs, the acquisition unit 20 can also perform filtering using a multimodal LLM. In this case, in addition to the information shown in Figure 2, images included in the submitted information are further input to the multimodal LLM. In this case as well, the acquisition unit 20 excludes submitted information that does not output "yes" from the multimodal LLM.

[0049] [Location Information Processing Unit 21] The location information processing unit 21 acquires location information of the same user over a certain period of time from the location information storage unit 10, and based on the acquired location information, it performs a determination of whether the user is outside their living area and detects events outside their living area at regular intervals. The living area is the user's daily living area. The location information processing unit 21 also calculates a distance score. The location information processing unit 21 outputs the detected events outside the living area and the calculated distance score.

[0050] For example, the location information processing unit 21 determines which clusters are considered the user's living area and which are considered outside the living area, based on the clusters obtained by clustering multiple consecutive locations of the user. The location information processing unit 21 then extracts the time when the user is continuously located outside the living area as an outside-living-area event, based on the location information outside the living area. The location information processing unit 21 also calculates a distance score, which is a score based on the distance from the location indicated by the location information within the living area to the location indicated by the location information outside the living area.

[0051] Figure 3 shows an example of the configuration of the location information processing unit 21 in the processing unit 4 of the information processing device 1 according to the embodiment. As shown in Figure 3, the location information processing unit 21 includes a living area detection unit 30, a living area event extraction unit 31, and a distance score calculation unit 32. The living area detection unit 30, the living area event extraction unit 31, and the distance score calculation unit 32 will be described below in that order.

[0052] [Outside Living Area Determination Unit 30] The Outside Living Area Determination Unit 30 acquires location information of the same user over a certain period of time from the location information storage unit 10, and makes a determination of being outside the living area based on the acquired location information. The location information over a certain period of time includes multiple location information that are consecutive over a certain period of time.

[0053] The outside-living-area determination unit 30 determines, for example, that among multiple clusters obtained by clustering multiple consecutive location information of the user, several clusters in descending order of location information are considered to be within the living area, and the remaining clusters are considered outside the living area.

[0054] Figure 4 is a flowchart showing the processing flow of the outside-living-area determination unit 30 in the location information processing unit 21 shown in Figure 3. As shown in Figure 4, the outside-living-area determination unit 30 acquires location information data that continues for a certain period of time (step S10).

[0055] The outside-living-area determination unit 30 classifies multiple location information by hierarchical clustering and assigns clusters to C in order of the number of location information. 1 , C 2 , ..., C m This is done (Step S11). Then, the living area determination unit 30 determines that the union U of the family of sets shown in equation (1) below is the living area at the smallest k that satisfies equation (2) described later, and the remainder is outside the living area (Step S12), and terminates the process shown in Figure 4.

[0056]

[0057] As shown in Figure 3, the outside-living-area determination unit 30 comprises an acquisition unit 40, a clustering unit 41, and a determination processing unit 42. The acquisition unit 40, the clustering unit 41, and the determination processing unit 42 will be described in that order below.

[0058] [Acquisition Unit 40] The acquisition unit 40 acquires multiple location information that are continuous for a certain period of time as the user's location information at regular intervals. "Regular period of time" is, for example, one day, one week, or one month, but is not limited to these examples.

[0059] "A fixed time" is set to an appropriate length depending on, for example, the calculations of processing unit 4 or the memory resources. "A fixed time" may also be set dynamically, but it is desirable to match it to the information update time interval required for information presentation. Furthermore, "continuous" means discrete continuity, and "continuous position information" means position information obtained at fixed intervals (for example, in seconds), although the interval does not necessarily have to be constant.

[0060] [Clustering Unit 41] The clustering unit 41 performs a clustering process at regular intervals, which is the process of clustering multiple location information that has been acquired by the acquisition unit 40 for a certain period of time.

[0061] For example, the clustering unit 41 classifies a plurality of pieces of position information into a plurality of clusters by hierarchical clustering, and sorts the clusters in descending order of the number of pieces of position information as C 1 , C 2 , ..., C m . m is a value equal to the total number of clusters. The clustering unit 41 uses the Ward's method as the hierarchical clustering, but is not limited to such an example.

[0062] [Determination Processing Unit 42] The determination processing unit 42 determines, for each of the plurality of C 1 , C 2 , ..., C m obtained by clustering by the clustering unit 41, whether the cluster is a living area or outside the living area.

[0063] For example, the determination processing unit 42 uses the following formula (2) to calculate the minimum k that satisfies the following formula (2). In the following formula (2), "n(C i )" represents the number of pieces of position information of the cluster C i . Further, in the following formula (2), "T" is determined according to the ratio of the total number of pieces of position information acquired by the acquisition unit 40 that is designated as outside the living area. For example, when less than 5% of the total number of pieces of position information is designated as outside the living area, T is set to 0.95 (= 1 - 0.05).

[0064]

[0065] The determination processing unit 42 determines the union U of the family of sets represented by the above formula (1) as a living area, and determines the rest as outside the living area. That is, the determination processing unit 42 merges the clusters C 1 , C 2 , ..., C k into a living area cluster, and determines each of the clusters C k+1 , ..., C m as a cluster outside the living area. Hereinafter, the living area cluster obtained by merging the clusters C 1 , C 2 , ..., C k may be referred to as a living area cluster C x .

[0066] Figure 5 shows an image of the determination result of being outside the living area by the determination processing unit 42 shown in Figure 3. In Figure 5, the movement trajectory shown by the continuous location information of a certain user and the distinction between the living area and outside the living area are shown.

[0067] [Out-of-Life Area Event Extraction Unit 31] The Out-of-Life Area Event Extraction Unit 31 extracts the time when a user is continuously located outside the living area as an Out-of-Life Area Event, based on location information outside the living area.

[0068] The Out-of-Life Area Event Extraction Unit 31 extracts out-of-life area events based on multiple location information included in clusters determined to be outside the living area by the Determination Processing Unit 42, for example, when a user is continuously located outside the living area. The Out-of-Life Area Event Extraction Unit 31 outputs out-of-life area event information, which is information about the extracted out-of-life area events. The out-of-life area event information includes, but is not limited to, time information indicating the out-of-life area event and information indicating the location where the out-of-life area event occurred.

[0069] Figure 6 is a flowchart showing the processing flow of the out-of-habitation-area event extraction unit 31 in the location information processing unit 21 shown in Figure 3. As shown in Figure 6, the out-of-habitation-area event extraction unit 31 selects one unprocessed location information from outside the living area (step S20). The location information from outside the living area is location information included in the cluster that has been determined to be outside the living area.

[0070] Next, the out-of-habitation-area event extraction unit 31 extracts the time period during which the out-of-habitation-area location information selected in step S20 and the consecutive out-of-habitation-area location information exist as a single out-of-habitation-area event (step S21).

[0071] Next, the out-of-habitation-area event extraction unit 31 determines whether or not there is unprocessed location information outside the living area (step S22). If the out-of-habitation-area event extraction unit 31 determines that there is unprocessed location information outside the living area (step S22: Yes), it proceeds to step S20. If it determines that there is no unprocessed location information outside the living area (step S22: No), it terminates the process shown in Figure 6.

[0072] In this way, the out-of-habitation-area event extraction unit 31 extracts, for example, the time period during which a selected out-of-habitation-area location information and consecutive out-of-habitation-area location information exist as a single out-of-habitation-area event. In the following, out-of-habitation-area location information may be referred to as out-of-habitation-area location information.

[0073] Figure 7 shows an example of a simplified algorithm for extracting events outside the living area in the event extraction unit 31 of the location information processing unit 21 shown in Figure 3. In the simplified algorithm shown in Figure 7, location information is arranged in chronological order from the past to the future, and then p 1 ~p n To the left, the location information selected by the outside-living-area event extraction unit 31 is p k That is what they say.

[0074] Figure 8 shows an example of multiple location information from which events outside the living area are extracted by the simplified algorithm shown in Figure 7. As shown in Figure 8, the event extraction unit 31 outputs event information outside the living area, which includes time information indicating an event outside the living area in a format in which the event start time and event end time are uniquely determined.

[0075] For example, as shown in Figure 8, the out-of-habitation-area event extraction unit 31 outputs a combination of a list of out-of-habitation-area location information and time information indicating an out-of-habitation-area event as out-of-habitation-area event information. In the example shown in Figure 8, the out-of-habitation-area event is "2025 / 1 / 8 11:10:01 to 17:30:00 (6 hours 19 minutes 59 seconds)", and the list of out-of-habitation-area location information includes multiple out-of-habitation-area location information from "11:10:01" to "17:30:00" on 2025 / 1 / 8.

[0076] Figure 9 shows another example of a simplified algorithm for extracting events outside the living area in the event extraction unit 31 of the location information processing unit 21 shown in Figure 3. In the simplified algorithm shown in Figure 9, location information is arranged in chronological order from the past to the future, and p 1 ~p n To the left, the location information selected by the outside-living-area event extraction unit 31 is p kThis is the case. Also, in the simplified algorithm shown in Figure 9, p i+1 and p i The distance between |p i+1 -p i It is set as |.

[0077] The simplified algorithm shown in Figure 9 extracts a single "out-of-habitation event" by selecting an out-of-habitation location from among multiple out-of-habitation location data points and identifying the time period during which the out-of-habitation location data points and the travel speed are consecutive and below a threshold.

[0078] For example, if we define an event as only the time when the vehicle is not moving at car speed, then the difference in location information (= |p) is used. i+1 -p i A threshold Tv is set such that if the speed obtained by dividing the distance [km] by the time difference exceeds 10 km / h, the user is considered to be moving by car or other means. In this case, the threshold Tv is given by, for example, the following equation (3). In the following equation (3), "t i+1 -t i " is location information p i+1 Detection time and location information p i This is the difference [h] between the detection time and the detection time. Tv = 10 × (t i+1 -t i ) ... (3)

[0079] Figure 10 shows an example of multiple location information from which an event outside the living area is extracted using the simplified algorithm shown in Figure 9. In the example shown in Figure 10, the event outside the living area is "2025 / 1 / 8 11:10:01-11:15:03 (0 hours 5 minutes 2 seconds)", and the list of location information outside the living area includes multiple location information from "11:10:01" to "11:15:03" on 2025 / 1 / 8.

[0080] Furthermore, the out-of-habitation-area event extraction unit 31 may, in place of or in addition to the list of out-of-habitation-area location information, include information indicating the location range shown by the out-of-habitation-area location information as out-of-habitation-area location information in the out-of-habitation-area event information.

[0081] [Distance Score Calculation Unit 32] The distance score calculation unit 32 calculates a distance score, which is a score based on the distance from a location indicated by location information within the living area to a location indicated by location information outside the living area.

[0082] The distance score calculation unit 32 calculates, for example, a distance score for each location information outside the living area based on the distance from the location information included in the set of location information determined to be within the living area by the living area determination unit 30.

[0083] For example, the distance score calculation unit 32 calculates the living area cluster C xに The included position P in,i Regarding the center of gravity position P in,G , and the variance-covariance matrix Σ is calculated. Then, the distance score calculation unit 32 uses the following formula (4) to calculate the distance score d k Calculate the following equation (4): "P out,k This indicates a location outside of one's usual living area.

[0084]

[0085] Distance score d k This is called the Mahalanobis distance, and can be considered a distance standardized over the principal components obtained from principal component analysis of the living area. That is, the Mahalanobis distance d k This refers to location P, which is located outside the living area. out,k This score represents how far away the location is from the user's living area.

[0086] Thus, the distance score calculation unit 32 calculates the position P indicated by location information outside the living area. out,k The degree of distance from the living area is measured by a distance score d. k It will be scored as follows.

[0087] Figure 11 shows a visualized image of the distance score calculated by the distance score calculation unit 32 shown in Figure 3. As shown in Figure 11, the principal components obtained from the principal component analysis of the living area are shown as an elliptical density distribution.

[0088] Furthermore, the distance score calculation unit 32 can also calculate a distance score for each event outside the living area based on a list of location information outside the living area included in the event information outside the living area. For example, the distance score calculation unit 32 can calculate the average of the distance scores of multiple locations outside the living area included in the list of location information outside the living area as the distance score for the event outside the living area.

[0089] [Event Information Processing Unit 22] The event information processing unit 22 obtains events outside the living area output from the location information processing unit 21 and event-related information corresponding to these events from the event-related information storage unit 11. Then, based on the acquired event-related information, the event information processing unit 22 performs candidate event detection processing and score calculation processing, and outputs the resulting candidate event information.

[0090] The candidate event detection process detects events outside the user's local area that are relevant to the information presented to the user as candidate events. The score calculation process calculates a score for the candidate events outside the user's local area.

[0091] The event information processing unit 22 performs candidate event detection processing and score calculation processing based on information such as user information related to an event outside the living area, which includes one or more pieces of information from the distance score, the user's purchase amount, the user's count value, and the aesthetic score of the user's photos.

[0092] The event information processing unit 22 stores the candidate event information of the candidate events detected in the candidate event detection process in the candidate event information storage unit 12, based on the results of the candidate event detection process and the results of the score calculation process.

[0093] Figure 12 is a flowchart showing the processing flow of the event information processing unit 22 shown in Figure 1. As shown in Figure 12, the event information processing unit 22 acquires one unprocessed event information from outside the living area (step S30).

[0094] Next, the event information processing unit 22 acquires event-related information corresponding to the event outside the living area (step S31). The event-related information corresponding to the event outside the living area is information about the actions taken by the user during the time indicated by the event outside the living area.

[0095] Next, the event information processing unit 22 performs candidate event detection processing based on the event-related information acquired in step S31 (step S32). The event information processing unit 22 also performs score calculation processing based on the event-related information acquired in step S31 (step S33).

[0096] Next, the event information processing unit 22 determines whether or not there is any unprocessed event information outside the living area (step S34). If the event information processing unit 22 determines that there is any unprocessed event information outside the living area (step S34: Yes), it proceeds to step S30. If there is no unprocessed event information outside the living area (step S34: No), it terminates the process shown in Figure 12.

[0097] Figure 13 shows an example of the configuration of the event information processing unit 22 in the processing unit 4 of the information processing device 1 according to the embodiment. As shown in Figure 13, the event information processing unit 22 includes an acquisition unit 50, an event detection unit 51, and a score calculation unit 52. The acquisition unit 50, the event detection unit 51, and the score calculation unit 52 will be described in that order below.

[0098] [Acquisition Unit 50] The acquisition unit 50 acquires event-related information corresponding to the out-of-habitation area event output from the location information processing unit 21 from the event-related information storage unit 11.

[0099] For example, suppose the event outside the living area included in the information on events outside the living area is "2025 / 1 / 8 11:10:01 to 17:30:00". In this case, the acquisition unit 50 acquires event-related information that occurred within the range of "2025 / 1 / 8 11:10:01 to 17:30:00" from the event-related information storage unit 11 as event-related information corresponding to the event outside the living area.

[0100] [Event Detection Unit 51] Based on the event-related information corresponding to the event outside the user's living area, the event detection unit 51 detects events that are likely to be memorable or that the user would like to recall as candidate events, which are used to generate the presented information.

[0101] For example, the event detection unit 51 performs event detection and determination using thresholds for event-related information. Thresholds may be determined from the data distribution of individuals such as the same user or similar users as the user being judged, or from a database or statistical data containing data from multiple people. Thresholds are determined for each type of event-related information using data from a certain past period (e.g., the past month or year from the present). Thresholds are set, for example, by the value per unit time, mean, median, percentile value, etc., within a predetermined period (e.g., 15 minutes, 30 minutes, or 1 hour) that includes the time when the user performed the action that is the subject of the event-related information, but are not limited to these examples.

[0102] The event detection unit 51 calculates the value per unit time for the event-related information to be judged if the threshold is a value per unit time, and otherwise selects the average, maximum, minimum, or median value of the event and compares it with the threshold to make a judgment, but is not limited to these examples.

[0103] The event detection unit 51 detects an event outside the living area as a candidate event if, for example, the value indicated by the event-related information exceeds or falls below a threshold.

[0104] For example, if the event-related information is purchase information, the event detection unit 51 can detect events outside the living area with high expenditure if they exceed the threshold, and events outside the living area with low expenditure if they fall below the threshold, in an event detection process using a threshold. The event detection unit 51 can designate events with high expenditure or events outside the living area with low expenditure as candidate events.

[0105] Figure 14 shows an example of event detection using a threshold by the event detection unit 51 of the event information processing unit 22 shown in Figure 13. Figure 14 shows an example where the event-related information to be judged is purchase information.

[0106] In Example 1 shown in the figure, the event detection unit 51 detects as candidate events events that occur outside the user's living area, where the purchase amount per unit time (e.g., 1 hour) exceeds a threshold (e.g., the purchase amount per past hour).

[0107] In Example 2 shown in Figure 14, the event detection unit 51 detects events outside the user's living area as candidate events, where the maximum purchase amount exceeds a threshold (for example, the median of past purchase amounts).

[0108] In Example 3 shown in Figure 14, the event detection unit 51 detects events outside the user's living area as candidate events, where the median purchase amount exceeds a threshold (for example, the 80th percentile of past purchase amounts).

[0109] In Example 4 shown in Figure 14, the event detection unit 51 detects events outside the user's living area as candidate events, where the median purchase amount falls below a threshold (for example, the 20th percentile of past purchase amounts).

[0110] If event-related information is purchase information, thresholds can be set for each type of product or store. This allows for event detection using thresholds, excluding purchase information from everyday convenience stores and the like. While product and store types are pre-included as attributes in the purchase information, the system is not limited to these examples. The event detection unit 51 can also use classifications output by inputting part (e.g., product name) or all of the purchase information into the LLM if the purchase information includes a product name, for example.

[0111] While the above mainly described thresholding when event-related information is purchase information, similar thresholding can be used to detect candidate events even when event-related information is a numerical value or an aesthetic score of a photograph.

[0112] Furthermore, the event detection unit 51 can utilize multiple pieces of event-related information or the distance score calculated by the location information processing unit 21. In this case, the event detection unit 51 can detect events to be detected by combining multiple thresholds or by using a machine learning model. The machine learning model is, for example, a model generated by machine learning such as a support vector machine or a neural network.

[0113] The machine learning model is, for example, a model trained to take information including information for multiple stores and distance scores as input and output a result determining whether it is an event to be detected, but is not limited to such an example. The event detection unit 51 can use a machine learning model to detect candidate events instead of a threshold-based determination process.

[0114] Figure 15 shows another example of event detection using a threshold by the event detection unit 51 of the event information processing unit 22 shown in Figure 13. Figure 15 shows an example where the event-related information to be judged is purchase information.

[0115] In Example 1 shown in Figure 15, the event detection unit 51 detects as candidate events events that occur outside the user's living area if the purchase amount at stores other than convenience stores per unit time (e.g., 1 hour) exceeds a threshold (e.g., the purchase amount at stores other than convenience stores in the past hour).

[0116] In Example 2 shown in Figure 15, the event detection unit 51 detects as candidate events events that occur outside the user's living area in one or more stores of different store types, where the maximum purchase amount exceeds a threshold (for example, the median of past purchase amounts).

[0117] In Example 3 shown in Figure 15, the event detection unit 51 determines whether an event is a candidate event based on the output of the machine learning model when the purchase amount for each store type (for example, store type α and store type β) is input to the machine learning model.

[0118] Figure 16 is a diagram illustrating the machine learning model used for event detection by the event detection unit 51 of the event information processing unit 22 shown in Figure 13. In the example shown in Figure 16, store type α information (purchase amount), store type β information (purchase amount), and distance score are input to the machine learning model, and the machine learning model outputs a determination result of whether the event is a target event for detection. The determination result is 1 if it is a target event for detection, and 0 if there is no target event for detection, but is not limited to this example. The machine learning model shown in Figure 16 is, for example, a support vector machine or a neural network, but is not limited to this example.

[0119] The event detection unit 51 can also detect events as candidate events if the score calculated by the score calculation unit 52 is equal to or greater than a threshold. In this case, the score calculation unit 52 calculates a score for each event that occurs outside the living area.

[0120] [Score Calculation Unit 52] The score calculation unit 52 calculates a score for each candidate event detected by the event detection unit 51, indicating the degree to which the user is likely to remember or want to recall it.

[0121] First, we will explain how to calculate the score of a candidate event corresponding to event-related information of a single attribute. The score calculation unit 52 can calculate the score of a candidate event corresponding to event-related information of a single attribute using, for example, the following formula (5). The score increases as the event-related information is greater than the median. s = (a - m) / NIQR ... (5)

[0122] In equation (5) above, "m" is the median of the data distribution for event-related information over a certain period in the past, "a" is the event-related information, and "s" is the score of the event corresponding to the event-related information. Furthermore, "NIQR" in equation (5) above is the value obtained by normalizing the interquartile range, which is the difference between the third quartile and the first quartile in the data distribution for event-related information over a certain period in the past, by dividing the IQR by the IQR in a normal distribution (1.3489).

[0123] Next, we will explain how to calculate the score of candidate events corresponding to event-related information with multiple attributes. Here, we assume that the multiple attributes are attributes 1 to n (where n is an integer of 2 or more). The score calculation unit 52 calculates the score for each attribute 1 to n using the following formula (6). In the following formula (6), the subscript "i" represents the attribute number. i = (a i -m i ) / NIQR...(6)

[0124] Next, the score calculation unit 52 calculates the sum of the scores for attributes 1 to n, as shown in formula (7) below, and uses this as the score for the candidate event.

[0125]

[0126] Furthermore, the score calculation unit 52 can also calculate a score by inputting event-related information of multiple attributes into a machine learning model. The machine learning model is, for example, a support vector machine or a neural network, which has been trained to output a score by taking a combination of event-related information of multiple attributes as input.

[0127] The score calculation unit 52 normalizes the event-related information of each attribute in advance, as shown in the score s of equation (5) above, before inputting it into the machine learning model. The output training data (labels) are collected using a Likert scale (4-point or 5-point scale), etc.

[0128] Figure 17 is a diagram illustrating the machine learning model used for score calculation by the score calculation unit 52 of the event information processing unit 22 shown in Figure 13. In the example shown in Figure 17, store type α information (purchase amount), store type β information (purchase amount), and distance score are input to the machine learning model, and the machine learning model outputs a score for the candidate event. The machine learning model shown in Figure 17 is, for example, a support vector regression, a support vector machine, or a neural network, but is not limited to these examples.

[0129] Next, the method for calculating scores based on event-related information of the count values ​​will be explained. In the following, event-related information of the count values ​​may be referred to as event-related count values. The score calculation unit 52 can, for example, assume that the data distribution of event-related count values ​​over a certain period in the past follows a Poisson distribution with a mean value λ, and calculate the score of candidate events corresponding to event-related count values ​​based on the cumulative distribution function using the following formula (8).

[0130]

[0131] Furthermore, the score calculation unit 52 can also calculate a score that emphasizes novel experiences. For example, the score calculation unit 52 can calculate the score of a candidate event corresponding to event-related information using the following formula (9): s = d + |s purchase | + s step +s photo ... (9)

[0132] In formula (9) above, "d" is the distance score, and "s purchase " is a score calculated using, for example, formula (7) above, and "s step " is a score calculated using the above formula (5) for the number of steps, for example, and "s photo " represents, for example, the number of photos, and "| |" represents the absolute value.

[0133] The score s shown in equation (9) above takes a higher value for novelty as the distance from the normalized living area, purchase amount, number of steps, and number of photos increase for each distribution. For purchase amount, the absolute value is taken so that a smaller amount also results in a higher score as it represents a novelty experience with less expenditure.

[0134] Furthermore, the score calculation unit 52 can also calculate a score that emphasizes shared experiences with others. For example, the score calculation unit 52 can calculate the score of a candidate event corresponding to event-related information using the following formula (10): s = n person +n reaction ... (10)

[0135] In the above formula (10), "n person" is, for example, the number of people extracted from the photograph, and "n reaction " is, for example, the total number of replies and quotes to a social media post. The score s shown in formula (10) above will be higher as the number of people extracted from the photo, i.e., companions, and the number of people who shared the experience via social media increase, as it represents an experience shared with others.

[0136] Furthermore, the score calculation unit 52 can also calculate a score that emphasizes aesthetic experience. For example, the score calculation unit 52 can calculate the score of a candidate event corresponding to event-related information using the following formula (11): s = ReLU(s purchase )+a...(11)

[0137] In the above formula (11), "s purchase " is a score calculated using, for example, formula (7) above, "a" is, for example, an aesthetic score, and "ReLU" represents the ramp function. The score s shown in formula (11) above takes a higher value as the purchase amount and aesthetic score increase, indicating a higher aesthetic experience. The reason why the score s increases with the purchase amount is that expensive meals and accommodations tend to be aesthetic experiences.

[0138] Furthermore, the score calculation unit 52 can also calculate a score that emphasizes the travel experience. For example, the score calculation unit 52 can calculate the score of a candidate event corresponding to the event-related information using the following formula (12): s = d + s step ... (12)

[0139] In the above formula (12), "d" is the distance score, and "s step " is a score calculated, for example, using formula (5) above for the number of steps. The score s shown in formula (12) above takes a higher value as a movement experience because the greater the distance from the living area and the greater the number of steps, the more movement occurs.

[0140] Furthermore, the score calculation unit 52 can also calculate scores that emphasize novel experiences, shared experiences, aesthetic experiences, and travel experiences. For example, the score calculation unit 52 can calculate the score of candidate events corresponding to event-related information using the following formula (13).

[0141]

[0142] In the above formula (13), "s novelty " is the score calculated using the above formula (9), "s share " is the score calculated using the above formula (10). Also, in the above formula (13), "s aesthetic " is the score calculated using the formula (11) described above, and "stransportation" is the score calculated using the formula (12) described above.

[0143] Furthermore, in the above formula (13), "w novelty " is "s novelty The weight of "" and "w share " is "s share The weight of "" and "w aesthetic " is "s aesthetic The weight of "wtransportation" is the weight of "transportation".

[0144] These weights should ideally be set to values ​​appropriate for the user and application. For example, for users who travel frequently, "wtransportation" might be set to a small value close to 0, or for applications used by passenger transport operators where the primary output should be the travel experience, "wtransportation" might be set relatively high compared to the other weights. Alternatively, machine learning models such as support vector machines or neural networks may be used to determine the aforementioned weights.

[0145] [Response Processing Unit 23] The response processing unit 23 takes the response target information as input, estimates the degree of memory retention, understanding, and engagement, calculates an update score for candidate events based on the estimated results, and updates the score of the candidate events.

[0146] The information to be responded to is information entered by the user, such as the user-provided information and response information mentioned above. The information to be responded to includes at least one of the following: the user's spoken voice, the text entered by the user, the text obtained by speech recognition processing of the user's spoken voice, and the user's video. Speech recognition processing can be performed using, for example, the Google® Cloud Speech-to-text API, but is not limited to such examples. The events to be updated in the score are, for example, events retrieved based on the user-provided information, or events referenced by the information presentation unit 25.

[0147] The response processing unit 23 updates the candidate event information, using the information to be updated from the candidate event information as update information. The update information includes, but is not limited to, the update score, information added as candidate event-related information, and the cumulative number of detections described later.

[0148] The response processing unit 23 estimates one or more of the user's memory retention, comprehension, and engagement level based on, for example, the user-provided information and the response information, when a relevant candidate event is found among multiple candidate events based on the user-provided information described above. The response processing unit 23 then updates the score of the candidate event found as described above and the score of the candidate event that was the basis of the response, or both, based on the estimated result of one or more of the memory retention, comprehension, and engagement levels.

[0149] Figure 18 shows an example of the configuration of the response processing unit 23 shown in Figure 1. As shown in Figure 18, the response processing unit 23 comprises a first processing unit 60 and a second processing unit 61. The first processing unit 60 and the second processing unit 61 will be described in that order below.

[0150] [First Processing Unit 60] The first processing unit 60 detects an event search key from the user-provided information, searches for candidate events using the detected event search key, and updates the score of the candidate events found in the search. The reason for updating the score of candidate events in this way is that, for example, the memory of a candidate event may be reinforced when the user interacts with the candidate event.

[0151] Figure 19 is a flowchart showing an example of the processing flow by the first processing unit 60 shown in Figure 18. As shown in Figure 19, the first processing unit 60 acquires user-presented information (step S40).

[0152] Next, the first processing unit 60 detects an event search key from the user-presented information acquired in step S40 (step S41). Then, the first processing unit 60 searches for the relevant event from among the multiple candidate events stored in the candidate event information storage unit 12 using the event search key detected in step S41 (step S42). The relevant event is a candidate event related to the user-presented information.

[0153] The first processing unit 60 determines whether or not the relevant event was found in the search in step S42 (step S43). If it determines that the relevant event was found (step S43: Yes), it proceeds to step S44.

[0154] In step S44, the first processing unit 60 estimates a value of 1 or higher from among memory retention, comprehension, and engagement based on the user-provided information, and uses the estimated result to recalculate the score of the retrieved candidate event and update the score. Also in step S44, the first processing unit 60 updates the event-related information corresponding to the retrieved candidate event based on the user-provided information obtained in step S40.

[0155] The first processing unit 60 terminates the process shown in Figure 19 when the processing in step S44 is completed, or when it determines that the relevant event has not been hit (step S43: No).

[0156] As shown in Figure 18, the first processing unit 60 includes an acquisition unit 70, an information analysis unit 71, an event search unit 72, an estimation unit 73, and a score update unit 74. The acquisition unit 70, information analysis unit 71, event search unit 72, estimation unit 73, and score update unit 74 will be described in that order below.

[0157] [Acquisition Unit 70] The acquisition unit 70 acquires user-presented information as information to be responded to. For example, the acquisition unit 70 acquires user-presented information transmitted from the user's terminal device via the communication unit 2.

[0158] [Information Analysis Unit 71] The information analysis unit 71 detects event search keys from the user-presented information acquired by the acquisition unit 70. For example, if the user-presented information is spoken audio, the information analysis unit 71 converts the spoken audio into text using speech recognition processing, and divides the converted text into multiple morphemes using morphological analysis. The information analysis unit 71 detects specific parts of speech (e.g., proper nouns, common nouns, time nouns) among the morphemes divided by morphological analysis as event search keys. The information analysis unit 71 outputs the detected event search keys.

[0159] Furthermore, the information analysis unit 71 can detect event search keys using LLM instead of morphological analysis. For example, the information analysis unit 71 can input user-provided information and an instruction statement that instructs the detection of event search keys from the user-provided information into the LLM as input information, and detect one or more strings output from the LLM as event search keys.

[0160] [Event Search Unit 72] The event search unit 72 uses the event search key detected by the information analysis unit 71 to search for a candidate event corresponding to the event search key from among multiple candidate events stored in the candidate event information storage unit 12. The event search unit 72 outputs information about the corresponding event obtained through the search.

[0161] Furthermore, the event search unit 72 performs a keyword search using the event search key detected by the information analysis unit 71, but it can also perform a vector search instead of or in addition to the keyword search. For example, the event search unit 72 can vectorize some or all of the user-presented information and perform a vector search using the vectorized information.

[0162] [Estimation Unit 73] When the estimation unit 73 finds a relevant candidate event from among multiple candidate events based on the user-provided information, it estimates one or more of the user's memory retention, understanding, and engagement with the relevant event based on the user-provided information. The user's memory retention, understanding, and engagement with the relevant event are, for example, the user's memory retention, understanding, and engagement with the user's actions and events in the relevant event.

[0163] The estimation unit 73 calculates one or more of the following as estimation results for the degree of user retention, understanding, and engagement with the event in question: a score indicating the degree of user retention, understanding, and engagement with the event in question.

[0164] First, let's explain the memory retention score, which is a score indicating the degree of memory retention. In order to calculate the memory retention score, the estimation unit 73 determines the number of matches between words in the user-presented information and positive words in the word polarity dictionary, based on the results of morphological analysis by the information analysis unit 71, and also determines the number of proper nouns in the user-presented information.

[0165] The word polarity dictionary contains a list of positive words. The word polarity dictionary is generated by, for example, the technology described in Reference 3, but is not limited to such examples. Alternatively, the estimation unit 73 can use a machine learning model that determines whether a noun is a positive word to determine the number of positive words among the words included in the user's response. The estimation unit 73 also detects the length of the user's utterance or the duration of the utterance audio indicated in the user-presented information. The length of the utterance is the length of the text obtained when the utterance audio is transcribed, if the user-presented information is audio.

[0166] Reference 3: Masahiko Higashiyama et al., "Acquisition of Noun Evaluative Polarity Focusing on Predicate Selection Preference," Proceedings of the 14th Annual Meeting of the Association for Natural Language Processing, 2008, pp. 584-587.

[0167] If the event search unit 72 obtains the relevant event, the estimation unit 73 calculates the user's memory retention score for that event. For example, the estimation unit 73 calculates the memory retention score using the following formula (14). m =w d n d +w p n p +w u n u +w t n t ... (14)

[0168] The memory retention score s shown in formula (14) above m This score is based on the characteristics of autobiographical memories, which are important to the user (repeatedly recalled memories, memories associated with emotions, vivid memories; see, for example, reference 4).

[0169] Reference 4: Koichi Sato, "Function and Recall Characteristics of Autobiographical Memory," Bulletin of the Faculty of Education, Gunma University, Humanities and Social Sciences, 2007, Vol. 56, pp. 333-357.

[0170] In the above formula (14), "n d " represents the cumulative number of candidate events detected, and "n p" is the number of matches between words included in user presentation information and positive words in a word polarity dictionary, and "n u " is the number of proper nouns included in the user presentation information, and "n t " is the sentence amount of utterance or the duration of utterance voice. The cumulative detected count of the corresponding event is the total number of times the related information (candidate event information) of the corresponding event has been referenced through search or the like.

[0171] In the above formula (14), "w d " is the weight of "n d ", "w p " is the weight of "n p ", "w u " is the weight of "n u ", "w t " is the weight of "n t ".

[0172] The greater the number of times a corresponding event is searched by the event search unit 72, the more the memory is repeatedly recalled and easily consolidated, so "n" is used to calculate the update score d " is used. In addition, the more positive words in the user presentation information, the more the memory with emotion is easily consolidated, so "n" is used to calculate the update score p " is used.

[0173] In addition, the more proper nouns in the user presentation information, the more the memory is repeatedly recalled and easily consolidated, so "n" is used to calculate the update score u " is used. Furthermore, the greater the sentence amount of the utterance or the longer the duration of the utterance voice, the clearer and more easily consolidated the memory becomes, so "n" is used to calculate the update score t " is used.

[0174] Next, the comprehension score, which is a score indicating the degree of comprehension, will be described. The estimation unit 73 can obtain a comprehension score as the estimation result of the user's comprehension degree using, for example, a comprehension estimation model (see, for example, Reference Document 5).

[0175] Reference 5: Shunichi Kinoshita et al., "A Study of Prediction of Listener's Comprehension Based on Multimodal Information," Proc. the 23rd ACM IVA '23, Article, 2023, No.30, pp.1-4.

[0176] The comprehension estimation model is a machine learning model such as gradient boosting, support vector machine, random forest, or neural network, and is a model trained using a dataset that includes information on how well the user understands what is being said and the content of the utterance at that time. The estimation unit 73 inputs the content of the utterance indicated by the user-presented information into the comprehension estimation model, and the comprehension estimation model can output the user's level of comprehension.

[0177] Furthermore, if the user-presented information includes the user's video, the estimation unit 73 can also estimate the user's level of comprehension using a comprehension estimation model that has been trained on a dataset that includes information on how well the user understands what is being said, as well as their facial expressions and body movements at that time.

[0178] The comprehension score output from the comprehension estimation model is, for example, a score in the range of -2 to +2, with the score approaching +2 indicating a higher level of comprehension, but is not limited to this example. The estimation unit 73 can also estimate the user's level of comprehension from user-provided information using the LLM. For example, the estimation unit 73 can input user-provided information and an instruction statement that instructs the output of a comprehension score into the LLM as input information, and have the LLM output a comprehension score. The information input into the LLM may also include related information (candidate event information) of the relevant event.

[0179] Next, we will explain the engagement score, which is a score indicating the degree of engagement. The estimation unit 73 can obtain an engagement score that indicates the estimated degree of user engagement with the relevant event using an engagement estimation model (see, for example, reference 6).

[0180] Reference 6: Takashi Inoue et al., "Estimation of dialogue engagement based on listener behavior using a latent character model," Transactions of the Japanese Society for Artificial Intelligence, 2018, Vol. 33, No. 1

[0181] The engagement score indicates, for example, the degree to which a user is interested, motivated, and wants to continue the conversation. Engagement estimation models are machine learning models such as support vector machines, random forests, or neural networks.

[0182] An engagement estimation model is trained using a dataset that includes, for example, engagement judgments for each annotator and corresponding listener behaviors (e.g., nodding, laughter, eye contact, etc.), but is not limited to such examples. For instance, an engagement estimation model may be trained using a dataset that includes information on the degree of engagement and the content of the utterance at that time.

[0183] The engagement score output from the engagement estimation model is, for example, a score in the range of 0 to 1, with the score approaching 1 as engagement is higher, but is not limited to this example. The estimation unit 73 can also obtain the engagement score using an LLM. In this case, the estimation unit 73 inputs information including user-presented information and an instruction statement that instructs the output of an engagement score into the LLM as input information, and has the LLM output an engagement score. The information input into the LLM may include related information (candidate event information) of the event in question.

[0184] [Score Update Unit 74] The score update unit 74 determines the updated score for the relevant event based on the estimation result from the estimation unit 73. The score update unit 74 updates the score for the relevant event with the determined updated score. The updated score for the relevant event is obtained by updating the score in the candidate event information for the relevant event stored in the candidate event information storage unit 12.

[0185] The score update unit 74 can calculate an updated score using, for example, one of the memory retention score, comprehension score, and engagement score, along with the score before the update. For example, the score update unit 74 calculates the updated score using the following formula (15). In the following formula (15), "s'" is the updated score for the event, and "s" is the score for the event before the update. m " is the memory retention score (for example, 0 ≤ s m ≤ 1) and "γ" is the renewal coefficient. s' = s{1 + γ・s m} ... (15)

[0186] Furthermore, the score update unit 74 can also calculate the updated score using a value obtained by weighted addition or weighted multiplication of two or more scores from the memory retention score, comprehension score, and engagement score, along with the score before the update.

[0187] For example, the score update unit 74 can calculate the updated score s' using the following formula (16): s' = s * (s r / 2+1)・(2・s e ) ... (16)

[0188] In the above formula (16), "s r " is the comprehension score, which is an estimated result of the level of understanding (for example, -2 ≤ s r ≤ 2) and "s e " is the engagement score, which is the estimated result of engagement (for example, 0 ≤ s e The value is ≤1), and "s" is the score before the update.

[0189] Thus, the estimation unit 73 is configured to set a function (program) that increases the update score as the level of understanding and engagement increases, and decreases the update score as the level of understanding and engagement decreases.

[0190] Furthermore, the score update unit 74 updates the cumulative number of detections of the relevant event in the candidate event information of the relevant event stored in the candidate event information storage unit 12, and stores the user-presented information in the candidate event information storage unit 12 as information related to the event.

[0191] [Example of processing by the first processing unit 60] Figure 20 is a diagram illustrating the processing by the first processing unit 60 shown in Figure 18. In the example shown in Figure 20, the user-presented information acquired by the acquisition unit 70 includes "The XX cafe I went to last week was great" and "I like Ethiopian coffee beans."

[0192] The information analysis unit 71 uses morphological analysis to divide "The XX cafe I went to last week was great" and "I like coffee made from Ethiopian beans" into multiple morphemes. In the example shown in Figure 20, the information analysis unit 71 detects "last week," "Ethiopia," "beans," and "coffee" as event search keys from among the multiple morphemes divided by morphological analysis.

[0193] The event search unit 72 uses "last week," "Ethiopia," "beans," and "coffee" as event search keys to search for the relevant event among multiple candidate events stored in the candidate event information storage unit 12. In the example shown in Figure 20, the relevant event is the candidate event shown for 2025 / 01 / 22 from 14:12:22 to 17:20:10.

[0194] The estimation unit 73 calculates a memory retention score s based on the user-provided information. m The score update unit 74 calculates the memory retention score s obtained by the estimation unit 73. m Based on this, the update score for the relevant event is calculated, and the calculated update score is set as the score for that event. In the example shown in Figure 20, in the user-presented information after morphological analysis, proper nouns are underlined, and words that are the same as positive words in the word polarity dictionary are italicized.

[0195] In the example shown in Figure 20, in the above formula (14), n d = 2, n p = 2, n u = 2, n t = 16, w d = 1, w p = 1, w u = 0.5, w t = 0.1, and s m= 6.6 (= 1 × 2 + 1 × 2 + 0.5 × 2 + 0.1 × 16). In this way, the score update unit 74 calculates the memory retention score s m Therefore, 6.6 is calculated. Also, in the example shown in Figure 20, in the above formula (15), s m = 6.6, γ = 0.05, s = 0.60, and s' = 0.798. Thus, the score update unit 74 calculates 0.798 as the update score for the event.

[0196] In the example shown in Figure 20, the score update unit 74 updates the score of the relevant event from 0.60 to 0.798 and updates the cumulative number of detected candidate events from 1 to 2. The score update unit 74 also stores the user-provided information, "The XX cafe I went to last week was great" and "I like Ethiopian coffee beans," in the candidate event information storage unit 12 as information related to the relevant event.

[0197] [Other examples of processing by the first processing unit 60] The information analysis unit 71 can extract direct keywords related to understanding and engagement, such as "I want to talk," "I don't want to talk," and "I don't understand," from the user-presented information. The estimation unit 73 uses the keywords extracted by the information analysis unit 71 to calculate the engagement score s e It is possible to estimate this.

[0198] For example, if the user-presented information contains the keyword "I want to talk", the estimation unit 73 will determine s e = 1.5, and if the user-provided information includes the keyword "I don't want to talk", s e Let = 0.5. Also, if the user-presented information contains the keywords "I don't know" or "I don't remember", the estimation unit 73 will determine s e Let's assume it equals 0.1.

[0199] In this case, the score update unit 74 calculates the updated score using, for example, the following formula (17): s' = s・s e ... (17)

[0200] Figure 21 shows an example of the score update rules in the first processing unit 60 shown in Figure 18. In the example shown in Figure 21, the first processing unit 60 multiplies the score by 1.5 if the user-provided information contains the keyword "I want to talk," and multiplies the score by 0.5 if the user-provided information contains the keyword "I don't want to talk." In addition, the first processing unit 60 multiplies the score by 0.1 if the user-provided information contains the keywords "I don't know" or "I don't remember."

[0201] [Second Processing Unit 61] The second processing unit 61 updates the score of candidate events used to generate the presented information based on the user's reaction to the presented information generated by the presented information generation unit 24 and presented to the user by the information presentation unit 25.

[0202] Figure 22 is a flowchart showing an example of the processing flow by the processing unit 4, which includes the second processing unit 61 shown in Figure 18. As shown in Figure 22, the presentation information generation unit 24 acquires candidate event information to be used for generating presentation information (step S50).

[0203] Next, the presentation information generation unit 24 generates presentation information using information related to the candidate event, and the information presentation unit 25 presents the presentation information generated by the presentation information generation unit 24 to the user (step S51).

[0204] The second processing unit 61 acquires response information (for example, one or more of spoken audio, text, or video) that indicates the user's response to the information presented in step S51 (step S52). Based on the response information acquired in step S52, the second processing unit 61 estimates one or more of the following: memory retention level, comprehension level, and engagement level. Using the estimated results, it calculates an update score and updates the score of the candidate event that is the basis of the response (step S53), and then terminates the process shown in Figure 22.

[0205] As shown in Figure 18, the second processing unit 61 includes an acquisition unit 80, a reaction analysis unit 81, an estimation unit 82, and a score update unit 83. In Figure 18, the acquisition unit 62 includes the acquisition unit 70 and the acquisition unit 80, the estimation unit 63 includes the estimation unit 73 and the estimation unit 82, and the score update unit 64 includes the score update unit 74 and the score update unit 83.

[0206] [Acquisition Unit 80] The acquisition unit 80 acquires response information as response target information, which is generated by the presentation information generation unit 24 using information related to the candidate events included in the candidate event information and presented to the user by the information presentation unit 25, and which is generated by the presentation information generation unit 24. For example, the acquisition unit 80 acquires response information which is a response to the presentation information from the user's terminal device, etc., via the communication unit 2.

[0207] [Response Analysis Unit 81] The response analysis unit 81 detects specific keywords from response information that shows the user's response and outputs the detected keywords as response detection results. For example, the response analysis unit 81 can detect keywords in the user's response that are directly related to the level of understanding or engagement, such as "I want to talk," "I don't want to talk," or "I don't understand," as response detection results.

[0208] Furthermore, if the presented information is information that directly asks the user, such as "Is it alright if I talk about <event>?" or "Shall we talk about <event>?", the reaction analysis unit 81 can detect keywords such as "yes" or "no" in the user's response as a reaction detection result.

[0209] The aforementioned "<event>" includes information about candidate events that form the basis of the user's response. For example, in the case of response information that shows the user's response to presented information generated using candidate event information for candidate event A, the information included in the information related to candidate event A (e.g., proper nouns, etc.) is included.

[0210] [Estimation Unit 82] Based on response information indicating the user's response, the estimation unit 82 estimates one or more of the user's memory retention, understanding, and engagement with the candidate event that is the basis of the user's response. In the following, the candidate event that is the basis of the user's response may be referred to as the relevant event.

[0211] The estimation unit 82 calculates one or more of the following as estimation results for the degree of user retention, understanding, and engagement with the event in question: a score indicating the degree of user retention, understanding, and engagement with the event in question.

[0212] For example, if the response information acquired by the acquisition unit 80 is spoken speech, the estimation unit 82 converts the spoken speech into text using speech recognition processing or the like, and then divides the converted text into multiple morphemes using morphological analysis. The estimation unit 82 then determines the number of matches between the words in the response information and the positive words in the word polarity dictionary, and determines the number of proper nouns in the response information.

[0213] The word polarity dictionary contains a list of positive words. Alternatively, the estimation unit 82 can use a machine learning model that determines whether a noun is a positive word to determine the number of positive words among the words included in the response information, instead of the word polarity dictionary. The estimation unit 82 also detects the length of the user's utterance or the duration of the utterance as indicated in the response information. The length of the utterance is the length of the text obtained when the utterance is converted into text using speech recognition processing or the like, if the user-presented information is an utterance.

[0214] The estimation unit 82 can calculate the update score s' using the above formula (15), for example, similar to the estimation unit 73.

[0215] Furthermore, the estimation unit 82 can calculate comprehension scores and engagement scores, similar to the estimation unit 73. For example, the estimation unit 82 can obtain a comprehension score as an estimation result of the user's comprehension using the comprehension estimation model or LLM described above. Also, the estimation unit 82 can obtain an engagement score as an estimation result of the degree of the user's engagement with the response information using the engagement estimation model or LLM described above.

[0216] [Score Update Unit 83] The score update unit 83 updates the score of the relevant event with the updated score calculated by the estimation unit 82. The score update unit 83 also updates the cumulative number of detected candidate events and stores the response information in the candidate event information storage unit 12 as information related to the relevant event.

[0217] The score update unit 83, like the score update unit 74, can calculate an updated score using one of the memory retention score, comprehension score, and engagement score obtained by the estimation unit 82, along with the score before the update.

[0218] Furthermore, the score update unit 83, similar to the score update unit 74, can also calculate the updated score using a value obtained by weighted addition or weighted multiplication of two or more scores from the memory retention score, comprehension score, and engagement score obtained by the estimation unit 82, and the score before the update. For example, the score update unit 83 can calculate the updated score s' using the above formula (16).

[0219] [Example of processing by the second processing unit 61] Figure 23 is a diagram illustrating the processing by the processing unit 4, which includes the second processing unit 61 shown in Figure 18. In the example shown in Figure 23, the response information acquired by the acquisition unit 80 includes "The XX cafe I went to last week was great" and "I like Ethiopian coffee beans."

[0220] The estimation unit 82 calculates a memory retention score s based on the response information. m The score update unit 83 calculates the memory retention score s obtained by the estimation unit 82. mBased on this, the update score for the relevant event is calculated, and the calculated update score is set as the score for that event. In the example shown in Figure 23, in the response information after morphological analysis, proper nouns are underlined, and words that are the same as positive words in the word polarity dictionary are italicized.

[0221] In the example shown in Figure 23, in the above formula (14), n d = 2, n p = 2, n u = 2, n t = 16, w d = 1, w p = 1, w u = 0.5, w t = 0.1, and s m = 6.6 (= 1 × 2 + 1 × 2 + 0.5 × 2 + 0.1 × 16). In this way, the score update unit 83 calculates the memory retention score s m Therefore, 6.6 is calculated. Also, in the example shown in Figure 23, in the above formula (15), s m = 6.6, γ = 0.05, s = 0.60, and s' = 0.798. Thus, the score update unit 83 calculates 0.798 as the update score for the event.

[0222] In the example shown in Figure 23, the score update unit 83 updates the score of the relevant event from 0.60 to 0.798 and updates the cumulative number of detected candidate events from 1 to 2. The score update unit 83 also stores "The XX cafe I went to last week was great" and "I like Ethiopian coffee beans" in the candidate event information storage unit 12 as information related to the relevant event.

[0223] [Other examples of processing by the second processing unit 61] Furthermore, if the reaction detection result detected by the reaction analysis unit 81 is a direct keyword related to understanding or engagement, such as "I want to talk," "I don't want to talk," or "I don't understand," the estimation unit 82 can use such keyword to calculate an engagement score according to predetermined rules.

[0224] For example, if the response information contains the keyword "I want to talk", the estimation unit 82 will determine s e= 1.5, and if the response information contains the keyword "I don't want to talk", s e Let = 0.5. Also, if the response information contains the keyword "don't know" or "don't remember", the estimation unit 82 will... e Let it be = 0.1. The score update unit 83 calculates the update score using, for example, the above formula (17).

[0225] Figure 24 shows an example of the score update rule in the second processing unit 61 shown in Figure 18. In the example shown in Figure 24, the second processing unit 61 multiplies the score by 1.5 if the response detection result is "I want to talk", multiplies the score by 0.5 if the response detection result is "I don't want to talk", and multiplies the score by 0.1 if the response detection result is "I don't know" or "I don't remember".

[0226] Furthermore, if the reaction detection result detected by the reaction analysis unit 81 is a keyword such as "good" or "bad," the estimation unit 82 can use that keyword to calculate an update score according to predetermined rules.

[0227] For example, if the reaction detection result is "good", the estimation unit 82 will determine s e = 1.5, and if the reaction detection result is "no", s e Let it be = 0.5. The score update unit 83 calculates the update score using, for example, the above formula (17).

[0228] Figure 25 shows another example of the score update rule in the second processing unit 61 shown in Figure 18. In the example shown in Figure 25, the second processing unit 61 multiplies the score by 1.5 if the reaction detection result is "good," and multiplies the score by 0.5 if the reaction detection result is "bad."

[0229] [Presentation Information Generation Unit 24] The presentation information generation unit 24 obtains candidate event information of candidate events necessary for information presentation from the candidate event information storage unit 12, and uses the obtained candidate event information to generate presentation information (for example, one or more of spoken audio and text).

[0230] As mentioned above, candidate event information includes, for example, time information indicating a candidate event, location information corresponding to the candidate event, and event-related information such as information about events related to the candidate event.

[0231] For example, when initiating a user interaction, the presentation information generation unit 24 searches for candidate event information that satisfies predetermined conditions from among multiple candidate event information stored in the candidate event information storage unit 12.

[0232] Candidate event information that satisfies predetermined conditions may, for example, be the candidate event information of the most recent candidate event, or the candidate event information of the candidate event with the highest score among multiple candidate events up to a predetermined period prior to the present information, but is not limited to these examples. For example, candidate event information that satisfies predetermined conditions may also be randomly selected candidate event information whose score is above a threshold.

[0233] Furthermore, if the presentation information generation unit 24 is, for example, responding to a user (for example, in the middle of a conversation with a user), it searches for candidate event information related to user presentation information from among multiple candidate event information stored in the candidate event information storage unit 12. The search by the presentation information generation unit 24 is performed by keyword search, but it may also be performed by vector search.

[0234] The presentation information generation unit 24 can generate presentation information using the candidate event information with the highest score m (where m is 1 or greater) when multiple candidate event information matches the search. The presentation information generation unit 24 can also search for candidate events with scores above a threshold.

[0235] Furthermore, the information generation unit 24 can also search for candidate event information that satisfies the predetermined conditions described above, for example, when it is responding to a user (for example, in the middle of a conversation with a user), in order to provide a topic.

[0236] The presentation information generation unit 24 can, for example, cause the LLM to output presentation information by inputting information including candidate event information obtained in the search described above and an instruction sentence that instructs the generation of presentation information based on such candidate event information as input information to the LLM. In this case, the presentation information generation unit 24 can also input the user's dialogue history (for example, the history of the dialogue from the start of the dialogue to the present) to the LLM. The presentation information generation unit 24 can also generate presentation information using natural language processing other than LLM.

[0237] The presentation information generation unit 24 converts the text into speech using speech synthesis processing if the presentation information generated by LLM or natural language processing is text. This generates presentation information in speech format. The speech synthesis function may also be incorporated into the LLM.

[0238] Figure 26 shows an example of the method for generating presentation information by the presentation information generation unit 24 shown in Figure 1, and Figure 27 shows another example of the method for generating presentation information by the presentation information generation unit 24 shown in Figure 1.

[0239] Figure 26 shows an example of generating information to present to a user while interacting with them (for example, in the middle of a conversation with the user). The example shows the generation of information in response to a user's utterance, "I wonder where I went last year." The information generation unit 24 searches for candidate event information for a candidate event from the same period last year among the multiple candidate event information stored in the candidate event information storage unit 12.

[0240] The presentation information generation unit 24 generates presentation information using the candidate event information with the highest score m when multiple candidate event information matches the search. In the example shown in Figure 26, the event-related information included in the candidate event information used by the presentation information generation unit 24 is "SNS post: "I went to a hot spring!"" and "Purchase information: {Content: "Hotel", Amount: 25,000 yen}". The presentation information generation unit 24 uses "SNS post: "I went to a hot spring!"" to generate presentation information with the utterance "You went to a hot spring." as a message on the system side (information processing device 1 side), as shown in Figure 26.

[0241] Figure 27 shows an example of generating presentation information to be presented to the user when initiating a dialogue with the user. In the example shown in Figure 27, the event-related information included in the candidate event information used by the presentation information generation unit 24 is "Purchase information: {Content: "Accessories", Amount: 45,000 yen}".

[0242] The presentation information generation unit 24 uses "Purchase Information: {Content: "Accessories", Amount: 45,000 yen}" to generate presentation information as a message from the system side (information processing device 1 side), as shown in Figure 27, which is "It seems you recently purchased some accessories. Did you find a design you like?"

[0243] The presentation information generation unit 24 outputs the candidate event information used to generate the presentation information to the second processing unit 61 of the response processing unit 23. The second processing unit 61 performs update processing on the candidate event information output from the presentation information generation unit 24, as described above.

[0244] [Information Presentation Unit 25] The information presentation unit 25 presents the information to the user by transmitting the information generated by the information presentation unit 24 to the user's terminal device via the communication unit 2.

[0245] For example, if the information presented by the information generation unit 24 is spoken audio, the information presentation unit 25 presents the information to the user in a dialogue format. If the information presented by the information generation unit 24 is text, the information presentation unit 25 presents the information to the user in a chat format.

[0246] [Modification] In the above example, we mainly described an example of processing for updating the score of candidate events based on voice dialogue and chat dialogue in the response processing unit 23, but the example is not limited to this.

[0247] For example, the response processing unit 23 can update the score of candidate events based on dialogue that includes sign language and gestures in the video. For example, the response processing unit 23 can analyze sign language using known video analysis processing technology, convert it to text, and then update the score of candidate events using the converted text. In this way, the response processing unit 23 can update the score of candidate events based on the interaction between the person and the information processing device 1. Furthermore, if the presentation information is text, the presentation information generation unit 24 can present the voice uttered by the character as presentation information to the user through speech synthesis processing and video generation processing.

[0248] Furthermore, the acquisition unit 80 can also acquire information related to score updates (for example, information on memory retention, comprehension, engagement, etc.) as response information via a GUI (Graphical User Interface). The information presentation unit 25 can present information to the user that includes a GUI (for example, buttons, sliders, toggle switches, or radio buttons) for inputting information related to score updates.

[0249] Figure 28 is a diagram illustrating an example of a GUI presented to the user by the information display unit 25 shown in Figure 1. The message box 90 shown in Figure 28 includes a label with the string "Shall we continue this conversation?" and two buttons 91 and 92. Button 91 has the label "I'd like you to continue," and button 92 has the label "I'd like to talk about something else."

[0250] When the user presses button 91 as a response to the presented information using candidate event information of candidate events, the response processing unit 23 calculates an updated score by multiplying the score of the candidate event corresponding to the candidate event information used to generate the presented information by 1.5. In this case, the estimation unit 82 determines, for example, the engagement score corresponding to the button pressed by the user. The score update unit 83 calculates the updated score from the engagement score determined by the estimation unit 82 using the above formula (17).

[0251] Furthermore, as a response from the user to the presented information using candidate event information of the candidate event, the response processing unit 23 calculates an updated score when the user presses button 92, by multiplying the score of the candidate event corresponding to the candidate event information used to generate the presented information by 0.5.

[0252] Figure 29 is a diagram illustrating another example of a GUI presented to the user by the information display unit 25 shown in Figure 1. The message box 95 shown in Figure 29 includes a slider 96. The slider 96 includes a track 97, a handle 98, and labels for the string "How was the conversation?", the string "I didn't really want to talk", and the string "I wanted to talk".

[0253] The response processing unit 23 calculates an update score based on the position of the handle 98 of the slider 96 when the user changes the position of the handle 98 as a response to the user's response to the presented information using candidate event information of the candidate event. In this case, the estimation unit 82 determines, for example, an engagement score corresponding to the position of the handle 98. The score update unit 83 calculates an update score from the engagement score determined by the estimation unit 82 using the above formula (17).

[0254] For example, the left end of track 97 is set to 0 and the right end to 2, and the scale is equally spaced. The response processing unit 23 can calculate the update score using, for example, the following formula (18). In the following formula (18), "s e " is the engagement score corresponding to the position of the handle 98 moved by the user, "s" is the score before the update, and "s'" is the score after the update. s' = s * s e ... (18)

[0255] Furthermore, some or all of the functions of the information processing device 1 may be incorporated into the user's terminal device. In this case, the user's terminal device functions as part or all of the information processing device 1.

[0256] Furthermore, in the example described above, the score update unit 64 calculates the score of the searched candidate event based on the estimation results based on the user-provided information, and calculates the score of the candidate event that was the source of the response based on the estimation results based on the response information, but is not limited to this example. For example, the score update unit 64 can also estimate one or more of the user's memory retention level, understanding level, and engagement level based on the user-provided information and response information, and calculate the score of the searched candidate event that was the source of the response based on these estimation results.

[0257] [Effects of the Embodiment] As described above, the information processing device 1 is an information processing device that holds information related to a plurality of candidate events used for information presentation, and comprises an acquisition unit 62, an estimation unit 63, and a score update unit 64. The acquisition unit 62 acquires either or both of user-presented information, which is information presented by the user, and response information, which indicates the user's response to the information presentation using information related to the candidate events. The estimation unit 63 estimates one or more of the user's memory retention level, comprehension level, and engagement level based on either or both of the user-presented information and the response information when a relevant candidate event is searched from among a plurality of candidate events based on the user-presented information. The score update unit 64 updates either or both of the score of the searched candidate event and the score of the candidate event that was the source of the response, based on the estimation result by the estimation unit 63. As a result, the information processing device 1 can accurately score candidate events. For example, the information processing device 1 can accurately score information that will give a good impression to the user when presented. Furthermore, when interacting with a user through dialogue or other means, the information processing device 1 can accurately score information about memorable events that create a positive impression on the user, with the aim of entertaining the user and improving their satisfaction.

[0258] Furthermore, the information processing device 1 includes an outside-living-area determination unit 30, an outside-living-area event extraction unit 31, and a distance score calculation unit 32. The outside-living-area determination unit 30 determines that multiple clusters obtained by clustering multiple consecutive location information of the user are within the living area, in order of the number of location information, and the remaining clusters are outside the living area. The distance score calculation unit 32 extracts the time when the user is continuously located outside the living area as an outside-living-area event, based on the location information outside the living area. The distance score calculation unit 32 calculates a distance score, which is a score based on the distance from the location indicated by the location information within the living area to the location indicated by the location information outside the living area. As a result, the information processing device 1 can accurately detect outside-living-area events, which are events that occur outside the user's living area, and can also detect a score related to the distance of outside-living-area events. Furthermore, for example, in the technology described in Non-Patent Literature 1 above, there are many data input steps involving the user that are necessary for extraction (such as the start and end times of the event and data on all people who were at the event), which places a heavy burden on the user and makes data input errors likely. On the other hand, the information processing device 1 can reduce the burden on the user by using automatically collectible location information, and can also avoid user data input that may be prone to errors. Furthermore, for example, the technologies described in Non-Patent Documents 1 and 2 mentioned above both require manual setting of criteria such as the user's home to extract information, making it difficult to keep up with user characteristics and changes in daily life. For example, because it determines places the user does not visit often based on the distance from their home, it is difficult to accommodate changes in locations other than home, such as workplaces, or relocation. On the other hand, the information processing device 1 can keep up with user characteristics and changes in daily life by determining that multiple clusters are within the user's living area in order of the amount of location information, and the remaining clusters are outside the living area.

[0259] Furthermore, the information processing device 1 includes an event information processing unit 22. The event information processing unit 22 detects events outside the user's living area and calculates a score for those events, based on user information related to events outside the user's living area, including one or more pieces of information such as distance score, user purchase amount, user count, and aesthetic score of the user's photos. This allows the information processing device 1 to accurately detect and score information about impressive events that will give a good impression to the user when presented. In addition, for example, the technology described in Non-Patent Document 1 above involves many data input steps requiring user intervention for extraction, which places a heavy burden on the user and makes data input errors likely. On the other hand, the information processing device 1 reduces the burden on the user by using automatically collectible location information and avoids user data input that may be prone to errors.

[0260] [System Configuration, etc.] Furthermore, the components of each part shown in the diagram are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown in the diagram, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. In addition, all or any part of the processing functions performed by each device can be realized by a CPU and the program executed on that CPU, or by hardware using wired logic.

[0261] Furthermore, among the processes described in the embodiments above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0262] [Program] A program can also be created that describes the processing performed by the information processing device 1 according to the above embodiment in a language that can be executed by a computer. In one embodiment, the information processing device 1 can be implemented by installing an information processing program that performs the above information processing as packaged software or online software on a desired computer. For example, by having the information processing device execute the above information processing program, the information processing device can be made to function as the information processing device 1. The information processing device referred to here is a general-purpose computer, but is not limited to this example, and may be a general-purpose server (for example, an on-premise server or a cloud server), or any other computer.

[0263] Figure 30 shows an example of a computer that executes an information processing program. Computer 1000 includes, for example, memory 1010, CPU 1020, hard disk drive interface 1030, disk drive interface 1040, serial port interface 1050, video adapter 1060, and network interface 1070. These components are connected by a bus 1080.

[0264] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1031. The disk drive interface 1040 is connected to the disk drive 1041. A removable storage medium, such as a magnetic disk or optical disk, is inserted into the disk drive 1041. A serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. A video adapter 1060 is connected to, for example, a display 1061.

[0265] Here, the hard disk drive 1031 stores, for example, the OS 1091, the application program 1092, the program module 1093, and the program data 1094. The information described in the above embodiment is stored, for example, in the hard disk drive 1031 or the memory 1010.

[0266] Furthermore, the information processing program is stored in the hard disk drive 1031 as a program module 1093 containing instructions to be executed by the computer 1000, for example. Specifically, the program module 1093 containing instructions for each process executed by the information processing device 1 described in the above embodiment is stored in the hard disk drive 1031.

[0267] Furthermore, the data used for information processing by the information processing program is stored as program data 1094, for example, in the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as needed and executes the procedures described above.

[0268] Furthermore, the program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1031; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 related to the information processing program may be stored in another computer connected via a network such as a LAN or WAN and read by the CPU 1020 via a network interface 1070.

[0269] The functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0270] There is a memory for storing a computer program that includes computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0271] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention.

[0272] 1 Information Processing Unit 2 Communication Unit 3 Storage Unit 4 Processing Unit 10 Location Information Storage Unit 11 Event-Related Information Storage Unit 12 Candidate Event Information Storage Unit 20, 40, 50, 62, 70, 80 Acquisition Unit 21 Location Information Processing Unit 22 Event Information Processing Unit 23 Response Processing Unit 24 Presentation Information Generation Unit 25 Information Presentation Unit 30 Outside Living Area Determination Unit 31 Outside Living Area Event Extraction Unit 32 Distance Score Calculation Unit 41 Clustering Unit 42 Determination Processing Unit 51 Event Detection Unit 52 Score Calculation Unit 60 First Processing Unit 61 Second Processing Unit 63, 73, 82 Estimation Unit 64, 74, 83 Score Update Unit 71 Information Analysis Unit 72 Event Search Unit 81 Reaction Analysis Unit 90, 95 Message Box 91, 92 Button 96 Slider 97 Track 98 Handle

Claims

1. An information processing device that holds information related to a plurality of candidate events used for presenting information, comprising: an acquisition unit that acquires one or both of user-presented information, which is information presented by the user, and response information, which indicates the user's response to the information presentation using the information related to the candidate events; an estimation unit that estimates one or more of the user's memory retention level, understanding level, and engagement level based on one or both of the user-presented information and the response information when a related candidate event is found among the plurality of candidate events based on the user-presented information; and a score update unit that updates one or both of the score of the found candidate event and the score of the candidate event that was the source of the response, based on the estimation result by the estimation unit.

2. An information processing device according to claim 1, comprising: a living area determination unit that determines, among a plurality of clusters obtained by clustering a plurality of consecutive location information of the user, to be a living area in order of the number of location information, and the remaining clusters to be outside the living area; an living area event extraction unit that extracts the time when the user is continuously located outside the living area as an living area event based on the location information outside the living area; and a distance score calculation unit that calculates a distance score which is a score based on the distance from a location indicated by the location information within the living area to a location indicated by the location information outside the living area.

3. The information processing device according to claim 2, further comprising an event information processing unit that detects an event outside the user's living area and calculates a score for the event outside the user's living area, based on information relating to the event outside the user's living area, which includes one or more pieces of information from the distance score, the user's purchase amount, the user's count value, and the aesthetic score of the user's photograph, and in which the relevant information is used as a candidate event in presenting information to the user.

4. An information processing method performed by an information processing device that holds information related to a plurality of candidate events used for information presentation, characterized in that: it acquires one or both of user-presented information, which is information presented by the user, and response information, which indicates the user's response to the information presentation using information related to the candidate events; when a relevant candidate event is found among the plurality of candidate events based on the user-presented information, it estimates one or more of the user's memory retention level, comprehension level, and engagement level based on one or both of the user-presented information and the response information; and updates one or both of the score of the found candidate event and the score of the candidate event that was the source of the response based on the estimated result.