Generation device, generation method, and generation program

Abstract

A generation device (10) includes: a selection unit (122) that selects a definition document corresponding to a user query from among a plurality of definition documents including at least an analysis method for an article and a first prompt instructing generation of a report related to the article; an external information analysis unit (123) that analyzes a plurality of articles by using the analysis method included in the definition document selected by the selection unit (122); and a generation unit (124) that causes a large language model (LLM) 20 to generate a report on the basis of the first prompt included in the definition document selected by the selection unit (122) and the analysis result by the external information analysis unit (123).

Description

Generation Device, Generation Method, and Generation Program

[0001] The present invention relates to a generation device, a generation method, and a generation program.

[0002] In the investigation of cyber threats in the company's system environment and the consideration of daily vulnerabilities and the company's countermeasures against cyberattacks, there is a task of collecting information and reporting it. For example, a responsible analyst collects security threat trend information inside and outside the security news site and the company's system, and generates a report based on their findings. Report generation involves the collection from a huge information source and the work of summarizing them according to the needs, which becomes the operation load of expert analysts, and reducing the operation is an issue.

[0003] In order to solve such problems, a technology for collecting open-source information and generating a security report in a story composition has been proposed (for example, Non-Patent Document 1). In this technology, when a user asks a question about what they want to know, information is collected from a database (DB), a prompt is constructed in a story composition, and a sentence according to the prompt is generated by a large language model (LLM: Large Language Models).

[0004] Priyanka Ranade, Anupam Joshi, “FABULA: Intelligence Report Generation Using Retrieval-Augmented Narrative Construction,” 2023 IEEE / ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), [Searched on October 24, 2024], Internet <URL: https: / / arxiv.org / pdf / 2310.13848>

[0005] FIG. 27 is a diagram for explaining the prior art. In the prior art (for example, Non-Patent Document 1), key elements are extracted from external information by natural language analysis (FIG. 27(1)) and stored in a DB (FIG. 27(2)).

[0006] Conventional technology generates a narrative-style intelligence report using LLM based on accumulated information in response to inquiries from security personnel regarding "cyberattacks on company xx." Conventional technology extracts and aggregates elements from multiple article data (hereinafter referred to as articles), such as news articles, when generating a report. Conventional technology performs extraction and aggregation, for example, extracting 5W1H elements from one article and combining duplicates from multiple articles. Conventional technology generates text by giving the prompt to LLM a narrative structure (lead, body, conclusion) (Figure 27 (3)).

[0007] Conventional technologies focus on accumulating information related to specific incidents and events and generating reports. In other words, conventional technologies extract and organize 5W1H information from individual news articles and intelligence reports. Therefore, conventional technologies cannot accurately answer specific questions such as, "Which vulnerabilities were frequently reported in articles in September 2024?"

[0008] However, with conventional technology, for example, words like "attention" or "related" are not stored in a direct relationship with the articles. Therefore, conventional technology has the challenge of not being able to derive accurate answers to abstract questions such as "vulnerabilities that have recently attracted attention" or "the number of threats related to our company in the past month."

[0009] The present invention has been made in view of the above, and aims to provide a generation device, generation method, and generation program that can generate highly accurate reports even for abstract questions.

[0010] To solve the above-mentioned problems and achieve the objective, the generation device according to the present invention is characterized by comprising: a selection unit that selects a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a method for analyzing an article and a first prompt for instructing the generation of a report relating to the article; an analysis unit that analyzes a plurality of articles using the analysis method of the definition document selected by the selection unit; and a generation unit that causes a large-scale language model to generate a report based on the first prompt of the definition document selected by the selection unit and the analysis results by the analysis unit.

[0011] Furthermore, the generation method according to the present invention is a generation method executed by a generation device, and is characterized by including the steps of: selecting a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a method for analyzing an article and a first prompt for instructing the generation of a report relating to the article; analyzing a plurality of articles using the analysis method of the definition document selected in the selection step; and causing a large-scale language model to generate a report based on the first prompt of the definition document selected in the selection step and the analysis results in the analysis step.

[0012] Furthermore, the generation program according to the present invention includes the steps of: selecting a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a method for analyzing an article and a first prompt for instructing the generation of a report relating to the article; analyzing a plurality of articles using the analysis method of the definition document selected in the selection step; and causing a large-scale language model to generate a report based on the first prompt of the definition document selected in the selection step and the analysis results in the analysis step; and having a computer execute these steps.

[0013] According to the present invention, it is possible to generate highly accurate reports even for abstract questions.

[0014] Figure 1 is a diagram illustrating the outline of an embodiment. Figure 2 is a diagram showing an example of the configuration of the generation device according to the embodiment. Figure 3 is a flowchart showing the processing procedure of the information collection process in the embodiment. Figure 4 is a flowchart showing the processing procedure of the report generation process according to the embodiment. Figure 5 is a diagram showing an example of the definition of the collected content. Figure 6 is a diagram showing an example of the collected results. Figure 7 is a diagram illustrating the processing of the external information collection unit shown in Figure 2. Figure 8 is a diagram showing the processing flow of the report generation unit shown in Figure 2. Figure 9-1 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-2 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-3 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-4 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-5 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-6 is a diagram showing an example of data used or generated in the processing of the report generation unit shown in Figure 2. Figure 9-7 is a diagram showing an example of data used or generated during the processing of the report generation unit shown in Figure 2. Figure 10 is a diagram illustrating the processing of the selection unit shown in Figure 2. Figure 11 is a diagram illustrating the report definition document. Figure 12-1 is a diagram showing an example of input and output data of the external information analysis unit shown in Figure 2. Figure 12-2 is a diagram showing an example of input and output data of the external information analysis unit shown in Figure 2. Figure 12-3 is a diagram showing an example of input and output data of the external information analysis unit shown in Figure 2. Figure 13 is a diagram illustrating the processing flow of the external information analysis unit shown in Figure 2. Figure 14 is a flowchart showing the grouping procedure executed by the external information analysis unit shown in Figure 2. Figure 15 is a diagram illustrating data obtained from the external information DB by the external information analysis unit shown in Figure 2. Figure 16 is a diagram illustrating the processing of the external information analysis unit shown in Figure 2. Figure 17 is a diagram showing an example of an article group extracted by the external information analysis unit shown in Figure 2. Figure 18 is a diagram showing an example of summary information for each article in the article group. Figure 19 is a diagram showing an example of prompts used for report generation.Figure 20 shows an example of a report generated by LLM. Figure 21 shows an example of a report generated by the generation unit shown in Figure 2. Figure 22 shows an example of a report generated by the generation unit shown in Figure 2. Figure 23 shows an example of a report generated by conventional technology. Figure 24 shows an example of a report output by the generation device shown in Figure 2. Figure 25 shows an example of a report created by a security expert. Figure 26 shows an example of a computer in which the generation device is realized by the execution of a program. Figure 27 is a diagram illustrating conventional technology.

[0015] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0016] [Embodiment] [Overview of the Embodiment] Figure 1 is a diagram illustrating the overview of the embodiment. The generation device according to the embodiment collects multiple article data (hereinafter referred to as articles) from an external site, etc., and stores the collected articles in an external information DB after vectorization processing and / or feature extraction processing (Figure 1(1), (2)).

[0017] This section describes the case where the generator receives an abstract question from a user (for example, "What are the recent notable vulnerabilities?"). In this case, the generator has the LLM select a report definition document (hereinafter referred to as the report definition document) designed to derive an accurate answer to the user's question (user query) (Figure 1 (3)), and then searches for articles in the external information database using the search key that resolves the user query (Figure 1 (4), (5)). The report definition document includes prompts (first prompt) that instruct how to analyze the articles and generate the report.

[0018] The generation device performs an analysis method (for example, aggregate analysis (grouping) of the searched articles) according to the selected report definition document for the searched articles (Figure 1 (5)). By performing the analysis method, the generation device aggregates and analyzes the articles, and allows the LLM to refer to article groups that include context such as frequency information. This allows the generation device to enable the LLM to generate appropriate reports even for abstract questions such as questions about trends (Figure 1 (6)). The generation device outputs the report generated by the LLM to the user.

[0019] [Generation Apparatus] Next, a generation apparatus according to the embodiment will be described. Figure 2 is a diagram showing an example of the configuration of the generation apparatus according to the embodiment.

[0020] The generation device 10 according to this embodiment is a generation device that is realized by loading a predetermined program into a computer including, for example, ROM (Read Only Memory), RAM (Random Access Memory), CPU (Central Processing Unit), etc., and having the CPU execute the predetermined program. The generation device 10 also has a communication interface for sending and receiving various information with other devices connected via a network, etc. The generation device 10 communicates with a user terminal used by the user and a server that provides a collection site 40 for collecting news articles, etc.

[0021] The generation device 10 includes an information collection unit 11, a report generation unit 12, an LLM 20, and an external information DB 30.

[0022] LLM20 is a natural language processing model that processes input text data in natural language according to a first prompt set in LLM20, and for example, generates and outputs a report.

[0023] The external information DB30 stores various articles, such as news articles, which are considered external information.

[0024] The information gathering unit 11 accesses the collection site 40, collects news articles and other articles as external information, and stores them in the external information DB 30. The information gathering unit 11 also has an external information gathering unit 111.

[0025] The external information collection unit 111 accesses the collection site 40 in accordance with periodic execution instructions and / or execution orders and collects articles, which are external information. The external information collection unit 111 stores the collected articles in the external information DB 30.

[0026] When the report generation unit 12 receives a user query as user input, it searches the external information DB 30 for articles corresponding to the user query. Based on the analysis results obtained from analyzing the searched articles and a first prompt instructing the LLM 20 to generate a report that answers the user query, the report generation unit 12 instructs the LLM 20 to generate a report that answers the user query. The report generation unit 12 outputs the generated report to the user.

[0027] The report generation unit 12 includes a report definition storage unit 121, a selection unit 122, an external information analysis unit 123 (analysis unit), and a generation unit 124.

[0028] The report definition storage unit 121 stores multiple report definitions. Each report definition is designed to guide the user to a precise response based on the prompts provided. Each report definition includes at least a first prompt that instructs the system on how to analyze an article and generate a report about the article. Report definitions are set up, for example, for each category.

[0029] The selection unit 122 selects a report definition document that corresponds to the user query from among multiple report definition documents stored in the report definition document storage unit 121.

[0030] The external information analysis unit 123 analyzes multiple articles using the analysis method specified in the report definition document selected by the selection unit 122. The external information analysis unit 123 searches for articles corresponding to user queries from the external information DB 30, which stores multiple articles. The external information analysis unit 123 analyzes the retrieved articles using the analysis method specified in the report definition document selected by the selection unit 122.

[0031] The generation unit 124 causes the LLM 20 to generate a report based on the first prompt of the report definition selected by the report definition storage unit 121 and the analysis results from the external information analysis unit 123. The generation unit 124 outputs the report generated by the LLM 20.

[0032] [Processing Procedure for Information Collection] Figure 3 is a flowchart showing the processing procedure for information collection in the embodiment.

[0033] As shown in Figure 3, when the external information collection unit 111 receives a periodic execution order or execution command (step S1), it accesses the collection site 40 to collect articles, which are external information (step S2), and stores them in the external information DB 30 (step S3).

[0034] [Processing Procedure for Report Generation] Figure 4 is a flowchart showing the processing procedure for report generation according to the embodiment.

[0035] When the report generation unit 12 receives a user query as user input (step S11), the selection unit 122 selects a report definition document that corresponds to the user query from among multiple report definition documents (step S12).

[0036] The external information analysis unit 123 searches the external information DB 30 for articles corresponding to the user query, and analyzes the searched articles using the analysis method of the report definition document selected by the selection unit 122 (step S13).

[0037] The generation unit 124 instructs the LLM 20 to generate a report that answers the user query, based on the first prompt of the report definition selected by the selection unit 122 and the analysis results from the external information analysis unit 123 (step S14). The report generation unit 12 outputs the report generated by the LLM 20 to the user (step S15).

[0038] [External Information Collection Unit] Next, the processing of the external information collection unit 111 will be explained. Figure 5 is a diagram showing an example of the definition of the collection contents. Figure 6 is a diagram showing an example of the collection results.

[0039] The external information collection unit 111 collects information (articles) from news sites searchable on the internet, according to the definition D1 of the collection content shown in Figure 5. Definition D1 defines the collection content as the title, category of information to be obtained from the web, date, detailed description, and URL (Uniform Resource Locator) information of the information provider. Definition D1 is predefined.

[0040] Figure 6 shows an example of information collected from, for example, "https: / / thehackernews.com". The external information collection unit 111 saves the collected articles in, for example, RDB (relational database) format. Alternatively, the external information collection unit 111 may perform embedding (embedded vector) processing on the content of the articles and save them in VectorDB (external information DB 30).

[0041] Furthermore, the external information collection unit 111 may process the collected articles. Figure 7 is a diagram illustrating the processing performed by the external information collection unit 111 shown in Figure 2.

[0042] For example, the external information collection unit 111 causes the LLM 20 to extract CVE (Common Vulnerabilities and Exposures) numbers and / or keywords contained in the article and stores them in the external information DB 30 as metadata. The external information collection unit 111 provides the LLM 20 with a Context and a Question based on a Template, and causes the LLM 20 to extract CVE numbers and keywords from the Context in order to answer the Question. As a result, the LLM 20 extracts the CVE numbers and keywords shown in frames W11 to W14 from the Context. The external information collection unit 111 stores the metadata D15 containing each of the extracted pieces of information in the external information DB 30.

[0043] [Report Generation Unit] Next, the processing of the report generation unit 12 will be explained. First, the processing flow of the report generation unit 12 will be explained. Figure 8 is a diagram showing the processing flow of the report generation unit 12 shown in Figure 2. Figures 9-1 to 9-7 are diagrams showing examples of data used or generated during the processing of the report generation unit 12 shown in Figure 2.

[0044] First, the user query D21 input by the user is input to the selection unit 122 (Figure 8(1)). The user query D21 is, for example, a question such as "Tell me about the vulnerability information that was noticed in one month in March 2024" as shown in Figure 9-1, and contains an abstract expression "noticed".

[0045] The selection unit 122 selects a report definition document corresponding to the user query from the report definition document storage unit 121 (Figure 8(2)). The selection unit 122 gives the selected report definition document and the user query D21 to the LLM 20, and based on the content of the report definition document, generates search queries D22, D23 (Figure 9-2, Figure 9-3) corresponding to the user query. The selection unit 122 gives the search queries D22, D23 including search keys (keywords, search period) and the analysis method of the selected report definition document to the external information analysis unit 123.

[0046] The external information analysis unit 123 searches for articles from the external information DB 30 according to the search queries D22, D23 (Figure 8(3)). Subsequently, the external information analysis unit 123 performs analysis on the retrieved articles using the analysis method (for example, aggregation analysis) of the report definition document (Figure 8(4)), and obtains, for example, an analysis result D24 (Figure 9-4).

[0047] The external information analysis unit 123 gives the sum of the analysis result D24 and the report definition document to the LLM 20, and outputs an analysis instruction text D25 (Figure 9-5) to the LLM 20. The external information analysis unit 123 analyzes the analysis result D24 and the retrieved articles according to the analysis instruction text D25, and obtains an analysis result D26 (Figure 9-6).

[0048] The external information analysis unit 123 outputs the information of the prompt (first prompt) of the selected report definition document (Figure 8(5)) and the analysis result D26 to the generation unit 124 for generating an article (report) (Figure 8(6)).

[0049] The generation unit 124 causes the LLM 20 to generate a report based on the prompts in the report definition selected by the report definition storage unit 121 and the analysis result D26 from the external information analysis unit 123. For example, the LLM 20 generates report D27 (Figure 9-7) according to the prompts including the analysis result D26. Report D27 contains the item "Vulnerabilities that attracted attention in March 2024" and its explanation in response to the question "Tell me about the vulnerabilities that attracted attention in March 2024". The generation unit 124 outputs the generated report D27 to the user (Figure 8 (7)).

[0050] [Selection Section] Next, the processing of the selection section 122 will be explained. Figure 10 is a diagram illustrating the processing of the selection section 122 shown in Figure 2.

[0051] The selection unit 122 uses the LLM 20 to select a report definition document from a plurality of predefined report definition documents in order to output a report from a perspective suitable for the user query. Each report definition document has items such as type, description, and structure, as shown in the report definition document group R31 in Figure 10.

[0052] The selection unit 122 extracts one or more items from the report definition document, provides the extracted one or more items and the user query to the LLM 20, and has the LLM 20 select a report definition document that is closely related to the user query based on the similarity between the user query and the one or more items, and / or a search using common keywords.

[0053] For example, the selection unit 122 extracts items (for example, only the explanation section) from the report definition document group R31 and provides the LLM20 with a prompt P32 in which the extracted items (for example, "Check the status of the company's systems," etc.) are written in the Context section (line P33) and the user query is written in the Question section. The LLM20 outputs the item "Check the status of the company's systems" D34, which is closely related to the user query, according to prompt P32. The selection unit 122 selects the definition 1 that matches the item "Check the status of the company's systems" D34 as the report definition document.

[0054] Next, we will explain the structure of the report definition document. Figure 11 is a diagram illustrating the report definition document. Each report definition document has items (elements) for type, description, and report structure, as mentioned above.

[0055] The "typename" field contains category information. Examples of types include "Quarterly briefing for management," "Vulnerability topics for employees," and "Emergency response report," as illustrated in Figure 11.

[0056] The description field should contain a natural language description of the specific content of the report. This field is used by LLM20 to infer which report output is appropriate in response to a user query.

[0057] The report structure (quest_file) contains prompt statements for each item in the report, which are used to generate text appropriate to each item. The report structure contains prompt statements (second prompts) that cause the LLM20 to generate a search query to the external information DB30 based on the user query, instructions for the analysis method (aggregation analysis of articles) of the external information analysis unit 123, and prompt statements (first prompts) that specify the output format of the final generated report.

[0058] The prompts and analysis method instructions include LLM configuration instructions, the names of the functions to be executed (corresponding to report items), and the input information to be passed to the functions. Furthermore, when performing trend analysis, information such as which items to sort the acquired data by for analysis is provided within the analysis method instructions. When using LLM, the prompts to be set in LLM are also provided within the analysis method instructions. Additionally, the prompts and analysis method instructions may include information called "tools," which allows LLM to autonomously execute functions.

[0059] [External Information Analysis Unit] Next, the processing of the external information analysis unit 123 will be explained. Figures 12-1 to 12-3 show examples of input and output data of the external information analysis unit 123 shown in Figure 2.

[0060] The external information analysis unit 123 receives the user query D51 (Figure 12-1) input to the generation device 10 and the report definition document R52 (Figure 12-2) selected by the selection unit 122. The external information analysis unit 123 analyzes the acquired articles using the analysis method (for example, grouping) (row R53) of the report definition document R52 and outputs the analysis result D55 (Figure 12-3) to the generation unit 124.

[0061] One possible method for grouping reports is keyword matching. For example, LLM20 could be used to generate keywords from multiple articles, and the articles could be grouped based on the degree of keyword matching between them. However, in many security reports, keywords are assigned after the report has been published. Therefore, in this embodiment, we will explain, as an example, a method of vectorizing articles and grouping them using the vectors.

[0062] Next, an example of the processing flow of the external information analysis unit 123 will be described. Figure 13 is a diagram illustrating the processing flow of the external information analysis unit 123 shown in Figure 2. Figure 13 shows an example in which information for a predetermined period is searched from the external information DB 30, the obtained articles are grouped, and the CVE number with the most articles is output.

[0063] The external information analysis unit 123 receives the user query D61 and the selected report definition information D62 as input (step S21). The selected report definition indicates that the analysis method involves grouping articles and obtaining statistical data for the grouped article groups. Along with this, the selected report definition includes a first prompt instructing the external information analysis unit 123 to create a report in a predetermined output format by referring to the analysis results. For example, the report definition indicates that the analysis method involves grouping articles based on their similarity.

[0064] The external information analysis unit 123 determines the search period (start date - end date, 1 month) for article searches in the external information DB 30 based on the user query D61 (step S22).

[0065] The external information analysis unit 123 searches the external information DB 30 for articles for the search period determined in step S22 (step S23). The external information analysis unit 123 searches the external information DB 30 for articles using a search query. The search query is generated by setting the second prompt of the report definition document selected by the selection unit 122 to LLM20.

[0066] The external information analysis unit 123, for example, groups the multiple articles obtained by the search in step S23 based on the similarity between the multiple articles (step S24).

[0067] The external information analysis unit 123 then performs statistical information analysis on the grouped article groups and obtains statistical data for the article groups as analysis results (step S25). For example, the external information analysis unit 123 ranks each group in order of the number of articles and outputs analysis results D64 and D65, which include the identification information and number of articles for the top N groups. The number of groups to output N is set in the report definition document, and may also be adjusted according to user queries and analysis results.

[0068] If the external information analysis unit 123 has another target for analysis (step S26: Yes), it returns to the analysis to perform the analysis on the next target (step S28).

[0069] If there are no further analysis targets (step S26: No), the external information analysis unit 123 sends the analysis results D64 and D65 to the report generation unit (step S27) and terminates the analysis process.

[0070] [Grouping] Next, an example of the grouping procedure will be described. Figure 14 is a flowchart showing the grouping procedure performed by the external information analysis unit 123 shown in Figure 2.

[0071] First, the external information analysis unit 123 retrieves a group of articles Ga from the external information DB 30 by specifying a period (step S31). Note that if the external information analysis unit 123 is only retrieving vulnerabilities, it may add the requirement that CVE numbers are assigned to the search conditions.

[0072] The external information analysis unit 123 selects one article Rb from the article group Ga (step S32) and obtains the vector Vb of article Rb (step S33). In this process, the external information analysis unit 123 uses two vectors: vector information B-1 of the entire article group Ga and vector information B-2 of the entire metadata.

[0073] The external information analysis unit 123 calculates the similarity between the vectors included in the records of the article group Ga and the vector Vb obtained in step S33 (step S34), and obtains all articles Rc of records whose calculated similarity exceeds a predetermined threshold (step S35). Since not all articles may have a category such as a CVE number assigned to them, the external information analysis unit 123 groups the articles based on the overall similarity of the articles.

[0074] The external information analysis unit 123 assigns the group Gd category to articles Rb and Rc and saves them (step S36). The external information analysis unit 123 removes group Gd from article group Ga (step S37).

[0075] The external information analysis unit 123 determines whether or not there is data in the article group Ga (step S38). If there is data in the article group Ga (step S38: Yes), the external information analysis unit 123 returns to step S32 and continues the grouping process. If there is no data in the article group Ga (step S38: No), the external information analysis unit 123 outputs the category (for example, CVE number) for each group (step S39).

[0076] The data actually processed by the external information analysis unit 123 will now be explained. Figure 15 is an example of the data obtained from the external information DB 30 by the external information analysis unit 123 shown in Figure 2. As shown in Figure 15, the external information analysis unit 123 obtains articles from the external information DB 30 that are within the search period and whose metadata (cve_number) starts with CVE.

[0077] Figure 16 is a diagram illustrating the processing of the external information analysis unit 123 shown in Figure 2. As shown in Figure 16, the external information analysis unit 123 searches for similarity within the external information DB 30 based on the vector information B-1 (embedding1) of the entire article group Ga and the vector information B-2 (embedding2) of the entire metadata, and acquires those that exceed a predetermined threshold.

[0078] The external information analysis unit 123 extracted articles that satisfy, for example, a vector similarity of the entire article > 0.7 and a vector similarity of the metadata > 0.5 as a group. Figure 17 shows an example of an article group extracted by the external information analysis unit 123 shown in Figure 2. As shown in each article group G1 to G4 in Figure 17, it was confirmed that no articles with different content were included within the same article group. Also, as shown in each article group G1 to G4, it was confirmed that the necessary articles were grouped together in one article group.

[0079] Figure 18 shows an example of summary information for each article in an article group. The external information analysis unit 123 generated summary information (Figure 18) for each article in a grouped article group, and visually confirmed that all the data within the same article group described the same event separately.

[0080] In this way, the external information analysis unit 123 can accurately group articles that introduce the same content into a single article group by performing grouping. The external information analysis unit 123 then ranks each article group in order of the number of articles and outputs the CVE numbers and number of articles of the top N groups as analysis information to the generation unit 124. This processing by the external information analysis unit 123 can be said to be presenting CVE numbers corresponding to the "highlighted" vulnerabilities of the user query in order of their level of attention by presenting the CVE numbers in order of the number of articles. In other words, the external information analysis unit 123 presents CVE numbers corresponding to the abstract questions of the user query by outputting the CVE numbers of the groups in order of the number of articles after grouping the articles.

[0081] In addition, the external information analysis unit 123 may calculate the number of articles in an article group as statistical data, and may also calculate the frequency of keyword occurrences in the article group as statistical data.

[0082] In this case, the report definition document indicates that the frequency of keyword occurrences in the article group and / or the number of articles in the article group will be calculated as statistical data. The external information analysis unit 123 then calculates the frequency of keyword occurrences and / or the number of articles in the article group as statistical data and outputs the calculated statistical data in association with the article group as an analysis result. When calculating the frequency of keyword occurrences as statistical data, the external information analysis unit 123 may, for example, have the LLM 20 write out the keywords of multiple articles and group them based on the degree of keyword matching between articles.

[0083] [Generation Unit] Next, the processing of the generation unit 124 will be described. The generation unit 124 sets the first prompt of the report definition document selected by the selection unit to the LLM 20, and, referring to the statistical data of the article group by the external information analysis unit 123, causes the LLM 20 to generate a report.

[0084] Figure 19 shows an example of prompts used for report generation. Figure 20 shows an example of a report generated by LLM20. The report in Figure 20 was generated by LLM20 according to the prompts shown in Figure 19.

[0085] As shown in Figure 19, the generation unit 124 provides the LLM 20 with the prompts (instructions and output format settings) described in the report definition document selected by the selection unit 122 and the analysis results (analysis data) from the external information analysis unit 123 as input P71 (Figure 19), causing it to generate a report. The LLM 20 refers to the analysis data of input P71 and generates a report D72 (Figure 20) that includes the CVE number and its description as "Information on vulnerabilities that attracted attention in March 2024". The generation unit 124 outputs the report D72 to the user.

[0086] Figures 21 and 22 show examples of reports generated by the generation unit 124 shown in Figure 2. If the user input is "Tell me about vulnerabilities that attracted attention in the month of April 2024," the generation unit 124 generates report R81 (Figure 21), which shows the CVE number and the number of articles, as shown in frame W71, as information about vulnerabilities that attracted attention in the month of April 2024. If the user input is "Tell me about vulnerabilities that attracted attention in the last month," the generation unit 124 generates report R82 (Figure 22), which lists the CVE numbers, as shown in frame W82, as information about vulnerabilities that attracted attention in the last month.

[0087] [Effects of the Embodiment] The generation device 10 according to the embodiment selects a report definition document corresponding to a user query from among a plurality of report definition documents, each of which includes at least a method for analyzing articles and a first prompt for instructing the generation of a report. The generation device 10 analyzes the plurality of articles using the analysis method of the selected report definition document. Then, the generation device 10 causes the LLM 20 to generate a report based on the first prompt of the selected report definition document and the analysis results.

[0088] In this way, the generation device 10 generates a report by having the LLM 20 refer to the analysis results, which include statistical processing of multiple articles. As a result, the generation device 10 generates a report that takes into account trends and analysis information such as which articles covered which topics, and can therefore generate a highly accurate report even for abstract questions.

[0089] Figure 23 shows an example of a report generated by conventional technology. As shown in Figure 23, with conventional technology, if the user input is "Tell me about the vulnerabilities that were highlighted in the month of April 2024," the report is generated listing CVE-2024-0156 (OpenSSL) and CVE-2024-0211 (M) as vulnerability information.

[0090] However, in the report in Figure 23, CVE-2024-0156 is the identifier for a buffer overflow CVE from Company D that was disclosed in March, and CVE-2024-0211(M) is the identifier for a CVE from Company W that was disclosed in January. The information described in Figure 23 also differs in its disclosure timing, and the information selected as a vulnerability of interest is different from the vulnerability of interest in April. For this reason, the report generated using conventional technology was inaccurate.

[0091] Figure 24 shows an example of a report output by the generation device 10 shown in Figure 2. The generation device 10 generated the report in Figure 24 in response to the same user input as the report in Figure 23.

[0092] Figure 25 shows an example of a report created by a security expert. The report in Figure 25 was created in response to a request to "tell us about the vulnerabilities that attracted attention during the month of April 2024." The security expert themselves searched for news articles related to vulnerabilities and presented the vulnerability information based on the search results.

[0093] Comparing the report in Figure 24, output by the generator 10, with the report in Figure 25, the report output by the generator 10 includes the same items as the report in Figure 25, specifically regarding vulnerabilities that attracted attention due to their release in April. Therefore, it can be said that the generator 10 can produce reports equivalent to those produced by experts. Thus, the generator 10 can generate highly accurate reports even for abstract questions.

[0094] [Regarding the System Configuration of the Embodiment] Each component of the generation device 10 is a functional concept and does not necessarily have to be physically configured as shown in the figure. In other words, the specific forms of distribution and integration of the functions of the generation device 10 are not limited to those shown in the figure, and all or part of it can be configured by functionally or physically distributing or integrating in any unit according to various loads and usage conditions.

[0095] Furthermore, each process performed in the generation device 10 may be implemented in whole or in part by a CPU, a GPU (Graphics Processing Unit), and a program that is analyzed and executed by the CPU and GPU. Alternatively, each process performed in the generation device 10 may be implemented as hardware using wired logic.

[0096] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated may be changed as appropriate unless otherwise specified.

[0097] [Program] Figure 26 shows an example of a computer in which the generation device 10 is realized when a program is executed. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These parts are connected by a bus 1080.

[0098] Memory 1010 includes ROM 1011 and RAM 1012. ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0099] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the generation device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing the same processes as the functional configuration in the generation device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0100] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes them.

[0101] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0102] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in these embodiments. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on these embodiments are included in the scope of the present invention.

[0103] 10 Generator 11 Information Collection Unit 12 Report Generation Unit 20 LLM 30 External Information DB 40 Collection Site 111 External Information Collection Unit 121 Report Definition Storage Unit 122 Selection Unit 123 External Information Analysis Unit 124 Generation Unit

Claims

1. A generation device characterized by comprising: a selection unit that selects a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a first prompt that instructs the generation of a report relating to the article and a method for analyzing the article; an analysis unit that analyzes a plurality of the articles using the analysis method of the definition document selected by the selection unit; and a generation unit that causes a large-scale language model to generate a report based on the first prompt of the definition document selected by the selection unit and the analysis results from the analysis unit.

2. The generation apparatus according to claim 1, characterized in that the selection unit extracts one or more items from the definition document, provides the extracted one or more items and the user query to a large-scale language model, and causes the large-scale language model to select a definition document that is closely related to the user query based on the similarity between the user query and the one or more items, and / or a search using common keywords.

3. The generation apparatus according to claim 1, characterized in that the definition document indicates that the analysis method is to group the articles and obtain statistical data of the grouped article group, and includes the first prompt which instructs the analysis unit to generate a report in a predetermined output format by referring to the analysis results, the analysis unit groups the plurality of articles and obtains statistical data of the grouped article group as the analysis result, and the generation unit sets the first prompt of the definition document selected by the selection unit to the large-scale language model and causes the large-scale language model to generate the report by referring to the statistical data of the article group.

4. The generation apparatus according to claim 3, wherein the definition document indicates a method for grouping the articles based on the similarity between the articles, and the analysis unit groups the plurality of articles based on the similarity between the plurality of articles.

5. The generation apparatus according to claim 3, characterized in that the definition document indicates that the frequency of keyword occurrence in the article group and / or the number of articles in the article group are calculated as statistical data, the analysis unit calculates the frequency of keyword occurrence in the article group and / or the number of articles in the article group as statistical data, and outputs the calculated statistical data in association with the article group as the analysis result.

6. The generation device according to claim 1, wherein the definition document includes a second prompt that causes the large-scale language model to generate a search query to a database storing the multiple articles based on the user query, and the analysis unit searches the database for the articles using the search query generated by setting the second prompt of the definition document selected by the selection unit to the large-scale language model.

7. A generation method executed by a generation device, comprising: a step of selecting a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a method for analyzing an article and a first prompt for instructing the generation of a report relating to the article; a step of analyzing a plurality of articles using the analysis method of the definition document selected in the selection step; and a step of causing a large-scale language model to generate a report based on the first prompt of the definition document selected in the selection step and the analysis results in the analysis step.

8. A step of selecting a definition document corresponding to a user query from among a plurality of definition documents, each containing at least a first prompt that instructs a method for analyzing an article and the generation of a report relating to the article; a step of analyzing a plurality of articles using the analysis method of the definition document selected in the selection step; a step of causing a large-scale language model to generate a report based on the first prompt of the definition document selected in the selection step and the analysis results in the analysis step; and a generation program to be executed by a computer.

Images