Server-based combined display of sensing information and 3D image information of a space

Abstract

An information processing system includes a first server that manages information related to a sensor, the information including sensing information, a second server that manages three-dimensional image information of a space in which the sensor is placed, and a terminal device including a display control unit to display a display screen including the information related to the sensor and the three-dimensional image information. The second server includes a processing unit that executes a process to associate the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information. The display control unit of the terminal device displays, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.

Description

FN202502303[DESCRIPTION][Title of Invention]INFORMATION PROCESSING SYSTEM, FIRST SERVER, SECOND SERVER, INFORMATION PROCESSING METHOD, AND RECORDING MEDIUM[Technical Field]

[0001] The present disclosure relates to an information processing system, a first server, a second server, an information processing method, and a recording medium.[Background Art]

[0002] A first server and a second server may each manage related information. In this case, there is a known technique for allowing a terminal device to display information managed by the first server and information managed by the second server.Patent Literature (PTL) 1 discloses a system for causing a communication terminal to display related information of a real estate property transmitted from a link information management system and a spherical image of the real estate property transmitted from an image management system.[Citation List][Patent Literature]

[0003] [PTL 1]lapanese Patent No. 6954410[Summary of Invention][Technical Problem]

[0004] The present disclosure provides a technique for performing a process based on information managed by a first server and information managed by a second server without adding a processing function to the first server.[Solution to Problem]

[0005] The present disclosure described herein provides an information processing system including a first server, a second server, and a terminal device communicably connected with the first server and the second server. The first server manages information related to a sensor, the information including sensing information. The second server manages three-dimensional image information of a space in which the sensor is placed. The terminal device includes a display control unit. The display control unit displays a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server. The second server includes a processing unit. The processing unit executes a process to associate the three-dimensional image information with one of the sensing information and generated information generated basedFN202502303on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information. The display control unit of the terminal device displays, on the display screen, the three-dimensional image information received from the second server and the one of the sensing information and the generated information associated with the three-dimensional image information.The present disclosure described herein provides a second server in the above-described information processing system. The second server includes the above-described processing unit and a transmission unit. The processing unit executes the process to associate the three-dimensional image information with the one of the sensing information and the generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information. The transmission unit transmits screen information to the terminal device. The screen information is for displaying the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information on the display screen.The present disclosure described herein provides a first server in the above-described information processing system. The first server includes a transmission unit. The transmission unit transmits the sensing information to the terminal device to display the sensing information on the display screen. The transmission unit furthers transmit the sensing information one of the terminal device and the second server, to display, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information. In a case where a request from the terminal device having received a request from the second server is received, the transmission unit transmits the sensing information to the terminal device. In a case where a request from the second server is received, the transmission unit transmits the sensing information to the second server.The present disclosure described herein provides an information processing method performed by a second server in a system. The system includes: a first server that manages information related to a sensor, the information including sensing information; the second server that manages three-dimensional image information of a space in which the sensor is placed; and a terminal device communicably connected with the first server and the second server to display a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server. The information processing method includes: associating the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information; and transmitting screen information to the terminal device, the screen information for displaying, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.FN202502303The present disclosure described herein provides carrier means carrying computer readable code for controlling a second server in a system to carry out an information processing method. The system includes: a first server that manages information related to a sensor, the information including sensing information; a second server that manages three-dimensional image information of a space in which the sensor is placed; and a terminal device communicably connected with the first server and the second server to display a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server. The information processing method includes: associating the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information; and transmitting screen information to the terminal device, the screen information for displaying, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.[Advantageous Effects of Invention]

[0006] According to one aspect of the present disclosure, a process based on information managed by a first server and information managed by a second server is performed without adding a processing function to the first server.[Brief Description of Drawings]

[0007] A more complete appreciation of embodiments of the present disclosure and many of the attendant advantages and features thereof can be readily obtained and understood from the following detailed description with reference to the accompanying drawings.FIG. l is a diagram illustrating an example general arrangement of an information processing system.FIG. 2 is a diagram illustrating an example hardware configuration of an image management server, a property information management server, and a terminal device.FIG. 3 is a diagram illustrating an example functional configuration of functions of the image management server, the property information management server, and the terminal device in the information processing system.FIG. 4 is an illustration of an example of an image information management table.FIG. 5 is an illustration of an example of a property information management table.FIG. 6 is an illustration of an example of a sensor information management table.FIGs. 7A and 7B are diagrams illustrating an example of screens displayed on the terminal device in a model update process and a text information generation process, respectively. FIGs. 8A and 8B are diagrams illustrating an example of screens displayed on the terminal device in the model update process and the text information generation process, respectively.FN202502303FIG. 9 is a sequence diagram illustrating an example of a process for generating screen information of a screen on which inspection information, three-dimensional image information, and sensing information are arranged, as a process based on the inspection information, the three-dimensional image information, and the sensing information.FIG. 10 is a diagram illustrating an example of a property designation screen.FIG. 11 is a diagram illustrating an example of a property management screen.FIG. 12 is a diagram illustrating an example of a property display screen.FIG. 13 is a diagram illustrating an example of a property display screen.FIG. 14 is a diagram illustrating an example of an inspection image screen.FIG. 15 is a diagram illustrating an example of an inspection image screen.FIG. 16 is a sequence diagram illustrating an example of a process for generating screen information based on inspection information, three-dimensional image information, and sensing information.FIG. 17 is a sequence diagram illustrating an example of the model update process.FIG. 18 is a diagram illustrating an example of a property display screen to be displayed on the terminal device.FIG. 19 is a sequence diagram illustrating an example of the text information generation process.FIG. 20 is a diagram illustrating an example of an inspection image screen in an inference phase.FIG. 21 is a diagram illustrating an example of text information displayed on a property display screen.FIG. 22 is a diagram illustrating an example functional configuration of functions of the image management server, the property information management server, and the terminal device in the information processing system.FIG. 23 is a sequence diagram illustrating an example of a process for generating text information and image information.FIG. 24 is a diagram illustrating an example of generated image information displayed on a property display screen.The accompanying drawings are intended to depict embodiments of the present disclosure and should not be interpreted to limit the scope thereof. The accompanying drawings are not to be considered as drawn to scale unless explicitly noted. Also, identical or similar reference numerals designate identical or similar components throughout the several views.[Description of Embodiments]

[0008] In describing embodiments illustrated in the drawings, specific terminology is employed for the sake of clarity. However, the disclosure of this specification is not intended to be limited to the specific terminology so selected and it is to be understood that each specific element includes all technical equivalents that have a similar function, operate in a similar manner, and achieve a similar result.FN202502303Referring now to the drawings, embodiments of the present disclosure are described below. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0009] Hereinafter, an information processing system, a first server, a second server, an information processing method, and a program according to an embodiment of the present disclosure will be described for illustrative purposes with reference to the drawings.Supplementary Information Related to Tacit KnowledgeIn the fields of civil engineering and architecture, the implementation of building information modeling (BIM) / construction information modeling (CIM) has been promoted for, for example, coping with the demographic shift towards an older population and enhancing labor efficiency and productivity.

[0010] BIM is a solution that involves utilizing a database of buildings, in which attribute data such as cost, finishing details, and management information is added to a three-dimensional (3D) digital model of a building. This model is created on a computer and utilized throughout every stage of the architectural process, including design, construction, and maintenance. The three-dimensional digital model is referred to as a 3D model in the following description.

[0011] CIM is a solution that has been proposed for the field of civil engineering (covering general infrastructure such as roads, electricity, gas, and water supply) following BIM, which has been advancing in the field of architecture. Similar to BIM, CIM is an approach aimed at improving the efficiency and sophistication of a series of construction production systems by sharing information through a 3D model among the parties involved.

[0012] A challenge in promoting the implementation of BIM and CIM is how to utilize the constructed BIM and CIM.

[0013] Specifically, 3D models reconstructed through BIM and CIM can be utilized for design and construction purposes and other tasks such as maintenance and site inspections. That is, in addition to use as design drawings, other uses are conceivable such as leaving records in a 3D model and sharing the 3D model with others.

[0014] Since tasks performed on the 3D model can be recorded as logs, tacit knowledge extracted based on the logs may be effectively utilized for purposes such as technology transfer from experienced to younger or less experienced technicians. This is expected to contribute to, for example, front-loading of operations and human resource development.

[0015] Focusing on the transfer of tacit knowledge, effectively transferring such tacit knowledge across different tasks and between users with varying levels of expertise becomes a challengeFN202502303not only with 3D models but also with two-dimensional (2D) data such as spherical images or planar images.

[0016] Specifically, since tacit knowledge is qualitative in nature and difficult to quantify, even if a tacit knowledge model is generated from tacit knowledge, it is challenging to secure user confidence in the tacit knowledge model. As a result, it has been difficult to promote use of such tacit knowledge models. For example, if the field of expertise of the user differs from the field of expertise of the tacit knowledge model, the tacit knowledge model has no practical value for the user, no matter how excellent the tacit knowledge model is. Similarly, if the knowledge level of the tacit knowledge model is lower than the knowledge level of the user, the tacit knowledge model lacks value for the user.

[0017] However, it is also true that tacit knowledge models can provide users with new perspectives and insights. By utilizing tacit knowledge models, even users with little experience have the potential to acquire operational expertise and technical capabilities for effective use in their tasks.

[0018] In addition, it may be desirable that a function for displaying an image such as a 3D model (three-dimensional image information) corresponding to property management information (e.g., information related to a sensor, including sensing information) be added to a system including a terminal device and a first server that stores the property management information.

[0019] One possible method for achieving this is that the first server acquires a function for displaying property management information and an image such as a 3D model. However, this method involves adding the function to the first server, which results in an increase in cost.

[0020] Accordingly, in one or more embodiments of the present disclosure, a process based on property management information managed by the first server and three-dimensional image information of a property managed by the second server is executed by the second server to allow a user's terminal device to display the three-dimensional image information and the property management information, which is acquired from the first server, in association with each other on a single screen. The property management information includes information on the sensor including the sensing information, and / or a captured image obtained by actually capturing an image of the property. Further, the second server can cause the terminal device not only to display two items of information but also to display tacit knowledge related to the property generated based on the sensing information, the captured image and the three-dimensional image information, such as text information, in association with the three-dimensional image information. Accordingly, the terminal device can display the tacitFN202502303knowledge related to the property in association with the three-dimensional image information without substantially adding a function to the first server.

[0021] TerminologyThe term "user" refers to a person who uses text information or non-text content, such as images, generated or output by a tacit knowledge model. The term "data provider" refers to a person who provides data to be used by a tacit knowledge model for training, such as audio information, character information, operation information, images, and 3D data.

[0022] Tacit knowledge (or implicit knowledge) is knowledge that is based on personal experience, intuition, and the like. The term "tacit knowledge model" refers to a model that learns tacit knowledge and outputs an answer to a question based on the learned tacit knowledge. The term "model" refers to a mechanism or artificial intelligence (Al) that learns the correspondence between input data and output data and outputs output data for input data. The output data may or may not be labeled data.

[0023] The term "property" refers to any space in which an article can be placed, such as a facility or a room in a facility. The term "article" refers to an object placed in a property, such as a sensor including sensing information. Articles to be placed vary depending on the functions of the facility.

[0024] Examples of such properties include, but are not limited to, real estate, industrial plants, construction sites, research institutions, healthcare facilities, agricultural land, storage facilities, and other infrastructure involving maintenance and management. Examples of such articles include, but are not limited to, furniture, construction materials, equipment, heavy machinery, tools, instruments, raw materials, biological cultures, and food products.

[0025] The term "information related to an article" is used to include, for example, the name of the article and the name of a property in which the article is placed, and is preferably information that can be tacitly known. Examples of such information related to an article include, but are not limited to, information known to a limited number of users. In embodiments disclosed herein, information related to an article is represented by the term "inspection information". Other examples of such information related to an article may include an instruction manual of the article, a daily report of work performed on the article, an estimate for the article, and a drawing of the article. The inspection may include, for example, investigation, testing, verification, review, and checking.

[0026] The term "three-dimensional image information of an article" refers to an image of the article displayed on a display. Three-dimensional image information of an article is a planar image obtained by two-dimensionally projecting the three-dimensional model shape of the article,FN202502303and the user can change the viewpoint of the image. Three-dimensional image information of an article may be a spherical image of the article.

[0027] The term "display screen" refers to a screen on which, for example, inspection information, information on the sensor, and three-dimensional image information of an article are displayed at a time. In embodiments disclosed herein, screens illustrated in FIGs. 12 to 15 are described as examples.

[0028] The term "generated information" refers to information generated based on three-dimensional image information and the sensing information. Alternatively, generated information may be information generated based on, for example, sensing information included in information related to an article such as a sensor. The generated information may be generated by a tacit knowledge model. In embodiments disclosed herein, generated information is represented by the term "tacit knowledge comment" or "text information".

[0029] First EmbodimentExample System ConfigurationFIG. l is a diagram illustrating an example general arrangement of an information processing system 100. The information processing system 100 includes a terminal device 10, which is an example of an input / output device, an image management server 40, and a property information management server 20. The terminal device 10 may be external to the information processing system 100 as long as the terminal device 10 can be connected to the image management server 40 or the property information management server 20 as appropriate.

[0030] The image management server 40 (an example of a second server) includes one or more information processing apparatuses that can communicate with the terminal device 10 via a communication network N. The image management server 40 manages three-dimensional image information of a property and includes a tacit knowledge model and a large language model. The image management server 40 uses the tacit knowledge model and the large language model to return text information including tacit knowledge to a user. The image management server 40 may be a web server that returns a processing result to the terminal device 10 in response to a request from the terminal device 10. The term "server" refers to a computer or software that implements a function for providing information or a processing result in response to a request from a client.

[0031] The image management server 40 may support cloud computing. Cloud computing is a mode of use that allows resources on a network to be used without identifying specific hardware resources. Cloud computing may take any of various forms including Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (laaS). Accordingly, theFN202502303image management server 40 may be housed in one or more housings or provided as one or more apparatuses. The functions of the image management server 40 may be distributed to a plurality of information processing apparatuses, or each of the plurality of information processing apparatuses may have all the functions of the image management server 40 and the information processing apparatus to be used for processing may be switched according to load balancing or the like. The image management server 40 may be a server residing in an onpremises environment.

[0032] Instead of including the tacit knowledge model and the large language model, the image management server 40 may call an application programming interface (API) published by an external system and use at least one of the tacit knowledge model and the large language model.

[0033] The property information management server 20 (an example of a first server) includes one or more information processing apparatuses that can communicate with the terminal device 10 via the communication network N. The property information management server 20 manages property management information. The property management information includes, for example, a character string such as text, information on the sensor, and a captured image captured during, for example, inspection. The property information management server 20 allows a user to update property management information as appropriate to manage, for example, the progress of construction of a property or the arrangement of articles.

[0034] The property information management server 20 may be a web server that returns a processing result to the terminal device 10 in response to a request from the terminal device 10. The property information management server 20 can communicate with the image management server 40 via the communication network N. The property information management server 20 may support either cloud computing or on-premises.

[0035] The image management server 40 and the property information management server 20 are preferably linked together in a way in which single sign-on can be achieved. The image management server 40 can communicate with the property information management server 20 via an API published by the property information management server 20. Alternatively, the image management server 40 and the property information management server 20 may cooperate with each other in performing business processes.

[0036] The terminal device 10 is a general-purpose information processing terminal used by a user of the information processing system 100. In the terminal device 10, a web browser or a native application dedicated to the image management server 40 or the property information management server 20 operates. In a case where the terminal device 10 executes a web browser, the terminal device 10 and the image management server 40 or the propertyFN202502303information management server 20 execute a web application. The web application is an application that operates in cooperation with a program written in a programming language (e.g., JavaScript®) operating on a web browser and a program on the web server (e.g., the image management server 40). In a case where the web application is executed, the processing according to the present embodiment may be performed by the image management server 40 or the property information management server 20 or may be performed by the terminal device 10 that has received the web application.

[0037] An application that is installed and executed locally on the terminal device 10 is referred to as a native application. Also in the present embodiment, the application executed on the terminal device 10 may be either a web application or a native application. In a case where the native application is executed, the processing according to the present embodiment may be performed by the image management server 40 or the terminal device 10 that executes the native application.

[0038] In one example, the terminal device 10 is a personal computer (PC), a smartphone, a personal digital assistant (PDA), or a tablet terminal. The terminal device 10 is any device on which a web browser or a native application operates. The terminal device 10 may be an electronic whiteboard, a television receiver, a glasses device, or a wearable device. A plurality of terminal devices 10 may be present.

[0039] The terminal device 10 can communicate with the image management server 40 and the property information management server 20 via the communication network N. The communication network N is implemented by, for example, the Internet, a local area network (LAN), or a provider service.The communication network N may include a wired communication network and a wireless LAN-based network or a mobile communication network such as a third generation (3G), Worldwide Interoperability for Microwave Access (WiMAX), or Long Term Evolution (LTE) network. The terminal device 10 also supports communication using short-range communication technology such as Bluetooth® or near field communication (NFC®).

[0040] In FIG. 1, the image management server 40, the property information management server 20, and the terminal device 10 communicate with one another via the communication network N. However, a user may directly operate the image management server 40 or the property information management server 20 from a console.

[0041] Example Hardware ConfigurationFIG. 2 is a diagram illustrating an example hardware configuration of the image management server 40, the property information management server 20, and the terminal device 10. The hardware elements of the image management server 40 and the property informationFN202502303management server 20 are designated by reference numerals in the 400 series. The hardware elements of the terminal device 10 are designated by reference numerals in the 100 series.

[0042] The following describes the hardware elements of the terminal device 10. Since the hardware elements of the image management server 40 and the property information management server 20 are similar to those of the terminal device 10, the description thereof will be omitted.

[0043] The terminal device 10 is implemented by a computer. As illustrated in FIG. 2, the terminal device 10 includes a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random-access memory (RAM) 103, a hard disk (HD) 104, a hard disk drive (HDD) controller 105, a display interface (I / F) 106, and a communication I / F 107.

[0044] The CPU 101 controls the overall operation of the terminal device 10. The ROM 102 stores a program used for booting the CPU 101, such as an initial program loader (IPL). The RAM 103 is used as a work area for the CPU 101.

[0045] The HD 104 stores various data such as a program. The HDD controller 105 controls reading or writing of various data from or to the HD 104 under the control of the CPU 101.

[0046] The display I / F 106 is a circuit that controls a display 106a to display an image.The display 106a is a type of display unit such as a liquid crystal display or an organic electroluminescent (EL) display that displays various types of information such as a cursor, a menu, a window, characters, or an image. The communication I / F 107 is an interface used for communication with another device.

[0047] When the terminal device 10 is a glasses device, the terminal device 10 may use a circuit that controls a member having transmissive and reflective properties, such as a lens, to display an image as an alternative to the display I / F 106.

[0048] The communication I / F 107 is, for example, a network interface card (NIC) in compliance with Transmission Control Protocol / Intemet Protocol (TCP / IP).

[0049] The terminal device 10 further includes a sensor I / F 108, an audio input / output I / F 109, an input I / F 110, a media I / F 111, and a digital versatile disc rewritable (DVD-RW) drive 112.

[0050] The sensor I / F 108 is an interface that receives detection information of various sensors 108a and 408a. The detection information is information related to a sensor, including information on the sensor including sensing information. The audio input / output I / F 109 is a circuit that processes the input of audio signals from a microphone 109b and the output of audio signalsFN202502303to a speaker 109a under the control of the CPU 101. The input I / F 110 is an interface for connecting predetermined input means to the terminal device 10.

[0051] A keyboard 110a is a type of input means including multiple keys for inputting, for example, characters, numerical values, or various instructions. A mouse 110b is a type of input means for selecting or executing various instructions, selecting a target for processing, moving a cursor being displayed, or performing an operation on a display screen.

[0052] The media I / F 111 controls reading or writing (storing) of data from or to a recording medium Illa such as flash memory. The DVD-RW drive 112 controls reading or writing of various data from or to a DVD-RW 112a, which is an example of a removable recording medium. In place of the DVD-RW 112a, a digital versatile disc recordable (DVD-R) may be used. In place of the DVD-RW drive 112, a Blu-ray drive that controls reading or writing of various data from or to a Blu-ray Disc® may be used.

[0053] The terminal device 10 further includes a bus line 113. Examples of the bus line 113 include an address bus and a data bus. The bus line 113 electrically connects the components of the terminal device 10, such as the CPU 101, to one another.

[0054] The programs described above may be stored in recording media such as an HD and a compact disc read-only memory (CD-ROM), and the recording media may be distributed domestically or internationally as program products. For example, the terminal device 10 executes a program according to an embodiment of the present disclosure to implement an information processing method according to an embodiment of the present disclosure.

[0055] FunctionsFIG. 3 is a diagram illustrating an example functional configuration of functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100.

[0056] Terminal DeviceAs illustrated in FIG. 3, the terminal device 10 includes a transmission / reception unit 11, an input reception unit 12, a display control unit 13, an audio control unit 14, a conversion unit 15, and a storing / reading unit 19. Each of these units is a function implemented by or means caused to function by any one or more of the hardware elements illustrated in FIG. 2 operating in accordance with instructions from the CPU 101 according to a program loaded onto the RAM 103 from the HD 104. The terminal device 10 further includes a storage unit 1000, which is implemented by at least one of the RAM 103 and the HD 104 illustrated in FIG. 2.

[0057] FN202502303The transmission / reception unit 11 is an example of a transmission unit or a reception unit and is implemented by instructions from the CPU 101 illustrated in FIG. 2 and by the communication I / F 107 illustrated in FIG. 2. The transmission / reception unit 11 transmits and receives various data (or information) to and from another terminal, device, apparatus, or system via the communication network N.

[0058] The input reception unit 12 is an example of input reception means and is implemented by instructions from the CPU 101 illustrated in FIG. 2 and by the input I / F 110 and the audio input / output I / F 109 illustrated in FIG. 2. The input reception unit 12 receives various inputs from the user through the microphone 109b, the keyboard 110a, and the mouse 110b.Specifically, in a state where three-dimensional image information of an article (such as a sensor) and sensing information or generated information corresponding to the three-dimensional image information are displayed on a display screen, the input reception unit 12 accepts selection of an article (such as another sensor) different from the article whose three-dimensional image information is being displayed.

[0059] The display control unit 13 is an example of display control means and output means and is implemented by instructions from the CPU 101 illustrated in FIG. 2 and by the display I / F 106 illustrated in FIG. 2. The display control unit 13 controls the display 106a, which is an example of a display unit, to display various images and screens. When the terminal device 10 is a glasses device, the display control unit 13 controls a member having transmissive and reflective properties, such as a lens, to display a virtual image as an alternative to the display I / F 106. Specifically, the display control unit 13 displays a display screen including information related to an article, which is received from the property information management server 20, and three-dimensional image information received from the image management server 40. The display control unit 13 also displays, on the display screen, the three-dimensional image information received from the image management server 40 and sensing information corresponding to the three-dimensional image information or generated information generated based on the sensing information.

[0060] For example, the display control unit 13 may display a display screen including a first display area and a second display area. The first display area is a display area that displays information related to an article, which is received from the property information management server 20. The second display area is a display area that displays the three-dimensional image information received from the image management server 40 and sensing information or generated information corresponding to the three-dimensional image information. The three-dimensional image information to be displayed on the display screen is obtained by two-dimensionally projecting a 3D model shape, and may include an image of an article with a changed viewpoint. For example, the display control unit 13 may display, on the display screen, other three-dimensional image information received from the image managementFN202502303server 40 and other sensing information or other generated information corresponding to the other three-dimensional image information. The other sensing information is sensing information included in a sensor whose selection is accepted by the input reception unit 12. The other generated information is information generated based on the other sensing information.

[0061] The audio control unit 14 is an example of audio control means and output means and is implemented by instructions from the CPU 101 illustrated in FIG. 2 and by the audio input / output I / F 109 illustrated in FIG. 2. The audio control unit 14 controls the speaker 109a, which is an example of an audio reproduction unit, to reproduce audio.

[0062] The conversion unit 15 is an example of processing means and is implemented by instructions from the CPU 101 illustrated in FIG. 2. The conversion unit 15 performs processing for converting character information into audio information or processing for converting audio information into character information.

[0063] The storing / reading unit 19 is an example of storage control means and is implemented by instructions from the CPU 101 illustrated in FIG. 2 and by the HD 104, the media I / F 111, and the DVD-RW drive 112 illustrated in FIG. 2. The storing / reading unit 19 stores various data in the storage unit 1000, the recording medium 11 la, or the DVD-RW 112a and reads various data from the storage unit 1000, the recording medium Illa, or the DVD-RW 112a.

[0064] Functional Configuration of Image Management ServerThe image management server 40 includes a transmission / reception unit 41, a screen generation unit 42, a determination unit 43, an identifying unit 44, a text information generation unit 45, an update unit 46, a processing unit 47, and a storing / reading unit 49. Each of these units is a function implemented by or means caused to function by any one or more of the hardware elements illustrated in FIG. 2 operating in accordance with instructions from the CPU 401 according to a program loaded onto the RAM 403 from the HD 404. The image management server 40 further includes a storage unit 4000, which is implemented by the HD 404 illustrated in FIG. 2. The storage unit 4000 is an example of storage means.

[0065] In FIG. 3, the single image management server 40 has all the functions described above. The image management server 40 may be configured to implement the functions in a distributed manner across multiple computers.

[0066] The transmission / reception unit 41 is an example of a transmission unit or a reception unit and is implemented by instructions from the CPU 401 illustrated in FIG. 2 and by the communication I / F 407 illustrated in FIG. 2. The transmission / reception unit 41 transmits and receives various data (or information) to and from another terminal, device, apparatus, orFN202502303system via the communication network N. Specifically, the transmission / reception unit 41 is an example of a transmission unit that transmits, to the terminal device 10, screen information for displaying three-dimensional image information and sensing information or generated information generated based on the sensing information on a display screen of the terminal device 10.

[0067] The screen generation unit 42 is an example of screen generation means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The screen generation unit 42 generates various screens. In a case where the terminal device 10 executes a web application, screen information is created by Hypertext Markup Language (HTML), Extensible Markup Language (XML), Cascading Style Sheets (CSS), JavaScript®, or the like. Thus, the screen information may be referred to as a web application. In a case where the terminal device 10 executes a client application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in the form of, for example, XML.

[0068] The determination unit 43 is an example of determination means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The determination unit 43 performs various determinations described below.

[0069] The identifying unit 44 is an example of identifying means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The identifying unit 44 identifies a target image.

[0070] The text information generation unit 45 is an example of text information generation means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The text information generation unit 45 acquires a tacit knowledge comment from a tacit knowledge model or generates text information, based on a large language model 4005.

[0071] The update unit 46 is an example of update means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The update unit 46 updates a tacit knowledge model 4004 described below. Specifically, the update unit 46 updates the tacit knowledge model 4004 by training the tacit knowledge model 4004 to learn the correspondence among three-dimensional image information, sensing information transmitted from the property information management server 20, and a tacit knowledge comment associated with the sensing information. Alternatively, the update unit 46 may update the tacit knowledge model 4004 by training the tacit knowledge model 4004 to learn the correspondence among three-dimensional image information, sensing information transmitted from the property information management server 20, a tacit knowledge comment associated with the sensing information, and input information received from the terminal device 10.

[0072] FN202502303The processing unit 47 performs processing to associate, based on sensing information transmitted from the property information management server 20 and three-dimensional image information of an article, the three-dimensional image information of the article with the sensing information or generated information generated based on the sensing information. The processing unit 47 may transmit a request for sensing information to the property information management server 20 and acquire the sensing information from the property information management server 20 as a response to the request. The processing unit 47 further performs processing to associate the three-dimensional image information with the sensing information. The process based on inspection information or the sensing information and the three-dimensional image information includes, for example, processing to display the inspection information and the three-dimensional image information or the sensing information and the three-dimensional image information on one screen, or processing to acquire a tacit knowledge comment, which is an example of text information, from the tacit knowledge model 4004 using the inspection information or the sensing information and the three-dimensional image information. Alternatively, the processing unit 47 may acquire sensing information transmitted from the property information management server 20 and a tacit knowledge comment, which is an example of comment data (text information) associated with the sensing information. For example, the processing unit 47 may acquire, as generated information, a tacit knowledge comment generated by the tacit knowledge model 4004, based on three-dimensional image information and sensing information for which a selection is accepted by the terminal device 10. The processing unit 47 may perform processing to associate, based on other sensing information of another sensor transmitted from the property information management server 20 and other three-dimensional image information of a space in which the other sensor is placed, the other three-dimensional image information with the other sensing information or other generated information generated based on the other sensing information. The processing unit 47 requests, for example, the screen generation unit 42 or the text information generation unit 45 to perform processing in accordance with the content of the processing.

[0073] The storing / reading unit 49 is an example of storage control means and is implemented by instructions from the CPU 401 illustrated in FIG. 2 and by the HD 404, the media I / F 411, and the DVD-RW drive 412 illustrated in FIG. 2. The storing / reading unit 49 stores various data in the storage unit 4000, the recording medium 411a, or the DVD-RW 412a and reads various data from the storage unit 4000, the recording medium 411a, or the DVD-RW 412a. The storage unit 4000, the recording medium 411a, and the DVD-RW 412a are examples of storage means. Specifically, the storing / reading unit 49 stores sensing information transmitted from the property information management server 20 in the storage unit 4000 in association with three-dimensional image information.

[0074] FN202502303The storage unit 4000 includes an image information management DB 4001, a model shape management DB 4002, a caption model 4003, the tacit knowledge model 4004, and the large language model 4005.

[0075] The image information management DB 4001 manages three-dimensional image information related to a property. The three-dimensional image information is information on visual representations of articles (also referred to as models) placed in the property. The model shape management DB 4002 manages three-dimensional model shape information of the articles placed in the property. The image management server 40 can generate three-dimensional image information related to the property, based on the three-dimensional model shape information. The three-dimensional model shape information is information for drawing the articles in three dimensions, such as three-dimensional point clouds or three-dimensional models of the articles. The three-dimensional model shape information may include, for example, polygons or computer-aided design (CAD) models. The image information management DB 4001 or the model shape management DB 4002 preferably stores a wide-view image such as a spherical image of the property. The image information management DB 4001 stores captured images and inspection information copied from a property information management DB 2001 of the property information management server 20.

[0076] The caption model 4003 is generated by executing a learning process using combinations of images and caption comments as training data, and causes a computer to function to output a caption comment based on an image. The caption comment is explicit knowledge and is used as a term corresponding to implicit knowledge or tacit knowledge. The caption comment is text data and is a comment describing an image among, for example, comments expressed by audio or in characters. A caption comment related to a property or an article is associated with identification information of the property or the article.

[0077] The tacit knowledge model 4004 is generated by executing a learning process using, as training data, the correspondence among three-dimensional image information, sensing information, and comments (comments other than caption comments) on the three-dimensional image information and the sensing information, and causes a computer to function to output a tacit knowledge comment based on an image. The tacit knowledge model 4004 is an example of a model trained on the correspondence among three-dimensional image information, sensing information, and a tacit knowledge comment.Alternatively, the tacit knowledge model 4004 may be an example of a model trained on the correspondence among three-dimensional image information, sensing information, a tacit knowledge comment, and input information received from the terminal device 10. The tacit knowledge model 4004 may also be trained on the correspondence among information other than comments, three-dimensional image information, and sensing information as follows:FN202502303The correspondence among three-dimensional image information, sensing information, and inspection information.The correspondence among three-dimensional image information, sensing information, comments, and inspection information.A tacit knowledge comment is text data and is a comment excluding a caption comment, that is, a comment regarding content not represented in an image, among the comments expressed by audio or in characters.

[0078] The large language model 4005 is a computer language model generated by executing a learning process using a vast amount of unlabeled text as training data. The large language model 4005 includes an artificial neural network having a large number of parameters. The large language model 4005 is sufficiently trained by a method for learning context, such as next sentence prediction or a masked language model, to capture much of the syntax and meaning of human language. The next sentence prediction understands context by determining whether sentence 1 and sentence 2 are consecutive. The masked language model understands context by masking a word in a sentence and predicting the masked word from the words before and after the masked word.

[0079] Image Information Management TableFIG. 4 is an illustration of an example of an image information management table according to the present embodiment. The storage unit 4000 stores the image information management DB 4001 including the image information management table as illustrated in FIG. 4.

[0080] In the image information management table, model identification information, position information, article information, inspection information, three-dimensional image information, and sensing information are stored in association with property identification information. The information identifying the captured images may also be stored in the image information management table. The term "property" refers to any space in which an article can be placed, such as a facility or a room in a facility. The articles to be placed vary depending on the functions of the facility.

[0081] The property identification information is identification information for uniquely identifying a property.

[0082] The model identification information is identification information for identifying an article placed in the property. The model identification information links property management information to a three-dimensional model shape in the model shape management DB 4002. The model identification information is the same as the model identification information in the property information management server 20. However, the model identificationFN202502303information may be associated with the model identification information in the property information management server 20 through a conversion table.

[0083] The position information is information indicating the position of an article in a property generated as a virtual space by using three-dimensional XYZ coordinates. The XYZ coordinates may be the coordinates of the center of gravity of the article or the coordinates of the vertices of a cube circumscribing the article.

[0084] The article information is a name of an article placed in a property. Examples of such article names include equipment, an illuminance sensor, a temperature sensor, a humidity sensor, a table, a floor, piping, and a voltage sensor. While article names assumed for use at a construction site or a building site are presented in FIG. 4, the article names vary depending on the property. While a property and articles are associated with each other in FIG. 4, articles may be registered independently irrespective of properties.

[0085] The inspection information is information (an example of comment data) related to inspection when a user inspects an article in a property. The inspection information varies depending on the article or the purpose of the inspection, and may be, for example, information indicating whether there is an abnormality in the article, the dimensions of the article, an inspection date and time, and an inspector. The inspection information is acquired from the property information management server 20. The property information management server 20 stores master inspection information, and the image management server 40 acquires the inspection information from the property information management server 20 and stores the inspection information. This is because inspection of an article is performed with respect to the property information management server 20. Thus, the inspection information in the image information management table may be present for some articles, or the inspection information in the image information management table, if any, may be inspection information that is not the latest.

[0086] The inspection information may be photographic information obtained by taking a photograph of an inspected article. In this case, the update unit 46 analyzes the photographic information, extracts features such as an article in the photograph and the state of a scratch on the article, and converts the extracted features into inspection information represented by character strings.

[0087] In addition to the inspection information, an instruction manual, a daily report, an estimate, drawings, and the like may be registered in the image information management table.

[0088] The three-dimensional image information is information on a visual representation of an article, such as three-dimensional model shape information or a spherical image of the article.FN202502303In FIG. 4, the image information management table contains the three-dimensional image information. The three-dimensional image information may be stored in the model shape management DB 4002.

[0089] The captured images are two-dimensional images acquired from the property information management server 20 together with inspection information. The property information management server 20 may store master captured images. A captured image is, for example, a planar image of an article captured by an inspector during inspection. Thus, when the image information management DB 4001 stores inspection information, a captured image is also stored in many cases. A plurality of captured images, such as the first captured image and the second captured image, are stored because an image of an article may be captured each time the article is inspected. Three or more captured images may be stored. Each of the first and second captured images has a timestamp indicating the date and time of capture.

[0090] The sensing information is included in various sensors such as an illuminance sensor, a temperature sensor, a humidity sensor, and a voltage sensor, which are examples of articles, and is an example of information related to such sensors.

[0091] Functional Configuration of Property Information Management ServerReference is made back to FIG. 3. The property information management server 20 includes a transmission / reception unit 21, a screen generation unit 22, a sensor management unit 23, and a storing / reading unit 29. Each of these units is a function implemented by or means caused to function by any one or more of the hardware elements illustrated in FIG. 2 operating in accordance with instructions from the CPU 401 according to a program loaded onto the RAM 403 from the HD 404. The property information management server 20 further includes a storage unit 2000, which is implemented by the HD 404 illustrated in FIG. 2. The storage unit 2000 is an example of storage means.

[0092] In FIG. 3, the single property information management server 20 has all the functions described above. The property information management server 20 may be configured to implement the functions in a distributed manner across multiple computers.

[0093] The transmission / reception unit 21 is an example of a transmission unit or a reception unit and is implemented by instructions from the CPU 401 illustrated in FIG. 2 and by the communication I / F 407 illustrated in FIG. 2. The transmission / reception unit 21 transmits and receives various data (or information) to and from another terminal, device, apparatus, or system via the communication network N. Specifically, the transmission / reception unit 21 is an example of a transmission unit that transmits sensing information to the terminal device 10 for display on a display screen of the terminal device 10. The transmission / reception unit 21 transmits sensing information to the terminal device 10 in response to a request from theFN202502303terminal device 10 that has received a request from the image management server 40 in order to display screen information, or transmits sensing information to the image management server 40 in response to a request from the image management server 40.

[0094] The screen generation unit 22 is an example of screen generation means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The screen generation unit 22 generates various screens. In a case where the terminal device 10 executes a web application, screen information is created by HTML, XML, CSS, JavaScript®, or the like. Thus, the screen information may be referred to as a web application. In a case where the terminal device 10 executes a client application, the screen information is stored in the terminal device 10, and the information to be displayed is transmitted in the form of, for example, XML.

[0095] The storing / reading unit 29 is an example of storage control means and is implemented by instructions from the CPU 401 illustrated in FIG. 2 and by the HD 404, the media I / F 411, and the DVD-RW drive 412 illustrated in FIG. 2. The storing / reading unit 29 stores various data in the storage unit 2000, the recording medium 411a, or the DVD-RW 412a and reads various data from the storage unit 2000, the recording medium 411a, or the DVD-RW 412a. The storage unit 2000, the recording medium 411a, and the DVD-RW 412a are examples of storage means.

[0096] Property Information Management TableFIG. 5 is an illustration of an example of a property information management table according to the present embodiment. In the storage unit 2000, the property information management DB 2001 includes the property information management table as illustrated in FIG. 5.

[0097] In the property information management table, model identification information, position information, article information, and inspection information, are stored in association with property identification information. The information identifying one or more captured images may also be stored in the property information management table. The property identification information, the model identification information, the position information, the article information, and the inspection information may be referred to as property management information. The content of the property management information may be similar to that of the image information management table. The property information management table does not include three-dimensional image information. The inspection information is recorded for all the articles that have been inspected at least once. The inspection information in the property information management server 20 is inspection information directly registered by a user, and is thus a master from which (a copy of) the inspection information in the image management server 40 is made. The captured images may be recorded in a similar manner. The one or more captured images may be captured images of a plurality of articles or may be images other than captured images, such as layout diagrams.FN202502303

[0098] Instead of including the property information management table described above, the property information management DB 2001 may include a property information management table in which position information, article information, inspection information, and one or more captured images are managed in association with model identification information rather than in association with property identification information.

[0099] FIG. 6 is an illustration of an example of a sensor information management table according to the present embodiment. The storage unit 2000 stores a sensor information management DB 2002 including the sensor information management table as illustrated in FIG. 6.

[0100] In the sensor information management table, sensing information is managed in association with article information of a sensor. The sensing information is information related to a sensor and includes, for example, timestamp information indicating a date and time.

[0101] Example of Model Update and Text Information GenerationA model update method and a text information generation method will be described with reference to FIGs. 7A, 7B, 8A, and 8B. While inspection information is not used for model update and text information generation in FIGs. 7A, 7B, 8A, and 8B, utterances described below, such as an utterance QI, may be replaced with inspection information or inspection information may be added to the utterances described below to perform learning in a similar manner.

[0102] FIGs. 7A and 7B are diagrams illustrating screens displayed on the terminal device 10 in a model update process and a text information generation process, respectively. FIG. 7A illustrates the model update process. The display control unit 13 of the terminal device 10 controls the display 106a to display a display screen 900 received from the image management server 40. The display screen 900 includes a target image 1100 and text 1200.

[0103] The input reception unit 12 of the terminal device 10 receives audio information from the microphone 109b as input information input by data providers on the displayed display screen 900. The audio information indicates utterances QI, Al, Q2, and A2 made by data providers Ml and M2. The data providers Ml and M2 preferably have a wealth of knowledge including tacit knowledge regarding the business. The tacit knowledge model 4004 is updated based on such interactions between the data providers Ml and M2, thus allowing a user to obtain useful tacit knowledge comments.

[0104] The identifying unit 44 identifies the target image 1100, which is a portion excluding the text 1200 from the display screen 900.

[0105] FN202502303Then, the determination unit 43 determines the levels of relevance between a caption comment acquired from the caption model 4003 using the target image 1100 and the utterances QI, Al, Q2, and A2.

[0106] The update unit 46 updates the tacit knowledge model 4004 using the target image 1100 or the like and, as training data, a tacit knowledge comment that is a comment determined to have a low level of relevance among the utterances QI, Al, Q2, and A2, and updates the caption model 4003 using the target image 1100 and, as training data, a caption comment that is a comment determined to have a high level of relevance among the utterances QI, Al, Q2, and A2.

[0107] Thus, the tacit knowledge model 4004 is trained on the correspondence between the target image 1100 and the utterances QI, Al, Q2, and A2. Features of the target image 1100 are extracted using several feature extraction models suitable for images, such as a convolutional neural network (CNN) model. The features represent, for example, objects that appear in an image and positions of the objects appearing in the image, or operations that are being performed in the image. Thus, the tacit knowledge model 4004 can learn the correspondence between the features of the image and the utterances QI, Al, Q2, and A2.

[0108] FIG. 7B illustrates the text information generation process. The display control unit 13 of the terminal device 10 controls the display 106a to display a display screen 900 received from the image management server 40. The display screen 900 includes an image 1110 and text 1210.

[0109] The input reception unit 12 of the terminal device 10 receives audio information via the microphone 109b as input information input by a user on the displayed display screen 900. The audio information indicates questions Qll and Q12 uttered by a user M3.

[0110] The identifying unit 44 identifies the image 1110, which does not include the text 1210, as a target image.

[0111] The text information generation unit 45 uses the image 1110 to acquire a tacit knowledge comment, based on the tacit knowledge model 4004. The tacit knowledge model 4004 extracts features from the image 1110, determines that the features of the image 1110 illustrated in FIG. 7B are similar to those of the image 1100 at the time of update illustrated in FIG. 7A, and can identify the utterances QI, Al, Q2, and A2 related to the image 1110. The utterances QI, Al, Q2, and A2 are set as tacit knowledge comments.

[0112] Further, the text information generation unit 45 uses, for example, the tacit knowledge comments (i.e., the utterances QI, Al, Q2, and A2) and the questions Qll and Q12 toFN202502303generate text information regarding answers All and A12 to the questions Qll and Q12, respectively, based on the large language model 4005.

[0113] The display control unit 13 of the terminal device 10 controls the display 106a to display text information regarding the answers All and A12 received from the image management server 40.

[0114] FIGs. 8 A and 8B are diagrams illustrating other screens displayed on the terminal device 10 in the model update process and the text information generation process, respectively, according to the present embodiment. FIGs. 8A and 8B illustrate a case in which no question sentence is used for model update and text information generation.

[0115] FIG. 8A illustrates the model update process. In an example illustrated in FIG. 8A, the tacit knowledge model 4004 is updated using audio information of one data provider and a partial image, rather than a conversation between data providers.

[0116] The display control unit 13 of the terminal device 10 controls the display 106a to display a display screen 900 received from the image management server 40. The display screen 900 includes a first image 1100 A and a second image 1100B.

[0117] The input reception unit 12 of the terminal device 10 receives character information from the keyboard 110a as input information input by a data provider on the displayed display screen 900. The character information indicates comments Cl to C4 made by a data provider M4.

[0118] The input reception unit 12 also receives operation information from the mouse 110b as input information input by the data provider M4 on the displayed display screen 900. The operation information indicates an operation performed by the data provider M4 to identify a partial image 1100B1 in the second image 1100B.

[0119] The identifying unit 44 may identify the partial image 1100B1 as the target image, or may identify the first image 1100 A or the second image 1100B as the target image.

[0120] Then, the determination unit 43 determines the levels of relevance between the caption comment acquired from the caption model 4003 using the target image and the comments Cl to C4.

[0121] The update unit 46 updates the tacit knowledge model 4004 using the partial image 1100B1 or the like and, as training data, a tacit knowledge comment that is a comment determined to have a low level of relevance among the comments Cl to C4, and updates the caption modelFN2025023034003 using the partial image 1100B1 and, as training data, a caption comment that is a comment determined to have a high level of relevance among the comments Cl to C4.

[0122] Thus, the tacit knowledge model 4004 is trained on the correspondence between the partial image 1100B1 and the comments Cl to C4. Features of the partial image 1100B1 are extracted using several feature extraction models suitable for images, such as a CNN model. The features represent, for example, objects that appear in an image and positions of the objects appearing in the image, or operations that are being performed in the image. Thus, the tacit knowledge model 4004 can learn the correspondence between the features of the image and the comments Cl to C4.

[0123] FIG. 8B illustrates the text information generation process. The display control unit 13 of the terminal device 10 controls the display 106a to display a display screen 900 received from the image management server 40. The display screen 900 includes an image 1110.

[0124] A user M5 does not perform an input on the displayed display screen 900, and the input reception unit 12 does not receive input information input by a user on the displayed display screen 900. The identifying unit 44 identifies the image 1110, which is the entire display screen 900, as the target image.

[0125] When the user M5 performs an operation to identify the partial image 1110B 1 on the display screen 900, the input reception unit 12 receives, as input information, operation information indicating the operation of identifying the partial image 1100B1, from the mouse 110b. In this case, the identifying unit 44 identifies the partial image 1100B1 on the display screen 900 as the target image in accordance with the operation information.

[0126] The text information generation unit 45 uses the partial image 1110B1 to acquire a tacit knowledge comment, based on the tacit knowledge model 4004. The tacit knowledge model 4004 determines that the features of an image 1110B1 illustrated in FIG. 8B are similar to those of the image 1100B1 at the time of update illustrated in FIG. 8 A, and can identify the comments Cl to C4 related to the image 1110B1. The tacit knowledge model 4004 extracts the comments Cl to C4 as tacit knowledge comments.

[0127] The text information generation unit 45 uses, for example, the tacit knowledge comments to generate text information regarding comments Cll to Cl 4, based on the large language model 4005. The text information generation unit 45 may generate the text information using a preset standard question when no question sentence is input, rather than using a method that does not use any questions at all.

[0128] FN202502303The display control unit 13 of the terminal device 10 controls the display 106a to display the text information regarding the comments Cl 1 to C 14 received from the image management server 40.

[0129] Operations or ProcessesNext, a method for displaying inspection information, three-dimensional image information, and sensing information on a single screen will be described as an example of a process based on the inspection information, the three-dimensional image information, and the sensing information. That is, the tacit knowledge model 4004 is not used.

[0130] FIG. 9 is a sequence diagram illustrating an example of a process for generating screen information of a screen on which inspection information, three-dimensional image information, and sensing information are arranged, as a process based on the inspection information, the three-dimensional image information, and the sensing information.

[0131] SI: A user performs a login operation on the terminal device 10. The login operation is to log in to the property information management server 20. The input reception unit 12 of the terminal device 10 receives input for the login operation. Any existing login method may be used. The following description is given on the assumption that the login is successful.

[0132] The user logs in to the property information management server 20 and then logs in to the image management server 40. Alternatively, the user may log in to the image management server 40 first and then log in to the property information management server 20.

[0133] S2: In response to a successful login, the transmission / reception unit 11 of the terminal device 10 transmits a request for a property designation screen 200 to the property information management server 20.

[0134] S3: The transmission / reception unit 21 of the property information management server 20 receives the request for the property designation screen 200. The screen generation unit 22 generates the property designation screen 200, and the transmission / reception unit 21 transmits screen information of the property designation screen 200 to the terminal device 10.

[0135] S4: The transmission / reception unit 11 of the terminal device 10 receives the screen information of the property designation screen 200. The display control unit 13 displays the property designation screen 200 (see FIG. 10). The user enters property identification information (e.g., V0001) on the displayed property designation screen 200. The input reception unit 12 of the terminal device 10 receives the property identification information.

[0136] FN202502303S5: The transmission / reception unit 11 of the terminal device 10 designates the property identification information and transmits a request for property management information to the property information management server 20.

[0137] S6: The transmission / reception unit 21 of the property information management server 20 receives the request for property management information, and the storing / reading unit 29 searches the property information management DB 2001 using the property identification information. The property management information is information related to a property associated with the property identification information. The screen generation unit 22 of the property information management server 20 generates a property management screen 210 for displaying the property management information, and the transmission / reception unit 21 transmits screen information of the property management screen 210 to the terminal device 10.

[0138] In response to the request for property management information, the transmission / reception unit 21 also transmits an image request program to the terminal device 10 so that the terminal device 10 can acquire three-dimensional image information. The image request program is, for example, a web application. The web application is installed in the property information management server 20 by the operator of the image management server 40 obtaining the permission of the operator of the property information management server 20. Alternatively, a uniform resource locator (URL) at which the image request program is available may be transmitted to the terminal device 10. The web application, which is configured to acquire three-dimensional image information from the image management server 40, has a function of connecting to the image management server 40 and requesting or displaying the three-dimensional image information.

[0139] S7: The transmission / reception unit 11 of the terminal device 10 receives the screen information of the property management screen 210 and the image request program. The display control unit 13 displays the property management screen 210 (see FIG. 11). As a result, the property management information is displayed. The user performs an operation (such as pressing an image acquisition button 213) on the displayed property management information to request three-dimensional image information of the property. The input reception unit 12 of the terminal device 10 accepts an operation for requesting the three-dimensional image information of the property. The three-dimensional image information of the property is three-dimensional image information of articles placed in the property, which is generated as a virtual space. The articles are represented by 3D model shape information. Since the property has been identified, a request for the three-dimensional image information of the property may be transmitted to the image management server 40 without an operation by the user.

[0140] FN202502303The property management screen 210 includes a first display area 214 and a second display area 215. The first display area 214 displays information related to an article, which is acquired from the property information management server 20. The second display area 215 displays three-dimensional image information of the article, which is acquired from the image management server 40. In step S7, the property management information is displayed in the first display area 214, whereas no information is displayed in the second display area 215.

[0141] S8: If the user has not logged in to the image management server 40, the user performs a login operation on the terminal device 10. The login operation is to log in to the image management server 40. The input reception unit 12 of the terminal device 10 receives input for the login operation. Any existing login method may be used. The following description is given on the assumption that the login is successful. The login operation by the user may be omitted using, for example, single sign-on.

[0142] S9: The terminal device 10 executes the image request program to request three-dimensional image information. Accordingly, the transmission / reception unit 11 designates the property identification information of the property selected by the user and transmits a request for the three-dimensional image information of the property to the image management server 40. Preferably, the transmission / reception unit 11 transmits the URL of the property information management server 20 to the image management server 40 so that the terminal device 10 can be redirected to the property information management server 20. The three-dimensional image information of the property is an image of articles placed in the property defined as a virtual space. Since the articles are represented by 3D model shape information, the terminal device 10 projects three-dimensional model shapes of the articles into two dimensions to generate a planar image. The user can view any article while changing the viewpoint. The transmission / reception unit 11 may transmit the property management information acquired from the property information management server 20 to the image management server 40. For example, the image request program receives the property management information as a URL parameter from a web application connected to the property information management server 20.

[0143] S10: The transmission / reception unit 41 of the image management server 40 receives the request for the three-dimensional image information of the property. The storing / reading unit 49 searches the image information management DB 4001 using the property identification information and acquires three-dimensional image information of each article. The processing unit 47 requests the screen generation unit 42 to generate a screen including the three-dimensional image information of the property and information related to the articles. The screen generation unit 42 generates a screen corresponding to the second display area 215 in which the three-dimensional image information of each article and sensing information or inspection information are arranged on a single screen. The screen generation unit 42 furtherFN202502303generates a screen corresponding to the first display area 214 in which the information related to the articles is arranged. The transmission / reception unit 41 transmits screen information of the screen corresponding to the first display area 214 and the screen corresponding to the second display area 215 to the terminal device 10. The three-dimensional image information of each article, which is included in the screen information, is three-dimensional image information in which all the articles included in the property are placed in the property, and the user can change the viewpoint as desired.

[0144] SI 1 : The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the first display area 214 and the screen corresponding to the second display area 215, and the display control unit 13 displays a property display screen 220 including the first display area 214 and the second display area 215 (see FIG. 12). In step S 11, the three-dimensional image information of each article is displayed in the second display area 215, and, for example, a list of articles placed in the property is displayed in the first display area 214. The list of articles is generated from the property management information. Thus, the list of articles and the three-dimensional image information of the property are displayed on a single screen. The list of articles is included in the property management information. Subsequently, the user identifies any article from the three-dimensional image information of the property. When the user identifies an article, the user can request inspection information and sensing information of the article. The input reception unit 12 of the terminal device 10 accepts the operation of identifying the article. The article may be identified by, for example, the coordinates of a position clicked by the user, or model identification information may be identified using the coordinates.

[0145] S12: When the user presses an information display button 225, the transmission / reception unit 11 of the terminal device 10 transmits information for identifying the article to the image management server 40.

[0146] S13: The transmission / reception unit 41 of the image management server 40 receives the information for identifying the article. The storing / reading unit 49 identifies, from the image information management DB 4001, the model identification information of an article having the closest position information, based on the information for identifying the article (here, position information). The transmission / reception unit 41 transmits the model identification information to the terminal device 10. When the information for identifying the article transmitted in step S12 is model identification information, the image information management DB 4001 is not searched. The image management server 40 transmits the model identification information in order to acquire the latest inspection information from the property information management server 20.

[0147] FN202502303S 14: The transmission / reception unit 11 of the terminal device 10 receives the model identification information. For example, the image management server 40 notifies the terminal device 10 of the URL of the property information management server 20 and redirects the terminal device 10 to the property information management server 20. As a result, the transmission / reception unit 11 of the terminal device 10 designates the model identification information and transmits a request for inspection information and sensing information of the article to the property information management server 20.

[0148] S 15 : The transmission / reception unit 21 of the property information management server 20 receives the request for inspection information and sensing information of the article, and the storing / reading unit 29 searches the property information management DB 2001 and the sensor information management DB 2002 using the model identification information to acquire the inspection information and the sensing information. The transmission / reception unit 21 transmits the inspection information and the sensing information to the terminal device 10. The transmission / reception unit 21 may additionally transmit the captured image.

[0149] SI 6: In response to receiving the inspection information and the sensing information, the transmission / reception unit 11 of the terminal device 10 designates the model identification information and transmits the inspection information and the sensing information to the image management server 40. In response to the transmission / reception unit 41 of the image management server 40 receiving the inspection information and the sensing information as a response to the request in step SI 3, the storing / reading unit 49 identifies the article using the model identification information and stores the latest inspection information and the latest sensing information in the image information management DB 4001 (or by overwriting existing ones). That is, inspection information and sensing information that do not yet exist in the image information management DB 4001 are stored.

[0150] SI 7: Subsequently, the processing unit 47 requests the screen generation unit 42 to generate a screen that displays the three-dimensional image information of the article corresponding to the model identification information, the latest inspection information, and the latest sensing information in association with one another. The screen generation unit 42 performs a process based on information related to the article and the three-dimensional image information of the article to generate a screen corresponding to the second display area 215 that displays inspection information, the three-dimensional image information of the article, and sensing information in association with one another. The screen generation unit 42 may perform an update process for adding the inspection information to the screen corresponding to the second display area 215. The screen generation unit 42 further generates a screen that displays article information in the first display area 214. The transmission / reception unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10.FN202502303

[0151] SI 8: The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays an inspection image screen 230 including the first display area 214 and the second display area 215 (see FIG. 14). In step S18, an article list (article information) is displayed in the first display area 214, and the three-dimensional image information of the article, inspection information corresponding to the three-dimensional image information, sensing information, and generated information are displayed in the second display area 215.

[0152] Example ScreensFIG. 10 illustrates an example of the property designation screen 200 for inputting property identification information. The property designation screen 200 includes a property identification information input field 201 and a search button 202. In response to the user entering property identification information in the property identification information input field 201 and pressing the search button 202, a room number list is displayed on the property management screen 210, as illustrated in FIG. 11.

[0153] FIG. 11 illustrates an example of the property management screen 210. The property management screen 210 includes a first display area 214 and a second display area 215. The first display area 214 displays information related to an article, which is acquired from the property information management server 20. The second display area 215 displays three-dimensional image information of the article, which is acquired from the image management server 40. The first display area 214 is an area other than the second display area 215. The first display area 214 includes a room number list 211. The room number list 211 is a list of room numbers in the property identified by the property identification information. The room number list 211 corresponds to the property management information and includes, for example, articles included in each room of the property.Depending on the property, the display of room numbers may be omitted, and the property designation screen 200 illustrated in FIG. 10 may transition to a screen illustrated in FIG. 12 to display the three-dimensional image information of the property. The user uses a mouse cursor 212 to select a room number for which the three-dimensional image information is to be displayed. In response to the user pressing the image acquisition button 213, the property display screen 220 is displayed.

[0154] While the second display area 215 is an area other than the first display area 214, display may be implemented by a program on a web application such as an ifirame.

[0155] FIGs. 12 and 13 are diagrams illustrating an example of the property display screen 220. In response to an article confirmation button being pressed on the property display screen 220,FN202502303the first display area 214 and the second display area 215 are displayed. The first display area 214 of the property display screen 220 displays an article list 221.Articles in the room are a table, a cylindrical object, a door, and a storage box.

[0156] The second display area 215 of the property display screen 220 displays three-dimensional image information 222 of these articles.

[0157] Subsequently, the user can select an article for which inspection information is to be displayed from the three-dimensional image information 222 using the mouse cursor 212. As a result, coordinates of the article are determined as information for identifying the article.Alternatively, the user may identify an article from the article list 221. In this case, model identification information is identified at the time of selection. In response to the user pressing the information display button 225, an inspection image screen 230 is displayed (see, for example, FIG. 14). As described below, the information display button 225 is used to display text information generated based on a tacit knowledge comment, in addition to displaying inspection information.In response to the user pressing an information update button 226, the tacit knowledge model 4004 is updated.

[0158] In FIG. 12, a size (floor area) 224 is displayed as information related to the property.The size (floor area) 224 may be a measured value or may be included in the property management information.

[0159] As illustrated in FIG. 12, the terminal device 10 can display the property management information, which is managed by the property information management server 20, and the three-dimensional image information 222 of the property, which is managed by the image management server 40, on a single screen. The user can check the three-dimensional image information 222 of the property while viewing the property management information.

[0160] As illustrated in FIG. 13, in response to a sensor confirmation button being pressed on the property display screen 220, information related to sensors is displayed as the article list 221 in the first display area 214 of the property display screen 220. Sensors in the room include a temperature sensor, an illuminance sensor, and a humidity sensor. The second display area 215 of the property display screen 220 displays three-dimensional image information 222 of these sensors.

[0161] FIG. 14 is a diagram illustrating an example of the inspection image screen 230. The inspection image screen 230 includes the first display area 214 and the second display area 215. In FIG. 14, three-dimensional image information of a sensor (e.g., an illuminance sensor Sei) selected by the user as an example of an article and sensing information (e.g., sensingFN202502303information of the illuminance sensor Sei) are displayed in the second display area 215. The second display area 215 also displays inspection information 232 associated with the sensor selected by the user. As an example, the inspection information 232 states, "The illuminance abnormally increased, but there was no sensor abnormality".

[0162] As described above, the terminal device 10 can display, on a single screen, three-dimensional image information of articles such as various sensors, the inspection information 232 of an article for which the three-dimensional image information is displayed, and sensing information of the article.

[0163] When the user moves the mouse cursor 212 to another sensor (e.g., the temperature sensor) in the article list 221 illustrated in FIG. 14 and clicks the other sensor, a notification indicating that the other sensor has been clicked is transmitted to the image management server 40. Then, the screen generation unit 42 generates a screen corresponding to the second display area 215 that displays inspection information of the other sensor and three-dimensional image information and sensing information of the article (i.e., the other sensor) in association with one another. The transmission / reception unit 41 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10.

[0164] FIG. 15 is a diagram illustrating an example of switching of three-dimensional image information and inspection information of an article to be displayed in the second display area 215. In FIG. 15, an article that is the temperature sensor is selected. Accordingly, three-dimensional image information of the temperature sensor, inspection information 228 and sensing information of the temperature sensor are displayed in the second display area 215.

[0165] As in FIG. 14, in a case where three-dimensional image information of a first article (e.g., the illuminance sensor Sei) is displayed, information related to the first article (the inspection information 232 and the sensing information) associated with the three-dimensional image information of the first article is displayed. When in this state, the display is changed from the three-dimensional image information of the first article to three-dimensional image information of a second article (e.g., a temperature sensor Se2), the display is changed to information related to the second article (the inspection information 228 and the sensing information) associated with the three-dimensional image information of the second article.

[0166] Example in which Image Management Server Acquires Inspection Information from Property Information Management ServerIn FIG. 9, the image management server 40 acquires, from the terminal device 10, the inspection information acquired by the terminal device 10 from the property information management server 20. However, the image management server 40 may acquire the inspection information directly from the property information management server 20.FN202502303

[0167] FIG. 16 is a sequence diagram illustrating an example of a process for generating screen information based on inspection information and three-dimensional image information. In FIG. 16, differences from FIG. 9 may be described. The processing of steps SI to S 12 may be similar to that in FIG. 9.

[0168] S21 : The transmission / reception unit 41 of the image management server 40 receives the information for identifying the article. The storing / reading unit 49 identifies, from the image information management DB 4001, the model identification information of an article having the closest position information, based on the information for identifying the article (here, position information). The processing unit 47 requests the transmission / reception unit 41 to make a request for inspection information and sensing information. The transmission / reception unit 41 calls an API of the property information management server 20, thereby designating the model identification information and transmitting the request for inspection information and sensing information to the property information management server 20. When the information for identifying the article is model identification information, the image information management DB 4001 is not searched.

[0169] S22: The transmission / reception unit 21 of the property information management server 20 receives the request for inspection information and sensing information. The storing / reading unit 29 searches the property information management DB 2001 and the sensor information management DB 2002 using the model identification information and acquires the inspection information and the sensing information. The transmission / reception unit 21 transmits the inspection information and the sensing information to the image management server 40. The transmission / reception unit 41 acquires the inspection information and the sensing information from the property information management server 20 as a response to the request in step S21.

[0170] The subsequent processing S23 and S24 may be similar to that of S17 and S18 in FIG. 9. The process as illustrated in FIG. 16 can reduce the number of operations performed by the terminal device 10 to change a connection destination, thereby shortening the time until the inspection image screen 230 is displayed.

[0171] In the present embodiment, the image management server 40 performs a process based on inspection information and three-dimensional image information. Thus, the terminal device 10 can display, in the second display area 215 of the inspection image screen 230, three-dimensional image information and inspection information and sensing information corresponding to the three-dimensional image information. The image management server 40 can perform a process based on inspection information and sensing information managed by the property information management server 20 and three-dimensional image informationFN202502303managed by the image management server 40, without adding a processing function to the property information management server 20.

[0172] The property information management server 20 may perform part of a process based on inspection information and sensing information managed by the property information management server 20 and three-dimensional image information managed by the image management server 40. Even in this case, the processing load on the property information management server 20 is reduced as compared with a case where the property information management server 20 performs the entire process based on inspection information and sensing information managed by the property information management server 20 and three-dimensional image information managed by the image management server 40.

[0173] The three-dimensional image information and the inspection information corresponding to the three-dimensional image information may be displayed so as not to overlap each other or may be displayed so as to overlap each other.The first display area 214 and the second display area 215 may be displayed so as not to overlap each other or may be displayed so as to overlap each other.Furthermore, the first display area 214 and the second display area 215 may be each divided into a plurality of sub-areas and displayed, and each of the plurality of sub-areas may be displayed in a mixed manner.

[0174] Second EmbodimentA second embodiment of the present disclosure describes an image management server 40 that acquires a tacit knowledge comment from a tacit knowledge model using inspection information, three-dimensional image information, and sensing information and generates text information based on the tacit knowledge comment.

[0175] In the present embodiment, reference is also made to the hardware configuration diagram of FIG. 2 and the functional block diagram of FIG. 3, which have been described in the first embodiment.

[0176] Operations or ProcessesLearning Phase (Model Update)First, a model update process in which the tacit knowledge model 4004 is trained on data will be described with reference to FIG. 17. FIG. 17 is a sequence diagram illustrating an example of the model update process. In the description of FIG. 17, differences from FIG. 9 may be described. The processing of steps S31 to S40 may be similar to that of steps SI to S 10 in FIG. 9.

[0177] FN202502303S41 : The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215, and the display control unit 13 displays the property display screen 220 including the first display area 214 and the second display area 215 (see FIG. 12). For example, the article list 221 indicating articles placed in the property and the three-dimensional image information 222 of the property are displayed. Subsequently, the user identifies any article from the three-dimensional image information 222. When the user identifies an article, the user can request inspection information and sensing information of the article. The input reception unit 12 of the terminal device 10 accepts the operation of identifying the article. The article may be identified by, for example, the coordinates of a position clicked by the user, or model identification information may be identified using the coordinates. The user inputs comments related to the article, such as the comments (character information or audio) described with reference to FIGs. 7A, 7B, 8 A, and 8B, to the terminal device 10. The comments may be referred to as input information. The comments may be tacit knowledge comments. The comments may include a caption comment describing the article.

[0178] S42: The transmission / reception unit 11 of the terminal device 10 transmits the information for identifying the article and the input information to the image management server 40.

[0179] The subsequent processing of steps S43 to S46 may be similar to that of steps S 13 to S 16 in FIG. 9.

[0180] S47: The transmission / reception unit 41 of the image management server 40 receives the inspection information and the sensing information from the terminal device 10. The storing / reading unit 49 identifies the article using the model identification information and stores the latest inspection information and the latest sensing information in the image information management DB 4001. The determination unit 43 acquires a caption comment identified by the model identification information from the caption model 4003, and determines a level of relevance between the caption comment and a comment included in the input information received in step S42. In one example, the determination unit 43 may determine the level of relevance of the entire comment included in the input information received in step S42 to the acquired caption comment. In another example, the determination unit 43 may divide the comment included in the input information received in step S42 into multiple portions and determine a level of relevance of each of the divided portions of the comment to the acquired caption comment.

[0181] S48: The update unit 46 updates the caption model 4003 by associating a comment determined to have a high level of relevance in step S47, as a caption comment, with the model identification information. The update unit 46 further updates the tacit knowledge model 4004 using, as training data, a comment determined to have a low level of relevance inFN202502303step S47, the inspection information (received in step S46), the three-dimensional image information (identified in step S42) of an article related to the comment and the inspection information, and the sensing information (received in step S46) of the article. That is, the correspondence among three-dimensional image information and sensing information of an article, a comment, and inspection information is learned. Features of the three-dimensional image information and sensing information of the article are extracted using several feature extraction models suitable for images, such as a CNN model. The features represent, for example, objects that appear in an image and positions of the objects appearing in the image, or operations that are being performed in the image. Accordingly, the tacit knowledge model 4004 can learn the correspondence among three-dimensional image information and sensing information of an article, a comment, and inspection information.

[0182] Both a comment and inspection information are not to be used, and at least one of them can be used to update the tacit knowledge model 4004.

[0183] In FIG. 17, the image management server 40 acquires inspection information and sensing information from the terminal device 10. Alternatively, as in FIG. 16, the image management server 40 may acquire inspection information and sensing information from the property information management server 20.

[0184] Example Screens in Learning PhaseScreens to be displayed on the terminal device 10 in the learning phase are substantially the same as those in FIGs. 10 to 15, except that the user can enter input information on the property display screen 220 illustrated in FIG. 12. Information corresponding to the property display screen 220 illustrated in FIG. 12 that additionally includes the inspection information is displayed in an inference phase described below.

[0185] FIG. 18 is a diagram illustrating an example of the property display screen 220 to be displayed on the terminal device 10. The property display screen 220 illustrated in FIG. 18 includes the first display area 214 and the second display area 215, and the size (floor area) 224 is displayed in the first display area 214 as information related to the property.

[0186] The second display area 215 displays three-dimensional image information of the illuminance sensor Sei, the temperature sensor Se2, and a humidity sensor Se3 and input information 241 that is input in association with the three-dimensional image information. The input information 241 states, "Since an abnormality in illuminance and an abnormality in temperature have occurred almost simultaneously, the abnormalities can be derived from a common cause". The image management server 40 can update the tacit knowledge model 4004 using the input information 241, inspection information, and the three-dimensionalFN202502303image information. The size (floor area) 224 serving as the information related to the property may be a caption comment.

[0187] Inference Phase (Text Information Generation)Next, a text information generation process using the tacit knowledge model 4004 will be described with reference to FIG. 19. FIG. 19 is a sequence diagram illustrating an example of the text information generation process. In the description of FIG. 19, differences from FIG. 9 may be described. The processing of steps S31 to S46 may be similar to that of steps SI to S16 in FIG. 9. Note that, in step S41, the user enters a question sentence regarding an article (see FIG. 20).

[0188] S51: The transmission / reception unit 41 of the image management server 40 receives the inspection information. The storing / reading unit 49 identifies the article using the model identification information and stores the latest inspection information in the image information management DB 4001. The processing unit 47 requests the text information generation unit 45 to generate text information. The text information generation unit 45 acquires a tacit knowledge comment corresponding to the three-dimensional image information and sensing information of the article from the tacit knowledge model 4004. The tacit knowledge model 4004 extracts features of the three-dimensional image information of the article and can identify at least one of a comment and inspection information corresponding to the features and the sensing information. The tacit knowledge model 4004 extracts at least one of the comment and the inspection information as a tacit knowledge comment.

[0189] S52: Subsequently, the text information generation unit 45 acquires text information created by the large language model 4005 using the tacit knowledge comment, the input information (question sentence), and the inspection information. The large language model 4005 can generate more detailed text information using the tacit knowledge comment, the input information (question sentence), and the inspection information. The text information generation unit 45 may convert audio information included in the input information into character information. The text information generated by the text information generation unit 45 may be either audio information or character information.

[0190] The text information generation unit 45 may generate text information without using any inspection information or question sentence. Alternatively, the text information generation unit 45 may generate a fixed question within the information processing system 100 in advance and use the fixed question. In this case, the question sentence is invisible to the user. Alternatively, the text information generation unit 45 may generate fixed questions within the information processing system 100 in advance, which are then displayed on the display unit to prompt the user to select any of the fixed questions, and use the selected question.FN202502303

[0191] As described above, inspection information is optional. However, using inspection information to generate text information from the large language model 4005 provides more detailed information related to an article. For example, when inspection information includes the severity of a scratch on an article, text information including appropriate measures to be taken in accordance with the severity of the scratch can be generated.

[0192] S53: The processing unit 47 requests the screen generation unit 42 to generate a screen that displays the three-dimensional image information of the article corresponding to the model identification information, and the text information in association with each other. The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the article and text information corresponding to the three-dimensional image information and the sensing information of the article. The screen generation unit 42 may perform an update process for adding only the text information to the screen corresponding to the second display area 215. The transmission / reception unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0193] S54: The display control unit 13 of the terminal device 10 displays the property display screen 220 (see FIG. 21) including the first display area 214 and the second display area 215 that includes the text information. Alternatively, the conversion unit 15 converts the received text information into audio information, and the audio control unit 14 controls the speaker 109a to reproduce the converted text information. When the received text information is audio information, the text information is reproduced by the speaker 109a, or the conversion unit 15 converts the received text information into character information and the display 106a displays the converted text information.

[0194] In FIG. 19, the image management server 40 acquires inspection information from the terminal device 10. Alternatively, as in FIG. 16, the image management server 40 may acquire inspection information from the property information management server 20.

[0195] Example Screens in Inference PhaseScreens to be displayed on the terminal device 10 in the inference phase are substantially the same as those in FIGs. 10 to 12, except that the user can enter input information (a question sentence) on the property display screen 220 illustrated in FIG. 12.

[0196] FN202502303FIG. 20 is a diagram illustrating an example of the inspection image screen 230 in the inference phase. The inspection image screen 230 includes the first display area 214 and the second display area 215. FIG. 20 illustrates substantially the same configuration as that illustrated in FIG. 14, except that the user has entered a question sentence as input information (question sentence) 234. The second display area 215 displays three-dimensional image information of the illuminance sensor Sei, sensing information, and the input information (question sentence) 234 corresponding to the three-dimensional image information of the illuminance sensor Sei. For example, the input information (question sentence) 234 illustrated in FIG. 20 states, "What could be causing the abnormal illuminance value?" The user presses the information display button 225 to make a request to generate text information using the tacit knowledge model 4004, together with the input information (question sentence) 234.

[0197] FIG. 21 is a diagram illustrating an example of text information displayed on the property display screen 220. The property display screen 220 includes the first display area 214 and the second display area 215. The second display area 215 displays the three-dimensional image information 222 of the illuminance sensor Sei, the temperature sensor Se2, and the humidity sensor Se3 and text information 235 corresponding to the three-dimensional image information 222 of the various sensors, namely, the illuminance sensor Sei, the temperature sensor Se2, and the humidity sensor Se3. The text information 235 states, "When an abnormal illuminance value and an abnormal temperature value have occurred simultaneously, a possible cause is a window close to two sensors". The text information 235 is generated by the large language model 4005 from the three-dimensional image information of the articles (the various sensors), the inspection information, and the input information (question sentence) 234. For example, when the presence of a scratch on an article is detected from three-dimensional image information of articles including various sensors, a tacit knowledge comment related to the scratch on the article is extracted. The tacit knowledge comment, a question sentence related to the scratch, and inspection information that specifies the current state of the scratch are input to the large language model 4005, and thus appropriate text information corresponding to the current state of the scratch can be generated.

[0198] The text information 235 is not identical to the inspection information but includes at least one of text information generated by the tacit knowledge model 4004 that has learned the inspection information and text information generated by the large language model 4005 based on the inspection information. The text information 235 can be regarded as a processing result of a process based on the inspection information and the three-dimensional image information.

[0199] Third EmbodimentFN202502303A third embodiment of the present disclosure describes an image management server 40 that generates an image from sensing information and text information.

[0200] FIG. 22 is a diagram illustrating an example functional configuration of functions of the image management server 40, the property information management server 20, and the terminal device 10 in the information processing system 100. In the description of FIG. 22, differences from FIG. 3 will be described.

[0201] The image management server 40 illustrated in FIG. 22 further includes an image generation unit 48, and the storage unit 4000 of the image management server 40 further includes an image generation model 4006. The other elements may be the same as those in FIG. 3.

[0202] The image generation unit 48 is an example of image generation means and is implemented by instructions from the CPU 401 illustrated in FIG. 2. The image generation unit 48 inputs text data to the image generation model 4006 or inputs text data and an image to the image generation model 4006 to generate image information.

[0203] The image generation model 4006 is a machine learning model (generative Al) that generates an image from text data or from text data and an image. The image generation model 4006 is trained using, for example, training data including text data and images. The training data includes, for example, text data or text data and an image for learning as input, and an image as ground truth for output. For example, the image generation model 4006 may be trained such that an image generated by the image generation model 4006 that has received text data or text data and an image included in training data as input becomes close to an image as ground truth included in the training data.

[0204] Learning PhaseA process in the learning phase may be similar to that illustrated in FIG. 17. In step S48, the update unit 46 updates the tacit knowledge model 4004 so as to train the tacit knowledge model 4004 to learn the correspondence between inputs representing a comment determined to have a low level of relevance in step S47 and inspection information and outputs representing the three-dimensional image information of the article. Alternatively, the update unit 46 updates the tacit knowledge model 4004 so as to train the tacit knowledge model 4004 to learn the correspondence between inputs representing the comment, the inspection information, and the three-dimensional image information of the article and an output representing the three-dimensional image information.

[0205] Inference Phase (Text Information Generation)FN202502303FIG. 23 is a sequence diagram illustrating an example of a process for generating text information and image information. In the description of FIG. 23, differences from FIG. 19 may be described. FIG. 23 further illustrates step S52-1.

[0206] S52-1 : The image generation unit 48 inputs the sensing information and the text information created by the large language model 4005 to the image generation model 4006 to generate image information. The image generation unit 48 may use the text information created by the large language model 4005, without using the captured image, to acquire the image information created by the image generation model 4006.

[0207] The storing / reading unit 49 stores the text information created by the large language model 4005 and the image information created by the image generation model 4006 in the image information management DB 4001 (or overwrites the information stored in the image information management DB 4001 with the text information created by the large language model 4005 and the image information created by the image generation model 4006) in association with the inspection information stored in the image information management DB 4001 in step S46.

[0208] S53: The processing unit 47 requests the screen generation unit 42 to generate a screen that displays the three-dimensional image information of the article corresponding to the model identification information, the generated image information, and the text information in association with one another.The screen generation unit 42 generates a screen corresponding to the second display area 215 that displays the three-dimensional image information of the article, the generated image information, and text information corresponding to the three-dimensional image information of the article.The transmission / reception unit 41 of the image management server 40 transmits screen information of the screen corresponding to the second display area 215 to the terminal device 10. The transmission / reception unit 11 of the terminal device 10 receives the screen information of the screen corresponding to the second display area 215 transmitted from the image management server 40.

[0209] Example Screen in Inference PhaseFIG. 24 is a diagram illustrating an example of generated image information displayed on the property display screen 220. In the description of FIG. 24, differences from FIG. 21 will be described.

[0210] The property display screen 220 illustrated in FIG. 24 displays a generated image. The generated image is a generated image 2401 generated by the image generation model 4006 based on the text information 235. For example, the generated image 2401 is an image of aFN202502303marker indicating the position of a door. Thus, the generated image 2401 includes the marker indicating the position of the door.

[0211] An effect of generating text information using inspection information as in the present embodiment will be described.

[0212] 1. Comparative Example 1 (Case of Using General Large Language Model)Question sentence: The user asks a question, "How should I repair a crack?"Tacit knowledge comment: You can use tape or filler to repair it.

[0213] 2. Comparative Example 2 (Case of Learning from Three-Dimensional Image Information) Learning PhaseInput image: Three-dimensional image informationComment: Please use tape for a large width crack and filler for a small width crack.Inference PhaseInput image: Three-dimensional image informationQuestion sentence: "How should I repair a crack?"Tacit knowledge comment: There are a large width crack and a small width crack, so the use of tape is recommended for the large width crack and the use of filler is recommended for the small width crack.

[0214] 3. Present Embodiment (Three-Dimensional Image Information)Learning PhaseInput image: Three-dimensional image informationInspection information: Crack in tape at comerInference PhaseInput image: Three-dimensional image informationQuestion sentence: "How should I repair a crack?"Tacit knowledge comment: There are a large width crack and a small width crack, so the use of tape is recommended for the large width crack and the use of filler is recommended for the small width crack. Please be careful when applying tape to the comer, as a crack may occur. That is, an effect obtained by learning from the inspection information is the comment, stating "Please be careful when applying tape to the corner, as a crack may occur".

[0215] 4. Present Embodiment (Three-dimensional Image Information and Input Information) Learning PhaseInput image: Three-dimensional image informationInput information: Please use tape for a large width crack and filler for a small width crack. Inspection information: Crack in tape at comerInference PhaseFN202502303Input image: Three-dimensional image informationQuestion sentence: "How should I repair a crack?"Tacit knowledge comment: There are a large width crack and a small width crack, so the use of tape is recommended for the large width crack and the use of filler is recommended for the small width crack. Please be careful when applying tape to the comer, as a crack may occur. That is, an effect obtained by learning from the inspection information is the comment, stating "Please be careful when applying tape to the corner, as a crack may occur".

[0216] Multimodal ModelsSeveral examples of combinations of input information and tacit knowledge comments will be described. The model described above is assumed to be a large language model. However, the present embodiment may use a multimodal model that receives a plurality of data formats (e.g., image, text, and gesture) as input and outputs a predetermined data format.Case where the input information is a character string and content other than text information is generated as a tacit knowledge commentA character string is input, and an image is generated. A character string is input, and a moving image is generated. A character string is input, and audio is generated. A character string is input, and a 3D model is generated.Case where the input information includes a character string and non-character string information and text information is generated as a tacit knowledge commentAn image and a character string are input, and text information is generated. A 3D model and a character string are input, and text information is generated. Audio and a character string are input, and text information is generated.Case where the input information includes a character string and non-character string information and content other than text information is generated as a tacit knowledge commentAn image and a character string are input, and an image is generated. A moving image and a character string are input, and a moving image is generated. A 3D model and a character string are input, and a 3D model is generated. Audio and a character string are input, and audio is generated.An image management server according to the present embodiment performs a process based on inspection information, sensing information, and three-dimensional image information to update a tacit knowledge model using the inspection information, the sensing information, and the three-dimensional image information. Thus, a terminal device according to the present embodiment can display a tacit knowledge comment corresponding to the inspection information, the sensing information, and the three-dimensional image information.Even if acquisition of inspection information has failed at the time of generation of text information, the tacit knowledge model can output a tacit knowledge comment that takes the inspection information into consideration, based on the acquired three-dimensional image information and the sensing information.FN202502303

[0217] The above-described embodiments are illustrative and do not limit the present invention. Thus, numerous additional modifications and variations are possible in light of the above teachings. For example, elements and / or features of different illustrative embodiments may be combined with each other and / or substituted for each other within the scope of the present invention. Any one of the above-described operations may be performed in various other ways, for example, in an order different from the one described above.The image management server 40 described in the above-described embodiments is an example, and it is to be understood that various system configurations may be implemented depending on the application or purpose.

[0218] For example, the above-described embodiments illustrate an example in which a tacit knowledge model for use in an industry such as civil engineering or architecture answers a question. However, the tacit knowledge model may be used in any industry in which tacit knowledge is effective, such as medical care, dental care, or investment decision-making.

[0219] In the above-described embodiments, furthermore, the large language model 4005 generates text information based on a tacit knowledge comment. In another example, the large language model 4005 is not used, and a tacit knowledge comment may be used as text information.

[0220] The tacit knowledge model 4004 may be trained on tacit knowledge comments, using three-dimensional image information and inspection information as input and input information as output. That is, the tacit knowledge model 4004 may receive, as input, information in different modalities, namely an image and text.

[0221] In the above-described embodiments, furthermore, the information processing system 100 is a client-server system. However, the functions of the image management server 40 may be installed in the terminal device 10 as an application. That is, the user may be allowed to use the functions illustrated in the above-described embodiments in a stand-alone manner.

[0222] In the example configurations such as the example configuration illustrated in FIG. 3, each configuration is divided according to main functions to facilitate understanding of processing performed by the image management server 40. No limitation on the present disclosure is intended by how the functions are divided by process or by the name of the functions. The processing of the image management server 40 may be divided into more processing units in accordance with the content of the processing. Further, the division may be made such that each processing unit includes more processing operations.

[0223] The present invention can be implemented in any convenient form, for example using dedicated hardware, or a mixture of dedicated hardware and software. The present inventionFN202502303may be implemented as computer software implemented by one or more networked processing apparatuses. The processing apparatuses include any suitably programmed apparatuses such as a general purpose computer, a personal digital assistant, a Wireless Application Protocol (WAP) or third -generation (3G)-compliant mobile telephone, and so on. Since the present invention can be implemented as software, each and every aspect of the present invention thus encompasses computer software implementable on a programmable device. The computer software can be provided to the programmable device using any conventional carrier medium (carrier means). The carrier medium includes a transient carrier medium such as an electrical, optical, microwave, acoustic or radio frequency signal carrying the computer code. An example of such a transient medium is a Transmission Control Protocol / Internet Protocol (TCP / IP) signal carrying computer code over an IP network, such as the Internet. The carrier medium may also include a storage medium for storing processor readable code such as a floppy disk, a hard disk, a compact disc read-only memory (CD-ROM), a magnetic tape device, or a solid state memory device.

[0224] The functionality of the elements disclosed herein may be implemented using circuitry or processing circuitry which includes general purpose processors, special purpose processors, integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or combinations thereof which are configured or programmed, using one or more programs stored in one or more memories, to perform the disclosed functionality. Processors are considered processing circuitry or circuitry as they include transistors and other circuitry therein. In the disclosure, the circuitry, units, or means are hardware that carry out or are programmed to perform the recited functionality. The hardware may be any hardware disclosed herein which is programmed or configured to carry out the recited functionality.There is a memory that stores a computer program which includes computer instructions. These computer instructions provide the logic and routines that enable the hardware (e.g., processing circuitry or circuitry) to perform the method disclosed herein. This computer program can be implemented in known formats as a computer-readable storage medium, a computer program product, a memory device, a record medium such as a CD-ROM or DVD, and / or the memory of an FPGA or ASIC.

[0225] The apparatuses or devices described in one or more embodiments of the present disclosure are merely examples of computing environments that can implement the embodiments disclosed herein. In some embodiments, the image management server 40 includes multiple computing devices, such as a server cluster. The multiple computing devices communicate with one another through any type of communication link including a network, shared memory, or the like and perform the processes disclosed herein.

[0226] FN202502303Further, the image management server 40 may perform the processing steps disclosed herein in various combinations. The components of the image management server 40 may be integrated into one apparatus or divided into a plurality of apparatuses. The processes performed by the image management server 40 may be performed by the terminal device 10.

[0227] The present disclosure includes the following aspects.In Aspect 1, an information processing system includes a first server, a second server, and a terminal device communicable with the first server and the second server. The first server manages information related to a sensor. The information includes sensing information. The second server manages three-dimensional image information of a space in which the sensor is placed. The terminal device includes a display control unit. The display control unit displays a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server. The second server includes a processing unit. The processing unit executes a process to associate, based on the sensing information transmitted from the first server and the three-dimensional image information, the three-dimensional image information with the sensing information or generated information generated based on the sensing information. The display control unit of the terminal device displays, on the display screen, the three-dimensional image information received from the second server and the sensing information or the generated information corresponding to the three-dimensional image information.According to Aspect 2, in the information processing system of Aspect 1, the display control unit of the terminal device displays the display screen including a first display area and a second display area. The first display area is for displaying the information related to the sensor received from the first server. The second display area is for displaying the three-dimensional image information received from the second server and the sensing information or the generated information corresponding to the three-dimensional image information. According to Aspect 3, in the information processing system of Aspect 1 or Aspect 2, the second server causes a storage unit to store the sensing information transmitted from the first server in association with the three-dimensional image information.According to Aspect 4, in the information processing system of Aspect 1, the processing unit transmits a request for the sensing information to the first server and acquires the sensing information from the first server as a response to the request.According to Aspect 5, in the information processing system of any one of Aspects 1 to 4, the processing unit of the second server acquires the sensing information transmitted from the first server and comment data associated with the sensing information.According to Aspect 6, in the information processing system of Aspect 5, the second server further includes a model trained on correspondence among the three-dimensional image information, the sensing information, and the comment data, and the processing unit acquires the comment data generated as the generated information by the model based on the three-FN202502303dimensional image information and the sensing information for which a selection is accepted by the terminal device.According to Aspect 7, in the information processing system of Aspect 5, the second server further includes a model trained on correspondence among the three-dimensional image information, the sensing information, the comment data, and input information received from the terminal device, and the processing unit acquires the comment data generated as the generated information by the model based on the three-dimensional image information and the sensing information for which a selection is accepted by the terminal device.According to Aspect 8, in the information processing system of Aspect 6, the second server further includes an update unit. The update unit updates the model by training the model to learn correspondence among the three-dimensional image information, the sensing information, and the comment data.According to Aspect 9, in the information processing system of Aspect 7, the second server further includes an update unit. The update unit updates the model by training the model to learn correspondence among the three-dimensional image information, the sensing information, the comment data, and the input information received from the terminal device. According to Aspect 10, in the information processing system of any one of Aspects 1 to 9, the three-dimensional image information included in the display screen displayed by the display control unit represents a two-dimensional projection of a three-dimensional model shape and includes an image of the sensor with a changed viewpoint.According to Aspect 11, in the information processing system of any one of Aspects 1 to 10, the terminal device further includes an input reception unit. In a state where the three-dimensional image information and the sensing information or the generated information corresponding to the three-dimensional image information are displayed on the display screen by the display control unit, the input reception unit accepts a selection of another sensor different from the sensor for which the three-dimensional image information is being displayed. The processing unit executes a process to associate, based on other sensing information of said another sensor transmitted from the first server and other three-dimensional image information of a space in which said another sensor is placed, the other three-dimensional image information with the other sensing information or other generated information generated based on the other sensing information. The display control unit displays, on the display screen, the other three-dimensional image information received from the second server and the other sensing information or the other generated information corresponding to the other three-dimensional image information.

[0228] This patent application is based on and claims priority to Japanese Patent Application No. 2025-025165, filed on February 19, 2025, and 2026-018112, filed on February 5, 2026, in the Japan Patent Office, the entire disclosure of which is hereby incorporated by reference herein.[Reference Signs List]

[0229] FN20250230310 Terminal device20 Property information management server 40 Image management server100 Information processing system

Claims

1. FN202502303[CLAIMS]1. An information processing system comprising:a first server that manages information related to a sensor, the information including sensing information;a second server that manages three-dimensional image information of a space in which the sensor is placed; anda terminal device communicably connected with the first server and the second server, whereinthe terminal device includes a display control unit configured to display a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server,the second server includes a processing unit configured to execute a process to associate the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information, andthe display control unit of the terminal device is configured to display, on the display screen, the three-dimensional image information received from the second server and the one of the sensing information and the generated information associated with the three-dimensional image information.

2. The information processing system according to claim 1, whereinthe display control unit of the terminal device is configured to display the display screen including a first display area and a second display area,the first display area displaying the information related to the sensor received from the first server,the second display area displaying the three-dimensional image information received from the second server and the one of the sensing information and the generated information associated with the three-dimensional image information.

3. The information processing system according to claim 1 or 2, wherein the second server causes a storage unit to store the sensing information transmitted from the first server in association with the three-dimensional image information.

4. The information processing system according to claim 1, whereinthe processing unit is configured to transmit a request for the sensing information to the first server and acquire the sensing information from the first server as a response to the request.FN2025023035. The information processing system according to any one of claims 1 to 4, whereinthe processing unit of the second server is configured to acquire the sensing information transmitted from the first server and comment data associated with the sensing information.

6. The information processing system according to claim 5, whereinthe second server is further configured to execute a model trained on correspondence between the three-dimensional image information, the sensing information, and the comment data, andthe processing unit is configured to acquire the comment data generated as the generated information by the model based on the three-dimensional image information and the sensing information that is selected by the terminal device.

7. The information processing system according to claim 5, whereinthe second server is further configured to execute a model trained on correspondence between the three-dimensional image information, the sensing information, the comment data, and input information received from the terminal device, andthe processing unit is configured to acquire the comment data generated as the generated information by the model based on the three-dimensional image information and the sensing information that is selected by the terminal device.

8. The information processing system according to claim 6, whereinthe second server further includes an update unit configured to update the model by training the model to learn the correspondence between the three-dimensional image information, the sensing information, and the comment data.

9. The information processing system according to claim 7, whereinthe second server further includes an update unit configured to update the model by training the model to learn the correspondence between the three-dimensional image information, the sensing information, the comment data, and the input information received from the terminal device.

10. The information processing system according to any one of claims 1 to 9, whereinthe three-dimensional image information included in the display screen displayed by the display control unit represents a two-dimensional projection of a three-dimensional model shape and includes an image of the sensor with a changed viewpoint.FN20250230311. The information processing system according to any one of claims 1 to 10, whereinthe terminal device further includes an input reception unit configured to, while the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information are being displayed on the display screen by the display control unit, receive a selection of another sensor different from the sensor for which the three-dimensional image information is being displayed,the processing unit is configured to execute a process to associate another three-dimensional image information with one of another sensing information and another generated information generated based on the other sensing information, based on the other sensing information of the other sensor transmitted from the first server and the other three-dimensional image information of a space in which the other sensor is placed, andthe display control unit is configured to display, on the display screen, the other three-dimensional image information received from the second server and the one of the other sensing information and the other generated information associated with the other three-dimensional image information.

12. A second server in the information processing system according to any one of claims 1 to 11, comprising:the processing unit configured to execute the process to associate the three-dimensional image information with the one of the sensing information and the generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information; anda transmission unit configured to transmit screen information to the terminal device for displaying, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.

13. A first server in the information processing system according to any one of claims 1 to 11, comprising a transmission unit configured to transmit the sensing information to the terminal device to display the sensing information on the display screen, wherein the transmission unit is configured to further transmit the sensing information one of the terminal device and the second server, to display, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.wherein, in a case where a request from the terminal device having received a request from the second server is received, the transmission unit is configured to transmit the sensing information to the terminal device, andFN202502303in a case where a request from the second server is received, the transmission unit is configured to transmit the sensing information to the second server.

14. An information processing method performed by a second server in a system, the system including:a first server that manages information related to a sensor, the information including sensing information;the second server that manages three-dimensional image information of a space in which the sensor is placed; anda terminal device communicably connected with the first server and the second server to display a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server, the information processing method comprising:associating the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information; andtransmitting screen information to the terminal device, the screen information for displaying, on the display screen, the three-dimensional image information and the one of the sensing information and the generated information associated with the three-dimensional image information.

15. Carrier means carrying computer readable code for controlling a second server in a system to carry out an information processing method,the system including:a first server that manages information related to a sensor, the information including sensing information;a second server that manages three-dimensional image information of a space in which the sensor is placed; anda terminal device communicably connected with the first server and the second server to display a display screen including the information related to the sensor received from the first server and the three-dimensional image information received from the second server, the information processing method comprising:associating the three-dimensional image information with one of the sensing information and generated information generated based on the sensing information, based on the sensing information transmitted from the first server and the three-dimensional image information; andtransmitting screen information to the terminal device, the screen information for displaying, on the display screen, the three-dimensional image information and the one ofFN202502303the sensing information and the generated information associated with the three-dimensionalimage information.

Images