Equipment control device, equipment control system, equipment control method, and program

Abstract

An equipment control device (20) comprises: an acquisition unit (221) that acquires natural language or an image from equipment used by a user (40); an identification unit (222) that analyzes the natural language or the image acquired by the acquisition unit (221) to identify equipment (10) targeted by the natural language or the image and an equipment control signal for instructing the targeted equipment (10) to execute an operation indicated by the natural language or the image; and an output unit (223) that outputs the equipment control signal identified by the identification unit (222) to the identified equipment (10).

Description

Machine control device, machine control system, machine control method, and program

[0001] The present disclosure relates to a machine control device, a machine control system, a machine control method, and a program.

[0002] Patent Document 1 discloses a technology for a user to remotely operate home appliances using a smartphone or the like.

[0003] Japanese Patent Application Laid-Open No. 2020-036276

[0004] Here, in the technology disclosed in Patent Document 1, when a user remotely controls a home appliance using a smartphone, there is a problem that it takes time to operate because there are many items that require input.

[0005] An object of the present disclosure is to provide a machine control device, a machine control system, a machine control method, and a program that can simplify the operation when a user remotely controls a machine.

[0006] One aspect of the machine control device according to the present disclosure includes an acquisition unit that acquires natural language or an image from a machine used by a user, and by analyzing the natural language or the image acquired by the acquisition unit, the natural language or the image is directed to a machine, and a specifying unit that specifies a machine control signal for instructing execution of an operation indicated by the natural language or the image on the target machine, and an output unit that outputs the machine control signal specified by the specifying unit to the specified machine.

[0007] In addition, one aspect of the machine control system according to the present disclosure includes one or more machines owned by a user and the above-described machine control device.

[0008] In addition, one aspect of the machine control method according to the present disclosure includes an acquisition step of acquiring natural language or an image from a machine used by a user, and by analyzing the natural language or the image acquired in the acquisition step, the natural language or the image is directed to a machine, and a specifying step of specifying a machine control signal for instructing execution of an operation indicated by the natural language or the image on the target machine, and an output step of outputting the machine control signal specified in the specifying step to the specified machine.

[0009] Furthermore, this disclosure can be implemented not only as the above-mentioned device control method, but also as a program that causes a computer to execute the above-mentioned device control method. Moreover, it can also be implemented as a computer-readable recording medium that stores the program.

[0010] The equipment control device, equipment control system, equipment control method, and program related to this disclosure can simplify the operation when a user remotely controls equipment.

[0011] Figure 1 is a conceptual diagram showing an overview of the device control system according to the embodiment. Figure 2 is a block diagram showing the functional configuration of the device control system according to the embodiment. Figure 3 is a diagram showing an example of a device database according to the embodiment. Figure 4 is a diagram showing an example of a command database according to the embodiment. Figure 5 is a flowchart explaining the operation of the device control device according to the embodiment. Figure 6A is a diagram showing an example of a screen for instruction information according to the embodiment. Figure 6B is a diagram showing another example of a screen for instruction information according to the embodiment. Figure 7 is a diagram showing an example of a screen when querying a target device according to the embodiment. Figure 8 is a diagram showing another example of a screen when querying a target device according to the embodiment. Figure 9 is a diagram showing an example of a screen when ignoring natural language according to the embodiment. Figure 10A is a diagram showing an example of a screen when querying a command according to the embodiment. Figure 10B is a diagram showing another example of a screen when querying a command according to the embodiment. Figure 10C is a diagram showing yet another example of a screen when querying a command according to the embodiment. Figure 11 is a diagram showing an example of a screen for confirming a device control signal according to the embodiment. Figure 12 is a diagram showing an example of a screen when querying a target device according to a modified example of the embodiment.

[0012] (Embodiments) Hereinafter, embodiments of the equipment control device, equipment control system, equipment control method, and program relating to this disclosure will be described in detail with reference to the drawings. The embodiments described below are all preferred specific examples of this disclosure. The numerical values, components, arrangement positions and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit this disclosure. Furthermore, among the components in the following embodiments, components not described in an independent claim will be described as any component constituting a preferred configuration.

[0013] Please note that each figure is a schematic diagram and not necessarily a strictly accurate representation. Furthermore, in each figure, substantially identical components are denoted by the same reference numerals, and redundant explanations may be omitted or simplified.

[0014] Furthermore, unless otherwise specified, ordinal numbers such as "first," "second," etc., in this disclosure do not refer to the number or order of components, etc., but are used for the purpose of avoiding confusion and distinguishing similar components, etc.

[0015] [Overview] First, an overview of the equipment control system according to the embodiment will be described. Figure 1 is a conceptual diagram showing an overview of the equipment control system 1 according to the embodiment.

[0016] As shown in Figure 1, the device control system 1 comprises one or more devices 10 and a device control device 20. The one or more devices 10 and the device control device 20 are connected to each other so as to be able to communicate via a public network 50 such as the Internet. Furthermore, the device control device 20 is connected to a terminal device 30 used by a user 40 so as to be able to communicate via the public network 50.

[0017] In other words, in the device control system 1, the device control device 20 communicates with the device 10 and the terminal device 30 to support the exchange of information between the device 10 and the terminal device 30.

[0018] One or more devices 10 include, for example, household electrical appliances (hereinafter also referred to as home appliances) installed in a house 100. Figure 1 shows a washing machine 10a and a refrigerator 10b as examples of devices 10. Device 10 may be a vacuum cleaner such as a cleaning robot, an air conditioning unit such as an air conditioner, or a dishwasher. As long as it can communicate with the device control unit 20 via the public network 50, the device 10 is not limited to home appliances, and its specific form is not particularly limited.

[0019] [Configuration] First, the configuration of the device control system according to this embodiment will be described. Figure 2 is a block diagram showing the functional configuration of the device control system 1 according to this embodiment. The device control system 1 identifies the device 10 targeted by the natural language or image input by the user 40, and the content of the control, and can control the identified device 10 with the identified content of the control.

[0020] The equipment control system 1 comprises one or more pieces of equipment 10 and an equipment control device 20.

[0021] The device 10 includes home appliances installed in the house 100. The device 10 is, for example, a washing machine 10a or a refrigerator 10b. The device 10 includes a communication unit 11.

[0022] The communication unit 11 is a wireless communication module for the device 10 to communicate wirelessly with the device control device 20 via the public network 50. The communication unit 11 communicates wirelessly according to a communication standard such as NB-IoT (Narrow Band Internet of Things) or HTTP (Hypertext Transfer Protocol). The communication standard used by the communication unit 11 is not particularly limited.

[0023] The device control unit 20 is a device that communicates with the device 10 and the terminal device 30 used by the user 40. The device control unit 20 is implemented, for example, by a cloud server. Specifically, the device control unit 20 comprises a communication unit 21, an information processing unit 22, and a storage unit 23.

[0024] The communication unit 21 is a wireless communication module for the device control device 20 to wirelessly communicate with the device 10 and terminal device 30 via the public network 50. The communication unit 21 performs wireless communication according to a communication standard such as LTE (Long Term Evolution) (registered trademark). The communication standard used for communication by the communication unit 21 is not particularly limited.

[0025] The information processing unit 22 performs information processing to control the device 10 based on natural language or images input by the user 40. The information processing unit 22 is implemented by, for example, a microcomputer, but may also be implemented by a processor or dedicated circuit. The functions of the information processing unit 22 are realized by the execution of a software program, such as a predetermined application program, stored in the storage unit 23 by hardware such as a microcomputer or processor. Specifically, the information processing unit 22 includes an acquisition unit 221, a specification unit 222, and an output unit 223.

[0026] The acquisition unit 221 uses the communication unit 21 to acquire natural language or images from the terminal device 30. Natural language refers to the language that the user 40 uses on a daily basis. Natural language is, for example, the language that the user 40 uses when freely inputting text into the terminal device 30. Images are photographs, but may also be illustrations.

[0027] The identification unit 222 analyzes the natural language or image acquired by the acquisition unit 221 to identify the device 10 targeted by the natural language or image, and the device control signal that instructs the target device 10 to perform the action indicated by the natural language or image. The identification unit 222, for example, uses a machine learning model (a trained model) to identify the device 10 targeted by the natural language or image, and identifies the device control signal that instructs the target device 10 to perform the action indicated by the natural language or image. Note that the machine learning model is an example of AI (Artificial Intelligence), and AI that does not use a machine learning model may also be used. AI refers to artificial intelligence, and for example, a machine learning model such as a trained neural network.

[0028] The output unit 223 outputs the device control signal identified by the identification unit 222 to the identified device 10.

[0029] The memory unit 23 is a storage device that stores information necessary for information processing, such as computer programs to be executed. The memory unit 23 is implemented using semiconductor memory.

[0030] The storage unit 23 further stores a device database 61 and a command database 62. Figure 3 is a diagram showing an example of a device database according to the embodiment. Figure 4 is a diagram showing an example of a command database according to the embodiment.

[0031] As shown in Figure 3, the device database 61 associates information indicating a registered user with information about one or more devices 10. The device database 61 is configured for each piece of information indicating a registered user.

[0032] The information indicating a registered user is, for example, information used to identify user 40 who registered the device 10. The information indicating a registered user is, for example, ID information assigned to user 40 when user 40 registered to use the services provided by the device control system 1.

[0033] The information indicating a registered user may also be attribute information. Attribute information is information used to identify the group to which user 40 belongs. Attribute information is, for example, a unique number assigned to the group to which user 40 belongs. In other words, the same attribute information value is associated with multiple users 40 belonging to the same group. In the device database 61 shown in Figure 3, attribute information values ​​may be associated instead of information indicating a registered user.

[0034] Information about the device 10 includes, for example, device identification information, device type, and installation location. For each of the multiple devices 10 stored in the device database 61, device identification information, device type, and installation location are associated. Device identification information is information for identifying the device 10. Device type is, for example, the general name of the device 10. Device type may also be the model number of the device 10 and information indicating the functions that the device 10 has. Device type is associated, for example, by the device control device 20 based on the device identification information. Installation location is information indicating where the device 10 is installed in the house 100. Installation location is associated, for example, by user 40 input.

[0035] In the equipment database 61 shown in Figure 3, the information indicating a registered user, "User A," is associated with equipment 10 whose equipment identification information is "Washing Machine A," equipment type is "Washing Machine," and installation location is "Laundry Room." In addition, the information indicating a registered user, "User A," is associated with equipment 10 whose equipment identification information is "Refrigerator A," equipment type is "Refrigerator," and installation location is "Kitchen." This indicates that user 40, whose user information is "User A," has registered equipment 10. User 40 can control equipment 10.

[0036] As shown in Figure 4, the command database 62 associates device identification information with information indicating the control content that the device 10 indicated by the device identification information can execute. The command database 62 is configured for each device 10.

[0037] The information indicating the control content that the device 10, as indicated by the device identification information, can execute is, for example, information indicating the model number, but it may also be information indicating a list of control content. It may also be a combination of these. The command database 62 is associated by the device control device 20 based on, for example, the catalog information of the device 10. In addition, control content may be added to the command database 62 by input from the user 40.

[0038] In the command database 62 shown in Figure 4, the device identification information "Washing Machine A" is associated with the model number "ABC01" and information showing a list of control options such as "Normal Course," "Powerful Waterfall Course," and "Delicate Course."

[0039] Here, the device control unit 20 can assist in the exchange of message information between multiple users 40 whose attribute information values ​​match. For example, the device control unit 20 can assist in the exchange of message information between two users 40a and user 40b. Users 40a and user 40b are users to whom the same attribute information values ​​are associated. The communication unit 21 exchanges message information with each of users 40a and user 40b. Furthermore, the communication unit 21 transmits the message information communicated with user 40a to user 40b. Specifically, the communication unit 21 can transmit the message information communicated with user 40a to user 40b by, for example, referring to the attribute information values ​​described above and obtaining user 40b, which is indicated by user information to which the same attribute information values ​​as user information "User A" of user 40a are associated. Similarly, the communication unit 21 can transmit the message information communicated with user 40b to user 40a. In this way, users 40a and user 40b can exchange message information via the device control unit 20.

[0040] Terminal device 30 is an information terminal used by user 40. Terminal device 30 is, for example, a smartphone. Terminal device 30 transmits natural language or images input by user 40 to the device control device 20. Terminal device 30 may also be a tablet terminal or a personal computer, etc. Terminal device 30 is just one example of a device used by user 40.

[0041] For example, the user 40 uses the terminal device 30 to use an external network service and communicates with the device control device 20 via the external network service. The external network service is, for example, an SNS (Social Network Service), but it may also be a mail system or a dedicated application program related to the device 10 and the device control device 20. The specific form of the external network service is not particularly limited as long as the device control device 20 can transmit information to the user 40 via wireless communication. Specifically, the terminal device 30 includes a communication unit 31, a display unit 32, and an operation reception unit 33.

[0042] The communication unit 31 is a communication circuit (wireless communication module) for the terminal device 30 to perform wireless communication with the device control device 20. Specifically, the communication unit 31 performs wireless communication according to a communication standard such as LTE. The communication standard of the communication performed by the communication unit 31 is not particularly limited.

[0043] The display unit 32 is a screen on which various images are displayed. The display unit 32 is realized by, for example, a display panel such as a liquid crystal panel or an organic EL (Electro Luminescence) panel.

[0044] The operation reception unit 33 receives the operations of the user 40. The operation reception unit 33 is realized by, for example, a touch panel or the like.

[0045] [Operation] Next, the operation of the device control system 1 according to this embodiment will be described. Figure 5 is a flowchart illustrating the operation of the device control device 20 according to this embodiment. Figures 6A to 12 are diagrams showing examples of screens displayed on the display unit 32 of the terminal device 30 in the operation described below. Regarding the message information shown by the bubbles in Figures 6A to 12, the bubbles with reference numerals 610a, 610b, and 611 to 618 indicate message information indicating instruction information. The bubbles with reference numerals 621, 622, and 624 to 628 indicate message information indicating inquiry information. The bubbles with reference numerals 631, 632, and 634 to 638 indicate message information indicating response information. The bubbles with reference numerals 640a, 640b, and 641 to 648 indicate message information indicating device control signals. The bubble with reference numeral 653 indicates message information indicating non-instruction information.

[0046] First, the acquisition unit 221 acquires natural language or an image (S11). The terminal device 30 accepts natural language or image input from the user 40 using the operation reception unit 33. The natural language or image input by the user 40 is transmitted using the communication unit 31 and received by the device control device 20 via the public network 50. The terminal device 30 accepts natural language input, for example, by displaying a text box on the display unit 32 and having the user 40 input text using the operation reception unit 33. The method of accepting natural language input is not particularly limited here.

[0047] When natural language or images are acquired by the acquisition unit 221, the identification unit 222 identifies the device 10 that the natural language or images target (S12). Specifically, the identification unit 222 identifies the device 10 that the natural language or images target by analyzing the acquired natural language or images using AI. The device 10 that the natural language or images target is the device 10 that has been analyzed to be controlled by the user 40 using the natural language or images. For example, the identification unit 222 identifies the device 10 by referring to the device type associated with the device database 61 based on the results of the analysis of the natural language or images.

[0048] For example, when the device database 61 is as shown in FIG. 3, it is assumed that the user 40 with user information "User A" receives a natural language input of "Start washing". The specifying unit 222 obtains "washing machine" as a candidate for the device type by analyzing "Start washing", and specifies the device identification information "Washing Machine A" whose device type is "washing machine" by referring to the device type associated with the device database 61.

[0049] Next, the specifying unit 222 determines whether the device 10 can be specified (S13).

[0050] If the specifying unit 222 can specify the device 10 (Yes in S13), the process proceeds to the process of step S16.

[0051] In the process of step S13, if the device 10 can be specified, the natural language or image input by the user 40 in step S11 is instruction information. The instruction information is the natural language or image input by the user 40 to control the device 10.

[0052] FIG. 6A is a diagram showing an example of a screen for the instruction information according to the embodiment. As shown in FIG. 6A, when the user 40 inputs a natural language "Start washing" as the instruction information 610a, the specifying unit 222 specifies the "washing machine", which is the device 10 targeted by the natural language, and "start of washing", which is the operation indicated by the natural language or image for the washing machine. That is, the specifying unit 222 specifies a device control signal. The specifying unit 222 transmits "Start washing" as the information 640a indicating the specified device control signal and causes it to be displayed on the display unit 32 of the terminal device 30.

[0053] Figure 6B shows another example of a screen for instruction information according to the embodiment. As shown in Figure 6B, when user 40 inputs an image of a person about to do laundry as instruction information 610b, the identification unit 222 identifies the device 10 targeted by the natural language as "washing machine" and the action indicated by the natural language or image to the washing machine as "start washing". In other words, the identification unit 222 identifies the device control signal. The identification unit 222 transmits "start washing" as information 640b indicating the identified device control signal, and displays it on the display unit 32 of the terminal device 30.

[0054] If the identification unit 222 is unable to identify the device 10 (No in S13), the identification unit 222 determines whether or not a candidate for the device 10 exists (S14).

[0055] If it is determined that there are candidates for device 10 (Yes in S14), in other words, if there are multiple devices 10 that could be the target of natural language or images and therefore device 10 could not be identified in the processing of step S12, the identification unit 222 sends inquiry information to the terminal device 30 (S15). The inquiry information is information sent to the terminal device 30 used by the user 40 in order to identify device 10 from among the candidates for device 10. Based on the user 40's response to the inquiry information, the identification unit 222 can identify the device 10 that the instruction information targets.

[0056] For example, in the process of step S11, if the content of the natural language acquired by the acquisition unit 221 is ambiguous, there may be multiple candidates for the device 10, making it impossible to identify the device 10.

[0057] Figure 7 shows an example of a screen when querying the target device according to the embodiment. As shown in Figure 7, when user 40 inputs the natural language "start washing" as instruction information 611, the identification unit 222 obtains "washing machine" and "dishwasher" as the target device 10 of the instruction information 611, and therefore cannot identify the device 10. In this case, the identification unit 222 sends "washing machine or dishwasher?" as inquiry information 621 and displays it on the display unit 32 of the terminal device 30. User 40 inputs "washing machine" as response information 631 to the inquiry information 621. The input of the response information 631 may be done by text input by user 40, or by presenting a list of options and allowing user 40 to select one. Based on the response information 631, the identification unit 222 can identify that the device 10 targeted by the instruction information 611 is the device with the device identification information "washing machine A".

[0058] Furthermore, for example, if multiple devices 10 of the same device type are associated with the device database 61, there may be multiple candidates for device 10, making it impossible to identify the correct device 10.

[0059] Figure 8 shows another example of a screen when querying a target device according to the embodiment. As shown in Figure 8, when user 40 inputs the natural language "Turn on the air conditioner" as instruction information 612, the identification unit 222 obtains "Air conditioner" as the target device 10 of the instruction information 612. However, in the device database 61 of Figure 3, there are three devices 10 with the device type "Air conditioner": device identification information "Air conditioner A", "Air conditioner B", and "Air conditioner C". Therefore, the identification unit 222 cannot identify the device 10. In this case, the identification unit 222 sends the inquiry information 622 "There are three air conditioners - living room, bedroom, and children's room - which air conditioner would you like to operate?" and displays it on the display unit 32 of the terminal device 30. User 40 inputs "Living room" as response information 632 to the inquiry information 622. Based on the response information 632, the identification unit 222 can identify that the device 10 targeted by the instruction information 612 is the device with the device identification information "Air conditioner A" associated with the installation location "Living room".

[0060] If it is determined that no candidate device 10 exists (No in S14), and if device 10 could not be identified in the processing of step S12, the device control device 20 terminates its operation. In other words, the output unit 223 does not produce output based on natural language or images if it is unable to identify the target device 10 because the target device 10 does not exist in the natural language or image analysis performed by the identification unit 222.

[0061] For example, if user 40 inputs natural language or an image that is not intended to control device 10, there is no device 10 that could be the target of the natural language or image. Also, for example, if user 40 inputs natural language or an image intended to control device 10 that they do not possess, there is no device 10 that could be the target of the natural language or image.

[0062] For example, the device control unit 20 can assist in the exchange of message information between user 40a and user 40b, as described above. When message information can be exchanged between user 40a and user 40b, the device control unit 20 may receive natural language or images (hereinafter also referred to as non-instructive information), such as greetings or notices. In such cases, the device control unit 20 can choose not to output anything based on the non-instructive information. Figure 9 shows an example of a screen when natural language is ignored according to the embodiment.

[0063] In Figure 9, terminal device 30a is an information terminal used by user 40a, and terminal device 30b is an information terminal used by user 40b. Figure 9 shows user 40a greeting user 40b with "Good morning." As shown in Figure 9, when user 40a inputs the natural language "Good morning" as non-instructive information 653, the identification unit 222 cannot identify the target device 10 because the target device 10 does not exist in the analysis of natural language or images. In this case, the device control device 20 considers the natural language "Good morning" as non-instructive information and terminates its operation without performing the processing necessary to control the device 10. On the other hand, as shown in Figure 9, when user 40 inputs the natural language "Start washing" as instruction information 613, the identification unit 222 identifies the device 10 targeted by the natural language as "washing machine" and the action indicated by the natural language or image to the washing machine as "Start washing," and transmits "Start washing" as information 643 indicating the identified device control signal, which is displayed on the display unit 32 of the terminal device 30.

[0064] After the identification unit 222 identifies the device 10, the identification unit 222 identifies a command (S16). Specifically, the identification unit 222 identifies a command by analyzing the acquired natural language or image using AI. A command is an action indicated by the natural language or image for the identified device 10.

[0065] For example, let's describe a situation where, if the command database 62 is as shown in Figure 4, the natural language command "Start washing" is acquired, and the device 10, whose device identification information is "Washing Machine A," is identified. The identification unit 222 analyzes "Start washing" and, by referring to the command database 62, identifies "Normal Course" from the information indicating the control content that the device 10 can execute.

[0066] Next, the identification unit 222 determines whether or not it was able to identify the command (S17).

[0067] If the command is identified by the identification unit 222 (Yes in S17), the process proceeds to step S19.

[0068] If the identification unit 222 fails to identify a command (No in S17), the identification unit 222 sends inquiry information (S18). In other words, if the identification unit 222 is unable to identify a command because there are multiple actions indicated by natural language or images for the identified device 10, the identification unit 222 sends inquiry information. The inquiry information is sent to the terminal device 30 used by the user 40 in order to identify a command from among the command candidates. Based on the user 40's response to the inquiry information, the identification unit 222 can identify the command targeted by the instruction information.

[0069] For example, the specific unit 222 may send query information to prompt the user 40 to input natural language corresponding to a command.

[0070] Figure 10A shows an example of a screen when querying a command according to the embodiment. As shown in Figure 10A, when user 40 inputs the natural language "Start washing" as instruction information 614, the identification unit 222 can identify that the device 10 is "Washing Machine A", but cannot identify the command. In this case, the identification unit 222 sends "Which washing course would you like?" as inquiry information 624 and displays it on the display unit 32 of the terminal device 30. User 40 inputs "Leave it to the machine" as response information 634 to inquiry information 624. The input of response information 634 may be done by text input by user 40, or by presenting a list of options for user 40 to select. Based on the response information 634, the identification unit 222 can identify that the command indicated by instruction information 614 is the "automatic course".

[0071] Furthermore, for example, the identification unit 222 may send query information to prompt the user 40 to input relevant information for identifying a command. With such query information, the identification unit 222 can identify a command without the user 40 having to input natural language corresponding to the command.

[0072] Figure 10B shows another example of a screen when querying a command according to the embodiment. As shown in Figure 10B, when user 40 inputs the natural language "Start washing" as instruction information 615, the identification unit 222 can identify that the device 10 is "Washing Machine A", but cannot identify the command. In this case, the identification unit 222 sends "What would you like to wash?" as inquiry information 625 and displays it on the display unit 32 of the terminal device 30. User 40 inputs an image of a chiffon blouse as response information 635 to the inquiry information 625. The input of the response information 635 may be done by sending an image or by inputting natural language text. By analyzing the response information 635 using AI, the identification unit 222 can identify that the command indicated by the instruction information 615 is the "delicate cycle".

[0073] Furthermore, for example, the identification unit 222 may transmit query information based on commands performed during the period from before the current predetermined period to the present. For example, the storage unit 23 stores equipment control signals output to the identified equipment 10 during the period from before the current predetermined period to the present, and the identification unit 222 may transmit a list of commands corresponding to the equipment control signals output during the period from before the current predetermined period to the present as query information. Here, the predetermined period is, for example, one week, but it may also be two weeks or one month. The length of the predetermined period is not particularly limited. With such query information, the identification unit 222 can present the controls performed by the user 40 during the period from before the current predetermined period to the present as options, thereby reducing the hassle for the user 40 when selecting a command.

[0074] Figure 10C shows yet another example of the screen when querying a command according to the embodiment. As shown in Figure 10C, when user 40 inputs the natural language "Start washing" as instruction information 616, the identification unit 222 can identify that the device 10 is "Washing Machine A", but cannot identify the command. In this case, the identification unit 222 sends inquiry information 626, "Which course would you like? The following courses are frequently used: Powerful Waterfall, Automatic, Delicate, Other", and displays it on the display unit 32 of the terminal device 30. User 40 inputs response information 636 "Powerful Waterfall" in response to inquiry information 626. Based on the response information 636, the identification unit 222 can identify that the command indicated by instruction information 616 is the "Powerful Waterfall course".

[0075] When the identification unit 222 identifies a command, the output unit 223 outputs a device control signal corresponding to the identified command to the identified device 10 (S19). The output device control signal is received by the device 10 via the public network 50. The device 10 uses the communication unit 11 to perform control based on the received device control signal.

[0076] At this time, before the output unit 223 outputs the device control signal, the identification unit 222 may transmit information indicating the identified device 10 and the device control signal to the terminal device 30 and obtain confirmation from the user 40. By obtaining confirmation from the user 40 before transmitting the identified device control signal to the identified device 10, it is possible to prevent the execution of incorrect control.

[0077] Figure 11 shows an example of a screen for confirming a device control signal according to an embodiment. As shown in Figure 11, when user 40 inputs the natural language "Start washing" as instruction information 617, the identification unit 222 identifies the device 10 targeted by the natural language as "washing machine" and the action indicated by the natural language or image on the washing machine as "Start washing". In other words, the identification unit 222 identifies the device control signal. The identification unit 222 sends the inquiry information 627 "Washing will begin. Are you ready? Yes / No" and displays it on the display unit 32 of the terminal device 30. User 40 inputs "Yes" as response information 637 to the inquiry information 627. Based on the response information 637, the identification unit 222 confirms that the identified device 10 and device control signal are correct. The output unit 223 transmits the identified device control signal to the identified device 10. Finally, it sends the information 647 indicating the confirmed device control signal "Washing has started" and displays it on the display unit 32 of the terminal device 30.

[0078] Based on the above operation, the device control device 20 can identify the device and device control signals targeted by the natural language or images obtained from the device used by the user 40. Therefore, such a device control device 20 can make it easier for the user 40 to remotely control the device 10.

[0079] [Modification] In the above embodiment, the user 40 who responds to the inquiry information was the user 40 who sent the instruction information. However, the inquiry information may be responded to by a user 40 who did not send the instruction information. In other words, the response information to the instruction information entered by user 40a may be entered by user 40b instead of user 40a.

[0080] Figure 12 shows an example of a screen when inquiring about a target device according to a modified embodiment. Figure 12 corresponds to Figure 8.

[0081] As shown in Figure 12, when user 40a inputs the natural language command "Turn on the air conditioner" as instruction information 618, the device database 61 in Figure 3 contains multiple devices 10 with the device type "air conditioner," so the identification unit 222 cannot identify the device 10. In this case, the identification unit 222 sends the inquiry information 628, "There are three air conditioners. Which air conditioner would you like to operate: the one in the living room, the one in the bedroom, or the one in the children's room?" The inquiry information 628 is displayed on the display unit 32a of the terminal device 30a used by user 40a and on the display unit 32b of the terminal device 30b used by user 40b.

[0082] Next, user 40b inputs "children's room" as response information 638 to inquiry information 628. Based on the response information 638, the identification unit 222 identifies the equipment 10 targeted by instruction information 618 as equipment 10 whose equipment identification information "air conditioner C" is associated with the installation location "children's room".

[0083] With such a device control device 20, a user 40 who does not input natural language or images to control the device 10 can control the device 10.

[0084] In Figure 12, another user responded to the inquiry information for identifying device 10, but another user may also respond to the inquiry information for identifying a command, or to the inquiry information for confirming a device control signal.

[0085] [Effects, etc.] The technologies derived from the disclosures in this specification include, for example, the following technologies. The technologies derived from the disclosures in this specification will be described below, along with the effects, etc., that can be obtained by such technologies.

[0086] Technology 1 is an equipment control device 20 comprising: an acquisition unit 221 that acquires natural language or images from equipment used by user 40; an identification unit 222 that analyzes the natural language or images acquired by the acquisition unit 221 to identify the equipment 10 targeted by the natural language or images, and an equipment control signal that instructs the target equipment 10 to perform the action indicated by the natural language or images; and an output unit 223 that outputs the equipment control signal identified by the identification unit 222 to the identified equipment 10. In this embodiment, the terminal equipment 30 is an example of equipment used by user 40.

[0087] With such a device control device 20, the user 40 can perform desired control of the device 10 by inputting natural language or images. Therefore, such a device control device 20 can make it easier for the user 40 to control the device 10 remotely.

[0088] Technology 2 is a device control device 20 of Technology 1, in which the identification unit 222 transmits inquiry information for identification to the device used by the user 40 when it is not possible to identify at least one of the target device 10 and the device control signal.

[0089] Such an equipment control device 20 can more reliably and accurately identify the equipment 10 and the equipment control signals by querying the user 40.

[0090] Technology 3 is a device control device 20 according to Technology 1 or 2, in which the identification unit 222 identifies the target device 10 and the device control signal, and then transmits information indicating the identified device 10 and the device control signal to the device used by the user 40.

[0091] With such an equipment control device 20, it is possible to prevent incorrect control from being executed by obtaining confirmation from the user 40 before transmitting the specified equipment control signal to the specified equipment 10.

[0092] Technology 4 is a device control device 20 according to any of Technologies 1 to 3, which includes a communication unit that communicates with a server that provides an external network service used by user 40, and an acquisition unit 221 that acquires natural language or images input by user 40 via the external network service.

[0093] Such a device control device 20 can receive natural language or image input from a user 40 via an external network service such as an SNS.

[0094] Technology 5 is the device control device 20 of Technology 4, wherein the output unit 223 does not produce output based on natural language or images when the target device 10 cannot be identified because the target device 10 does not exist in the analysis of natural language or images by the identification unit 222.

[0095] Such a device control device 20 can, for example, refrain from controlling the device 10 if the user 40 inputs natural language or an image that is not intended to control the device 10. Therefore, such a device control device 20 can be applied to services in which users 40 interact with each other in ways that are not related to controlling the device 10.

[0096] Technology 6 is a device control device 20 according to any of Technologies 1 to 5, in which the specific unit 222 identifies the device 10 targeted by natural language or images obtained from a device used by user 40, based on a device database 61 in which device identification information identifying the device 10 and user information identifying the user 40 are associated.

[0097] Such an equipment control device 20 can identify the equipment 10 based on the equipment database 61.

[0098] Technology 7 is a device control device 20 according to any of Technologies 1 to 6, in which the identification unit 222 identifies a device control signal based on a command database 62 in which the model number of the device 10 is associated with a device control signal that can be executed by the device 10.

[0099] With such an equipment control device 20, equipment control signals can be identified based on the command database 62.

[0100] Technology 8 is an equipment control system 1 comprising one or more devices 10 owned by the user 40 and an equipment control device 20 of any of technologies 1 to 7.

[0101] Such a device control system 1 can assist user 40 in performing desired control of the device 10 by inputting natural language or images. Therefore, such a device control system 1 can make it easier for user 40 to control the device 10 remotely.

[0102] Technology 9 is a device control method that includes an acquisition step of acquiring natural language or images from a device used by user 40; an identification step of identifying a device 10 targeted by the natural language or images and a device control signal that instructs the target device 10 to perform the action indicated by the natural language or images by analyzing the natural language or images acquired in the acquisition step; and an output step of outputting the device control signal identified in the identification step to the identified device 10.

[0103] With this device control method, the user 40 can perform desired control of the device 10 by inputting natural language or images. Therefore, this device control method makes it easier for the user 40 to control the device 10 remotely.

[0104] Technology 10 is a program that causes a computer to execute the equipment control method of Technology 9.

[0105] Such a program can help simplify operations when user 40 remotely controls device 10.

[0106] (Other Embodiments) Although embodiments have been described above, this disclosure is not limited to the embodiments described above.

[0107] For example, in the above embodiment, the equipment control device was implemented by a single device, but it may be implemented by multiple devices. When the equipment control device is implemented by multiple devices in this way, the components of the equipment control device (in particular, the functional components) may be distributed among the multiple devices in any manner.

[0108] Furthermore, the method of communication between devices in the above embodiment is not particularly limited. In addition, relay devices (such as gateway devices), which are not shown, may be involved in the communication between devices.

[0109] Furthermore, in the above embodiment, a process executed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, or multiple processes may be executed in parallel.

[0110] Furthermore, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0111] Furthermore, each component may be implemented by hardware. For example, each component may be a circuit (or integrated circuit). These circuits may form a single circuit as a whole, or they may be separate circuits. Also, each of these circuits may be a general-purpose circuit or a dedicated circuit.

[0112] Furthermore, the general or specific embodiments of this disclosure may be implemented as a system, apparatus, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM. They may also be implemented as any combination of a system, apparatus, method, integrated circuit, computer program, and recording medium.

[0113] For example, this disclosure may be implemented as a device control system of the above embodiment, or as a device control method executed by a computer such as the device control system (device control device) of the above embodiment, or as a program (in other words, a computer program product) for causing a computer to execute the device control method. Furthermore, this disclosure may be implemented as a computer-readable non-temporary recording medium on which such a program is recorded.

[0114] Furthermore, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art could conceive, or forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of this disclosure.

[0115] This disclosure can be used in technologies that enable users to operate communication devices remotely.

[0116] 1 Equipment control system 10 Equipment 10a Washing machine 10b Refrigerator 11 Communication unit 20 Server (equipment control device) 21 Communication unit 22 Information processing unit 221 Acquisition unit 222 Identification unit 223 Output unit 23 Storage unit 30, 30a, 30b Terminal equipment 31 Communication unit 32, 32a, 32b Display unit 33 Operation reception unit 40, 40a, 40b User 50 Public network 61 Equipment database 62 Command database 610a, 610b, 611, 612, 613, 614, 615, 616, 617, 618 Instruction information 621, 622, 624, 625, 626, 627, 628 Inquiry information 631, 632, 634, 635, 636, 637, 638 Response information 640a, 640b, 641, 642, 643, 644, 645, 646, 647, 648 Information indicating equipment control signals 653 Non-indication information 100 Housing

Claims

1. A device control device comprising: an acquisition unit that acquires natural language or images from a device used by a user; an identification unit that identifies the device targeted by the natural language or images and a device control signal that instructs the target device to perform the action indicated by the natural language or images by analyzing the natural language or images acquired by the acquisition unit; and an output unit that outputs the device control signal identified by the identification unit to the identified device.

2. The device control device according to claim 1, wherein if the identification unit cannot identify at least one of the target device and the device control signal, it transmits inquiry information for identification to the device used by the user.

3. The device control device according to claim 1 or 2, wherein the identifying unit identifies the target device and the device control signal, and then transmits information indicating the identified device and the device control signal to a device used by the user.

4. The device control device according to claim 1 or 2, comprising a communication unit that communicates with a server that provides an external network service used by the user, wherein the acquisition unit acquires natural language or images input by the user via the external network service.

5. The device control device according to claim 4, wherein the output unit does not perform output based on the natural language or image when the target device cannot be identified in the analysis of the natural language or image by the identification unit because the target device does not exist.

6. The device control device according to claim 1 or 2, wherein the identifying unit identifies the device to which the natural language or image obtained from the device used by the user is intended, based on a device database in which device identification information for identifying the device and user information for identifying the user are associated.

7. The device control device according to claim 1 or 2, wherein the identifying unit identifies the device control signal based on a command database in which the model number of the device and the device control signals that can be executed by the device are associated.

8. A device control system comprising one or more devices owned by a user, and a device control device according to claim 1 or 2.

9. A device control method comprising: an acquisition step of acquiring natural language or images from a device used by a user; an identification step of identifying the device targeted by the natural language or images and a device control signal that instructs the target device to perform the action indicated by the natural language or images by analyzing the natural language or images acquired in the acquisition step; and an output step of outputting the device control signal identified in the identification step to the identified device.

10. A program that causes a computer to execute the device control method described in claim 9.

Images