Image search method, image search program, and image search device

Abstract

This image search method, performed by a computer, includes: receiving input of first text; generating a plurality of first images based on the content of the input first text; searching for, from among a plurality of images stored in a storage unit, a second image that is similar to at least one of the plurality of first images; and displaying the second image as the search result.

Description

Image search method, image search program, and image search device

[0001] The present invention relates to an image search method, an image search program, and an image search device.

[0002] When a user wants to search for an image capturing a desired subject or scene from a large amount of images, an image search system has been proposed in which a user prepares a search source image in advance and searches for images similar to the search source image (for example, Patent Document 1).

[0003] It is desired to easily find an image capturing a desired subject or scene.

[0004] Japanese Unexamined Patent Application Publication No. 2011-90476

[0005] According to an aspect of the first disclosure, an image search method is executed by a computer, the method comprising: receiving an input of a first text; generating a plurality of first images based on content of the input first text; searching for a second image similar to at least one image among the plurality of first images from a plurality of images stored in a storage unit; and displaying the second image as a search result.

[0006] According to an aspect of the second disclosure, an image search program causes a computer to execute a process comprising: receiving an input of text; generating a plurality of first images representing content of the input text; searching for a second image similar to one image among the plurality of first images from a plurality of images stored in a storage unit; and displaying the second image as a search result.

[0007] According to an aspect of the third disclosure, an image search device comprises: a reception unit that receives an input of text; a generation unit that generates a plurality of first images representing content of the input text; a search unit that searches for a second image similar to one image among the plurality of first images from a plurality of images stored in a storage unit; and a display control unit that causes a display unit to display the second image as a search result.

[0008] Furthermore, the configuration of the embodiments described later may be modified as appropriate, and at least a part of it may be replaced with other components. Moreover, the configuration elements whose arrangement is not particularly limited may be arranged in positions that can achieve their function, not limited to the arrangement disclosed in the embodiments.

[0009] Figure 1 is a block diagram showing the configuration of the image search system according to the first embodiment. Figure 2(A) is a diagram showing the hardware configuration of the user terminal, and Figure 2(B) is a diagram showing the hardware configuration of the service server. Figure 3 is a functional block diagram of the service server. Figure 4 is a flowchart showing an example of the processing of the service server. Figures 5(A) and 5(B) are diagrams showing examples of screens displayed on the user terminal. Figure 6 is a diagram showing an example of an image stored in the image DB. Figure 7 is a diagram showing an example of displaying an image generated by the image generation unit. Figure 8 is a diagram showing an example of displaying the image search results. Figure 9 is a functional block diagram of the service server according to the second embodiment. Figure 10(A) is an example of a histogram showing the aggregated results of composition recognition, Figure 10(B) is an example of a histogram showing the aggregated results of time period recognition, and Figure 10(C) is an example of a histogram showing the aggregated results of weather recognition. Figure 11 is a diagram for explaining the relationship between the image generation text generated by the text query reception unit and the image generated by the image generation unit. Figure 12 is a diagram showing an example of displaying an image generated by the image generation unit. Figure 13 is a functional block diagram of the service server according to the third embodiment. Figure 14 is a diagram illustrating tagging. Figure 15(A) is a diagram showing an example of a data table stored in the tag DB, and Figure 15(B) is a diagram showing another example of a data table stored in the tag DB. Figure 16 is a functional block diagram of the service server according to the fourth embodiment. Figure 17 is a diagram showing an example of displaying image search results.

[0010] 《First Embodiment》 The image search system SYS according to the first embodiment will be described in detail below with reference to Figures 1 to 8. Figure 1 shows the configuration of the image search system SYS in a block diagram.

[0011] The image search system SYS is a system for searching for images that a user desires from among the images that the user has taken. As shown in Figure 1, the image search system SYS comprises a user terminal 10 and a service server 20.

[0012] The user terminal 10 and the service server 20 are connected via a network NW that includes public wireless LANs, the internet, and mobile phone networks.

[0013] User terminal 10 is a terminal used by user USR1 who uses the image search service provided by service server 20, and is, for example, a smartphone, tablet, notebook PC (Personal Computer), or desktop PC.

[0014] Figure 2(A) shows the hardware configuration of the user terminal 10. As shown in Figure 2(A), the user terminal 10 includes a CPU (Central Processing Unit) 190, ROM (Read Only Memory) 192, RAM (Random Access Memory) 194, a storage unit (in this case, an SSD (Solid State Drive) or HDD (Hard Disk Drive)) 196, a network interface 197, a display unit 193, an input unit 195, and a portable storage medium drive 199 capable of reading the portable storage medium 191.

[0015] Each component of the user terminal 10 is connected to the bus 198. The display unit 193 includes a liquid crystal display, and the input unit 195 includes a keyboard, mouse, touch panel, etc.

[0016] User USR1 can access the website provided by the service server 20 from a web browser installed on the user terminal 10 and store the images captured by the camera 50 in the storage area assigned to a predetermined account (user USR1's account). In other words, the service server 20 provides a storage service. Furthermore, user USR1 can search for the image they want from among the images stored in the storage area assigned to user USR1's account on the website provided by the service server 20.

[0017] The service server 20 is a server that provides storage services and image search services. Based on a request from the user terminal 10, the service server 20 stores images received from the user terminal 10 in the storage device provided by the service server 20, or searches for and presents the image requested by the user from the images stored in the storage device. Alternatively, the images received from the user terminal 10 may be stored in a storage server separate from the service server 20, and the service server 20 may search for and present the image requested by user USR1 from the images stored in the storage server.

[0018] Figure 2(B) shows the hardware configuration of the service server 20. As shown in Figure 2(B), the service server 20 includes a CPU 290, ROM 292, RAM 294, storage unit (in this case, SSD or HDD) 296, network interface 297, and a portable storage medium drive 299, etc. These components of the service server 20 are connected to a bus 298. In the service server 20, the CPU 290 executes programs (including image search programs) stored in the ROM 292 or storage unit 296, or programs (including image search programs) read from the portable storage medium 291 by the portable storage medium drive 299, thereby realizing the functions of each component shown in Figure 3. Note that the functions of each component in Figure 3 may be realized by integrated circuits such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays).

[0019] Figure 3 shows a functional block diagram of the service server 20. In the service server 20, the CPU 290 executes a program, and as shown in Figure 3, it functions as a text query receiving unit 21, an image generation unit 22, a display control unit 23, an image query receiving unit 24, and an image search unit 25. Figure 3 also illustrates the image DB (Database) 26, which serves as a storage unit and is stored in the storage unit 296 of the service server 20.

[0020] The text query reception unit 21 receives text input from user USR1 that represents the image they are looking for. The text query reception unit 21 outputs the input text to the image generation unit 22 as text for generating the image to be searched (image generation text).

[0021] The image generation unit 22 generates an image representing the content of the text received by the text query reception unit 21. The image generation unit 22 is, for example, an image generation AI (Artificial Intelligent). Examples of technologies for generating images from text (image generation AI) include Stable Diffusion and Midjourney.

[0022] The display control unit 23 controls the content of the screen displayed on the user terminal 10 (for example, a website for using an image search service). The display control unit 23, for example, displays an input field on the screen for the user to enter text indicating the content of the image they are looking for, or displays an image generated by the image generation unit 22 on the screen. The display control unit 23 also displays the image search results from the image search unit 25, which will be described later, on the screen.

[0023] The image query receiving unit 24 accepts the selection of an image to be used as the search source from among the images generated by the image generation unit 22.

[0024] The image search unit 25 searches the image database 26, which stores images taken by user USR1, for images similar to the source image received by the image query reception unit 24.

[0025] (Processing by Service Server 20) Next, we will explain in detail the processing performed by the service server 20.

[0026] Figure 4 is a flowchart illustrating an example of the processing performed by the service server 20. The processing shown in Figure 4 begins when, for example, the screen shown in Figure 5(A) is displayed on the user terminal 10. The following describes the case where user USR1 is searching for an image of their pet dog with flowers. It is assumed that the image database 26 contains multiple images, as shown in Figure 6.

[0027] In the process shown in Figure 4, the text query receiver 21 first waits until text is entered into the input field EF1 on the screen shown in Figure 5(A) (step S11 / NO). For example, when user USR1 enters the text "Image of a dog with flowers" into the input field EF1 as shown in Figure 5(B), the text query receiver 21 determines that text has been entered. At this time, the text query receiver 21 retrieves the text from the input field EF1.

[0028] When text is entered (step S11 / YES), the image generation unit 22 determines whether or not it has received an image generation instruction (step S13). The image generation unit 22 repeats the process in step S13 until it receives an image generation instruction (step S13 / NO). For example, in Figure 5(B), the image generation unit 22 determines that it has received an image generation instruction when the button BTN1 that instructs image generation is clicked.

[0029] When an image generation instruction is received (step S13 / YES), the image generation unit 22 generates an image corresponding to the text entered in the input field EF1 (step S15), and the display control unit 23 displays the generated image (step S17). For example, as shown in Figure 7, the image generation unit 22 generates multiple images including a dog and a flower. The display control unit 23 displays the generated image in the display field DF1. Step S15 is the process of generating the image to be used as the search source when searching for an image from among multiple images stored in the image DB 26, and step S17 is the process of allowing user USR1 to confirm the generated image. In Figure 7, four images are generated, but the number of images generated by the image generation unit 22 is not limited to four. The number of images generated by the image generation unit 22 may be three or less, or five or more. Alternatively, the number of images to be generated may be specified by user USR1.

[0030] Next, the image query reception unit 24 determines whether or not one image has been selected from among the multiple generated images displayed in the display field DF1 (step S19). For example, the image query reception unit 24 determines that an image has been selected when one image is clicked from among the multiple generated images displayed in the display field DF1. Although not shown, the display control unit 23 may, for example, clearly indicate the selected image with a thick border, or display a check mark above the selected image.

[0031] If no generated image is selected (step S19 / NO), the text query reception unit 21 determines whether or not it has received an image regeneration instruction from user USR1 (step S21). The text query reception unit 21 determines that it has received an image regeneration instruction, for example, when the text entered in input field EF1 is changed and the button BTN1 that instructs image generation is clicked. The text query reception unit 21 also determines that it has received an image regeneration instruction if the button BTN1 that instructs image generation is clicked while the text entered in input field EF1 has not been changed. In this embodiment, user USR1 can check the image generated by the text entered by user USR1 and determine whether or not the generated image can be adopted as the image to be searched.

[0032] If no instruction to regenerate the image has been received (step S21 / NO), the process returns to step S19. On the other hand, if an instruction to regenerate the image has been received (step S21 / YES), the process returns to step S15, and the image generation unit 22 generates an image corresponding to the text entered in input field EF1. User USR1 can repeat the image generation and confirmation process until an image similar to the one the user is looking for is generated.

[0033] On the other hand, when one image is selected from among the multiple generated images displayed in the display field DF1 (step S19 / YES), the image search unit 25 determines whether or not it has received an image search instruction (step S23).

[0034] The image search unit 25 determines that it has received an image search instruction when, for example, the button BTN2 that instructs an image search on the screen shown in Figure 7 is clicked. If no image search instruction has been received (step S23 / NO), the process returns to step S19.

[0035] On the other hand, when an image search instruction is received (step S23 / YES), the image search unit 25 searches for images similar to the image selected in step S19 from among the images stored in the image DB 26 (step S25). For example, suppose the upper left image is selected from the generated images displayed in the display field DF1 shown in Figure 8, and the button BTN2 is clicked. In this case, the image search unit 25 calculates the similarity by comparing, for example, the feature quantities of the selected generated image with the feature quantities of each image stored in the image DB 26 shown in Figure 6, and extracts images whose similarity is, for example, a predetermined value or higher. Cosine similarity can be used as a method for calculating the similarity.

[0036] The display control unit 23 displays images with a similarity score equal to or greater than a predetermined value as search results (step S27), and terminates the process shown in Figure 4. The display control unit 23 displays the search results in the display field DF2, for example, as shown in Figure 8. The display control unit 23 may display the extracted images in descending order of similarity, or in descending order of similarity. If no images with a similarity score equal to or greater than a predetermined value exist, the image search unit 25 may extract a predetermined number of images as search results in descending order of similarity. If the user USR1 finds an image that matches their request among the images displayed as search results, they select the image and, for example, download it.

[0037] If user USR1 does not find the image they were looking for among the images displayed as search results, they can perform the process shown in Figure 4 again. That is, user USR1 can generate a search source image again from the text and search for images similar to the generated image. In this way, the service server 20 can extract an image from the images stored in the image DB 26 that is similar to the image in user USR1's mind and present it to user USR1. The display control unit 23 may also choose not to display any images other than the selected image once an image has been selected.

[0038] As described in detail above, according to the first embodiment, the service server 20 includes a text query reception unit 21 that accepts text input, an image generation unit 22 that generates an image representing the content of the input text, an image search unit 25 that searches for an image similar to the generated image from among a plurality of images stored in the image DB 26, and a display control unit 23 that displays the search results on the screen. In a system where a user inputs text to search for an image in a database, for example, an image similar to the content of the text is extracted based on the similarity between the tags attached to the image and the input text. Here, there are various algorithms for attaching tags to images, and the tags attached to the same image may differ depending on the algorithm. Therefore, even if an image is the image the user is looking for, if the text entered by the user has a low similarity to the tags attached to that image, the image the user is looking for may not be extracted. Also, it may be difficult to express the image the user is looking for using only text, and the image the user is looking for may not be found in the database. On the other hand, in this embodiment, by performing image generation as an intermediate process from text, it is possible to search for an image based on the feature quantities of the generated image and the feature quantities of the images stored in the database. Therefore, tagging is unnecessary for images stored in the database, and images similar to the generated image can be extracted regardless of the tags attached to the image. This increases the likelihood that the user will find the image they are looking for. In addition, in this embodiment, the image generation unit 22 generates an image according to the text entered by the user, so the user does not need to prepare an image to search for. This improves user convenience.

[0039] Furthermore, in the first embodiment, the display control unit 23 displays multiple generated images, and the image query reception unit 24 accepts the selection of one image from among the displayed generated images. This allows user USR1 to check the images generated from the text entered by user USR1 and determine whether or not the generated images can be used as the image to be searched. In addition, user USR1 can select an image from among the generated images that is similar to an image in user USR1's mind as the image to be searched.

[0040] 《Second Embodiment》 In the first embodiment, the text query receiving unit 21 output the text entered by user USR1 to the image generation unit 22 as text for generating the image to be searched (image generation text), but it is not limited to this. In the second embodiment, the text query receiving unit 21 outputs not only the text entered by user USR1, but also text created based on the text entered by user USR1 and text representing the content of the image stored in the image DB 26, as image generation text to the image generation unit 22.

[0041] Figure 9 is a functional block diagram of the service server 20A according to the second embodiment. The service server 20A differs from the service server 20 according to the first embodiment in that it includes an image analysis unit 27 and the processing performed by the text query reception unit 21. Figure 9 also shows the analysis DB 28 stored in the storage unit 296, etc., of the service server 20A.

[0042] The image analysis unit 27 recognizes pre-set elements in each image. These pre-set elements represent the content of the image (the scene in which the image was taken), and are not limited to, but include, for example, the composition of the image, the time of day the image was taken, and the weather conditions when the image was taken. For example, the main subject of the image (people, animals, vehicles, etc.) may be set as an element representing the content of the image.

[0043] The image analysis unit 27 has previously learned in advance, for example, a dataset for recognizing composition, a dataset for recognizing time periods, and a dataset for recognizing weather, as training data. Based on the learning results of the dataset, the image analysis unit 27 recognizes the composition of an image, the time period when the image was captured, and the weather at the time the image was captured.

[0044] The image analysis unit 27 aggregates recognition results of predetermined elements, and creates a histogram for the preset elements. For example, the image analysis unit 27 creates a histogram for each of the image composition, the time period when the image was captured, and the weather at the time the image was captured.

[0045] FIG. 10A is an example of a histogram showing aggregated results of composition recognition results, FIG. 10B is an example of a histogram showing aggregated results of time period recognition results, and FIG. 10C is an example of a histogram showing aggregated results of weather recognition results. In FIG. 10A, "Center", "Curved", and the like are texts representing the content of the image (the composition of the image). In the example shown in FIG. 10A, among the images stored in the image DB 26, the composition in which the subject is located at the center is most frequently adopted, and the occurrence frequency of "Center" is the highest.

[0046] In FIG. 10B, "Night", "Morning", and the like are texts representing the content of the image (the time period when the image was captured). In the example shown in FIG. 10B, among the images stored in the image DB 26, the number of images captured at night is the largest, and the occurrence frequency of "Night" is the highest.

[0047] In FIG. 10C, "Sunny", "Cloudy", and the like are texts representing the content of the image (the weather at the time the image was captured). In the example shown in FIG. 10C, among the images stored in the image DB 26, the number of images captured in sunny weather is the largest, and the occurrence frequency of "Sunny" is the highest.

[0048] The aggregated results of the recognition results of each element obtained by the image analysis unit 27 are stored in the analysis DB 28.

[0049] The text query reception unit 21 uses the text received from user USR1 and the aggregated recognition results stored in the analysis DB 28 to generate text (image generation text) that represents the content of the image to be generated by the image generation unit 22. The text query reception unit 21 also uses the text received from user USR1 as image generation text.

[0050] Figure 11 is a diagram illustrating the relationship between the image generation text generated by the text query reception unit 21 and the image generated by the image generation unit 22.

[0051] For example, the text query receiver 21 generates, in addition to the text TX1 entered in the input field EF1, image generation text TX2 which combines the entered text with the most frequently used composition (Center) in images taken by user USR1, image generation text TX3 which combines the entered text with the most frequently taken time of day (Night) in images taken by user USR1, and image generation text TX4 which combines the entered text with the most frequently taken weather (Sunny) in images taken by user USR1. Image generation text TX2 is, for example, "an image with a dog in the center with flowers", image generation text TX3 is, for example, "an image with a dog at night with flowers", and image generation text TX4 is, for example, "an image with a dog on a sunny day with flowers".

[0052] The image generation unit 22 generates image GIM1 based on text TX1, image GIM2 based on image generation text TX2, image GIM3 based on image generation text TX3, and image GIM4 based on image generation text TX4. The generated images are displayed in the display field DF1, as in the first embodiment (see Figure 7).

[0053] In this way, by generating images based on text (center, night, clear) derived from the recognition results of images stored in the image DB26, and adding this text to the text entered by user USR1, it becomes possible to generate a variety of images that reflect the user's shooting tendencies and preferences. This increases the likelihood that images similar to the image the user is looking for will be generated as the source image for the search, thus increasing the likelihood that the user will find the image they are looking for.

[0054] The image generation unit 22 may generate multiple images for each image generation text, and the display control unit 23 may display multiple images for each element (composition, time of day, weather) as shown in Figure 12. In Figure 12, the baseline column displays images generated based on the text entered by user USR1 (text entered by user USR1 with no other elements added). For example, the text query reception unit 21 may generate multiple image generation texts for each element. For example, the text query reception unit 21 may generate multiple image generation texts using the entered text and texts whose occurrence frequency for a given time of day is above a predetermined value. In this case, the text query reception unit 21 may generate images such as "image of a dog with flowers at night," "image of a dog with flowers during the day," and "image of a dog with flowers in the morning" as image generation texts.

[0055] Furthermore, the text query receiver 21 may, for example, generate image generation text by combining two or more elements with the input text. For example, if the text query receiver 21 receives input from user USR1, "an image of a dog with flowers," it may combine the text "an image of a dog with flowers" with composition and time of day to generate image generation text such as "an image of a dog in the center with flowers at night."

[0056] (Modification) When text is entered, the text query receiver 21 may generate image generation text by replacing words in the entered text with synonyms or related words. For example, if the entered text contains "book", the image generation text may be generated by replacing "book" with "publication", "book," "book," etc. The text query receiver 21 may also generate image generation text by replacing words in the entered text with semantically related words. For example, if the entered text contains "book", the image generation text may be generated by replacing "book" with "novel", "comic", "textbook," etc.

[0057] Furthermore, the text query receiver 21 may normalize the input text and generate image generation text. For example, if the input text contains the half-width word "programming", the image generation text may be generated with the half-width word "programming" replaced by the full-width word "programming". Also, if the input text contains spelling mistakes or typos, the text query receiver 21 may generate image generation text with the spelling mistakes or typos corrected.

[0058] By doing so, the image generation unit 22 can generate a variety of images, increasing the likelihood that an image similar to the one requested by user USR1 will be generated as the source image for the search.

[0059] <Third Embodiment> The text query receiving unit 21 may assist user USR1 in text input by utilizing tags attached to images in the database. That is, tags attached to images in the database may be used to assist in text input.

[0060] Figure 13 is a functional block diagram of the service server 20B according to the third embodiment. The service server 20B differs from the service server 20 according to the first embodiment in that it includes a tagging unit 29 and the processing performed by the text query receiving unit 21. Figure 13 also shows the tag DB 30 stored in the storage unit 296, etc., of the service server 20B.

[0061] When user USR1 selects an image from the images displayed as search results, the tagging unit 29 performs tagging on the selected image. For example, suppose the user selects the image shown in Figure 14. In this case, the tagging unit 29 performs image captioning using, for example, CoCa (Contrastive Captioners are Image Text Foundation Models (2022)). Other image captioning methods may also be used. Suppose that as a result of image captioning, the image in Figure 14 is given the caption "A photograph of a white dog sitting in front of flowers." The tagging unit 29 uses natural language processing to extract words of specific parts of speech as tags from the caption "A photograph of a white dog sitting in front of flowers." For example, from the caption "A photograph of a white dog sitting in front of flowers," the tags extracted for the image in Figure 14 are "white," "dog," "flower," "in front," and "sitting." The tagging unit 29 stores the extracted tags in the tag DB 30.

[0062] Figure 15(A) shows an example of a data table stored in the tag DB 30. The data table shown in Figure 15(A) includes fields for tags and extraction count. The tag field stores the words extracted as tags. The extraction count field stores the number of times the word stored in the tag field has been extracted. For example, if five images are tagged with "white", then "5" will be stored in the extraction count field. Each time user USR1 selects an image from the images displayed as search results, the data table in the tag DB 30 is updated.

[0063] When user USR1 enters text into input field EF1 shown in Figure 5(A), the text query receiver 21 suggests words to add to the currently entered text based on the tags stored in the tag database 30. For example, the text query receiver 21 suggests words registered as tags in the tag database 30 as candidates for text to be entered into input field EF1, in order from the words with the most extractions. For example, when user USR1 places the cursor in input field EF1, it suggests "dog," which is the word with the most extractions among the words registered as tags in the tag database 30.

[0064] As shown in Figure 15(B), a related tags field may be provided, and each tag may be associated with and registered with words (tags) that were included in the same caption. In this case, for example, a predetermined number of tags may be registered in the related tags field in order of frequency of occurrence. When user USR1 enters a word registered in the tag field of the tag DB 30 into the input field EF1, the text query receiver 21 suggests words stored in the related tags field as candidate text to be entered into input field EF1. For example, if user USR1 enters "white" into input field EF1, the text query receiver 21 suggests "dog," "center," and "walking" as candidate text to be entered into input field EF1. As a result, words that match user USR1's shooting tendencies are suggested, increasing the likelihood that text that can generate an image close to the image user USR1 is looking for will be entered.

[0065] Furthermore, for words that are extracted frequently, the text query receiving unit 21 may determine that they are images that user USR1 potentially likes, and generate image generation text by adding the words that are extracted frequently to the input text.

[0066] 《Fourth Embodiment》 Figure 16 is a functional block diagram of the service server 20C according to the fourth embodiment. In the fourth embodiment, the image generation unit 22 differs from the first embodiment in that, when generating the image to be searched, it reflects the trends of the images stored in the image DB 26 in the generated image.

[0067] Specifically, the image generation unit 22 customizes (fine-tunes) the generated image according to the images stored in the image database 26. For example, as shown in Figure 6, even though the image database 26 only contains images of a breed of dog called a Bichon Frise, the subject of the image generated by the image generation unit 22 is a breed of dog other than a Bichon Frise, as shown in Figure 7. In this case, the image generation unit 22 generates an image so that the subject is a Bichon Frise. Specifically, the AI ​​model used by the image generation unit 22 when generating an image according to the image generation text is corrected based on the images stored in the image database 26. As a result, the image generation unit 22 generates an image that reflects the images stored in the image database 26.

[0068] Furthermore, for example, suppose the image database 26 contains only images of basketball games, and the text "A player in a white uniform is jumping" is input as the text for image generation. In this case, suppose the image generated by the image generation unit 22 using a predetermined AI model is an image of a player in a white uniform jumping in a sport other than basketball (baseball, American football, tennis, etc.). In this case, the AI ​​model used by the image generation unit 22 when generating the image is corrected with the basketball game images stored in the image database 26. As a result, for example, the image generation unit 22 will generate an image of a player in a white uniform jumping in a basketball game.

[0069] In this way, by customizing the generated images to match the images stored in the image database 26, it is possible to present user USR1 with images that closely match the image USR1 is looking for as candidate images in the search source.

[0070] In the first to fourth embodiments described above, the user searches for desired images from images taken by user USR1 stored on a cloud server, but the invention is not limited to this. For example, an image search program may be installed on the user terminal 10 so that it can search for images desired by user USR1 from images taken by user USR1 stored on the user terminal 10. Alternatively, user USR1 may connect to the service server 20 using a dedicated application installed on the user terminal 10, and store images taken with the camera 50 in a storage area assigned to a predetermined account (user USR1's account) on the service server 20, or search for images desired by the user. In this case, the service server 20 functions as a storage unit.

[0071] Furthermore, in the first to fourth embodiments described above, if the number of images stored in the image DB 26 is less than a predetermined number (for example, 50), the image search unit 25 may search for an image that represents the content of the text from among the images stored in the image DB 26 based on the text received by the text query reception unit 21, without performing image generation by the image generation unit 22 (for example, cross-modal image search). This is because, when the number of images stored in the image DB 26 is less than a predetermined number, there is a high probability that the image requested by user USR1 can be extracted even with a text search. Note that cross-modal image search is a method of searching for an image based on the features of the text query and the features of the image in the database, but instead of cross-modal image search, a method may be adopted in which the text of the tags (or captions) attached to the image is made into features, and the image is searched based on the features of the text query and the features of the tags, etc., attached to the image.

[0072] On the other hand, if the number of images stored in the image DB 26 is greater than or equal to a predetermined number (for example, 50), then, as described in the first to fourth embodiments, the image generation unit 22 may be used to generate images, and an image search may be performed based on the generated images.

[0073] Furthermore, in the first to fourth embodiments described above, the image query receiving unit 24 may accept the selection of multiple images from among the images generated by the image generation unit 22. In this case, the image search unit 25 searches the image DB 26 for images similar to each of the selected multiple images. At this time, the display control unit 23 only needs to display the search results for each of the selected multiple images. Alternatively, for example, as shown in Figure 17, the search results for each selected image may be displayed together with the selected image. This allows user USR1 to see what kinds of images are found to be similar to the image used as the source image. Also, since images similar to each of the multiple source images are displayed as search results, the likelihood of extracting the image that user USR1 is looking for increases.

[0074] Furthermore, in the first to fourth embodiments described above, the image search unit 25 searches the images stored in the image database 26 using an image selected from the images generated by the image generation unit 22 as the source image, but it is not limited to this. The image search unit 25 may, for example, determine one of the images generated by the image generation unit 22 as the source image and search the images stored in the image database 26. In other words, the service server 20 may perform a series of processes after the user inputs text (generation of the source image, selection (determination) of the source image, searching for images similar to the selected source image, and displaying the search results). The image search unit 25 may, for example, perform image captioning on the images generated by the image generation unit 22 and select the image with the highest similarity between the caption attached to each image and the text entered by user USR1 as the source image.

[0075] Furthermore, the images stored in service servers 20-20C are not limited to images taken by user USR1. The image database 26 of service servers 20-20C may store images taken by photographers other than user USR1. For example, service servers 20-20C may be servers that provide an image sales service. In this case, user USR1 can search for the image they want to purchase from the images stored in the image database 26, and if the image is found in the search results, they can download the image after paying a predetermined amount. The images stored in service servers 20-20C may not be photographs but illustrations. That is, service servers 20-30C may be servers that provide an illustration sales service.

[0076] Furthermore, for example, service servers 20 to 20C may be servers that provide a search service for figurative trademarks. Also, service servers 20 to 20C may be servers that provide medical images. For example, image DB 26 stores X-ray images and endoscopic images of cases, and can be used by doctors to search for images similar to the image being read when interpreting medical images. Also, service servers 20 to 20C may be servers that provide diagnostic services at inspection sites such as waterways and tunnels. For example, image DB 26 stores images that indicate conditions requiring repair or maintenance, and inspectors can generate images by describing the condition of the waterway or tunnel they have confirmed at the inspection site in text, and then search for images similar to the generated image to determine whether or not repair or maintenance is necessary for the waterway or tunnel.

[0077] Furthermore, although the first to fourth embodiments described above describe the search for still images, the first to fourth embodiments can also be applied to the search for videos. In this case, the image generation unit 22 generates a video corresponding to the text input by the user USR1, and the image search unit 25 searches the image DB 26 for a video similar to the video selected from the videos generated by the image generation unit 22.

[0078] Furthermore, the first to fourth embodiments described above can be freely combined. That is, the service server may have all of the functions described in the first to fourth embodiments, or it may have two or more of the functions described in the first to fourth embodiments.

[0079] The above processing functions can be implemented by a computer. In this case, a program describing the processing content of the functions that the processing unit (CPU) should have is provided. By executing this program on the computer, the above processing functions are implemented on the computer. The program describing the processing content can be recorded on a computer-readable recording medium (except for carrier waves).

[0080] When distributing a program, it may be sold in the form of a portable storage medium such as a DVD (Digital Versatile Disc) or CD-ROM (Compact Disc Read Only Memory) on which the program is recorded. Alternatively, the program can be stored in the storage device of a server computer and transferred from the server computer to other computers via a network.

[0081] A computer executing a program stores the program, for example, on a portable storage medium or transferred from a server computer, in its own memory. The computer then reads the program from its memory and executes the processing according to the program. Alternatively, the computer can directly read the program from the portable storage medium and execute the processing according to that program. Furthermore, the computer can sequentially execute the processing according to the program received each time it is transferred from a server computer.

[0082] The embodiments described above are preferred examples of the present invention and can be combined as appropriate. However, the invention is not limited thereto, and various modifications are possible without departing from the spirit of the invention.

[0083] SYS Image Search System 10 User Terminal 20 Service Server 21 Text Query Reception Unit 22 Image Generation Unit 23 Display Control Unit 24 Image Query Reception Unit 25 Image Search Unit 26 Image Database 27 Image Analysis Unit 29 Tagging Unit

Claims

1. An image search method in which a computer performs the following steps: receiving input of a first text; generating a plurality of first images based on the content of the input first text; searching for a second image similar to at least one of the plurality of first images from among a plurality of images stored in a memory unit; and displaying the second image as a search result.

2. The image search method according to claim 1, wherein the computer performs the following: displaying a plurality of first images and receiving the selection of at least one image from among the displayed plurality of first images.

3. The image retrieval method according to claim 1 or 2, wherein the computer performs the following steps to generate the first image: acquiring a third text from among the second texts indicating the contents of the plurality of images stored in the memory unit, the computer generates the plurality of first images based on the content obtained by adding the third text to the input first text.

4. The image search method according to claim 3, wherein the computer performs the following actions: there are multiple third texts, and for each of the multiple first images is displayed.

5. The image retrieval method according to any one of claims 1 to 4, wherein if the number of images stored in the storage unit is equal to or greater than a first number, a second image similar to at least one image is searched for, and if the number of images stored in the storage unit is less than a first number, a third image based on the input first text is searched for from among the images stored in the storage unit.

6. An image search method according to any one of claims 1 to 5, comprising: accepting the selection of a second image displayed as a search result when the first text is entered; generating a sentence describing the selected second image and storing words contained in the sentence; and suggesting the words when the input of the first text is accepted.

7. The image search method according to claim 6, wherein the generation of the plurality of first images is based on the content of the input first text and the content obtained by adding frequently occurring words from the stored words to the input first text.

8. The image retrieval method according to any one of claims 1 to 7, wherein the generation of the plurality of first images includes correcting a model for generating an image representing the content of the input first text based on the plurality of images stored in the storage unit, and generating the plurality of first images based on the corrected model.

9. An image search program that causes a computer to perform a process including: receiving text input; generating a plurality of first images representing the content of the input text; searching for a second image similar to one of the plurality of first images from a plurality of images stored in a memory unit; and displaying the second image as a search result.

10. An image search device comprising: a reception unit for receiving text input; a generation unit for generating a plurality of first images representing the content of the input text; a search unit for a second image similar to one of the plurality of first images from among a plurality of images stored in a storage unit; and a display control unit for displaying the second image as a search result.

Images